System and method for autonomously cleaning piles

Abstract

A controller-implemented method for automatically controlling at least one of an autonomous or semi-autonomous machine (100) to clean at least one pile of material (172, 192) along a work surface. The method includes detecting the pile of material (172, 192) using a visual perception sensor (166), generating a plurality of pile visual detection signals, determining a position of the pile of material (172, 192) relative to the machine (100) based on the plurality of pile visual detection signals, and generating a machine control command to clean the pile of material (172, 192).

Description

Technical Field

[0001] The present patent disclosure relates generally to a method and steering system for an autonomous machine, and more particularly to a method and system for steering an autonomously operated machine to clear a pile of material. Background Art

[0002] Machines such as bulldozers, motor graders, wheel loaders, and the like are used to perform various tasks such as road clearing, excavating, loosening, moving, etc., different materials at a work site, such as those associated with mining, earth moving, and other industrial activities. Performing operations such as road clearing can result in the formation of a pile of material as the material is moved by a tool such as a blade.

[0003] The machine may be operated in an autonomous, semi-autonomous, or manual manner to perform these tasks in response to commands generated as part of the machine's work plan. Autonomously and semi-autonomously operated machines may provide certain advantages over manually operated machines. For example, autonomously and semi-autonomously operated machines may remain productive at all times regardless of human operators or environmental conditions. In addition, autonomous systems may allow operation in environments where a human operator is unsuitable or undesirable. Autonomous or semi-autonomous systems may also compensate for inexperienced human operators and the inefficiencies associated with repetitive tasks.

[0004] Autonomous or semi-autonomous systems may also be needed when performing complex tasks or in operating machines that require simultaneous manipulation of various controls. For example, an articulated machine such as a motor grader includes a front frame member, a rear frame member, and an articulated joint connecting the front and rear frame members, and a movable implement in the form of a blade. As in a motor grader, a machine having a wide axle wheelbase with two pairs of rear wheels and a pair of spaced-apart front wheels presents additional steering and operating challenges. Steering of a machine such as a motor grader, including control and movement of the articulated joint, and maintaining traction on the wheels while turning, faces a challenge that is further complicated by the operation of the movable implement while preferably maintaining a relatively constant load on the blade. Therefore, performing tasks that require manipulation of the machine, as well as operating the machine based on visual or blurred features such as clearing a pile of material on a worksite, can be particularly challenging.

[0005] U.S. Publication No. 2019 / 0106862A1 to Ono et al. discloses a control system for clearing a pile of materials by an autonomous machine with a work tool. The '862 publication uses mmWave sensors to locate the pile of materials and associates them with corresponding GPS locations. These GPS locations are stored at a base station. The autonomous machine then reads the GPS points from the base station and follows the GPS locations to clear the pile of materials. These systems require relatively expensive high-precision equipment. In addition, the operation of such systems may be limited by the use of GPS, which may provide unreliable results in cities, tunnels, etc.

[0006] The foregoing background discussion is intended only to aid the reader. It is not intended to limit the innovations described herein, nor is it intended to limit or extend the prior art discussed. Therefore, the foregoing discussion should not be taken to indicate that any particular element of an existing system is unsuitable for use with the innovations described herein, nor should it be taken to indicate that any element is necessary in implementing the innovations described herein. The implementation and application of the innovations described herein are defined by the appended claims. Summary of the invention

[0007] The present invention relates to a method for autonomously clearing a pile of materials along a working surface of a machine. The method includes detecting the pile of materials by a visual perception sensor connected to a controller, generating a plurality of visual detection signals of the pile of materials by the controller, determining a position of the pile of materials relative to the machine based on the plurality of visual detection signals of the pile of materials, and generating a plurality of machine control commands by the controller to clear the pile of materials.

[0008] The present invention also relates to a system for automatically controlling a machine to clean at least one pile of material along a work surface. The system includes a visual perception sensor and a controller. The visual perception sensor is mounted on the machine. The visual perception sensor is adapted to provide three-dimensional data and generate a plurality of pile detection signals. The controller is configured to receive the plurality of pile detection signals and determine a position of the pile relative to the machine. The controller is further configured to access a steering parameter of the machine and generate a machine control command to clean the pile of material. The machine control command includes at least one of a steering command, a propulsion command, and an implement control command.

[0009] The present invention also relates to a machine comprising a prime mover, a ground engaging work tool for engaging a work surface along a path, a visual perception sensor, and a controller. The machine is mounted with a visual perception sensor. The visual perception sensor is adapted to provide three-dimensional data and generate a plurality of pile detection signals. The controller is configured to receive the plurality of pile detection signals and determine a position of the pile relative to the machine. The controller is further configured to access a steering parameter of the machine and generate a machine control command to clear the pile. The machine control command comprises at least one of a steering command, a propulsion command, and a tool control command. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a side view of a machine incorporating a control arrangement according to aspects of the present invention.

[0011] Figure 2 is a schematic diagram of a system incorporating aspects of the control arrangement of the present invention.

[0012] Figure 3 is a flow chart of a control arrangement according to aspects of the present invention.

[0013] Figure 4 is a diagrammatic representation of an implementation of a system and control arrangement according to aspects of the present invention.

[0014] Figure 5 is a flow chart of an exemplary control arrangement according to aspects of the present invention.

[0015] Figure 6 is a flow chart of another exemplary control arrangement according to aspects of the present invention. DETAILED DESCRIPTION

[0016] Turning now to the accompanying drawings, Figure 1 An exemplary embodiment of a machine 100 in the form of a motor grader 101 is shown in FIG. Figure 1 1, a motor grader 101 is shown, but the term "machine" may refer to any machine that performs some type of operation in which a windrow may be formed in a work surface. For example, the machine 100 may alternatively be a front loader, a harvesting device, a snow blower, etc.

[0017] The motor grader 101 includes a main frame 102 supporting a ground engaging work implement 103. The main frame 102 has a rear frame portion 104 and a front frame portion 106. The rear frame portion 104 and the front frame portion 106 may be optionally articulated at an articulation joint 108, which includes a hinge 109. The main frame 102 is supported on a plurality of ground engaging members 110. In the illustrated embodiment, the ground engaging members 110 include a pair of front wheels 112 spaced apart from a plurality of rear wheels 114, 116 disposed in pairs along opposite sides of the rear frame portion 104. However, it will be appreciated that the ground engaging members 110 may include alternative arrangements, for example, the rear wheels 114, 116 may instead be track assemblies, as is known in the art.

[0018] The front frame portion 106 includes a front frame section 120 supported between a hinge 109 and a front ground engaging member 110, here a pair of front wheels 112 as shown. A ground engaging work implement 103 in the form of a blade assembly 122 is mounted along the front frame section 120 and can be used for grading. The blade assembly 122 includes a blade 124 and a linkage assembly 126 that allows the blade 124 to move to a variety of different positions relative to the front frame section 120. Although the exemplary blade 124 is shown as an attached implement, the motor grader 101 may include additional implements such as a plow, a scarifier, and a ripper.

[0019] Returning to the illustrated embodiment, an operator cab 128 may be supported along the front frame segment 120. The cab 128 may include, for example, a seat 130, a control console 136, and various operator controls, such as a steering mechanism 132, a speed throttle, or a control lever 134. An operator occupying the cab 128 controls various functions and movements of the motor grader 101, such as by using the steering mechanism 132 to set the direction of travel of the motor grader 101, or by using the control lever 134 to set the travel speed of the machine. It will be appreciated that the representation of various control mechanisms given herein is general and is meant to include all possible mechanisms or devices for transmitting operator commands to the machine, including, for example, so-called joystick operation. Although an operator cab 128 is shown in the illustrated embodiment, the inclusion of such a cab and associated seat, control mechanisms, and control console is optional, as the machine may alternatively be autonomous or semi-autonomous, i.e., the machine may be partially or completely controlled by a control system that does not require operation by an onboard human operator.

[0020] The rear frame section 104 includes a rear frame section 138 supported on a plurality of ground engaging members 110 along either side of the machine 100. In the illustrated embodiment, the ground engaging members 110 supporting the rear frame section 138 include two pairs of rear wheels 114, 116. Figure 1 104. Although the ground engaging member 110 may alternatively be coupled directly to the rear frame portion 104, in the illustrated embodiment, the pair of rear wheels 114, 116 are rotatably mounted on a tandem support 140, which itself is pivotally mounted along either side of the rear frame segment 138 at pivots 144. Thus, each rear wheel 114, 116 rotates and the tandem support 140 pivots about a corresponding axis. Those skilled in the art will appreciate that the ground engaging member 110 may include alternative or additional structure, such as a belt (not shown) disposed about the pair of rear wheels 114, 116.

[0021] To assist in steering the machine 100, in some embodiments, the pairs of wheels 114, 116 along either side of the machine 100 may be individually controlled such that the pairs of wheels 114, 116 along one side or the other of the machine 100 may be rotated at different speeds by the drive unit to facilitate steering the machine 100 along a desired path. For example, the wheels 114, 116 along one side of the machine 100 may be rotated at a different speed than the wheels 114, 116 along the opposite side of the machine 100. Figure 1 100) rotates faster to help turn the machine 100 or keep the machine 100 on track.

[0022] The machine 100 includes a prime mover 146 that may be of any suitable design. For example, the prime mover 146 may include an engine 148 adapted to propel the machine 100 through operation of the ground engaging members 110. The prime mover 146 may also be coupled to a hydraulic system 150. The hydraulic system 150 may include one or more pumps (not shown) to drive or actuate machine operations, such as steering of the ground engaging members 110 (e.g., wheels 114, 116) and operation of the linkage assembly 126 to control the position of the blade 124 relative to the frame 102.

[0023] A control module or electronic controller 160 is connected to the machine 100 and is arranged to receive information from various sensors on the machine 100, process the information, and issue commands to various components within the system during operation. Although the electronic controller 160 may be located remotely, in at least one embodiment, the electronic controller 160 may be disposed on the machine 100.

[0024] The electronic controller 160 can be any conventional design with hardware and software configured to perform calculations and send and receive appropriate signals to execute the disclosed logic. The controller 160 can include one or more controller units and can be configured to execute only the disclosed strategies, or to execute the disclosed strategies and other independent processes of the machine 100. The electronic controller 160 can be of any suitable structure and can include a processor (not shown) and a memory component (not shown). The processor can be a microprocessor or other processor known in the art. In some embodiments, the processor can be composed of multiple processors. In one example, the controller 160 includes a digital processor system, which includes a microprocessor circuit with data input and control output, and the microprocessor circuit operates according to computer-readable instructions stored on a computer-readable medium. Typically, the processor will have associated with it a long-term (non-volatile) memory for storing program instructions, and a short-term (volatile) memory for storing operands and results during processing (or generated by it).

[0025] The processor may execute instructions for the operation of the various systems of the machine 100. In alternative embodiments, hardwired circuits may be used in place of or in combination with software instructions to implement the desired operations. Such instructions may be read into or incorporated into a computer-readable medium, such as a memory component or provided externally to the processor. The memory component may include any form of computer-readable medium as described above. The memory component may include multiple memory components. Therefore, the embodiments are not limited to any specific combination of hardware circuitry and software.

[0026] The term "computer readable medium" as used herein refers to any medium or combination of media that participates in providing instructions to a processor for execution. Such media can take a variety of forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks. Volatile media include dynamic memory. Transmission media include coaxial cables, copper wire, and optical fiber.

[0027] Common forms of computer-readable media include, for example, floppy disks, diskettes, hard disks, magnetic tape or any other magnetic medium, CD-ROMs, or any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, and EPROM, Flash-EPROM, any other memory chip or cartridge, or any other medium from which a computer or processor can read.

[0028] refer to Figure 2 , a number of connections relevant to the present description are shown, but it is understood that a greater number of other connections may be present with respect to the controller 160. For example, the controller 160 may be connected to receive signals from a number of sensors, generally shown at 162, and control inputs, generally shown at 164. The sensors 162 may include, for example, sensors related to the relationship of the machine 100 to the surrounding environment, such as ground speed sensors, and sensors related to the operation or position of machine components, such as position sensors related to the position of the blade 124 or the position of the swash plate of the corresponding pump of the hydraulic drive system 150. The control inputs 164 may include, for example, the steering mechanism 132 and a speed throttle or control lever 134 that may be moved by an operator of the machine 100 to set a desired direction setting for the machine 100. The control signals relayed from the control inputs 164 to the controller 160 may be used in calculations along with other parameters, such as the speed of the engine 148, the ground speed of the machine 100, and the additional sensors 162, to produce a desired operation of the machine 100.

[0029] The windrow may be formed by a tool, such as a blade 124 that moves material from the work surface. According to one aspect of the invention, the machine 100 may operate autonomously or semi-autonomously to clear such windrows. To facilitate autonomous or semi-autonomous clearing of such windrows, the machine 100 is provided with at least one visual perception sensor 166 adapted to provide three-dimensional data. In at least one embodiment, the visual perception sensor 166 is disposed at a forward position on the front frame segment 120 so as to provide an unobstructed view of the work surface in front of the machine 100 when viewed from the direction of travel. However, it will be appreciated that the at least one visual sensor 166 may alternatively be located elsewhere / in another position, so long as the visual perception sensor 166 provides an appropriate view of the work surface from which the windrow is to be cleared.

[0030] The at least one visual perception sensor 166 may include any suitable visual perception sensor suitable for providing three-dimensional data, such as a stereo camera, a three-dimensional camera, a lidar sensor, etc. In addition, the at least one visual perception sensor 166 may include a plurality of such visual perception sensors 166 and may include a sensor suitable for providing two-dimensional data.

[0031] Reference Figure 2 , the visual perception sensor 166 observes the surrounding work surface 170, for example, located in front of the machine 100 to detect a pile of material 172. The pile detection signal is relayed to the electronic controller 160, which locates the pile of material 172 from the electronic controller. The electronic controller 160 generates control commands to instruct the machine 100 to clear the pile of material. As examples, such control commands may include one or more steering commands 174, one or more propulsion commands 176, and / or one or more implement control commands 178. The generated control commands then generate any necessary adjustments in the corresponding control system machine 100, such as the steering system 180, the propulsion system 182, and / or the blade control system 184, to properly steer or propel the machine 100 and / or adjust the position and arrangement of the blade 124. For example, the controller 160 may identify a desired speed of a pair of wheels 114, 116 of the propulsion and steering machine 100 based on the positioning of the stockpile 172 relative to the machine 100 and various inputs received by the controller 160 to control, for example, the displacement of a hydraulic pump, the displacement or speed of a hydraulic motor, the speed of the rear wheels 114, 116, the speed of the front wheels 112, the steering angle of the front wheels 112, and, if applicable, the articulation of the rear and front frame portions 104, 106.

[0032] It will be appreciated by those skilled in the art that features external to the stockpile may cause irregularities or noise in the three-dimensional data generated by the visual perception sensor 166. For example, in some applications, environmental features such as trees may cause irregularities or noise in the three-dimensional data. Therefore, noise reduction techniques or at least one visual perception sensor 166 may utilize noise reduction techniques or devices. In at least one embodiment, in addition to a three-dimensional camera, at least one visual perception sensor 166 may include a two-dimensional camera adapted to provide two-dimensional data. In such an arrangement, the two-dimensional camera may be physically separated from the three-dimensional camera, or may be installed with a three-dimensional camera or sensor, such as as part of a stereo camera or a lidar sensor. In such an arrangement, the two-dimensional camera of at least one visual perception sensor 166 may be used to initially identify the general location of the stockpile. It will be appreciated by those skilled in the art that a system for identifying stockpile using two-dimensional images may be developed through machine learning and the use of a large number of images. It should also be appreciated that identifying stockpile using two-dimensional data can help reduce the noise associated with the three-dimensional data, allowing the system to focus on the position data associated with the detected stockpile.

[0033] Industrial Applicability

[0034] The present invention may be applied to a machine 100 that may be used to clean a pile of material autonomously or semi-autonomously.

[0035] Go to Figure 3 , shows a method implemented by an exemplary controller according to the present invention. A depth map is obtained from the viewed image using at least one visual perception sensor 166 (stage 185). The stockpile is located in the depth map, if any (stage 186). A steering angle is then calculated based on the orientation of the stockpile (stage 187), and then a machine control command is generated to cause the machine 100 to clear the stockpile (stage 188). As the machine 100 is pushed and turned to clear the stockpile, the method continues, generating continuous machine control commands to clear the stockpile.

[0036] Go to Figure 4 , shows an exemplary arrangement for determining the location of a pile of material using a two-dimensional camera and a three-dimensional camera and determining a desired steering angle for clearing the pile of material. Figure 4Panel 190 of FIG. 1 shows a two-dimensional view of the work surface in front of the machine in which a pile of materials 192 can be seen. Panel 194 shows a corresponding top view of a three-dimensional depth map generated from a three-dimensional camera. The vertical axis of the depth map represents the distance forward from the camera, while the horizontal axis represents the lateral direction. Based on the data provided from the identification of the pile of materials 192 in the two-dimensional view, the three-dimensional depth map shown in panel 194 has been cleaned up after noise removal. The pile of materials 192 can be seen as a slightly shaded portion 196 on the depth map. Using a perspective transform, the coordinates of the pile of materials 192 from the two-dimensional view of panel 190 are also shown on the depth map of panel 194 as a box surrounded by dashed lines (generally identified as 210 in panel 194). Those skilled in the art will understand that the coordinates 210 of the two-dimensional view follow the shaded portion 196 identified on the depth map relatively closely.

[0037] From the depth map, the depth of the working surface is determined at an identifiable distance in front of the camera. In this illustration, depths of 3.5 meters, 6 meters, and 9 meters in front of the camera are shown in the cross-sections of the depth maps in panels 198, 200, and 202, respectively. In each cross-section of the depth map, a bold vertical line identifies the highest peak (204, 206, 208, respectively) of the stockpile 192 at the distance identified in front of the camera. The locations where the cross-sections were taken in panels 198, 200, 202 are shown in the depth map of panel 194 as horizontal lines at 3.5 meters, 6 meters, and 9 meters. Although the cross-sections shown are taken at 3.5 meters, 6 meters, and 9 meters, it should be understood that these distances are exemplary and that the cross-sections can be taken at alternative or additional distances from the front of the camera. The location of the peaks 204, 206, 208 of each corresponding cross section of the depth map (panels 198, 200, 202) and multiple other depths are used to identify the location of the bank 192 in the depth map (panel 194) having multiple circles. Curve 212 connects the identified peaks 204, 206, 208 of the bank 192 at different distances in the depth map of panel 194 to graphically illustrate the outline of the bank 192 within the depth map (panel 194).

[0038] Thus, with reference to the depth map of panel 194, the curve 212 connecting the locations of the identified peaks of the stockpile shows the determined location of the stockpile 192. The desired steering angle can then be determined based on the location of the stockpile 192 determined in the depth map (panel 194). In this embodiment, the desired steering angle is determined based on a tangent to the calculated location of the stockpile 192, as shown by the bold hatching 214 in panel 194. Those skilled in the art will appreciate that the steering angle is continuously updated as the machine 100 moves forward, and continuous images are provided by the two-dimensional and three-dimensional cameras. Those skilled in the art will appreciate that similar two-dimensional and three-dimensional information can be provided by a stereo camera or a lidar sensor.

[0039] Now go to Figure 5 , showing an embodiment of an exemplary method according to the present invention. At panel 220, the visual perception sensor inspects the work surface to obtain a two-dimensional image and a three-dimensional depth map (panel 222). It is then determined whether a pile of materials is detected (panel 224). If no pile of materials is detected, the visual perception sensor repeatedly continues to inspect the work surface (panel 220), obtain a two-dimensional image and a three-dimensional depth map (panel 222), and determine whether a pile of materials is detected (panel 224) until a pile of materials is detected at panel 224. Based on the two-dimensional image, the visual perception sensor generates a pile detection signal, from which the position of the detected pile of materials relative to the machine is determined (panel 226). The positioning of the pile of materials based on the two-dimensional image recognition is identified on the three-dimensional depth map (panel 228). This information transmission from the two-dimensional image to the depth map can be based on, for example, a perspective transformation.

[0040] In order to more specifically identify the location of the pile in the depth map, the depth of the pile is determined at a predetermined distance from the visual perception sensor, taking into account the transmitted two-dimensional image (panel 230). Figure 4 The depth is determined by looking at the cross-sections of the depth map illustrated in panel 194 of FIG. The location of the windrow peaks in each cross-section is identified in the depth map and connected by lines (panel 232) to more accurately identify the location of the windrow relative to the machine. The steering angle for clearing the windrow is then determined based on the identified location of the windrow in the depth map (panel 234).

[0041] Machine control commands are then generated at panel 236 based on the detected pile location and the control and steering parameters of the machine. Based on the machine control commands, the machine is then turned and propelled along the identified cleaning direction at panel 238. The system then continues to inspect the work surface at panel 220 to continuously detect and identify the location of the pile, determine the steering angle, generate and execute machine control commands, and adjust the steering direction and propulsion as needed.

[0042] exist Figure 6 Another embodiment of an exemplary method according to the present invention is shown in the flowchart of . At panel 320, one or more visual perception sensors inspect the work surface to obtain a two-dimensional image and a three-dimensional depth map. At panel 322, machine learning is used to detect piles in the resulting two-dimensional image. At panel 324, it is determined whether piles are detected in the two-dimensional image. Conversely, if piles are not detected, the system relies solely on data obtained from the three-dimensional perception sensor (panel 326). If piles are detected in the two-dimensional image, the piles from the two-dimensional image are transformed into a three-dimensional depth map to remove noise data in the three-dimensional depth map (panel 328).

[0043] To more specifically identify the location of the pile in the depth map, the depth of the pile is determined at regular intervals from the visual perception sensor (panel 330). Figure 4 The depth is determined by taking multiple cross sections of the depth map illustrated in panel 194 of FIG. The location of the windrow peak in each cross section is identified in the depth map and connected by lines (panel 332) to more accurately identify the location of the windrow relative to the machine. The steering angle for clearing the windrow is then determined based on the identified location of the windrow in the depth map (panel 334).

[0044] Machine control commands are then generated at panel 336 based on the detected pile location and the control and steering parameters of the machine. Based on the machine control commands, the machine is then turned and propelled along the identified cleaning direction at panel 338. The system then continues to repeatedly check the work surface at panel 320 to continuously detect and identify the location of the pile, determine the steering angle, generate and execute machine control commands, and adjust the steering direction and propulsion as needed.

[0045] The method and system of the present invention can be applied to machines used in clearing piles of materials. In this regard, the method and system are particularly useful for machines used in road clearing, such as motor graders.

[0046] Some of the disclosed methods and systems may be used to facilitate machines to remain productive at all times regardless of human operators or environmental conditions.

[0047] Furthermore, autonomous systems can allow operation in environments where human operators are unsuitable or undesirable. Autonomous or semi-autonomous systems can also compensate for inexperienced human operators and the inefficiencies associated with repetitive tasks.

[0048] Some of the disclosed methods and systems may be used for machines that need to perform complex tasks or tasks that require simultaneous manipulation of various controls. Some of the disclosed systems and methods may also be used for machines that present specific steering challenges, such as machines that include a front frame member, a rear frame member, and an articulated joint connecting the front frame member and the rear frame member, and a movable implement.

[0049] Some of the disclosed methods and systems may provide arrangements that may be adjusted to changing conditions by dynamically determining a desired steering path.

[0050] Some disclosed methods and systems may not require a separate control center because the detection, analysis, and generation of control commands are performed by onboard equipment and software.

[0051] Some methods and systems according to the present invention may provide an economical alternative to more complex arrangements that require more expensive mmWave sensors or high-precision GPS systems on machines. Additionally, some methods and systems may utilize hardware and software that already exists on the machine. Furthermore, because the disclosed methods and systems do not require GPS, the methods and systems are not subject to the limitations of GPS. For example, some disclosed methods and systems are particularly useful in positioning where GPS may provide inaccurate results, such as in cities or tunnels.

[0052] It is obvious to those skilled in the art that various modifications and variations can be made to the stockpile detection and cleaning system and process of the present invention. In view of the disclosure herein, other embodiments of the described methods and systems will be apparent to those skilled in the art. The description and examples of the present invention are considered to be exemplary only, and the true scope of the present invention is indicated by the appended claims and their equivalents. It is conceivable that other implementations of the present invention may differ from the foregoing examples in detail. All references to the present invention or its examples are intended to reference the specific examples discussed at that point and are not intended to imply any limitation on the scope of the present invention more generally. All language regarding the distinction and disparagement of certain features is intended to lack a preference for these features, but unless otherwise indicated, it is not completely excluded from the scope of the present invention.

[0053] Recitation of ranges of values ​​herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context.

[0054] Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Additionally, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or clearly contradicted by context.

Claims

1. A method of autonomously cleaning a pile of material (172, 192) along a work surface using a machine (100), the method comprising: observing the work surface via at least one visual perception sensor (166) coupled to a controller (160), the visual perception sensor (166) being adapted to provide a three-dimensional depth map of the work surface and a two-dimensional view of the work surface; determining whether a pile of material is detected in the two-dimensional view of the work surface (172, 192); In the event that no pile of material (172, 192) is detected in the two-dimensional view of the working surface, detecting the pile of material (172, 192) using only the three-dimensional depth map of the working surface; In the event that a pile of material is detected in the two-dimensional view of the work surface, transforming the pile of material from the two-dimensional view into a three-dimensional depth map; identifying a location of a stockpile peak for each of a plurality of cross sections of the three-dimensional depth map; connecting the stockpile peaks in the three-dimensional depth map by lines; Generate a plurality of visual detection signals of the pile of materials (172, 192) via the controller (160) using the three-dimensional depth map of the work surface or the two-dimensional view of the work surface according to whether the pile of materials (172, 192) is detected in the two-dimensional view of the work surface; determining a position of the pile (172, 192) relative to the machine (100) based on the plurality of pile visual detection signals; as well as Based on the determined position of the pile (172, 192) relative to the machine (100), a plurality of machine control commands are generated via the controller (160) to clear the pile (172, 192).

2. The method of claim 1, wherein the machine (100) is an autonomous machine, and wherein utilizing at least one visual perception sensor (166) comprises utilizing at least one of a stereo camera, a three-dimensional camera, and a lidar sensor coupled to the machine (100). 3 . The method of claim 1 , wherein generating machine control commands includes generating at least one steering command.

4. The method of any one of claims 1-3, wherein generating machine control commands further comprises generating at least one propulsion command.

5. The method of any one of claims 1-3, wherein generating machine control commands further comprises generating at least one implement control command.

6. The method of claim 1, wherein the pile detection signal indicates an orientation of the pile (172, 192) relative to a frame (102) of the visual perception sensor (166).

7. The method of claim 1, wherein: The method further includes: in response to the generated machine control command, pushing the machine (100) to a continuous machine position, and continuously detecting the pile (172, 192) using the visual perception sensor (166), generating a continuous plurality of pile detection signals, determining continuous positions of the pile (172, 192) relative to the machine (100), and generating continuous control commands to clean the pile (172, 192).

8. A system for performing the method of any one of claims 1 to 7 for automatically controlling a machine (100) to clean a pile of material (172, 192) along a work surface, the system comprising: a visual perception sensor system adapted to generate a plurality of pile detection signals, the visual perception sensor system adapted to provide a three-dimensional depth map of the work surface and a two-dimensional view of the work surface, the visual perception sensor system being mounted to the machine (100); and A controller (160) configured to: observing the work surface via the visual perception sensor system; determining whether a pile of material is detected in the two-dimensional view of the work surface (172, 192); In the event that no pile of material (172, 192) is detected in the two-dimensional view of the working surface, detecting the pile of material (172, 192) using only the three-dimensional depth map of the working surface; In the event that a pile of material is detected in the two-dimensional view of the work surface, transforming the pile of material from the two-dimensional view into a three-dimensional depth map; identifying a location of a stockpile peak for each of a plurality of cross sections of the three-dimensional depth map; connecting the stockpile peaks in the three-dimensional depth map by lines; receiving the plurality of material pile detection signals based on a three-dimensional depth map of the working surface or a two-dimensional view converted to a three-dimensional space according to whether material piles (172, 192) are detected in the two-dimensional view of the working surface; determining a position of the stockpile (172, 192) relative to the machine (100); accessing steering parameters of the machine (100); generating a machine control command to clear the pile (172, 192) based on the determined position of the pile (172, 192) relative to the machine (100) and the accessed steering parameter of the machine (100), the machine control command comprising at least one of a steering command, a propulsion command, and an implement control command, The visual perception sensor system includes at least a two-dimensional sensor for providing a two-dimensional view of the working surface and a three-dimensional sensor for providing a three-dimensional depth map of the working surface.

9. A machine (100) comprising: a frame (102) supported by ground engaging members (110); Prime mover (146); A ground engaging work tool (103) for engaging a work surface along a path; and The system of claim 8, wherein the controller (160) is disposed with the frame (102).

Images