Automatic steering using machine vision

Abstract

A machine vision based autosteering method receives a point cloud generated using a stereo camera. A train position is determined based on the point cloud, and a steering angle is generated based on the train position. The train center is detected using a Hough transform detection algorithm and a horizontal projection of the point cloud.

Description

[Technical Field]

[0001] The subject matter of this disclosure relates generally to machine automation, and more particularly to automatic steering using machine vision. [Background technology]

[0002] Agricultural machinery typically requires an experienced operator to perform agricultural tasks. Tasks such as harvesting fruit in vineyards or orchards require the operator to navigate the vehicle while maintaining a parallel alignment with the rows of plants. While some agricultural tasks can be automated using Global Navigation Satellite System (GNSS) hardware, the location of objects cannot be determined based on GNSS information alone. For example, the boundaries of an orchard or vineyard can be known, but the actual location of the plants growing within the boundaries cannot be determined. Furthermore, plant growth cannot be predicted because branches grow in various directions. A method is needed to automate agricultural equipment to perform tasks related to plants when the plant's location is unknown. Summary of the Invention [Problem to be solved by the invention]

[0003] A method for automatically steering an agricultural machine includes receiving point cloud data from a camera at a machine control device. A position of a row of plants is determined based on the point cloud data. A steering angle is generated for the agricultural machine based on the row position. In one embodiment, the determining includes determining a row centerline based on a Hough transform detection algorithm. In one embodiment, the Hough transform detection algorithm determines the row centerline based on a horizontal projection of the point cloud data. In one embodiment, the horizontal projection of the point cloud is based on points of the point cloud that are above ground level, found using a random sample consensus algorithm. The determination of the row centerline is further based on a frontal projection of the point cloud. The steering angle can be further generated based on a heading error, which is the angle between the longitudinal axis of the agricultural machine and a midline, and in some embodiments, can be generated based on an X-track, which is the distance from the midline to the center of the rear axle of the agricultural machine. In one embodiment, the midline is the centerline of the row. In another embodiment, the midline is a centerline located between a row and an adjacent row that is parallel to but offset from the row. Also disclosed is an apparatus having a steering control device, a steering actuation device, a camera, and a machine control device, the apparatus being configured to operate for automatic steering of an agricultural machine. Also disclosed is a computer-readable medium for automatic steering of an agricultural machine. [Brief explanation of the drawings]

[0004] [Figure 1] Figure 1 shows a plot of land with rows of plants on either side of the walkway. [Figure 2] Figure 2 shows the tractor moving down the path. [Figure 3] Figure 3 shows a harvester moving along a row of plants. [Figure 4] Figure 4(A) shows the angle and height parameters of the camera installed on the tractor, Figure 4(B) shows the angle and centerline-lens distance parameters of the camera installed on the tractor, and Figure 4(C) shows the angle parameters of the camera installed on the tractor. [Figure 5]Figure 5(A) shows the angle and height parameters of the camera installed on the harvester, Figure 5(B) shows the angle and centerline-lens distance parameters of the camera installed on the harvester, and Figure 5(C) shows the angle parameters of the camera installed on the harvester. [Figure 6] Figure 6 shows the heading and trajectory parameters of the tractor navigating the aisle. [Figure 7] Figure 7 shows the heading and trajectory parameters of the harvester as it moves along the row. [Figure 8] Figure 8 shows the point cloud of a row of plants. [Figure 9] FIG. 9 shows the point cloud of a passage bounded by columns on either side. [Figure 10] Figure 10 shows the local coordinate system of the camera. [Figure 11] Figure 11 shows the horizontal projection of the point cloud for row detection. [Figure 12] Figure 12 shows the horizontal projection of the point cloud for aisle detection. [Figure 13] FIG. 13 shows a point cloud front projection for use with a harvester. [Figure 14] FIG. 14 shows a graph with a line generated using median averaging based on the point cloud frontal projection of FIG. [Figure 15] FIG. 15 shows a graph with curves generated using different averaging algorithms. [Figure 16] FIG. 16 shows a graph with various curves generated using different averaging algorithms. [Figure 17] FIG. 17 shows a flowchart of a method according to one embodiment. [Figure 18] FIG. 18 illustrates an automatic steering system according to one embodiment. [Figure 19] FIG. 19 illustrates a high-level block diagram of a computer for performing the operations of the elements described by the embodiments in this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0005] 1 shows a farm having an alley 102 bounded on either side by rows 104A, 104B of plants. The plants in rows 104A, 104B require various agricultural operations to be performed to grow the plants from seed to maturity and then harvest them. Machinery is frequently used on farms to perform the agricultural operations necessary to grow the plants.

[0006] Figures 2 and 3 show agricultural machinery controlled by a human operator. Figure 2 shows a tractor 202 traveling along a path 102 bounded on either side by rows 104A and 104B. The tractor 202 is configured to travel along the path 102 without damaging the plants in the rows 104A and 104B located on either side of the path 102. Figure 3 shows a harvester 302 traveling along a row 304. Wheels 301A of the harvester 302 are shown traveling along a path 306A located on a first side of the row 304, and wheels 301B are shown traveling along a path 306B located on a second side of the row 304. The tractor 202 and harvester 302 typically travel at a speed of about 1-3 m / s, and the paths between the rows of plants are typically 2-4 m wide.

[0007] Automation enables efficient and consistent operation of agricultural machinery. In one embodiment, agricultural machinery is automated using data from cameras that may be mounted on the machinery. Figures 4(A)-(C) and 5(A)-(C) show camera orientation parameters based on how each camera is mounted on its associated vehicle.

[0008] FIG. 4(A) shows a tractor 402 with a camera 404A mounted on an upper body member (e.g., roof 410) and a camera 404B mounted near the front of the tractor 402 (e.g., hood 412). In one embodiment, cameras 404A and 404B are stereo cameras with each camera having two lenses. In other embodiments, cameras 404A and 404B can be any type of three-dimensional (3D) sensor, such as a time-of-flight (ToF) camera or a 3D lidar (LiDAR) sensor. Camera 404A is located at height H1 406A above the ground 414 on which the tractor 402 operates, and camera 404B is located at height H2 406B above the ground 414 on which the tractor 402 operates. Both cameras 404A and 404B are pointed downward so that the field of view of each camera includes the area of ​​the ground 414 in front of the tractor 402. In one embodiment, cameras 404A, 404B compute a 3D point cloud of objects within each camera's respective field of view. Camera 404A's longitudinal axis 420 (i.e., the axis along which the camera views the environment) is angled downward from horizontal 422 at angle β1 408A. Camera 404B's longitudinal axis 420 is angled downward from horizontal 422 at angle β2 408B. Note that angle β1 408A may be the same as or different from angle β2 408B. Also, a single camera is sufficient for the system to operate, and FIG. 4(A) shows two variations of possible positions for such a camera (i.e., the positions of camera 404A and camera 404B).

[0009] 4B illustrates additional orientation parameters for camera 404B. Displacement D 410 is the offset of the center 418 of either of camera 408B's two lenses from the longitudinal centerline 416 of tractor 402. Note that in one embodiment, cameras 404A and 404B are mounted such that a point between each camera's two lenses is located on the centerline of the machine on which the camera is mounted. γ 412 is the angle between level 422 and the horizontal axis 424 of camera 404B based on how camera 404B is mounted.

[0010] FIG. 4C shows angle α 414, which is the angle between the longitudinal axis 420 of camera 404B and the longitudinal centerline 416 of tractor 402.

[0011] FIGS. 5(A)-(C) show orientation parameters for a camera 504 mounted on an upper body member (e.g., roof 528) of a harvester 502. FIG. 5(A) shows the camera 504 positioned at height H 506 above the ground 414 on which the harvester 502 operates. The camera 504 is tilted downward at angle β 508, with its longitudinal axis 516 below level 518. FIG. 5(B) shows offset D 510, which is the distance of the center 522 of the camera's lens 505B from the longitudinal centerline 524 of the harvester 502. FIG. 5(B) also shows angle γ 512, which is the angle between level 518 and the horizontal axis 520 of the camera 504 based on how the camera 504 is mounted. FIG. 5(C) shows angle α 514, which is the angle between the longitudinal axis 516 of the camera 504 and the longitudinal centerline 526 of the harvester 502.

[0012] In one embodiment, the displacement D is based on the distance between the vehicle's centerline and the projection of the origin of the stereo camera's local coordinate system onto the ground. In one embodiment, the origin is located in the upper left corner of the camera's left sensor (see FIG. 10). The displacement D, height H, and one or more angles described above are used to relate the vehicle's coordinate system to the camera's coordinate system.

[0013] In one embodiment, as described in more detail below, orientation parameters of cameras on the tractor 402 and harvester 502 are used in conjunction with point clouds of data acquired using the cameras to steer the tractor 402 and harvester 502. The motion and orientation parameters of the tractor 402 and harvester 502 are described with reference to Figures 6 and 7.

[0014] FIG. 6 shows a tractor 402 moving along an alley 602 located between plant rows 604A and 604B. An alley centerline 606 (also referred to as the midline, vineyard alley center, or orchard alley center) is located substantially equidistant from rows 604A and 604B and is shown as the dashed line between points A and B. Heading error 608 is the angle between the longitudinal axis 416 of the tractor 402 and the midline 606. X-track 610 is the distance from the midline 606 to the center 614 of the rear axle 612 of the tractor 402 (i.e., how far the tractor's longitudinal axis is from the midline. For example, the tractor 402 can translate relative to the midline 606 without its longitudinal centerline intersecting the midline 606). L 616 is the distance between the camera 404B and the rear axle 612 of the tractor 402.

[0015] 7 shows harvester 502 moving along row 702B between rows 702A and 702C. A centerline 706 (also referred to as the midline) of row 702B is located along row 702B between points A and B. Heading error 708 is the angle between the longitudinal axis 526 of harvester 502 and midline 706. X-track 710 is the distance from midline 706 to the center 714 of the rear axle 712 of harvester 502. Distance L 716 is the distance between camera 504 and center 714 of rear axle 712 of harvester 502.

[0016] A camera (such as one of cameras 404A, 404B, and 504 shown in FIGS. 4A-C, 5A-C, 6, and 7) is used to generate a point cloud of data about the environment in which the tractor and / or harvester operates. A point cloud of data has a plurality of points in three-dimensional space, each point representing a portion of a physical object. In one embodiment, the points of the point cloud are used to determine objects within the field of view of the camera. In one embodiment, point clouds 802, 902 (shown in FIGS. 8 and 9, respectively) are used to determine the location of tracks and plant rows relative to agricultural machinery such as tractor 402 or harvester 502. The point clouds can be generated in real time as the agricultural vehicle associated with the camera moves, or they can be generated in advance of operations performed by the vehicle.

[0017] FIG. 8 shows point cloud 802 including row 804 with aisles 806A and 806B located on either side. In one embodiment, point cloud 802 is generated using data obtained from a camera (such as one of cameras 404A, 404B, and 504 shown in FIGS. 4(A)-(C), 5(A)-(C), 6 and 7). In one embodiment, as shown in FIG. 7, point cloud 902 is generated based on information from camera 504 as harvester 502 moves along row 702B.

[0018] Figure 9 shows a point cloud 902 that includes a path 904 bounded on either side by plant rows 906A, 906B. In one embodiment, point cloud 902 is generated using data obtained from a camera (such as one of cameras 404A, 404B). In one embodiment, point cloud 802 shown in Figure 8 is generated based on information from camera 504 as harvester 502 moves along vineyard rows 706, as shown in Figure 7.

[0019] In one embodiment, the coordinates of the point cloud are defined in the camera's local coordinate system. FIG. 10 illustrates the camera's local coordinate system according to an embodiment in which the coordinate system is based on the camera's image sensor. Camera 1002 is shown having a left lens 1004A and a right lens 1004B. Each lens focuses an image onto its respective image sensor. Lens 1004A focuses an image onto image sensor 1006A, and lens 1004B focuses an image onto image sensor 1006B. The coordinate system is located at point O, where the X, Y, and Z axes intersect. In the embodiment shown in FIG. 10, point O is located at the upper left corner of the left image sensor of the stereo camera. The horizontal axis X is orthogonal to the vertical axis Y, and both the X and Y axes are contained in the plane of the left image sensor and are orthogonal to the Z axis.

[0020] In one embodiment, a ground plane that coincides with the ground plane on which the agricultural machine moves is found as a 3D point cloud using a random sample consensus (RANSAC) algorithm. Because the approximate position and orientation of the camera placement (height and tilt) are known, it is possible to impose constraints on the desired plane, significantly speeding up the search for the ground plane. As a result, the 3D sensor position and orientation relative to the ground plane can be estimated. In this way, the sensor height, roll (tilt) and pitch (pitch) angles can be estimated.

[0021] In one embodiment, the obtained point cloud is transformed by translation and rotation using estimates of the camera's height, roll and pitch angles in such a way that the ground plane found above is contained in the XOZ plane of the camera's local coordinate system.

[0022] In one embodiment, a Hough transform detection algorithm is used to detect the centerline of a plant row. There are two modes of operation for this algorithm: harvester mode, in which the camera needs to be positioned above the row and detect only one row, and tractor mode, in which the camera needs to be positioned between the rows and detect both rows from the left and right sides of the sensor.

[0023] In one embodiment, a horizontal projection (top view) of the point cloud is used to find the columns. When constructing the horizontal projection, only points that lie above the ground plane are used; the higher a point is above the ground plane, the more it contributes to the projection. In this way, in the horizontal projection, the columns are clearly visible and the ground plane is removed.

[0024] Figure 11 shows a horizontal projection 1102 for detecting row positions to control harvester operation relative to the row positions of plants. A horizontal plane formed by an X axis 1104 and a Z axis 1106 (representing the X and Z axes shown in Figure 10, respectively) is used to display the point cloud data forming the rows. In one embodiment, a Hough transform detection algorithm is used to identify centerlines 1108 of the plant rows.

[0025] Figure 12 shows a horizontal projection 1202 for detecting row position to control tractor operation relative to row position. Row detection used with a tractor detects the two rows that separate the aisle. A horizontal plane formed by the X-axis 1204 and Z-axis 1206 (representing the X-axis and Z-axis shown in Figure 10, respectively) is used to represent the point cloud data that forms the two rows. In one embodiment, a Hough transform detection algorithm is used to identify centerlines 1208A, 1208B, which are opposite sides of the aisle.

[0026] The centerline identified as shown in Figures 11 and 12 can be used to estimate the heading error, which is the angle between the vehicle's longitudinal axis and the median line (i.e., the centerline of the row or aisle) and the X-track.

[0027] A frontal projection of the point cloud is generated after correction using a previously determined heading error, determined based on determining the centerline of the row or the centerline of the aisle between two rows.

[0028] Figure 13 shows a point cloud front projection 1302 for use with a harvester. The X and Y axes shown in Figure 13 correspond to the X and Y axes shown in Figure 10, respectively. The point cloud front projections are averaged to identify the tops of the rows (i.e., row medians). Figure 14 shows a graph 1402 in which a line 1404 is generated based on the edge shape (i.e., median averaging).

[0029] A sliding window averaging algorithm can be used to determine a curve whose maximum value corresponds to the row centerline. FIG. 15 shows a graph 1502 consisting of various lines generated using different algorithms. Line 1504 is generated using median averaging (as shown in FIG. 14). Line 1506 is generated using sliding window averaging. Line 1508 shows the projection of the maximum sliding window averaging function on the X-axis. Based on the data in FIG. 15, the lateral displacement of the vehicle relative to the centerline of the row (i.e., the X-track) can be calculated based on the row centerline information and the camera position information. Lines 1504, 1506, and 1508 are used to determine the row centerline. Information about the row centerline, along with vehicle orientation information, can be used to determine how the vehicle should be steered to travel a desired path (i.e., along the row centerline for the harvester).

[0030] FIG. 16 shows a graph 1602 of various lines generated using different algorithms. Graph 1602 is for use with a tractor, and two rows are selected. Based on the two rows, the lateral offset of the tractor relative to the centerline between the two rows can be determined. Line 1604 is generated using median averaging (as shown in FIG. 14). Line 1606 is generated using sliding window averaging. Lines 1608A and 1608B represent the projection of the maximum sliding window averaging function on the X-axis. Line 1610 is determined to be located midway between lines 1608A and 1608B. Thus, lines 1604 and 1606 are used to determine lines 1608A and 1608B, which are used to determine line 1610 (i.e., the centerline of the aisle) located midway between lines 1608A and 1608B. Information about the row centerline, along with vehicle orientation information, can be used to determine how the vehicle should be steered to proceed along the desired path (i.e., along the row centerline for the tractor).

[0031] In one embodiment, the detection of rows can be made more precise and accurate. In some embodiments, the row estimates obtained based on the previous methods described above are not accurate enough. This is due to the fact that rows contain grapevines that clearly deviate from one another. To eliminate such anomalies in the linear approximation of the lanes, a least squares method is used.

[0032] Because the density and accuracy of a 3D point cloud decreases with distance from the stereo camera, and because higher points characterize a sequence better than lower points, in one embodiment a least squares method is used, taking into account the weighting of the measurements: the closer and higher the points that characterize a sequence, the more weight they receive.

[0033] In addition to estimating the heading error and X track based on the sequence, we can also calculate an estimate of the reliability of the results produced. There are several measures of reliability:

[0034] In the case of harvesters, there are three criteria: row continuity, row height constancy, and row relative smoothness (i.e., how much the data points associated with a row differ from the fitted line).

[0035] In the case of tractors, two more criteria are added: the parallelism of two adjacent rows and the correspondence of the distance between them with the expected value (the actual distance between the rows on the farm).

[0036] In one embodiment, row, aisle, and vehicle position information is used to execute vehicle steering commands. In one embodiment, tractor speed is used to determine steering commands. Speed ​​information is obtained from the vehicle's standard odometry system or calculated using a navigation GNSS sensor. Alternatively, vehicle speed is obtained with the aid of a three-dimensional optical sensor and optical flow methods.

[0037] In one embodiment, the X track and heading error parameters, which are based on the vehicle position and the row or aisle position, are used to generate commands for the steering system. In one embodiment, the following equations are used:

[0038] U=-atan(L / V * (K1 * α-K2 * In D)),

[0039] U is the steering angle of the vehicle's steering wheels in radians;

[0040] L is the distance between the front and rear wheel axles of the vehicle in metres

[0041] V is the vehicle speed in meters per second,

[0042] α is the heading error in radians,

[0043] D is X track in meters

[0044] K1 and K2 are scale factors.

[0045] 17 is a flowchart of a method 1700 for automatic steering of an agricultural vehicle. In one embodiment, a machine controller associated with the agricultural vehicle performs the method 1700. In step 1702, a point cloud of data is received by the machine controller. In one embodiment, the point cloud is generated using a stereo camera mounted on the vehicle.

[0046] In step 1704, the position of the queue is determined based on the point cloud. In one embodiment, the position of the queue relative to the vehicle is determined using the steps previously described.

[0047] In step 1706, a steering angle is generated based on the position of the column relative to the position of the vehicle. In one embodiment, the steering angle is generated based on the heading error and X track determined based on the position of the vehicle relative to the column. In one embodiment, the steering angle is determined using the equations described above.

[0048] FIG. 18 illustrates an automatic steering system 1800 including a machine controller 1802 mounted on an agricultural vehicle capable of automatically steering. In one embodiment, the machine controller 1802 is a processor that controls the operation of the associated vehicle and, in some embodiments, additional peripheral equipment. The machine controller 1802 communicates with a camera 1804. In one embodiment, the camera 1804 is one or more of the cameras 404A, 404B, and 504 shown in FIGS. 4(A)-(C), 5(A)-(C), 6, and 7. In one embodiment, the camera 1804 transmits data that is used to generate a point cloud of data about objects within its field of view. In another embodiment, the camera 1804 generates a point cloud of data that is transmitted to the machine controller 1802. The machine controller 1802 also communicates with a steering controller 1806, which receives steering commands transmitted from the machine controller 1802. When the machine controller 1802 is operating to automatically steer the agricultural vehicle, the steering control device 1806 communicates with a steering actuator 1808, which steers the agricultural vehicle. The steering actuator 1808 can be an electric, hydraulic, or pneumatic device used to actuate steering associated with the machine to which the steering actuator 1808 is attached. Note that in some embodiments, the functionality of the machine controller 1802, steering control device 1806, steering actuator device 1808, and camera 1804 can be omitted or combined to form one or more devices. For example, an agricultural machine may have a combination of steering control and actuator devices that perform the operations described herein as being performed by the machine controller, steering control device, and steering actuator device. An agricultural machine may have a machine controller that performs the operations described herein as being performed by the machine controller, steering control device, and steering actuator device.

[0049] In one embodiment, the automatic steering system 1800 is calibrated once installation into the agricultural machine is complete. Calibration may be performed at other times as desired or needed. In one embodiment, calibration is performed by the agricultural vehicle, with the machine controller in calibration mode, traveling along a flat calibration surface having three straight reference lines. The reference lines are spaced apart by known distances. Based on information contained in the point cloud generated by the system while traveling across the flat calibration surface, the automatic steering system 1800 can determine what adjustments are needed to operate the automatic steering system accurately.

[0050] In one embodiment, the accuracy of the X-track calculation is 5 cm or less, and the accuracy of the heading error calculation is 1 degree or less. In one embodiment, the distance range for the aisle width calculation is 0.5 to 15 m. In one embodiment, the distance between rows of plants on the farm is substantially constant and known.

[0051] In one embodiment, a computer is used to cause the machine controller 1802 to perform the method of FIG. 17 . In one embodiment, the computer is capable of implementing the method and providing calculated steering angles at a frequency of 5 Hz or greater. The computer may also be used to operate the steering controller 1806, steering actuator 1808, and / or camera 1804. A high-level block diagram of such a computer is shown in FIG. 19 . The computer 1902 includes a processor 1904 that controls the overall operation of the computer 1902 by executing computer program instructions that define such operation. The computer program instructions may be stored in a storage device 1912 or other computer-readable medium (e.g., magnetic disk, CR-ROM, etc.) and read into memory 1910 when execution of the computer program instructions is desired. Thus, the elements and equations described in this disclosure may be defined by computer program instructions stored in memory 1910 and / or storage device 1912 and controlled by the processor 1904 executing the computer program instructions. For example, the computer program instructions may be implemented as computer-executable code programmed by one skilled in the art to implement the algorithms defined by the elements and equations set forth in this disclosure. Thus, by executing the computer program instructions, the processor 1904 performs the method illustrated in FIG. 17. The computer 1902 also includes one or more network interfaces 1906 for communicating with other devices over a network. The computer 1902 also includes input / output devices 1908 (e.g., a screen, keyboard, mouse, speakers, buttons, etc.) that allow a user to interact with the computer 1902. Those skilled in the art will recognize that an actual computer implementation may include other elements as well, and that FIG. 19 is a high-level representation of some of the elements of such a computer for illustrative purposes. In one embodiment, the computer 1902 is implemented using an NVIDIA Xavier or Olin processor.

[0052] The foregoing detailed description should be understood in all respects to be illustrative and exemplary, and not restrictive, and the scope of the inventive concepts disclosed in this disclosure should be construed in accordance with the full breadth allowed by patent law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the inventive concepts, and that various modifications may be made by those skilled in the art without departing from the scope and spirit of the inventive concepts. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the inventive concepts.

Claims

1. An automatic steering method for an agricultural machine, comprising: Receive point cloud data from the camera, determining the position of a row of plants based on the point cloud data; A method for automatically steering an agricultural machine, comprising generating a steering angle for the agricultural machine based on the position of the row.

2. 2. The method of claim 1, wherein said determining comprises determining the centerline of said column based on a Hough transform detection algorithm.

3. The method of claim 2 , wherein the Hough transform detection algorithm determines the centerline of the column based on a horizontal projection of the point cloud data.

4. 4. The method of claim 3, wherein the horizontal projection of the point cloud data is based on points of the point cloud that are above ground level found using a random sample consensus algorithm.

5. The method of claim 4 , wherein the determination of the centerline of the column is further based on a frontal projection of the point cloud data.

6. 10. The method of claim 1, wherein the generation of a steering angle is further based on a heading error.

7. 7. The method of claim 6, wherein the heading error is the angle between the longitudinal axis of the agricultural machine and a median line.

8. 8. The method of claim 7, wherein the generation of steering angles is further based on an X-track.

9. 9. The method of claim 8, wherein the X-track is the distance from the median line to the center of the rear wheel axle of the agricultural machine.

10. 10. The method of claim 9, wherein the intermediate line is a centerline of the row.

11. 10. The method of claim 9, wherein the intermediate line is a centerline located between the row and an adjacent row that is parallel to and offset from the row.

12. A steering actuator; A camera and an apparatus comprising a steering control device in communication with the steering actuation device and the camera, The steering control device includes: receiving point cloud data from the camera; determining the position of a row of plants based on the point cloud data; generating a steering angle for the agricultural machine based on the position of the row.

13. 13. The apparatus of claim 12, wherein said determining comprises determining the centerline of said column based on a Hough transform detection algorithm.

14. The apparatus of claim 13 , wherein the Hough transform detection algorithm determines the centerline of the column based on a horizontal projection of the point cloud data.

15. 15. The apparatus of claim 14, wherein the horizontal projection of the point cloud data is based on points of the point cloud that are above ground level found using a random sample consensus algorithm.

16. 16. The apparatus of claim 15, wherein the determination of the centerline of the column is further based on a frontal projection of the point cloud data.

17. 13. The apparatus of claim 12, wherein the generation of the steering angle is further based on heading error.

18. 18. The apparatus of claim 17, wherein the generation of steering angles is further based on an X-track.

19. 1. A computer-readable medium storing computer program instructions for automatic steering of an agricultural machine, the computer program instructions, when executed by a processor, Receive point cloud data from the camera, determining the position of a row of plants based on the point cloud data; 11. A computer-readable medium configured to cause the processor to perform the operation of: generating a steering angle for the agricultural machine based on the position of the row.

20. 20. The computer-readable medium of claim 19, wherein the determining determines the centerline of the column based on a Hough transform detection algorithm.

21. 21. The computer-readable medium of claim 20, wherein the Hough transform detection algorithm determines the centerline of the column based on a horizontal projection of the point cloud data.

22. 22. The computer-readable medium of claim 21, wherein the horizontal projection of the point cloud data is based on points of the point cloud that are above a ground plane found using a random sample consensus algorithm.

23. 23. The computer-readable medium of claim 22, wherein the determination of the centerline of the column is further based on a frontal projection of the point cloud data.

24. 20. The computer-readable medium of claim 19, wherein the generation of a steering angle is further based on a heading error.

25. 25. The computer-readable medium of claim 24, wherein the generation of steering angles is further based on an X-track.

Images