Automatic steering system for guiding an agricultural vehicle across a field and corresponding procedure

Abstract

Automatic steering system for guiding an agricultural vehicle (10) across a field on which a track (56) was left during a previous work operation, wherein the steering system comprises a camera (32) looking at the track, an image processing system (36) for processing image signals from the camera (32), and a steering control (34) to which a position signal generated by the image processing system (36) with respect to a detected position of the track (56) can be supplied, and the image processing system (36) is programmed to convert the image signal from the camera (32) into a binary image and to recognize the track (56) by means of a morphological operation applied to the binary image, in which it is checked at which points in the image a structural element (50) representing the track (56) fits exactly into,wherein the driving lane (56) consists of two depressions in the ground arranged at a predetermined distance and the structural element (50) consists of two approximately perpendicular lines (52, 54) or rectangles arranged at a lateral distance corresponding to the predetermined distance of the depressions in the image of the camera (32).

Description

[0001] The invention relates to an automatic steering system for guiding an agricultural vehicle across a field and a corresponding method. State of the art

[0002] In agriculture, there is a need for vehicles to steer themselves autonomously across a field to ease the operator's workload and allow them to concentrate on monitoring and controlling tasks performed while driving across the field, such as tillage, application of substances, or harvesting crops. In addition to steering systems that generate a steering signal using a satellite-based positioning system and a map of the tracks to be traversed (EP 0 856 453 A2), optical steering systems are also known. These optical systems use a laser rangefinder scanning the field in front of the vehicle (DE 197 43 884 A1) or a camera image evaluated by an image processing system (DE 103 51 861 A1, US 8 706 341 B2, EP 0 878 121 A1) to detect features in the field, which are then used to evaluate a route to be traveled.These features may be cultivation boundaries that entail a change in the soil profile, or the edge of a still standing crop stand.

[0003] While steering systems based on satellite-based positioning systems are relatively complex and therefore expensive, especially if they are to have the (usually required) accuracy in the centimeter range, and also require data regarding the lanes to be traveled (i.e., the map), and the laser rangefinders are also relatively expensive, the hardware for a steering system based on image processing can be presented in a more cost-effective manner.

[0004] Many existing image processing-based steering systems rely on first learning which element of the captured image should be considered the tracking structure. For example, in DE 103 51 861 A1, a sample image is first captured, and a neural network is then informed via human input which areas of this image should be considered the swath to be tracked. The neural network can then recognize a swath based on a real image and generate a steering signal accordingly. US 8 706 341 B2 uses a similar approach. The algorithm for recognizing the image features to be tracked is relatively complex and requires significant computing power. The steering system of EP 0 878 121 A1 is based on the detection of a crop edge in the image.

[0005] Another approach (K. Hanawa et al., Development of a Stereo Vision System to Assist the Operation of Agricultural Tractors, JARQ 46 (4), 287 - 293 (2012) http: / / www.jircas.affrc.go.jp) for an image processing-based steering system involves leaving a predetermined track in the field using a track marker or similar device during a previous operation. A three-dimensional image captured with a stereo camera is then examined for features that correspond to the predetermined track. This involves performing pattern matching on a two-dimensional image spanning the width of the field in front of the vehicle, i.e., successively examining the degree of correspondence between a predetermined pattern and the individual areas of the image. The disadvantages of this approach are that the predetermined track must first be generated, and the pattern recognition algorithm is also relatively computationally intensive.

[0006] Finally, it was proposed (J. Xue et al., Variable field-of-view machine vision based row guidance of an agricultural robot, Computers and Electronics in Agriculture 84 (2012) 85-91) to distinguish maize plants in a field from the ground based on their color. If there are not enough plants in the field of view, the system switches to a so-called close-up mode, in which the camera is lowered. The objects to be detected are then only the plant stems, which are identified using morphological operations with a rectangular structural element to identify and segment the stems. The lower ends of the stems are connected by a line, and a path to be followed is placed in the middle between adjacent lines. While this approach is less computationally intensive than pattern recognition, it is limited to cases where plant stems can be detected in a field.

[0007] Image processing systems for plant recognition have been proposed (US 5,253,302 A) in which a binary image is first generated based on color. A learning process is then performed using an image containing plants marked by an operator, followed by a clustering process. Morphological operations are applied to the identified clusters to give them a more closed shape. Based on the learned clusters, the image is decomposed into a binary image to subsequently identify plant parts based on geometric features or shape.

[0008] Similarly, US 5 978 498 A describes an application of morphological operations to an image of cell groupings in a biological sample to prepare the image for further processing. Object of the invention

[0009] The object underlying the invention is seen as being to provide an automatic steering system for guiding a vehicle across a field that requires as little computing power as possible, and a corresponding method that makes it possible to use existing tracks from previous processing operations in the field for steering. Solution

[0010] This problem is solved according to the invention by the teaching of claims 1 and 6, wherein further claims list features which advantageously develop the solution further.

[0011] An automatic steering system for guiding an agricultural vehicle across a field where a track has been left during a previous operation comprises a camera facing the track, an image processing system for processing the camera's image signals, and a steering control unit that can receive a position signal generated by the image processing system regarding the detected position of the track. The image processing system is programmed to convert the camera's image signal into a binary image and to recognize the track using a structural element representing the track, based on a morphological operation applied to the binary image. Thus, the camera image is converted into a binary image, and a morphological operation is used to determine precisely where in the image the structural element fits.The resulting points represent the detected lane in the image, which is fed directly or after further processing, e.g., to determine the position of the lane's center and its direction, to the steering control system. Based on this detected lane, the steering control system can generate a steering signal to steer the vehicle along the lane.

[0012] This method provides a relatively simple and computationally efficient way to detect the driving path. The steering system is suitable, for example, for self-propelled, mounted, or trailed sprayers that are guided along existing tramlines. These tramlines are created when sowing the field and are subsequently driven over multiple times for fertilizing and / or controlling unwanted plants or insects, etc., so that they become clearly defined in the soil and can be easily detected.

[0013] The lane consists of two depressions in the ground arranged at a predetermined distance. The structural element comprises two approximately perpendicular lines or rectangles arranged at a lateral distance corresponding to the predetermined distance between the depressions in the camera image.

[0014] If the camera is mounted on the vehicle tilted forward and downward, it is advisable to program the image processing system to perform a perspective correction to correct the tilt of the camera.

[0015] The camera can be a color camera and the image processing system can be programmed to generate the binary image based on the colors of the image.

[0016] The camera can be a stereo camera and the image processing system can be programmed to generate the binary image based on a disparity image of the images from the stereo camera or based on a color image from one or both of the stereo cameras.

[0017] It is also possible to switch between a binary image generated from the colors of the image and a disparity image of the stereo camera's images, using the camera's signals. Example of implementation

[0018] The drawings illustrate an embodiment of the invention, which is described in more detail below. It shows: Fig. 1. A side view of an agricultural vehicle with an automatic steering system Fig. 2. A flowchart illustrating how the steering system operates. Fig. 3. A flowchart illustrating the image processing and lane detection step. Fig. 4. A flowchart illustrating the image processing of color signals. Fig. 5 a flowchart illustrating the image processing of disparity signals, Fig. 6. A flowchart illustrating the image preprocessing of color signals. Fig. 7 a flowchart illustrating the image preprocessing of disparity signals, Fig. 8-11 Examples of color images of a field before, during and after preprocessing, Fig. 11-20 Examples of disparity images of a field before, during and after preprocessing, and Fig. 21 examples of an image of the field during the detection process.

[0019] The Fig. Figure 1 shows a side view of a self-propelled agricultural vehicle 10 in the form of an agricultural tractor and an implement 12 in the form of a field sprayer mounted on a three-point linkage 14 of the vehicle 10. The vehicle 10 is built on a supporting frame 16, which rests on steerable front wheels 18 and driven rear wheels 20 and carries a cab 22 containing an operator workstation 24. The operator workstation 24 includes a steering wheel 26, a seat 28, pedals (not shown), and a virtual terminal 30.

[0020] The vehicle 10 includes an automatic steering system comprising a stereo camera 32, an image processing system 36 connected to the image sensor of the camera 32, and a steering control unit 34. The image processing system 36 could be integrated into the housing of the camera 32 or into the steering control unit 34. The steering control unit 34 is connected to a valve 38 for controlling a steering actuator 40 and to a steering angle sensor 44 for detecting the current steering angle of the front wheels 18. During operation, the camera 32 views the field in front of the vehicle 10 and transmits its image signals to the image processing system 36, which detects an existing track 56 in the field and sends a signal regarding its position to the steering control unit 34.The steering control unit 34 controls the steering actuator 40 via the valve 38 in such a way that the vehicle 10 automatically follows the existing lane 56, taking into account the signals from the steering angle sensor 44 as feedback signals. For the hardware of the steering system, reference is made to the disclosure in DE 103 51 861 A1, which is incorporated into the present documents by reference.

[0021] The Fig. Figure 2 shows a flowchart illustrating the operation of the steering system. After initial data acquisition of the image signal from camera 32 (step 100), image processing and a detection process take place in step 102, a lane 56 to be driven is generated in step 104, and a steering signal is generated in step 106, which is used in step 110 to steer the vehicle 10. In step 112, the results of steps 100 to 106 can be displayed on the virtual terminal 30.

[0022] The Fig. Figure 3 shows step 102 of the image processing and detection process in more detail. In the first step, 120, initial values ​​are detected and assigned by finding the position and size of the window in the image from camera 32 that contains a lane 56. The colors in the image and parameters of lane 56 (color, difference from the rest of the image) are also evaluated. In step 122, it is then checked whether the system is ready for automatic driving, i.e., whether a lane 56 could be detected in the image. If so, step 124 follows, in which it is checked whether the image should be examined based on color or based on a disparity image. In the first case, step 128 follows, which is described in the Fig. 4 and Fig. 6 is shown in more detail, and in the second case, step 130, which is in the Fig. 5 and Fig. 7 is shown in more detail. In step 132, which follows both steps 128 and 130, lane 56 is detected; in step 134, the reliability of the detection of lane 56 and points to be traversed by vehicle 10 are determined; and in step 136, the list of points to be traversed is generated and validated, i.e., checked for plausibility.

[0023] The Fig. Figure 4 shows step 128, i.e., the detailed processing of (only the) color signals from camera 32. After image acquisition in step 140, image preprocessing takes place in step 142, then lane 56 is detected in step 144, and then steps 146 and 148, which correspond to steps 134 and 136.

[0024] The Fig. Figure 5 shows step 130, i.e., the detailed processing of (only the) disparity signals from camera 32. After image acquisition in step 150, image preprocessing takes place in step 152, then lane 56 is detected in step 154, and then steps 156 and 158, which correspond to steps 134 and 136.

[0025] The Fig. Figure 6 shows the procedure of step 142 (and thus that of step 128) in detail. In step 160, an image transformation is performed to correct the downward tilt of camera 32. In step 162, color differences are magnified, and in step 164, discrimination is performed, i.e., the image is divided into two values ​​based on a threshold. Fig. Figure 8 shows an example of an unprocessed image from camera 32, which would originally be in color and shows lane 56 in a different color (especially brown) than the plants (usually green). Fig. Figure 9 shows the image after the cropping process of step 120 and the perspective correction of step 160, which Fig. 10 the image after step 162 and the Fig. 11 the picture after step 164.

[0026] The Fig. Figure 7 shows the procedure of step 152 (and thus that of step 130) in detail. In step 170, outliers are removed; in step 172, the tilt of the vehicle 10 (e.g., detected by an inertial navigation system) is corrected in the forward and lateral directions; in step 174, any gaps are filled; in step 176, an image transformation is performed to correct the downward tilt of the camera 32; and in step 178, a comparison is made with a threshold, as in the Fig. 11 examples are shown. Fig. Figure 12 shows an example of an unprocessed disparity image from camera 32 in two-dimensional representation, while the Fig. 13 the image of the Fig. 12 in three-dimensional representation shows the Fig. 14 the image of the Fig. 13 after correction of the pitch angle and the Fig. 15. The image after correction of the roll angle (step 172). Fig. 16 shows the image after Fig. 15 in two-dimensional representation. In Fig. 17 holes were filled according to step 174 and in Fig. 18. The perspective correction of step 176 was performed while the Fig. 19 the image after step 178 shows and the Fig. 20 the image after step 154.

[0027] The procedure according to the Fig. 4 and Fig. Option 6 is always suitable when lane 56 is sufficiently distinguishable in color from the rest of the field, e.g., when it is dark (brown, black, etc.) and the rest of the field is green (covered with vegetation), and when disparity images are not informative or are unavailable due to the lack of a functioning stereo camera. The procedure according to the Fig. 5 and Fig. Option 7 is always suitable when lane 56 has sufficient spatial contours, i.e., is sufficiently deep. In step 124, it is therefore checked which of the aforementioned approaches is more promising and the more suitable one is selected.

[0028] The result of steps 128 and 130 is each a pre-processed, binary image of the field in front of vehicle 10, in which lane 56 has been identified as clearly as possible. It now needs to be determined at which points in the image the existing lane 56 is located, which is done in steps 132, 144, and 154. The [unclear text] in the Fig. Step 132, shown as an example in Figure 21, is always the same regardless of the image preprocessing after steps 128 or 130. As shown in the Fig. 21 shows the original image from camera 32 ( Fig. 21a) preprocessed and binarized, as in Fig. 21b is shown. Subsequently, in step 132 (which corresponds to steps 144 and 154), a morphological operation is performed, in which it is checked into which positions of the binary image a value in the Fig. The structural element 50 shown in 21c fits exactly into it (“fit”), which represents lane 56.

[0029] The structural element 50 comprises two vertical lines 52, 54 or rectangles arranged side by side at a distance corresponding to the two depressions of the existing lane 56. The distance between the vertical lines 52, 54 can be automatically detected by the image processing system 36 or entered by the operator via the virtual terminal 30, as can the width of the lines 52, 54. If the lane 56 extends straight across the entire image in the vertical direction, the length of the lines 52, 54 could correspond to the image height, whereas the length of the lines 52, 54 will be shorter if the lane 56 is curved and depends on the curve radius of the lane 56. The length of the lines 52, 54 can be automatically detected by the image processing system 36 or entered by the operator via the virtual terminal 30. Furthermore, the lines 52, 54 or rectangles of the structural element 50 do not have to be oriented exactly vertically, but can be (e.g.,(when driving around a curve) it may be tilted to the left or right. This angle can also be detected by the image processing system 36 or entered by the operator.

[0030] The result of step 132 is all areas of the image into which structural element 50 fits exactly, as shown in the Fig. 21d is shown. Fig. 21e shows the images of the Fig. 21b and Fig. 21d superimposed. Now, information is quickly and easily obtained indicating the locations of a lane 56 within the image. After removing outliers that are smaller than the expected value in the vertical direction and determining the center point, the positions of the depressions and, based on these, the position of the center of lane 56 can be calculated, which are then fed to the steering control 34 (steps 134, 136 of the Fig. 3).

Claims

[1] Automatic steering system for guiding an agricultural vehicle (10) across a field on which a track (56) has been left during a previous work operation, wherein the steering system comprises a camera (32) looking at the track, an image processing system (36) for processing image signals from the camera (32) and a steering control (34) to which a position signal generated by the image processing system (36) with respect to a detected position of the track (56) can be supplied and the image processing system (36) is programmed to convert the image signal from the camera (32) into a binary image and to recognize the track (56) by means of a morphological operation applied to the binary image, in which it is checked at which points in the image a structural element (50) representing the track (56) fits exactly into,wherein the driving lane (56) consists of two depressions in the ground arranged at a predetermined distance and the structural element (50) consists of two approximately perpendicular lines (52, 54) or rectangles arranged at a lateral distance corresponding to the predetermined distance of the depressions in the image of the camera (32). [2] Steering system according to claim 1, wherein the camera (32) is mounted on the vehicle inclined towards and downwards and the image processing system (36) is programmed to perform a perspective correction to correct the inclination of the camera (32). [3] Steering system according to one of claims 1 or 2, wherein the camera (32) is a color camera and the image processing system (36) is programmed to generate the binary image based on the colors of the image. [4] Steering system according to one of claims 1 to 3, wherein the camera (32) is a stereo camera and the image processing system (36) is programmed to generate the binary image based on a disparity image of the images from the stereo camera. [5] Steering system according to claim 4, wherein the image processing system (36) is programmed to switch between a binary image generated from the colors of the image and a disparity image of the images of the stereo camera based on the signals from the camera (32). [6] Method for automatically guiding an agricultural vehicle (10) across a field on which a track (56) has been left during a previous work operation, by means of an automatic steering system comprising a camera (32) looking at the track (56), an image processing system (36) for processing image signals from the camera (32) and a steering control (34) to which a position signal generated by the image processing system (36) with respect to a detected position of the track is supplied, wherein the image processing system (36) converts the image signal from the camera (32) into a binary image and recognizes the track (56) by means of a morphological operation applied to the binary image, in which it is checked at which points in the image a structural element (50) representing the track (56) fits exactly into,wherein the driving lane (56) consists of two depressions in the ground arranged at a predetermined distance and the structural element (50) consists of two approximately perpendicular lines (52, 54) or rectangles arranged at a lateral distance corresponding to the predetermined distance of the depressions in the image of the camera (32).

Images