Crop row detection system, agricultural machine equipped with the crop row detection system, and crop row detection method

Abstract

This crop row detection system comprises: an imaging device that is attached to an agricultural machine and that photographs a ground surface on which the agricultural machine travels to acquire time-series color images of at least a portion of said ground surface; and a processing device which performs image processing on the time-series color images. The processing device: generates, from each of the time-series color images, a highlight image in which the color of a crop row to be detected is highlighted; generates, from the highlight image, a top view image which is seen from above the ground surface and in which classification is performed between first pixels that have a greater index value for the color of the crop row than a threshold value and second pixels having an index value less than the threshold value; and determines the position of edge lines for the crop row on the basis of the index value of the first pixels.

Description

[Technical Field]

[0001] The present disclosure relates to a crop row detection system, an agricultural machine including the crop row detection system, and a crop row detection method. [Background technology]

[0002] Research and development is underway to automate work vehicles such as tractors used in farm fields. For example, work vehicles that run with automatic steering using positioning systems such as the Global Navigation Satellite System (GNSS), which enables precise positioning, have been put into practical use. Work vehicles that not only automatically steer but also automatically control speed have also been put into practical use.

[0003] In addition, vision guidance systems are being developed that use imaging devices such as cameras to detect crop rows or furrows in a field and control the movement of work vehicles along the detected crop rows or furrows.

[0004] Patent Document 1 discloses a work machine that travels along rows of cultivated land where crops are planted in the ridges. Patent Document 1 describes a method of binarizing an original image obtained by photographing the cultivated land from diagonally above with an on-board camera, and then generating a planar projectively transformed image. The technology disclosed in Patent Document 1 rotates the planar projectively transformed image to generate multiple rotated images with different orientations, and detects work paths between the ridges. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-208871 Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology of detecting crop rows or furrows using an imaging device, the detection accuracy may be reduced due to disturbance factors such as sunlight conditions.

[0007] The present disclosure provides a crop row detection system, an agricultural machine equipped with the crop row detection system, and a crop row detection method that can solve these problems. [Means for solving the problem]

[0008] In an exemplary, non-limiting embodiment, a crop row detection system according to the present disclosure includes an imaging device attached to an agricultural machine, which captures images of the ground on which the agricultural machine travels to acquire time-series color images including at least a portion of the ground, and a processing device that processes the time-series color images. The processing device generates an enhanced image from the time-series color images in which the color of the crop row to be detected is enhanced, generates a top-view image from the enhanced image as viewed from above the ground, in which pixels having a crop row color index value equal to or greater than a threshold value and pixels having a crop row color index value less than the threshold value, and determines the position of an edge line of the crop row based on the index value of the first pixel.

[0009] In an exemplary, non-limiting embodiment, an agricultural machine according to the present disclosure is an agricultural machine equipped with the above-mentioned crop row detection system, and includes a running gear including a steering wheel, and an automatic steering device that controls the steering angle of the steering wheel based on the position of the edge line of the crop row determined by the crop row detection system.

[0010] In an exemplary, non-limiting embodiment, the crop row detection method according to the present disclosure is a computer-implemented crop row detection method that causes a computer to perform the following steps: acquire, from an imaging device attached to an agricultural machine, time-series color images including at least a portion of the ground surface on which the agricultural machine is traveling, and generate, from the time-series color images, an enhanced image that enhances the color of the crop row being detected; generate, from the enhanced image, a top-view image viewed from above the ground surface, in which the crop row color index value is classified into first pixels that are equal to or greater than a threshold value and second pixels that are less than the threshold value; and determine the position of the edge line of the crop row based on the index value of the first pixel.

[0011] A general or specific aspect of the present disclosure may be realized by an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable non-transitory storage medium, or any combination thereof. The computer-readable storage medium may include a volatile storage medium or a non-volatile storage medium. An apparatus may be composed of multiple devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. [Effects of the Invention]

[0012] According to the embodiments of the present disclosure, it is possible to suppress a decrease in detection accuracy due to disturbance factors such as sunlight conditions, and to increase robustness. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram schematically illustrating an image of the ground captured by an imaging device attached to an agricultural machine. [Figure 2] FIG. 2 is a perspective view showing a schematic relationship between a body coordinate system Σb and a camera coordinate system Σc fixed to an agricultural machine, and a world coordinate system Σw fixed to the ground. [Figure 3]1 is a top view showing a schematic diagram of a part of a field with multiple rows of crops on the ground. FIG. [Figure 4] 4 is a diagram schematically illustrating an example of an image captured by an imaging device of the agricultural machine illustrated in FIG. 3. FIG. [Figure 5] FIG. 10 is a top view schematically showing a state in which the position and orientation (angle in the yaw direction) of the agricultural machine have been adjusted. [Figure 6] 6 is a diagram showing an example of an image captured by an imaging device of the agricultural machine in the state shown in FIG. 5. [Figure 7] 1 is a block diagram illustrating a basic configuration example of a crop row detection system according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram schematically illustrating a configuration example of a processing device according to an embodiment of the present disclosure. [Figure 9] This is a monochrome image corresponding to one frame of a time-series color image captured by an on-board camera mounted on a tractor. [Figure 10] FIG. 10 is a diagram showing an enhanced image obtained by converting the RGB values ​​of one frame of a time-series color image into an excess green index (ExG=2×grb). [Figure 11] 11 is a histogram of the excess green index (ExG) in the image of FIG. 10. [Figure 12] FIG. 10 is a diagram showing an example of a top-view image (bird's-eye view image) classified into first pixels (for example, crop pixels) and second pixels (background pixels). [Figure 13] 10 is a perspective view schematically showing the positional relationship between each of the camera coordinate system Σc1 and the camera coordinate system Σc2 and the reference plane Re. FIG. [Figure 14] FIG. 10 is a schematic diagram showing an example in which the direction of the crop rows in the top view image is parallel to the direction of the scanning lines. [Figure 15] 15 is a diagram schematically illustrating an example of an integrated value histogram obtained for the top view image of FIG. 14. FIG. [Figure 16] 10 is a schematic diagram showing an example in which the direction of the crop rows in the top view image intersects with the direction of the scanning lines. FIG. [Figure 17] FIG. 17 is a diagram schematically illustrating an example of an integrated value histogram obtained for the top view image of FIG. 16. [Figure 18] 10 is a flowchart illustrating an example of an algorithm by which a processing device determines edge lines of a crop row according to an embodiment of the present disclosure. [Figure 19] 13 is a diagram showing an integrated value histogram obtained from the top view image of FIG. 12. FIG. [Figure 20] FIG. 2 is a block diagram illustrating a process executed by a processing device according to an embodiment of the present disclosure. [Figure 21] FIG. 10 is a diagram for explaining a form in which a top view image is divided into a plurality of blocks. [Figure 22] 22 is a diagram schematically illustrating the relationship between the position of a scanning line and the integrated value of index values ​​in each block of FIG. 21. FIG. [Figure 23] 23 is a diagram showing an example of a crop row center in each block of FIG. 22 and an approximation line to the crop row center. FIG. [Figure 24] FIG. 24 is a top view showing an example of an edge line of a crop row determined based on the approximation line of FIG. 23. [Figure 25] FIG. 10 is a diagram illustrating a method for dividing a part or all of a top-view image into multiple blocks and determining the position of an edge line for each of the multiple blocks when a crop row includes a curved portion. [Figure 26] 26 is a diagram schematically illustrating the relationship between the position of the scanning line and the integrated value (histogram) of the index value in each block of FIG. 25. FIG. [Figure 27] FIG. 27 is a diagram showing an example of the crop row centers in each block of FIG. 26 and an approximation line to the crop row centers. [Figure 28] FIG. 28 is a top view showing an example of an edge line of a crop row determined based on the approximation curve of FIG. 27. [Figure 29] FIG. 1 is a perspective view illustrating an example of the appearance of an agricultural machine according to an embodiment of the present disclosure. [Figure 30] 1 is a side view schematically illustrating an example of an agricultural machine with a work implement attached thereto. FIG. [Figure 31] FIG. 1 is a block diagram showing an example of a schematic configuration of an agricultural machine and a work machine. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present disclosure will be described. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims. In the following description, components having the same or similar functions are designated by the same reference numerals.

[0015] The following embodiments are examples, and the technology of the present disclosure is not limited to the following embodiments. For example, the numerical values, shapes, materials, steps, the order of the steps, the layout of the display screen, and the like shown in the following embodiments are merely examples, and various modifications are possible as long as no technical contradiction occurs. Furthermore, one aspect can be combined with another aspect as long as no technical contradiction occurs.

[0016] In this disclosure, the term "agricultural machinery" broadly includes machines that perform basic agricultural tasks in fields, such as tilling, planting, and harvesting. Agricultural machinery is a machine with the function and structure to perform agricultural tasks on the ground in a field, such as tilling, sowing, pest control, fertilizing, planting crops, or harvesting. These agricultural tasks are sometimes referred to as "ground work" or simply "work." Agricultural machinery does not necessarily have to have a traveling device for its own movement; it may travel by being attached to or towed by another vehicle equipped with a traveling device. Furthermore, the term "agricultural machinery" is not limited to cases where a work vehicle, such as a tractor, functions alone as an "agricultural machine." The term "agricultural machinery" may also include cases where the entire work vehicle and an implement attached to or towed by the work vehicle function as a single "agricultural machine." Examples of agricultural machinery include tractors, riding cultivators, vegetable transplanters, mowers, and mobile field robots.

[0017] (Embodiment 1) A crop row detection system and a crop row detection method according to a first exemplary embodiment of the present disclosure will be described.

[0018] The crop row detection system of this embodiment includes an imaging device attached to an agricultural machine for use. The imaging device is fixed to the agricultural machine so as to capture images of the ground on which the agricultural machine travels and acquire time-series color images including at least a portion of the ground.

[0019] FIG. 1 schematically shows how an imaging device 120 attached to an agricultural machine 100, such as a tractor or a riding cultivation machine, captures an image of the ground 10. In the example of FIG. 1, the agricultural machine 100 includes a travelable vehicle body 110, and the imaging device 120 is fixed to the vehicle body 110. For reference, FIG. 1 also shows a body coordinate system Σb having Xb, Yb, and Zb axes that are orthogonal to each other. The body coordinate system Σb is a coordinate system fixed to the agricultural machine 100, and the origin of the body coordinate system Σb can be set, for example, near the center of gravity of the agricultural machine 100. For ease of viewing, the figure shows the origin of the body coordinate system Σb as if it were located outside the agricultural machine 100. In the body coordinate system Σb in the present disclosure, the Xb axis coincides with the traveling direction (the direction of arrow F) when the agricultural machine 100 travels straight. The Yb axis corresponds to the direction directly to the right when looking from the coordinate origin in the positive direction of the Xb axis, and the Zb axis corresponds to the downward vertical direction.

[0020] The imaging device 120 is, for example, an in-vehicle camera having a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) image sensor. The imaging device 120 in this embodiment is, for example, a monocular camera capable of capturing video at a frame rate of 3 frames per second (fps) or higher.

[0021] 2 is a perspective view schematically showing the relationship between the above-mentioned body coordinate system Σb, the camera coordinate system Σc of the image capture device 120, and the world coordinate system Σw fixed to the ground 10. The camera coordinate system Σc has an Xc axis, a Yc axis, and a Zc axis that are orthogonal to each other, and the world coordinate system Σw has an Xw axis, a Yw axis, and a Zw axis that are orthogonal to each other. In the example of FIG. 2, the Xw axis and the Yw axis of the world coordinate system Σw are on a reference plane Re that extends along the ground 10.

[0022] The imaging device 120 is attached to a predetermined position on the agricultural machine 100 so as to face in a predetermined direction. Therefore, the position and orientation of the camera coordinate system Σc relative to the body coordinate system Σb are fixed to a known state. The Zc axis of the camera coordinate system Σc is on the camera optical axis λ1. In the example shown in the figure, the camera optical axis λ1 is inclined from the traveling direction F of the agricultural machine 100 toward the ground 10, and the depression angle Φ is greater than 0°. The traveling direction F of the agricultural machine 100 is approximately parallel to the ground 10 on which the agricultural machine 100 is traveling. The depression angle Φ can be set, for example, in the range of 0° to 60°. When the position at which the imaging device 120 is attached is close to the ground 10, the depression angle Φ may be set to a negative value, in other words, the orientation of the camera optical axis λ1 may be set so that the depression angle Φ has a positive elevation angle.

[0023] When the agricultural machine 100 is traveling on the ground 10, the body coordinate system Σb and the camera coordinate system Σc translate relative to the world coordinate system Σw. If the agricultural machine 100 rotates or swings in the pitch, roll, or yaw directions while traveling, the body coordinate system Σb and the camera coordinate system Σc may rotate relative to the world coordinate system Σw. For simplicity's sake, in the following description, it is assumed that the agricultural machine 100 does not rotate in the pitch or roll directions, but moves approximately parallel to the ground 10.

[0024] FIG. 3 is a top view schematically illustrating a portion of a farm field in which multiple crop rows 12 are laid on the ground 10. The crop rows 12 are rows formed by planting crops continuously in one direction on the ground 10 of the farm field. In other words, the crop rows 12 are groups of crops planted in ridges in the field. Because each crop row 12 is thus formed by a group of crops planted in the field, the shape of the crop rows is complex, strictly speaking, depending on the shape and arrangement of the crops. The width of the crop rows 12 changes depending on the growth of the crops.

[0025] Between adjacent crop rows 12, there is a strip-shaped intermediate region 14 where no crops are planted. Each intermediate region 14 is an area sandwiched between two edge lines E facing each other between two adjacent crop rows 12. When multiple crops are planted in a single ridge in the width direction of the ridge, multiple crop rows 12 will be formed on the ridge. In other words, multiple crop rows 12 will be formed between the rows of the ridge. In such a case, the edge line E of the crop row 12 located at the end of the ridge in the width direction among the multiple crop rows 12 formed on the ridge serves as the reference for the intermediate region 14. In other words, the intermediate region 14 is located between the edge lines E of the crop rows 12 located at the end of the ridge in the width direction among the edge lines E of the multiple crop rows 12.

[0026] The intermediate area 14 functions as an area (work path) through which the wheels of the agricultural machine 100 pass, and therefore the "intermediate area" may be referred to as the "work path."

[0027] In this disclosure, the "edge line" of a crop row refers to a reference line segment (which may include a curve) for defining a target path along which an agricultural machine travels. Such a reference line segment may be defined as both ends of a strip-shaped area (work path) through which the wheels of the agricultural machine are allowed to pass. A specific method for determining the "edge line" of a crop row will be described later.

[0028] 3 shows a schematic diagram of an agricultural machine 100 entering a field where a crop row 12 is laid. This agricultural machine 100 has left and right front wheels 104F and left and right rear wheels 104R as traveling devices, and pulls an implement 300. The front wheels 104F are steered wheels.

[0029] In the example of FIG. 3 , thick dashed arrows L and R are drawn on the work paths 14 located on both sides of a single central crop row 12. When the agricultural machine 100 travels along the target path indicated by the solid arrow C, the front wheels 104F and rear wheels 104R of the agricultural machine 100 are required to move along the arrows L and R in the work path 14 so as not to step on the crop row 12. In this embodiment, the image capture device 120 attached to the agricultural machine 100 can detect the edge line E of the crop row 12, making it possible to control the steering and traveling of the agricultural machine 100 so that the front wheels 104F and rear wheels 104R move along the work path 14 in the arrows L and R. Controlling the steering and traveling of the agricultural machine 100 in this way based on the edge line E of the crop row may be referred to as "row-following travel control."

[0030] Fig. 4 is a diagram schematically illustrating an example of an image 40 acquired by the imaging device 120 of the agricultural machine 100 shown in Fig. 3. Theoretically, the multiple crop rows 12 and the intermediate area (work path) 14 extending parallel to one another on the ground 10 intersect at a vanishing point P0 on the horizon 11. The reason that the vanishing point P0 is located in the area on the right side of the image 40 is because, as shown in Fig. 3, the traveling direction F of the agricultural machine 100 is inclined with respect to the direction in which the crop rows 12 extend (the direction parallel to the arrow C).

[0031] In this embodiment, using a method described below, it is possible to accurately detect the crop row 12 from such an image 40 and determine the edge line E of the crop row 12, even if the sunlight conditions or the growth state of the crop change. Then, based on the edge line E, it is possible to appropriately generate a path (target path) for the agricultural machine 100 to follow. As a result, it becomes possible to control the travel of the agricultural machine 100 by automatic steering so that the front wheels 104F and rear wheels 104R of the agricultural machine 100 move along the arrows L and R in the work passage 14 (row-following travel control). Such row-following travel control enables precise automatic steering according to the growth state of the crop, which is difficult to achieve with automatic steering technology that uses a positioning system such as GNSS.

[0032] FIG. 5 is a top view that schematically shows a state in which the agricultural machine 100 is steered to adjust the position and orientation (angle in the yaw direction) of the agricultural machine 100 so as to reduce a position error with respect to the target path (arrow C). FIG. 6 is a diagram showing an example of an image 40 captured by the imaging device 120 of the agricultural machine 100 in such a state. The front wheels 104F and rear wheels 104R of the agricultural machine 100 in the state of FIG. 5 are positioned on lines indicated by arrows L and R within the work path 14, respectively. When the agricultural machine 100 travels along the target path C indicated by arrow C in the center, the automatic steering device of the agricultural machine 100 controls the steering angles of the steering wheels so that the front wheels 104F and rear wheels 104R do not deviate from the work path 14.

[0033] The configuration and operation of a crop row detection system according to an embodiment of the present disclosure will be described in detail below.

[0034] 7 , the crop row detection system 1000 according to this embodiment includes the above-described imaging device 120 and a processing device 122 that performs image processing of time-series color images acquired from the imaging device 120. The processing device 122 can be connected to, for example, an automatic steering device 124 that is included in the agricultural machine 100. The automatic steering device 124 is included in, for example, an automatic driving device that controls the traveling of the agricultural machine 100.

[0035] The processing device 122 may be realized by an electronic control unit (ECU) for image recognition. The ECU is an in-vehicle computer. The processing device 122 is connected to the image capture device 120 by a serial signal line such as a wire harness so as to receive image data output by the image capture device 120. A part of the image recognition process performed by the processing device 122 may be performed inside the image capture device 120 (inside the camera module).

[0036] 8 is a block diagram showing an example of the hardware configuration of the processing device 122. The processing device 122 includes a processor 20, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 24, a communication device 26, and a storage device 28. These components are connected to each other via a bus 30.

[0037] The processor 20 is a semiconductor integrated circuit, also referred to as a central processing unit (CPU) or microprocessor. The processor 20 may include a graphics processing unit (GPU). The processor 20 sequentially executes a computer program containing a predetermined set of instructions stored in a read-only memory (ROM) 22, thereby implementing the processing required for the crop row detection of the present disclosure. Part or all of the processor 20 may be a field programmable gate array (FPGA), application specific integrated circuit (ASIC), or application specific standard product (ASSP) equipped with a CPU.

[0038] The communication device 26 is an interface for performing data communication between the processing device 122 and an external computer. The communication device 26 can perform wired communication using a CAN (Controller Area Network) or the like, or wireless communication conforming to the Bluetooth (registered trademark) standard and / or the Wi-Fi (registered trademark) standard.

[0039] The storage device 28 can store data of images acquired from the imaging device 120 or images in the process of being processed. Examples of the storage device 28 include a hard disk drive or a nonvolatile semiconductor memory.

[0040] The hardware configuration of the processing device 122 is not limited to the above example. It is not necessary for part or all of the processing device 122 to be mounted on the agricultural machine 100. By using the communication device 26, it is also possible to have one or more computers located outside the agricultural machine 100 function as part or all of the processing device 122. For example, a server computer connected to a network can function as part or all of the processing device 122. On the other hand, a computer mounted on the agricultural machine 100 may perform all of the functions required of the processing device 122.

[0041] In this embodiment, the processing device 122 acquires time-series color images from the imaging device 120 and executes the following operations S1, S2, and S3. (S1) An enhanced image is generated from the time-series color images, in which the color of the crop row being detected is enhanced. (S2) From the enhanced image, a top view image of the ground is generated, in which first pixels having a crop row color index value equal to or greater than a threshold value and second pixels having this index value less than the threshold value are classified. (S3) Determine the position of the edge line of the crop row based on the index value of the first pixel.

[0042] Specific examples of operations S1, S2, and S3 will be described in detail below.

[0043] A time-series color image is a collection of images captured by the imaging device 120 in a time series. Each image is composed of a group of pixels per frame. For example, if the imaging device 120 outputs images at a frame rate of 30 frames per second, the processing device 122 can acquire a new image at intervals of approximately 33 milliseconds. The speed at which an agricultural machine 100, such as a tractor, travels in a field is relatively slow compared to the speed of an ordinary automobile traveling on a public road, and may be, for example, approximately 10 kilometers per hour or less. At a speed of 10 kilometers per hour, the distance traveled in approximately 33 milliseconds is approximately 6 centimeters. For this reason, the processing device 122 may acquire images at intervals of, for example, 100 to 300 milliseconds, and does not need to process all frames of images captured by the imaging device 120. The acquisition interval for images to be processed by the processing device 122 may be automatically changed by the processing device 122 depending on the traveling speed of the agricultural machine 100.

[0044] FIG. 9 is an image corresponding to image 40 of one frame in a time-series color image sequence acquired by an imaging device (in this example, a monocular camera) mounted on an agricultural machine. The image in FIG. 9 shows rows of crops (crop rows) planted in rows on the ground in a farm field. In this example, the rows of crops are arranged approximately parallel and at equal intervals on the ground, and the camera optical axis of the imaging device faces the traveling direction of the agricultural machine. As described above, the camera optical axis does not need to be parallel to the traveling direction of the agricultural machine, and may be incident on the ground ahead of the traveling direction of the agricultural machine. The mounting position of the imaging device is not limited to this example. When multiple imaging devices are mounted on an agricultural machine, the camera optical axis of some of the imaging devices may be facing in the opposite direction to the traveling direction or in a direction intersecting the traveling direction.

[0045] In operation S1, the processing device 122 in FIG. 7 generates an image (enhanced image) in which the color of the crop rows, which are the detection target, is enhanced based on the time-series color images acquired from the imaging device 120. Crops contain chlorophyll (chlorophyll) to perform photosynthesis when exposed to sunlight (white light). Chlorophyll has a lower light absorption rate for green than for red and blue. Therefore, the spectrum of sunlight reflected by crops exhibits relatively higher values ​​in the green wavelength range compared to the spectrum of sunlight reflected by the soil surface. As a result, the color of crops generally contains a large amount of green components, and the "color of the crop rows" is typically green. However, as will be described later, the "color of the crop rows" is not limited to green.

[0046] The image sensor in the imaging device 120 has a large number of photodetector cells arranged in rows and columns. Each photodetector cell corresponds to a picture element (pixel) that makes up an image and includes an R subpixel that detects the intensity of red light, a G subpixel that detects the intensity of green light, and a B subpixel that detects the intensity of blue light. The light outputs detected by the R subpixel, G subpixel, and B subpixel in each photodetector cell are referred to as the R value, G value, and B value, respectively. Hereinafter, the R value, G value, and B value may be collectively referred to as the "pixel value" or "RGB value." When the R value, G value, and B value are used, a color can be specified by coordinate values ​​in the RGB color space.

[0047] When the color of the crop rows being detected is green, an enhanced image that emphasizes the color of the crop rows is an image in which the RGB values ​​of each pixel in a color image captured by an imaging device are converted into pixel values ​​in which the weight of the G value is relatively large. This conversion of pixel values ​​to generate an enhanced image is defined, for example, as "(2×G value - R value - B value) / (R value + G value + B value)." Here, the denominator (R value + G value + B value) is a normalization factor. Hereinafter, the normalized RGB values ​​are referred to as rgb values, and are defined as r = R value / (R value + G value + B value), g = G value / (R value + G value + B value), and b = B value / (R value + G value + B value). "2×grb" is called the Excess Green Index (ExG).

[0048] FIG. 10 is a diagram showing an enhanced image 42 obtained by converting the RGB values ​​in the image of FIG. 9 into "2×grb." As a result of this conversion, pixels in the image 42 of FIG. 10 where "r+b" is relatively small compared to g are displayed brighter, and pixels where "r+b" is relatively large compared to g are displayed darker. This conversion results in an image (enhanced image) 42 in which the color of the crop rows to be detected ("green" in this example) is emphasized. Relatively bright pixels in the image of FIG. 10 are pixels with a relatively large green component, and belong to the crop area.

[0049] As a "color index value" for enhancing the color of crops, other indices such as the green-red vegetation index (G value - R value) / (G value + R value) may be used in addition to the excess green index (ExG). Also, if the imaging device can also function as an infrared camera, the NDVI (Normalized Difference Vegetation Index) may be used as the "color index value of crop rows."

[0050] Note that each row of crops may be covered with a sheet called "mulch." In such cases, the "color of the crop row" refers to the "color of the object that is arranged in a row covering the crops." Specifically, if the color of the sheet is achromatic black, the "color of the crop row" refers to "black." Also, if the color of the sheet is red, the "color of the crop row" refers to "red." In this way, the "color of the crop row" refers not only to the color of the crop itself, but also to the color of the area that defines the crop row (a color that can be distinguished from the color of the soil surface).

[0051] To generate an enhanced image that emphasizes the "color of the crop rows," conversion from the RGB color space to the HSV color space can be used. The HSV color space is a color space consisting of three components: hue, saturation, and value. Using color information converted from the RGB color space to the HSV color space, low-saturation colors such as black or white can be detected. To detect "black" using the OpenCV library, simply set the hue to its maximum range (0-179), the saturation to its maximum range (0-255), and the value to its maximum range (0-30). To detect "white," simply set the hue to its maximum range (0-179), the saturation to its maximum range (0-255), and the value to its maximum range (200-255). Pixels with hue, saturation, and value values ​​within these ranges are the pixels whose color is to be detected. For example, to detect green pixels, the hue range can be set to, for example, 30-90.

[0052] By generating an image (enhanced image) in which the color of the target crop rows is emphasized, it becomes easier to separate (extract) the crop row area from the other background areas (segmentation).

[0053] Next, operation S2 will be described.

[0054] In operation S2, the processing device 122 generates a top-view image from the enhanced image 42, in which pixels are classified into first pixels having a crop row color index value equal to or greater than a threshold value and second pixels having this index value less than the threshold value. The top-view image is an image viewed from above the ground.

[0055] In this embodiment, the aforementioned excess green index (ExG) is used as the index value for the color of the crop row, and a discrimination threshold is determined using discriminant analysis (Otsu's binarization). FIG. 11 shows a histogram of the excess green index (ExG) in the enhanced image 42 of FIG. 10. The horizontal axis of the histogram represents the excess green index (ExG), and the vertical axis represents the number of pixels in the image (corresponding to frequency of occurrence). FIG. 11 also shows a dashed line indicating the threshold value Th calculated by the discriminant analysis algorithm. The pixels in the enhanced image 42 are classified into two classes based on this threshold value Th. To the right of the dashed line indicating the threshold value Th, the frequency of occurrence of pixels whose excess green index (ExG) is equal to or greater than the threshold value is shown. These pixels are presumed to belong to the crop class. In contrast, to the left of the dashed line indicating the threshold value Th, the frequency of occurrence of pixels whose excess green index (ExG) is less than the threshold value is shown. These pixels are presumed to belong to the class of cover crops such as soil. In this example, the first pixel, whose index value is equal to or greater than the threshold value, corresponds to a "crop pixel." On the other hand, the second pixel whose index value is less than the threshold corresponds to a "background pixel." The background pixel corresponds to an object other than the detection target, such as the surface of the soil, and the intermediate area (work passage) 14 described above may be composed of background pixels. Note that the method for determining the threshold is not limited to the above example, and the threshold may be determined using other methods that utilize machine learning, for example.

[0056] By assigning each pixel constituting the enhanced image 42 to either a "first pixel" or a "second pixel," it is possible to extract a detection target area from the enhanced image 42. In addition, by assigning "zero" to the pixel value of the "second pixel" or by removing the data of the second pixel from the image data, it is possible to mask areas other than the detection target. When determining the area to be masked, a process may be performed in which pixels showing locally high values ​​of the excess green index (ExG) are included in the mask area as noise.

[0057] FIG. 12 shows an example of a top-view image 44, viewed from above the ground, classified into first and second pixels. The top-view image 44 in FIG. 12 was created from the enhanced image 42 in FIG. 10 using the image conversion technology described below. In this top-view image 44, the second pixels, whose crop row color index value (in this example, the green-excess index) is less than the threshold Th, are black pixels (pixels with a brightness set to zero). The area formed by the second pixels is primarily an area where the soil surface is visible. In the top-view image 44 in FIG. 12, black triangular areas exist at the left and right corners of the bottom edge. These triangular areas correspond to areas not visible in the enhanced image 42 in FIG. 10. Note that in the image 40 in FIG. 9 and the enhanced image 42 in FIG. 10, distortion of lines that should be straight is observed in the periphery of the image. This image distortion occurs due to the performance of the camera lens and can be corrected using the camera's internal parameters. Processing such as crop area enhancement, masking, distortion correction, etc. can be called pre-processing, but pre-processing may also include other processing.

[0058] The top-view image 44 in Figure 12 is a bird's-eye view image of a reference plane Re parallel to the ground, viewed from directly above in the normal direction of the reference plane Re. This bird's-eye view image can be generated from the enhanced image 42 in Figure 10 by homography transformation (planar projective transformation). Homography transformation is a type of geometric transformation that can transform points on a certain plane in three-dimensional space into points on any other plane.

[0059] 13 is a perspective view schematically illustrating the positional relationship between the reference plane Re and the camera coordinate system Σc1 of the imaging device in a first posture (position and orientation: pose) and the camera coordinate system Σc2 of the imaging device in a second posture. In the illustrated example, the camera coordinate system Σc1 is inclined so that its Zc axis intersects the reference plane Re at an angle. The imaging device in the first posture corresponds to an imaging device attached to an agricultural machine. In contrast, the camera coordinate system Σc2 has its Zc axis perpendicular to the reference plane Re. In other words, the camera coordinate system Σc2 is positioned so that a bird's-eye view image of the reference plane Re viewed from directly above in the normal direction of the reference plane Re can be obtained.

[0060] A virtual image plane Im1 exists at a position on the Zc axis away from the origin O1 of the camera coordinate system Σc1 by the camera's focal length. The image plane Im1 is orthogonal to the Zc axis and the camera optical axis λ1. Pixel positions on the image plane Im1 are defined by an image coordinate system having mutually orthogonal u and v axes. For example, assume that the coordinates of points P1 and P2 located on the reference plane Re are (X1, Y1, Z1) and (X2, Y2, Z2) in the world coordinate system Σw, respectively. In the example of Figure 13, the Xw and Yw axes of the world coordinate system Σw are on the reference plane Re. Therefore, Z1 = Z2 = 0. The reference plane Re is set to extend along the ground.

[0061] Points P1 and P2 on the reference plane Re are transformed into points p1 and p2 on the image plane Im1 of the imaging device in the first attitude by perspective projection of the pinhole camera model. In the image plane Im1, points p1 and p2 are located at pixel positions indicated by the coordinates (u1, v1) and (u2, v2), respectively.

[0062] When the imaging device is in the second orientation, a virtual image plane Im2 exists at a position that is the focal length of the camera on the Zc axis from the origin O2 of the camera coordinate system Σc2. In this example, the image plane Im2 is parallel to the reference plane Re. The pixel positions on the image plane Im2 are determined by the mutually orthogonal u * axis and v *The image coordinate system has axes P1 and P2 on the reference plane Re, which are projected by perspective projection onto the image plane Im2. * and point p2 * In the image plane Im2, the point p1 * and point p2 * are respectively, (u1 * ,v1 * ) and (u2 * ,v2 * ) is located at the pixel position indicated by the coordinates.

[0063] Given the positional relationship of the camera coordinate systems Σc1 and Σc2 with respect to the reference plane Re (world coordinate system Σw), a homography transformation can be used to transform any point (u, v) on the image plane Im1 to the corresponding point (u * ,v * ) can be obtained. When the coordinates of the points are expressed in a homogeneous coordinate system, such a homography transformation is defined by a 3-row x 3-column transformation matrix H.

number

[0064] The contents of the transformation matrix H are as follows: 11 , h 12 , , h 32 It is defined by the numerical value of

number

[0065] 8 numbers (h 11 , h 12 , , h 32 ) can be calculated using a known algorithm by capturing an image of a calibration board placed on the reference plane Re using the imaging device 120 attached to the agricultural machine 100.

[0066] When the coordinates of a point on the reference plane Re are (X, Y, 0), the coordinates of the corresponding points on the image planes Im1 and Im2 of the respective cameras are associated with the point (X, Y, 0) by the respective homography transformation matrices H1 and H2, as shown in the following equations 3 and 4.

number

number

[0067] The following equation is derived from the above two equations. As is clear from this equation, the transformation matrix H is H2H1 -1 Equal to H1 -1 is the inverse matrix of H1.

number

[0068] The contents of the transformation matrices H1 and H2 depend on the reference plane Re, so when the position of the reference plane Re changes, the contents of the transformation matrix H also change.

[0069] By using such homography transformation, it is possible to generate a top-view image of the ground from an image of the ground captured by an imaging device in a first orientation (an imaging device attached to an agricultural machine). In other words, homography transformation allows the coordinates of any point on the image plane Im1 of the imaging device 120 to be converted into the coordinates of a point on the image plane Im2 of a virtual imaging device in a predetermined orientation with respect to the reference plane Re.

[0070] After calculating the contents of the transformation matrix H, the processing device 122 executes a software program based on the above algorithm to generate an overhead image of the ground 10 from the time-series color images or preprocessed images of the time-series color images.

[0071] In the above description, it was assumed that all points in three-dimensional space (e.g., P1, P2) are located on the reference plane Re (e.g., Z1 = Z2 = 0). If the height of the crop relative to the reference plane Re is not 0, the positions of corresponding points will shift from their correct positions in the top-view image after homography transformation. To suppress an increase in the amount of shift, it is desirable that the height of the reference plane Re be close to the height of the crop to be detected. The ground 10 may have unevenness such as ridges, furrows, and grooves. In such cases, the reference plane Re may be displaced upward from the bottom of such unevenness. The distance of the displacement can be appropriately set according to the unevenness of the ground 10 on which the crop is cultivated.

[0072] Furthermore, when the agricultural machine 100 is traveling on the ground 10, if a roll or pitch movement occurs in the vehicle body 110 (see FIG. 1), the attitude of the image capture device 120 changes, which may change the contents of the transformation matrix H1. In such a case, if the roll and pitch rotation angles of the vehicle body 110 are measured by the IMU, it is possible to correct the transformation matrix H1 and the transformation matrix H in accordance with the change in attitude of the image capture device.

[0073] The processing device 122 in this embodiment generates a top view image of the ground viewed from above using the above-mentioned method, in which the crop row color index value is classified into first pixels whose index value is greater than or equal to a threshold value and second pixels whose index value is less than the threshold value. Then, the processing device 122 executes operation S3.

[0074] Next, operation S3 will be described.

[0075] In operation S3, the processing unit 122 determines the position of the edge line of the crop row based on the index value of the first pixel. Specifically, the processing unit 122 accumulates the index values ​​of the first pixels (pixels whose color index value is equal to or greater than a threshold) along multiple scan lines in the top-view image.

[0076] Fig. 14 is an example of a top-view image 44 showing three crop rows 12. In this example, the direction of the crop rows 12 is parallel to the vertical direction of the image (v-axis direction). Fig. 14 shows a number of scan lines (dashed lines) S that are parallel to the vertical direction of the image (v-axis direction). The processing device 122 accumulates the index values ​​of pixels located on multiple scan lines S for each scan line S to obtain an accumulated value.

[0077] FIG. 15 is a diagram schematically illustrating the relationship between the position of the scanning line S and the integrated value of the index value (a histogram of the integrated value) obtained for the top-view image of FIG. 14. The horizontal axis of FIG. 15 indicates the position of the scanning line S in the horizontal direction of the image (the u-axis direction). In the top-view image 44, if many of the pixels crossed by the scanning line S are first pixels belonging to the crop rows 12, the integrated value of the scanning line S will be large. On the other hand, if many of the pixels crossed by the scanning line S are second pixels (background pixels) belonging to the intermediate region (work passage) 14 between the crop rows 12, the integrated value of the scanning line S will be small. Note that in this embodiment, the intermediate region (work passage) 14 is masked, and the index value of the second pixel is zero.

[0078] The histogram in FIG. 15 includes concave regions where the integrated value is zero or close to zero, and convex regions separated by these concave regions. The concave regions correspond to the intermediate region (work passage) 14, and the convex regions correspond to the crop rows 12. In this embodiment, predetermined positions on both sides of the peak integrated value in the convex region—specifically, the position of the scan line S having an integrated value that is a predetermined percentage of the peak integrated value (for example, a value selected from the range of 60% to 90%)—are determined as the position of the edge line of the crop row 12. The two ends of the arrow W in FIG. 15 indicate the position of the edge line of each crop row 12. Note that in the example of FIG. 15, the position of the edge line of each crop row 12 is the position of the scan line S having a value that is 80% of the peak integrated value of each crop row 12.

[0079] In this embodiment, the second pixels are masked, and then the index values ​​of the crop row color on each scan line S are accumulated. That is, the number of first pixels (pixel count) is not counted for a top-view image binarized based on the classification of the first and second pixels. When counting the number of first pixels, for example, if there are many pixels (classified as first pixels) that slightly exceed the threshold Th due to fallen leaves or weeds, the count value of the first pixels will increase. In contrast, accumulating the index values ​​of the crop row color of the first pixels, rather than the number of first pixels, as in the embodiment of the present disclosure, reduces erroneous determinations due to fallen leaves or weeds and improves the robustness of crop row detection.

[0080] Fig. 16 is an example of a top-view image 44 in which the crop rows 12 extend at an angle. As described with reference to Figs. 3 and 4 , depending on the orientation of the agricultural machine 100, the direction in which the crop rows 12 extend in the image 40 acquired by the imaging device 120 may be tilted to the right or left within the image. When a top-view image 44 is generated from such an image by homography transformation, the direction of the crop rows 12 is tilted from the vertical direction of the image (v-axis direction), as in the example of Fig. 16 .

[0081] FIG. 16 also shows a number of scanning lines (dashed lines) S parallel to the vertical direction (v-axis direction) of the image. When the processing device 122 accumulates the index values ​​of pixels located on these multiple scanning lines S for each scanning line S to obtain an accumulated value, a histogram of the accumulated values ​​is obtained, as shown in FIG. 17. FIG. 17 is a diagram schematically showing the relationship between the position of the scanning line S and the accumulated value of the index values, obtained for the top-view image of FIG. 16. The edge lines of the crop rows 12 cannot be determined from this histogram.

[0082] FIG. 18 is a flowchart showing an example of a procedure for searching for a direction (angle) of the scanning line S parallel to the direction of the crop rows 12 by changing the direction (angle) of the scanning line S.

[0083] In step S10, the direction (angle) of the scanning line S is set. Here, the clockwise angle with respect to the u axis in the image coordinate system is set as θ (see FIGS. 14 and 16). The search for the angle θ can be set within a range of, for example, 60 to 120 degrees, with angle increments of, for example, 1 degree. In this case, in step S1, the angles θ of the scanning line S are set to 60, 61, 62, . . . , 119, and 120 degrees.

[0084] In step S12, the index values ​​for the pixels on the scanning line S extending in the direction of each angle θ are integrated to create a histogram of the integrated values. The histogram will show a different distribution depending on the angle θ.

[0085] In step S14, from the multiple histograms thus obtained, a histogram is selected in which the boundary between the convex and concave portions is sharp, as shown in FIG. 15, and the crop rows 12 are clearly separated from the intermediate region 14, and the angle θ of the scanning line S that generates that histogram is determined.

[0086] In step S16, an edge line of each crop row 12 is determined from the peak value of the histogram corresponding to the angle θ determined in step S14. As described above, the position of the scan line S having an integrated value that is, for example, 0.8 times the peak may be adopted as the edge line.

[0087] When searching for the direction (angle) of the scanning line S, a histogram of the integrated values ​​on the scanning line S at each angle θ may be created each time the angle θ is changed by 1 degree within the search range. Feature quantities (e.g., depth of recesses / height of protrusions, differential values ​​of envelopes, etc.) may be calculated from the waveform of the histogram, and it may be determined whether the direction of the crop rows 12 and the direction of the scanning line S are parallel or not based on the feature quantities.

[0088] The method for determining the angle θ is not limited to the above example. If the direction in which the crop rows extend is known through measurement, the direction of the agricultural machine 100 may be measured using an inertial measurement unit (IMU) mounted on the agricultural machine 100, and the angle θ with respect to the direction in which the crop rows extend may be determined.

[0089] FIG. 19 is a diagram showing an example of an integrated value histogram created from the top-view image of FIG. 12. For the convex portion of the histogram located in the center, the scanning line position 0.8 times the peak value is set as the position of edge line E. In this histogram, the peak of the convex portion becomes lower and wider as the scanning line position moves left and right from the center. This is because, as is clear from the image of FIG. 12, there is little image distortion in the center of the top-view image, but the image distortion increases as the distance from the center to the left and right increases, and the black triangular areas located on both sides of the bottom side decrease the integrated value.

[0090] When crop row detection is used to guide agricultural machinery, the crop rows that need to be accurately detected are in the center or its periphery of the image, so distortions in the areas near the left and right ends of the top-view image can be ignored.

[0091] FIG. 20 is a block diagram showing a series of processes executed by the processing device 122 in this embodiment. As shown in FIG. 20, the processing device 122 executes image acquisition 32, enhanced image generation 33, crop row extraction 34, and homography transformation 35 to obtain a top-view image 44, such as that shown in FIG. 16. The processing device 122 further executes scan line direction determination 36 and edge line position determination 37 to obtain the positions of the edge lines of the crop rows. Subsequently, the processing device 122, or a path generation device that has received information indicating the positions of the edge lines from the processing device 122, can execute target path generation 38 for the agricultural machine based on the edge lines. The target path can be generated so that the wheels of the agricultural machine are maintained within the intermediate region (work path) 14 between the edge lines E. For example, the target path can be generated so that the center of the tire in the width direction passes through the center of two edge lines located at both ends of the intermediate region (work path) 14. According to such a target path, even if the agricultural machine deviates from the target path by a few centimeters while traveling, it is possible to reduce the possibility of the tires encroaching on the crop rows.

[0092] It has been confirmed that the embodiments of the present disclosure enable highly accurate detection of crop rows by suppressing the effects of sunlight conditions that change depending on weather conditions such as front lighting, back lighting, sunny days, cloudy days, and fog, as well as the time of day. It has also been confirmed that highly robust detection of crop rows is possible even when the type of crop (cabbage, broccoli, radish, carrot, lettuce, Chinese cabbage, etc.), growth state (from seedling to mature state), presence or absence of disease, presence or absence of fallen leaves and weeds, and soil color change.

[0093] In the above embodiment, the homography transformation is performed after determining a binarization threshold and extracting a crop region based on pixels equal to or greater than the threshold. However, the step of extracting a crop region may be performed after the homography transformation. Specifically, in the series of processes shown in FIG. 20 , the homography transformation 35 may be performed between the enhanced image generation 33 and the crop row extraction 34, or between the image acquisition 32 and the enhanced image generation 33.

[0094] Hereinafter, another embodiment of a crop row detection method executed by the crop detection system according to the present disclosure will be described.

[0095] FIG. 21 is a diagram for explaining a method of dividing a part or the whole of a top-view image into a plurality of blocks and determining the position of an edge line for each of the plurality of blocks.

[0096] In this embodiment, the processing device 122 divides a part or all of the top-view image 44 into a plurality of blocks. Then, for each of the plurality of blocks, the position of the edge line E of the crop row 12 is determined. In the example shown, three blocks B1, B2, and B3 form a continuous band shape in the horizontal direction of the top-view image. The processing device 122 can determine the edge line of the crop row based on the band shape in a direction different from the traveling direction of the agricultural machine 100.

[0097] FIG. 22 is a diagram showing a schematic diagram of the relationship between the position of the scanning line S and the accumulated index value (accumulated value histogram) for each of blocks B1, B2, and B3 in the top-view image of FIG. 21. When accumulating, the scanning line S is always parallel to the vertical direction of the image. The index value is accumulated on a block-by-block basis, and there is no need to change the direction (angle) of the scanning line S. By shortening the length of the scanning line S, it becomes possible to properly detect the area of ​​the second pixel (background pixel) resulting from the intermediate region (work passage) 14, even if the crop row 12 extends at an angle. This eliminates the need to change the angle of the scanning line S.

[0098] The two ends of the arrow W in Figure 22 indicate the positions of the edge lines of the crop rows determined in each of the blocks B1, B2, and B3. In the example shown in Figure 21, the direction of the crop rows 12 is inclined with respect to the direction of the scan line S. For this reason, as described above, if the scan line position showing a value 0.8 times the peak value of the integrated value histogram is used as the position of the edge line E of the crop row 12, then the position of such edge line E corresponds to both ends of the "width" that passes through the vicinity of the center of the crop row 12 in each of the blocks B1, B2, and B3.

[0099] FIG. 23 shows the crop row centers Wc for each of the blocks B1, B2, and B3 in FIG. 22. The crop row centers Wc are determined from the centers of the arrows W that define the edge lines of the crop rows determined from the integrated value histogram in FIG. 22, and are located at the center of each block in the vertical direction of the image. FIG. 23 shows an example of an approximation line 12C for the crop row centers Wc that belong to the same crop row 12. The approximation line 12C is, for example, a straight line determined so as to minimize the root mean square of the distance (error) from multiple crop row centers Wc for each crop row 12. Such an approximation line 12C corresponds to a line that passes through the center of the crop row 12.

[0100] Fig. 24 is a top view showing an example of edge lines E of crop rows 12 determined from the approximation line 12C in Fig. 23. In this example, the two edge lines E associated with each crop row 12 are spaced apart by the same distance as the length of the arrow W and are equidistant from the approximation line 12C.

[0101] According to this embodiment, it is not necessary to change the direction (angle) of the scanning line, and the edge line E of the crop row 12 can be determined with a smaller amount of calculation. The length of each block in the vertical direction of the image can be set to correspond to a distance of 1 to 2 meters on the ground, for example. In this embodiment, one image is divided into three blocks to determine the integrated value histogram, but the number of blocks may be four or more. The shape of the blocks is not limited to the above example. Within the top-view image, the blocks may have a continuous band shape extending in either the horizontal or vertical direction of the image. The processing device 122 can determine the edge line of the crop row by dividing the image into band-shaped blocks extending in a direction different from the traveling direction of the agricultural machine 100.

[0102] Fig. 25 schematically shows a state in which the crop rows 12 in the top-view image 44 include a curved portion. Fig. 26 schematically shows integrated value histograms for each of blocks B1, B2, and B3 in the top-view image 44 in Fig. 25.

[0103] Figure 27 is a diagram showing examples of the crop row centers Wc in each of the blocks B1, B2, and B3 in Figure 26, and an approximation line 12C for each crop row center Xc. In this example, the approximation line 12C is a curve (e.g., a higher-order curve such as a cubic curve) determined so as to minimize the root mean square of the distance (error) from the crop row center Wc of each crop row 12. Such an approximation line 12C corresponds to a curved line that passes through the center of the crop row 12, which has a curved portion.

[0104] Fig. 28 is a top view showing an example of edge lines E of the crop rows 12 determined from the approximation lines of Fig. 27. The edge lines E are generated in a manner similar to that described with reference to Fig. 24. That is, the two edge lines E associated with each crop row 12 are spaced apart by the same distance as the length of the arrow W and are equidistant from the approximation line 12C.

[0105] As explained above, by dividing the top-view image into multiple blocks and generating a histogram of integrated values ​​for each block, it becomes easy to determine the direction of the crop row, and even if the direction of the crop row changes along the way, it becomes possible to know the direction to which it has changed.

[0106] Any of the above crop row detection methods can be implemented in a computer and performed by causing the computer to perform the desired operations.

[0107] (Embodiment 2) Next, an embodiment of an agricultural machine equipped with the crop row detection system of the present disclosure will be described.

[0108] The agricultural machine in this embodiment is equipped with the above-described crop row detection system. The agricultural machine also includes a control system that performs control to realize automatic steering operation. The control system is a computer system that includes a storage device and a control device, and is configured to control the steering, traveling, and other operations of the agricultural machine.

[0109] In a normal automatic steering operation mode, the control device identifies the position of the agricultural machine using the positioning device, and controls the steering of the agricultural machine so that the agricultural machine travels along a target route based on a previously generated target route. Specifically, the control device controls the steering angle of the steering wheels (e.g., front wheels) of the agricultural machine so that the work vehicle travels along the target route in the field. The agricultural machine in this embodiment is equipped with an automatic steering device configured to not only operate in this normal automatic steering mode, but also to perform automatic travel using "row-following travel control" in a field where crop rows are laid.

[0110] The positioning device includes, for example, a GNSS receiver. Such a positioning device can determine the position of a work vehicle based on signals from GNSS satellites. However, when crop rows are present in a field, even if the positioning device can accurately measure the position of the agricultural machine, the spacing between the crop rows is narrow, and depending on the planting method and growth conditions of the crops, there is a high possibility that the agricultural machine's running gear, such as wheels, may extend beyond the crop rows. However, in this embodiment, by using the aforementioned crop row detection system, it is possible to detect actual crop rows and perform appropriate automatic steering. That is, the automatic steering device provided in the agricultural machine in the embodiment of the present disclosure is configured to control the steering angle of the steering wheels based on the position of the crop row edge line determined by the crop row detection system.

[0111] Furthermore, in the agricultural machine of this embodiment, the processing device of the crop row detection system can monitor the positional relationship between the edge line of the crop row and the steering wheel based on the time-series color images. If a position error signal is generated from this positional relationship, the automatic steering device of the agricultural machine can appropriately adjust the steering angle to reduce the position error signal.

[0112] Fig. 29 is a perspective view showing an example of the appearance of the agricultural machine 100 according to this embodiment. Fig. 30 is a side view schematically showing an example of the agricultural machine 100 with the work implement 300 attached. The agricultural machine 100 according to this embodiment is an agricultural tractor (work vehicle) with the work implement 300 attached. The agricultural machine 100 is not limited to a tractor, and does not necessarily have to be equipped with the work implement 300. The crop row detection technology according to the present disclosure can be used to great effect in small cultivators and vegetable transplanters that can be used for inter-furrow work such as ridge creation, inter-cultivation, hilling, weeding, top dressing, and pest control, for example.

[0113] The agricultural machine 100 in this embodiment includes an imaging device 120, a positioning device 130, and an obstacle sensor 136. Although one obstacle sensor 136 is illustrated in Fig. 29, the obstacle sensor 136 may be provided at multiple locations on the agricultural machine 100.

[0114] As shown in FIG. 30, the agricultural machine 100 includes a vehicle body 110, a prime mover (engine) 102, and a transmission 103. The vehicle body 110 is provided with tires 104 (wheels) and a cabin 105. The tires 104 include a pair of front wheels 104F and a pair of rear wheels 104R. A driver's seat 107, a steering device 106, an operation terminal 200, and a group of switches for operation are provided inside the cabin 105. One of the front wheels 104F and the rear wheels 104R may be a crawler instead of a tire. The agricultural machine 100 may be a four-wheel drive vehicle equipped with four tires 104 as drive wheels, or may be a two-wheel drive vehicle equipped with a pair of front wheels 104F or a pair of rear wheels 104R as drive wheels.

[0115] The positioning device 130 in this embodiment includes a GNSS receiver. The GNSS receiver includes an antenna that receives signals from GNSS satellites and a processing circuit that determines the position of the agricultural machine 100 based on the signals received by the antenna. The positioning device 130 receives GNSS signals transmitted from GNSS satellites and performs positioning based on the GNSS signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, for example, Michibiki), GLONASS, Galileo, and BeiDou. The positioning device 130 in this embodiment is provided on top of the cabin 105, but may be provided in another position.

[0116] The positioning device 130 can further supplement the position data by using signals from an inertial measurement unit (IMU). The IMU can measure the tilt and minute movements of the agricultural machine 100. By supplementing the position data based on the GNSS signals with data acquired by the IMU, it is possible to improve the positioning performance.

[0117] In the examples shown in FIGS. 29 and 30 , an obstacle sensor 136 is provided at the rear of the vehicle body 110. The obstacle sensor 136 may also be provided at a location other than the rear of the vehicle body 110. For example, one or more obstacle sensors 136 may be provided at any of the side and front of the vehicle body 110, and the cabin 105. The obstacle sensor 136 detects objects present around the agricultural machine 100. The obstacle sensor 136 may include, for example, a laser scanner or an ultrasonic sonar. The obstacle sensor 136 outputs a signal indicating the presence of an obstacle when an object is present closer than a predetermined distance from the obstacle sensor 136. Multiple obstacle sensors 136 may be provided at different positions on the body of the agricultural machine 100. For example, multiple laser scanners and multiple ultrasonic sonars may be provided at different positions on the body. By providing such a large number of obstacle sensors 136, it is possible to reduce blind spots in monitoring obstacles around the agricultural machine 100.

[0118] The prime mover 102 is, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 103 can change the propulsive force and travel speed of the agricultural machine 100 by changing speeds. The transmission 103 can also switch the agricultural machine 100 between forward and reverse travel.

[0119] The steering device 106 includes a steering wheel, a steering shaft connected to the steering wheel, and a power steering device that assists steering by the steering wheel. The front wheels 104F are steerable wheels, and the traveling direction of the agricultural machine 100 can be changed by changing the turning angle (also referred to as the "steering angle"). The steering angle of the front wheels 104F can be changed by operating the steering wheel. The power steering device includes a hydraulic device or an electric motor that supplies an assisting force for changing the steering angle of the front wheels 104F. When automatic steering is performed, the steering angle is automatically adjusted by the force of the hydraulic device or electric motor under control of a control device arranged in the agricultural machine 100.

[0120] A coupling device 108 is provided at the rear of the vehicle body 110. The coupling device 108 includes, for example, a three-point support device (also referred to as a "three-point link" or "three-point hitch"), a PTO (Power Take Off) shaft, a universal joint, and a communication cable. The working implement 300 can be attached to and detached from the agricultural machine 100 using the coupling device 108. The coupling device 108 can control the position or attitude of the working implement 300 by raising and lowering the three-point link using, for example, a hydraulic device. Power can also be transmitted from the agricultural machine 100 to the working implement 300 via the universal joint. The agricultural machine 100 can cause the working implement 300 to perform a predetermined task while towing the working implement 300. The coupling device may be provided at the front of the vehicle body 110. In this case, the working implement can be connected to the front of the agricultural machine 100.

[0121] The implement 300 shown in Figure 30 is, for example, a rotary cultivator. The implement 300, which is towed by or attached to a work vehicle such as a tractor when traveling through crop rows, can be any implement that can be used for inter-row work such as ridge creation, inter-cultivation, soiling, weeding, top dressing, and pest control.

[0122] 31 is a block diagram showing an example of a schematic configuration of the agricultural machine 100 and the work implement 300. The agricultural machine 100 and the work implement 300 can communicate with each other via a communication cable included in the coupling device 108.

[0123] In the example of FIG. 31 , the agricultural machine 100 includes an imaging device 120, a positioning device 130, an obstacle sensor 136, and an operation terminal 200, as well as a drive unit 140, a steering wheel sensor 150, a turning angle sensor 152, a control system 160, a communication interface (IF) 190, an operation switch group 210, and a buzzer 220. The positioning device 130 includes a GNSS receiver 131 and an inertial measurement unit (IMU) 125. The control system 160 includes a storage device 170 and a control device 180. The control device 180 includes multiple electronic control units (ECUs) 181 to 186. The work implement 300 includes a drive unit 340, a control device 380, and a communication interface (IF) 390. Note that FIG. 31 shows components that are relatively closely related to the automatic steering or automatic driving operation of the agricultural machine 100, and does not show other components.

[0124] The positioning device 130 uses GNSS to locate the position of the agricultural machine 100. If the positioning device 130 is equipped with an RTK receiver, correction signals transmitted from a reference station are used in addition to GNSS signals transmitted from multiple GNSS satellites. The reference station may be installed around the field in which the agricultural machine 100 travels (for example, within 10 km of the agricultural machine 100). The reference station generates correction signals based on the GNSS signals received from the multiple GNSS satellites and transmits them to the positioning device 130. A GNSS receiver 131 in the positioning device 130 receives GNSS signals transmitted from the multiple GNSS satellites. The positioning device 130 performs positioning by calculating the position of the agricultural machine 100 based on the GNSS signals and the correction signals. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position information including latitude, longitude, and altitude information is acquired through high-precision positioning using RTK-GNSS. The positioning method is not limited to RTK-GNSS, and any positioning method (such as interferometric positioning or relative positioning) that can obtain position information with the required accuracy can be used. For example, positioning may be performed using a Virtual Reference Station (VRS) or a Differential Global Positioning System (DGPS).

[0125] The IMU 135 includes a three-axis acceleration sensor and a three-axis gyroscope. The IMU 135 may also include a direction sensor such as a three-axis geomagnetic sensor. The IMU 135 functions as a motion sensor and can output signals indicating various quantities such as the acceleration, velocity, displacement, and attitude of the agricultural machine 100. The positioning device 130 can estimate the position and orientation of the agricultural machine 100 with higher accuracy based on the signal output from the IMU 135 in addition to the GNSS signal and correction signal. The signal output from the IMU 135 can be used to correct or complement the position calculated based on the GNSS signal and correction signal. The IMU 135 outputs a signal at a higher frequency than the GNSS signal. The high-frequency signal can be used to measure the position and orientation of the agricultural machine 100 at a higher frequency (for example, 10 Hz or higher). Instead of the IMU 135, a three-axis acceleration sensor and a three-axis gyroscope may be provided separately. The IMU 135 may also be provided as a device separate from the positioning device 130.

[0126] The positioning device 130 may include other types of sensors in addition to the GNSS receiver 131 and the IMU 135. Depending on the environment in which the agricultural machine 100 travels, the position and orientation of the agricultural machine 100 can be estimated with high accuracy based on data from these sensors.

[0127] By using such a positioning device 130, it is also possible to create a map of the crop rows detected by the above-described crop row detection system 1000.

[0128] The drive device 140 includes various devices necessary for the travel of the agricultural machine 100 and the drive of the work implement 300, such as the aforementioned prime mover 102, transmission 103, differential device including a differential lock mechanism, steering device 106, and coupling device 108. The prime mover 102 includes an internal combustion engine such as a diesel engine. The drive device 140 may include an electric motor for traction instead of or in addition to the internal combustion engine.

[0129] The steering wheel sensor 150 measures the rotation angle of the steering wheel of the agricultural machine 100. The turning angle sensor 152 measures the turning angle of the front wheels 104F, which are steered wheels. The measurement values ​​of the steering wheel sensor 150 and the turning angle sensor 152 are used for steering control by the control device 180.

[0130] The storage device 170 includes one or more storage media, such as a flash memory or a magnetic disk. The storage device 170 stores various data generated by each sensor and the control device 180. The data stored in the storage device 170 may include map data of the environment in which the agricultural machine 100 travels, and target route data for automatic steering. The storage device 170 also stores computer programs that cause each ECU in the control device 180 to execute various operations, which will be described later. Such computer programs may be provided to the agricultural machine 100 via a storage medium (e.g., a semiconductor memory or an optical disk) or an electric communication line (e.g., the Internet). Such computer programs may be sold as commercial software.

[0131] The control device 180 includes multiple ECUs. The multiple ECUs include an ECU 181 for image recognition, an ECU 182 for speed control, an ECU 183 for steering control, an ECU 184 for automatic steering control, an ECU 185 for implement control, an ECU 186 for display control, and an ECU 187 for buzzer control. The ECU 181 for image recognition functions as a processing device for the crop row detection system. The ECU 182 controls the speed of the agricultural machine 100 by controlling the prime mover 102, the transmission 103, and the brakes included in the drive unit 140. The ECU 183 controls the steering of the agricultural machine 100 by controlling the hydraulic device or the electric motor included in the steering device 106 based on measurement values ​​of the steering wheel sensor 150. The ECU 184 performs calculations and control to achieve automatic steering operation based on signals output from the positioning device 130, the steering wheel sensor 150, and the turning angle sensor 152. During automatic steering operation, the ECU 184 sends a command to change the steering angle to the ECU 183. The ECU 183 changes the steering angle by controlling the steering device 106 in response to the command. The ECU 185 controls the operation of the coupling device 108 to cause the work implement 300 to perform a desired operation. The ECU 185 also generates a signal to control the operation of the work implement 300 and transmits the signal from the communication IF 190 to the work implement 300. The ECU 186 controls the display of the operation terminal 200. For example, the ECU 186 realizes various displays on the display device of the operation terminal 200, such as a map of the farm field, detected crop rows, the position and target route of the agricultural machine 100 on the map, pop-up notifications, and setting screens. The ECU 187 controls the output of a warning sound by the buzzer 220.

[0132] Through the operation of these ECUs, the control device 180 realizes operation by manual steering or automatic steering. During normal automatic steering operation, the control device 180 controls the drive device 140 based on the position of the agricultural machine 100 measured or estimated by the positioning device 130 and the target route stored in the storage device 170. In this way, the control device 180 causes the agricultural machine 100 to travel along the target route. On the other hand, in a row-following travel control mode in which the agricultural machine 100 travels along crop rows, the image recognition ECU 181 determines the edge line of the crop row from the detected crop row and generates a target route based on this edge line. The control device 180 executes operations according to this target route.

[0133] The multiple ECUs included in the control device 180 can communicate with each other in accordance with a vehicle bus standard such as CAN (Controller Area Network). In FIG. 31, each of the ECUs 181 to 187 is shown as an individual block, but the functions of each of these may be realized by multiple ECUs. Also, an on-board computer that integrates at least some of the functions of the ECUs 181 to 187 may be provided. The control device 180 may include ECUs other than the ECUs 181 to 187, and any number of ECUs may be provided depending on the functions. Each ECU includes a control circuit including one or more processors.

[0134] The communication IF 190 is a circuit that communicates with the communication IF 390 of the work implement 300. The communication IF 190 transmits and receives signals compliant with ISOBUS standards such as ISOBUS-TIM to and from the communication IF 390 of the work implement 300. This makes it possible to cause the work implement 300 to perform desired operations and to obtain information from the work implement 300. The communication IF 190 may communicate with an external computer via a wired or wireless network. The external computer may be, for example, a server computer in a farming support system that centrally manages information about farm fields on the cloud and supports agriculture by utilizing data on the cloud.

[0135] The operation terminal 200 is a terminal through which a user performs operations related to the traveling of the agricultural machine 100 and the operation of the work implement 300, and is also referred to as a virtual terminal (VT). The operation terminal 200 may include a display device such as a touch screen and / or one or more buttons. By operating the operation terminal 200, a user can perform various operations, such as switching the automatic steering mode on / off, switching the cruise control on / off, setting the initial position of the agricultural machine 100, setting a target route, recording or editing a map, switching between 2WD and 4WD, switching the differential lock on / off, and switching the work implement 300 on / off. At least some of these operations can also be realized by operating the operation switch group 210. The display on the operation terminal 200 is controlled by the ECU 186.

[0136] The buzzer 220 is an audio output device that emits a warning sound to notify the user of an abnormality. For example, the buzzer 220 emits a warning sound when the agricultural machine 100 deviates from the target route by a predetermined distance or more during automatic steering operation. Instead of the buzzer 220, a similar function may be realized by a speaker of the operation terminal 200. The buzzer 220 is controlled by the ECU 186.

[0137] The drive device 340 in the work machine 300 performs operations necessary for the work machine 300 to perform a predetermined task. The drive device 340 includes devices appropriate for the intended use of the work machine 300, such as a hydraulic device, an electric motor, or a pump. The control device 380 controls the operation of the drive device 340. The control device 380 causes the drive device 340 to perform various operations in response to signals transmitted from the agricultural machine 100 via the communication IF 390. The control device 380 can also transmit signals appropriate to the state of the work machine 300 from the communication IF 390 to the agricultural machine 100.

[0138] In the above embodiments, the agricultural machine 100 may be an unmanned work vehicle that operates autonomously. In that case, components that are only required for manned operation, such as a cabin, a driver's seat, a steering wheel, and an operation terminal, may not be provided in the agricultural machine 100. The unmanned work vehicle may perform operations similar to those in the above embodiments by autonomous driving or by remote control by a user.

[0139] A system providing various functions in the embodiments can also be retrofitted to an agricultural machine that does not have those functions. Such a system can be manufactured and sold independently of the agricultural machine. A computer program used in such a system can also be manufactured and sold independently of the agricultural machine. The computer program can be provided, for example, by being stored on a computer-readable non-transitory storage medium. The computer program can also be provided by downloading via a telecommunications line (for example, the Internet). [Industrial Applicability]

[0140] The technology of the present disclosure can be applied to agricultural machinery such as riding tillers, vegetable transplanters, and tractors, for example. [Explanation of symbols]

[0141] 10 Ground, 12 Crop rows, 14 Intermediate area (work passage), 40 Image, 42 Enhanced image, 44 Top view image, 100 Agricultural machinery, 110 Vehicle body, 120 Imaging device, 122 Processing device, 124 Automatic steering device

Claims

1. an imaging device attached to an agricultural machine, which captures images of the ground on which the agricultural machine travels and acquires time-series color images including at least a portion of the ground; a processing device that processes the time-series color images; Equipped with The processing device includes: generating an enhanced image in which the color of the crop row being the detection target is enhanced from the time-series color image; generating a top view image of the ground from above, the top view image being classified into first pixels having an index value of the color of the crop row equal to or greater than a threshold value and second pixels having the index value less than the threshold value, from the enhanced image; integrating the index values ​​of the first pixel along a plurality of scanning lines extending in each of a plurality of angular directions in the top view image to obtain an integrated value, and creating a histogram that associates the positions of the scanning lines with the integrated values ​​for each of the plurality of angular directions; a crop row detection system that refers to the histogram for each of the plurality of angles, selects an angle of a scanning line extending in the direction of the crop row from the plurality of angles, and determines the position of an edge line of the crop row from predetermined positions on both sides of the peak of the integrated value in the histogram associated with the selected angle.

2. The processing device includes: The crop row detection system according to claim 1 , wherein a part or all of the top-view image is divided into a plurality of blocks, and the position of the edge line is determined for each of the plurality of blocks.

3. the plurality of blocks have a continuous band shape in either a horizontal direction or a vertical direction of the image in the top view image, The crop row detection system according to claim 2 , wherein the processing device determines an edge line of the crop row based on a band shape in a direction different from the traveling direction of the agricultural machine.

4. The crop row detection system according to claim 2 or 3, wherein the processing unit determines the direction in which the crop rows extend based on the position of the edge line in each of the plurality of blocks. Tem.

5. the top-view image is a bird's-eye view image of a reference plane along the ground viewed from directly above in a normal direction of the reference plane, The crop row detection system according to claim 1 , wherein the processing device generates the overhead image from the time-series color images or preprocessed images of the time-series color images by homography transformation.

6. 6. The crop row detection system according to claim 5, wherein the reference plane is displaced upward from the bottom of the unevenness in the ground by a predetermined distance set according to the unevenness of the ground on which the crop is planted.

7. The crop row detection system according to claim 1 , wherein the processing device generates and outputs a target path based on the position of the edge line of the crop row.

8. An agricultural machine equipped with a crop row detection system according to any one of claims 1 to 7, a running device including a steering wheel; an automatic steering device that controls the steering angle of the steering wheel based on the position of the edge line of the crop row determined by the crop row detection system; Agricultural machinery equipped with

9. 9. The agricultural machine according to claim 8, wherein the processing device of the crop row detection system monitors a positional relationship between the edge line of the crop row and the steering wheel based on the time-series color images, and provides a position error signal to the automatic steering device.

10. 1. A computer-implemented method for crop row detection, comprising: acquiring, from an imaging device attached to an agricultural machine, time-series color images of the ground on which the agricultural machine is traveling, the color images including at least a portion of the ground; generating an enhanced image in which the color of the crop row being the detection target is enhanced from the time-series color image; generating a top view image of the ground from above, the top view image being classified into first pixels having an index value of the color of the crop row equal to or greater than a threshold value and second pixels having the index value less than the threshold value, from the enhanced image; integrating the index values ​​of the first pixel along a plurality of scanning lines extending in each of a plurality of angular directions in the top view image to obtain an integrated value, and creating a histogram that associates the positions of the scanning lines with the integrated values ​​for each of the plurality of angular directions; referencing the histogram for each of the plurality of angles, selecting an angle of a scanning line extending in the direction of the crop row from the plurality of angles, and determining a position of an edge line of the crop row from predetermined positions on both sides of a peak of the integrated value in the histogram associated with the selected angle; A crop row detection method that causes a computer to execute the above.

Images