Row detection system and agricultural machinery

Abstract

To efficiently execute line detection processing by appropriately selecting a region of interest.SOLUTION: A line detection system includes: a first imaging device which is mounted on an agricultural machine, images the ground and generates a first image of a first region on the ground; a second imaging device which is mounted on the agricultural machine, images the ground and generates a second image of a second region shifted rearward relative to the first region; and a processing device which performs image processing on the first image and the second image. The second imaging device is provided such that at least a portion of each front wheel and at least a portion of each rear wheel are included in the second image. The processing device converts the first image into a first top-view image, converts the second image into a second top-view image, selects a region of interest from the first top-view image on the basis of the position of each front wheel and the position of each rear wheel in the second top-view image and executes line detection processing on the region of interest.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to a row detection system and agricultural machinery equipped with a row detection system. [Background technology]

[0002] Research and development are underway to automate work vehicles such as tractors used in fields. For example, work vehicles that use positioning systems such as GNSS (Global Navigation Satellite System), which enables precise positioning, to automatically steer have been put into practical use. Work vehicles that automatically control speed in addition to automatic steering have also been put into practical use.

[0003] Furthermore, vision guidance systems are being developed that use imaging devices such as cameras to detect rows or ridges of crops in a field, and control the movement of work vehicles along the detected rows or ridges.

[0004] Patent Document 1 discloses a work machine that travels along the ridges of a cultivated field where crops are planted in rows. Patent Document 1 describes a process in which a raw image obtained by photographing the cultivated field from diagonally above with an on-board camera is binarized, and then a planar projection transformed image is generated. In the technology disclosed in Patent Document 1, a number of rotated images with different orientations are generated by rotating the planar projection transformed image, and the work paths between the ridges are detected. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2016-208871 [Overview of the project] [Problems that the invention aims to solve]

[0006] When agricultural machinery uses image recognition technology to automatically steer along rows of crops or furrows, it is necessary to detect these rows with high positional accuracy.

[0007] This disclosure provides a column detection system capable of improving the detection accuracy of column regions, and an agricultural machine equipped with the column detection system. [Means for solving the problem]

[0008] A row detection system according to one aspect of the present disclosure includes: a first imaging device attached to an agricultural machine having a plurality of wheels including a pair of front wheels and a pair of rear wheels, which photographs the ground and generates a first image of a first region on the ground; a second imaging device attached to the agricultural machine, which photographs the ground and generates a second image of a second region on the ground that is shifted behind the first region; and a processing device that performs image processing on the first image and the second image. The second imaging device is provided so as to include at least a portion of each front wheel and at least a portion of each rear wheel in the second image. The processing device converts the first image into a first top view image viewed from above the ground, converts the second image into a second top view image viewed from above the ground, selects a region of interest from the first top view image based on the positions of each front wheel and each rear wheel in the second top view image, and performs a column detection process targeting the region of interest.

[0009] Agricultural machinery according to other aspects of the present disclosure includes the row detection system described above, and an automatic steering device that controls the direction of travel of the agricultural machinery based on the position of crop rows or ridges detected by the row detection system.

[0010] General or specific aspects of the present disclosure may be implemented by an apparatus, a system, a method, an integrated circuit, a computer program, a non-transitory computer-readable storage medium, or any combination of these. The computer-readable storage medium may include volatile storage media, and may also include non-volatile storage media. The apparatus may be constituted by a plurality of devices. When the apparatus is constituted by two or more devices, the two or more devices may be disposed within one device, or may be separately disposed within two or more separate devices. Effects of the Invention

[0011] According to embodiments of the present disclosure, it becomes possible to improve the detection accuracy of row regions such as crop rows or ridges. Brief Description of the Drawings

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of a row detection system according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 2 is a flowchart showing an outline of operations executed by a processing device. [Figure 3A] FIG. 3 is a diagram showing an example of a first image acquired by a first imaging device attached to an agricultural machine. [Figure 3B] FIG. 4 is a diagram showing an example of a second image acquired by a second imaging device attached to an agricultural machine. [Figure 4A] FIG. 5 is a diagram showing an example of a first top-view image viewed from above the ground, which is created by converting the first image. [Figure 4B] FIG. 6 is a diagram showing an example of a second top-view image viewed from above the ground, which is created by converting the second image. [Figure 5] FIG. 7 is a diagram showing a pair of front wheel regions and a pair of rear wheel regions detected from the second top-view image. [Figure 6] FIG. 8 is a diagram showing an example of an edge image detected from the second top-view image. [Figure 7]It is a diagram showing an example of arrangement of front wheel reference points and rear wheel reference points obtained from the second top-view image. [Figure 8] It is a drawing describing an example of a region of interest in the first top-view image. [Figure 9] It is a side view schematically showing how the first imaging device and the second imaging device attached to an agricultural machine photograph the ground. [Figure 10] It is a perspective view schematically showing the relationship among the vehicle coordinate system Σb, the camera coordinate system Σc1 of the first imaging device, the camera coordinate system Σc2 of the second imaging device, and the world coordinate system Σw fixed to the ground. [Figure 11] It is a top view schematically showing a part of a farm field where a plurality of crop rows are provided on the ground. [Figure 12] It is a diagram schematically showing an example of an image acquired by the imaging device of the agricultural machine shown in FIG. 11. [Figure 13] It is a top view schematically showing a state where the traveling direction of the agricultural machine is inclined with respect to the extending direction of the crop rows. [Figure 14] It is a diagram schematically showing an example of an image acquired by the imaging device of the agricultural machine shown in FIG. 13. [Figure 15] It is a top view schematically showing a part of a farm field where a plurality of curved crop rows are provided on the ground. [Figure 16] It is a block diagram showing an example of the hardware configuration of a processing device. [Figure 17] It is a flowchart showing an example of the operation of a processing device. [Figure 18] It is a perspective view schematically showing the positional relationship between the camera coordinate system Σc1 of the imaging device in the first posture, the camera coordinate system Σc3 of the virtual imaging device in the second posture, and the reference plane Re, respectively. [Figure 19] It is a diagram showing examples of the first image, the second image, the first top-view image, and the second top-view image. [Figure 20] It is a diagram showing an example of a composite image generated in the example of FIG. 19. [Figure 21] It is a diagram showing an example of a state where a calibration object placed on the ground is photographed by two imaging devices. [Figure 22] This figure shows an example of an image obtained by photographing a subject for calibration. [Figure 23] This figure shows a composite-enhanced image obtained by converting the RGB values ​​in the composite image shown in Figure 20 to "2 × grb". [Figure 24] This figure shows an example of a binary image obtained by binarizing the composite enhancement image shown in Figure 23. [Figure 25] Figure 23 shows a histogram of the green excess index (ExG) in the composite-weighted image. [Figure 26] This diagram schematically illustrates an example of an image showing three rows of crops. [Figure 27] This figure schematically shows the relationship between the scan line position and the integrated index value obtained for the image shown in Figure 26. [Figure 28] This is an example image showing rows of crops extending diagonally. [Figure 29] This figure schematically shows the relationship between the scan line position and the integrated index value obtained for the image shown in Figure 28. [Figure 30] This flowchart shows an example of a procedure for searching for a scanning line direction parallel to the crop row direction by changing the direction of the scanning line. [Figure 31] This is a perspective view showing an example of the appearance of agricultural machinery. [Figure 32] This is a schematic side view showing an example of agricultural machinery with an implement attached. [Figure 33] This block diagram shows an example of a schematic configuration of agricultural machinery and implements. [Modes for carrying out the invention]

[0013] Embodiments of the present disclosure are described below. However, descriptions that are unnecessarily detailed may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid the following description becoming unnecessarily verbose and to facilitate understanding for those skilled in the art. The inventors provide the accompanying drawings and the following description so that those skilled in the art can fully understand the present disclosure, and not to limit the subject matter described in the claims. In the following description, components having the same or similar function are denoted by the same reference numerals.

[0014] The embodiments described below are illustrative, and the technology of this disclosure is not limited to the embodiments described below. For example, the numerical values, shapes, materials, steps, the order of those steps, the layout of the display screen, etc., shown in the embodiments below are merely examples, and various modifications are possible as long as they do not result in a technical inconsistency. Furthermore, it is possible to combine one embodiment with another as long as it does not result in a technical inconsistency.

[0015] In this disclosure, “agricultural machinery” broadly includes machines that perform basic agricultural tasks in the field, such as “plowing,” “planting,” “harvesting,” and “spraying pesticides.” Agricultural machinery is a machine that has the function and structure to perform agricultural tasks such as tilling, sowing, pest control, fertilizing, planting crops, or harvesting on the ground in the field. These agricultural tasks may be referred to as “ground operations” or simply “operations.” It is not limited to cases where a work vehicle such as a tractor functions as “agricultural machinery” on its own, but rather the entire work vehicle and implements attached to or towed by the work vehicle may function as a single “agricultural machine.” Examples of agricultural machinery include tractors, riding cultivators, vegetable transplanters, lawnmowers, agricultural drones, and field mobile robots.

[0016] (1. Overview of the column detection system) A row detection system according to an embodiment of the present disclosure comprises a plurality of imaging devices attached to an agricultural machine having a plurality of wheels, including a pair of front wheels and a pair of rear wheels. The plurality of imaging devices include a first imaging device that photographs the ground and generates a first image of a first region on the ground, and a second imaging device that photographs the ground and generates a second image of a second region on the ground that is shifted behind the first region. A processing device processes the first and second images to select a region of interest from the first image. The processing device is configured to perform a row detection process targeting the region of interest. According to the row detection system of the present disclosure, the row detection process can be performed efficiently and the computational load can be reduced because the region of interest is selected based on the position of each front wheel and each rear wheel.

[0017] The processing unit may be configured to determine a target path based on the detected row area and output information of the target path to the automatic steering system of the agricultural machinery. The automatic steering system controls the steering of the agricultural machinery so that it moves along the target path. This makes it possible to move the agricultural machinery along crop rows or furrows.

[0018] Figure 1 is a block diagram showing the configuration of a row detection system 1000 according to an exemplary embodiment of the present disclosure. The row detection system 1000 comprises a first imaging device 120, a second imaging device 121, and a processing device 122. Specifically, the first imaging device 120 is mounted on an agricultural machine and photographs the ground to generate a first image of a first region on the ground. The second imaging device 121 is mounted on an agricultural machine and photographs the ground to generate a second image of a second region on the ground that is shifted behind the first region. The agricultural machine comprises a plurality of wheels, including a pair of front wheels and a pair of rear wheels. The second imaging device 121 is configured to include at least a portion of each front wheel and at least a portion of each rear wheel in the second image. The first imaging device 120 and the second imaging device 121 may be mounted on the bottom of the agricultural machine 100. Alternatively, the first imaging device 120 may be provided at the front or top of the agricultural machine 100, and the second imaging device 121 may be provided at the center or rear of the bottom of the agricultural machine 100. In such a configuration, the agricultural machine 100 may be configured to travel across the crop rows 12.

[0019] The processing unit 122 may be connected, for example, to an automatic steering system 124 of an agricultural machine. The processing unit 122 converts the first image into a first top view image viewed from above the ground, and converts the second image into a second top view image viewed from above the ground. The processing unit 122 is configured to select a region of interest from the first top view image based on the positions of each front wheel and each rear wheel in the second top view image, and to perform column detection processing on the region of interest. The processing unit 122 may also be configured to generate a composite image, such as a panoramic planar image, by combining the first top view image and the second top view image.

[0020] Figure 2 is a flowchart outlining the operations performed by the processing unit 122. The processing unit 122 detects crop rows or ridges on the ground by performing the operations in steps S10, S20, and S30 shown in Figure 2.

[0021] In step S10, the processing unit 122 acquires a first image from the first imaging device 120 and a second image from the second imaging device 121. In step S20, the processing unit 122 converts the first image into a first top-view image viewed from above the ground, and converts the second image into a second top-view image viewed from above the ground. These conversions can be performed by homography transformation (planar projection transformation) as described later. In step S30, the processing unit 122 selects a region of interest from the first top-view image based on the positions of each front wheel and each rear wheel in the second top-view image, and performs column detection processing targeting the region of interest.

[0022] Next, we will explain the details of the process described above in step S30.

[0023] Figure 3A shows an example of a first image 41A acquired from the first imaging device 120, and Figure 3B shows an example of a second image 42A acquired from the second imaging device 121. The first image 41A includes at least a portion of each front wheel 4F. The second image 42A includes at least a portion of each front wheel 4F and at least a portion of each rear wheel 4R. In other words, the second image 42A includes at least a portion of each of the four wheels 4F and 4R. The first image 41A and the second image 42A are acquired in step S10 of Figure 2, respectively.

[0024] Figure 4A shows an example of the first top view image 41B, and Figure 4B shows an example of the second top view image 42B. The first top view image 41B and the second top view image 42B are acquired in step S20 of Figure 2, respectively.

[0025] The processing device 122 in this disclosure is configured to detect the regions of a pair of front wheels 4F and a pair of rear wheels 4R in the second top view image 42B. Hereinafter, for simplicity, the "region of the pair of front wheels 4F" may be referred to as the "front wheel region 4F" and the "region of the pair of rear wheels 4R" may be referred to as the "rear wheel region 4R".

[0026] In the example shown in Figure 4B, the front wheel 4F and rear wheel 4R are formed from rims that hold black rubber tires. The rims are formed from, for example, a painted metal material and have a surface that is brighter than the ground and the tires, for example, white. Therefore, by selecting pixels with relatively high brightness from the second top view image 42B, the front wheel region 4F and the rear wheel region 4R in the second top view image 42B can be detected. Pixels with relatively high brightness are images with brightness exceeding a predetermined value. This predetermined value may be fixed in advance, or it may be automatically determined based on the brightness distribution of the pixels in the second top view image 42B. From a collection of pixels with relatively high brightness, a region with high brightness can be determined.

[0027] Furthermore, if the first image 41A and the second image 42A are color images, the processing unit 122 may determine the front wheel region 4F and the rear wheel region 4R in the second top view image 42B based on the color information of the pair of front wheels 4F and the pair of rear wheels 4F. For example, if the rim surface color of the pair of front wheels 4F and the pair of rear wheels 4F is yellow, the front wheel region 4F and the rear wheel region 4R in the second top view image 42B can be detected by selecting yellow pixels from the second top view image 42B, which is a color image. It is also possible to detect the front wheel region 4F and the rear wheel region 4R in the second top view image 42B based on the color information of the tire surface rather than the rims of the pair of front wheels 4F and the pair of rear wheels 4F. However, the tire surface is generally black, or often covered with dirt or mud, so it is not always easy to distinguish from the ground. For this reason, it is easier to accurately determine the front wheel region 4F and the rear wheel region 4R by performing detection processing based on the brightness or color of the rim.

[0028] The processing unit 122 can extract a pair of front wheel reference points FP1 and FP2 from the front wheel region 4F, and a pair of rear wheel reference points RP1 and RP2 from the rear wheel region 4R. In Figure 4B, these front wheel reference points FP1 and FP2 and rear wheel reference points RP1 and RP2 are schematically represented by black squares. The front wheel reference points FP1 and FP2 are pixels that define the distance between the left front wheel region 4F and the right front wheel region 4F among the pixels that make up the front wheel region 4F. The rear wheel reference points RP1 and RP2 are pixels that define the distance between the left rear wheel region 4R and the right rear wheel region 4R among the pixels that make up the rear wheel region 4R. The front wheel reference points FP1 and FP2 are selected as representative points that define the distance between the pair of front wheels 4F. The rear wheel reference points RP1 and RP2 are selected as representative points that define the distance between the pair of rear wheels 4R. As will be described later, it is preferable to determine the region of interest as a region that includes the area between the left and right wheels of the agricultural machine in its center. By selecting such a region of interest and performing image processing on the selected region of interest, crops or furrows on the target path of the agricultural machine can be efficiently detected.

[0029] The specific process for extracting a pair of front wheel reference points FP1 and FP2 from the front wheel region 4F and a pair of rear wheel reference points RP1 and RP2 from the rear wheel region 4R can be performed, for example, as follows.

[0030] First, as shown in Figure 4B, the processing unit 122 determines a vertical reference line VL that divides the second top view image 42B into a left portion and a right portion. The processing unit 122 selects a pair of pixels closest to the vertical reference line VL from the front wheel region 4F as a pair of front wheel reference points FP1 and FP2, and selects a pair of pixels closest to the vertical reference line VL from the rear wheel region 4R as a pair of rear wheel references RP1 and RP2. The processing unit 122 also determines a horizontal reference line LL that is perpendicular to the vertical reference line VL. The horizontal reference line LL includes the front wheel region in the upper portion and the rear wheel region in the lower portion. In this way, by dividing the left portion and the right portion of the second top view image 42B into an upper portion and a lower portion, the processing unit 122 can extract four reference points (FP1, FP2, RP1, RP2) from four portions of a single top view image 42B.

[0031] The width of the region of interest may be formed from a pair of front wheel reference points FP1 and FP2, from a pair of rear wheel reference points RP1 and RP2, or from a pair of front wheel reference points FP1 and FP2 and a pair of rear wheel reference points RP1 and RP2. In this disclosure, the "region of interest" may be, for example, a rectangular region having a predetermined width that extends vertically through the center of the first top view image 41B. The region of interest may have various shapes such as a trapezoid or a sector. Below, a method for determining the predetermined width will be described for the case where the region of interest is a rectangle having a predetermined width.

[0032] The method for determining the pixel positions that define the four reference points (FP1, FP2, RP1, RP2) described above is arbitrary. In embodiments of this disclosure, brightness and edge information of a top-view image are used to extract the four reference points (FP1, FP2, RP1, RP2) from the rims of the front wheel 4F and the rear wheel 4R.

[0033] Figure 5 is an image from the second top view image 42B of Figure 4B, with areas of high brightness shown in white. In the image of Figure 5, the white areas correspond to the front wheel region 4F and the rear wheel region 4R. To identify the multiple pixels that define the contours of these front wheel region 4F and rear wheel region 4R, the processing unit 122 generates an edge image showing the edges of the second top view image 42B using edge detection technology. Figure 6 shows an example of the edge image 42D generated in this way.

[0034] The processing unit 122 determines the overlapping portion between the front wheel region 4F and the rear wheel region 4R and the edge in the second top view image 42B. Specifically, the processing unit 122 determines a group of pixels (reference candidate pixels) located in the overlapping positions of the white area in Figure 5, i.e., the front wheel region 4F and the rear wheel region 4R, and the edge in Figure 6. Next, the processing unit 122 selects a pair of pixels from the front wheel region 4F that are closest to the vertical reference line VL as a pair of front wheel reference points FP1 and FP2, and selects a pair of pixels from the rear wheel region 4R that are closest to the vertical reference line VL as a pair of rear wheel reference points RP1 and RP2. In Figure 6, the front wheel reference points FP1 and FP2, and the rear wheel reference points RP1 and RP2 are schematically represented by black squares.

[0035] Figure 7 shows the front wheel spacing FD, which is defined by the distance between the front wheel reference point FP1 located on the left and the front wheel reference point FP2 located on the right, and the rear wheel spacing RD, which is defined by the distance between the rear wheel reference point RP1 located on the left and the rear wheel reference point RP2 located on the right.

[0036] The processing unit 122 can determine the width of the region of interest based on either the front wheel spacing FD or the rear wheel spacing RD. For example, the width of the region of interest can be determined based on the relatively larger of the two wheel spacings, FD and RD. In this case, the processing unit 122 may use a value obtained by multiplying either the front wheel spacing FD or RD by, for example, a number between 0.9 and 2.0 as the width of the region of interest.

[0037] Figure 8 shows an example of a region of interest (ROI) in the first top view image 41B. The width of this ROI is set to 1.0 times the front wheel distance FD. The ROI selected from the first top view image 41B can also be placed on the first image 41A in Figure 3A by performing a coordinate transformation. This coordinate transformation is the inverse transformation of the transformation used to generate the first top view image 41B from the first image 41A. In the example in Figure 8, the ROI has a rectangular shape with long sides on the left and right that extend parallel to the direction of travel of the agricultural machinery. The length of the short side of this rectangle corresponds to the width of the ROI. The position of the ROI can be set such that the line connecting the midpoints of the pair of short sides passes through the center of the agricultural machinery (e.g., the center of gravity) in the top view. Alternatively, the left long side of the ROI may be determined to pass through the front wheel reference point FP1 or the rear wheel reference point RP1, and the right long side of the ROI may be determined to pass through the front wheel reference point FP2 or the rear wheel reference point RP2. The width and position of the region of interest (ROI) are not limited to the example in Figure 8. As mentioned above, the ROI width may be a value obtained by multiplying either the front wheel spacing (FD) or the rear wheel spacing (RD) by a number between 0.9 and 2.0.

[0038] In the example in Figure 8, the length of the long side of the region of interest (ROI) in the direction of travel of the agricultural machinery is smaller than the size (length) of the first top view image 41B in the direction of travel; however, the length of the long side of the region of interest (ROI) is not limited to this example. If the first image includes an image of the ground extending forward in the direction of travel, as shown in Figure 3A, the region of interest (ROI) may extend forward from the front of the agricultural machinery, for example, 2 meters or more.

[0039] Examples of agricultural machinery 100 on which the row detection system of this disclosure may be installed include work vehicles such as tractors or riding cultivators. The agricultural machinery 100 is configured to travel along crop rows 12 and perform agricultural tasks such as planting, sowing, fertilizing, pest control, harvesting, or plowing crops. The agricultural machinery 100 can detect crop rows 12 and travel along them with automatic steering.

[0040] The first imaging device 120 is mounted at a first position on the agricultural machine 100. The second imaging device 121 is mounted at a second position further back than the first position on the agricultural machine 100. For example, the first imaging device 120 is mounted on the side of the agricultural machine 100 in front of its center of gravity, and the second imaging device 121 is mounted on the side of the agricultural machine 100 further back than its center of gravity. The agricultural machine 100 is configured to travel along the crop rows 12 in a field where crop rows 12 exist, using automatic steering. Note that the positions and orientations of the first imaging device 120 and the second imaging device 121 are not limited to the illustrated examples. The first imaging device 120 may be located at the front of the agricultural machine 100, and the second imaging device 121 may be located at the rear end of the agricultural machine 100 or an implement.

[0041] The processing unit 122 detects the crop rows 12 based on the first top view image, or the first and second top view images. For example, if the first and second images are color images, the processing unit 122 generates an enhanced image based on the first and second images, emphasizing the color (e.g., green) of the crop rows. Then, the processing unit 122 can detect the crop rows 12 on the ground based on the enhanced image. A more detailed example of how to detect the crop rows 12 will be described later.

[0042] After detecting the crop rows 12, the processing unit 122 determines an approximate line (straight or curved) of the crop rows 12 and determines the target path of the agricultural machine 100 along the approximate line. The processing unit 122 outputs the information of the determined target path to the automatic steering device 124 of the agricultural machine 100. The automatic steering device 124 controls the steering of the agricultural machine 100 so that it travels along the target path. This allows the agricultural machine 100 to travel along the crop rows 12.

[0043] As described above, in this embodiment, crop rows 12 are detected from a region of interest of a predetermined width based on a first top-view image generated by the first imaging device 120 and a second top-view image generated by the second imaging device 121. As illustrated, crop rows 12 may also be detected based on a panoramic composite image obtained by combining the first top-view image and the second top-view image. The panoramic composite image contains information for a wider area than each of the first top-view image and the second top-view image. Therefore, compared to the case where crop rows 12 are detected based on only one of the first top-view image or the second top-view image, the approximation line of crop rows 12 can be determined more accurately. In particular, when detecting crop rows 12 such as relatively small seedlings, a decrease in detection accuracy due to missing plants can be suppressed.

[0044] The processing device 122 detects the crop rows 12, but it may also be configured to detect furrows instead of, or in addition to, the crop rows 12.

[0045] The row detection system 1000 may include three or more imaging devices. By detecting crop rows or furrows based on images acquired by the three or more imaging devices, it may be possible to determine the approximate lines of the crop rows or furrows more accurately.

[0046] (2. Specific examples of column detection systems) Next, a more specific example of the row detection system in the embodiments of this disclosure will be described. In this embodiment, "row detection" is performed as detection of crop rows.

[0047] As shown in Figure 1, the row detection system 1000 in this embodiment comprises a first imaging device 120, a second imaging device 121, and a processing device 122. Each of the first imaging device 120 and the second imaging device 121 is fixed to agricultural machinery to acquire a time-series color image including at least a portion of the ground.

[0048] Figure 9 schematically shows how a first imaging device 120 and a second imaging device 121, attached to an agricultural machine 100, photograph the ground 10. In the example in Figure 9, the agricultural machine 100 has a drivable vehicle body 110, and the first imaging device 120 and the second imaging device 121 are fixed to the vehicle body 110. For reference, Figure 9 shows a vehicle coordinate system Σb having mutually orthogonal Xb, Yb, and Zb axes. The vehicle coordinate system Σb is a coordinate system fixed to the agricultural machine 100, and the origin of the vehicle coordinate system Σb may be set, for example, near the center of gravity of the agricultural machine 100. In Figure 9, for ease of viewing, the origin of the vehicle coordinate system Σb is depicted as being located outside the agricultural machine 100. In the vehicle coordinate system Σb in this disclosure, the Xb axis coincides with the direction of travel (direction of arrow F) when the agricultural machine 100 is moving in a straight line. The Yb axis coincides with the direction directly to the right when viewed from the coordinate origin in the positive direction of the Xb axis, and the Zb axis coincides with the direction vertically downward.

[0049] Each of the first imaging device 120 and the second imaging device 121 is, for example, an in-vehicle camera having a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor. Each of the imaging devices 120 and 121 in this embodiment is a monocular camera capable of capturing video at a frame rate of, for example, 3 frames per second (fps) or higher.

[0050] The image sensors in imaging devices 120 and 121 have a large number of light detection cells arranged in rows and columns. Each light detection cell corresponds to a pixel that makes up an image and includes an R subpixel for detecting the intensity of red light, a G subpixel for detecting the intensity of green light, and a B subpixel for detecting the intensity of blue light. The light outputs detected by the R subpixel, G subpixel, and B subpixel in each light detection cell will be called the R value, G value, and B value, respectively. Hereinafter, the R value, G value, and B value may be collectively referred to as "pixel value" or "RGB value". When using the R value, G value, and B value, the color can be defined by the coordinate values ​​in the RGB color space.

[0051] Figure 10 is a schematic perspective view showing the relationship between the vehicle coordinate system Σb described above, the camera coordinate system Σc1 of the first imaging device 120, the camera coordinate system Σc2 of the second imaging device 121, and the world coordinate system Σw fixed to the ground 10. The camera coordinate system Σc1 of the first imaging device 120 has mutually orthogonal Xc1, Yc1, and Zc1 axes. The camera coordinate system Σc2 of the second imaging device 121 has mutually orthogonal Xc2, Yc2, and Zc2 axes. The world coordinate system Σw has mutually orthogonal Xw, Yw, and Zw axes. In the example in Figure 10, the Xw and Yw axes of the world coordinate system Σw lie on a reference plane Re that extends along the ground 10.

[0052] The first imaging device 120 is mounted on the agricultural machine 100 at a first position and facing a first direction. The second imaging device 121 is mounted at a second position behind the first position of the agricultural machine 100 and facing a second direction. Therefore, the positions and orientations of the camera coordinate systems Σc1 and Σc2 with respect to the vehicle coordinate system Σb are fixed in a known state. The Zc1 axis of the camera coordinate system Σc1 of the first imaging device 120 lies on the camera optical axis λ1 of the first imaging device 120. The Zc2 axis of the camera coordinate system Σc2 of the second imaging device 121 lies on the camera optical axis λ2 of the second imaging device 121. In the illustrated example, the camera optical axes λ1 and λ2 are inclined toward the ground 10 from the direction of travel F of the agricultural machine 100, and their depression angle is greater than 0°. The direction of travel F of the agricultural machine 100 is approximately parallel to the ground 10 on which the agricultural machine 100 is traveling. The depression angle of the camera optical axis λ1 of the first imaging device 120 (i.e., the angle between the direction of travel F and the camera optical axis λ1) can be set, for example, in the range of 0° to 90°. The depression angle of the camera optical axis λ2 of the second imaging device 121 (i.e., the angle between the direction of travel F and the camera optical axis λ2) can be set, for example, in the range of 45° to 135°. In the example shown in Figure 9, the depression angle of the camera optical axis λ1 of the first imaging device 120 is approximately 40°, and the depression angle of the camera optical axis λ2 of the second imaging device 121 is approximately 90°. As in this example, the first imaging device 120 can be mounted on the agricultural machine 100 at a first position facing diagonally downwards forward. The second imaging device 121 can be mounted on the agricultural machine 100 at a second position facing downwards. Here, "diagonally downwards forward" refers to the camera optical axis depression angle being in the range of 10° to 80°. "Downward" refers to a camera optical axis having a depression angle between 80° and 100°.

[0053] In the example shown in Figure 9, both the first imaging device 120 and the second imaging device 121 are mounted on the bottom of the vehicle body 110. The second imaging device 121 is mounted further back than the first imaging device 120. That is, the Xb coordinate value of the second imaging device 121 in the vehicle coordinate system Σb is smaller than the Xb coordinate value of the first imaging device 120. In the example in Figure 9, both imaging devices 120 and 121 are located near the center with respect to the width direction (Yb direction) of the vehicle body 110. Each of the imaging devices 120 and 121 may be mounted not only on the bottom of the vehicle body 110, but also on other locations such as the side, front, or rear. When the imaging devices 120 and 121 are provided on the bottom of the vehicle body 110, as in this embodiment, it is possible to detect crop rows or furrows below the agricultural machinery 100.

[0054] Figure 9 illustrates the imaging areas of the first imaging device 120 and the second imaging device 121 with radially extending dotted lines. Each imaging area of ​​the first imaging device 120 and the second imaging device 121 includes the portion of the ground 10 located below the agricultural machinery 100. Each imaging area of ​​the first imaging device 120 and the second imaging device 121 also includes at least a portion of the wheels of the agricultural machinery 1100. Each imaging area of ​​the first imaging device 120 and the second imaging device 121 may have a size that is longer in the front-to-back direction than in the left-to-right direction of the agricultural machinery 100. It is preferable that the imaging areas of the first imaging device 120 and the second imaging device 121 partially overlap. If there is an overlap between the first image generated by the first imaging device 120 and the second image generated by the second imaging device 121, it becomes possible to create a planar panoramic image from the first and second images.

[0055] In this embodiment, the imaging area of ​​the first imaging device 120 includes the portion of the ground 10 located directly below the front axle 125F of the agricultural machine 100. On the other hand, the imaging area of ​​the second imaging device 121 includes the portion of the ground 10 located directly below the rear axle 125R of the agricultural machine 100. Therefore, crop rows or ridges near the front wheel 4F and rear wheel 4R can be detected with high accuracy.

[0056] When the agricultural machine 100 is traveling on the ground 10, the vehicle coordinate system Σb and the camera coordinate systems Σc1 and Σc2 are translated relative to the world coordinate system Σw. If the agricultural machine 100 rotates or oscillates in the pitch, roll, and yaw directions while traveling, the vehicle coordinate system Σb and the camera coordinate system Σc rotate relative to the world coordinate system Σw. For the sake of simplicity, in the following description, the agricultural machine 100 will not rotate in the pitch and roll directions, but will move approximately parallel to the ground 10.

[0057] Figure 11 is a schematic top view showing a portion of a field where multiple crop rows 12 are planted on the ground 10. A crop row 12 is a row formed by planting crops continuously in one direction on the ground 10 of the field. For example, a crop row 12 may be a collection of crops planted in the ridges of the field. Thus, since each crop row 12 is a row formed by a collection of crops planted in the field, the shape of the crop row is, strictly speaking, complex and depends on the shape and arrangement of the crops. The width of the crop row 12 changes according to the growth of the crops. Between adjacent crop rows 12, there is a band-shaped intermediate area 14 where no crops are planted. Each intermediate area 14 is the area sandwiched between two opposing boundary lines E between two adjacent crop rows 12. Note that if multiple crops are planted in the width direction of a ridge, multiple crop rows 12 will be formed on a single ridge. In such cases, the boundary line E of the crop row 12 located at the end of the ridge in the width direction of the ridge becomes the reference point for the intermediate area 14. In other words, the intermediate area 14 is the area between the boundary lines E of the crop row 12 located at the end of the ridge in the width direction of the ridge. Since the intermediate area 14 functions as an area through which the wheels of the agricultural machinery 100 pass, the "intermediate area" is sometimes referred to as a "work passage."

[0058] In this disclosure, the “boundary line” of a crop row means a reference line segment (which may include curves) that defines the target path when agricultural machinery travels along it. Such a reference line segment may be defined as the two ends of a strip-shaped area (working path) through which the wheels of agricultural machinery are permitted to pass. A specific method for determining the “boundary line” of a crop row will be described later.

[0059] Figure 11 schematically shows an agricultural machine 100 traveling through a field where rows of crops 12 are provided. This agricultural machine 100 is equipped with left and right front wheels 104F and left and right rear wheels 104R as its running gear, and is towing an implement 300. The front wheels 104F are steering wheels.

[0060] In the example shown in Figure 11, thick dashed arrows L and R are indicated on the work passages 14 located on either side of a central crop row 12. When the agricultural machine 100 travels along the target path indicated by the solid arrow C, the front wheels 104F and rear wheels 104R of the agricultural machine 100 are required to move along arrows L and R within the work passages 14 so as not to run over the crop row 12. In this embodiment, the boundary line E of the crop row 12 can be detected using imaging devices 120 and 121 attached to the agricultural machine 100. Therefore, it is possible to control the steering and travel of the agricultural machine 100 so that the front wheels 104F and rear wheels 104R move along arrows L and R within the work passages 14. Controlling the steering and travel of the agricultural machine 100 based on the boundary line E of the crop row 12 in this way can be called "row-following travel control".

[0061] Figure 12 schematically shows an example of an image 40 acquired by the first imaging device 120 of the agricultural machine 100 shown in Figure 11. For clarity, the front wheels 104F of the agricultural machine 100, which may be included in the image 40, are omitted from Figure 12. Multiple crop rows 12 and intermediate areas (work passages) 14 extending parallel to each other on the ground 10 theoretically intersect at a vanishing point P0 on the horizon 11. The vanishing point P0 is located in the central area of ​​the image 40.

[0062] Figure 13 is a schematic top view showing a state in which the direction of travel F of the agricultural machine 100 is inclined with respect to the direction in which the crop rows 12 extend. Figure 14 is a schematic diagram showing an example of an image 40 acquired by the first imaging device 120 of the agricultural machine 100 shown in Figure 13. When the direction of travel F of the agricultural machine 100 is inclined with respect to the direction in which the crop rows 12 extend (the direction parallel to arrow C), the vanishing point P0 is located in the right or left region of the image 40. In the example shown in Figure 14, the vanishing point P0 is located in the right region of the image 40.

[0063] The agricultural machine 100 is equipped with a row detection system 1000 and an automatic steering device 124 as shown in Figure 1. The processing device 122 in the row detection system 1000 generates a first top view image and a second top view image based on a first image generated by a first imaging device 120 and a second image generated by a second imaging device 121. In the embodiment of this disclosure, the processing device 122 determines a region of interest by performing the above-described processing. The processing device 122 can also detect crop rows 12 included in the region of interest and obtain an approximation line by linearly approximating the detected crop rows 12. The processing device 122 determines a target path for the agricultural machine 100 along the approximation line. The automatic steering device 124 performs steering control so that the agricultural machine 100 travels along the target path. In this way, the agricultural machine 100 becomes capable of "row-following travel" along the detected crop rows.

[0064] The automatic steering system 124 controls the steering of the agricultural machine 100 to reduce the positional and azimuth deviations of the agricultural machine 100 relative to the target path (arrow C shown in Figure 13). As a result, for example, the agricultural machine 100 in the state shown in Figure 13 has its position and orientation (yaw angle) adjusted to approach the state shown in Figure 11. In the state shown in Figure 11, the left and right wheels of the agricultural machine 100 are located on the lines indicated by arrows L and R in the work path 14, respectively. When the agricultural machine 100 travels along the target path indicated by the central arrow C, the automatic steering system 124 in the agricultural machine 100 controls the steering angle of the steering wheels so that the front wheels 104F and rear wheels 104R do not deviate from the work path 14.

[0065] Figure 15 is a schematic top view showing a portion of a field where multiple curved rows of crops 12 are arranged on the ground 10. According to this embodiment, even in a field where curved rows of crops 12 are formed, the position of the crop rows 12 can be accurately detected from two images acquired by two imaging devices 120 and 121, enabling precise control of the steering and driving of agricultural machinery 100 along the crop rows 12.

[0066] Here, the configuration and operation of the processing unit 122 in the column detection system 1000 will be described in more detail.

[0067] In this embodiment, the processing unit 122 performs image processing on time-series color images acquired from the imaging devices 120 and 121. The processing unit 122 is connected to the automatic steering device 124 of the agricultural machine 100. The automatic steering device 124 may be included, for example, in a control device that controls the movement of the agricultural machine 100.

[0068] The processing unit 122 may be implemented by an electronic control unit (ECU) for image processing. The ECU is an in-vehicle computer. The processing unit 122 is connected to the imaging devices 120 and 121 by serial signal lines, such as a wire harness, to receive image data output by the imaging devices 120 and 121. Some of the image processing performed by the processing unit 122 may be performed inside the imaging devices 120 and 121 (within the camera module).

[0069] Figure 16 is a block diagram showing an example of the hardware configuration of the processing unit 122. The processing unit 122 comprises a processor 20, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 24, a communication device 26, and a storage device 28. These components are interconnected via a bus 30.

[0070] The processor 20 is a semiconductor integrated circuit, also referred to as a central processing unit (CPU) or microprocessor. The processor 20 may include an image processing unit (GPU). The processor 20 sequentially executes a computer program describing a predetermined set of instructions stored in the ROM 22 to realize the processing necessary for the column detection described herein. Part or all of the processor 20 may be an FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), or ASSP (Application Specific Standard Product) equipped with a CPU.

[0071] The communication device 26 is an interface for data communication between the processing unit 122 and an external computer. The communication device 26 can perform wired communication such as CAN (Controller Area Network), or wireless communication compliant with the Bluetooth® standard and / or Wi-Fi® standard.

[0072] The storage device 28 can store images acquired from the imaging device 120, or image data in the process of processing. Examples of the storage device 28 include a hard disk drive or a non-volatile semiconductor memory.

[0073] The hardware configuration of the processing unit 122 is not limited to the example described above. It is not necessary for part or all of the processing unit 122 to be mounted on the agricultural machine 100. By utilizing the communication device 26, one or more computers located outside the agricultural machine 100 can function as part or all of the processing unit 122. For example, a server computer connected to a network may function as part or all of the processing unit 122. Alternatively, a computer mounted on the agricultural machine 100 may perform all the functions required of the processing unit 122.

[0074] Figure 17 is a flowchart showing an example of the operation of the processing unit 122 in this embodiment. The processing unit 122 detects rows of crops on the ground and determines a target path for the agricultural machine 100 along the rows of crops by performing the operations from steps S110 to S160 shown in Figure 17.

[0075] In step S110, the processing unit 122 acquires a first image from the first imaging device 120 and a second image from the second imaging device 121. In this embodiment, each of the first and second images is a time-series color image. A time-series color image is a collection of images generated chronologically by the imaging devices 120 and 121 through photography. Each image is composed of a group of pixels in frame units. For example, if the imaging devices 120 and 121 output images at a frame rate of 30 frames / second, the processing unit 122 can acquire a new image at a cycle of approximately 33 milliseconds. The speed at which agricultural machinery 100 such as a tractor travels in a field is relatively low compared to the speed of a typical automobile traveling on a public road, and may be, for example, 10 kilometers per hour or less. At a speed of 10 kilometers per hour, the distance traveled in approximately 33 milliseconds is approximately 6 centimeters. Therefore, the processing unit 122 may acquire images at a period of, for example, 100 to 300 milliseconds, and it is not necessary to process images of all frames captured by the imaging devices 120 and 121. The image acquisition period for which the processing unit 122 is to be processed may be automatically changed by the processing unit 122 according to the travel speed of the agricultural machine 100.

[0076] In step S120, the processing unit 122 performs homography transformation on the first and second images, which were captured at approximately the same time, to generate the first top view image and the second top view image. If necessary, the processing unit 122 controls the imaging devices 120 and 121 to synchronize the acquisition time of the first image and the acquisition time of the second image.

[0077] The top view image is an overhead view of a reference plane parallel to the ground, viewed from directly above in the direction of the normal to the reference plane. The overhead view image can be generated from the first and second images by homography transformation. Homography transformation is a type of geometric transformation that can transform a point on one plane in three-dimensional space to a point on any other plane. Below, an example of the process of converting the first image acquired by the first imaging device 120 into the first top view image is described. The process of converting the second image acquired by the second imaging device 121 into the second top view image is performed in a similar manner.

[0078] Figure 18 is a schematic perspective view showing the arrangement of the camera coordinate system Σc1 of the imaging device 120 in a first orientation (position and orientation: pose), and the camera coordinate system Σc3 of a hypothetical imaging device in a second orientation, with respect to the reference plane Re. The vehicle coordinate system Σb is also shown in Figure 18. In the example shown in Figure 18, the camera coordinate system Σc1 is tilted so that its Zc axis intersects the reference plane Re at an angle. In contrast, the camera coordinate system Σc3 has its Zc axis perpendicular to the reference plane Re. When the agricultural machine 100 does not rotate in the pitch and roll directions, the plane containing the Xb and Yb axes of the vehicle coordinate system Σb (hereinafter referred to as the "vehicle coordinate system plane") is parallel to the reference plane Re. In this case, the Zc axis of the camera coordinate system Σc3 is also perpendicular to the vehicle coordinate system plane. That is, the camera coordinate system Σc3 is positioned so that an overhead view image can be obtained in the direction of the normals of the reference plane Re and the vehicle coordinate system plane.

[0079] A virtual image plane Im1 exists at a position separated from the origin O1 of the camera coordinate system Σc1 in the Zc axis direction by the focal length of the camera. The image plane Im1 is orthogonal to the Zc axis and the camera optical axis λ1. A pixel position on the image plane Im1 is defined by an image coordinate system having a u-axis and a v-axis that are orthogonal to each other. For example, assume that the coordinates of a point P1 and a point P2 located on the reference plane Re are (X1,Y1,Z1) and (X2,Y2,Z2) in the world coordinate system Σw, respectively. In the example of Fig. 18, the Xw axis and Yw axis of the world coordinate system Σw lie on the reference plane Re. For this reason, Z1=Z2=0. The reference plane Re is set so as to extend along the ground surface.

[0080] The point P1 and the point P2 on the reference plane Re are respectively converted by perspective projection of the pinhole camera model into a point p1 and a point p2 on the image plane Im1 of the imaging device 120 in a first posture. In the image plane Im1, the point p1 and the point p2 are respectively located at pixel positions indicated by coordinates (u1,v1) and (u2,v2).

[0081] When it is assumed that the imaging device is in a second posture, a virtual image plane Im2 exists at a position separated from the origin O3 of the camera coordinate system Σc3 in the Zc axis direction by the focal length of the camera. In this example, the image plane Im2 is parallel to the reference plane Re and the vehicle coordinate system plane. A pixel position on the image plane Im2 has a mutually orthogonal u * axis and v * axis, and is defined by the image coordinate system. This image coordinate system moves together with the vehicle coordinate system Σb relative to the world coordinate system Σw. For this reason, the pixel position on the image plane Im2 can also be defined by the vehicle coordinate system Σb. The point P1 and the point P2 on the reference plane Re are respectively converted by perspective projection into a point p1 * and a point p2 * on the image plane Im2. In the image plane Im2, the point p1 * and the point p2 * are respectively located at (u1 * ,v1 * ) and (u2 * ,v2 *It is located at the pixel position indicated by the coordinates of ).

[0082] Given the arrangement relationship between the camera coordinate systems Σc1 and Σc3 with respect to the reference plane Re in the world coordinate system Σw, a homography transformation allows us to transform any point (u,v) on the image plane Im1 to the corresponding point (u) on the image plane Im2. * ,v * This allows us to determine the homography transformation. When the coordinates of a point are expressed in a homogeneous coordinate system, such a homography transformation is defined by a 3x3 transformation matrix H.

number

[0083] The contents of the transformation matrix H are as follows: 11 h 12 , , , h 32 It is determined by the numerical value.

number

[0084] 8 numbers (h 11 h 12 , , , h 32 The result can be calculated using a known algorithm by photographing a calibration board placed on a reference plane Re with an imaging device 120 attached to the agricultural machine 100.

[0085] If the coordinates of a point on the reference plane Re are (X,Y,0), then the coordinates of the corresponding point in the image planes Im1 and Im2 of each camera are associated with the point (X,Y,0) by the individual homography transformation matrices H1 and H2, as shown in equations 3 and 4 below.

number

number

[0086] From the two equations above, the following equation can be derived. As is clear from this equation, the transformation matrix H is H²H¹ -1 It is equal to H1 -1 This is the inverse matrix of H1.

number

[0087] Since the contents of transformation matrices H1 and H2 depend on the reference plane Re, the contents of transformation matrix H also change when the position of the reference plane Re changes.

[0088] By utilizing this homography transformation, a top-view image of the ground can be generated from an image of the ground acquired by the imaging device 120 in a first orientation. In other words, the homography transformation allows the coordinates of any point on the image plane Im1 of the imaging device 120 to be converted to the coordinates of a point on the image plane Im2 of a virtual imaging device in a predetermined orientation with respect to the reference plane Re.

[0089] After calculating the contents of the transformation matrix H, the processing unit 122 executes a software program based on the above algorithm to generate an overhead view image of the ground 10 from the time-series images output from the imaging device 120. Before generating the overhead view image, preprocessing such as white balance and noise reduction may be applied to the time-series images.

[0090] In the above explanation, it was assumed that points in 3D space (e.g., P1, P2) were both located on the reference plane Re (e.g., Z1=Z2=0). If the height of the crop relative to the reference plane Re is not 0, the position of the corresponding point will shift from its correct position in the top view image after homography transformation. To suppress the increase in the amount of shift, it is desirable that the height of the reference plane Re is close to the height of the crop being detected. The ground 10 may have irregularities such as ridges, levees, and ditches. In such cases, the reference plane Re may be displaced upward from the bottom of such irregularities. The displacement distance can be appropriately set according to the irregularities of the ground 10 on which the crop is planted.

[0091] Furthermore, when the agricultural machine 100 is traveling on the ground 10, if the vehicle body 110 (see Figure 9) undergoes roll or pitch motion, the posture of the imaging device 120 changes, which can change the contents of the transformation matrix H1. In such cases, by measuring the roll and pitch rotation angles of the vehicle body 110 using an inertial measurement unit (IMU), the transformation matrix H1 and transformation matrix H can be corrected according to the change in the posture of the imaging device 120.

[0092] The processing unit 122 can convert the second image acquired by the second imaging device 121 into a second top-view image in the same manner as the conversion from the first image acquired by the first imaging device 120 to the first top-view image. Both the first top-view image and the second top-view image are generated as images on a virtual image plane Im2. The first top-view image and the second top-view image may also be represented in terms of xb and yb coordinates in the vehicle coordinate system Σb. As shown in Figure 9, the imaging area of ​​the first imaging device 120 and the imaging area of ​​the second imaging device 121 partially overlap, so the first top-view image and the second top-view image include overlapping portions (overlapping regions).

[0093] The processing unit 122 may generate a first top-view image and a second top-view image, and then combine them to generate a composite image such as a planar panoramic image. For example, the processing unit 122 may generate a composite image by a synthesis process that includes a process of weighting the pixel values ​​of each pixel in the first overlapping region of the first top-view image that overlaps with the second top-view image, and the pixel values ​​of the corresponding pixels in the second overlapping region of the second top-view image that overlaps with the first top-view image, according to the position of each pixel.

[0094] As shown in Figure 9, when the imaging devices 120 and 121 are installed relatively close to the ground 10, the imaging devices 120 and 121 may be cameras that generate wide-angle images, such as fisheye cameras. Even if the image distortion is large, the image acquired by shooting can be converted into a top-view image by appropriately setting the internal parameters of the camera.

[0095] Figure 19 shows examples of the first image, the second image, the first top view image, and the second top view image. In Figure 19, the upper left figure shows the first image, the lower left figure shows the second image, the upper right figure shows the first top view image, and the lower right figure shows the second top view image. In this example, the first image shows the front wheels of the agricultural machine 100 and the area on the ground located between the front wheels. The second image shows the front and rear wheels of the agricultural machine 100 and the area on the ground located between the front and rear wheels. The processing unit 122 converts the first and second images into the first top view image and the second top view image, respectively, as shown on the right of Figure 19. The lower part of the first top view image and the upper part of the second top view image contain a common subject (the front wheels, the ground and crops between them). This part corresponds to the overlapping area.

[0096] Refer to Figure 17 again. In step S130, the processing unit 122 selects a region of interest from the first top view image based on the positions of each front wheel and each rear wheel in the second top view image. Specifically, the processing unit 122 performs the process described with reference to Figure 7, etc., to select a region of interest ROI as illustrated in Figure 8 from the first top view image.

[0097] In this embodiment, the processing unit 122 can generate a composite image by interpolating pixel values ​​in overlapping regions from the first top-view image and the second top-view image. Note that the creation of the composite image may be omitted.

[0098] Figure 20 shows an example of a composite image generated in the example shown in Figure 19. The region of interest (ROI) is indicated in Figure 20. In this example, the ROI is extended not only to the first top view image but also to the composite second top view image. By detecting crop rows based on such a composite image, it is possible to calculate the approximate line of the crop rows more accurately, even if, for example, there are missing plants.

[0099] Here, an example of a calibration operation to determine the above transformation matrix is ​​described. Calibration is performed based on two images acquired by imaging devices 120 and 121 by photographing a specific subject on the ground. Calibration may be performed before starting to use the agricultural machine 100, or if the position or orientation of the imaging devices 120 and 121 deviates from their initial state. In calibration, the processing unit 122 performs the following operations S1, S2, and S3. (S1) A first reference image is obtained by the first imaging device 120 capturing a specific subject located on the ground, and a second reference image is obtained by the second imaging device 121 capturing the same subject. (S2) Multiple feature points of the subject are extracted from the first reference image and the second reference image, respectively. (S3) A transformation matrix is ​​generated or updated based on the relationship between the positions of multiple feature points in the first reference image and the positions of the corresponding multiple feature points in the second reference image.

[0100] The specific object used for calibration may be, for example, a board with a distinctive pattern drawn on it, such as one used as an AR (Augmented Reality) marker. Alternatively, the wheel of agricultural machinery 100 may be used as the specific object. In the following description, the specific object used for calibration will be referred to as the "calibration object".

[0101] Figure 21 shows an example of a calibration object 18 placed on the ground 10 being photographed by imaging devices 120 and 121. In this example, the calibration object 18 is a board with multiple marks having a distinctive pattern drawn on it, and is placed between the two front wheels of an agricultural machine 100. With the agricultural machine 100 stopped, the first imaging device 120 and the second imaging device 121 photograph the ground on which the calibration object 18 is placed, and generate a first reference image and a second reference image, respectively.

[0102] Figure 22 shows an example of an image obtained by photographing the calibration subject 18. The first imaging device 120 generates a first reference image, as shown in the upper left of Figure 22, by photographing the calibration subject 18. The second imaging device 121 generates a second reference image, as shown in the lower left of Figure 22, by photographing the calibration subject 18. The processing device 122 detects multiple feature points on the calibration subject 18 (for example, the four corners of the board, or each of the four marks) from the first reference image and the second reference image, and calculates a transformation matrix based on their positions.

[0103] The upper right image in Figure 22 shows an example of a top-view image converted from the first reference image. The lower right image in Figure 22 shows an example of a top-view image converted from the second reference image. The processing unit 122 determines the range corresponding to the overlapping region by superimposing multiple feature points of the calibration subject 18 in these top-view images and records this information in the storage device. Subsequent synthesis processing is performed based on this recorded information.

[0104] Refer to Figure 17 again. In step S140, the processing device 122 detects rows of crops on the ground based on the composite image. The processing device 122 can, for example, generate a composite enhanced image from the composite image that emphasizes the color (e.g., green) of the rows of crops, and then detect the boundary lines of the rows of crops on the ground based on the composite enhanced image.

[0105] In step S150, the processing device 122 determines an approximation line for the detected crop row. For example, the processing device 122 determines the approximation line to be a line that passes through the center of the boundary lines at both ends of the detected crop row.

[0106] In step S160, the processing unit 122 determines the target path of the agricultural machine 100 in the vehicle coordinate system along the approximation line. The target path may be determined to overlap, for example, the approximation line of a crop row. The target path may also be set parallel to the approximation line of a crop row or furrow, at a predetermined distance from the approximation line. The processing unit 122 outputs the information of the determined target path to the automatic steering device 124 of the agricultural machine 100. The automatic steering device 124 controls the steering of the agricultural machine 100 so that it travels along the target path.

[0107] Here, a specific example of how to detect crop rows in step S140 will be described. Once the processing device 122 generates a composite image, it can detect crop rows from the composite image by performing the following operations S1, S2, and S3. (S1) A composite-enhanced image is generated from the composite image, with the color of the crop row to be detected emphasized. (S2) From the composite-weighted image, a binary image is generated in which the first pixel of the crop row is classified into a color index value greater than or equal to the threshold, and a second pixel is classified into a color index value less than the threshold. (S3) Based on the index value of the first pixel, the position of the boundary line of the crop row is determined.

[0108] The following provides specific examples of operations S1, S2, and S3.

[0109] The composite image shown in Figure 20 displays rows of crops planted in a field. In this example, the rows of crops are arranged almost parallel and at equal intervals on the ground.

[0110] In operation S1, the processing unit 122 generates a composite-enhanced image based on the composite image, emphasizing the color of the crop row to be detected. Crops have chlorophyll because they perform photosynthesis in response to sunlight (white light). Chlorophyll has a lower absorption rate of green light compared to red and blue light. Therefore, the spectrum of sunlight reflected by crops shows relatively higher values ​​in the green wavelength range compared to the spectrum of sunlight reflected by the soil surface. As a result, the color of crops generally contains a large amount of green components, and the "color of the crop row" is typically green. However, as will be described later, the "color of the crop row" is not limited to green.

[0111] When the color of the crop row being detected is green, the enhanced image, which emphasizes the color of the crop row, is an image in which the RGB values ​​of each pixel in the color image are transformed into pixel values ​​with a relatively large weight for the G value. Such a transformation of pixel values ​​for generating an enhanced image is defined, for example, by "(2 × G value - R value - B value) / (R value + G value + B value)". Here, the denominator (R value + G value + B value) is a factor for normalization. Hereafter, the normalized RGB value will be called the rgb value, and will be defined as r = R value / (R value + G value + B value), g = G value / (R value + G value + B value), and b = B value / (R value + G value + B value). "2 × grb" is called the Excess Green Index (ExG).

[0112] Figure 23 shows a composite-enhanced image obtained by converting the RGB values ​​in the composite image shown in Figure 20 to "2 × grb". This conversion causes pixels in the composite image shown in Figure 20 where "r+b" is relatively small compared to g to appear brighter, and pixels where "r+b" is relatively large compared to g to appear darker. This conversion results in an image (composite-enhanced image) that emphasizes the color of the target crop row (in this example, "green"). In the image shown in Figure 23, the relatively bright pixels are those with a relatively large green component and belong to the crop region.

[0113] In addition to the Excess Green Index (ExG), other indicators such as the Green-Red Vegetation Index (G-value - R-value) / (G-value + R-value) may be used as "color indicator values" to emphasize the color of crops. Furthermore, if the imaging device can also function as an infrared camera, the NDVI (Normalized Difference Vegetation Index) may be used as the "color indicator value for crop rows."

[0114] In addition, each row of crops may be covered with a sheet called a "mulch" (mulching sheet). In such cases, the "color of the crop row" refers to the "color of the object that covers the crops and is arranged in rows." Specifically, if the sheet is achromatic (black), the "color of the crop row" means "black." If the sheet is red, the "color of the crop row" means "red." Thus, the "color of the crop row" refers not only to the color of the crop itself, but also to the color of the area that defines the crop row (a color that can be distinguished from the color of the soil surface).

[0115] To generate enhanced images that emphasize the "color of crop rows," conversion from the RGB color space to the HSV color space may be used. The HSV color space is a color space composed of three components: Hue, Saturation, and Value. By using color information converted from the RGB color space to the HSV color space, it is possible to detect "colors" with low saturation, such as black or white. When detecting "black" using the OpenCV library, set the Hue to the maximum range (0-179), the Saturation to the maximum range (0-255), and the Value range to 0-30. Similarly, when detecting "white," set the Hue to the maximum range (0-179), the Saturation to the maximum range (0-255), and the Value range to 200-255. Pixels with Hue, Saturation, and Value within these set ranges are the pixels that have the color to be detected. For example, to detect green pixels, set the Hue range to, for example, 30-90.

[0116] By generating a composite-enhanced image that emphasizes the color of the target crop row, it becomes easier to separate (extract) the region of the crop row from the rest of the background (segmentation).

[0117] Next, we will explain operation S2.

[0118] In operation S2, the processing unit 122 generates a binary image from the composite-enhanced image, in which the first pixel is classified into a crop row color index value equal to or greater than a threshold, and the second pixel is classified into a crop row color index value less than a threshold. Figure 24 shows an example of a binary image.

[0119] In this embodiment, the aforementioned green excess index (ExG) is used as the color index value for crop rows, and the discrimination threshold is determined by discriminant analysis (Otsu's binarization). Figure 25 is a histogram of the green excess index (ExG) in the composite-weighted image shown in Figure 23. The horizontal axis of the histogram is the green excess index (ExG), and the vertical axis is the number of pixels in the image (corresponding to the frequency of occurrence). Figure 25 shows a dashed line indicating the threshold Th calculated by the discriminant analysis algorithm. Pixels in the composite-weighted image are classified into two classes by this threshold Th. To the right of the dashed line indicating the threshold Th, the frequency of occurrence of pixels with a green excess index (ExG) greater than or equal to the threshold is shown, and these pixels are estimated to belong to the crop class. Conversely, to the left of the dashed line indicating the threshold Th, the frequency of occurrence of pixels with a green excess index (ExG) less than the threshold is shown, and these pixels are estimated to belong to the crop class, such as soil. In this example, the first pixel, which has an index value greater than or equal to the threshold, corresponds to a "crop pixel". On the other hand, the second pixel whose index value is below the threshold corresponds to a "background pixel." Background pixels correspond to objects other than the target of detection, such as the surface of the soil, and the intermediate region (working path) 14 mentioned above may be composed of background pixels. Note that the method for determining the threshold is not limited to the above example, and the threshold may be determined using other methods, such as machine learning.

[0120] By assigning each pixel constituting the composite-enhanced image to either the "first pixel" or the "second pixel," the region to be detected can be extracted from the composite-enhanced image. Furthermore, by assigning "zero" to the pixel value of the "second pixel" or by removing the data of the second pixel from the image data, regions other than the target of detection can be masked. When determining the region to be masked, pixels showing locally high green excess index (ExG) values ​​may be included as noise in the masked region. Through such processing, a binary image classified into first and second pixels can be generated, as shown in Figure 20.

[0121] Next, we will explain operation S3.

[0122] In operation S3, the processing unit 122 determines the position of the boundary line of the crop row 12 based on the index value of the first pixel in the binary image.

[0123] Figure 26 schematically shows an example of an image 44 in which three crop rows 12 are visible in the first top view image. In this example, the direction of the crop rows 12 is parallel to the vertical direction of the image (v-axis direction). Figure 22 shows a number of scan lines (dashed lines) S parallel to the vertical direction of the image (v-axis direction). The processing unit 122 obtains an integrated value by accumulating the index values ​​of pixels located on multiple scan lines S within the region of interest ROI for each scan line S.

[0124] Figure 27 schematically shows the relationship between the position of the scanning line S within the region of interest (ROI) and the integrated index value obtained for image 44 shown in Figure 26. The horizontal axis of Figure 27 indicates the position of the scanning line S in the horizontal direction (u-axis direction) of the image. In image 44, if many of the pixels crossed by the scanning line S are first pixels belonging to the crop row 12, the integrated value of the scanning line S becomes large. On the other hand, if many of the pixels crossed by the scanning line S are second pixels (background pixels) belonging to the intermediate region (working passage) 14 between the crop rows 12, the integrated value of the scanning line S becomes small. In this embodiment, the intermediate region (working passage) 14 is masked, and the index value of the second pixels is zero.

[0125] In the example in Figure 27, there are recessed areas where the cumulative value is zero or close to zero, and convex areas separated by these recessed areas. The recessed areas correspond to the intermediate area (work passage) 14, and the convex areas correspond to the crop rows 12. In this embodiment, the positions of the scanning lines S having cumulative values ​​at predetermined positions on both sides of the peak of the cumulative value in the convex areas, specifically, values ​​at a predetermined ratio (for example, a value selected from the range of 60% to 90%) of the peak of the cumulative value, are determined as the positions of the boundary lines of the crop rows 12. The ends of the arrows W in Figure 23 indicate the positions of the boundary lines of each crop row 12. In the example in Figure 23, the position of the boundary line of each crop row 12 is the position of the scanning line S having a value of 80% of the peak of the cumulative value of each crop row 12.

[0126] In this embodiment, even if the first top-view image contains multiple crop rows, the computational load is reduced because the calculation only needs to be performed on the region of interest (ROI).

[0127] Figure 28 shows an example of a binary image 44 in which multiple crop rows 12 extend diagonally within the first top view image. Depending on the orientation of the agricultural machine 100, the direction in which the crop rows 12 extend in the images acquired by the imaging devices 120 and 121 may be tilted to the right or left within the image. When a top view image is generated from such an image by homography transformation, the direction of the crop rows 12 is tilted from the vertical direction of the image (v-axis direction), as in the example in Figure 28.

[0128] Figure 28 also shows numerous scan lines (dashed lines) S parallel to the vertical direction (v-axis direction) of the image. The processing unit 122 accumulates the index values ​​of pixels located on these multiple scan lines S for each scan line S to obtain an accumulated value for each scan line. Figure 29 schematically shows the relationship between the position of the scan line S and the accumulated index value obtained for the image 44 shown in Figure 28.

[0129] The processing device 122 searches for a scanning line S direction parallel to the direction of the crop row 12 by changing the direction (angle) of the scanning line S. Figure 30 is a flowchart showing an example of a method for searching for a scanning line S direction parallel to the direction of the crop row 12.

[0130] In step S131, the direction (angle) of the scan line S is set. Here, θ is defined as the clockwise angle relative to the u-axis in the image coordinate system (see Figures 22 and 24). The search for the angle θ can be performed by setting the range to, for example, 60 to 120 degrees and the angle step to, for example, 1 degree. In this case, in step S131, the angles θ of the scan line S are given as 60, 61, 62, ..., 119, and 120 degrees.

[0131] In step S132, index values ​​are accumulated for pixels on the scan line S extending in the direction of each angle θ, and data on the distribution of the accumulated values ​​in the direction perpendicular to the scan line is obtained. This data will show different distributions depending on the angle θ.

[0132] In step S133, from the data of the cumulative value distribution in multiple directions obtained in this way, a distribution is selected such that the boundary of the unevenness is steep, as shown in Figure 27, and the crop row 12 is most clearly separated from the intermediate region 14, and the angle θ of the scanning line S that generates that distribution is determined.

[0133] In step S134, the boundary lines for each crop row 12 are determined from the peak values ​​of the distribution corresponding to the angle θ obtained in step S133. As mentioned above, the position of the scanning line S having an integrated value of, for example, 0.8 times the peak can be adopted as the boundary line.

[0134] When searching for the direction (angle) of the scanning line S, the distribution of the cumulative values ​​on the scanning line S for each angle θ is changed by 1 degree within the search range. Features (for example, the depth of the concave portion / height of the convex portion, the derivative of the envelope, etc.) can be calculated from the waveform of the cumulative value distribution, and based on these features, it can be determined whether the direction of the crop row 12 and the direction of the scanning line S are parallel.

[0135] The method for determining the angle θ is not limited to the example above. If the direction in which the crop rows extend is known by measurement, the direction of the agricultural machine 100 may be measured using an IMU mounted on the agricultural machine 100, and the angle θ with respect to the direction in which the crop rows extend may be determined.

[0136] The computational load for calculating the angle θ is also reduced because only processing for the region of interest (ROI) within the first top-view image needs to be performed.

[0137] According to the method described above, the influence of sunlight conditions that vary depending on weather conditions such as direct sunlight, backlighting, sunny, cloudy, and foggy, as well as the time of day, can be suppressed, enabling highly accurate detection of crop rows. Furthermore, it is possible to detect crop rows with high robustness even when the type of crop (such as cabbage, broccoli, radish, carrot, lettuce, and Chinese cabbage), growth stage (from seedling to mature), presence or absence of diseases, presence or absence of fallen leaves and weeds, and soil color changes.

[0138] In the above embodiment, the processing device 122 generates a composite image of the first top view image and the second top view image generated by homography transformation from the first image and the second image, then determines the binarization threshold in the composite image, and extracts crop regions using pixels above the threshold. Alternatively, the processing device 122 may detect crop rows by performing the following steps S11 to S16. (S11) A first enhanced image is generated from the first image, with the colors of the crop rows emphasized. (S12) A second-weighted image is generated from the second image, with the colors of the crop rows emphasized. (S13) From the first weighted image, a first top view image is generated, which is obtained by classifying the pixels of the crop rows into those whose index value is above a threshold and those whose index value is below a threshold. (S14) A second top-view image is generated from the second-weighted image, in which pixels of the crop rows are classified into those whose index value is above a threshold and those whose index value is below a threshold. (S15) Determine the region of interest based on the second top view image. (S16) Based on the first and second top images, rows of crops on the ground are detected.

[0139] This method generates a first top-view image and a second top-view image in which the areas of the crop rows are highlighted, and a region of interest containing the crop rows is selected from these images. This method, like the method described above, can detect crop rows with high accuracy.

[0140] In the example above, crop rows are detected from the composite image, but it is also possible to detect furrows from the composite image. To detect furrows, 3D information (undulation information) of the ground can be obtained using ToF (Time of Flight) technology. Even in that case, it is possible to reduce the computational load by using 3D information within the region of interest.

[0141] Examples of methods for detecting crop rows or furrows as described above are described in detail in the applicant's unpublished international application PCT / JP2022 / 004548. All disclosures of PCT / JP2022 / 004548 are incorporated herein by reference.

[0142] The method for detecting crop rows or ridges formed in a field is not limited to the examples described above, and can be performed using a wide range of known algorithms. For example, a method for linearly approximating crop rows is described in detail in Japanese Patent Publication No. 2624390, of which the applicant is the patent holder. All disclosures of Japanese Patent Publication No. 2624390 are incorporated herein by reference. In addition, a method for detecting lines formed by steps or grooves of ridges is described in Japanese Patent Application Publication No. 2016-146061. All disclosures of Japanese Patent Application Publication No. 2016-146061 are incorporated herein by reference.

[0143] (3. Examples of agricultural machinery configurations) Next, we will explain an example of an agricultural machine configuration.

[0144] The agricultural machinery is equipped with the aforementioned row detection system and a control system that performs control to achieve automatic steering operation. The control system is a computer system comprising a memory device and a control device, and is configured to control the steering, driving, and other operations of the agricultural machinery.

[0145] The control device may be configured to determine the position of the agricultural machinery using a positioning device in a normal automatic steering operation mode, and to control the steering of the agricultural machinery so that it travels along a pre-generated target path. Specifically, the steering angle of the steering wheels (e.g., front wheels) of the agricultural machinery may be controlled so that the work vehicle travels along a target path within the field. The agricultural machinery in this embodiment is equipped with an automatic steering device configured not only for such a normal automatic steering mode, but also for travel using "row-following driving control" in a field where rows of crops or furrows are provided.

[0146] The positioning device includes, for example, a GNSS receiver. Such a positioning device can determine the position of a work vehicle based on signals from GNSS satellites. However, if rows are present in the field, even if the positioning device can accurately measure the position of the agricultural machinery, the spaces between the rows are narrow, and depending on how the crops are planted and their growth, there is a high possibility that the wheels or other running gear of the agricultural machinery may encroach on the rows. However, in this embodiment, by using the row detection system described above, it is possible to detect the rows that actually exist and perform appropriate automatic steering. That is, the automatic steering system of the agricultural machinery in the embodiment of this disclosure is configured to control the steering angle of the steering wheels based on the position of the row boundary line determined by the row detection system.

[0147] Furthermore, in the agricultural machinery of this embodiment, the processing unit of the row detection system can monitor the positional relationship between the row boundary lines and the steering wheels based on a time-series color image. By generating a position error signal from this positional relationship, the automatic steering system of the agricultural machinery can appropriately adjust the steering angle to minimize the position error signal.

[0148] Figure 31 is a perspective view showing an example of the appearance of the agricultural machine 100 in this embodiment. Figure 32 is a schematic side view showing an example of the agricultural machine 100 with the implement 300 attached. The agricultural machine 100 in this embodiment is an agricultural tractor (work vehicle) with the implement 300 attached. The agricultural machine 100 is not limited to a tractor, nor does it need to have the implement 300 attached. The row detection technology in this disclosure can be used, for example, in small cultivators used for inter-row work such as ridging, intertillage, hilling, weeding, top dressing, and pest control, and can be used in vegetable transplanters, demonstrating excellent results.

[0149] The agricultural machine 100 in this embodiment includes imaging devices 120 and 121, a positioning device 130, and an obstacle sensor 136. Although one obstacle sensor 136 is shown as an example in Figure 31, the obstacle sensor 136 may be provided at multiple locations on the agricultural machine 100.

[0150] As shown in Figure 32, the agricultural machine 100 comprises a vehicle body 110, a prime mover (engine) 102, and a transmission 103. The vehicle body 110 is provided with wheels 104 with tires and a cabin 105. The wheels 104 include a pair of front wheels 104F and a pair of rear wheels 104R. Inside the cabin 105 are a driver's seat 107, a steering device 106, an operating terminal 200, and a group of switches for operation. Either the front wheels 104F or the rear wheels 104R may be replaced with multiple wheels (crawlers) fitted with tracks instead of wheels with tires. The agricultural machine 100 may be a four-wheel drive vehicle with four wheels 104 as drive wheels, or a two-wheel drive vehicle with a pair of front wheels 104F or a pair of rear wheels 104R as drive wheels.

[0151] The positioning device 130 in this embodiment includes a GNSS receiver. The GNSS receiver includes an antenna that receives signals from GNSS satellites and a processing circuit that determines the position of the agricultural machine 100 based on the signals received by the antenna. The positioning device 130 receives GNSS signals transmitted from GNSS satellites and performs positioning based on the GNSS signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, e.g., Michibiki), GLONASS, Galileo, and BeiDou. The positioning device 130 in this embodiment is located on top of the cabin 105, but it may be located in other positions.

[0152] The positioning device 130 can further supplement position data using signals from the IMU. The IMU can measure the tilt and minute movements of the agricultural machine 100. By using the data acquired by the IMU to supplement the position data based on the GNSS signal, the positioning performance can be improved.

[0153] In the examples shown in Figures 31 and 32, an obstacle sensor 136 is provided at the rear of the vehicle body 110. The obstacle sensor 136 may also be located at locations other than the rear of the vehicle body 110. For example, one or more obstacle sensors 136 may be provided at any of the locations of the side, front, and cabin 105 of the vehicle body 110. The obstacle sensor 136 detects objects present around the agricultural machine 100. The obstacle sensor 136 may include, for example, a laser scanner and / or ultrasonic sonar. The obstacle sensor 136 outputs a signal indicating the presence of an obstacle when an obstacle is present within a predetermined detection area (search area) from the obstacle sensor 136. Multiple obstacle sensors 136 may be provided at different locations on the body of the agricultural machine 100. For example, multiple laser scanners and multiple ultrasonic sonars may be provided at different locations on the body. By providing many such obstacle sensors 136, blind spots in monitoring obstacles around the agricultural machine 100 can be reduced.

[0154] The prime mover 102 is, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 103 can change the propulsion force and travel speed of the agricultural machine 100 by shifting gears. The transmission 103 can also switch the agricultural machine 100 between forward and reverse.

[0155] The steering system 106 includes a steering wheel, a steering shaft connected to the steering wheel, and a power steering system that assists steering by the steering wheel. The front wheels 104F are steering wheels, and the direction of travel of the agricultural machine 100 can be changed by changing their steering angle (also referred to as the "steering angle"). When manual steering is performed, the steering angle of the front wheels 104F can be changed by the operator operating the steering wheel. The power steering system includes a hydraulic system or electric motor that supplies auxiliary force to change the steering angle of the front wheels 104F. When automatic steering is performed, the steering angle is automatically adjusted by the force of the hydraulic system or electric motor (steering motor) under control from a control device located inside the agricultural machine 100.

[0156] A coupling device 108 is provided at the rear of the vehicle body 110. The coupling device 108 includes, for example, a three-point support device (also referred to as a "three-point link" or "three-point hitch"), a PTO (Power Take Off) shaft, a universal joint, and a communication cable. The coupling device 108 allows the implement 300 to be attached to and detached from the agricultural machine 100. The coupling device 108 can control the position or posture of the implement 300 by raising and lowering the three-point link, for example, by a hydraulic system. Power can also be supplied from the agricultural machine 100 to the implement 300 via the universal joint. The agricultural machine 100 can pull the implement 300 and cause the implement 300 to perform a predetermined task. The coupling device may be provided at the front of the vehicle body 110. In that case, the implement can be connected to the front of the agricultural machine 100.

[0157] The implement 300 shown in Figure 32 is, for example, a rotary cultivator. The implement 300, which is towed or attached to a work vehicle such as a tractor when traveling in a row, can be any type that can be used for inter-row work such as ridging, intertillage, hilling, weeding, top dressing, and pest control.

[0158] Figure 33 is a block diagram showing an example of a schematic configuration of the agricultural machine 100 and the implement 300. The agricultural machine 100 and the implement 300 can communicate with each other via a communication cable included in the coupling device 108.

[0159] In the example shown in Figure 33, the agricultural machine 100 includes a first imaging device 120, a second imaging device 121, a positioning device 130, an obstacle sensor 136, and an operating terminal 200, as well as a drive device 140, a steering wheel sensor 150, a steering angle sensor 152, a control system 160, a communication interface (IF) 190, a group of operating switches 210, and a buzzer 220. The positioning device 130 includes a GNSS receiver 131 and an IMU 135. The control system 160 includes a storage device 170 and a control device 180. The control device 180 includes a plurality of electronic control units (ECUs) 181 to 186. The work machine 300 includes a drive device 340, a control device 380, and a communication interface (IF) 390. Note that Figure 33 shows components that are relatively highly relevant to the automatic steering or automatic driving operation of the agricultural machine 100, and other components are not shown.

[0160] The positioning device 130 uses GNSS to position the agricultural machinery 100. If the positioning device 130 is equipped with an RTK receiver, correction signals transmitted from a base station are used in addition to GNSS signals transmitted from multiple GNSS satellites. The base station may be installed around the field where the agricultural machinery 100 is operating (for example, within 10 km of the agricultural machinery 100). The base station generates correction signals based on the GNSS signals received from multiple GNSS satellites and transmits them to the positioning device 130. The GNSS receiver 131 in the positioning device 130 receives GNSS signals transmitted from multiple GNSS satellites. The positioning device 130 performs positioning by calculating the position of the agricultural machinery 100 based on the GNSS signals and correction signals. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, an error of a few centimeters. Position information, including latitude, longitude, and altitude information, is acquired by high-precision positioning using RTK-GNSS. Furthermore, the positioning method is not limited to RTK-GNSS; any positioning method that can obtain the necessary accuracy of positional information (such as interferometric positioning or relative positioning) can be used. For example, positioning using VRS (Virtual Reference Station) or DGPS (Differential Global Positioning System) may be performed.

[0161] The IMU135 includes a 3-axis accelerometer and a 3-axis gyroscope. The IMU135 may also include a compass sensor, such as a 3-axis geomagnetic sensor. The IMU135 functions as a motion sensor and can output signals indicating various quantities such as acceleration, velocity, displacement, and attitude of the agricultural machine 100. The positioning device 130 can estimate the position and orientation of the agricultural machine 100 with higher accuracy based on the signals output from the IMU135, in addition to the GNSS signals and correction signals. The signals output from the IMU135 can be used to correct or complement the position calculated based on the GNSS signals and correction signals. The IMU135 outputs signals at a higher frequency than the GNSS signals. This high-frequency signal can be used to measure the position and orientation of the agricultural machine 100 at a higher frequency (e.g., 10 Hz or higher). Instead of the IMU135, a 3-axis accelerometer and a 3-axis gyroscope may be provided separately. The IMU135 may be provided as a separate device from the positioning device 130.

[0162] The positioning device 130 may include other types of sensors in addition to the GNSS receiver 131 and IMU 135. Depending on the environment in which the agricultural machine 100 operates, the position and orientation of the agricultural machine 100 can be estimated with high accuracy based on data from these sensors.

[0163] By utilizing such a positioning device 130, it is also possible to create maps of crop rows and furrows detected by the row detection systems 1000 and 2000 described above.

[0164] The drive system 140 includes various devices necessary for driving the agricultural machinery 100 and the implements 300, such as the prime mover 102, the transmission 103, the differential including a differential lock mechanism, the steering system 106, and the coupling device 108. The prime mover 102 is an internal combustion engine, such as a diesel engine. The drive system 140 may also include an electric motor for traction, either in place of or in conjunction with the internal combustion engine.

[0165] The steering wheel sensor 150 measures the rotation angle of the steering wheel of the agricultural machine 100. The steering angle sensor 152 measures the steering angle of the front wheel 104F, which is the steering wheel. The values ​​measured by the steering wheel sensor 150 and the steering angle sensor 152 are used for steering control by the control device 180.

[0166] The storage device 170 includes one or more storage media, such as flash memory or magnetic disks. The storage device 170 stores various data generated by each sensor and the control device 180. The data stored in the storage device 170 may include map data of the environment in which the agricultural machine 100 travels, and data of the target route for automatic steering. The storage device 170 also stores computer programs that cause each ECU in the control device 180 to perform various operations described later. Such computer programs may be provided to the agricultural machine 100 via a storage medium (e.g., semiconductor memory or optical disk) or a telecommunications line (e.g., the Internet). Such computer programs may be sold as commercial software.

[0167] The control device 180 includes a plurality of ECUs. The plurality of ECUs include ECU 181 for image recognition, ECU 182 for speed control, ECU 183 for steering control, ECU 184 for automatic steering control, ECU 185 for implement control, ECU 186 for display control, and ECU 187 for buzzer control. ECU 181 for image recognition functions as a processing unit for the row detection system. ECU 182 controls the speed of the agricultural machine 100 by controlling the prime mover 102, transmission 103, and brakes included in the drive unit 140. ECU 183 controls the steering of the agricultural machine 100 by controlling the hydraulic system or electric motor included in the steering unit 106 based on the measurement value of the steering wheel sensor 150. ECU 184 performs calculations and controls to realize automatic steering operation based on signals output from the positioning device 130, steering wheel sensor 150, and steering angle sensor 152. During automatic steering operation, ECU184 sends a command to ECU183 to change the steering angle. ECU183 changes the steering angle by controlling the steering device 106 in response to the command. ECU185 controls the operation of the coupling device 108 to cause the implement 300 to perform the desired operation. ECU185 also generates signals to control the operation of the implement 300 and transmits these signals to the implement 300 from the communication IF190. ECU186 controls the display on the operation terminal 200. ECU186 enables the display device on the operation terminal 200 to display various things, such as a field map, detected crop rows or furrows, the position and target path of the agricultural machine 100 on the map, pop-up notifications, and a settings screen. ECU187 controls the output of a warning sound by the buzzer 220.

[0168] Through the operation of these ECUs, the control unit 180 enables operation by manual or automatic steering. In normal automatic steering operation, the control unit 180 controls the drive unit 140 based on the position of the agricultural machine 100 measured or estimated by the positioning device 130 and the target path stored in the storage device 170. In this way, the control unit 180 makes the agricultural machine 100 travel along the target path. On the other hand, in the row-following driving control mode, the ECU 181 for image recognition determines the boundary lines of crop rows or furrows from the detected crop rows or furrows and generates a target path based on these boundary lines. The control unit 180 then performs operations according to this target path.

[0169] Multiple ECUs included in the control unit 180 can communicate with each other according to a vehicle bus standard such as CAN (Controller Area Network). In Figure 29, each of the ECUs 181 to 187 is shown as an individual block, but each of these functions may be realized by multiple ECUs. In addition, an on-board computer integrating at least some of the functions of ECUs 181 to 187 may be provided. The control unit 180 may also include ECUs other than ECUs 181 to 187, and any number of ECUs can be provided depending on the function. Each ECU includes a control circuit that includes one or more processors.

[0170] Communication IF190 is a circuit that communicates with the communication IF390 of the implement 300. Communication IF190 transmits and receives signals compliant with the ISOBUS standard, such as ISOBUS-TIM, to and from the communication IF390 of the implement 300. This allows the implement 300 to perform desired operations or to acquire information from the implement 300. Communication IF190 may also communicate with an external computer via a wired or wireless network. The external computer may be, for example, a server computer in an agricultural support system that centrally manages field-related information on the cloud and supports agriculture by utilizing the data on the cloud.

[0171] The control terminal 200 is a terminal used by the operator to perform operations related to the driving of the agricultural machine 100 and the operation of the implement 300, and is also called a virtual terminal (VT). The control terminal 200 may be equipped with a display device such as a touchscreen and / or one or more buttons. By operating the control terminal 200, the operator can perform various operations such as switching the automatic steering mode on / off, switching the cruise control on / off, setting the initial position of the agricultural machine 100, setting a target route, recording or editing a map, switching between 2WD / 4WD, switching the differential lock on / off, and switching the implement 300 on / off. At least some of these operations can also be performed by operating the control switch group 210. The display on the control terminal 200 is controlled by the ECU 186.

[0172] The buzzer 220 is an audio output device that emits a warning sound to notify the operator of an abnormality. For example, during automatic steering operation, the buzzer 220 emits a warning sound if the agricultural machine 100 deviates from the target path by a predetermined distance or more. Instead of the buzzer 220, a similar function may be achieved by the speaker of the operation terminal 200. The buzzer 220 is controlled by the ECU 186.

[0173] The drive unit 340 in the implement 300 performs the operations necessary for the implement 300 to carry out a predetermined operation. The drive unit 340 includes devices such as a hydraulic system, an electric motor, or a pump, depending on the application of the implement 300. The control device 380 controls the operation of the drive unit 340. The control device 380 causes the drive unit 340 to perform various operations in response to signals transmitted from the agricultural machine 100 via the communication IF 390. It can also transmit signals corresponding to the status of the implement 300 to the agricultural machine 100 via the communication IF 390.

[0174] In the embodiments described above, the agricultural machine 100 may be an unmanned, autonomously operated work vehicle. In that case, components necessary only for manned operation, such as a cabin, driver's seat, steering wheel, and operating terminal, may not be provided on the agricultural machine 100. The unmanned work vehicle may perform operations similar to those in each of the embodiments described above, either through autonomous driving or remote operation by an operator.

[0175] The systems providing the various functions in the embodiments can also be retrofitted to agricultural machinery that does not possess those functions. Such systems can be manufactured and sold independently of the agricultural machinery. Computer programs used in such systems can also be manufactured and sold independently of the agricultural machinery. Computer programs can be provided, for example, by being stored in a computer-readable non-temporary storage medium. Computer programs can also be provided by download via telecommunications lines (e.g., the Internet).

[0176] In the embodiments described above, the agricultural machinery 100 is an agricultural work vehicle, but the agricultural machinery 100 is not limited to a work vehicle. The agricultural machinery 100 may be, for example, an agricultural unmanned aerial vehicle (e.g., a drone). Such an unmanned aerial vehicle can be equipped with the row detection system of this disclosure to detect row areas such as rows of crops or furrows on the ground. Such an unmanned aerial vehicle can perform agricultural tasks such as spraying pesticides or fertilizers while flying along the detected row areas.

[0177] As described above, this disclosure includes the column detection system and agricultural machinery described in the following items.

[0178] [Item 1] A first imaging device, which is attached to an agricultural machine having multiple wheels including a pair of front wheels and a pair of rear wheels, and which photographs the ground and generates a first image of a first region on the ground, A second imaging device attached to the agricultural machine, which photographs the ground and generates a second image of a second region on the ground that is shifted backward from the first region, A processing device that performs image processing on the first image and the second image, Equipped with, The second imaging device is provided such that at least a portion of each front wheel and at least a portion of each rear wheel are included in the second image. The aforementioned processing apparatus is The first image is converted into a first top view image as seen from above the ground, The second image is converted into a second top view image as seen from above the ground, A column detection system that selects a region of interest from the first top view image based on the positions of each front wheel and each rear wheel in the second top view image, and performs a column detection process targeting the region of interest.

[0179] [Item 2] The aforementioned processing apparatus is The regions of the pair of front wheels and the regions of the pair of rear wheels in the second top view image are detected. A row detection system according to item 1, which determines the width of the region of interest based on the regions of the pair of front wheels and the regions of the pair of rear wheels.

[0180] [Item 3] The column detection system according to item 1 or 2, wherein the region of interest is a rectangular region having the width that extends vertically through the center of the first top view image.

[0181] [Item 4] The first and second images are color images. The aforementioned processing apparatus is A column detection system according to any one of items 1 to 3, which determines the region of the pair of front wheels and the region of the pair of rear wheels in the second top view image based on the color information of the pair of front wheels and the pair of rear wheels.

[0182] [Item 5] The aforementioned processing apparatus is A pair of front wheel reference points are extracted from the aforementioned region of the pair of front wheels. A pair of rear wheel reference points are extracted from the region of the pair of rear wheels. The column detection system according to item 4, which determines the width of the region of interest based on one or both of the pair of front wheel reference points and the pair of rear wheel reference points.

[0183] [Item 6] The aforementioned processing apparatus is A vertical reference line is determined to divide the second top view image into a left portion and a right portion. From the area of ​​the pair of front wheels, the pair of pixels closest to the vertical reference line is selected as the pair of front wheel reference points. From the region of the pair of rear wheels, the pair of pixels closest to the vertical reference line is selected as the pair of rear wheel reference points. The row detection system according to item 5, which determines the width of the region of interest based on either the front wheel spacing defined by the distance between the pair of front wheel reference points, or the rear wheel spacing defined by the distance between the pair of rear wheel reference points.

[0184] [Item 7] The aforementioned processing apparatus is The column detection system according to item 6, which determines the width of the region of interest based on the distance between the front wheels and the distance between the rear wheels, whichever is not relatively smaller.

[0185] [Item 8] The aforementioned processing apparatus is The column detection system according to item 6, wherein the width of the region of interest is obtained by multiplying one of the front wheel spacing and the rear wheel spacing by a value between 0.9 and 2.0.

[0186] [Item 9] The aforementioned processing apparatus is A horizontal reference line perpendicular to the vertical reference line is determined, which divides the left and right portions of the second top view image into an upper and lower portion, with the region of the pair of front wheels included in the upper portion and the region of the pair of rear wheels included in the lower portion. Edge detection technology is used to detect edges in the second top view image, The overlapping portion of the region of the pair of front wheels and the region of the pair of rear wheels and the edge in the second top view image is determined. The column detection system according to item 6, wherein in each of the four parts divided by the vertical reference line and the horizontal reference line, the pixel closest to the vertical reference line in the overlapping portion is selected as the front wheel reference point and the rear wheel reference point.

[0187] [Item 10] The first and second images are color images. The aforementioned processing apparatus is From the region of interest in the first top view image, a first top view enhanced image is generated in which the colors of the crop rows are emphasized. From the first top-view enhanced image, a first top-view binarized image is generated in which pixels of the crop row are classified into those whose color index value is greater than or equal to a threshold and those whose index value is less than the threshold. Based on the first top-view binarized image, the rows of crops on the ground are detected. A column detection system as described in any one of items 1 through 9.

[0188] [Item 11] The aforementioned processing apparatus is The region of interest in the first top view image is extended into the second top view image. From the region of interest in the second top view image, a second top view enhanced image is generated in which the colors of the crop rows are emphasized. From the second top-view enhanced image, a second top-view binarized image is generated in which pixels of the crop row are classified into those whose index value is greater than or equal to the threshold and those whose index value is less than the threshold. A row detection system according to item 10, which detects rows of crops on the ground based on the first top-view binarized image and the second top-view binarized image.

[0189] [Item 12] The first imaging device is mounted facing diagonally downwards and forwards, The second imaging device is a column detection system according to any one of items 1 to 11, mounted facing downward.

[0190] [Item 13] The aforementioned agricultural machinery is a work vehicle, The second imaging device is a row detection system according to any one of items 1 to 12, mounted on the underside of the work vehicle.

[0191] [Item 14] The processing device generates a target path based on the location of the crop row or furrow determined by the row detection, according to any one of items 1 to 12.

[0192] [Item 15] A column detection system described in any one of items 1 through 14, An automatic steering device that controls the direction of travel of the agricultural machine based on the position of the crop row or furrow detected by the row detection system, Agricultural machinery equipped with [specific features / equipment].

[0193] [Item 16] Further equipped with a running gear including a steering wheel, The agricultural machine according to item 15, wherein the automatic steering device controls the steering angle of the steering wheel based on the position of the crop row or furrow detected by the row detection system. [Industrial applicability]

[0194] The technology disclosed herein can be applied to agricultural machinery such as riding cultivators, vegetable transplanters, tractors, or agricultural drones. [Explanation of Symbols]

[0195] 10...Ground, 12...Crop rows, 14...Intermediate area (working passage), 16...Furrows, 40...Image, 42...Enhanced image, 44...Top view image, 100...Agricultural machinery, 110...Vehicle body, 120, 121...Imaging device, 122...Processing device, 124...Automatic steering device, 1000...Row detection system

Claims

1. A first imaging device, which is attached to an agricultural machine having multiple wheels including a pair of front wheels and a pair of rear wheels, and which photographs the ground and generates a first image of a first region on the ground, A second imaging device attached to the agricultural machine, which photographs the ground and generates a second image of a second region on the ground that partially overlaps with the first region and is shifted backward from the first region, A processing apparatus that performs image processing on the first image and the second image, Equipped with, The second imaging device is provided such that at least a portion of each front wheel and at least a portion of each rear wheel are included in the second image. The aforementioned processing apparatus is The first image is converted into a first top view image as seen from above the ground, The second image is converted into a second top view image as seen from above the ground, Based on the positions of each front wheel and each rear wheel in the second top view image, a region of interest is selected from the first top view image, and a column detection process is performed targeting the region of interest. Column detection system.

2. The region of interest is a rectangular region having a width that extends vertically through the center of the first top view image, The aforementioned processing apparatus is The regions of the pair of front wheels and the regions of the pair of rear wheels in the second top view image are detected, Based on the regions of the pair of front wheels and the regions of the pair of rear wheels, the width of the region of interest is determined. The column detection system according to claim 1.

3. The first and second images are color images. The aforementioned processing apparatus is Based on the color information of the pair of front wheels and the pair of rear wheels, the regions of the pair of front wheels and the regions of the pair of rear wheels in the second top view image are determined. The column detection system according to claim 2.

4. The aforementioned processing apparatus is A pair of front wheel reference points are extracted from the aforementioned region of the pair of front wheels. A pair of rear wheel reference points are extracted from the region of the pair of rear wheels. The width of the region of interest is determined based on one or both of the pair of front wheel reference points and the pair of rear wheel reference points. The column detection system according to claim 3.

5. The aforementioned processing apparatus is A vertical reference line is determined to divide the second top view image into a left portion and a right portion. From the area of ​​the pair of front wheels, the pair of pixels closest to the vertical reference line is selected as the pair of front wheel reference points. From the region of the pair of rear wheels, the pair of pixels closest to the vertical reference line is selected as the pair of rear wheel reference points. The width of the region of interest is determined based on either the distance between the front wheels, which is determined by the distance between the pair of front wheel reference points, or the distance between the rear wheels, which is determined by the distance between the pair of rear wheel reference points. The column detection system according to claim 4.

6. The aforementioned processing apparatus is The width of the region of interest is determined based on the distance between the front wheels and the distance between the rear wheels, whichever is not relatively smaller. The column detection system according to claim 5.

7. The aforementioned processing apparatus is The width of the region of interest is obtained by multiplying either the front wheel distance or the rear wheel distance by a value between 0.9 and 2.

0. The column detection system according to claim 5.

8. The aforementioned processing apparatus is A horizontal reference line perpendicular to the vertical reference line is determined, wherein the left portion and the right portion of the second top view image are each divided into an upper portion and a lower portion, the upper portion includes the area of ​​the pair of front wheels, and the lower portion includes the area of ​​the pair of rear wheels. Edge detection technology is used to detect edges in the second top view image, The overlapping portion of the region of the pair of front wheels and the region of the pair of rear wheels and the edge in the second top view image is determined. In each of the four sections divided by the vertical and horizontal reference lines, the pixel closest to the vertical reference line in the overlapping section is selected as the front wheel reference point and the rear wheel reference point. The column detection system according to claim 5.

9. The first imaging device is mounted facing diagonally downwards and forwards, The second imaging device is mounted facing downwards. The column detection system according to claim 1.

10. The aforementioned agricultural machinery is a work vehicle, The second imaging device is mounted on the underside of the work vehicle. The column detection system according to claim 1.

11. The processing device generates a target path based on the location of the crop row or furrow detected by the row detection process. The column detection system according to claim 1.

12. A column detection system according to any one of claims 1 to 11, An automatic steering device that controls the direction of travel of the agricultural machine based on the position of the crop row or furrow detected by the row detection system, Agricultural machinery equipped with [specific features / equipment].

13. Further equipped with a running gear including a steering wheel, The automatic steering system controls the steering angle of the steering wheel based on the position of the crop row or furrow detected by the row detection system. The agricultural machinery according to claim 12.

Images