agricultural machinery

Abstract

This agricultural machine comprises: an image recognition system that detects at least one column region among a ridge and crops provided on the ground of a farm field; a traveling device including a steering wheel; and a control device that can operate in a column following travel mode for controlling the traveling device such that the image recognition system travels along the detected column region. The control device continues traveling for a prescribed time or by a prescribed distance when the image recognition system has detected a defect location in the column region or the end of the farm field during operation in the column following travel mode.

Description

[Technical Field]

[0001] The present disclosure relates to agricultural machinery. [Background technology]

[0002] Research and development is underway to automate work vehicles such as tractors used in farm fields. For example, work vehicles that run with automatic steering using positioning systems such as the Global Navigation Satellite System (GNSS), which enables precise positioning, have been put into practical use. Work vehicles that not only automatically steer but also automatically control speed have also been put into practical use.

[0003] In addition, vision guidance systems are being developed that use imaging devices such as cameras to detect crop rows or ridges in a field and control the movement of work vehicles along the detected crop rows or ridges.

[0004] Patent Document 1 discloses a work machine that travels along rows of ridges in cultivated land where crops are planted. Patent Document 1 describes that the cultivated land is photographed from diagonally above with an on-board camera, and the original image is then binarized, after which a planar projectively transformed image is generated. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-208871 Summary of the Invention [Problem to be solved by the invention]

[0006] When an agricultural machine uses image recognition technology to automatically steer along a row area such as a crop row or furrow, the image recognition system may not always be able to continuously detect the row area.

[0007] The present disclosure provides an agricultural machine that can solve such problems. [Means for solving the problem]

[0008] In an exemplary, non-limiting embodiment, an agricultural machine according to the present disclosure includes an image recognition system that detects row areas of at least one of crops and ridges laid on the ground of a field from acquired images; a traveling device including a steering wheel; and a control device that controls the traveling device and is capable of operating in a row-following traveling mode that controls the traveling device to travel along the row area detected by the image recognition system, and when the image recognition system detects a missing portion in the row area or the end of the field while operating in the row-following traveling mode, the control device continues traveling for a predetermined time or a predetermined distance.

[0009] A general or specific aspect of the present disclosure may be realized by an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable non-transitory storage medium, or any combination thereof. The computer-readable storage medium may include a volatile storage medium or a non-volatile storage medium. An apparatus may be composed of multiple devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. [Effects of the Invention]

[0010] According to an embodiment of the present disclosure, even if the image recognition system detects a missing portion in a row region or the end of a field while the agricultural machine is automatically steering along a row region such as a crop row or ridge, the automatic steering mode is not suddenly canceled, so the driver does not panic. As a result, the driver has enough time to take action to switch to manual steering, and can smoothly perform operations such as traveling with manual steering or stopping the vehicle. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram schematically illustrating an example of a basic configuration of an agricultural machine according to the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of an image recognition system. [Figure 3] FIG. 1 is a diagram schematically illustrating an example of the arrangement of switches and the like provided around a steering wheel of an agricultural machine. [Figure 4] FIG. 10 is a perspective view showing another example of a display device. [Figure 5] FIG. 2 is a diagram schematically illustrating a display screen of an operation terminal that displays the operating state of an agricultural machine. [Figure 6] 10 is a diagram schematically showing a display screen of the operation terminal that displays other operating states of the agricultural machine. FIG. [Figure 7] FIG. 10 is a diagram schematically showing a display screen of the operation terminal that displays still another operating state of the agricultural machine. [Figure 8] FIG. 10 is a diagram schematically showing a display screen of the operation terminal that displays still another operating state of the agricultural machine. [Figure 9] FIG. 10 is a diagram schematically showing a display screen of the operation terminal that displays still another operating state of the agricultural machine. [Figure 10] FIG. 10 is a diagram schematically showing a display screen of the operation terminal that displays still another operating state of the agricultural machine. [Figure 11] FIG. 1 is a diagram schematically illustrating an image of the ground captured by an imaging device attached to an agricultural machine. [Figure 12] FIG. 2 is a perspective view showing a schematic relationship between a body coordinate system Σb and a camera coordinate system Σc fixed to an agricultural machine, and a world coordinate system Σw fixed to the ground. [Figure 13] 1 is a top view showing a schematic diagram of a part of a field with multiple rows of crops on the ground. FIG. [Figure 14] 14 is a diagram schematically illustrating an example of an image captured by an imaging device of the agricultural machine illustrated in FIG. 13. FIG. [Figure 15] FIG. 10 is a top view schematically showing a state in which the position and orientation (angle in the yaw direction) of the agricultural machine have been adjusted. [Figure 16]16 is a diagram showing an example of an image captured by an imaging device of the agricultural machine in the state shown in FIG. 15. FIG. [Figure 17] 1 is a block diagram schematically illustrating a configuration example of a processing device according to a first embodiment of the present disclosure. [Figure 18] This is a monochrome image corresponding to one frame of a time-series color image captured by an on-board camera mounted on a tractor. [Figure 19] FIG. 10 is a diagram showing an enhanced image obtained by converting the RGB values ​​of one frame of a time-series color image into an excess green index (ExG=2×grb). [Figure 20] 20 is a histogram of the excess green index (ExG) in the image of FIG. 19. [Figure 21] FIG. 10 is a diagram showing an example of a top-view image (bird's-eye view image) classified into first pixels (for example, crop pixels) and second pixels (background pixels). [Figure 22] 10 is a perspective view schematically showing the positional relationship between each of the camera coordinate system Σc1 and the camera coordinate system Σc2 and the reference plane Re. FIG. [Figure 23] FIG. 10 is a schematic diagram showing an example in which the direction of the crop rows in the top view image is parallel to the direction of the scanning lines. [Figure 24] FIG. 24 is a diagram schematically illustrating an example of an integrated value histogram obtained for the top view image of FIG. 23. [Figure 25] FIG. 10 is a schematic diagram showing an example in which the direction of the crop rows in the top view image intersects with the direction of the scanning lines. [Figure 26] FIG. 26 is a diagram schematically illustrating an example of an integrated value histogram obtained for the top view image of FIG. 25. [Figure 27] 10 is a flowchart illustrating an example of an algorithm by which a processing device determines edge lines of a crop row according to an embodiment of the present disclosure. [Figure 28] FIG. 22 is a diagram showing an integrated value histogram obtained from the top view image of FIG. 21. [Figure 29] FIG. 2 is a block diagram illustrating a process executed by a processing device according to an embodiment of the present disclosure. [Figure 30]FIG. 10 is a schematic diagram showing an example of a top view image in which a part of a crop row is missing. [Figure 31] 31 is a diagram schematically showing an example of an integrated value histogram obtained for the top view image of FIG. 30. FIG. [Figure 32] FIG. 10 is a diagram for explaining a form in which a top view image is divided into a plurality of blocks. [Figure 33] 33 is a diagram schematically illustrating the relationship between the position of the scanning line and the integrated value of the index value in each block of FIG. 32. FIG. [Figure 34] FIG. 34 is a diagram showing an example of the crop row center in each block of FIG. 33 and an approximation line to the crop row center. [Figure 35] FIG. 35 is a top view showing an example of an edge line of a crop row determined based on the approximation line of FIG. 34. [Figure 36] FIG. 10 is a diagram illustrating a method for dividing a part or all of a top-view image into multiple blocks and determining the position of an edge line for each of the multiple blocks when a crop row includes a curved portion. [Figure 37] 38 is a diagram schematically illustrating the relationship between the position of the scanning line and the integrated value (histogram) of the index value in each block of FIG. 37. FIG. [Figure 38] FIG. 38 is a diagram showing an example of the crop row centers in each block of FIG. 37 and an approximation line to the crop row centers. [Figure 39] FIG. 39 is a top view showing an example of an edge line of a crop row determined based on the approximation curve of FIG. 38. [Figure 40] FIG. 1 is a perspective view showing a row of ridges provided on the ground. [Figure 41] FIG. 2 is a diagram showing an image acquired from an imaging device at time t. [Figure 42] FIG. 2 is a diagram schematically illustrating the correspondence relationship of feature points between an image acquired from an imaging device at time t and an image acquired at time t+1. [Figure 43] FIG. 10 is a perspective view schematically showing the movement of feature points on the ridges and intermediate area (work passage) shown in an image captured by an imaging device. [Figure 44]10 is a diagram schematically illustrating the relationship between the movement amount (first movement amount) of a feature point projected onto an image plane and the movement amount (second movement amount) of a feature point projected onto a reference plane. FIG. [Figure 45] FIG. 10 is a block diagram showing processing executed by a processing device according to a second embodiment of the present disclosure. [Figure 46] FIG. 10 is a diagram illustrating the relationship between the average height of feature points on a scan line and the position of the scan line. [Figure 47] FIG. 10 is a diagram illustrating an example of a basic configuration of an image recognition system according to a third embodiment of the present disclosure. [Figure 48] FIG. 10 is a diagram illustrating an example of an image acquired by a processing device from an imaging device. [Figure 49] FIG. 49 shows a portion of the image in FIG. 48. [Figure 50] 1 is a top view showing a schematic view of a part of the ground on which rows of crops are provided; FIG. [Figure 51] 10 is a diagram showing a schematic diagram of the positional relationship between points P3 and P4 included in a part of the front wheel shown in the image and corresponding points P3' and P4' on the reference plane Re. FIG. [Figure 52] FIG. 1 is a block diagram schematically illustrating an example of a basic configuration of an agricultural machine according to the present disclosure. [Figure 53] FIG. 2 is a top view schematically showing an imaging range on the ground of time-series images acquired by an imaging device of the agricultural machine. [Figure 54] 1 is a diagram showing an example of the arrangement of crop rows on the ground of a farm field, and a schematic diagram showing the state of an agricultural machine at multiple positions during movement. FIG. [Figure 55] 55 is a plan view schematically showing the positional relationship between a search area and a crop row when the agricultural machine is at position P10 in FIG. 54. FIG. [Figure 56] FIG. 55 is a diagram showing a schematic diagram of the results of area classification of each block constituting the search area when the agricultural machine is at position P10 in FIG. 54. [Figure 57] 55 is a plan view schematically showing the positional relationship between a search area and crop rows when the agricultural machine is at position P11 in FIG. 54. FIG. [Figure 58]FIG. 55 is a diagram showing a schematic diagram of the results of area classification of each block constituting the search area when the agricultural machine is at position P11 in FIG. 54. [Figure 59] 55 is a plan view schematically showing the positional relationship between the search area and the crop rows when the agricultural machine is at position P12 in FIG. 54. FIG. [Figure 60] FIG. 55 is a diagram schematically showing the results of area classification of each block constituting the search area when the agricultural machine is at position P12 in FIG. 54. [Figure 61] FIG. 10 is a diagram showing an example of a search area in a top view image showing three rows of crops. [Figure 62] 62 is a diagram schematically showing the relationship between the position of the scanning line and the integrated value of the index value (histogram of the integrated value), obtained for each block in the search region in the top-view image of FIG. 61. FIG. [Figure 63] FIG. 63 is a diagram schematically illustrating an example of a two-dimensional block arrangement pattern formed by the “row regions” detected within the search region of FIG. 62. [Figure 64] FIG. 10 is a diagram showing an example of a search area in a top view image showing two diagonally extending rows of crops. [Figure 65] FIG. 65 is a diagram schematically showing the relationship between the position of the scanning line and the integrated value of the index value (histogram of the integrated value), obtained for each block in the search region in the top-view image of FIG. 64. [Figure 66] FIG. 66 is a diagram schematically illustrating an example of a two-dimensional block arrangement pattern formed by the "row regions" detected within the search region of FIG. 65. [Figure 67] 10 is a flowchart illustrating an example of a procedure for the processing device to detect a row region. [Figure 68] FIG. 2 is a block diagram illustrating a process executed by a processing device according to an embodiment of the present disclosure. [Figure 69] FIG. 10 is a diagram showing blocks associated with paths taken by the wheels of an agricultural machine within a search area. [Figure 70] FIG. 10 is a diagram showing another example of the arrangement of blocks within a search area. [Figure 71] FIG. 10 is a diagram showing yet another example of the arrangement of blocks within a search area. [Figure 72] FIG. 1 is a perspective view illustrating an example of the appearance of an agricultural machine according to an embodiment of the present disclosure. [Figure 73] 1 is a side view schematically illustrating an example of an agricultural machine with a work implement attached thereto. FIG. [Figure 74] FIG. 1 is a block diagram showing an example of a schematic configuration of an agricultural machine and a work machine. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims. In the following description, components having the same or similar functions are designated by the same reference numerals.

[0013] The following embodiments are examples, and the technology of the present disclosure is not limited to the following embodiments. For example, the numerical values, shapes, materials, steps, the order of the steps, the layout of the display screen, and the like shown in the following embodiments are merely examples, and various modifications are possible as long as no technical contradiction occurs. Furthermore, one aspect can be combined with another aspect as long as no technical contradiction occurs.

[0014] In this disclosure, "agricultural machinery" broadly includes machines that perform basic agricultural tasks in fields, such as tilling, planting, and harvesting. Agricultural machinery is a machine that has the function and structure to perform agricultural tasks on the ground in a field, such as tilling, sowing, pest control, fertilizing, planting crops, or harvesting. These agricultural tasks are sometimes referred to as "ground work" or simply "work." In addition, the term "agricultural machinery" does not necessarily refer to cases where a work vehicle such as a tractor functions alone as an "agricultural machine," but rather the entire work vehicle and an implement attached to or towed by the work vehicle may function as a single "agricultural machine." Examples of agricultural machinery include tractors, riding cultivators, vegetable transplanters, mowers, and mobile field robots.

[0015] <Basic configuration> Before describing the embodiments of the present disclosure in detail, an example of the basic configuration and operation of an agricultural machine according to the present disclosure will be described.

[0016] FIG. 1 is a block diagram schematically illustrating an example of the basic configuration of an agricultural machine according to the present disclosure. The agricultural machine 100 in this example includes an image recognition system 1000, a traveling device 145 including a steering wheel, and a control device 180 that controls the traveling device 145. As shown in FIG. 2, for example, the image recognition system 1000 includes an imaging device 120 such as a camera attached to the agricultural machine 100, and a processing device 122 that may be implemented by at least one computer. The image recognition system 1000 is configured to detect row regions of at least one of crops and ridges provided on the ground of a field from an image acquired by the imaging device 120. The method for detecting row regions using the image recognition system 1000 will be described in detail later.

[0017] The agricultural machine 100 according to the present disclosure can perform "row-following traveling" in addition to traveling by normal manual steering. "Row-following traveling" refers to traveling by automatic steering along a row area detected by the image recognition system 1000. In the row-following traveling mode, the control device 180 controls the traveling device 145 so that the machine travels along the row area detected by the image recognition system 1000. The direction of the steered wheels during row-following traveling is automatically controlled, for example, by a steering motor, without a human operating the steering wheel. Such row-following traveling is performed by the control device 180 controlling the traveling device 145 so that the wheels of the traveling device (all wheels including the steered wheels) move through an area (work passage) between two adjacent row areas. Therefore, during row-following traveling, the image recognition system 1000 can monitor the relationship between the position of the row area to be followed (for example, the position of the "edge" of the row area) and the position of the wheels with a high accuracy of about several centimeters.

[0018] In this disclosure, "wheel" means "wheel with tire" or "wheel with tracks." Hereinafter, when referring to the tire portion of a wheel, for example, the term "tire" will be used, and when referring to the metal "wheel" portion, for example, the term "metal wheel" will be used.

[0019] In this way, the image recognition system 1000 is configured not only to detect row areas from within an image, but also to calculate with high accuracy the relative position of the detected row area with respect to the agricultural machine 100 through calculations. The relative position of the row area with respect to the agricultural machine 100 is, for example, the coordinates of the row area in a local coordinate system fixed to the agricultural machine 100. When performing row-following traveling, the coordinates of the row area do not need to be converted into coordinates in a world coordinate system fixed to the ground. Therefore, while performing row-following traveling, the agricultural machine 100 does not need to accurately measure its own position (e.g., latitude and longitude) in the world coordinate system. However, if the agricultural machine 100 is equipped with a self-position estimation device, the coordinates of the row area in the local coordinate system fixed to the agricultural machine 100 may be converted into coordinates in a world coordinate system fixed to the ground to generate a map of the row area.

[0020] During trail following travel, the travel speed of the agricultural machine 100 can be controlled in accordance with the positions of operating members such as an accelerator pedal and a brake pedal operated by the driver (operator). However, the travel speed of the agricultural machine 100 may also be automatically controlled by the control device 180.

[0021] 1, the agricultural machine 100 according to the present disclosure may include a start switch 112 that commands the start of row-following travel. More specifically, the start switch 112 commands the start of row-following travel in which the control device 180 controls the traveling device 145 so that the agricultural machine 100 travels along the row area detected by the image recognition system 1000. In the example of FIG. 1, the start switch 112 is connected to the control device 180, but may also be connected to the image recognition system 1000.

[0022] The start switch 112 can be provided around the driver's seat of the agricultural machine 100 or around the steering wheel.

[0023] FIG. 3 is a diagram schematically showing an example of the arrangement of switches and other components provided around the steering wheel 118 of the agricultural machine 100. In this example, the start switch 112 is a lever that can be switched, for example, from a neutral position to up, down, forward, or backward. The start switch 112 is one of the members operated by the driver of the agricultural machine 100. The agricultural machine 100 is provided with various switches that are operated by the driver. Hereinafter, the driver of the agricultural machine 100 will be referred to as the "operator."

[0024] Referring again to FIG. 1 , the control device 180 in the present disclosure is configured to continue traveling for a predetermined time or a predetermined distance when the image recognition system 1000 detects a missing portion of a row region or the end of the field while operating in row-following mode. Here, examples of a “missing portion of a row region” include a portion of the crop row missing due to withering of the crop, a portion of a ridge that is partially destroyed, or a portion of the crop row or ridge that is partially covered by some object. Furthermore, the “missing portion of a row region” includes not only actual missing portions of the crop row or ridge, but also areas that could not be correctly detected as row regions due to erroneous recognition by the image recognition system 1000. Therefore, “when the image recognition system detects a missing portion of a region or the end of the field while operating in row-following mode” is interpreted as including “when the image recognition system loses sight of the row region.” Note that the end of the field during row-following travel is possible by detecting that multiple row regions, including the row region being followed, all have their ends while the agricultural machine 100 is traveling.

[0025] If the image recognition system 1000 detects a "missing portion of the row area," and the automatic steering mode is suddenly canceled, normal steering will not be performed, which may result in unintended driving problems. However, according to the present disclosure, even if the image recognition system 1000 loses sight of part of the row area, driving in the automatic steering mode will not immediately stop. This gives the driver ample time to take action to switch to manual steering, allowing for smooth operation such as driving with manual steering or stopping driving. The same effect is achieved when the agricultural machine 100 approaches the end of the field, because the automatic steering mode is not suddenly canceled.

[0026] After the agricultural machine 100 has stopped, it becomes possible for the operator to manually drive the agricultural machine. When the operator wishes to start file-following traveling again, he or she can issue a command to start file-following traveling by operating the start switch 112 described above. In this case, the image recognition system 1000 determines whether file-following traveling is possible, and only when file-following traveling is possible can file-following traveling be resumed in automatic steering mode.

[0027] The "predetermined time" for which automatic steering travel continues after the image recognition system 1000 loses sight of the row area may be set, for example, within a range of 1 second to 5 seconds. The "predetermined distance" may be set, for example, within a range of 1 meter to 5 meters, or based on the length of the agricultural machine 100. The control device 180 may determine these "predetermined time" and "predetermined distance" to different values ​​depending on the traveling speed of the agricultural machine 100. The control device 180 may store a table or a calculation formula that defines the relationship between the "predetermined time" and / or the "predetermined distance" and the traveling speed in a storage device. The distance traveled by the agricultural machine 100 may be determined by the control device 180 based on, for example, the number of wheel rotations acquired from the traveling device 145, or may be determined using a positioning system such as GNSS, or self-position estimation technology. The time traveled by the agricultural machine 100 may be determined by measuring it using a timer. The control device 180 may also determine the distance traveled by the agricultural machine 100 based on the product of the traveling speed and the traveling time.

[0028] Generally, the orientation of crop rows and furrows does not change significantly and discontinuously, and the edges of the row region are made up of straight lines or curves whose direction changes gradually. For this reason, after the image recognition system 1000 loses sight of the row region, the control device 180 preferably controls the traveling device 145 so that the agricultural machine 100 travels along an extension of the path it has traveled. The control device 180 may continue traveling in this manner and then stop the agricultural machine 100.

[0029] The control device 180 may be configured to control the traveling device 145 to travel along a new row area if the image recognition system 1000 detects a new row area while continuing to travel for a predetermined time or a predetermined distance after the image recognition system 1000 detects a missing portion in a row area or the end of the field.

[0030] Note that, when a work implement is attached to or towed by the agricultural machine 100, the control device 180 may determine the above-mentioned "predetermined time" or "predetermined distance" depending on the work implement. For example, the "predetermined distance" may be based on the sum of the length of the agricultural machine 100 and the length of the work implement, and may be the distance it takes for the work implement to pass the missing part in the row area or the end of the field after detecting the missing part in the row area or the end of the field. Also, the "predetermined time" may be determined as the time it takes for the work implement to pass the missing part in the row area or the end of the field, taking into account the work speed and length of the work implement according to the type of work implement. Furthermore, when the work history is stored and the agricultural machine 100 approaches the end of the field, the "predetermined distance" or "predetermined time" may be set to the distance or time it takes for the work implement to pass the adjacent end point of the previous work.

[0031] The agricultural machine 100 may be equipped with a notification means that notifies the operator of a warning to stop traveling when the image recognition system 1000 detects the end of the field. Examples of such notification means include a display device that displays an icon, text, or symbol, a light-emitting device such as an LED, and an acoustic device that emits sound or vibration, such as a buzzer or speaker. When the image recognition system 1000 detects the ends of multiple row regions and consequently detects the end of the field, an icon, text, or symbol indicating "row following travel possible" may be displayed on the display screen. When row following travel is not possible, an icon, text, or symbol indicating "row following travel not possible" may be displayed on the display screen. Furthermore, when row following travel is not possible and the operator operates the start switch 112 to issue a command to start row following travel, a voice such as "row following travel is currently not possible" may be emitted from the audio device or text may be displayed on the display.

[0032] In the example of Fig. 3, a symbol indicating whether "row following driving" is possible may be displayed as a notification 116 on a display device 117 located near the steering wheel 118. As described above, when the image recognition system 1000 detects a missing portion in the row area or the end of the field, a notification 116 indicating that the agricultural machine 100 is approaching the end of the field may be displayed on the display device 117.

[0033] Fig. 4 is a perspective view showing another example of the display device 117. The display device 117 may include an operation terminal 200 arranged near the driver's seat. For example, as shown in Fig. 5, the display device 117 of this operation terminal 200 may display a notification 116 such as characters or symbols indicating that the end of the field is approaching. In Fig. 5, the row area displayed on the display device 117 is schematically depicted as a hatched bar-shaped area.

[0034] FIG. 6 shows an example of a display of a notice 116 that warns that the agricultural machine 100 has reached the end of the field and will soon stop.

[0035] 7 shows an example of the display of a notification 116 indicating that the agricultural machine 100 has passed the end of the field and stopped. As shown in FIG. 7, this notification 116 may include information indicating that automatic steering during row following travel has been switched to manual steering.

[0036] FIG. 8 shows an example of a notification 116 that is displayed when the image recognition system 1000 detects a missing part in a row area while following the rows in a field, and warns that the system will automatically stop after traveling a specified time or distance.

[0037] FIG. 9 shows an example of the display of a notification 116 indicating that the image recognition system 1000 has detected a missing portion in the row area while the vehicle is following the rows in a field, but that the vehicle will continue to follow the rows using automatic steering without stopping.

[0038] Note that, when the agricultural machine 100 passes the end of the field during row-following travel and stops, and then the operator manually steers the agricultural machine 100 to turn it and resume row-following travel, it is desirable that the image recognition system 1000 determine whether or not row-following travel is possible based on the position of the wheels of the agricultural machine 100. If the image recognition system 1000 determines that row-following travel is not possible, a notification 116 indicating this may be displayed on the display device 117.

[0039] FIG. 10 shows an example of a notification 116 that is displayed on the display device 117 when the image recognition system 1000 determines that the vehicle is not in a state where it can travel along a convoy.

[0040] The control device 180 may be configured to execute the line following travel mode when the image recognition system 1000 detects a line area and detects a work aisle area of ​​a predetermined width or more on either or both sides of the line area. The control device 180 may be configured to determine whether the steered wheels can pass through the work aisle area based on the work aisle area and the position of the steering wheel, and to stop the line following travel mode when it determines that the steered wheels cannot pass through the work aisle area.

[0041] 1 and 3, the agricultural machine 100 may be equipped with a mode switch 114 that switches between an automatic steering mode and a manual steering mode. When the automatic steering mode is selected by the mode switch 114, the control device 180 can permit the start of row-following traveling by the start switch 112. Specifically, when the automatic steering mode is selected by the mode switch 114, for example, power may be started to be supplied to the steering motor for automatic steering. Furthermore, the image recognition system 1000 may be configured to start image recognition processing for detecting the row region when the automatic steering mode is selected by the mode switch 114.

[0042] When the image recognition system 1000 starts image recognition processing in response to the automatic steering mode being selected by the mode switch 114, the image recognition system 1000 may be configured to perform the image processing in multiple stages.

[0043] In the example of FIG. 3 , the mode switch 114 is provided near the steering wheel 118. In this example, the mode switch 114 is a push button. The mode switch 114 is switched from an OFF state to an ON state, for example, when switching from a manual steering mode to an automatic steering mode. Conversely, the mode switch 114 is switched from an ON state to an OFF state when switching from an automatic steering mode to a manual steering mode. In other words, the mode switch 114 is a switch that determines the start and end of the automatic steering mode. However, in the present disclosure, the start of the automatic steering mode does not immediately mean that the vehicle will start following-file traveling by automatic steering. As described above, when it is determined that the vehicle is in a state where following-file traveling is possible based on the detection results of the image recognition system 1000 and the operator issues a command to start following-file traveling using the start switch 112, the control device 180 starts following-file traveling.

[0044] In one embodiment, when the automatic steering mode is enabled and file-following traveling is possible, the file-following traveling mode is started when the operator switches the start switch 112, for example, from the neutral position to the downward position. Furthermore, when the operator switches the start switch 112, for example, from the neutral position to the upward position during file-following traveling, the output of a signal commanding the start of file-following traveling that has been output from the start switch 112 may be stopped, or a signal commanding the stop of file-following traveling may be output from the start switch 112. As described above, the start switch 112 is a switch that turns on the file-following traveling function of the agricultural machine 100, but the control device 180 in the present disclosure may be configured not to immediately start file-following traveling even when a command to start file-following traveling is issued by the start switch 112. It is desirable that the image recognition system 1000 determine whether file-following traveling is possible, and enable the command from the start switch 112 only when it is determined that file-following traveling is possible. This process and operation prevents row-following travel from starting when row-following travel is not possible, making it possible to avoid a situation in which crop rows or furrows are trampled down by the wheels.

[0045] During trail following travel, the travel speed of the agricultural machine 100 can be controlled in accordance with the positions of operation members such as an accelerator pedal and a brake pedal operated by the operator. However, the travel speed of the agricultural machine 100 may also be automatically controlled by the control device 180.

[0046] An embodiment of the image recognition system 1000 shown in Fig. 1 will now be described, followed by a detailed description of an embodiment of the agricultural machine 100 according to the present disclosure.

[0047] (Image Recognition System Embodiment 1) First, an image recognition system according to a first exemplary embodiment of the present disclosure will be described. In this embodiment, crop row detection is performed as "row detection."

[0048] The image recognition system 1000 in this embodiment includes an imaging device 120 that is attached to the agricultural machine 100 when in use (FIG. 2). The imaging device 120 is fixed to the agricultural machine 100 so as to acquire time-series color images that include at least a portion of the ground.

[0049] FIG. 11 schematically shows how an imaging device 120 attached to an agricultural machine 100, such as a tractor or a riding cultivation machine, captures an image of the ground 10. In the example of FIG. 11, the agricultural machine 100 includes a travellable vehicle body 110, and the imaging device 120 is fixed to the vehicle body 110. For reference, FIG. 11 also shows a body coordinate system Σb having Xb-axis, Yb-axis, and Zb-axis that are orthogonal to each other. The body coordinate system Σb is a coordinate system fixed to the agricultural machine 100, and the origin of the body coordinate system Σb can be set, for example, near the center of gravity of the agricultural machine 100. For ease of viewing, the figure shows the origin of the body coordinate system Σb as if it were located outside the agricultural machine 100. In the body coordinate system Σb in the present disclosure, the Xb-axis coincides with the traveling direction (the direction of arrow F) when the agricultural machine 100 travels straight. The Yb axis corresponds to the direction directly to the right when looking from the coordinate origin in the positive direction of the Xb axis, and the Zb axis corresponds to the downward vertical direction.

[0050] The imaging device 120 is, for example, an in-vehicle camera having a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) image sensor. The imaging device 120 in this embodiment is, for example, a monocular camera capable of capturing video at a frame rate of 3 frames per second (fps) or higher.

[0051] 12 is a perspective view schematically showing the relationship between the above-mentioned body coordinate system Σb, the camera coordinate system Σc of the image capture device 120, and the world coordinate system Σw fixed to the ground 10. The camera coordinate system Σc has an Xc axis, a Yc axis, and a Zc axis that are orthogonal to each other, and the world coordinate system Σw has an Xw axis, a Yw axis, and a Zw axis that are orthogonal to each other. In the example of FIG. 12, the Xw axis and the Yw axis of the world coordinate system Σw are on a reference plane Re that extends along the ground 10.

[0052] The imaging device 120 is attached to a predetermined position on the agricultural machine 100 so as to face in a predetermined direction. Therefore, the position and orientation of the camera coordinate system Σc relative to the body coordinate system Σb are fixed to a known state. The Zc axis of the camera coordinate system Σc is on the camera optical axis λ1. In the example shown in the figure, the camera optical axis λ1 is inclined from the traveling direction F of the agricultural machine 100 toward the ground 10, and the depression angle Φ is greater than 0°. The traveling direction F of the agricultural machine 100 is approximately parallel to the ground 10 on which the agricultural machine 100 is traveling. The depression angle Φ can be set, for example, in the range of 0° to 60°. When the position at which the imaging device 120 is attached is close to the ground 10, the depression angle Φ may be set to a negative value, in other words, the orientation of the camera optical axis λ1 may be set so that the depression angle Φ has a positive elevation angle.

[0053] When the agricultural machine 100 is traveling on the ground 10, the body coordinate system Σb and the camera coordinate system Σc translate relative to the world coordinate system Σw. If the agricultural machine 100 rotates or swings in the pitch, roll, or yaw directions while traveling, the body coordinate system Σb and the camera coordinate system Σc may rotate relative to the world coordinate system Σw. For simplicity's sake, in the following description, it is assumed that the agricultural machine 100 does not rotate in the pitch or roll directions, but moves approximately parallel to the ground 10.

[0054] FIG. 13 is a top view schematically illustrating a portion of a farm field in which multiple crop rows 12 are laid on the ground 10. The crop rows 12 are rows formed by planting crops continuously in one direction on the ground 10 of the farm field. In other words, the crop rows 12 are groups of crops planted in ridges in the field. Because each crop row 12 is thus formed by a group of crops planted in the field, the shape of the crop rows is, strictly speaking, complex, depending on the shape and arrangement of the crops. The width of the crop rows 12 changes depending on the growth of the crops.

[0055] Between adjacent crop rows 12, there is a strip-shaped intermediate region 14 where no crops are planted. Each intermediate region 14 is an area sandwiched between two edge lines E facing each other between two adjacent crop rows 12. When multiple crops are planted in a single ridge in the width direction of the ridge, multiple crop rows 12 will be formed on the ridge. In other words, multiple crop rows 12 will be formed between the rows of the ridge. In such a case, the edge line E of the crop row 12 located at the end of the ridge in the width direction among the multiple crop rows 12 formed on the ridge serves as the reference for the intermediate region 14. In other words, the intermediate region 14 is located between the edge lines E of the crop rows 12 located at the end of the ridge in the width direction among the edge lines E of the multiple crop rows 12.

[0056] The intermediate area 14 functions as an area (work path) through which the wheels of the agricultural machine 100 pass, and therefore the "intermediate area" may be referred to as the "work path."

[0057] In this disclosure, the "edge line" of a crop row refers to a reference line segment (which may include a curve) that defines a target path when an agricultural machine follows the row using automatic steering. Such a reference line segment may be defined as both ends of a strip-shaped area (work path) through which the wheels of the agricultural machine are allowed to pass. A specific method for determining the "edge line" of a crop row will be described later.

[0058] 13 shows a schematic diagram of an agricultural machine 100 entering a field where a crop row 12 is laid. This agricultural machine 100 is equipped with left and right front wheels (steered wheels) 104F and left and right rear wheels 104R as traveling devices 145, and pulls a work machine (implement) 300. The front wheels 104F are steered wheels.

[0059] In the example of FIG. 13 , thick dashed arrows L and R are drawn on the work paths 14 located on both sides of a single central crop row 12. When the agricultural machine 100 travels along the target path indicated by the solid arrow C, the front wheels 104F and rear wheels 104R of the agricultural machine 100 are required to move along the arrows L and R within the work path 14 so as not to step on the crop row 12. In this embodiment, the image recognition system 1000 can detect the edge line E of the crop row 12 using the imaging device 120 attached to the agricultural machine 100. This enables the control device 180 and the traveling device 145 to control the steering and traveling of the agricultural machine 100 so that the front wheels (steered wheels) 104F and rear wheels 104R move along the work path 14 along the arrows L and R. Controlling the steering and traveling of the agricultural machine 100 based on the edge line E of the crop row in this way may be referred to as “row-following traveling control.” The agricultural machine 100 of this embodiment is capable of traveling in an automatic steering mode for performing file-following traveling, and in a normal manual steering mode. In order for the operator to switch between the automatic steering mode and the manual steering mode, the agricultural machine 100 may be provided with a mode switch 114 shown in FIGS. 1 and 3. When performing file-following traveling, the operator operates the mode switch 114 to select the automatic steering mode. When the automatic steering mode is selected by the mode switch 114, the image recognition system 1000 starts image recognition processing based on the image acquired by the imaging device 120.

[0060] Fig. 14 is a diagram schematically illustrating an example of an image 40 acquired by the imaging device 120 of the agricultural machine 100 shown in Fig. 13. Theoretically, the multiple crop rows 12 and the intermediate area (work path) 14 extending parallel to one another on the ground 10 intersect at a vanishing point P0 on the horizon 11. The reason that the vanishing point P0 is located in the area on the right side of the image 40 is because, as shown in Fig. 13, the traveling direction F of the agricultural machine 100 is inclined with respect to the direction in which the crop rows 12 extend (the direction parallel to the arrow C).

[0061] In this embodiment, using a method described below, it is possible to accurately detect the crop row 12 from such an image 40 and determine the edge line E of the crop row 12. Then, based on the edge line E, it is possible to appropriately generate a path (target path) to be followed by the agricultural machine 100 while traveling along the crop row. As a result, it is possible to control the traveling of the agricultural machine 100 by automatic steering so that the front wheels 104F and rear wheels 104R of the agricultural machine 100 move along the arrows L and R in the work passage 14 (row-following traveling control). Such row-following traveling control enables precise automatic steering according to the growing conditions of the crop, which is difficult to achieve with automatic steering technology that uses a positioning system such as GNSS.

[0062] However, as shown in FIG. 13 , if the orientation of the agricultural machine 100 and the direction in which the work passage 14 extends do not match, automatic steering may not be able to move the front wheels 104F and rear wheels 104R of the agricultural machine 100 through the work passage 14 along the arrows L and R. The image recognition system 1000 determines whether or not line-following traveling is possible based on the position and orientation of the steered wheels (front wheels 104F). If the image recognition system 1000 determines that line-following traveling is not possible, the operator is notified of this. In response to the notification, the operator adjusts the position and / or orientation of the agricultural machine 100 by manual steering.

[0063] Fig. 15 is a top view that schematically shows a state in which the agricultural machine 100 is steered to adjust the position and orientation (angle in the yaw direction) of the agricultural machine 100 so as to reduce the position error with respect to the target path (arrow C). Fig. 16 is a diagram showing an example of an image 40 captured by the imaging device 120 of the agricultural machine 100 in such a state. The front wheels 104F and rear wheels 104R of the agricultural machine 100 in the state shown in Fig. 15 are positioned on the lines indicated by arrows L and R in the work aisle 14, respectively.

[0064] After the image recognition system 1000 determines that line-following travel is not possible, if the state of the agricultural machine 100 changes to, for example, the state shown in Fig. 15 due to manual steering by the operator, the image recognition system 1000 may determine that line-following travel is possible and notify the operator of this. In this embodiment, after receiving the notification, the image recognition system 1000 stops on the spot until the operator issues a command to start line-following travel using the start switch 112 in Fig. 3. In this state, if the operator operates the start switch 112 to issue a command to start line-following travel, the control device 180 permits the start of line-following travel using automatic steering.

[0065] When the automatic steering-based tracking travel starts and the agricultural machine 100 travels along the target path C indicated by the arrow C in the center of FIG. 15, the control device 180 and the traveling device 145 in the agricultural machine 100 control the steering angles of the steering wheels so that the front wheels 104F and the rear wheels 104R do not deviate from the work passage 14.

[0066] The configuration and operation of an image recognition system according to an embodiment of the present disclosure will be described in detail below.

[0067] 2, the image recognition system 1000 according to this embodiment includes the above-described imaging device 120 and a processing device 122 that performs image processing of time-series color images acquired from the imaging device 120. The processing device 122 is connected to a control device 180 included in the agricultural machine 100.

[0068] 2 may be realized by an electronic control unit (ECU) for image recognition. The ECU is an in-vehicle computer. The processing device 122 is connected to the image capture device 120 by a serial signal line such as a wire harness so as to receive image data output by the image capture device 120. A part of the image recognition process performed by the processing device 122 may be performed inside the image capture device 120 (inside the camera module).

[0069] 17 is a block diagram showing an example of the hardware configuration of the processing device 122. The processing device 122 includes a processor 20, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 24, a communication device 26, and a storage device 28. These components are connected to each other via a bus 30.

[0070] The processor 20 is a semiconductor integrated circuit, also referred to as a central processing unit (CPU) or microprocessor. The processor 20 may include a graphics processing unit (GPU). The processor 20 sequentially executes a computer program containing a predetermined set of instructions stored in a read-only memory (ROM) 22, thereby implementing the processing required for the sequence detection of the present disclosure. Part or all of the processor 20 may be a field programmable gate array (FPGA), application specific integrated circuit (ASIC), or application specific standard product (ASSP) equipped with a CPU.

[0071] The communication device 26 is an interface for performing data communication between the processing device 122 and an external computer. The communication device 26 can perform wired communication using a CAN (Controller Area Network) or the like, or wireless communication conforming to the Bluetooth (registered trademark) standard and / or the Wi-Fi (registered trademark) standard.

[0072] The storage device 28 can store data of images acquired from the imaging device 120 or images in the process of being processed. Examples of the storage device 28 include a hard disk drive or a nonvolatile semiconductor memory.

[0073] The hardware configuration of the processing device 122 is not limited to the above example. It is not necessary for part or all of the processing device 122 to be mounted on the agricultural machine 100. By using the communication device 26, it is also possible to have one or more computers located outside the agricultural machine 100 function as part or all of the processing device 122. For example, a server computer connected to a network can function as part or all of the processing device 122. On the other hand, a computer mounted on the agricultural machine 100 may perform all of the functions required of the processing device 122.

[0074] In this embodiment, the processing device 122 acquires time-series color images from the imaging device 120 and executes the following operations S1, S2, and S3. (S1) An enhanced image is generated from the time-series color images, in which the color of the crop row being detected is enhanced. (S2) From the enhanced image, a top view image of the ground is generated, in which first pixels having a crop row color index value equal to or greater than a threshold value and second pixels having this index value less than the threshold value are classified. (S3) Determine the position of the edge line of the crop row based on the index value of the first pixel.

[0075] Specific examples of operations S1, S2, and S3 will be described in detail below.

[0076] A time-series color image is a collection of images captured by the imaging device 120 in a time series. Each image is composed of a group of pixels per frame. For example, if the imaging device 120 outputs images at a frame rate of 30 frames per second, the processing device 122 can acquire a new image at intervals of approximately 33 milliseconds. The speed at which an agricultural machine 100, such as a tractor, travels in a field is relatively slow compared to the speed of an ordinary automobile traveling on a public road, and may be, for example, approximately 10 kilometers per hour or less. At a speed of 10 kilometers per hour, the distance traveled in approximately 33 milliseconds is approximately 6 centimeters. For this reason, the processing device 122 may acquire images at intervals of, for example, 100 to 300 milliseconds, and does not need to process all frames of images captured by the imaging device 120. The acquisition interval for images to be processed by the processing device 122 may be automatically changed by the processing device 122 depending on the traveling speed of the agricultural machine 100.

[0077] FIG. 18 is an image corresponding to image 40 of one frame in a time-series color image sequence acquired by an imaging device (in this example, a monocular camera) mounted on an agricultural machine. The image in FIG. 18 shows rows of crops (crop rows) planted in rows on the ground in a farm field. In this example, the rows of crops are arranged approximately parallel and at equal intervals on the ground, and the camera optical axis of the imaging device faces the traveling direction of the agricultural machine. As described above, the camera optical axis does not need to be parallel to the traveling direction of the agricultural machine, and may be incident on the ground ahead of the traveling direction of the agricultural machine. The mounting position of the imaging device is not limited to this example. When multiple imaging devices are mounted on an agricultural machine, some of the imaging devices may face their camera optical axes in the opposite direction to the traveling direction or in a direction intersecting the traveling direction.

[0078] In operation S1, the processing device 122 in FIG. 2 generates an image (enhanced image) in which the color of the crop rows, which are the detection target, is enhanced based on the time-series color images acquired from the imaging device 120. Crops contain chlorophyll (chlorophyll) to perform photosynthesis when exposed to sunlight (white light). Chlorophyll has a lower light absorption rate for green than for red and blue. Therefore, the spectrum of sunlight reflected by crops exhibits relatively higher values ​​in the green wavelength range compared to the spectrum of sunlight reflected by the soil surface. As a result, the color of crops generally contains a large amount of green components, and the "color of the crop rows" is typically green. However, as will be described later, the "color of the crop rows" is not limited to green.

[0079] The image sensor in the imaging device 120 has a large number of photodetector cells arranged in rows and columns. Each photodetector cell corresponds to a picture element (pixel) that makes up an image and includes an R subpixel that detects the intensity of red light, a G subpixel that detects the intensity of green light, and a B subpixel that detects the intensity of blue light. The light outputs detected by the R subpixel, G subpixel, and B subpixel in each photodetector cell are referred to as the R value, G value, and B value, respectively. Hereinafter, the R value, G value, and B value may be collectively referred to as the "pixel value" or "RGB value." When the R value, G value, and B value are used, a color can be specified by coordinate values ​​in the RGB color space.

[0080] When the color of the crop rows being detected is green, an enhanced image that emphasizes the color of the crop rows is an image in which the RGB values ​​of each pixel in a color image captured by an imaging device are converted into pixel values ​​in which the weight of the G value is relatively large. This conversion of pixel values ​​to generate an enhanced image is defined, for example, as "(2×G value - R value - B value) / (R value + G value + B value)." Here, the denominator (R value + G value + B value) is a normalization factor. Hereinafter, the normalized RGB values ​​are referred to as rgb values, and are defined as r = R value / (R value + G value + B value), g = G value / (R value + G value + B value), and b = B value / (R value + G value + B value). "2×grb" is called the Excess Green Index (ExG).

[0081] FIG. 19 is a diagram showing an enhanced image 42 obtained by converting the RGB values ​​in the image of FIG. 18 into "2×grb." As a result of this conversion, pixels in the image 42 of FIG. 19 where "r+b" is relatively small compared to g are displayed brighter, and pixels where "r+b" is relatively large compared to g are displayed darker. This conversion results in an image (enhanced image) 42 in which the color of the crop rows to be detected ("green" in this example) is emphasized. Relatively bright pixels in the image of FIG. 19 are pixels with a relatively large green component, and belong to the crop area.

[0082] As a "color index value" for enhancing the color of crops, other indices such as the green-red vegetation index (G value - R value) / (G value + R value) may be used in addition to the excess green index (ExG). Also, if the imaging device can also function as an infrared camera, the NDVI (Normalized Difference Vegetation Index) may be used as the "color index value of crop rows."

[0083] Note that each row of crops may be covered with a sheet called "mulch." In such cases, the "color of the crop row" refers to the "color of the object that is arranged in a row covering the crops." Specifically, if the color of the sheet is achromatic black, the "color of the crop row" refers to "black." Also, if the color of the sheet is red, the "color of the crop row" refers to "red." In this way, the "color of the crop row" refers not only to the color of the crop itself, but also to the color of the area that defines the crop row (a color that can be distinguished from the color of the soil surface).

[0084] To generate an enhanced image that emphasizes the "color of the crop rows," conversion from the RGB color space to the HSV color space can be used. The HSV color space is a color space composed of three components: hue, saturation, and value. Using color information converted from the RGB color space to the HSV color space, low-saturation colors such as black or white can be detected. To detect "black" using the OpenCV library, simply set the hue to its maximum range (0-179), the saturation to its maximum range (0-255), and the value to its maximum range (0-30). To detect "white," simply set the hue to its maximum range (0-179), the saturation to its maximum range (0-255), and the value to its maximum range (200-255). Pixels with hue, saturation, and value values ​​within these ranges are the pixels whose color is to be detected. To detect green pixels, for example, simply set the hue range to 30-90.

[0085] By generating an image (enhanced image) in which the color of the target crop rows is emphasized, it becomes easier to separate (extract) the crop row area from the other background areas (segmentation).

[0086] Next, operation S2 will be described.

[0087] In operation S2, the processing device 122 generates a top-view image from the enhanced image 42, in which pixels are classified into first pixels having a crop row color index value equal to or greater than a threshold value and second pixels having this index value less than the threshold value. The top-view image is an image viewed from above the ground.

[0088] In this embodiment, the aforementioned excess green index (ExG) is used as the index value for the color of the crop row, and the discrimination threshold is determined using discriminant analysis (Otsu's binarization). FIG. 20 is a histogram of the excess green index (ExG) in the enhanced image 42 of FIG. 19. The horizontal axis of the histogram represents the excess green index (ExG), and the vertical axis represents the number of pixels in the image (corresponding to frequency of occurrence). FIG. 20 also shows a dashed line indicating the threshold value Th calculated using the discriminant analysis algorithm. The pixels in the enhanced image 42 are classified into two classes based on this threshold value Th. To the right of the dashed line indicating the threshold value Th, the frequency of occurrence of pixels whose excess green index (ExG) is equal to or greater than the threshold value is shown. These pixels are presumed to belong to the crop class. In contrast, to the left of the dashed line indicating the threshold value Th, the frequency of occurrence of pixels whose excess green index (ExG) is less than the threshold value is shown. These pixels are presumed to belong to the class of cover crops such as soil. In this example, the first pixel, whose index value is equal to or greater than the threshold value, corresponds to a "crop pixel." On the other hand, the second pixel whose index value is less than the threshold corresponds to a "background pixel." The background pixel corresponds to an object other than the detection target, such as the surface of the soil, and the intermediate area (work passage) 14 described above may be composed of background pixels. Note that the method for determining the threshold is not limited to the above example, and the threshold may be determined using other methods that utilize machine learning, for example.

[0089] By assigning each pixel constituting the enhanced image 42 to either a "first pixel" or a "second pixel," it is possible to extract a detection target area from the enhanced image 42. In addition, by assigning "zero" to the pixel value of the "second pixel" or by removing the data of the second pixel from the image data, it is possible to mask areas other than the detection target. When determining the area to be masked, a process may be performed in which pixels showing locally high values ​​of the excess green index (ExG) are included in the mask area as noise.

[0090] FIG. 21 shows an example of a top-view image 44 of the ground, viewed from above, classified into first and second pixels. The top-view image 44 in FIG. 21 was created from the enhanced image 42 in FIG. 19 using the image conversion technology described below. In this top-view image 44, the second pixels, whose crop row color index value (in this example, the green-excess index) is less than the threshold Th, are black pixels (pixels with a brightness set to zero). The area formed by the second pixels is primarily an area where the soil surface is visible. In the top-view image 44 in FIG. 21, black triangular areas exist at the left and right corners of the bottom edge. These triangular areas correspond to areas not visible in the enhanced image 42 in FIG. 19. Note that in the image 40 in FIG. 18 and the enhanced image 42 in FIG. 19, distortion of lines that should be straight is observed in the periphery of the image. This image distortion occurs due to the performance of the camera lens and can be corrected using the camera's internal parameters. Processing such as crop area enhancement, masking, distortion correction, etc. can be called pre-processing, but pre-processing may also include other processing.

[0091] The top-view image 44 in Figure 21 is a bird's-eye view image of a reference plane Re parallel to the ground, viewed from directly above in the normal direction of the reference plane Re. This bird's-eye view image can be generated from the enhanced image 42 in Figure 19 by homography transformation (planar projective transformation). Homography transformation is a type of geometric transformation that can transform points on a certain plane in three-dimensional space into points on any other plane.

[0092] 22 is a perspective view schematically illustrating the positional relationship between the reference plane Re and the camera coordinate system Σc1 of the imaging device in a first posture (position and orientation: pose) and the camera coordinate system Σc2 of the imaging device in a second posture. In the illustrated example, the camera coordinate system Σc1 is inclined so that its Zc axis intersects the reference plane Re at an angle. The imaging device in the first posture corresponds to an imaging device attached to an agricultural machine. In contrast, the camera coordinate system Σc2 has its Zc axis perpendicular to the reference plane Re. In other words, the camera coordinate system Σc2 is positioned so that a bird's-eye view image of the reference plane Re viewed from directly above in the normal direction of the reference plane Re can be obtained.

[0093] A virtual image plane Im1 exists at a position on the Zc axis away from the origin O1 of the camera coordinate system Σc1 by the camera's focal length. The image plane Im1 is orthogonal to the Zc axis and the camera optical axis λ1. Pixel positions on the image plane Im1 are defined by an image coordinate system having mutually orthogonal u and v axes. For example, assume that the coordinates of points P1 and P2 located on the reference plane Re are (X1, Y1, Z1) and (X2, Y2, Z2) in the world coordinate system Σw, respectively. In the example of Figure 22, the Xw and Yw axes of the world coordinate system Σw are on the reference plane Re. Therefore, Z1 = Z2 = 0. The reference plane Re is set to extend along the ground.

[0094] Points P1 and P2 on the reference plane Re are transformed into points p1 and p2 on the image plane Im1 of the imaging device in the first attitude by perspective projection of the pinhole camera model. In the image plane Im1, points p1 and p2 are located at pixel positions indicated by the coordinates (u1, v1) and (u2, v2), respectively.

[0095] When the imaging device is in the second orientation, a virtual image plane Im2 exists at a position that is the focal length of the camera on the Zc axis from the origin O2 of the camera coordinate system Σc2. In this example, the image plane Im2 is parallel to the reference plane Re. The pixel positions on the image plane Im2 are determined by the mutually orthogonal u * axis and v *The image coordinate system has axes P1 and P2 on the reference plane Re, which are projected by perspective projection onto the image plane Im2. * and point p2 * In the image plane Im2, the point p1 * and point p2 * are respectively, (u1 * ,v1 * ) and (u2 * ,v2 * ) is located at the pixel position indicated by the coordinates.

[0096] Given the positional relationship of the camera coordinate systems Σc1 and Σc2 with respect to the reference plane Re (world coordinate system Σw), a homography transformation can be used to transform any point (u, v) on the image plane Im1 to the corresponding point (u * ,v * ) can be obtained. When the coordinates of the points are expressed in a homogeneous coordinate system, such a homography transformation is defined by a 3-row x 3-column transformation matrix H.

number

[0097] The contents of the transformation matrix H are as follows: 11 , h 12 , , h 32 It is defined by the numerical value of

number

[0098] 8 numbers (h 11 , h 12 , , h 32 ) can be calculated using a known algorithm by capturing an image of a calibration board placed on the reference plane Re using the imaging device 120 attached to the agricultural machine 100.

[0099] When the coordinates of a point on the reference plane Re are (X, Y, 0), the coordinates of the corresponding points on the image planes Im1 and Im2 of the respective cameras are associated with the point (X, Y, 0) by the respective homography transformation matrices H1 and H2, as shown in the following equations 3 and 4.

number

number

[0100] The following equation is derived from the above two equations. As is clear from this equation, the transformation matrix H is H2H1 -1 Equal to H1 -1 is the inverse matrix of H1.

number

[0101] The contents of the transformation matrices H1 and H2 depend on the reference plane Re, so when the position of the reference plane Re changes, the contents of the transformation matrix H also change.

[0102] By using such homography transformation, it is possible to generate a top-view image of the ground from an image of the ground captured by an imaging device in a first orientation (an imaging device attached to an agricultural machine). In other words, homography transformation allows the coordinates of any point on the image plane Im1 of the imaging device 120 to be converted into the coordinates of a point on the image plane Im2 of a virtual imaging device in a predetermined orientation with respect to the reference plane Re.

[0103] After calculating the contents of the transformation matrix H, the processing device 122 executes a software program based on the above algorithm to generate an overhead image of the ground 10 from the time-series color images or preprocessed images of the time-series color images.

[0104] In the above description, it was assumed that all points in three-dimensional space (e.g., P1, P2) are located on the reference plane Re (e.g., Z1 = Z2 = 0). If the height of the crop relative to the reference plane Re is not 0, the positions of corresponding points will shift from their correct positions in the top-view image after homography transformation. To suppress an increase in the amount of shift, it is desirable that the height of the reference plane Re be close to the height of the crop to be detected. The ground 10 may have unevenness such as ridges, furrows, and grooves. In such cases, the reference plane Re may be displaced upward from the bottom of such unevenness. The distance of the displacement can be appropriately set according to the unevenness of the ground 10 on which the crop is cultivated.

[0105] Furthermore, when the agricultural machine 100 is traveling on the ground 10, if a roll or pitch movement occurs in the vehicle body 110 (see FIG. 11), the attitude of the image capture device 120 changes, which may change the contents of the transformation matrix H1. In such a case, if the roll and pitch rotation angles of the vehicle body 110 are measured by the IMU, it is possible to correct the transformation matrix H1 and the transformation matrix H in accordance with the change in attitude of the image capture device.

[0106] The processing device 122 in this embodiment generates a top view image of the ground viewed from above using the above-mentioned method, in which the crop row color index value is classified into first pixels whose index value is greater than or equal to a threshold value and second pixels whose index value is less than the threshold value. Then, the processing device 122 executes operation S3.

[0107] Next, operation S3 will be described.

[0108] In operation S3, the processing unit 122 determines the position of the edge line of the crop row based on the index value of the first pixel. Specifically, the processing unit 122 accumulates the index values ​​of the first pixels (pixels whose color index value is equal to or greater than a threshold) along multiple scan lines in the top-view image.

[0109] FIG. 23 is an example of a top-view image 44 showing three crop rows 12. In this example, the direction of the crop rows 12 is parallel to the vertical direction of the image (v-axis direction). FIG. 23 shows a number of scanning lines (dashed lines) S that are parallel to the vertical direction of the image (v-axis direction). The processing device 122 accumulates the index values ​​of pixels located on multiple scanning lines S for each scanning line S to obtain an accumulated value.

[0110] FIG. 24 is a diagram schematically illustrating the relationship between the position of the scanning line S and the integrated value of the index value (a histogram of the integrated value) obtained for the top-view image of FIG. 23. The horizontal axis of FIG. 24 indicates the position of the scanning line S in the horizontal direction of the image (the u-axis direction). In the top-view image 44, if many of the pixels crossed by the scanning line S are first pixels belonging to the crop rows 12, the integrated value of the scanning line S will be large. On the other hand, if many of the pixels crossed by the scanning line S are second pixels (background pixels) belonging to the intermediate region (work passage) 14 between the crop rows 12, the integrated value of the scanning line S will be small. Note that in this embodiment, the intermediate region (work passage) 14 is masked, and the index value of the second pixel is zero.

[0111] The histogram in FIG. 24 includes concave regions where the integrated value is zero or close to zero, and convex regions separated by these concave regions. The concave regions correspond to the intermediate region (work passage) 14, and the convex regions correspond to the crop rows 12. In this embodiment, predetermined positions on both sides of the peak integrated value in the convex region—specifically, the position of the scan line S having an integrated value that is a predetermined percentage of the peak integrated value (for example, a value selected from the range of 60% to 90%)—are determined as the position of the edge line of the crop row 12. The two ends of the arrow W in FIG. 24 indicate the position of the edge line of each crop row 12. In the example of FIG. 24, the position of the edge line of each crop row 12 is the position of the scan line S having a value that is 80% of the peak integrated value of each crop row 12.

[0112] In this embodiment, the second pixels are masked, and then the index values ​​of the color of the crop row on each scan line S are accumulated. That is, the number of first pixels (pixel count) is not counted for the top-view image binarized based on the classification of the first and second pixels. When the number of first pixels is counted, for example, if there are many pixels (classified as first pixels) that slightly exceed the threshold Th due to fallen leaves or weeds, the count value of the first pixels will increase. In contrast, accumulating the index values ​​of the crop row color of the first pixels, rather than the number of first pixels, as in the embodiment of the present disclosure, reduces erroneous determinations due to fallen leaves or weeds and improves the robustness of row detection.

[0113] Fig. 25 is an example of a top-view image 44 in which the crop rows 12 extend at an angle. As described with reference to Fig. 13 and Fig. 14 , depending on the orientation of the agricultural machine 100, the direction in which the crop rows 12 extend in the image 40 acquired by the imaging device 120 may be tilted to the right or left within the image. When a top-view image 44 is generated from such an image by homography transformation, the direction of the crop rows 12 is tilted from the vertical direction of the image (v-axis direction), as in the example of Fig. 25 .

[0114] FIG. 25 also shows a number of scanning lines (dashed lines) S parallel to the vertical direction (v-axis direction) of the image. When the processing device 122 accumulates the index values ​​of pixels located on these multiple scanning lines S for each scanning line S to obtain an accumulated value, a histogram of the accumulated values ​​is obtained, as shown in FIG. 26. FIG. 26 is a diagram schematically showing the relationship between the position of the scanning line S and the accumulated value of the index values, obtained for the top-view image of FIG. 25. The edge lines of the crop rows 12 cannot be determined from this histogram.

[0115] FIG. 27 is a flowchart showing an example of a procedure for searching for a direction (angle) of the scanning line S parallel to the direction of the crop rows 12 by changing the direction (angle) of the scanning line S.

[0116] In step S10, the direction (angle) of the scanning line S is set. Here, the clockwise angle with respect to the u axis in the image coordinate system is set as θ (see FIGS. 23 and 25). The range for searching the angle θ can be set to, for example, 60 to 120 degrees, and the angle increment can be set to, for example, 1 degree. In this case, in step S1, 60, 61, 62, . . . , 119, and 120 degrees are given as the angle θ of the scanning line S.

[0117] In step S12, the index values ​​for the pixels on the scanning line S extending in the direction of each angle θ are integrated to create a histogram of the integrated values. The histogram will show a different distribution depending on the angle θ.

[0118] In step S14, from the multiple histograms thus obtained, a histogram is selected in which the boundaries between convex and concave portions are sharp, as shown in FIG. 24, and the crop rows 12 are clearly distinguished from the intermediate region 14, and the angle θ of the scanning line S that generates that histogram is determined.

[0119] In step S16, an edge line of each crop row 12 is determined from the peak value of the histogram corresponding to the angle θ determined in step S14. As described above, the position of the scan line S having an integrated value that is, for example, 0.8 times the peak may be adopted as the edge line.

[0120] When searching for the direction (angle) of the scanning line S, a histogram of the integrated values ​​on the scanning line S at each angle θ may be created each time the angle θ is changed by 1 degree within the search range. Feature quantities (e.g., depth of recesses / height of protrusions, differential values ​​of envelopes, etc.) may be calculated from the waveform of the histogram, and it may be determined whether the direction of the crop rows 12 and the direction of the scanning line S are parallel or not based on the feature quantities.

[0121] The method for determining the angle θ is not limited to the above example. If the direction in which the crop rows extend is known through measurement, the direction of the agricultural machine 100 may be measured using an inertial measurement unit (IMU) mounted on the agricultural machine 100, and the angle θ with respect to the direction in which the crop rows extend may be determined.

[0122] FIG. 28 is a diagram showing an example of an integrated value histogram created from the top-view image of FIG. 21. For the convex portion of the histogram located in the center, the scanning line position 0.8 times the peak value is set as the position of edge line E. In this histogram, the peak of the convex portion becomes lower and wider as the scanning line position moves left and right from the center. This is because, as is clear from the image of FIG. 21, there is little image distortion in the center of the top-view image, but image distortion increases as the distance from the center to the left and right increases, and the black triangular areas located on both sides of the bottom side decrease the integrated value.

[0123] When crop row detection is used to guide agricultural machinery, the crop rows that need to be detected accurately are those in the center or its periphery of the image, so distortions in the areas near the left and right ends of the top-view image can be ignored.

[0124] FIG. 29 is a block diagram showing a series of processes executed by the processing device 122 in this embodiment. As shown in FIG. 29, the processing device 122 executes image acquisition 32, enhanced image generation 33, crop row extraction 34, and homography transformation 35 to obtain a top-view image 44, such as that shown in FIG. 25. The processing device 122 further executes scan line direction determination 36 and edge line position determination 37 to obtain the positions of the edge lines of the crop rows. Thereafter, the processing device 122, or a path generation device that has received information indicating the positions of the edge lines from the processing device 122, can execute target path generation 38 for the agricultural machine based on the edge lines. The target path can be generated so that the wheels of the agricultural machine are maintained within the intermediate region (work path) 14 between the edge lines E. For example, the target path can be generated so that the center in the width direction of a tire mounted on a metal wheel passes through the center of two edge lines located at both ends of the intermediate region (work path) 14. According to such a target route, even if the agricultural machine deviates from the target route by a few centimeters while traveling, it is possible to reduce the possibility of the tires entering the crop rows.

[0125] It has been confirmed that the embodiments of the present disclosure enable highly accurate detection of crop rows by suppressing the effects of sunlight conditions that change depending on weather conditions such as front lighting, back lighting, sunny days, cloudy days, and fog, as well as the time of day. It has also been confirmed that highly robust detection of crop rows is possible even when the type of crop (cabbage, broccoli, radish, carrot, lettuce, Chinese cabbage, etc.), growth state (from seedling to mature state), presence or absence of disease, presence or absence of fallen leaves and weeds, and soil color change.

[0126] In the above embodiment, the homography transformation is performed after determining a binarization threshold and extracting a crop region based on pixels equal to or greater than the threshold. However, the step of extracting a crop region may be performed after the homography transformation. Specifically, in the series of processes shown in FIG. 29, the homography transformation 35 may be performed between the enhanced image generation 33 and the crop row extraction 34, or between the image acquisition 32 and the enhanced image generation 33.

[0127] When detecting a crop row using the above method (algorithm), if a portion of the crop row is missing, as shown in FIG. 30, the integrated value of pixel values ​​decreases, as shown in FIG. 31. Therefore, in such an example, it is possible to detect the presence of a "missing portion" in the crop row or the "end of the field" based on the magnitude of the integrated value of pixel values. Furthermore, when the integrated value of pixel values ​​decreases and approaches the noise level, it becomes impossible to determine the edge line through calculation. This state corresponds to a state in which the crop row has been lost. According to the image recognition system 1000 of this embodiment, even if a portion of the crop row is lost due to a missing portion, the system can be configured to calculate an extension of the target path generated from the edge line used for row-following travel up to that point, and set this extension as the target path. This allows the agricultural machine 100 to temporarily travel along this extension. Furthermore, after detecting a "missing portion" or "end of the field" in a crop row, if the accumulated value of pixel values ​​increases while the vehicle continues traveling and a convex area appears in the accumulated value histogram, the image recognition system 1000 generates a target route using this convex area as a new crop row area.

[0128] A modified example of the sequence detection method executed by the image recognition system according to the present disclosure will now be described.

[0129] FIG. 32 is a diagram for explaining a method of dividing a part or the whole of a top-view image into a plurality of blocks and determining the position of an edge line for each of the plurality of blocks.

[0130] In this modified example, the processing device 122 divides part or all of the top-view image 44 into a plurality of blocks. Then, for each of the plurality of blocks, the position of the edge line E of the crop row 12 is determined. In the illustrated example, three blocks B1, B2, and B3 form a continuous band shape in the horizontal direction of the top-view image. The processing device 122 can determine the edge line of the crop row based on the band shape in a direction different from the traveling direction of the agricultural machine 100.

[0131] FIG. 33 is a diagram showing a schematic diagram of the relationship between the position of the scanning line S and the accumulated index value (accumulated value histogram) for each of blocks B1, B2, and B3 in the top-view image of FIG. 32. When accumulating, the scanning line S is always parallel to the vertical direction of the image. The index value is accumulated on a block-by-block basis, and there is no need to change the direction (angle) of the scanning line S. By shortening the length of the scanning line S, it becomes possible to properly detect the area of ​​the second pixel (background pixel) resulting from the intermediate region (work passage) 14, even if the crop row 12 extends at an angle. This eliminates the need to change the angle of the scanning line S.

[0132] The two ends of the arrow W in Figure 33 indicate the positions of the edge lines of the crop rows determined in each of the blocks B1, B2, and B3. In the example shown in Figure 32, the direction of the crop rows 12 is inclined with respect to the direction of the scan line S. For this reason, as described above, if the scan line position showing a value 0.8 times the peak value of the integrated value histogram is used as the position of the edge line E of the crop row 12, then the position of such edge line E corresponds to both ends of the "width" that passes through the vicinity of the center of the crop row 12 in each of the blocks B1, B2, and B3.

[0133] Figure 34 shows the crop row centers Wc for each of the blocks B1, B2, and B3 in Figure 33. The crop row centers Wc are determined from the centers of the arrows W that define the edge lines of the crop rows determined from the integrated value histogram in Figure 33, and are located at the center of each block in the vertical direction of the image. Figure 34 shows an example of an approximation line 12C for the crop row centers Wc that belong to the same crop row 12. The approximation line 12C is, for example, a straight line determined so as to minimize the root mean square of the distance (error) from multiple crop row centers Wc for each crop row 12. Such an approximation line 12C corresponds to a line that passes through the center of the crop row 12.

[0134] Figure 35 is a top view showing an example of edge lines E of crop rows 12 determined from the approximation line 12C in Figure 34. In this example, the two edge lines E associated with each crop row 12 are spaced apart by the same distance as the length of the arrow W and are equidistant from the approximation line 12C.

[0135] According to this modified example, it is not necessary to change the direction (angle) of the scanning line, and the edge line E of the crop row 12 can be determined with a smaller amount of calculation. The length of each block in the vertical direction of the image can be set to correspond to a distance of 1 to 2 meters on the ground, for example. In this modified example, one image is divided into three blocks to determine the integrated value histogram, but the number of blocks may be four or more. The shape of the blocks is also not limited to the above example. Within the top-view image, the blocks may have a continuous band shape in either the horizontal or vertical direction of the image. The processing device 122 can determine the edge line of the crop row by dividing the image into band-shaped blocks extending in a direction different from the traveling direction of the agricultural machine 100.

[0136] Fig. 36 schematically shows a state in which the crop rows 12 in the top-view image 44 include a curved portion. Fig. 37 schematically shows integrated value histograms for each of blocks B1, B2, and B3 in the top-view image 44 in Fig. 36.

[0137] Figure 38 is a diagram showing examples of the crop row centers Wc in each of the blocks B1, B2, and B3 in Figure 37, and an approximation line 12C for each crop row center Xc. In this example, the approximation line 12C is a curve (e.g., a higher-order curve such as a cubic curve) determined so as to minimize the root mean square of the distance (error) from the crop row center Wc of each crop row 12. Such an approximation line 12C corresponds to a curved line that passes through the center of the crop row 12, which has a curved portion.

[0138] Figure 39 is a top view showing an example of edge lines E of the crop rows 12 determined from the approximation line of Figure 38. The edge lines E are generated in a manner similar to that described with reference to Figure 35. That is, the two edge lines E associated with each crop row 12 are spaced apart by the same distance as the length of the arrow W and are equidistant from the approximation line 12C.

[0139] As explained above, by dividing the top-view image into multiple blocks and generating a histogram of integrated values ​​for each block, it becomes easy to determine the direction of the crop row, and even if the direction of the crop row changes along the way, it becomes possible to know the direction to which it has changed.

[0140] Any of the above sequence detection methods may be implemented in a computer and performed by causing the computer to perform the desired operations.

[0141] In this way, for example, by using a method of calculating the integrated values ​​for each of blocks B1, B2, and B3 as shown in top-view image 44 in Figure 36, it is possible to more accurately detect the location of the missing portion of the crop row. Furthermore, because the determined edge line is aligned with the crop row, even if the image recognition system 1000 loses sight of the crop row at the missing portion of the crop row, it is possible to continue traveling appropriately along the crop row in a straight line or curve during temporary traveling before stopping.

[0142] As can be seen from the above explanation, it is desirable to increase the number of scan lines and reduce the length of each block in the vertical direction of the image in order to determine the position of the edge line E of the crop row 12 with high positional accuracy. However, performing row detection using the above method for all of the time-series images acquired by the imaging device 120 or for the entire range of each image may increase the computational load on the processing device 122.

[0143] As described above, after the image recognition system 1000 starts image processing in response to the operator selecting the automatic steering mode using the mode switch 114, low-position accuracy detection processing may be performed until a row region such as a crop row is detected. This low-position accuracy detection processing may be performed, for example, on a portion of images selected from a time-series image acquired by the imaging device 120. When a row region is detected, it is desirable to start detection processing with increased position accuracy by increasing the number of scanning lines, reducing the length of each block in the vertical direction of the image, or increasing the number of blocks. Once a row region is detected and it is determined that row-following travel is possible, the operator may issue a command to start row-following travel using the start switch 112, and then detection processing with increased accuracy may be started.

[0144] In this way, by dividing the processing performed by the image recognition system 1000 into two or more stages, the computation load on the processing device 122 can be effectively reduced.

[0145] The above blocks may be set within a region (region of interest) selected from the image to determine line detection. When starting line-following traveling, the image recognition system 1000 may select a region of interest to be subjected to image recognition processing, and may perform processing to make the region of interest during line-following traveling smaller than the region of interest before line-following traveling. Specifically, when performing line-following traveling, a relatively narrow range that includes the line region to be followed may be selected as the region of interest.

[0146] (Image Recognition System Embodiment 2) Row detection by an image recognition system according to a second exemplary embodiment of the present disclosure will be described. In this embodiment, ridge detection is performed as "row detection."

[0147] FIG. 40 is a perspective view showing a row of ridges 16 on the ground 10. A "ridge" is a location where plants are planted in rows or rows. It is a convex, generally linear, raised area with soil mounded at intervals. The cross-sectional shape of the ridges 16 perpendicular to the direction in which the ridges 16 extend can be roughly trapezoidal, semicircular, or semicircular. FIG. 40 shows a schematic representation of a ridge 16 with a trapezoidal cross-section. Actual ridges do not have the simple shape shown in FIG. 40. Between two adjacent ridges 16 is an intermediate region 14, called a furrow space. The intermediate region 14 functions as a work passage. Crops may be planted on the ridges 16, or they may be unplanted, with only the soil exposed. Each ridge 16 may also be covered with mulch.

[0148] The height, width, and spacing of the ridges 16 do not need to be uniform and may vary depending on the location. The height of the ridges 16 is generally the difference in elevation between the ridges and the spacing between the ridges. In this specification, the "height" of the ridges 16 is defined as the distance from the reference plane Re described above to the top surface of each ridge 16.

[0149] In the example of Figure 40, the edge lines of the ridges 16 are clear. However, in reality, the ridges 16 are part of the ground surface 10 that continues from the intermediate region 14, and as described above, the "cross-sectional shapes" of the ridges 16 vary, so the boundaries between the ridges 16 and the intermediate region 14 are not necessarily clear. In the embodiment of the present disclosure, the edge lines of the ridges 16, i.e., the boundaries between the ridges 16 and the intermediate region 14, are defined as positions located on both sides of the peak of each ridge 16 and at a height that is a predetermined percentage of the peak. The position of the edge line is, for example, a position that has a height that is 0.8 times the peak of each ridge 16.

[0150] 2, the image recognition system 1000 according to this embodiment also includes an imaging device 120 and a processing device 122 that performs image processing of time-series color images acquired from the imaging device 120. The hardware configuration of the processing device 122 is the same as the configuration of the processing device 122 in the first embodiment.

[0151] In this embodiment, the processing device 122 acquires time-series images from the imaging device 120 and executes the following operations S21, S22, and S23. (S21) A first movement amount within the image plane of each of a plurality of feature points is obtained by feature point matching from a plurality of images acquired at different times in the time-series images. (S22) Each of the plurality of feature points is perspectively projected from the image plane onto a reference plane corresponding to the ground, and a second movement amount of each projected point within the reference plane is calculated based on the first movement amount. (S23) Based on the second movement amount, the heights of the plurality of feature points from the reference plane are estimated to detect ridges on the ground.

[0152] Specific examples of operations S21, S22, and S23 will be described in detail below.

[0153] First, operation S21 will be described. In operation S11, a first movement amount within an image plane of each of a plurality of feature points is determined by feature point matching from a plurality of images acquired at different times in the time-series images. The time-series images are a collection of images acquired in time series by the imaging device 120 through photography. The time-series images do not need to be color images, but may be color images. When the imaging device 120 outputs time-series color images, the processing device 122 may grayscale the color images to be processed in the time-series color images. As described in the first embodiment, each image is composed of a group of pixels in units of frames. The frame rate is also as described in the first embodiment.

[0154] Figure 41 shows image 40(t), one frame in a time-series of images captured at time t by an imaging device (monocular camera in this example) 122 mounted on agricultural machine 100. In this example, no crops are planted on ridge 16. The data for image 40(t) captured by the monocular camera does not contain depth information. For this reason, it is not possible to determine the difference in elevation between ridge 16 and intermediate region 14 from a single image 40(t).

[0155] The imaging device 120 acquires an image 40(t+1), an image 40(t+2), an image 40(t+3), and so on in chronological order not only at time t but also at other times, for example, at times t+1, t+2, t+3, and so on. The multiple images acquired in chronological order by the imaging device 120 while the agricultural machine 100 is traveling may each include the same area of ​​the ground 10 in a partially overlapping manner.

[0156] In this embodiment, the processing device 122 extracts feature points from image 40(t), image 40(t+1), .... A "feature point" is a pixel whose brightness value or color can be distinguished from surrounding pixels, and whose position can be identified within the image. Extracting feature points within an image makes it possible to match multiple images of the same scene. In areas of an image where brightness value and color are uniform, it is difficult to distinguish the pixels within that area from the surrounding pixels. For this reason, feature points are selected from areas of the image where brightness value or color changes locally. A feature point is a pixel or group of pixels that has a "local feature."

[0157] In this embodiment, the purpose of extracting feature points is to measure the amount of movement of feature points by performing feature point matching on the time-series images 40(t), 40(t+1), etc., acquired by the agricultural machine 100 while it is moving. The processing device 122 can extract feature points suitable for such feature point matching through image processing. Examples of image processing feature point extraction algorithms include SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Feature), KAZE, and A-KAZE (Accelerated-KAZE). Like SIFT or SURF, KAZE and A-KAZE are highly robust feature point extraction algorithms that are resistant to changes in scaling, rotation, and lighting. Unlike SIFT and SURF, KAZE and A-KAZE do not use a Gaussian filter. Therefore, KAZE and A-KAZE are less susceptible to changes in rotation, scale, and brightness, and can extract feature points even from areas of the image where brightness and color change are relatively small. This makes it easy to extract feature points suitable for feature point matching even from images such as soil surfaces. A-KAZE also has the advantage of being more robust and capable of increasing processing speed compared to KAZE. In this embodiment, feature points are extracted using the A-KAZE algorithm. However, the algorithm for feature point matching is not limited to this example.

[0158] 42 schematically shows the correspondence between feature points in an image 40(t) captured by an imaging device at time t and an image 40(t+1) captured at time t+1. Here, the time interval between time t and time t+1 can be, for example, 100 milliseconds to 500 seconds.

[0159] A feature point matching algorithm is used to associate multiple feature points extracted from image 40(t) with multiple corresponding feature points in image 40(t+1). In FIG. 42, eight pairs of corresponding feature points are connected by arrows. In this embodiment, the processing device 122 can extract, for example, hundreds to more than a thousand feature points from each of image 40(t) and image 40(t+1) using A-KAZE. The number of feature points to be extracted can be determined based on the number of images processed per second.

[0160] After performing this feature point matching, the processing device 122 calculates the amount of movement (first movement amount) within the image plane for each of the multiple feature points. The first movement amount calculated from the two images 40(t) and 40(t+1) does not have a common value for all feature points. The first movement amount indicates different values ​​due to the physical height difference of the feature points on the ground 10.

[0161] FIG. 43 is a perspective view schematically showing the movement of the ridge 16 and the intermediate region (work passage) 14 reflected in images acquired by the imaging device 120, and also shows an image 40(t) and an image 40(t+1) schematically. FIG. 43 also shows a state in which point F1 on the ridge 16 and point F2 on the intermediate region (between the furrows or the work passage) 14 move horizontally to the left side of the drawing. This horizontal movement is a relative motion caused by the imaging device 120, which is fixed to the agricultural machine 100, moving to the right together with the agricultural machine 100. For simplicity, FIG. 43 shows the state in which the origin O of the camera coordinate system Σc in the imaging device 120 is stationary, and the ground 10 moves toward the left side. The height of the origin O of the camera coordinate system Σc is Hc. In the example shown, the ridge 16 is a simplified ridge having a height dH.

[0162] Image 40(t) in Figure 43 shows feature point f1 of ridge 16 and feature point f2 of intermediate region 14. These feature points f1 and f2 are examples of many feature points extracted using a feature point extraction algorithm such as A-KAZE. Image 40(t+1) shows feature points f1 and f2 after movement. For reference, image 40(t+1) also shows arrow A1 indicating the movement of feature point f1 from time t to time t+1 and arrow A2 indicating the movement of feature point f2. The length of arrow A1 (corresponding to the first movement amount) is greater than the length of arrow A2 (corresponding to the first movement amount). Thus, the movement amount of a feature point in an image (first movement amount) varies depending on the distance from the origin O of the camera coordinate system Σc to the corresponding point on the subject. This is due to the geometric properties of perspective projection.

[0163] Feature points f1 and f2 in image 40(t) are respectively points obtained by perspectively projecting points F1 and F2 on the surface of the ground 10, which is the subject, onto the image plane Im1 of the imaging device 120. Similarly, feature points f1 and f2 in image 40(t+1) are respectively points obtained by perspectively projecting points F1* and F2* on the surface of the ground 10, which is the subject, onto the image plane Im1 of the imaging device 120. The center point of the perspective projection is the origin O of the camera coordinate system Σc of the imaging device 120. Because perspective projection is a bidirectional relationship, points F1 and F2 can also be said to be points obtained by perspectively projecting feature points f1 and f2 in image 40(t) onto the ground 10. Similarly, points F1* and F2* can also be said to be points obtained by perspectively projecting feature points f1 and f2 in image 40(t) onto the ground 10.

[0164] 43, between time t and time t+1, point F1 on the ridge 16 moves to the position of point F1*, and point F2 on the intermediate region 14 moves to the position of point F2*. The distances of these movements are both equal to the distance traveled by the agricultural machine 100 between time t and time t+1 (horizontal movement distance). In contrast, the amounts of movement of feature points f1 and f2 on the image plane Im1 of the imaging device 120 are different from each other.

[0165] 44 is a diagram schematically showing the relationship between the amount of movement (L) of point F1 on ridge 16 corresponding to feature point f1 on image plane Im1 of imaging device 120 and the amount of movement (second amount of movement L+dL) of point F1p projected onto reference plane Re (projected point). In this example, the height of reference plane Re is made to coincide with the height of intermediate region (inter-furrow) 14, and the height of ridge 16 is set to dH.

[0166] As can be seen from Figure 44, point F1 on the ridge 16 has moved to the left by a distance (movement amount) L equal to the traveling distance of the agricultural machine 100, but the movement amount (second movement amount) of point F1p perspectively projected onto reference plane Re (projected point) is expressed as L + dL, which is longer than L. This is because point F1 on the ridge 16 is located higher than reference plane Re and is closer to the center of perspective projection (origin O of the camera coordinate system). Corresponding to this extra length dL, the movement amount (first movement amount) on image plane Im1 also increases.

[0167] The following formula can be derived from the ratio of the side lengths (similarity ratio) of the two similar triangles shown in Figure 44.

number

[0168] By transforming the above equation, the following equation is obtained.

number

[0169] The processing device 122 in this embodiment executes operation S22 to estimate the magnitude of the unevenness in the ground surface 10 based on the above equation. That is, each of the multiple feature points is perspectively projected from the image plane onto a reference plane Re corresponding to the ground surface 10, and the second movement amount (L+dL) of each projected point within the reference plane Re is calculated based on the first movement amount. The distance L in the above equation can be obtained by measuring the travel distance of the agricultural machine 100. Furthermore, the height Hc of the origin O of the camera coordinate system from the reference plane Re is known. Therefore, once the second movement amount (L+dL) is known, the height dH of the ridge 16 can be calculated from equation 7. The second movement amount (L+dL) can then be calculated from the first movement amount.

[0170] After performing operation S22, the processing unit 122 performs operation S23.

[0171] In operation S23, the processing device 122 estimates the height dH of each characteristic point from the reference plane Re based on the second movement amount (L+dL) of each characteristic point, and detects the ridges 16 of the ground surface 10.

[0172] As described above, in this embodiment, when the height of the center point O of the perspective projection from the reference plane Re is Hc, the heights of multiple feature points from the reference plane Re are dH, the second movement amount of a feature point on the reference plane Re (where dH is zero) is L, and the second movement amount of a feature point where dH is greater than zero is L+dL, Hc·(1.0−L / (L+dL)) can be determined as the height of each feature point.

[0173] When determining the second movement amount from the first movement amount, a homography transformation can be used. Specifically, the inverse matrix H1 of the above-mentioned transformation matrix H1 -1The coordinates of each feature point on the image plane Im1 can be transformed into the coordinates of the corresponding point on the reference plane Re using the above equation. Therefore, the processing device 122 first obtains a first movement amount from the coordinates of each feature point on the image plane Im1 before and after movement. Next, the processing device 122 transforms the coordinates of each feature point into the coordinates of the corresponding point on the reference plane Re by homography transformation, and then obtains a second movement amount from the coordinates on the reference plane Re before and after movement.

number

[0174] Fig. 45 is a block diagram showing a series of processes executed by the processing device 122 in the second embodiment. As shown in Fig. 45, the processing device 122 executes image acquisition 52, feature point matching 53, movement amount calculation 54, and feature point height estimation 55. As a result, an estimate of the height from the reference plane Re can be obtained for each of a large number of feature points in the image. A two-dimensional map of such height estimates shows the distribution of height differences of the unevenness on the surface of the ground 10.

[0175] As described in the first embodiment, multiple scan lines are set in this embodiment. However, in this embodiment, the average height of the feature points is calculated along each scan line. Furthermore, by changing the direction (angle) of the scan line, the direction of the ridges can be found from the distribution of the average heights of the feature points. Once the direction of the ridges 16 can be determined, the edge lines of the ridges 16 can be determined using a method similar to that used to determine the edge lines of the crop rows 12. Note that, as described with reference to Figure 32 and other figures, if a method of dividing the image into multiple blocks is adopted, the scan line direction determination 56 can be omitted.

[0176] Thus, the processing device 122 in this embodiment executes scan line direction determination 56, edge line position determination 57, and target path generation 58, as shown in FIG.

[0177] Figure 46 is a diagram showing the relationship between the average height of feature points on a scan line parallel to the direction in which the ridges extend and the position of the scan line. In the graph of Figure 46, the horizontal axis represents the position of the scan line, and the vertical axis represents the average height of feature points on each scan line. As shown in the graph, the average height repeatedly increases and decreases as the position of the scan line moves from left to right. The position where the average height reaches its peak corresponds to the center of the ridge. The curve representing the average height forms a valley midway between two adjacent peaks. This valley corresponds to the vicinity of the center of the intermediate region (furrow space or work passage) 14.

[0178] In this embodiment, the processing device 122 determines, as the edge line of the ridge, positions on both sides of the peak position indicated by the height average value and having a height that is a predetermined percentage (e.g., 0.8 times) of the peak. Above the graph in FIG. 46, white arrows indicating the positions of the edge lines of two ridges in the image are shown. When detecting ridges using the above method (algorithm), for example, if a defect exists in a portion of the ridge, the height average values ​​will be approximately the same, making it impossible to determine the edge line by calculation. This state corresponds to a state in which the ridge has been lost. Note that, in this embodiment, as described with reference to FIGS. 32 to 39, the image may be divided into multiple blocks and the average height of feature points on the scanning line may be calculated for each block. Furthermore, in this embodiment, the image recognition system 1000 may be configured to calculate an extension of the target path generated from the edge line used for the previous ridge tracking travel, and to use this extension as the target path, even if a portion of the ridge is lost due to a defect. This allows the agricultural machine 100 to temporarily travel along this extension. Furthermore, after the image recognition system 1000 detects a "missing part" or "end of the field" of a ridge, the average height of each scanning line will begin to show differences while the system continues to travel, and when a peak area appears, the system generates a target route using this peak area as a new ridge row area.

[0179] According to this embodiment, row detection does not depend on the "color of the crop row," which has the advantage that it is less affected by the type of crop or sunlight conditions. We have confirmed that it is possible to detect not only high ridges such as the "raised ridges" often created in vegetable cultivation, but also relatively low ridges in the range of 5 to 10 centimeters in height.

[0180] The processing device 122 may simultaneously or selectively detect the crop row in the first embodiment and the ridge in the second embodiment. When a crop is planted in a ridge, the edge line of the crop row and the edge line of the ridge are determined. The target path of the agricultural machine can be determined based on both or one of the edge lines.

[0181] The processing device 122 may calculate the reliability of detection for each of the crop row detection and the ridge detection. The reliability of the crop row detection may be determined based on, for example, the distribution of the integrated values ​​of the index values ​​shown in FIG. 33 or the magnitude of the peak value. The reliability of the ridge detection may be determined based on, for example, the magnitude of the difference between the maximum value and the minimum value in the height distribution shown in FIG. 46. For example, when a target route is generated based on the edge lines of detected crop rows and the agricultural machine is traveling along the target route, ridge detection may be performed in the background at a point where crop row detection becomes impossible or the reliability falls below a predetermined level so that the target route is generated based on the edge lines of the ridges.

[0182] In addition, in the case where the processing device 122 is capable of performing both crop row detection and ridge detection, one or both of the crop row detection and ridge detection may be performed depending on the operator's selection.

[0183] (Image Recognition System Embodiment 3) The following describes the sequence detection in the third exemplary embodiment of the present disclosure, where the selection of the region of interest will be described in detail.

[0184] 47 shows an example of the basic configuration of an image recognition system 1000 according to this embodiment. The image recognition system 1000 includes a processing device 122 having the same hardware configuration as in the other embodiments. The processing device 122 selects a region of interest for detecting at least one of a crop row and a ridge from the time-series images. The region of interest has a size and shape that includes at least a portion of a wheel.

[0185] Figure 48 shows an example of an image 40 acquired by the processing device 122 from the imaging device 120. The image 40 is one of a series of images. The image 40 shows the crop rows 12, the intermediate region 14, part of the vehicle body 110 of the agricultural machine 100, and part of the front wheel 104F. For reference, edge lines are indicated by white lines in Figure 48.

[0186] Figure 49 is a diagram showing a portion of the image in Figure 48. In Figure 49, a portion of the vehicle body 110 of the agricultural machine 100 and a portion of the front wheel 104F shown in image 40 are surrounded by a white line. In image 40 in Figure 49, an example of a region of interest 60 is shown by a trapezoidal dashed line that includes a portion of the front wheel 104F. This region of interest 60 has a shape that includes, of at least one of the crop rows and ridges in image 40, the crop row or ridge located on the left side of the front wheel 104F to the crop row or ridge located on the right side.

[0187] As can be seen from the example image in Fig. 21, even in the top view image, distortion is greater in the periphery than in the center. As a result, as shown in Fig. 28, for example, the peak value decreases and the interval between peaks increases as the position of the scan line moves away from the center.

[0188] On the other hand, the crop rows or furrows to be detected, which are necessary for selecting the target path, are located near the front of the traveling agricultural machine. More specifically, it is sufficient to accurately detect the crop rows or furrows located around the wheels of the traveling device of the agricultural machine. In this embodiment, by performing row detection on only a partial area rather than the entire image acquired by the imaging device 120, it is possible to reduce the amount of calculations performed by the processing device 122 and the time required for calculation. In addition, outliers caused by distortion in the periphery of the image can be excluded, thereby improving the accuracy of row detection.

[0189] Selection of the region of interest 60 (region setting) depends on the position and orientation at which the imaging device 120 is attached to the agricultural machine 100, as well as the structure or shape of the agricultural machine 100. For example, after attaching the imaging device 120 to the agricultural machine 100, the extent (shape, size, position) of the region of interest 60 may be determined manually while checking an image obtained from the imaging device 120 on a monitor screen. Alternatively, the extent of the region of interest 60 may be determined based on the optical performance of the imaging device 120, the attachment position, the model of the agricultural machine, etc., and input to the processing device 122.

[0190] The processing device 122 in this embodiment may be configured to detect at least a portion of the wheel 10F from the image 40 as shown in Fig. 49 using, for example, image recognition technology. In this case, it becomes possible to adaptively change the range of the region of interest 60 so that an area including at least a portion of the detected front wheel 104F is selected as the region of interest 60.

[0191] The processing device 122 may estimate a positional relationship between the front wheel 104F and at least one of the detected crop rows 12 and ridges 16 based on an image of the portion of the front wheel 104F included in the region of interest 60. The processing device 122 may be configured to estimate a positional relationship between the agricultural machine 100 and at least one of the detected crop rows 12 and ridges 16 based on such a positional relationship.

[0192] There are cases where the processing device 122 does not have information indicating the accurate position of the front wheels 104F relative to the agricultural machine 100. Such information indicating the position is, for example, the coordinates of the front wheels 104F relative to the body coordinate system Σb fixed to the agricultural machine 100. Even if such coordinates are stored in advance in the storage device 28 of the processing device 122, accuracy will be lost if, for example, the operator changes the tire size of the front wheels 104F or changes the spacing between the left and right front wheels 104F. In such cases, the processing device 122 may detect a portion of the front wheels 104F included in the region of interest 60 and estimate the position of the front wheels 104F relative to the agricultural machine 100 based on an image of the detected portion of the front wheels 104F.

[0193] FIG. 50 is a top view diagram of a portion of the ground 10 on which the crop rows 12 are provided. A pair of front wheels 104F are depicted in FIG. 50. The rectangular area 62 in this top view diagram is a top-view image generated by performing the above-described homography transformation on the region of interest 60 of the image in FIG. 49. The vehicle body 110 shown in the region of interest 60 in FIG. 49 is omitted from FIG. 50. Furthermore, because the image of the portion of the front wheels 104F shown in the region of interest 60 is significantly deformed by the homography transformation, FIG. 50 depicts the shape of the front wheels 104F as "parallel projections" onto the reference plane Re. Furthermore, for reference, FIG. 50 also depicts a schematic representation of the tire treads (contact surfaces) CA where the front wheels 104F contact the ground 10. The center-to-center distance T between the left and right tire treads CA is the "tread width (track)."

[0194] Generally, the position of the tire tread CA relative to the vehicle body 110 of the agricultural machine 100 is known. Therefore, the positional relationship of the tire tread CA relative to the top view image (rectangular area) 62 of the region of interest 60 is also known. However, by setting the region of interest 60 to include at least a portion of one or more wheels, as in this embodiment, the following effects can be obtained.

[0195] The structure of the vehicle body 110 varies depending on the model, and the tread width T (the center-to-center distance of the tire tread CA) may also vary depending on the model. Furthermore, even for the same model, the operator may change the tread width T, as described above. Therefore, selecting the shape and size of the region of interest 60 so that it includes the wheels 104 shown in the image enables image processing that is compatible with a variety of models and can accommodate cases where the tread width T is changed by the operator.

[0196] It is no longer necessary to input the position of the tire tread CA as coordinates in the body coordinate system Σb in advance. Based on the image captured by the imaging device 120, it becomes possible to automatically obtain the coordinates of the front wheel 104F or the tire tread CA in the body coordinate system Σb.

[0197] It is possible to monitor the position error between the wheel and the edge line of the row determined by the image recognition system, or the target path generated based on the edge line, based on the image.

[0198] As mentioned above, when a top-view image of the ground is generated by homography transformation, the wheels are deformed. In order to accurately estimate the positional relationship of the wheels (particularly the tire tread CA) with respect to the row edge line or the target path, it is desirable to correct the homography transformation. This point will be explained below.

[0199] FIG. 51 is a diagram schematically illustrating the positional relationship between points P3 and P4 included in a portion of the front wheel 104F shown in the image 40 and corresponding points P3' and P4' obtained by perspectively projecting these points P3 and P4 onto the reference plane Re. The coordinates of the points P3 and P4 in the world coordinate system are (X3, Y3, Z3) and (X4, Y4, Z4), respectively. The coordinates of the corresponding points P3' and P4' in the world coordinate system are (X3', Y3', 0) and (X4', Y4', 0), respectively. As can be seen from FIG. 51, the points P3 and P4 are located higher than the reference plane Re. Therefore, when a top-view image viewed from directly above the reference plane Re is generated by homography transformation, the X and Y coordinates of the corresponding points P3' and P4' on the reference plane Re are shifted from the X and Y coordinates of the points P3 and P4, respectively. Therefore, when a homography transformation is performed on image 40, which shows part of the front wheel 104F, to generate a top-view image, the image of the front wheel 104F is formed in a distorted shape in the top-view image, making it difficult to estimate the accurate positional relationship.

[0200] To determine the positional relationship between the front wheel 104F and the edge line of the crop row 12 or ridge 16 based on such a top-view image, it is preferable to estimate the center of the tire tread CA based on the coordinates (X3', Y3', 0) and (X4', Y4', 0) of the corresponding points P3' and P4'.

[0201] 51, if the height Ht of the front wheel 104F is known, the positions of points P3 and P4 on the front wheel 104F shown in the image can be estimated from the shape in the image using a technique such as pattern matching. Once the positions of points P3 and P4 on the front wheel 104F are estimated, the center position of the tire tread CA can be estimated by correcting, for example, the coordinates (X3', Y3', 0) and (X4', Y4', 0) of the corresponding points P3' and P4'.

[0202] Thus, in this embodiment, by including at least a part of the wheels within the region of interest, it becomes possible to monitor the positional relationship of the wheels with respect to the row detected from within the region of interest using time-series images.

[0203] (Image Recognition System Embodiment 4) 52 is a block diagram schematically illustrating another example of the basic configuration of an agricultural machine according to the present disclosure. In this example, the agricultural machine 100 includes an image recognition system 1000, a traveling device 145 including a steering wheel, and a control device 180 that controls the traveling device 145.

[0204] Next, reference will be made to Fig. 53. Fig. 53 is a top view that schematically shows the imaging range of the ground in time-series images acquired by the imaging device 120 of the agricultural machine 100.

[0205] The processing device 122 included in the agricultural machine 100 according to the present disclosure selects a search area 314 divided into multiple blocks B(1,1), . . . , B(i,j), . . . , B(m,n) from the time-series images output by the imaging device 120. Here, B(i,j) is a code indicating the block located at row i and column j within the search area 314. i, j, m, and n are all positive integers that satisfy the relationships 1≦i≦m, 1≦j≦n, and 2≦m×n. i identifies the “row” to which block B(i,j) belongs, and j identifies the “column” to which block B(i,j) belongs. m is the number of rows of block B(i,j) included in the search area 314, and n is the number of columns of block B(i,j) included in the search area 314. Block B(i,j) refers to any block within the search area 314.

[0206] The search area 314 can be selected from an imaged ground area 111 included in an image acquired by the imaging device 120. The imaged ground area 111 indicates a portion of the ground in an image acquired by the imaging device 120. For simplicity, the shape of the imaged ground area 111 in FIG. 53 is a fan shape, but the shape of the actual imaged ground area 111 is not limited to this example.

[0207] In the example of FIG. 53, the search region 314 is rectangular, and the blocks B(i,j) are arranged in rows and columns. The rectangular search region 314 may be defined by a horizontal size (width) Hs and a vertical size (depth) Vs on the ground. The depth Vs of the search region 314 may be, for example, in the range of 1 m to 3 m, and the width Hs may be, for example, in the range of 1 m to 5 m. Furthermore, the nearest side of the search region 314 may be located forward from the imaging device 120, for example, a distance of 1 m to 3 m. When the block B(i,j) is rectangular (including a square), the length of the long side or short side may be, for example, in the range of 0.2 m to 0.5 m.

[0208] The search region 314 does not need to have a rectangular shape, and may generally have a trapezoid, a sector, or any other arbitrary shape. The shape and / or size of the search region 314 may change while the agricultural machine 100 is traveling. Furthermore, in the example of FIG. 53 , block B(i,j) has the same shape and size regardless of its position. However, the shape and / or size of block B(i,j) may differ depending on its position within the search region 314, or may change while the agricultural machine 100 is traveling. When the size or shape of the search region 314 changes, the number of blocks included in the search region 314 (the values ​​of m and / or n) may change. When the size and shape of the search region 314 do not change, the number of blocks included in the search region 314 (the values ​​of m and / or n) and the shape and / or size of block B(i,j) may change.

[0209] Information or data defining the shape and size of the search region 314 and the shape, size, number, and arrangement of the blocks B(i,j) may be stored in advance in a storage device within the processing device 122. The processing device 122 can read the information or data from the storage device when executing the algorithm described below. In one embodiment, the processing device 122 generates a top-view image of at least the search region 314 of the ground from the time-series images and divides the search region 314 into a plurality of blocks. The generation of the top-view image will be described later.

[0210] The processing device 122 performs area classification of the ground in each of the multiple blocks B(i,j). "Area classification" includes classifying each of the multiple blocks B(i,j) into, for example, a "crop area" or a "non-crop area." A "crop area" is, for example, an area within the block where crops are estimated to exist. In contrast, a "non-crop area" is, for example, an area within the block where crops are estimated to not exist. In one embodiment, a "crop area" is an area that includes a target to be tracked in the row tracking mode, and a "non-crop area" is an area that does not include a target to be tracked. A "non-crop area" may be further classified into multiple other areas. For example, it may be classified into various areas such as a "ridge area," a "soil surface area," a "waterway area," and a "headland area." Details of the "area classification" process will be described later.

[0211] The processing device 122 detects row regions of at least one of crops and furrows established on the ground of the field based on the results of the region classification within the search region 314.

[0212] Figure 54 is a diagram that schematically shows an example of the arrangement of crop rows on the ground of a farm field, and the state of the agricultural machine 100 at multiple positions while moving. A search area 314 is shown in front of the agricultural machine 100. Crop rows 12 exist on the ground of the farm field, and between adjacent crop rows 12 there are strips of intermediate areas 14 where no crops are planted. Figure 54 shows examples of ridges 316 and water channels 318.

[0213] FIG. 55 is a plan view schematically illustrating the positional relationship between the search area 314 and the crop row 12 when the agricultural machine 100 is located at position P10 in FIG. 54. FIG. 56 is a diagram schematically illustrating the results of area classification of each block constituting the search area 314 at this time. In FIG. 56, hatched blocks are blocks classified as "crop area" or "ridge area," while unhatched blocks are blocks not classified as "crop area" or "ridge area." As shown in FIG. 56, if a hatched block exists columnwise ahead of an unhatched block, it can be determined that a "crop area" or "ridge area" exists. If such hatched blocks are consecutively arranged columnwise ahead, the first hatched block can be determined to be the beginning of a "crop area" or "ridge area." In this case, the arrangement direction of the hatched blocks can be determined to be the direction in which the row area extends. When the processing device 122 detects the start of a row area or the existence of a row area, it can determine whether or not to start row-following travel based on the relationship between the positions of the left and right wheels of the agricultural machine 100 and the row area and their orientations. In the example shown in Figure 56, the row direction extends in a direction that is inclined relative to the traveling direction of the agricultural machine 100 (the up and down direction in Figure 56). In this case, it cannot be determined that the left and right wheels can pass through the middle area 14 where no crops are planted, so row-following travel will not be started even if requested by the operator.

[0214] Figure 57 is a plan view that schematically shows the positional relationship between the search area 314 and the crop rows 12 when the agricultural machine 100 is at position P11 in Figure 54. Figure 58 is a diagram that schematically shows the results of area classification of each block that makes up the search area 314 at this time. In Figure 58, too, hatched blocks are blocks that have been classified as "crop areas" or "ridge areas," and unhatched blocks are blocks that have not been classified as either "crop areas" or "ridge areas."

[0215] Among the multiple blocks in the search area 314, blocks classified as crop areas or ridge areas constitute "row areas." Therefore, the processing device 122 in the present disclosure is able to detect row areas with spatial resolution on a block-by-block basis. Blocks determined to be row areas form a two-dimensional block arrangement pattern within the search area 314. Such a two-dimensional block arrangement pattern can dynamically change as the search area 314 moves. In other words, the search area 314 scans the ground of the field as the agricultural machine 100 moves. The resulting two-dimensional block arrangement pattern provides information useful for estimating the relative position and orientation of the agricultural machine 100 with respect to the arrangement of row areas across a wide area of ​​the field.

[0216] The processing device 122 can be configured to detect the row area to be copied based on a two-dimensional block arrangement pattern formed within the search area 314 by the row area detected within the search area 314, and to generate a signal for controlling the travel of the agricultural machine 100 in accordance with the arrangement of the detected row area. Furthermore, the processing device 122 can determine the direction in which the row area extends and detect at least one of the edge line, start end, end end, and missing portion of the row area based on such a two-dimensional block arrangement pattern.

[0217] FIG. 59 is a plan view schematically showing the positional relationship between the search area 314 and the crop rows 12 when the agricultural machine 100 is at position P12 in FIG. 54. FIG. 60 is a diagram schematically showing the results of area classification of each block constituting the search area 314 at this time. In FIG. 60, the hatched blocks are also blocks classified as "crop areas" or "ridge areas," while the unhatched blocks are blocks not classified as "crop areas" or "ridge areas." When the agricultural machine 100 is at position P12, the processing device 122 determines that the two-dimensional block arrangement pattern formed by the row areas detected within the search area 314 has an unhatched block located ahead of the hatched block in the row direction. In the case of this pattern, it is possible to determine that the agricultural machine 100 has reached the end of the row area. Then, based on such a determination, the processing unit 122 can generate a driving control signal, such as reducing the driving speed of the agricultural machine 100 or issuing a warning to the operator.

[0218] In the example shown in Figure 60, there are unhatched blocks column-wise ahead of all the hatched blocks. In this case, it can be determined that the end of the field has been reached. Furthermore, in the example shown in Figure 63 (described later), there is an unhatched block column-wise ahead of the central hatched block, and hatched blocks are consecutively present column-wise ahead of the hatched blocks on both sides. In this case, it can be determined that the "crop area" or "ridge area" of the central hatched block is missing. In this way, various information can be read from the two-dimensional block arrangement pattern without being bound by the subtle differences in the shapes of the crops.

[0219] In one embodiment, the processing unit 122 may acquire time-series color images from the image capture device 120 and perform the following operations S1, S2, S3, S4, and S5. (S1) An enhanced image is generated from the time-series color images, in which the color of the crop row being detected is enhanced. (S2) From the enhanced image, a top view image of the ground is generated, in which first pixels having a crop row color index value equal to or greater than a threshold value and second pixels having this index value less than the threshold value are classified. (S3') A search region divided into a plurality of blocks is selected from the top-view image, and ground region classification is performed for each of the plurality of blocks. In this embodiment, each block is classified into a crop region and a non-crop region based on the index value of the first pixel in each block. (S4') Based on the results of the area classification within the search area, row areas of at least one of crops and ridges established on the ground of the field are detected. (S5') A signal for controlling the travel of the agricultural machine is generated based on the two-dimensional block arrangement pattern formed by the row regions within the search region.

[0220] Here, operations S1 and S2 are the same as those described above, so specific examples of operations S3'-S5' will be described.

[0221] In this embodiment, the processing device 122 generates a top view image of the ground from above, in which the crop row color index value is classified into first pixels whose index value is greater than or equal to a threshold value and second pixels whose index value is less than the threshold value, in the same manner as in the previously described embodiment, and then performs operation S3'.

[0222] In operation S3', the processing device 122 selects a search area divided into multiple blocks from the top-view image and performs ground area classification for each of the multiple blocks. In this embodiment, each block is classified into a crop area and a non-crop area such as soil based on the index value of the first pixel in each block. An example of area classification for each block is described below.

[0223] FIG. 61 is a diagram showing an example of a search region 314 in a top-view image 44 showing three crop rows 12. In this example, the crop rows 12 are parallel to the vertical direction of the image (v-axis direction). The central crop row 12 is discontinued within the search region 314. In the example of FIG. 61, the search region 314 is rectangular and is divided into blocks of 5 rows and 9 columns. In each block, the processing device 122 accumulates, for each scan line, the index values ​​of pixels located on multiple scan lines parallel to the vertical direction of the image (v-axis direction) to obtain an accumulated value.

[0224] FIG. 62 is a diagram schematically illustrating the relationship between the position of the scan line and the integrated value of the index value (a histogram of the integrated value) obtained for each block in the search region in the top-view image of FIG. 61. The bottom side (horizontal axis) of each block in FIG. 62 indicates the position of the scan line in the horizontal direction of the image (the u-axis direction). In the search region 314 of the top-view image 44, if many of the pixels crossed by the scan line are first pixels belonging to the crop rows 12, the integrated value of the scan line will be large. On the other hand, if many of the pixels crossed by the scan line are second pixels (background pixels) belonging to the intermediate region (work passage) 14 between the crop rows 12, the integrated value of the scan line will be small. Note that in this embodiment, the intermediate region (work passage) 14 is masked, and the index value of the second pixel is zero.

[0225] In this embodiment, the second pixels are masked, and then the index values ​​of the color of the crop row on each scan line are accumulated. That is, the number of first pixels (pixel count) is not counted for the top-view image binarized based on the classification of the first and second pixels. When the number of first pixels is counted, for example, if there are many pixels (classified as first pixels) that slightly exceed the threshold Th due to fallen leaves or weeds, the count value of the first pixels will increase. In contrast, accumulating the index values ​​of the crop row color of the first pixels, rather than the number of first pixels, as in the embodiment of the present disclosure, reduces erroneous determinations due to fallen leaves or weeds and improves the robustness of row detection.

[0226] As is clear from the image in Figure 21, there is little image distortion in the center of the top-view image, but the image distortion increases as you move away from the center to the left or right. When using crop row detection to guide agricultural machinery, the crop rows that need to be accurately detected are the center of the image or its periphery. The search area 314 is located in a position that is not affected by distortion in areas near the left and right ends of the top-view image.

[0227] Next, operation S4' will be described.

[0228] In operation S4', based on the results of the area classification within the search area, row areas of at least one of crops and ridges established on the ground of the field are detected.

[0229] As shown in FIG. 62, the index value histogram varies for each block. The processing device 122 classifies each block into regions based on the sum of the index values ​​within the block or features such as the shape of the sum histogram. For example, the processing device 122 may be configured to classify a block into a "row region" if the sum of the index values ​​within the block exceeds a predetermined threshold, or into a "non-row region" if the sum is below the threshold. In this manner, the processing device 122 can detect row regions, which are at least one of crops and ridges on the ground of the field. Here, the processing device 122 determines the range of the row region to be traced based on the results of crop detection. However, in another embodiment described below, the processing device 122 detects ridges from the unevenness of the ground and determines the range of the row region accordingly.

[0230] Figure 63 shows a schematic example of a two-dimensional block arrangement pattern formed within the search area 314 by "row areas" detected within the search area 314. In Figure 63, blocks classified as "row areas" are hatched. Blocks that are not hatched correspond to areas of ground where no crops are planted.

[0231] 64 is a diagram showing an example of a search region 314 in a top-view image 44 showing two diagonally extending crop rows 12. In this example, the direction of the crop rows 12 is inclined with respect to the vertical direction of the image (v-axis direction). Even in such a case, the processing device 122 accumulates, for each scan line, the index values ​​of pixels located on multiple scan lines parallel to the vertical direction of the image (v-axis direction) in each block to obtain an accumulated value.

[0232] Fig. 65 is a diagram schematically showing the relationship between the position of the scanning line and the integrated value of the index value (histogram of the integrated value) obtained for each block in the search area in the top-view image of Fig. 64. As described above, the processing device 122 classifies each block into areas based on the total value of the integrated value within the block or feature amount such as the shape of the integrated value histogram.

[0233] Figure 66 shows a schematic diagram of another example of a two-dimensional block arrangement pattern formed within the search area 314 by a "row area" detected within the search area 314. In Figure 66, the arrangement of blocks (hatched) classified as a "row area" represents a diagonally extending crop row 12.

[0234] Based on such a two-dimensional block arrangement pattern, the processing device 122 can detect at least one of the edge line, the start end, the end end, and missing portions of the row region, or determine the direction in which the row region extends.

[0235] The processing device 122 can determine the position of the edge line of the crop row based on the index value of the first pixel. Specifically, within the search area of ​​the top-view image, the processing device 122 integrates the index values ​​of the first pixel (pixels whose color index value is equal to or greater than a threshold) along multiple scan lines in all blocks belonging to the same row. Then, for example, the position of the scan line having a value that is 80% of the peak integrated value of each crop row 12 may be determined as the edge line. In this case, it becomes possible to estimate the position of the edge line of each crop row 12 at a scale smaller than the size of the block (higher resolution).

[0236] In a preferred embodiment, the processing unit 122 performs calculations on a block-by-block basis to detect a "column region" within the search region 314, and then performs calculations to determine an edge line for the row containing the detected "column region."

[0237] Next, operation S5' will be described.

[0238] In operation S5', a signal for controlling the traveling of the agricultural machine is generated based on the two-dimensional block arrangement pattern formed by the row regions within the search area. As shown in Fig. 54, when the agricultural machine 100 is at each of positions P10 to P19, for example, the row regions may form a characteristic two-dimensional block arrangement pattern within the search area 314. This makes it possible to obtain useful information for controlling the traveling state of the agricultural machine 100 from the two-dimensional block arrangement pattern.

[0239] For example, when the agricultural machine 100 is at each of positions P10 to P19, the travel control shown below can be executed based on the two-dimensional block arrangement pattern formed by the row areas within the search area 314. Position P10: Queue tracking will not start even if requested by the operator. Position P11: Following the procession continues. Position P12: Preparations to stop the procession (e.g. deceleration and notification to the operator) are initiated. Position P13: Preparations for stopping the line tracking (for example, deceleration and notification to the operator) and detection of the next line area to be tracked begins. Position P14: The line area is completed and line tracking continues. Position P15: Row tracking is performed along the central row area. Position P16: Row tracking is performed along the curved row area. Position P17: Headland travel is carried out along the crop row 12 or waterway 318 detected on one side. Position P18: Headland travel is performed along the ridge 316 detected on one side. Position P19: The wheels of the agricultural machine 100 are positioned outside the intermediate region 14, so that the following travel does not start.

[0240] The running control performed by the processing device 122 based on the two-dimensional block arrangement pattern of the row region is not limited to the above example. The processing device 122 determines the content of the running control based on the two-dimensional block arrangement pattern and generates a signal for the determined running control. This signal is provided to the control device 180 of FIG. 52.

[0241] FIG. 67 is a flowchart showing an example of a procedure for the processing unit 122 to detect a row region.

[0242] In step S20, a search region divided into blocks is selected from the image.

[0243] In step S22, the color index values ​​of the pixels on the scan line in each block are integrated, and a histogram of the integrated values ​​is created.

[0244] In step S24, each block is classified into a region based on the integrated value, and as a result of the region classification, a row region is determined.

[0245] In step S26, a signal for controlling the travel of the agricultural machine 100 is generated based on the two-dimensional block arrangement pattern formed by the row areas determined in step S24. As described above, the two-dimensional block arrangement pattern changes depending on the relative position and orientation of the agricultural machine 100 with respect to the arrangement of row areas such as the crop rows 12, the intermediate areas 14, and other areas, as shown in Fig. 54, and can contain a wealth of information regarding various situations of the agricultural machine 100 in the field.

[0246] Fig. 68 is a block diagram showing a series of processes executed by the processing device 122 in this embodiment. As shown in Fig. 68, the processing device 122 executes image acquisition 32, enhanced image generation 33, crop row extraction 34, and homography transformation 35, thereby obtaining a top-view image 44 as shown in Fig. 21, for example.

[0247] The processing device 122 selects a search region 144 divided into multiple blocks from the top-view image 44, and then performs an integrated value histogram calculation 36A and a region classification 37A for each block to detect the row region. The processing device 122 then performs a travel control signal generation 38A. During this process, information indicating the position of the edge line may be provided from the processing device 122 to the control device 180. The control device 180 may generate a target path for the agricultural machine 100 based on the edge line. The target path may be generated so that the wheels of the agricultural machine are maintained within the intermediate region (work path) 14 between the edge lines. For example, the target path may be generated so that the center in the width direction of a tire mounted on a metal wheel passes through the center of two edge lines located at both ends of the intermediate region (work path) 14. Using such a target path, it is possible to reduce the possibility of the tires entering the crop rows even if the agricultural machine deviates from the target path by a few centimeters while traveling.

[0248] It has been confirmed that the embodiments of the present disclosure enable highly accurate detection of crop rows by suppressing the effects of sunlight conditions that change depending on weather conditions such as front lighting, back lighting, sunny days, cloudy days, and fog, as well as working hours. Furthermore, highly robust detection of crop rows is possible even when the type of crop (such as cabbage, broccoli, radish, carrot, lettuce, or Chinese cabbage), growth state (from seedling to mature state), presence or absence of disease, presence or absence of fallen leaves or weeds, or soil color changes.

[0249] In the above embodiment, the homography transformation is performed after determining a binarization threshold and extracting a crop region based on pixels equal to or greater than the threshold. However, the step of extracting a crop region may be performed after the homography transformation. Specifically, in the series of processes shown in FIG. 68, the homography transformation 35 may be performed between the enhanced image generation 33 and the crop row extraction 34, or between the image acquisition 32 and the enhanced image generation 33.

[0250] As can be seen from the above explanation, to determine the position of the edge line of the crop row 12 with high positional accuracy, it is desirable to increase the number of scan lines and shorten the length of each block in the vertical direction of the image. However, performing row detection using the above method for all time-series images acquired by the imaging device 120 or for the entire range of each image can increase the computational load on the processing device 122. However, according to this embodiment, the computation is performed on image data within a specific search area 314 that includes the target to be tracked for row tracking, thereby reducing the computational load. Furthermore, if the computation for determining the edge line of the crop row 12 with high positional accuracy is performed after roughly detecting the row region, such as the crop row 12, on a block-by-block basis, the total computational load can be reduced.

[0251] As described above, after the image recognition system 1000 starts image processing in response to the operator selecting the automatic steering mode with the mode switch, it executes low-position accuracy detection processing on a block-by-block basis until it detects a row region such as a crop row. Once a row region is detected, it is desirable to start detection processing with improved position accuracy by increasing the number of scanning lines, shortening the length of each block in the vertical direction of the image, or increasing the number of blocks. Once a row region is detected and it is determined that row-following travel is possible, the operator may issue a command to start row-following travel with the start switch, and then detection processing with improved accuracy may be started.

[0252] In this way, by dividing the processing performed by the image recognition system 1000 into two or more stages, the computation load on the processing device 122 can be effectively reduced.

[0253] FIG. 69 is a diagram showing blocks within the search area 314 that are related to the path traveled by the wheels of the agricultural machine 100. In this example, the area (work path) traveled by the wheels of the agricultural machine 100 while traveling in a line-following manner is the hatched blocks B(i,2), B(i,3), B(i,7), and B(i,8) in the second, third, seventh, and eighth rows. Here, "i" is an integer from 1 to 5. The "row area" to be followed is likely to be any of the following blocks: block B(i,1) in the first row located to the left of the left wheel; blocks B(i,4), B(i,5), and B(i,6) in the fourth to sixth rows sandwiched between the left and right wheels; and block B(i,9) in the ninth row located to the right of the right wheel.

[0254] In such an example, if any of the blocks belonging to the hatched rows are classified as a "row region," there is a possibility that an obstacle will occur in row-following travel. Also, if the unhatched rows do not include any blocks classified as a "row region," there is a possibility that an obstacle will occur in row-following travel because the target for following is lost. In such a case where there is a possibility that an obstacle will occur in row-following travel, the processing device 122 in this embodiment can generate a signal to control the travel of the agricultural machine 100 to stop row-following travel.

[0255] Figure 70 is a diagram showing another example of the arrangement of blocks within the search area. In this example, the blocks B(i,j) included in the search area 314 are not uniform in size. The block arrangement in the example of Figure 70 makes it possible to improve the detection resolution for crop rows located between the left and right wheels of the agricultural machine 100.

[0256] FIG. 71 is a diagram showing yet another example of the arrangement of blocks within the search area 314. In this example, the size of the blocks B(i, j) is reduced, thereby increasing the detection resolution of the crop rows across the entire range of the search area 314. Such a block arrangement makes it possible to estimate with higher accuracy the direction in which the crop rows extend relative to the agricultural machine 100 (see FIG. 66). When the two-dimensional block arrangement pattern shown in FIG. 66 is obtained, the processing device 122 may be configured to reduce the size of each block B(i, j) as shown in FIG. 71.

[0257] The shape and size of the search region 144 do not need to be fixed. For example, when starting line-following traveling, the processing device 122 may perform processing to make the search region 144 during line-following traveling smaller than the search region before line-following traveling. Specifically, when line-following traveling, a relatively narrow range that includes the line region to be followed may be selected as the search region.

[0258] (Embodiment of agricultural machinery) Next, we will explain embodiments of agricultural machines equipped with the image recognition systems of embodiments 1 to 4. Note that the procedure or algorithm for row detection by the image recognition system is not limited to the procedure or algorithm described in embodiments 1 to 4 above.

[0259] The agricultural machine in this embodiment is equipped with the image recognition system described above. The agricultural machine also includes a control system that performs control to realize automatic steering operation. The control system is a computer system that includes a storage device and a control device, and is configured to control the steering, traveling, and other operations of the agricultural machine.

[0260] The control device may be configured to, in a normal automatic steering operation mode, identify the position of the agricultural machine using the positioning device, and, based on a target route generated in advance, control the steering of the agricultural machine so that the agricultural machine travels along a target route. Specifically, the control device may control the steering angle of the steering wheels (e.g., front wheels) of the agricultural machine so that the work vehicle travels along the target route in the field. The agricultural machine in this embodiment is equipped with an automatic steering device configured to travel not only in this normal automatic steering mode, but also using "row-following travel control" in a field where rows of crops and ridges are provided.

[0261] The positioning device includes, for example, a GNSS receiver. Such a positioning device can identify the position of a work vehicle based on signals from GNSS satellites. However, when rows are present in a field, even if the positioning device can accurately measure the position of the agricultural machine, the rows are narrow, and depending on the way the crops are planted and their growing conditions, there is a high possibility that the agricultural machine's running gear, such as wheels, may extend beyond the rows. However, in this embodiment, by using the image recognition system described above, it is possible to detect rows that actually exist and perform appropriate automatic steering. That is, the automatic steering device (control device) provided in the agricultural machine in the embodiment of the present disclosure is configured to control the steering angle of the steering wheels based on the position of the row edge line determined by the image recognition system.

[0262] Furthermore, in the agricultural machine of this embodiment, the processing device of the image recognition system can monitor the positional relationship between the row edge line and the steering wheel based on the time-series color images. If a position error signal is generated from this positional relationship, the automatic steering device of the agricultural machine can appropriately adjust the steering angle to reduce the position error signal.

[0263] Fig. 72 is a perspective view showing an example of the appearance of the agricultural machine 100 according to this embodiment. Fig. 73 is a side view schematically showing an example of the agricultural machine 100 with the work implement 300 attached. The agricultural machine 100 according to this embodiment is an agricultural tractor (work vehicle) with the work implement 300 attached. The agricultural machine 100 is not limited to a tractor, and does not necessarily have to be equipped with the work implement 300. The row detection technology according to the present disclosure can be used to great effect in small cultivators and vegetable transplanters that can be used for inter-row work such as ridge creation, inter-cultivation, hilling, weeding, top dressing, and pest control, for example.

[0264] The agricultural machine 100 in this embodiment includes an imaging device 120, a positioning device 130, and an obstacle sensor 136. Although one obstacle sensor 136 is illustrated in FIG. 72, the obstacle sensor 136 may be provided at multiple locations on the agricultural machine 100.

[0265] As shown in FIG. 73, the agricultural machine 100 includes a vehicle body 110, a prime mover (engine) 102, and a transmission 103. The vehicle body 110 is provided with wheels 104 and a cabin 105. The wheels 104 include a pair of front wheels 104F and a pair of rear wheels 104R. A driver's seat 107, a steering device 106, an operation terminal 200, and a group of switches for operation are provided inside the cabin 105. The group of switches may include a start switch 112 and a mode switch 114 shown in FIG. 1. One or both of the front wheels 104F and the rear wheels 104R may be replaced with wheels (crawlers) equipped with tracks instead of tires. The agricultural machine 100 may be a four-wheel drive vehicle equipped with four wheels 104 as drive wheels, or may be a two-wheel drive vehicle equipped with a pair of front wheels 104F or a pair of rear wheels 104R as drive wheels.

[0266] The positioning device 130 in this embodiment includes a GNSS receiver. The GNSS receiver includes an antenna that receives signals from GNSS satellites and a processing circuit that determines the position of the agricultural machine 100 based on the signals received by the antenna. The positioning device 130 receives GNSS signals transmitted from GNSS satellites and performs positioning based on the GNSS signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, for example, Michibiki), GLONASS, Galileo, and BeiDou. The positioning device 130 in this embodiment is provided on top of the cabin 105, but may be provided in another position.

[0267] The positioning device 130 can further supplement the position data by using signals from an inertial measurement unit (IMU). The IMU can measure the tilt and minute movements of the agricultural machine 100. By supplementing the position data based on the GNSS signals with data acquired by the IMU, it is possible to improve the positioning performance.

[0268] In the examples shown in FIGS. 72 and 73 , an obstacle sensor 136 is provided at the rear of the vehicle body 110. The obstacle sensor 136 may also be provided at a location other than the rear of the vehicle body 110. For example, one or more obstacle sensors 136 may be provided at any of the side and front of the vehicle body 110, and the cabin 105. The obstacle sensor 136 detects objects present around the agricultural machine 100. The obstacle sensor 136 may include, for example, a laser scanner and / or an ultrasonic sonar. When an obstacle is present within a predetermined detection area (search area) of the obstacle sensor 136, the obstacle sensor 136 outputs a signal indicating the presence of an obstacle. Multiple obstacle sensors 136 may be provided at different positions on the body of the agricultural machine 100. For example, multiple laser scanners and multiple ultrasonic sonars may be provided at different positions on the body. By providing such a large number of obstacle sensors 136, it is possible to reduce blind spots in monitoring obstacles around the agricultural machine 100.

[0269] The prime mover 102 is, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 103 can change the propulsive force and travel speed of the agricultural machine 100 by changing speeds. The transmission 103 can also switch the agricultural machine 100 between forward and reverse travel.

[0270] The steering device 106 includes a steering wheel, a steering shaft connected to the steering wheel, and a power steering device that assists steering by the steering wheel. The front wheels 104F are steerable wheels, and the traveling direction of the agricultural machine 100 can be changed by changing the turning angle (also referred to as the "steering angle"). When manual steering is performed, the steering angle of the front wheels 104F can be changed by the operator operating the steering wheel. The power steering device includes a hydraulic device or an electric motor that supplies an assisting force for changing the steering angle of the front wheels 104F. When automatic steering is performed, the steering angle is automatically adjusted by the force of the hydraulic device or the electric motor (steering motor) under control of a control device arranged in the agricultural machine 100.

[0271] A coupling device 108 is provided at the rear of the vehicle body 110. The coupling device 108 includes, for example, a three-point support device (also referred to as a "three-point link" or "three-point hitch"), a PTO (Power Take Off) shaft, a universal joint, and a communication cable. The coupling device 108 can attach and detach the work implement 300 to the agricultural machine 100. The coupling device 108 can raise and lower the three-point link using, for example, a hydraulic device, to control the position or attitude of the work implement 300. In addition, power can be transmitted from the agricultural machine 100 to the work implement 300 via the universal joint. The agricultural machine 100 can cause the work implement 300 to perform a predetermined task while towing the work implement 300. The coupling device may be provided at the front of the vehicle body 110. In this case, the work implement can be connected to the front of the agricultural machine 100.

[0272] The implement 300 shown in Figure 73 is, for example, a rotary cultivator. The implement 300, which is towed by or attached to a work vehicle such as a tractor when traveling along the rows, can be any implement that can be used for inter-row work such as ridge making, inter-cultivation, soiling, weeding, fertilizing, and pest control.

[0273] 74 is a block diagram showing an example of a schematic configuration of the agricultural machine 100 and the work implement 300. The agricultural machine 100 and the work implement 300 can communicate with each other via a communication cable included in the coupling device 108.

[0274] In the example of FIG. 74 , the agricultural machine 100 includes, in addition to the imaging device 120, the positioning device 130, the obstacle sensor 136, and the operation terminal 200, a drive unit 140, a steering wheel sensor 150, a turning angle sensor 152, a control system 160, a communication interface (IF) 190, an operation switch group 210, and a buzzer 220. The positioning device 130 includes a GNSS receiver 131 and an inertial measurement unit (IMU) 125. The control system 160 includes a storage device 170 and a control device 180. The control device 180 includes multiple electronic control units (ECUs) 181 to 186. The work implement 300 includes a drive unit 340, a control device 380, and a communication interface (IF) 390. Note that FIG. 74 shows components that are relatively closely related to the automatic steering or automatic driving operation of the agricultural machine 100, and does not show other components.

[0275] The positioning device 130 uses GNSS to locate the position of the agricultural machine 100. If the positioning device 130 is equipped with an RTK receiver, correction signals transmitted from a reference station are used in addition to GNSS signals transmitted from multiple GNSS satellites. The reference station may be installed around the field in which the agricultural machine 100 travels (for example, within 10 km of the agricultural machine 100). The reference station generates correction signals based on the GNSS signals received from the multiple GNSS satellites and transmits them to the positioning device 130. A GNSS receiver 131 in the positioning device 130 receives GNSS signals transmitted from the multiple GNSS satellites. The positioning device 130 performs positioning by calculating the position of the agricultural machine 100 based on the GNSS signals and the correction signals. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position information including latitude, longitude, and altitude information is acquired through high-precision positioning using RTK-GNSS. The positioning method is not limited to RTK-GNSS, and any positioning method (such as interferometric positioning or relative positioning) that can obtain position information with the required accuracy can be used. For example, positioning using a Virtual Reference Station (VRS) or a Differential Global Positioning System (DGPS) may be performed.

[0276] The IMU 135 includes a three-axis acceleration sensor and a three-axis gyroscope. The IMU 135 may also include a direction sensor such as a three-axis geomagnetic sensor. The IMU 135 functions as a motion sensor and can output signals indicating various quantities such as the acceleration, velocity, displacement, and attitude of the agricultural machine 100. The positioning device 130 can estimate the position and orientation of the agricultural machine 100 with higher accuracy based on the signal output from the IMU 135 in addition to the GNSS signal and correction signal. The signal output from the IMU 135 can be used to correct or complement the position calculated based on the GNSS signal and correction signal. The IMU 135 outputs a signal at a higher frequency than the GNSS signal. The high-frequency signal can be used to measure the position and orientation of the agricultural machine 100 at a higher frequency (for example, 10 Hz or higher). Instead of the IMU 135, a three-axis acceleration sensor and a three-axis gyroscope may be provided separately. The IMU 135 may also be provided as a device separate from the positioning device 130.

[0277] The positioning device 130 may include other types of sensors in addition to the GNSS receiver 131 and the IMU 135. Depending on the environment in which the agricultural machine 100 travels, the position and orientation of the agricultural machine 100 can be estimated with high accuracy based on data from these sensors.

[0278] By using such a positioning device 130, it is also possible to create a map of the crop rows and furrows detected by the image recognition system 1000 described above.

[0279] The drive device 140 includes various devices necessary for the travel of the agricultural machine 100 and the drive of the work implement 300, such as the aforementioned prime mover 102, transmission 103, differential device including a differential lock mechanism, steering device 106, and coupling device 108. The prime mover 102 includes an internal combustion engine such as a diesel engine. The drive device 140 may include an electric motor for traction instead of or in addition to the internal combustion engine.

[0280] The steering wheel sensor 150 measures the rotation angle of the steering wheel of the agricultural machine 100. The turning angle sensor 152 measures the turning angle of the front wheels 104F, which are steered wheels. The measurement values ​​of the steering wheel sensor 150 and the turning angle sensor 152 are used for steering control by the control device 180.

[0281] The storage device 170 includes one or more storage media, such as a flash memory or a magnetic disk. The storage device 170 stores various data generated by each sensor and the control device 180. The data stored in the storage device 170 may include map data of the environment in which the agricultural machine 100 travels, and target route data for automatic steering. The storage device 170 also stores computer programs that cause each ECU in the control device 180 to execute various operations, which will be described later. Such computer programs may be provided to the agricultural machine 100 via a storage medium (e.g., a semiconductor memory or an optical disk) or an electric communication line (e.g., the Internet). Such computer programs may be sold as commercial software.

[0282] The control device 180 includes multiple ECUs. The multiple ECUs include an ECU 181 for image recognition, an ECU 182 for speed control, an ECU 183 for steering control, an ECU 184 for automatic steering control, an ECU 185 for work implement control, an ECU 186 for display control, and an ECU 187 for buzzer control. The ECU 181 for image recognition functions as a processing device of the image recognition system. The ECU 182 controls the speed of the agricultural machine 100 by controlling the prime mover 102, the transmission 103, and the brakes included in the drive unit 140. The ECU 183 controls the steering of the agricultural machine 100 by controlling the hydraulic device or the electric motor included in the steering device 106 based on measurement values ​​of the steering wheel sensor 150. The ECU 184 performs calculations and control to achieve automatic steering operation based on signals output from the positioning device 130, the steering wheel sensor 150, and the turning angle sensor 152. During automatic steering operation, the ECU 184 sends a command to change the steering angle to the ECU 183. The ECU 183 changes the steering angle by controlling the steering device 106 in response to the command. The ECU 185 controls the operation of the coupling device 108 to cause the work implement 300 to perform a desired operation. The ECU 185 also generates a signal to control the operation of the work implement 300 and transmits the signal from the communication IF 190 to the work implement 300. The ECU 186 controls the display of the operation terminal 200. For example, the ECU 186 realizes various displays on the display device of the operation terminal 200, such as a map of the farm field, detected crop rows or furrows, the position and target route of the agricultural machine 100 on the map, pop-up notifications, and setting screens. The ECU 187 controls the output of a warning sound by the buzzer 220.

[0283] Through the operation of these ECUs, the control device 180 realizes operation by manual steering or automatic steering. During normal automatic steering operation, the control device 180 controls the drive device 140 based on the position of the agricultural machine 100 measured or estimated by the positioning device 130 and the target route stored in the storage device 170. In this way, the control device 180 causes the agricultural machine 100 to travel along the target route. On the other hand, in a row following travel control mode in which the agricultural machine 100 travels along the rows, the image recognition ECU 181 determines the edge lines of the crop rows or ridges from the detected crop rows or ridges, and generates a target route based on these edge lines. The control device 180 executes operations according to this target route.

[0284] The multiple ECUs included in the control device 180 can communicate with each other in accordance with a vehicle bus standard such as CAN (Controller Area Network). In FIG. 74, each of the ECUs 181 to 187 is shown as an individual block, but the functions of each of these may be realized by multiple ECUs. Also, an on-board computer that integrates at least some of the functions of the ECUs 181 to 187 may be provided. The control device 180 may include ECUs other than the ECUs 181 to 187, and any number of ECUs may be provided depending on the functions. Each ECU includes a control circuit including one or more processors.

[0285] The communication IF 190 is a circuit that communicates with the communication IF 390 of the work implement 300. The communication IF 190 transmits and receives signals compliant with ISOBUS standards such as ISOBUS-TIM to and from the communication IF 390 of the work implement 300. This makes it possible to cause the work implement 300 to perform desired operations and to obtain information from the work implement 300. The communication IF 190 may communicate with an external computer via a wired or wireless network. The external computer may be, for example, a server computer in a farming support system that centrally manages information about farm fields on the cloud and supports agriculture by utilizing data on the cloud.

[0286] The operation terminal 200 is a terminal through which the operator executes operations related to the traveling of the agricultural machine 100 and the operation of the work implement 300, and is also referred to as a virtual terminal (VT). The operation terminal 200 may include a display device such as a touch screen and / or one or more buttons. By operating the operation terminal 200, the operator can execute various operations, such as switching the automatic steering mode on / off, switching the cruise control on / off, setting the initial position of the agricultural machine 100, setting a target route, recording or editing a map, switching between 2WD and 4WD, switching the differential lock on / off, and switching the work implement 300 on / off. At least some of these operations can also be realized by operating the operation switch group 210. The display on the operation terminal 200 is controlled by the ECU 186.

[0287] The buzzer 220 is an audio output device that emits a warning sound to alert the operator of an abnormality. For example, the buzzer 220 emits a warning sound when the image recognition system 1000 detects a missing portion in the row area or the end of the field while operating in the row-following traveling mode. The buzzer 220 may also emit a warning sound when the agricultural machine 100 deviates from the target path by a predetermined distance or more. The buzzer 220 may also emit a warning sound when the operator operates the start switch 112 to issue a command to start row-following traveling when row-following traveling is not possible. Instead of the buzzer 220, a similar function may be achieved by a speaker of the operation terminal 200. The buzzer 220 is controlled by the ECU 186.

[0288] The drive device 340 in the work machine 300 performs operations necessary for the work machine 300 to perform a predetermined task. The drive device 340 includes devices appropriate for the intended use of the work machine 300, such as a hydraulic device, an electric motor, or a pump. The control device 380 controls the operation of the drive device 340. The control device 380 causes the drive device 340 to perform various operations in response to signals transmitted from the agricultural machine 100 via the communication IF 390. The control device 380 can also transmit signals appropriate to the state of the work machine 300 from the communication IF 390 to the agricultural machine 100.

[0289] In the above embodiments, the agricultural machine 100 may be an unmanned work vehicle that performs automatic driving. In that case, components that are only required for manned driving, such as a cabin, a driver's seat, a steering wheel, and an operation terminal, may not be provided in the agricultural machine 100. The unmanned work vehicle may perform operations similar to those in the above embodiments by autonomous driving or by remote control by an operator.

[0290] A system providing various functions in the embodiments can also be retrofitted to an agricultural machine that does not have those functions. Such a system can be manufactured and sold independently of the agricultural machine. A computer program used in such a system can also be manufactured and sold independently of the agricultural machine. The computer program can be provided, for example, by being stored on a computer-readable non-transitory storage medium. The computer program can also be provided by downloading via a telecommunications line (for example, the Internet). [Industrial Applicability]

[0291] The technology of the present disclosure can be applied to agricultural machinery such as riding tillers, vegetable transplanters, and tractors, for example. [Explanation of symbols]

[0292] 10···ground, 12···crop row, 14···intermediate area (work passage), 16···furrow, 40···image, 42···enhanced image, 44···top view image, 100···agricultural machinery, 110···vehicle body, 120···imaging device, 122···processing device, 1000···image recognition system

Claims

1. an image recognition system that detects row regions of at least one of crops and ridges provided on the ground of the field from the acquired image; a running device including a steering wheel; a control device for controlling the traveling device, the control device being operable in a line-following traveling mode in which the traveling device is controlled so as to travel along the line area detected by the image recognition system; Equipped with the control device, while operating in the row following traveling mode, continues traveling for a predetermined time or a predetermined distance when the image recognition system detects a missing portion in the row region or an end of a field, the control device executes the line-following travel mode when the image recognition system detects the line area and detects a work passage area having a predetermined width or more on either or both sides of the line area; and The control device determines whether or not the steering wheel can pass through the work passage area based on the work passage area and the position of the steering wheel, and when it determines that the steering wheel cannot pass through the work passage area, stops the following travel mode.

2. The agricultural machine according to claim 1 , wherein the control device controls the traveling device so that, when continuing the traveling, the agricultural machine travels along an extension of a route that the agricultural machine has traveled.

3. The agricultural machine according to claim 1 or 2, wherein the control device continues the traveling and then stops the traveling.

4. The agricultural machine according to claim 3 , further comprising a notification unit that notifies a user of a warning to stop traveling when the image recognition system detects an end of the field.

5. 3. The agricultural machine according to claim 1, wherein, when the image recognition system detects a new row area while the agricultural machine continues traveling, the control device controls the traveling device to travel along the new row area.

6. The agricultural machine according to claim 1 or 2, wherein the control device determines the predetermined time or the predetermined distance depending on a work implement towed by the agricultural machine.

Images