Route generation system and route generation method for automatic driving of agricultural machinery

Abstract

A route generation system, which is for automated travel of agricultural machine, is provided with a processing device that generates an automated travel route for the agricultural machine. The processing device acquires data indicating a travel trajectory from a vehicle that travels manually while recording the travel trajectory on a route on which the agricultural machine is scheduled to automatedly travel, removes from the travel trajectory a trajectory related to an avoidance action performed to avoid an oncoming vehicle, and generates an automated travel route for the agricultural machine on the basis of the travel trajectory from which the trajectory related to the avoidance action has been removed.

Description

[Technical Field]

[0001] The present disclosure relates to a route generation system and a route generation method for automatic travel of agricultural machinery. [Background technology]

[0002] Research and development is underway to automate agricultural machinery used in fields. For example, work vehicles such as tractors, combine harvesters, and rice transplanters that can navigate autonomously within fields using positioning systems such as the Global Navigation Satellite System (GNSS) have been put to practical use. Research and development is also underway on work vehicles that can navigate autonomously not only within fields but also outside of them.

[0003] Patent Documents 1 and 2 disclose examples of a system in which an unmanned work vehicle automatically travels between two farm fields separated by a road. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-073602 [Patent Document 2] Patent Publication No. 2021-029218 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to realize an autonomously driven agricultural machine, it is necessary to generate an autonomous driving route in advance. The present disclosure provides a technique for appropriately generating an autonomous driving route for an agricultural machine. [Means for solving the problem]

[0006] A route generation system according to an exemplary embodiment of the present disclosure is a route generation system for automated driving of an agricultural machine, and includes a processing device that generates an automated driving route for the agricultural machine. The processing device acquires data indicating a driving trajectory from a vehicle that manually drives a route along which the agricultural machine is to be automated while recording the driving trajectory, removes from the driving trajectory a trajectory associated with an avoidance operation performed to avoid an oncoming vehicle, and generates an automated driving route for the agricultural machine based on the driving trajectory from which the trajectory associated with the avoidance operation has been removed.

[0007] A route generation method according to another embodiment of the present disclosure is a route generation method for automatic driving of an agricultural machine, and includes acquiring data indicating a driving trajectory from a vehicle that manually drives a route along which the agricultural machine is planned to drive automatically while recording the driving trajectory, removing from the driving trajectory a trajectory related to an evasive action taken to avoid the oncoming vehicle, and generating an automatic driving route for the agricultural machine based on the driving trajectory from which the trajectory related to the evasive action has been removed.

[0008] A general or specific aspect of the present disclosure may be realized by an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable non-transitory storage medium, or any combination thereof. The computer-readable storage medium may include a volatile storage medium or a non-volatile storage medium. An apparatus may be composed of multiple devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. [Effects of the Invention]

[0009] According to an embodiment of the present disclosure, an automated driving route for an agricultural machine can be appropriately generated. [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram illustrating an example of a route generation system. [Figure 2] FIG. 2 is a block diagram showing an example of a more detailed configuration of the route generation system. [Figure 3] FIG. 1 is a diagram schematically illustrating a state in which a vehicle travels on a road outside a field while collecting data. [Figure 4] 10 is a flowchart illustrating an example of an operation for generating an automatic driving route. [Figure 5A] FIG. 10 is a diagram illustrating an example of an operation of a vehicle to avoid an oncoming vehicle. [Figure 5B] FIG. 10 is a diagram showing an example of a travel trajectory from which a trajectory related to an avoidance operation has been removed; [Figure 5C] FIG. 10 is a diagram illustrating a process of complementing a portion removed from a travel trajectory with a linear complementary path. [Figure 6] 10A and 10B are diagrams illustrating another example of the operation of a vehicle to avoid an oncoming vehicle. [Figure 7A] FIG. 10 is a diagram illustrating a display example of a display device. [Figure 7B] FIG. 10 is a diagram showing an example of a display screen when a user touches one of the portions enclosed by dotted lines. [Figure 7C] FIG. 10 is a diagram showing an example of a display screen in which one of the removed portions has been complemented. [Figure 7D] FIG. 10 is a diagram showing an example of a display screen in which all removed portions have been complemented and the automated driving route has been completed. [Figure 8] FIG. 1 is a diagram for explaining an overview of an agricultural management system according to an exemplary embodiment of the present disclosure. [Figure 9] 1 is a side view schematically showing an example of a work vehicle and an implement coupled to the work vehicle. FIG. [Figure 10] FIG. 2 is a block diagram showing an example of the configuration of a work vehicle and an implement. [Figure 11] FIG. 1 is a conceptual diagram showing an example of a work vehicle that performs positioning using RTK-GNSS. [Figure 12] 3A and 3B are diagrams illustrating an example of an operation terminal and an operation switch group provided inside a cabin. [Figure 13]FIG. 2 is a block diagram illustrating a schematic hardware configuration of a management device and a terminal device. [Figure 14] FIG. 1 is a diagram schematically illustrating an example of a work vehicle that automatically travels along a target route in a farm field. [Figure 15] 4 is a flowchart showing an example of the operation of steering control during automatic driving executed by the control device. [Figure 16A] FIG. 1 is a diagram illustrating an example of a work vehicle traveling along a target route. [Figure 16B] FIG. 10 is a diagram illustrating an example of a work vehicle at a position shifted to the right from the target route. [Figure 16C] FIG. 10 is a diagram illustrating an example of a work vehicle that is shifted to the left from the target route. [Figure 16D] FIG. 10 is a diagram illustrating an example of a work vehicle facing in an inclined direction relative to a target route. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Definition of terms) In this disclosure, "agricultural machinery" refers to machinery used for agricultural purposes. Examples of agricultural machinery include tractors, harvesters, rice transplanters, riding cultivators, vegetable transplanters, mowers, seed sowing machines, fertilizer applicators, and agricultural mobile robots. Not only can a work vehicle such as a tractor function alone as an "agricultural machinery," but the entire work vehicle and an implement attached to or towed by the work vehicle can also function as a single "agricultural machinery." Agricultural machinery performs agricultural work on the ground in a field, such as plowing, sowing, pest control, fertilizing, planting crops, or harvesting. These agricultural works are sometimes referred to as "ground work" or simply "work." Traveling while performing agricultural work by a vehicle-type agricultural machine is sometimes referred to as "work travel."

[0012] "Autonomous driving" refers to controlling the movement of an agricultural machine through the action of a control device, without manual operation by a driver. Agricultural machines that perform autonomous driving are sometimes called "autonomous agricultural machines" or "robotic agricultural machines." During autonomous driving, not only the movement of the agricultural machine but also the agricultural work operations (e.g., the operation of the implements) may be automatically controlled. When the agricultural machine is a vehicle-type machine, the movement of the agricultural machine through autonomous driving is referred to as "autonomous driving." The control device may control at least one of the steering, speed adjustment, and start and stop of movement required for the movement of the agricultural machine. When controlling a work vehicle equipped with implements, the control device may control operations such as raising and lowering the implements and starting and stopping their operation. Autonomous driving movement includes not only movement of the agricultural machine toward a destination along a predetermined route, but also movement of the agricultural machine following a tracking target. An autonomously driving agricultural machine may move partially based on user instructions. Furthermore, an autonomously driving agricultural machine may operate in a manual driving mode, in which it moves through manual operation by the driver, in addition to an autonomous driving mode. Steering an agricultural machine by the action of a control device, without manual operation, is called "automatic steering." Part or all of the control device may be external to the agricultural machine. Control signals, commands, data, and the like may be communicated between the agricultural machine and a control device external to the agricultural machine. An agricultural machine that performs automatic driving may move autonomously while sensing the surrounding environment, without a human being being involved in controlling the movement of the agricultural machine. An agricultural machine capable of autonomous movement can travel unmanned within a field or outside a field (e.g., on a road). During autonomous movement, the machine may detect obstacles and take action to avoid them.

[0013] An "environmental map" is data that represents the positions or areas of objects in the environment in which the agricultural machine moves using a specified coordinate system. An environmental map may be simply referred to as a "map" or "map data." The coordinate system that defines the environmental map may be, for example, a world coordinate system such as a geographic coordinate system fixed relative to the Earth. An environmental map may also include information other than the positions of objects in the environment (e.g., attribute information and other information). Environmental maps include maps in various formats, such as point cloud maps or grid maps. Data for local or partial maps that are generated or processed in the process of constructing an environmental map are also referred to as a "map" or "map data."

[0014] An "automated driving route" refers to data on a route connecting a starting point to a destination point when an agricultural machine is driving automatically. An automated driving route is also called a "global route" or a "target route." An automated driving route can be defined, for example, by the coordinate values ​​of multiple points on a map through which the agricultural machine must pass. A point through which the agricultural machine must pass is called a "waypoint," and a line segment connecting adjacent waypoints is called a "link." Waypoint data may include position and speed information. In this specification, generating data indicating an automated driving route (for example, data on multiple waypoints) is expressed as "generating an automated driving route."

[0015] (Embodiment) Hereinafter, embodiments of the present disclosure will be described. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims. In the following description, components having the same or similar functions are designated by the same reference numerals.

[0016] The following embodiments are examples, and the technology of the present disclosure is not limited to the following embodiments. For example, the numerical values, shapes, materials, steps, step order, display screen layout, etc. shown in the following embodiments are merely examples, and various modifications are possible as long as no technical contradiction occurs. Furthermore, one aspect can be combined with another aspect as long as no technical contradiction occurs.

[0017] FIG. 1 is a block diagram showing an example of a route generation system for autonomous driving of agricultural machinery. The route generation system 10 shown in FIG. 1 is used in combination with a vehicle 20 that collects data necessary for generating an autonomous driving route and an agricultural machine 30 capable of autonomous driving. The route generation system 10 is a computer system including a processing device 15. The processing device 15 generates an autonomous driving route for the agricultural machine 30 based on data collected by the vehicle 20. The vehicle 20 is a vehicle that collects data necessary for generating the autonomous driving route for the agricultural machine 30. The vehicle 20 may be, for example, a passenger car, a truck (lorry), a van, or an agricultural work vehicle. The agricultural machine 30 is an autonomous agricultural machine that automatically drives according to the autonomous driving route generated by the processing device 15. The agricultural machine 30 is, for example, an agricultural work vehicle such as a tractor. The agricultural machine 30 can automatically drive not only within a field but also on roads outside the field (for example, farm roads or public roads).

[0018] In the example shown in FIG. 1 , the vehicle 20 is a vehicle different from the agricultural machine 30, but the agricultural machine 30 may also have the functions of the vehicle 20. In other words, a single agricultural work vehicle capable of both automatic and manual driving may be used as the agricultural machine 30 and the vehicle 20. The route generation system 10 may be a system independent of the vehicle 20 and the agricultural machine 30 (for example, a cloud computing system), or may be mounted on the vehicle 20 or the agricultural machine 30. Here, an example will be described in which the vehicle 20 and the agricultural machine 30 are different vehicles and the route generation system 10 is a system independent of the vehicle 20 and the agricultural machine 30.

[0019] Fig. 2 is a block diagram showing an example of a more detailed configuration of the system shown in Fig. 1. In the example shown in Fig. 2, a route generation system 10 includes a processing device 15, an input interface (I / F) 11, an output interface 12, and a storage device 13. A vehicle 20 includes a positioning device 21, a camera 22, and a storage device 23. An agricultural machine 30 includes a self-position estimation device 31, a travel control device 32, and a storage device 33. Fig. 2 also shows, by example, a display device 45 that displays the automatic driving route generated by the processing device 15, and an input device 40 that a user uses to edit the automatic driving route.

[0020] FIG. 3 is a diagram schematically illustrating a state in which the vehicle 20 travels on a road 75 (e.g., a farm road) outside a field 70 while collecting data. FIG. 3 illustrates a plurality of fields 70, the surrounding roads 75, and a storage shed 78 for the agricultural machine 30. Before starting operation of the agricultural machine 30, a user drives the vehicle 20 along a route along which the agricultural machine 30 is scheduled to later travel automatically. The vehicle 20 travels while recording its own travel trajectory. For example, while traveling, the vehicle 20 records position data sequentially output from a positioning device 21 such as a GNSS receiver in the storage device 23 as data indicating the travel trajectory. The position data may include, for example, information on latitude and longitude in a geographic coordinate system. The data indicating the travel trajectory may include position data of the vehicle 20 and information on the corresponding time. In other words, the data indicating the travel trajectory may indicate a change in the position of the vehicle 20 over time. The data indicating the travel trajectory may include, in addition to information on the position of the vehicle 20 at each time, information on the travel speed of the vehicle 20 at each time. The information on the position and travel speed of the vehicle 20 may be recorded at relatively short intervals (for example, every few milliseconds to every few seconds).

[0021] In FIG. 3 , an example of the travel trajectory of the vehicle 20 is indicated by dashed arrows. In the example of FIG. 3 , the vehicle 20 travels from a storage shed 78 along roads 75 around multiple fields 70 where agricultural work by the agricultural machine 30 is scheduled, and then returns to the storage shed 78. The route that the vehicle 20 travels to collect data is determined according to the route along which the agricultural machine 30 is scheduled to travel. The vehicle 20 travels manually along the route along which the agricultural machine 30 is scheduled to travel automatically, while recording the travel trajectory. In this specification, the term "manually traveling" refers to the vehicle 20 traveling manually by a driver. The vehicle 20 may travel while capturing images of the surroundings of the vehicle 20 with the camera 22. In this case, the vehicle 20 travels while recording video captured by the camera 22 in the storage device 23.

[0022] After data collection by the vehicle 20 is completed, data indicating the travel trajectory is sent to the processing device 15. The data indicating the travel trajectory may be transmitted via a wired or wireless communication line, or may be provided to the processing device 15 via any recording medium. In either form, the processing device 15 directly or indirectly acquires the data indicating the travel trajectory from the vehicle 20. The processing device 15 generates an automated travel route for the agricultural machine 30 based on the acquired data indicating the travel trajectory. For example, the processing device 15 can approximate the travel trajectory of the vehicle 20 as a combination of multiple line segments on a map prepared in advance, and generate the combination of these line segments as an automated travel route.

[0023] In the example shown in FIG. 3 , there are no obstacles on road 75 on which vehicle 20 travels, and vehicle 20 travels in a straight line except when turning right or left. In such a case, an appropriate automated driving route can be generated by approximating the travel trajectory of vehicle 20 with multiple line segments. However, while vehicle 20 is traveling, an oncoming vehicle may be present ahead. In this case, if road 75 is narrow, vehicle 20 will take action to avoid the oncoming vehicle. For example, vehicle 20 may take evasive action to avoid contact with the oncoming vehicle by slowing down and moving to the edge of road 75, backing up, or temporarily stopping. In such a case, the travel trajectory related to the evasive action is also recorded. Therefore, if an automated driving route is simply generated based on data indicating the travel trajectory, an inappropriate automated driving route that reflects the evasive action will be generated.

[0024] In order to solve the above problem, the processing device 15 in this embodiment generates an automated driving route after performing a process to remove trajectories related to avoidance operations from the driving trajectory of the vehicle 20. An example of this process will be described below with reference to FIG. 4.

[0025] FIG. 4 is a flowchart showing an example of an operation of generating an automated driving route by the processing device 15. First, the processing device 15 acquires driving trajectory data recorded by the vehicle 20 (step S11). Next, the processing device 15 removes a trajectory related to an avoidance operation performed to avoid an oncoming vehicle from the driving trajectory indicated by the driving trajectory data (step S12). An example of a method for identifying a trajectory related to an avoidance operation from a driving trajectory will be described later. The processing device 15 generates an automated driving route for the agricultural machine 30 based on the driving trajectory from which the trajectory related to the avoidance operation has been removed (S13). For example, the automated driving route can be generated by performing interpolation processing such as approximating the removed portion with a line segment. Thereafter, the processing device 15 transmits data indicating the automated driving route to the agricultural machine 30 (step S14). Note that if the processing device 15 is mounted on the agricultural machine 30, the operation of step S14 may be omitted.

[0026] Here, a specific example of the operations in steps S12 and S13 will be described with reference to FIGS. 5A to 5C.

[0027] 5A shows an example of an operation in which vehicle 20 avoids oncoming vehicle 90. In this example, the driver of vehicle 20 first moves vehicle 20 to the left edge of road 75 to avoid contact with oncoming vehicle 90 approaching from the front, and then operates the steering wheel so that vehicle 20 returns to the center of road 75 after passing oncoming vehicle 90. Therefore, the travel trajectory recorded by vehicle 20 is a connection of two linear paths 91 and 93 and a non-linear path 92 resulting from the avoidance operation between them, as shown by the dashed arrows in FIG. 5A.

[0028] The path associated with the avoidance operation (hereinafter, sometimes referred to as the "avoidance path") is not limited to the path 92 shown in FIG. 5A. For example, as shown in FIG. 6, the avoidance path may include a path 95 for moving backward and a path 96 for moving forward thereafter. In the example of FIG. 6, the width of the road 75 is narrow, and the vehicle 20 and the oncoming vehicle 90 cannot pass each other. In such a case, the vehicle 20 will first move backward to return to a place wide enough for the two vehicles to pass each other, stop temporarily to allow the oncoming vehicle 90 to pass, and then move forward to return to the original path.

[0029] When the processing device 15 acquires the data of the travel trajectory of the vehicle 20, it extracts and removes trajectories related to avoidance operations from the travel trajectory indicated by the data. Fig. 5B shows an example of a travel trajectory from which trajectories related to avoidance operations have been removed.

[0030] The processing device 15 may extract a trajectory related to an avoidance operation based on data of a moving image captured by the camera 22 while the vehicle 20 is traveling. In this case, in step S11 shown in Fig. 4, the processing device 15 acquires data of the moving image in addition to data of the traveling trajectory. The processing device 15 detects an avoidance operation based on the moving image, and determines and removes a trajectory related to the avoidance operation from the traveling trajectory.

[0031] The processing device 15 may perform image recognition processing based on the moving image, and may determine a trajectory related to an avoidance operation based on the result of recognizing an oncoming vehicle 90 approaching the vehicle 20 from the moving image. For example, the processing device 15 may remove, from the traveling trajectory, a trajectory corresponding to at least a part of the period from when the oncoming vehicle 90 is recognized in the moving image to when the oncoming vehicle 90 is no longer recognized, as a trajectory related to an avoidance operation. Alternatively, the processing device 15 may remove, from the traveling trajectory, a trajectory corresponding to a predetermined time period (e.g., 10 seconds, 20 seconds, or 30 seconds) including the period from when it is recognized in the moving image that the oncoming vehicle 90 has approached to within a predetermined distance (e.g., 5 m, 10 m, or 20 m) from the vehicle 20 to when the oncoming vehicle 90 is no longer recognized, as a trajectory related to an avoidance operation.

[0032] The processing device 15 may also detect an evasive maneuver based on a temporal change in the position of the vehicle 20 indicated by the travel trajectory. For example, the processing device 15 may detect at least one of reversing, changing direction, accelerating, and decelerating, performed by the vehicle 20 to avoid the oncoming vehicle 90, as an evasive maneuver. As an example, the processing device 15 may extract, from the travel trajectory, a portion that follows a non-linear trajectory despite being a trajectory on a linear portion of the road 75, as a trajectory related to the evasive maneuver. Records of steering and / or acceleration / deceleration operations of the vehicle 20 may be used to extract, from the travel trajectory, a portion in which a large change in direction is made at a position other than an intersection on the road 75, as a trajectory related to the evasive maneuver. The processing device 15 may also extract, from the travel trajectory, a portion in which the vehicle 20 slows down or stops, or reverses and then moves forward again while traveling along the road 75, as a trajectory related to the evasive maneuver. A machine learning algorithm such as deep learning may be used to detect the evasive maneuver. The processing device 15 may extract a trajectory related to an avoidance operation from the travel trajectory based on the travel trajectory data acquired from the vehicle 20 and a pre-trained learned model.

[0033] The processing device 15 generates an automated driving path by removing trajectories related to avoidance operations and then performing a process of complementing the removed portions. For example, as shown in FIG. 5C , the processing device 15 may generate an automated driving path by complementing the portions removed from the driving path with a linear complementary path 94. Such complementing process may be performed automatically by the processing device 15 or in response to an operation from a user. For example, the processing device 15 may display the driving path from which trajectories related to avoidance operations have been removed on the display device 45, and complement the portions removed from the driving path in response to an operation by the user using the input device 40 to determine a complementary path.

[0034] FIG. 7A is a diagram showing an example of a display on the display device 45. In this example, the display device 45 is a computer with a built-in display, such as a tablet computer or a smartphone. The illustrated display device 45 is equipped with a touch screen and also functions as the input device 40. The display device 45 displays an environmental map of the surroundings of the farm field 70. A route in which the avoidance path has been removed from the travel trajectory of the vehicle 20 is displayed on the map. In FIG. 7A, the portion corresponding to the removed avoidance path is surrounded by a dotted line. The user can perform an operation to complement the route, for example, by touching the portion enclosed by the dotted line.

[0035] FIG. 7B shows an example of a display screen when a user touches one of the portions enclosed by the dotted line. In this example, a pop-up message asking "Do you want to complete the route?" appears, and the user can select "Yes" or "No." If the user selects "Yes," the processing device 15 generates a complementary route that completes the removed portion. For example, the processing device 15 completes the removed portion with a linear complementary route. Alternatively, the user may be allowed to specify a complementary route.

[0036] Figure 7C shows an example of one of the removed sections being completed. The completed section is indicated by a dashed arrow. The user can complete other removed sections in the same way.

[0037] FIG. 7D shows an example of a state in which all removed portions have been complemented and the automated driving route has been completed. The automated driving route may be defined by, for example, a plurality of waypoints. Each waypoint may include, for example, position and speed information. In FIG. 7D, the waypoints are represented by dots, and the links between the waypoints are represented by arrows. In this example, waypoints are set at locations where the agricultural machine 30 can change direction (intersections, near the entrance and exit of a field, the entrance and exit of a storage shed, etc.). The method of setting waypoints is not limited to the example shown in the figure, and the length of the links between waypoints can be set as desired.

[0038] The above-described operation can prevent the avoidance operation performed to avoid the oncoming vehicle 90 from being reflected in the automated driving route. This makes it possible to generate a more appropriate automated driving route for the agricultural machine 30.

[0039] Data indicating the generated automated driving route is sent to the agricultural machine 30 and recorded in the storage device 33. The driving control device 32 of the agricultural machine 30 controls the driving speed and steering of the agricultural machine 30 so that the agricultural machine 30 drives along the automated driving route. For example, if the automated driving route is defined by multiple waypoints, each of which includes position and speed information, the driving control device 32 controls the driving speed and steering so that the agricultural machine 30 passes through each waypoint at a specified speed. The driving control device 32 can estimate the degree to which the agricultural machine 30 deviates from the automated driving route based on the position and orientation of the agricultural machine 30 estimated by the self-position estimation device 31. The self-position estimation device 31 is a device that estimates its own position using sensors such as GNSS, an inertial measurement unit (IMU), a light detection and ranging (LiDAR), and / or a camera (including an image sensor). The driving control device 32 can realize driving along the automated driving route by performing steering control to reduce deviations in the position and / or orientation of the agricultural machine 30 from the automated driving route.

[0040] In this embodiment, the processing device 15 performs the above processing when generating a route for the agricultural machine 30 to travel automatically outside a field. The processing device 15 may perform similar processing when generating a route for the agricultural machine 30 to travel automatically within a field. Even within a field, other agricultural work vehicles may be present as oncoming vehicles, so the route generation method according to this embodiment is effective.

[0041] Next, an embodiment in which the technology of the present disclosure is applied to a work vehicle such as a tractor, which is an example of agricultural machinery, will be described. The technology of the present disclosure is not limited to work vehicles such as tractors, but can also be applied to other types of agricultural machinery.

[0042] FIG. 8 is a diagram illustrating an overview of an agricultural management system according to an exemplary embodiment of the present disclosure. The system illustrated in FIG. 8 includes a work vehicle 100, a terminal device 400, and a management device 600. The work vehicle 100 is an agricultural machine capable of autonomous driving. The terminal device 400 is a computer used by a user to remotely monitor the work vehicle 100. The management device 600 is a computer managed by a business operator that operates the system. The work vehicle 100, the terminal device 400, and the management device 600 can communicate with each other via a network 80. While FIG. 8 illustrates one work vehicle 100, the system may include multiple work vehicles or other agricultural machines. In this embodiment, the work vehicle 100 combines the functions of both the vehicle 20 and the agricultural machine 30 shown in FIG. 1. The management device 600 has the functions of the processing device 15 shown in FIG. 1.

[0043] The work vehicle 100 in this embodiment is a tractor. The work vehicle 100 can be fitted with an implement at either the rear or the front, or both. The work vehicle 100 can travel within a field while performing agricultural work according to the type of implement. The work vehicle 100 may also travel within or outside a field without an implement attached.

[0044] The work vehicle 100 has an automatic driving function. That is, the work vehicle 100 can travel by the operation of a control device, without manual operation. The control device in this embodiment is provided inside the work vehicle 100, and can control both the speed and steering of the work vehicle 100. The work vehicle 100 can travel automatically not only within a field, but also outside the field (for example, on a road).

[0045] The work vehicle 100 is equipped with devices used for positioning or self-location estimation, such as a GNSS receiver and a LiDAR sensor. A control device of the work vehicle 100 causes the work vehicle 100 to travel automatically based on the position of the work vehicle 100 and information about a target route. In addition to controlling the travel of the work vehicle 100, the control device also controls the operation of the implement. This allows the work vehicle 100 to perform agricultural work using the implement while traveling automatically within a field. Furthermore, the work vehicle 100 can automatically travel along roads outside the field (e.g., farm roads or public roads) along a target route. When traveling automatically along roads outside the field, the work vehicle 100 travels while generating a local route along the target route that can avoid obstacles, based on data output from sensing devices such as a camera or LiDAR sensor. Within the field, the work vehicle 100 may travel while generating a local route as described above, or may travel along a target route without generating a local route and stop if an obstacle is detected.

[0046] The management device 600 is a computer that manages agricultural work performed by the work vehicle 100. The management device 600 may be, for example, a server computer that centrally manages information about farm fields on the cloud and supports agriculture by utilizing data on the cloud. The management device 600 has functions equivalent to those of the processing device 15 shown in FIG. 1. That is, the management device 600 generates an automatic driving route (i.e., a target route) for the work vehicle 100. The management device 600 acquires data indicating the driving trajectory of the work vehicle 100 when it is manually driven, and generates an automatic driving route for the work vehicle 100 based on the acquired data. More specifically, before the work vehicle 100 begins automatic driving, the work vehicle 100 manually drives the route along which it plans to drive automatically while recording the driving trajectory. The work vehicle 100 records its driving trajectory by sequentially recording its own position using a positioning device such as a GNSS unit. After the driving for data recording is completed, the management device 600 acquires data indicating the driving trajectory from the work vehicle 100. The driving trajectory may include a trajectory related to an avoidance maneuver performed to avoid an oncoming vehicle on the road. The management device 600 removes the trajectory related to the avoidance maneuver performed to avoid an oncoming vehicle from the driving trajectory indicated by the acquired data using the method described above, and generates an automated driving route for the work vehicle 100 based on the driving trajectory from which the trajectory has been removed. This processing makes it possible to generate an appropriate automated driving route without reflecting the trajectory associated with the avoidance maneuver.

[0047] The management device 600 may further create a work plan for the work vehicle 100 and issue instructions to the work vehicle 100 to start and end autonomous driving in accordance with the work plan. The management device 600 may also generate an environmental map based on data collected by the work vehicle 100 or other vehicles using a sensing device such as a LiDAR sensor.

[0048] The management device 600 transmits data such as the generated automated driving route, work plan, and environmental map to the work vehicle 100. The work vehicle 100 automatically travels and performs farm work based on this data.

[0049] It should be noted that the generation of the automated driving route is not limited to the management device 600, and may be performed by other devices. For example, the control device of the work vehicle 100 may generate the automated driving route. In that case, the control device of the work vehicle 100 functions as a processing device that generates the automated driving route.

[0050] The terminal device 400 is a computer used by a user located remotely from the work vehicle 100. The terminal device 400 shown in FIG. 8 is a laptop computer, but is not limited to this. The terminal device 400 may be a stationary computer such as a desktop personal computer (PC), or a mobile terminal such as a smartphone or tablet computer. The terminal device 400 may be used to remotely monitor or operate the work vehicle 100. For example, the terminal device 400 can display on a display image captured by one or more cameras (imaging devices) equipped on the work vehicle 100. The user can view the image to check the situation around the work vehicle 100 and send instructions to the work vehicle 100 to stop or start. The terminal device 400 may further include the functions of the input device 40 and display device 45 shown in FIG. 2. In other words, the terminal device 400 may be used to edit the automated driving route generated by the management device 600.

[0051] The configuration and operation of the system in this embodiment will be described in more detail below.

[0052] [1. Configuration] FIG. 9 is a side view that schematically shows an example of a work vehicle 100 and an implement 300 coupled to the work vehicle 100. The work vehicle 100 in this embodiment can operate in both a manual driving mode and an automatic driving mode. In the automatic driving mode, the work vehicle 100 can travel unmanned. The work vehicle 100 can be driven automatically both inside and outside a field.

[0053] As shown in FIG. 9, work vehicle 100 includes a vehicle body 101, a prime mover (engine) 102, and a transmission 103. Vehicle body 101 is provided with wheels 104 with tires and a cabin 105. Wheels 104 include a pair of front wheels 104F and a pair of rear wheels 104R. Inside cabin 105, a driver's seat 107, a steering device 106, an operation terminal 200, and a group of switches for operation are provided. One or both of front wheels 104F and rear wheels 104R may be replaced with a plurality of wheels (crawlers) equipped with tracks rather than with tires.

[0054] The work vehicle 100 is equipped with a plurality of sensing devices that sense the surroundings of the work vehicle 100. In the example of Figure 9, the sensing devices include a plurality of cameras 120, a LiDAR sensor 140, and a plurality of obstacle sensors 130.

[0055] Cameras 120 may be installed, for example, on the front, rear, left, and right sides of work vehicle 100. Cameras 120 capture images of the environment around work vehicle 100 and generate image data. Images acquired by camera 120 may be transmitted to terminal device 400 for remote monitoring. These images may be used to monitor work vehicle 100 during unmanned driving. Camera 120 may also be used to generate images for recognizing surrounding features or obstacles, white lines, signs, or markings when work vehicle 100 travels on roads outside of fields (farm roads or public roads). For example, camera 120 may also be used to detect oncoming vehicles when work vehicle 100 is traveling manually while recording its travel trajectory.

[0056] In the example of FIG. 9 , the LiDAR sensor 140 is disposed on the lower front part of the vehicle body 101. The LiDAR sensor 140 may be disposed in another position. While the work vehicle 100 is traveling mainly outside the field, the LiDAR sensor 140 repeatedly outputs sensor data indicating the distance and direction to each measurement point of an object present in the surrounding environment, or the two-dimensional or three-dimensional coordinate values ​​of each measurement point. The sensor data output from the LiDAR sensor 140 is processed by a control device of the work vehicle 100. The control device can estimate the self-position of the work vehicle 100 by matching the sensor data with an environmental map. The control device can further detect objects such as obstacles present around the work vehicle 100 based on the sensor data, and generate a local path that the work vehicle 100 should actually travel along the target path. The control device may be configured to generate or edit an environmental map using an algorithm such as SLAM (Simultaneous Localization and Mapping). Work vehicle 100 may be equipped with multiple LiDAR sensors positioned at different locations and with different orientations.

[0057] The multiple obstacle sensors 130 shown in FIG. 9 are provided at the front and rear of the cabin 105. The obstacle sensors 130 may also be located in other locations. For example, one or more obstacle sensors 130 may be provided at any position on the side, front, or rear of the vehicle body 101. The obstacle sensors 130 may include, for example, a laser scanner or ultrasonic sonar. The obstacle sensors 130 are used to detect surrounding obstacles during autonomous driving and to stop or detour the work vehicle 100. A LiDAR sensor 140 may be used as one of the obstacle sensors 130.

[0058] The work vehicle 100 further includes a GNSS unit 110. The GNSS unit 110 includes a GNSS receiver. The GNSS receiver may include an antenna that receives signals from GNSS satellites and a processor that calculates the position of the work vehicle 100 based on the signals received by the antenna. The GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and performs positioning based on the satellite signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, e.g., Michibiki), GLONASS, Galileo, and BeiDou. In this embodiment, the GNSS unit 110 is provided on top of the cabin 105, but may be provided in another location.

[0059] The GNSS unit 110 may include an inertial measurement unit (IMU). Signals from the IMU can be used to complement position data. The IMU can measure the tilt and minute movements of the work vehicle 100. By complementing position data based on satellite signals with data acquired by the IMU, positioning performance can be improved.

[0060] The control device of the work vehicle 100 may use, in addition to the positioning results from the GNSS unit 110, sensing data acquired by sensing devices such as the camera 120 or LiDAR sensor 140 for positioning. If there are features that function as characteristic points in the environment in which the work vehicle 100 is traveling, such as farm roads, forest roads, public roads, or orchards, the position and orientation of the work vehicle 100 can be estimated with high accuracy based on the data acquired by the camera 120 or LiDAR sensor 140 and an environmental map that has been stored in advance in a storage device. By using the data acquired by the camera 120 or LiDAR sensor 140 to correct or complement position data based on satellite signals, the position of the work vehicle 100 can be determined with higher accuracy.

[0061] The prime mover 102 may be, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 103 can change the propulsive force and travel speed of the work vehicle 100 by changing gears. The transmission 103 can also switch the work vehicle 100 between forward and reverse travel.

[0062] The steering device 106 includes a steering wheel, a steering shaft connected to the steering wheel, and a power steering device that assists steering by the steering wheel. The front wheels 104F are steerable wheels, and the traveling direction of the work vehicle 100 can be changed by changing the turning angle (also referred to as the "steering angle"). The steering angle of the front wheels 104F can be changed by operating the steering wheel. The power steering device includes a hydraulic device or an electric motor that supplies an assisting force to change the steering angle of the front wheels 104F. When automatic steering is performed, the steering angle is automatically adjusted by the force of the hydraulic device or electric motor under control of a control device arranged inside the work vehicle 100.

[0063] A coupling device 108 is provided at the rear of the vehicle body 101. The coupling device 108 includes, for example, a three-point support device (also referred to as a "three-point link" or "three-point hitch"), a PTO (Power Take Off) shaft, a universal joint, and a communication cable. The coupling device 108 allows the implement 300 to be attached to and detached from the work vehicle 100. The coupling device 108 can raise and lower the three-point link using, for example, a hydraulic device, thereby changing the position or posture of the implement 300. Power can also be sent from the work vehicle 100 to the implement 300 via the universal joint. The work vehicle 100 can pull the implement 300 and cause the implement 300 to perform a predetermined task. The coupling device may be provided at the front of the vehicle body 101. In this case, the implement can be connected to the front of the work vehicle 100.

[0064] 9 is a rotary tiller, but the implement 300 is not limited to a rotary tiller. For example, any implement such as a seeder (seed sowing machine), a spreader (fertilizer applicator), a transplanter, a mower (grass cutter), a rake, a baler (grass collector), a harvester (harvesting machine), a sprayer, or a harrow can be connected to the work vehicle 100 and used.

[0065] The work vehicle 100 shown in FIG. 9 is capable of being driven by a driver, but may also be capable of being driven only unmanned. In that case, components required only for driven operation, such as the cabin 105, steering device 106, and driver's seat 107, do not need to be provided in the work vehicle 100. The unmanned work vehicle 100 can travel autonomously or by remote control by a user. When a work vehicle 100 without the capability of being driven by a driver is used, travel trajectory data for route generation is acquired by a manned vehicle other than the work vehicle 100.

[0066] 10 is a block diagram showing an example configuration of the work vehicle 100 and the implement 300. The work vehicle 100 and the implement 300 can communicate with each other via a communication cable included in the coupling device 108. The work vehicle 100 can communicate with the terminal device 400 and the management device 600 via the network 80.

[0067] In the example of FIG. 10 , the work vehicle 100 includes a GNSS unit 110, a camera 120, an obstacle sensor 130, a LiDAR sensor 140, and an operation terminal 200, as well as a group of sensors 150 that detect the operating state of the work vehicle 100, a control system 160, a communication device 190, a group of operation switches 210, a buzzer 220, and a drive unit 240. These components are communicatively connected to each other via a bus. The GNSS unit 110 includes a GNSS receiver 111, an RTK receiver 112, an inertial measurement unit (IMU) 115, and a processing circuit 116. The group of sensors 150 includes a steering wheel sensor 152, a turning angle sensor 154, and an axle sensor 156. The control system 160 includes a memory device 170 and a control device 180. The control device 180 includes multiple electronic control units (ECUs) 181 to 185. Implement 300 includes a drive unit 340, a control unit 380, and a communication unit 390. Note that Fig. 10 shows components that are relatively highly related to the operation of the autonomous driving by work vehicle 100, and does not show other components.

[0068] The GNSS receiver 111 in the GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated in a predetermined format, such as the NMEA-0183 format. The GNSS data may include, for example, values ​​indicating the identification number, elevation angle, azimuth angle, and reception strength of each satellite from which a satellite signal is received.

[0069] The GNSS unit 110 shown in FIG. 10 performs positioning of the work vehicle 100 using RTK (Real Time Kinematic)-GNSS. FIG. 11 is a conceptual diagram showing an example of a work vehicle 100 performing positioning using RTK-GNSS. Positioning using RTK-GNSS uses satellite signals transmitted from multiple GNSS satellites 50 as well as correction signals transmitted from a reference station 60. The reference station 60 may be installed near the field where the work vehicle 100 will be traveling (for example, within 10 km of the work vehicle 100). The reference station 60 generates correction signals, for example in RTCM format, based on the satellite signals received from the multiple GNSS satellites 50 and transmits them to the GNSS unit 110. The RTK receiver 112 includes an antenna and a modem and receives the correction signals transmitted from the reference station 60. The processing circuit 116 of the GNSS unit 110 corrects the positioning results obtained by the GNSS receiver 111 based on the correction signals. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position information including latitude, longitude, and altitude information is obtained through highly accurate positioning using RTK-GNSS. The GNSS unit 110 calculates the position of the work vehicle 100, for example, at a frequency of approximately 1 to 10 times per second.

[0070] The positioning method is not limited to RTK-GNSS, and any positioning method (such as interferometric positioning or differential positioning) that can obtain position information with the required accuracy can be used. For example, positioning may be performed using a Virtual Reference Station (VRS) or a Differential Global Positioning System (DGPS). If position information with the required accuracy can be obtained without using a correction signal transmitted from the reference station 60, the position information may be generated without using a correction signal. In this case, the GNSS unit 110 does not need to be equipped with the RTK receiver 112.

[0071] Even when RTK-GNSS is used, in places where correction signals from the reference station 60 cannot be obtained (for example, on a road far from a field), the position of the work vehicle 100 is estimated by other methods without relying on signals from the RTK receiver 112. For example, the position of the work vehicle 100 can be estimated by matching data output from the LiDAR sensor 140 and / or camera 120 with a highly accurate environmental map.

[0072] The GNSS unit 110 in this embodiment further includes an IMU 115. The IMU 115 may include a three-axis acceleration sensor and a three-axis gyroscope. The IMU 115 may also include a direction sensor such as a three-axis geomagnetic sensor. The IMU 115 functions as a motion sensor and can output signals indicating various quantities such as the acceleration, velocity, displacement, and attitude of the work vehicle 100. The processing circuit 116 can estimate the position and orientation of the work vehicle 100 with higher accuracy based on the signals output from the IMU 115 in addition to the satellite signals and correction signals. The signals output from the IMU 115 can be used to correct or complement the position calculated based on the satellite signals and correction signals. The IMU 115 outputs signals at a higher frequency than the GNSS receiver 111. Using these high-frequency signals, the processing circuit 116 can measure the position and orientation of the work vehicle 100 at a higher frequency (e.g., 10 Hz or higher). A three-axis acceleration sensor and a three-axis gyroscope may be provided separately instead of the IMU 115. The IMU 115 may be provided as a device separate from the GNSS unit 110.

[0073] The camera 120 is an imaging device that captures images of the environment surrounding the work vehicle 100. The camera 120 includes an image sensor, such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 120 may also include an optical system including one or more lenses and a signal processing circuit. The camera 120 captures images of the environment surrounding the work vehicle 100 while the work vehicle 100 is traveling and generates image (e.g., video) data. The camera 120 can capture video at a frame rate of, for example, 3 frames per second (fps) or higher. The images generated by the camera 120 can be used, for example, when a remote observer uses the terminal device 400 to check the environment surrounding the work vehicle 100. The images generated by the camera 120 may also be used for positioning or obstacle detection. For example, the images generated by the camera 120 while the work vehicle 100 is traveling to collect the aforementioned travel trajectory data can be used in the process of recognizing oncoming vehicles and detecting actions to avoid the oncoming vehicles. As shown in Fig. 9, multiple cameras 120 may be provided at different positions on the work vehicle 100, or a single camera may be provided. A visible light camera that generates visible light images and an infrared camera that generates infrared images may be provided separately. Both a visible light camera and an infrared camera may be provided as cameras that generate images for monitoring. The infrared camera may also be used to detect obstacles at night.

[0074] The obstacle sensor 130 detects objects present around the work vehicle 100. The obstacle sensor 130 may include, for example, a laser scanner or an ultrasonic sonar. The obstacle sensor 130 outputs a signal indicating the presence of an obstacle when an object is present closer than a predetermined distance from the obstacle sensor 130. Multiple obstacle sensors 130 may be provided at different positions on the work vehicle 100. For example, multiple laser scanners and multiple ultrasonic sonars may be arranged at different positions on the work vehicle 100. By providing such a large number of obstacle sensors 130, blind spots in monitoring obstacles around the work vehicle 100 can be reduced.

[0075] The steering wheel sensor 152 measures the rotation angle of the steering wheel of the work vehicle 100. The turning angle sensor 154 measures the turning angle of the front wheels 104F, which are the steered wheels. The measurement values ​​from the steering wheel sensor 152 and the turning angle sensor 154 are used for steering control by the control device 180.

[0076] The axle sensor 156 measures the rotational speed of the axle connected to the wheel 104, i.e., the number of rotations per unit time. The axle sensor 156 may be a sensor that uses, for example, a magnetoresistive element (MR), a Hall element, or an electromagnetic pickup. The axle sensor 156 outputs a numerical value that indicates, for example, the number of rotations per minute (unit: rpm) of the axle. The axle sensor 156 is used to measure the speed of the work vehicle 100.

[0077] The drive device 240 includes various devices necessary for the travel of the work vehicle 100 and the driving of the implement 300, such as the prime mover 102, transmission 103, steering device 106, and coupling device 108 described above. The prime mover 102 may be equipped with an internal combustion engine such as a diesel engine. The drive device 240 may be equipped with an electric motor for traction instead of or in addition to the internal combustion engine.

[0078] The buzzer 220 is an audio output device that emits a warning sound to notify of an abnormality. For example, the buzzer 220 emits the warning sound when an obstacle is detected during automatic driving. The buzzer 220 is controlled by the control device 180.

[0079] The storage device 170 includes one or more storage media, such as a flash memory or a magnetic disk. The storage device 170 stores various data generated by the GNSS unit 110, the camera 120, the obstacle sensor 130, the LiDAR sensor 140, the sensor group 150, and the control device 180. The data stored in the storage device 170 may include map data (environmental map) of the environment in which the work vehicle 100 travels, and data on an automatic driving route (target route) for automatic driving. The environmental map includes information on multiple fields in which the work vehicle 100 will perform agricultural work and the roads in their surroundings. The environmental map and target route may be generated by a processor in the management device 600. Note that the control device 180 in this embodiment has the function of generating or editing the environmental map and target route. The control device 180 can edit the environmental map and target route obtained from the management device 600 according to the driving environment of the work vehicle 100. The storage device 170 also stores computer programs that cause each ECU in the control device 180 to perform various operations, which will be described later. Such a computer program may be provided to the work vehicle 100 via a storage medium (e.g., a semiconductor memory or an optical disk) or a telecommunications line (e.g., the Internet). Such a computer program may also be sold as commercial software.

[0080] The control device 180 includes a plurality of ECUs, such as an ECU 181 for speed control, an ECU 182 for steering control, an ECU 183 for implement control, an ECU 184 for automatic driving control, and an ECU 185 for path generation.

[0081] The ECU 181 controls the speed of the work vehicle 100 by controlling the prime mover 102 , the transmission 103 , and the brakes included in the drive unit 240 .

[0082] The ECU 182 controls the steering of the work vehicle 100 by controlling the hydraulic device or electric motor included in the steering device 106 based on the measurement value of the steering wheel sensor 152 .

[0083] The ECU 183 controls the operation of the three-point link and PTO shaft included in the coupling device 108, etc., in order to cause the implement 300 to perform a desired operation. The ECU 183 also generates a signal that controls the operation of the implement 300, and transmits the signal from the communication device 190 to the implement 300.

[0084] The ECU 184 performs calculations and controls to achieve autonomous driving based on data output from the GNSS unit 110, the camera 120, the obstacle sensor 130, the LiDAR sensor 140, and the sensor group 150. For example, the ECU 184 determines the position of the work vehicle 100 based on data output from at least one of the GNSS unit 110, the camera 120, and the LiDAR sensor 140. Within a farm field, the ECU 184 may determine the position of the work vehicle 100 based solely on data output from the GNSS unit 110. The ECU 184 may also estimate or correct the position of the work vehicle 100 based on data acquired by the camera 120 or the LiDAR sensor 140. By using the data acquired by the camera 120 or the LiDAR sensor 140, the accuracy of positioning can be further improved. Outside of a farm field, the ECU 184 estimates the position of the work vehicle 100 using data output from the LiDAR sensor 140 or the camera 120. For example, ECU 184 may estimate the position of work vehicle 100 by matching data output from LiDAR sensor 140 or camera 120 with an environmental map. During autonomous driving, ECU 184 performs calculations necessary for work vehicle 100 to travel along a target path or a local path based on the estimated position of work vehicle 100. ECU 184 sends a speed change command to ECU 181 and a steering angle change command to ECU 182. In response to the speed change command, ECU 181 changes the speed of work vehicle 100 by controlling prime mover 102, transmission 103, or brakes. In response to the steering angle change command, ECU 182 changes the steering angle by controlling steering device 106.

[0085] While the work vehicle 100 is traveling along the target route, the ECU 185 sequentially generates local routes that can avoid obstacles. While the work vehicle 100 is traveling, the ECU 185 recognizes obstacles present around the work vehicle 100 based on data output from the camera 120, the obstacle sensor 130, and the LiDAR sensor 140. The ECU 185 generates local routes that avoid the recognized obstacles. The ECU 185 may have a function for generating a target route instead of the management device 600. In that case, the ECU 185 generates the target route based on data output from the GNSS unit 110, the camera 120, and / or the LiDAR sensor 140 while the work vehicle 100 is traveling for data collection. Examples of methods for generating the target route are as described with reference to FIGS. 3 to 7D. Note that the target route is not limited to being generated by the management device 600 or the ECU 185, and may also be generated by other devices, such as the operation terminal 200 or the terminal device 400.

[0086] Through the operation of these ECUs, control device 180 realizes autonomous driving. During autonomous driving, control device 180 controls drive device 240 based on the measured or estimated position of work vehicle 100 and the target route. In this way, control device 180 causes work vehicle 100 to travel along the target route.

[0087] The multiple ECUs included in the control device 180 can communicate with each other in accordance with a vehicle bus standard such as CAN (Controller Area Network). Instead of CAN, a faster communication method such as Automotive Ethernet (registered trademark) may be used. In FIG. 10, each of the ECUs 181 to 185 is shown as an individual block, but the functions of each of these may be realized by multiple ECUs. An on-board computer that integrates at least some of the functions of the ECUs 181 to 185 may be provided. The control device 180 may include ECUs other than the ECUs 181 to 185, and any number of ECUs may be provided depending on the functions. Each ECU includes a processing circuit including one or more processors.

[0088] The communication device 190 includes circuits for communicating with the implement 300, the terminal device 400, and the management device 600. The communication device 190 includes circuits for transmitting and receiving signals compliant with ISOBUS standards, such as ISOBUS-TIM, between the communication device 390 of the implement 300. This allows the implement 300 to perform desired operations and acquire information from the implement 300. The communication device 190 may further include an antenna and communication circuits for transmitting and receiving signals via the network 80 between the communication devices of the terminal device 400 and the management device 600. The network 80 may include, for example, a cellular mobile communication network such as 3G, 4G, or 5G, and the Internet. The communication device 190 may also have a function for communicating with a mobile device used by a supervisor near the work vehicle 100. Communication between such mobile terminals may be performed in accordance with any wireless communication standard, such as Wi-Fi (registered trademark), cellular mobile communication such as 3G, 4G or 5G, or Bluetooth (registered trademark).

[0089] The operation terminal 200 is a terminal through which a user performs operations related to the travel of the work vehicle 100 and the operation of the implement 300, and is also referred to as a virtual terminal (VT). The operation terminal 200 may include a display device such as a touch screen and / or one or more buttons. The display device may be, for example, a liquid crystal display or an organic light-emitting diode (OLED) display. By operating the operation terminal 200, a user can perform various operations, such as switching the autonomous driving mode on / off, recording or editing an environmental map, setting a target route, and switching the implement 300 on / off. At least some of these operations can also be achieved by operating the operation switch group 210. The operation terminal 200 may be configured to be detachable from the work vehicle 100. A user located remotely from the work vehicle 100 may operate the detached operation terminal 200 to control the operation of the work vehicle 100. Instead of the operation terminal 200, the user may control the operation of the work vehicle 100 by operating a computer, such as a terminal device 400, on which necessary application software is installed.

[0090] 12 is a diagram showing an example of operation terminal 200 and operation switch group 210 provided inside cabin 105. Switch group 210 including a plurality of switches that can be operated by the user is arranged inside cabin 105. Operation switch group 210 may include, for example, a switch for selecting the gear stage of the main transmission or auxiliary transmission, a switch for switching between automatic driving mode and manual driving mode, a switch for switching between forward and reverse, and a switch for raising and lowering implement 300. Note that if work vehicle 100 only performs unmanned operation and does not have the function of manned operation, work vehicle 100 does not need to be equipped with operation switch group 210.

[0091] The drive device 340 in the implement 300 shown in FIG. 10 performs the operations required for the implement 300 to perform a predetermined task. The drive device 340 includes devices appropriate for the application of the implement 300, such as a hydraulic device, an electric motor, or a pump. The control device 380 controls the operation of the drive device 340. The control device 380 causes the drive device 340 to perform various operations in response to signals transmitted from the work vehicle 100 via the communication device 390. In addition, a signal appropriate for the state of the implement 300 can also be transmitted from the communication device 390 to the work vehicle 100.

[0092] Next, the configurations of the management device 600 and the terminal device 400 will be described with reference to Fig. 13. Fig. 13 is a block diagram illustrating a schematic hardware configuration of the management device 600 and the terminal device 400.

[0093] The management device 600 includes a storage device 650, a processor 660, a read-only memory (ROM) 670, a random access memory (RAM) 680, and a communication device 690. These components are communicatively connected to each other via a bus. The management device 600 manages the schedule of agricultural work performed in the field by the work vehicle 100 and can function as a cloud server that supports agriculture by utilizing the data it manages. A user can input information necessary for creating a work plan using the terminal device 400 and upload that information to the management device 600 via the network 80. The management device 600 can create a schedule of agricultural work, i.e., a work plan, based on that information. The management device 600 can also generate or edit an environmental map and generate an automated driving route for the work vehicle 100. The environmental map may be distributed from a computer external to the management device 600.

[0094] The communication device 690 is a communication module for communicating with the work vehicle 100 and the terminal device 400 via the network 80. The communication device 690 can perform wired communication in accordance with communication standards such as IEEE1394 (registered trademark) or Ethernet (registered trademark). The communication device 690 may also perform wireless communication in accordance with the Bluetooth (registered trademark) standard or the Wi-Fi standard, or cellular mobile communication such as 3G, 4G, or 5G.

[0095] The processor 660 may be, for example, a semiconductor integrated circuit including a central processing unit (CPU). The processor 660 may be implemented by a microprocessor or a microcontroller. Alternatively, the processor 660 may be implemented by a field programmable gate array (FPGA) equipped with a CPU, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), or a combination of two or more circuits selected from these circuits. The processor 660 sequentially executes a computer program stored in the ROM 670, which describes a group of instructions for executing at least one process, to achieve the desired process.

[0096] The ROM 670 is, for example, a writable memory (e.g., a PROM), a rewritable memory (e.g., a flash memory), or a read-only memory. The ROM 670 stores a program that controls the operation of the processor 660. The ROM 670 does not need to be a single storage medium, but may be a collection of multiple storage media. Part of the collection of multiple storage media may be removable memory.

[0097] The RAM 680 provides a working area for temporarily loading the control program stored in the ROM 670 at boot time. The RAM 680 does not have to be a single storage medium, but may be a collection of multiple storage media.

[0098] The storage device 650 mainly functions as database storage. The storage device 650 may be, for example, a magnetic storage device or a semiconductor storage device. An example of a magnetic storage device is a hard disk drive (HDD). An example of a semiconductor storage device is a solid state drive (SSD). The storage device 650 may be a device independent of the management device 600. For example, the storage device 650 may be a storage device connected to the management device 600 via the network 80, such as a cloud storage device.

[0099] The terminal device 400 includes an input device 420, a display device 430, a storage device 450, a processor 460, a ROM 470, a RAM 480, and a communication device 490. These components are communicatively connected to one another via a bus. The input device 420 is a device for converting user instructions into data and inputting the data to a computer. The input device 420 may be, for example, a keyboard, a mouse, or a touch panel. The display device 430 may be, for example, a liquid crystal display or an organic EL display. The processor 460, the ROM 470, the RAM 480, the storage device 450, and the communication device 490 are described in the hardware configuration example of the management device 600, and therefore their description will be omitted.

[0100] [2. Operation] Next, an example of the operation of the automatic driving of the work vehicle 100 will be described. In this embodiment, the work vehicle 100 can automatically drive both inside and outside the field. In the field, the work vehicle 100 drives the implement 300 while traveling along a predetermined target route to perform predetermined agricultural work. If the obstacle sensor 130 detects an obstacle while traveling in the field, the work vehicle 100 stops traveling, emits a warning sound from the buzzer 220, and transmits a warning signal to the terminal device 400. In the field, the positioning of the work vehicle 100 is performed mainly based on data output from the GNSS unit 110. On the other hand, outside the field, the work vehicle 100 automatically drives along a target route set on a farm road or public road outside the field. While traveling outside the field, the work vehicle 100 drives while detecting obstacles based on data acquired by the camera 120 or LiDAR sensor 140. When work vehicle 100 detects an obstacle outside the field, it either avoids the obstacle or stops there. Outside the field, the position of work vehicle 100 is estimated based on the positioning data output from GNSS unit 110 as well as the data output from LiDAR sensor 140 or camera 120.

[0101] An example of the operation of the work vehicle 100 when it travels automatically within a farm field will now be described.

[0102] FIG. 14 is a schematic diagram illustrating an example of a work vehicle 100 automatically traveling through a field along a target route. In this example, the field includes a work area 72 where the work vehicle 100 performs work using an implement 300 and a headland 74 located near the outer edge of the field. The user can set in advance which areas of the field on the map correspond to the work area 72 or the headland 74. The target route in this example includes multiple parallel main routes P1 and multiple turning routes P2 connecting the multiple main routes P1. The main routes P1 are located within the work area 72, and the turning routes P2 are located within the headland 74. Although each main route P1 shown in FIG. 14 is a straight route, each main route P1 may also include curved portions. The dashed line in FIG. 14 represents the working width of the implement 300. The working width is set in advance and recorded in the storage device 170. The working width may be set and recorded by the user operating the operation terminal 200 or the terminal device 400. Alternatively, the working width may be automatically recognized and recorded when the implement 300 is connected to the work vehicle 100. The spacing between the multiple main paths P1 may be set to match the working width. A target route may be created based on user operation before automatic driving begins. The target route may be created to cover, for example, the entire work area 72 in a field. The work vehicle 100 automatically travels back and forth from the start point of work to the end point of work along a target route such as that shown in FIG. 14. Note that the target route shown in FIG. 14 is merely an example, and the target route may be defined in any manner.

[0103] Next, an example of control by the control device 180 during automatic operation will be described.

[0104] FIG. 15 is a flowchart showing an example of the operation of steering control during automatic driving executed by the control device 180. The control device 180 performs automatic steering by executing the operations of steps S121 to S125 shown in FIG. 15 while the work vehicle 100 is traveling. The speed may be maintained at a preset speed, for example, or may be adjusted according to the situation. While the work vehicle 100 is traveling, the control device 180 acquires data indicating the position of the work vehicle 100 generated by the GNSS unit 110 (step S121). Next, the control device 180 calculates the deviation between the position of the work vehicle 100 and the target route (step S122). The deviation represents the distance between the position of the work vehicle 100 at that time and the target route. The control device 180 determines whether the calculated position deviation exceeds a preset threshold (step S123). If the deviation exceeds the threshold, the control device 180 changes the steering angle by changing the control parameters of the steering device included in the drive device 240 so as to reduce the deviation. If the deviation does not exceed the threshold value in step S123, the operation of step S124 is skipped. In the following step S125, the control device 180 determines whether or not a command to end the operation has been received. A command to end the operation may be issued, for example, when a user remotely instructs the work vehicle 100 to stop autonomous driving, or when the work vehicle 100 reaches its destination. If a command to end the operation has not been issued, the process returns to step S121, and the same operation is performed based on the newly measured position of the work vehicle 100. The control device 180 repeats the operations of steps S121 to S125 until a command to end the operation is issued. The above operations are executed by the ECUs 182 and 184 in the control device 180.

[0105] 15, the control device 180 controls the drive device 240 based only on the deviation between the position of the work vehicle 100 identified by the GNSS unit 110 and the target route, but the control may also take into consideration the deviation in heading. For example, when the heading deviation, which is the angular difference between the orientation of the work vehicle 100 identified by the GNSS unit 110 and the direction of the target route, exceeds a preset threshold, the control device 180 may change the control parameters (e.g., steering angle) of the steering device of the drive device 240 in accordance with the deviation.

[0106] An example of steering control by the control device 180 will be described in more detail below with reference to FIGS. 16A to 16D.

[0107] FIG. 16A is a diagram showing an example of a work vehicle 100 traveling along a target route P. FIG. 16B is a diagram showing an example of a work vehicle 100 shifted to the right from the target route P. FIG. 16C is a diagram showing an example of a work vehicle 100 shifted to the left from the target route P. FIG. 16D is a diagram showing an example of a work vehicle 100 facing in an inclined direction with respect to the target route P. In these figures, the pose indicating the position and orientation of the work vehicle 100 measured by the GNSS unit 110 is expressed as r(x, y, θ). (x, y) are coordinates representing the position of the reference point of the work vehicle 100 in the XY coordinate system, which is a two-dimensional coordinate system fixed to the Earth. In the examples shown in FIGS. 16A to 16D, the reference point of the work vehicle 100 is located at the position where the GNSS antenna on the cabin is installed, but the position of the reference point is arbitrary. θ is an angle representing the measured orientation of the work vehicle 100. In the illustrated example, the target path P is parallel to the Y axis, but in general, the target path P is not necessarily parallel to the Y axis.

[0108] As shown in FIG. 16A, if the position and orientation of the work vehicle 100 do not deviate from the target route P, the control device 180 maintains the steering angle and speed of the work vehicle 100 unchanged.

[0109] As shown in Fig. 16B, when the position of work vehicle 100 has shifted to the right from target route P, control device 180 changes the steering angle so that the traveling direction of work vehicle 100 leans leftward and approaches route P. At this time, the speed may also be changed in addition to the steering angle. The magnitude of the steering angle can be adjusted, for example, according to the magnitude of position deviation Δx.

[0110] As shown in Fig. 16C, when the position of the work vehicle 100 has shifted to the left from the target route P, the control device 180 changes the steering angle so that the traveling direction of the work vehicle 100 tilts to the right and approaches the route P. In this case, too, the speed may be changed in addition to the steering angle. The amount of change in the steering angle may be adjusted, for example, according to the magnitude of the position deviation Δx.

[0111] As shown in FIG. 16D , when the position of the work vehicle 100 is not significantly deviated from the target route P but the heading is different from the direction of the target route P, the control device 180 changes the steering angle to reduce the azimuth deviation Δθ. In this case, the speed may also be changed in addition to the steering angle. The magnitude of the steering angle may be adjusted, for example, according to the magnitudes of the position deviation Δx and the azimuth deviation Δθ. For example, the smaller the absolute value of the position deviation Δx, the greater the amount of change in the steering angle according to the azimuth deviation Δθ. When the absolute value of the position deviation Δx is large, the steering angle will be changed significantly to return to the route P, which inevitably increases the absolute value of the azimuth deviation Δθ. Conversely, when the absolute value of the position deviation Δx is small, it is necessary to bring the azimuth deviation Δθ closer to zero. For this reason, it is appropriate to relatively increase the weight of the azimuth deviation Δθ (i.e., the control gain) used to determine the steering angle.

[0112] Control techniques such as PID control or MPC control (model predictive control) can be applied to the steering control and speed control of work vehicle 100. By applying these control techniques, it is possible to smooth the control that brings work vehicle 100 closer to target path P.

[0113] If an obstacle is detected by one or more obstacle sensors 130 while the work vehicle 100 is traveling, the control device 180 will stop the work vehicle 100. At this time, the buzzer 220 may be caused to emit a warning sound or a warning signal may be sent to the terminal device 400. If it is possible to avoid the obstacle, the control device 180 may control the drive device 240 to avoid the obstacle.

[0114] The work vehicle 100 in this embodiment is capable of autonomous driving not only within a field but also outside the field. Outside the field, the control device 180 performs steering control and speed control along the target route (autonomous driving route) generated by the method described above. The control device 180 can detect objects (e.g., other vehicles or pedestrians) that are located relatively far from the work vehicle 100 based on data output from the camera 120 or the LiDAR sensor 140. The control device 180 generates a local route to avoid detected objects, and performs speed control and steering control along the local route, thereby realizing autonomous driving on roads outside the field.

[0115] In this way, the work vehicle 100 in this embodiment can travel autonomously in and out of fields without a driver. The storage device 170 stores an environmental map of an area including multiple fields and the roads around them, as well as a target route. When the work vehicle 100 travels on a road, the work vehicle 100 travels along the target route with the implement 300 raised, while sensing the surroundings using sensing devices such as the camera 120 and LiDAR sensor 140. While traveling, the control device 180 sequentially generates local routes and causes the work vehicle 100 to travel along the local routes. This allows the work vehicle 100 to travel autonomously while avoiding obstacles. The target route may be changed during travel depending on the situation.

[0116] As described above, according to this embodiment, the actual driving route when the work vehicle 100 was manually driven can be used to generate a route for automated driving. The route for automated driving is generated excluding the actual driving route when the work vehicle 100 performed an operation to avoid an oncoming vehicle. This prevents inappropriate routes resulting from an avoidance operation from being included in the automated driving route, and makes it possible to generate an appropriate automated driving route. According to the automated driving route generated in this way, the work vehicle 100 can appropriately perform automated driving on roads around a field, for example.

[0117] The systems for generating automated driving routes or performing automated driving control in the above embodiments can also be retrofitted to agricultural machines that do not have these functions. Such systems can be manufactured and sold independently of the agricultural machines. The computer programs used in such systems can also be manufactured and sold independently of the agricultural machines. The computer programs can be provided, for example, by being stored on a computer-readable non-transitory storage medium. The computer programs can also be provided by downloading via a telecommunications line (for example, the Internet).

[0118] As described above, the present disclosure includes the route generation systems and route generation methods described in the following items.

[0119] [Item 1] A route generation system for automatic travel of an agricultural machine, a processing device for generating an automatic travel route for the agricultural machine; The processing device includes: acquiring data indicating a travel locus from a vehicle that manually travels along a route on which the agricultural machine is to be automatically traveled while recording the travel locus; remove a trajectory related to an avoidance operation performed to avoid an oncoming vehicle from the travel trajectory; generating an automated driving route for the agricultural machine based on the driving trajectory from which the trajectory related to the avoidance operation has been removed; Route generation system.

[0120] [Item 2] The processing device includes: acquiring data of moving images captured by a camera mounted on the vehicle while the vehicle is traveling; detecting the avoidance operation based on the moving image, and determining and removing a trajectory related to the avoidance operation from the traveling trajectory; Item 1. A pathway generation system according to item 1.

[0121] [Item 3] The processing device includes: Recognizing an oncoming vehicle approaching the vehicle from the moving image; removing, from the travel trajectory, a trajectory corresponding to at least a part of a period from when the oncoming vehicle is recognized until when the oncoming vehicle is no longer recognized, as a trajectory related to the avoidance operation; Item 2. A pathway generation system according to item 2.

[0122] [Item 4] Item 1. The path generation system according to item 1, wherein the processing device detects the avoidance operation based on a change in the position of the vehicle over time indicated by the travel trajectory, and determines and removes a trajectory related to the avoidance operation from the travel trajectory.

[0123] [Item 5] 5. A path generation system according to any one of items 1 to 4, wherein the processing device detects at least one of the following actions taken by the vehicle to avoid an oncoming vehicle: reversing, changing direction, accelerating, and decelerating as the avoidance action.

[0124] [Item 6] 6. The route generation system according to any one of items 1 to 5, wherein the processing device acquires position data sequentially output from a GNSS receiver mounted on the vehicle as data indicating the traveling trajectory.

[0125] [Item 7] 7. A route generation system according to any one of items 1 to 6, wherein the processing device generates as the automatic driving route a route defined by a plurality of waypoints, each of which includes position and speed information.

[0126] [Item 8] 8. The route generation system according to any one of items 1 to 7, wherein the processing device generates the automatic driving route by performing processing to complement the portion removed from the driving trajectory.

[0127] [Item 9] Item 9. The route generation system according to item 8, wherein the processing device generates the automatic driving route by complementing the portion removed from the driving trajectory with a linear complementary route.

[0128] [Item 10] the processing device causes a display device to display the travel path from which the path related to the avoidance operation has been removed; complementing the portion removed from the travel path in response to an operation performed by a user to determine a complementary route; Item 9. The pathway generation system according to item 8.

[0129] [Item 11] 11. A route generation system according to any one of items 1 to 10, wherein the processing device executes the process of generating the automatic driving route when generating a route for the agricultural machine to automatically drive outside a field.

[0130] [Item 12] A route generation method for automatic travel of an agricultural machine, comprising: acquiring data indicating a travel locus from a vehicle that manually travels along a route along which the agricultural machine is to be automatically driven while recording the travel locus; removing a trajectory related to an avoidance operation performed to avoid the oncoming vehicle from the travel trajectory; generating an automated travel route for the agricultural machine based on the travel trajectory from which the trajectory related to the avoidance operation has been removed; A route generation method comprising: [Industrial Applicability]

[0131] The technology disclosed herein can be applied to a system that generates automatic driving paths for agricultural machinery such as tractors, harvesters, rice transplanters, riding tillers, vegetable transplanters, mowers, seed sowing machines, fertilizer applicators, or agricultural robots. [Explanation of symbols]

[0132] 10... Path generation system, 11... Input interface, 12... Output interface, 15... Processing unit, 16... Storage device, 20... Vehicle, 21... GNSS receiver, 22... Camera, 23... Storage device, 30... Agricultural machinery, 31... Self-position estimation device, 32... Driving control device, 33... Storage device, 40... Input device, 45... Display device, 50... GNSS satellite, 60... Reference station, 70... Field, 72... Working area, 74... Pillow Ground, 75... Farm road, 78... Storage shed, 80... Network, 90... Oncoming vehicle, 91, 92... Route, 93... Avoidance route, 94... Complementary route, 100... Work vehicle, 101... Vehicle body, 102... Engine, 103... Transmission, 104... Wheels, 105... Cabin, 106... Steering gear, 107... Driver's seat, 108... Coupling device, 110... GNSS unit, 111... GNSS receiver, 112... RT K receiver, 115... inertial measurement unit (IMU), 116... processing circuit, 120... camera, 130... obstacle sensor, 140... LiDAR sensor, 150... sensor group, 152... steering wheel sensor, 154... turning angle sensor, 156... axle sensor, 160... control system, 170... storage device, 180... control device, 181 to 185... ECU, 190... communication device, 200... operation terminal, 210... operation switch group, 220... Buzzer, 240, drive unit, 300, implement, 340, drive unit, 380, control unit, 390, communication unit, 400, terminal unit, 420, input unit, 430, display unit, 450, storage unit, 460, processor, 470, ROM, 480, RAM, 490, communication unit, 600, management computer, 660, processor, 670, storage unit, 670, ROM, 680, RAM, 690, communication unit

Claims

1. A route generation system for automatic travel of an agricultural machine, a processing device for generating an automatic travel route for the agricultural machine; The processing device includes: acquiring data indicating a travel locus from a vehicle that manually travels along a route on which the agricultural machine is to be automatically traveled while recording the travel locus; remove a trajectory related to an avoidance operation performed to avoid an oncoming vehicle from the travel trajectory; generating an automated driving route for the agricultural machine based on the driving trajectory from which the trajectory related to the avoidance operation has been removed; Route generation system.

2. The processing device includes: acquiring data of moving images captured by a camera mounted on the vehicle while the vehicle is traveling; detecting the avoidance operation based on the moving image, and determining and removing a trajectory related to the avoidance operation from the traveling trajectory; The route generation system according to claim 1 .

3. The processing device includes: Recognizing an oncoming vehicle approaching the vehicle from the moving image; removing, from the travel trajectory, a trajectory corresponding to at least a part of a period from when the oncoming vehicle is recognized until when the oncoming vehicle is no longer recognized, as a trajectory related to the avoidance operation; The route generation system according to claim 2 .

4. The route generation system according to claim 1 , wherein the processing device detects the avoidance operation based on a change over time in the position of the vehicle indicated by the travel trajectory, and determines and removes a trajectory related to the avoidance operation from the travel trajectory.

5. The route generation system according to claim 1 , wherein the processing device detects at least one of an action of reversing, changing direction, accelerating, and decelerating that the vehicle performs to avoid an oncoming vehicle as the avoidance action.

6. The route generation system according to claim 1 , wherein the processing device acquires, as the data indicating the travel locus, position data sequentially output from a GNSS receiver mounted on the vehicle.

7. The route generation system according to claim 1 , wherein the processing device generates, as the automated driving route, a route defined by a plurality of waypoints, each of which includes position and speed information.

8. The route generation system according to claim 1 , wherein the processing device generates the automated driving route by performing a process of complementing a portion removed from the driving locus.

9. The route generation system according to claim 8 , wherein the processing device generates the automated driving route by complementing a portion removed from the driving trajectory with a linear complementary route.

10. the processing device causes a display device to display the travel path from which the path related to the avoidance operation has been removed; complementing the portion removed from the travel path in response to an operation performed by a user to determine a complementary route; The route generation system according to claim 8 .

11. 5. The route generation system according to claim 1, wherein the processing device executes the process of generating the automatic driving route when generating a route for the agricultural machine to automatically drive outside a field.

12. A route generation method for automatic travel of an agricultural machine, comprising: acquiring data indicating a travel locus from a vehicle that manually travels along a route along which the agricultural machine is to be automatically driven while recording the travel locus; removing a trajectory related to an avoidance operation performed to avoid an oncoming vehicle from the travel trajectory; generating an automated travel route for the agricultural machine based on the travel trajectory from which the trajectory related to the avoidance operation has been removed; A route generation method comprising:

Images