Video display system and work vehicle

Abstract

This video display system comprises: an imaging device that is attached to a work vehicle to which a work machine is connected, and generates time series image data by capturing images in the advancement direction of the work vehicle; a screen; and a control device that causes a video based on the time series image data to be displayed on the screen. When the work machine is connected to the work vehicle on the side opposite the advancement direction of the work vehicle, the control device causes a composite video, in which the course of the work machine is superimposed over the video, to be displayed on the screen.

Description

Technical Field

[0001] The present disclosure relates to a video display system and a work vehicle.

Background Art

[0002] Research and development for automating agricultural machinery used in fields are underway. For example, work vehicles such as tractors, combines, and rice transplanters that automatically travel within a field using a positioning system such as GNSS (Global Navigation Satellite System) have been put into practical use. Research and development of work vehicles that automatically travel not only within the field but also outside the field are also underway. Development of technology for remotely operating agricultural machinery is also underway.

[0003] Patent Documents 1 and 2 disclose an example of a system for automatically driving an unmanned work vehicle between two fields separated from each other by a road. Patent Document 3 discloses an example of a device for remotely operating a work vehicle that performs autonomous driving.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure provides a technique for displaying, on a screen, a composite video in which an image showing at least one of the trajectory of a working machine and the working trace after ground operation in the traveling direction of a work vehicle is superimposed on a video.

Means for Solving the Problems

[0006] A video display system according to one aspect of the present disclosure comprises an imaging device attached to a work vehicle to which a work machine is connected, which generates time-series image data by taking photographs in the direction of travel of the work vehicle; a screen; and a control device which displays a video based on the time-series image data on the screen, wherein when the work machine is connected to the opposite side of the direction of travel of the work vehicle, the control device displays a composite video on the screen in which the trajectory of the work machine is superimposed on the video.

[0007] An agricultural machine according to one aspect of this disclosure comprises a work vehicle, a work machine, and the above-mentioned video display system.

[0008] Another aspect of the present disclosure provides a video display system comprising: an imaging device attached to a work vehicle to which a work machine is attached, which generates time-series image data by taking photographs in the direction of travel of the work vehicle; a screen; and a control device that displays a video on the screen based on the time-series image data, wherein the control device displays a composite video on the screen in which images showing the work tracks after ground work in the direction of travel, which are predicted to occur while the work vehicle to which the work machine is attached is traveling, are superimposed on the video.

[0009] Another aspect of the present disclosure of agricultural machinery comprises a work vehicle, a work machine, and the above-described video display system.

[0010] The comprehensive or specific embodiments of this disclosure may be implemented by apparatus, systems, methods, integrated circuits, computer programs, or computer-readable non-temporary storage media, or any combination thereof. Computer-readable storage media may include volatile storage media or non-volatile storage media. Apparatus may consist of multiple devices. If apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. [Effects of the Invention]

[0011] According to embodiments of this disclosure, it is possible to display a composite image on a screen in which an image showing at least one of the trajectory of the work machine and the work trace after ground work in the direction of travel of the work vehicle is superimposed on the video. [Brief explanation of the drawing]

[0012] [Figure 1A] This diagram shows an example of the configuration of an agricultural management system. [Figure 1B] This figure shows other configuration examples for agricultural management systems. [Figure 2] This is a schematic side view showing an example of a work vehicle and an implement connected to the work vehicle. [Figure 3] This is a block diagram showing an example configuration of a work vehicle and implement. [Figure 4] This is a conceptual diagram showing an example of a work vehicle that performs positioning using RTK-GNSS. [Figure 5] This figure shows an example of an operating terminal and a group of operating switches installed inside the cabin. [Figure 6] This is a block diagram illustrating the hardware configuration of the management device and remote device. [Figure 7] This diagram schematically illustrates an example of a work vehicle that automatically travels along a target route within a field. [Figure 8] This flowchart shows an example of steering control operation during autonomous driving. [Figure 9A] This figure shows an example of a work vehicle traveling along a target route P. [Figure 9B] This figure shows an example of a work vehicle positioned to the right of the target path P. [Figure 9C] This figure shows an example of a work vehicle positioned to the left of the target path P. [Figure 9D] This figure shows an example of a work vehicle facing in a direction inclined with respect to the target path P. [Figure 10] This diagram schematically illustrates an example of multiple work vehicles automatically navigating roads both inside and outside a field. [Figure 11] It is a diagram showing an example of a display screen in the automatic driving mode. [Figure 12A] It is a diagram schematically showing an example of an image of the front of the work vehicle displayed on the screen. [Figure 12B] It is a diagram schematically showing an example of a composite image in which the trajectory of the working machine in the traveling direction of the work vehicle is superimposed on the image. [Figure 13] It is a diagram schematically showing the positional relationship between the reference point in the local coordinate system of the work vehicle and the positions of both end portions of the working machine. [Figure 14] It is a diagram schematically showing an example of a composite image in which the trajectory of the working machine and the trajectory of a pair of rear wheels are superimposed on the image. [Figure 15] It is a diagram schematically showing an example of a composite image in which the trajectory of the working machine and the target line along the target path are superimposed on the image. [Figure 16] It is a schematic diagram for explaining the deviation amount between the predicted trajectory of the working machine and the reference trajectory. [Figure 17] It is a diagram schematically showing an example of a composite image including a warning display that can be displayed on the screen when the deviation amount is greater than or equal to the threshold value. [Figure 18] It is a diagram schematically showing the positional relationship between the reference point in the local coordinate system of the work vehicle and the position of one side portion of the offset working machine. [[ID=2nd]] [Figure 19] It is a diagram schematically showing an example of a composite image in which the trajectory of the offset working machine is superimposed on the image. [Figure 20] It is a diagram schematically showing an example of an image in which a viaduct intersecting the road vertically is reflected in the traveling direction of the work vehicle traveling on a road outside the field. [Figure 21] It is a schematic diagram showing an example of the configuration of the HUD unit. [Figure 22] It is a diagram schematically showing an example of a composite image in which the working trace after the ground operation in the traveling direction is superimposed on the image. [Figure 23] It is a diagram schematically showing another example of a composite image in which the working trace after the ground operation in the traveling direction is superimposed on the image. [Figure 24]This diagram schematically shows an example of a composite image in which the work traces after ground work and the target line are superimposed on the video. [Figure 25] This diagram schematically illustrates an example of a composite video that includes guidance displays to direct users in accordance with the guidelines. [Figure 26] This diagram schematically illustrates an example of a composite video that includes a warning message indicating that the work prediction line does not conform to the guidelines. [Figure 27] This diagram schematically illustrates an example of a composite video that includes guidance displays prompting the user to change their target route. [Figure 28] This diagram schematically shows an example of a composite image that includes the display of a turning point located in the direction of travel of a work vehicle. [Modes for carrying out the invention]

[0013] (Definition of terms) In this disclosure, “agricultural machinery” means machinery used for agricultural purposes. Examples of agricultural machinery include tractors, harvesters, rice transplanters, riding cultivators, vegetable transplanters, mowers, seeders, fertilizer spreaders, agricultural drones (i.e., unmanned aerial vehicles: UAVs), and agricultural mobile robots. Not only may a work vehicle such as a tractor function as “agricultural machinery” on its own, but the entire work vehicle, including implements attached to or towed by the work vehicle, may function as a single “agricultural machinery.” Agricultural machinery performs agricultural work on the ground in a field, such as tilling, sowing, pest control, fertilizing, planting crops, or harvesting. These agricultural work may be referred to as “ground work” or simply “work.” When a vehicle-type agricultural machine moves while performing agricultural work, it may be referred to as “working while moving.”

[0014] "Autonomous driving" means controlling the movement of agricultural machinery through the operation of a control device, without manual operation by a driver. Agricultural machinery that performs autonomous driving is sometimes called "autonomous farm machinery" or "robot farm machinery." During autonomous driving, not only the movement of the agricultural machinery but also the actions of agricultural work (e.g., the operation of implements) may be controlled automatically. If the agricultural machinery is a vehicle-type machine, the movement of the agricultural machinery by autonomous driving is called "autonomous driving." The control device may control at least one of the following necessary actions for the movement of the agricultural machinery: steering, adjustment of movement speed, starting and stopping of movement. When controlling a work vehicle equipped with implements, the control device may also control actions such as raising and lowering the implements, and starting and stopping the operation of the implements. Movement by autonomous driving may include not only movement of agricultural machinery along a predetermined path toward a destination but also movement following a target. Agricultural machinery that performs autonomous driving may move partially based on user instructions. In addition to the autonomous driving mode, agricultural machinery that performs autonomous driving may also operate in a manual driving mode in which it is moved by manual operation by a driver. The steering of agricultural machinery by a control device, without manual intervention, is called "automatic steering." Part or all of the control device may be located outside the agricultural machinery. Communication, such as control signals, commands, or data, may take place between the external control device and the agricultural machinery. An autonomously operating agricultural machine may move autonomously while sensing its surroundings, without human intervention in controlling its movement. An autonomously moving agricultural machine can travel unmanned within or outside the field (e.g., on a road). Obstacle detection and obstacle avoidance maneuvers may be performed during autonomous movement.

[0015] "Remote control" or "remote operation" refers to the operation of agricultural machinery using a remote control device. Remote control can be performed by an operator located away from the agricultural machinery (e.g., a system administrator or user of the agricultural machinery). "Remote-controlled driving" refers to the agricultural machinery driving in response to signals transmitted from a remote control device. A remote control device may include devices with signal transmission capabilities, such as a personal computer (PC), laptop computer, tablet computer, smartphone, or remote controller (remote control). By operating the remote control device, the operator can give commands to the agricultural machinery, such as starting, stopping, accelerating, decelerating, or changing direction of travel. The mode in which the control device controls the driving of the agricultural machinery in response to these commands is called "remote control mode."

[0016] A "remote device" is a device with communication capabilities located away from the agricultural machinery. For example, a remote device could be a remote control device used by an operator to remotely control the agricultural machinery. The remote device may include or be connected to a display device. The display device can, for example, display an image (or video) that visualizes the surrounding environment of the agricultural machinery, based on sensor data (also called "sensing data") output from sensing devices such as cameras or LiDAR sensors installed on the agricultural machinery. The operator can understand the surrounding environment of the agricultural machinery by viewing the displayed image and, if necessary, operate the remote control device to remotely control the agricultural machinery.

[0017] A "work plan" is data that outlines the schedule for one or more agricultural tasks performed by agricultural machinery. A work plan may include, for example, information indicating the sequence of agricultural tasks performed by the agricultural machinery and the field in which each task will be performed. A work plan may also include information on the scheduled date and time for each task. A work plan containing information on the scheduled date and time for each task is specifically referred to as a "work schedule" or simply a "schedule." A work schedule may include information on the scheduled start and / or end times for each agricultural task performed on each workday. For each agricultural task, a work plan or work schedule may include information such as the nature of the task, the machinery used, and / or the type and quantity of agricultural materials used. Here, "agricultural materials" means the materials used in agricultural tasks performed by agricultural machinery. Agricultural materials are sometimes simply referred to as "materials." Agricultural materials may include, for example, materials consumed by agricultural tasks, such as pesticides, fertilizers, seeds, or seedlings. A work plan may be created by a processing device that communicates with agricultural machinery to manage agricultural tasks, or by a processing device installed on agricultural machinery. The processing device can, for example, create a work plan based on information entered by a user (such as an agricultural manager or farm worker) by operating a terminal device. In this specification, a processing device that communicates with agricultural machinery to manage agricultural work is referred to as a "management device." The management device may manage the agricultural work of multiple agricultural machines. In this case, the management device may create a work plan that includes information about each agricultural work performed by each of the multiple agricultural machines. The work plan may be downloaded by each agricultural machine and stored in a storage device. Each agricultural machine can automatically go to the field and perform the agricultural work in accordance with the work plan to carry out the scheduled agricultural work.

[0018] An "environmental map" is data that represents the location or area of ​​objects in the environment in which agricultural machinery operates, using a predetermined coordinate system. Environmental maps are sometimes simply referred to as "maps" or "map data." The coordinate system that defines an environmental map may be a world coordinate system, such as a geographic coordinate system fixed to the Earth. Environmental maps may include information other than location about objects in the environment (e.g., attribute information or other information). Environmental maps include various forms of maps, such as point cloud maps or grid maps. Data of local maps or submaps generated or processed in the process of constructing an environmental map is also called a "map" or "map data."

[0019] "Agricultural roads" refer to roads primarily used for agricultural purposes. Agricultural roads are not limited to asphalt-paved roads, but also include unpaved roads covered with soil or gravel. Agricultural roads include roads exclusively passable by agricultural machinery (such as tractors and other work vehicles) (including private roads) and roads that can also be used by general vehicles (passenger cars, trucks, buses, etc.). Work vehicles may travel automatically on general roads in addition to agricultural roads. General roads are roads that are maintained for the traffic of general vehicles.

[0020] (Embodiment) Embodiments of the present disclosure are described below. However, descriptions that are unnecessarily detailed may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid the following description becoming unnecessarily verbose and to facilitate understanding for those skilled in the art. The inventors provide the accompanying drawings and the following description so that those skilled in the art can fully understand the present disclosure, and not to limit the subject matter described in the claims. In the following description, components having the same or similar function are denoted by the same reference numerals.

[0021] The following embodiments are illustrative, and the technology of this disclosure is not limited to these embodiments. For example, the numerical values, shapes, materials, steps, order of steps, and display screen layouts shown in the following embodiments are merely examples, and various modifications are possible as long as they do not create a technical inconsistency. Furthermore, it is possible to combine one embodiment with another as long as it does not create a technical inconsistency.

[0022] The following primarily describes embodiments in which the technology of this disclosure is applied to work vehicles such as tractors, which are an example of agricultural machinery. The technology of this disclosure can be applied not only to tractors but also to other types of agricultural machinery (e.g., rice transplanters, combine harvesters, harvesters, riding cultivators, vegetable transplanters, lawnmowers, seeders, fertilizer spreaders, agricultural drones, and agricultural mobile robots), and is particularly suitable for agricultural machinery capable of remote operation. As an example, the following describes an embodiment in which a driving control system for realizing automatic driving and remote operation functions is mounted on a work vehicle. However, the technology of this disclosure does not necessarily require automatic driving or remote operation functions. At least some functions of the driving control system may be implemented in other devices that communicate with the work vehicle (e.g., remote control devices or servers).

[0023] Figure 1A is a diagram illustrating an overview of an exemplary agricultural management system according to the present disclosure. The agricultural management system shown in Figure 1A comprises a work vehicle 100, a remote device 400, and a management device 600. The remote device 400 is a computer used by a user to remotely monitor the work vehicle 100. The management device 600 is a computer managed by the operator of the agricultural management system. The work vehicle 100, the remote device 400, and the management device 600 can communicate with each other via a network 80. Although Figure 1A illustrates one work vehicle 100, the agricultural management system may include multiple work vehicles or other agricultural machinery. The agricultural management system in the embodiments of the present disclosure includes a remote control system for the work vehicle 100. The remote control system includes a sensing device, a communication device, and a control device in the work vehicle 100, and the remote device 400. The portion of the entire remote control system in the embodiments of the present disclosure that includes the sensing device and the communication device in the work vehicle 100 may be referred to as the “sensing system.” In other words, the sensing system is part of the remote control system.

[0024] In the embodiments of this disclosure, the work vehicle 100 is a tractor. The work vehicle 100 can be fitted with implements on either the rear or the front, or both. The work vehicle 100 can travel in a field while performing agricultural work according to the type of implement.

[0025] The work vehicle 100 in the embodiments of this disclosure is equipped with an automatic driving function. That is, the work vehicle 100 can be driven by the operation of a control device without manual operation. The control device in the embodiments of this disclosure is installed inside the work vehicle 100 and can control both the speed and steering of the work vehicle 100. The work vehicle 100 can automatically drive not only within the field but also outside the field (for example, on a road). The mode in which the control device causes the work vehicle 100 to drive automatically is referred to as the "automatic driving mode".

[0026] The work vehicle 100 is also equipped with a remote control function. The control device can change the travel speed and direction of the work vehicle 100 by controlling the travel device in response to remote operation by the user using the remote device 400. The work vehicle 100 can perform remote control operation not only within the field but also outside the field. The mode in which the control device remotely controls the work vehicle 100 is called "remote control mode".

[0027] The work vehicle 100 is equipped with devices used for positioning or self-position estimation, such as a GNSS receiver and a LiDAR sensor. In automatic driving mode, the control device of the work vehicle 100 automatically drives the work vehicle 100 based on the position of the work vehicle 100 and information on the target route generated by the management device 600. In addition to controlling the driving of the work vehicle 100, the control device also controls the operation of the implement. This allows the work vehicle 100 to perform agricultural work using the implement while automatically driving within the field. Furthermore, the work vehicle 100 can automatically drive along a target route on a road outside the field (e.g., a farm road or public road). When the work vehicle 100 is automatically driving along a road outside the field, it drives while generating a localized path that can avoid obstacles along the target route based on data output from sensing devices such as a camera or a LiDAR sensor. The work vehicle 100 may, within the field, travel while generating a local path as described above, or it may travel along a target path without generating a local path and stop when an obstacle is detected.

[0028] The management device 600 is a computer that manages agricultural work performed by the work vehicle 100. The management device 600 may be a server computer that centrally manages field-related information on the cloud and supports agriculture by utilizing the data on the cloud. For example, the management device 600 can create a work plan for the work vehicle 100 and generate a target route for the work vehicle 100 according to that work plan. Alternatively, the management device 600 may generate a target route for the work vehicle 100 in response to operations performed by the user using the remote device 400.

[0029] The remote device 400 is a computer used by a user located away from the work vehicle 100. The remote device 400 shown in Figure 1A is a laptop computer, but is not limited to this. The remote device 400 may be a stationary computer such as a desktop PC (personal computer), or a mobile device such as a smartphone or tablet computer.

[0030] The remote device 400 can be used to remotely monitor or remotely control the work vehicle 100. For example, the remote device 400 can display images captured by one or more cameras on the work vehicle 100 on a display. The user can view the images to check the situation around the work vehicle 100 and send instructions to the work vehicle 100, such as stopping, starting, accelerating, decelerating, or changing direction.

[0031] Figure 1B shows another example of an agricultural management system. The agricultural management system shown in Figure 1B includes multiple work vehicles 100. Three work vehicles 100 are illustrated in Figure 1B, but the number of work vehicles 100 is arbitrary. Other agricultural machinery (e.g., agricultural drones) may also be included in the system. The remote device 400 in the example in Figure 1B is not a household terminal device, but a computer located in a remote monitoring center for agricultural machinery. The remote device 400 may be connected to a remote control 500 and one or more displays 430 used by an operator at the remote monitoring center. Five displays 430 are illustrated in Figure 1B, but the number of displays 430 is arbitrary. The remote control 500 may include various devices for remotely controlling the work vehicles 100 (e.g., steering wheel, accelerator pedal, left and right brake pedals, clutch pedal, and various switches or levers). The remote control device 500 shown in Figure 1B is a device that mimics the operating equipment used for manually driving the work vehicle 100, but the remote control device 500 is not limited to such a device. For example, remote control may be performed by a controller such as a joystick. Each display 430 can display, for example, an environmental map of the area including the field where the work vehicle 100 performs agricultural work, and images (e.g., moving images) taken by one or more cameras mounted on the work vehicle 100. The operator can understand the situation around the work vehicle 100 by looking at the images displayed on the display 430. Depending on the situation around each work vehicle 100, the operator can switch between automatic driving mode and remote control mode, or remotely control each agricultural machine.

[0032] The configuration and operation of the system in the embodiments of this disclosure will be described in more detail below.

[0033] [1. Structure] Figure 2 is a schematic side view showing an example of a work vehicle 100 and a work machine 300 connected to the work vehicle 100. In the embodiments of this disclosure, the work vehicle 100 can operate in both manual and automatic driving modes. In automatic driving mode, the work vehicle 100 can travel unmanned. The work vehicle 100 can be operated automatically both inside and outside the field. In automatic driving mode, the control device can operate in an automatic driving mode that drives the work vehicle 100 along a preset target path, and a remote operation mode that drives the work vehicle 100 in response to operations by a user using a remote device 400. The user may be, for example, the user of the work vehicle 100 or an operator at a remote monitoring center. Switching between automatic driving mode and remote operation mode can be performed by the user performing a predetermined operation using the remote device 400. For example, in automatic driving mode, if the user performs an operation using the remote device 400 to instruct the start of remote operation, the system will switch to remote operation mode. Furthermore, in remote control mode, if the user uses the remote device 400 to instruct the system to start automatic driving, the system will switch to automatic driving mode.

[0034] As shown in Figure 2, the work vehicle 100 comprises a vehicle body 101, a prime mover (engine) 102, and a transmission 103. The vehicle body 101 is provided with a running gear including wheels 104 with tires, and a cabin 105. The running gear includes four wheels 104, axles that rotate the four wheels, and brakes that brake each axle. The wheels 104 include a pair of front wheels 104F and a pair of rear wheels 104R. Inside the cabin 105 are a driver's seat 107, a steering gear 106, an operating terminal 200, and a group of switches for operation. When the work vehicle 100 is performing work in a field, one or both of the front wheels 104F and the rear wheels 104R may be replaced with multiple wheels (crawlers) fitted with tracks instead of wheels with tires.

[0035] The work vehicle 100 can switch between a four-wheel drive (4W) mode in which all of the front wheels 104F and rear wheels 104R are driven wheels, and a two-wheel drive (2W) mode in which either the front wheels 104F or the rear wheels 104R are driven wheels. The work vehicle 100 can also switch between a state where the left and right brakes are connected and a state where they are disconnected. By disconnecting the left and right brakes, the left and right wheels 104 can be braked independently. This makes it possible to make turns with a small turning radius.

[0036] The work vehicle 100 is equipped with multiple sensing devices that sense the area around the work vehicle 100. In the example shown in Figure 2, the sensing device includes multiple cameras 120, a LiDAR sensor 140, and multiple obstacle sensors 130. The sensing device may include only some of the cameras 120, LiDAR sensors 140, and obstacle sensors 130. The sensing device senses the environment around the vehicle body and outputs sensing data.

[0037] Cameras 120 may be installed, for example, on the front, rear, left, and right sides of the work vehicle 100. Cameras 120 capture images of the environment surrounding the work vehicle 100 and generate image data. In this specification, the image data generated by camera 120 may be simply referred to as "images." The act of capturing images and generating them may be expressed as "acquiring images." Images acquired by camera 120 may be transmitted to a remote device 400 for remote monitoring. These images may be used to monitor the work vehicle 100 when it is operating unmanned. Cameras 120 may also be used to generate images for recognizing surrounding features or obstacles, white lines, signs, or markings when the work vehicle 100 is traveling on a road outside the field (agricultural road or public road).

[0038] In the example shown in Figure 2, the LiDAR sensor 140 is located at the lower front of the vehicle body 101. The LiDAR sensor 140 may be located in other positions. While the work vehicle 100 is mainly traveling outside the field, the LiDAR sensor 140 repeatedly outputs sensor data indicating the distance and direction to each measurement point of objects in the surrounding environment, or the 2D or 3D coordinate values ​​of each measurement point. The sensor data output from the LiDAR sensor 140 is processed by the control device of the work vehicle 100. The control device can perform self-position estimation of the work vehicle 100 by matching the sensor data with an environmental map. Furthermore, based on the sensor data, the control device can detect objects such as obstacles in the vicinity of the work vehicle 100 and generate a local path that the work vehicle 100 should actually take, along the target path (also called a global path). The control device can also generate or edit the environmental map using algorithms such as SLAM (Simultaneous Localization and Mapping). The work vehicle 100 may be equipped with multiple LiDAR sensors positioned at different locations and in different orientations.

[0039] The multiple obstacle sensors 130 shown in Figure 2 are located at the front and rear of the cabin 105. Obstacle sensors 130 may also be located in other areas. For example, one or more obstacle sensors 130 may be provided at any location on the sides, front, and rear of the vehicle body 101. Obstacle sensors 130 may include, for example, laser scanners or ultrasonic sonar. Obstacle sensors 130 are used to detect surrounding obstacles during autonomous driving and to stop or bypass the work vehicle 100. A LiDAR sensor 140 may be used as one of the obstacle sensors 130.

[0040] The work vehicle 100 further comprises a GNSS unit 110. The GNSS unit 110 includes a GNSS receiver. The GNSS receiver may include an antenna that receives signals from GNSS satellites and a processor that calculates the position of the work vehicle 100 based on the signals received by the antenna. The GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and performs positioning based on the satellite signals. In embodiments of this disclosure, the GNSS unit 110 is located on top of the cabin 105, but it may be located in other locations.

[0041] The GNSS unit 110 may include an inertial measuring unit (IMU). Signals from the IMU can be used to supplement positional data. The IMU can measure the tilt and minute movements of the work vehicle 100. By using the data acquired by the IMU to supplement positional data based on satellite signals, positioning performance can be improved.

[0042] The control device of the work vehicle 100 may use sensing data acquired by sensing devices such as a camera 120 or LiDAR sensor 140 for positioning, in addition to the positioning results from the GNSS unit 110. If there are features that function as characteristic points in the environment in which the work vehicle 100 travels, such as farm roads, forest roads, public roads, or orchards, the position and orientation of the work vehicle 100 can be estimated with high accuracy based on the data acquired by the camera 120 or LiDAR sensor 140 and an environmental map stored in a storage device in advance. By correcting or supplementing the position data based on satellite signals using the data acquired by the camera 120 or LiDAR sensor 140, the position of the work vehicle 100 can be determined with even higher accuracy.

[0043] The prime mover 102 may be, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 103 can change the propulsion force and travel speed of the work vehicle 100 by shifting gears. The transmission 103 can also switch the work vehicle 100 between forward and reverse.

[0044] The steering system 106 includes a steering wheel, a steering shaft connected to the steering wheel, and a power steering system that assists steering by the steering wheel. The front wheels 104F are steering wheels, and the direction of travel of the work vehicle 100 can be changed by changing their steering angle (also referred to as the "steering angle"). The steering angle of the front wheels 104F can be changed by operating the steering wheel. The power steering system includes a hydraulic system or electric motor that supplies auxiliary force to change the steering angle of the front wheels 104F. When automatic steering is performed, the steering angle is automatically adjusted by the force of the hydraulic system or electric motor under control from a control device located inside the work vehicle 100.

[0045] A coupling device 108 is provided at the rear of the vehicle body 101. The coupling device 108 includes, for example, a three-point support device (also referred to as a "three-point link" or "three-point hitch"), a PTO (Power Take Off) shaft, a universal joint, and a communication cable. The coupling device 108 allows the work implement 300 to be attached to and detached from the work vehicle 100. The coupling device 108 can change the position or orientation of the work implement 300 by raising and lowering the three-point link, for example, by a hydraulic system. Power can also be supplied from the work vehicle 100 to the work implement 300 via the universal joint. The work vehicle 100 can pull the work implement 300 and cause the work implement 300 to perform a predetermined operation. The coupling device may be provided at the front of the vehicle body 101. In that case, the work implement can be connected to the front of the work vehicle 100.

[0046] The implement 300 shown in Figure 2 is a rotary tiller, but the implement 300 is not limited to a rotary tiller. For example, any implement such as a seeder, spreader, transplanter, mower, rake, baler, harvester, spreader, or harrow can be connected to the work vehicle 100 and used.

[0047] The work vehicle 100 shown in Figure 2 is capable of being operated by a person, but may also be designed for unmanned operation only. In that case, components necessary only for manned operation, such as the cabin 105, steering system 106, and driver's seat 107, do not need to be provided on the work vehicle 100. The unmanned work vehicle 100 can be driven autonomously or by remote control by a user.

[0048] Figure 3 is a block diagram showing an example configuration of a work vehicle 100 and a work machine 300. The work vehicle 100 and the work machine 300 can communicate with each other via a communication cable included in the coupling device 108. The work vehicle 100 can communicate with the remote device 400 and the management device 600 via the network 80.

[0049] In the example shown in Figure 3, the work vehicle 100 includes a GNSS unit 110, a sensing device 250 (camera 120, obstacle sensor 130, LiDAR sensor 140), and an operating terminal 200, as well as a sensor group 150 for detecting the operating status of the work vehicle 100, a driving control system 160, a communication device 190, an operating switch group 210, a buzzer 220, and a drive unit 240. These components are connected to each other via a bus for communication. The GNSS unit 110 includes a GNSS receiver 111, an RTK receiver 112, an inertial measurement unit (IMU) 115, and a processing circuit 116. The sensor group 150 includes a steering wheel sensor 152, a steering angle sensor 154, and an axle sensor 156. The driving control system 160 includes a storage device 170 and a control device 180. The control device 180 includes a plurality of electronic control units (ECUs) 181 to 186. The work machine 300 includes a drive unit 340, a control device 380, and a communication device 390. Figure 3 shows components that are relatively highly related to the operation of the work vehicle 100's automatic driving and remote control, and other components are not shown.

[0050] The GNSS receiver 111 in the GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated in a predetermined format, such as NMEA-0183 format. The GNSS data may include, for example, the identification number, elevation angle, azimuth angle, and received signal strength of each satellite from which the satellite signal was received.

[0051] The GNSS unit 110 shown in Figure 3 uses RTK (Real Time Kinematic)-GNSS to position the work vehicle 100. Figure 4 is a conceptual diagram showing an example of a work vehicle 100 performing positioning using RTK-GNSS. In RTK-GNSS positioning, in addition to satellite signals transmitted from multiple GNSS satellites 50, a correction signal transmitted from a base station 60A is used. The base station 60A may be installed near the field where the work vehicle 100 is performing its work (for example, within 10 km of the work vehicle 100). Based on the satellite signals received from multiple GNSS satellites 50, the base station 60A generates a correction signal, for example, in RTCM format and transmits it to the GNSS unit 110. The RTK receiver 112 includes an antenna and a modem and receives the correction signal transmitted from the base station 60A. The processing circuit 116 of the GNSS unit 110 corrects the positioning result from the GNSS receiver 111 based on the correction signal. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position information, including latitude, longitude, and altitude, is acquired by high-precision positioning using RTK-GNSS. The GNSS unit 110 calculates the position of the work vehicle 100 at a frequency of, for example, 1 to 10 times per second.

[0052] Furthermore, the positioning method is not limited to RTK-GNSS; any positioning method that can obtain the necessary accuracy of positional information (such as interferometric positioning or relative positioning) can be used. For example, positioning using VRS (Virtual Reference Station) or DGPS (Differential Global Positioning System) may be performed. If the necessary accuracy of positional information can be obtained without using the correction signal transmitted from the base station 60A, the positional information may be generated without using the correction signal. In that case, the GNSS unit 110 does not need to be equipped with an RTK receiver 112.

[0053] Even when using RTK-GNSS, in locations where correction signals from the base station 60A cannot be obtained (for example, on a road far from the field), the position of the work vehicle 100 is estimated by other means, without relying on signals from the RTK receiver 112. For example, the position of the work vehicle 100 can be estimated by matching data output from the LiDAR sensor 140 and / or camera 120 with a high-precision environmental map.

[0054] The GNSS unit 110 in the embodiments of this disclosure further comprises an IMU 115. The IMU 115 may include a 3-axis accelerometer and a 3-axis gyroscope. The IMU 115 may also include an orientation sensor, such as a 3-axis geomagnetic sensor. The IMU 115 functions as a motion sensor and can output signals indicating various quantities such as acceleration, velocity, displacement, and attitude of the work vehicle 100. The processing circuit 116 can estimate the position and orientation of the work vehicle 100 with higher accuracy based on the signals output from the IMU 115 in addition to the satellite signals and correction signals. The signals output from the IMU 115 can be used to correct or complement the position calculated based on the satellite signals and correction signals. The IMU 115 outputs signals at a higher frequency than the GNSS receiver 111. Using these high-frequency signals, the processing circuit 116 can measure the position and orientation of the work vehicle 100 at a higher frequency (e.g., 10 Hz or higher). Instead of the IMU 115, a 3-axis accelerometer and a 3-axis gyroscope may be provided separately. The IMU115 may be provided as a separate device from the GNSS unit 110.

[0055] Camera 120 is an imaging device that captures the environment around the work vehicle 100. Camera 120 includes an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 120 may also include an optical system including one or more lenses and a signal processing circuit. While the work vehicle 100 is in motion, Camera 120 captures the environment around the work vehicle 100 and generates image (e.g., video) data. Camera 120 can capture video at a frame rate of, for example, 3 frames per second (fps) or higher. The images generated by Camera 120 can be used, for example, when a remote observer uses a remote device 400 to check the environment around the work vehicle 100. The images generated by Camera 120 may also be used for positioning or obstacle detection. As shown in Figure 2, multiple cameras 120 may be installed at different locations on the work vehicle 100, or a single camera may be installed. A visible light camera that generates visible light images and an infrared camera that generates infrared images may be provided separately. Both the visible light camera and the infrared camera may be provided as cameras that generate surveillance images. The infrared camera can also be used for detecting obstacles at night.

[0056] The obstacle sensor 130 detects objects present around the work vehicle 100. The obstacle sensor 130 may include, for example, a laser scanner or an ultrasonic sonar. The obstacle sensor 130 outputs a signal indicating the presence of an obstacle when an object is closer than a predetermined distance from the obstacle sensor 130. Multiple obstacle sensors 130 may be installed at different locations on the work vehicle 100. For example, multiple laser scanners and multiple ultrasonic sonars may be placed at different locations on the work vehicle 100. By providing many such obstacle sensors 130, blind spots in monitoring obstacles around the work vehicle 100 can be reduced.

[0057] The steering wheel sensor 152 measures the rotation angle of the steering wheel of the work vehicle 100. The steering angle sensor 154 measures the steering angle of the front wheels 104F, which are the steering wheels. The values ​​measured by the steering wheel sensor 152 and the steering angle sensor 154 are used for steering control by the control device 180.

[0058] The axle sensor 156 measures the rotational speed of the axle connected to the wheel 104, i.e., the number of rotations per unit time. The axle sensor 156 may be a sensor that utilizes, for example, a magnetoresistive element (MR), a Hall element, or an electromagnetic pickup. The axle sensor 156 outputs a numerical value indicating, for example, the number of rotations of the axle per minute (in rpm). The axle sensor 156 is used to measure the speed of the work vehicle 100.

[0059] The drive system 240 includes various devices necessary for the movement of the work vehicle 100 and the driving of the work equipment 300, such as the prime mover 102, the transmission 103, the steering system 106, and the coupling device 108. The prime mover 102 may be an internal combustion engine, such as a diesel engine. The drive system 240 may also be equipped with an electric motor for traction, either in place of or in conjunction with the internal combustion engine.

[0060] The buzzer 220 is an audio output device that emits a warning sound to notify of an abnormality. For example, the buzzer 220 emits a warning sound when an obstacle is detected during autonomous driving. The buzzer 220 is controlled by the control device 180.

[0061] The storage device 170 includes one or more storage media such as flash memory or magnetic disks. The storage device 170 stores various data generated by the GNSS unit 110, camera 120, obstacle sensor 130, LiDAR sensor 140, sensor group 150, and control device 180. The data stored in the storage device 170 may include map data of the environment in which the work vehicle 100 travels (environmental map) and data of a global route for autonomous driving (target route). The environmental map includes information on multiple fields where the work vehicle 100 performs agricultural work and the roads around them. The environmental map and target route may be generated by a processing unit (i.e., processor) in the management device 600. In this embodiment of the disclosure, the control device 180 may have a function to generate or edit the environmental map and target route. The control device 180 can edit the environmental map and target route acquired from the management device 600 according to the driving environment of the work vehicle 100.

[0062] The storage device 170 also stores work plan data received by the communication device 190 from the management device 600. The work plan includes information about multiple farming tasks to be performed by the work vehicle 100 over multiple work days. The work plan may be work schedule data, for example, including information about the scheduled time for each farming task to be performed by the work vehicle 100 on each work day. The storage device 170 also stores computer programs that cause each ECU in the control device 180 to perform various operations described later. Such computer programs may be provided to the work vehicle 100 via a storage medium (e.g., semiconductor memory or optical disc) or a telecommunications line (e.g., the Internet). Such computer programs may be sold as commercial software.

[0063] The control device 180 includes a plurality of ECUs. These plurality of ECUs include, for example, an ECU 181 for speed control, an ECU 182 for steering control, an ECU 183 for implement control, an ECU 184 for autonomous driving control, an ECU 185 for route generation, and an ECU 186 for map generation.

[0064] The ECU 181 controls the speed of the work vehicle 100 by controlling the prime mover 102, the transmission 103, and the brakes, which are included in the drive unit 240.

[0065] The ECU 182 controls the steering of the work vehicle 100 by controlling the hydraulic system or electric motor included in the steering device 106 based on the measurements of the steering wheel sensor 152.

[0066] The ECU 183 controls the movement of the three-point linkage and PTO shaft, etc., included in the coupling device 108, in order to make the work implement 300 perform the desired operation. The ECU 183 also generates signals to control the operation of the work implement 300 and transmits these signals from the communication device 190 to the work implement 300.

[0067] The ECU 184 performs calculations and controls to achieve autonomous driving based on data output from the GNSS unit 110, camera 120, obstacle sensor 130, LiDAR sensor 140, and sensor group 150. For example, the ECU 184 determines the position of the work vehicle 100 based on data output from at least one of the GNSS unit 110, camera 120, and LiDAR sensor 140. Within the field, the ECU 184 may determine the position of the work vehicle 100 based only on data output from the GNSS unit 110. The ECU 184 may estimate or correct the position of the work vehicle 100 based on data acquired by the camera 120 or LiDAR sensor 140. By utilizing data acquired by the camera 120 or LiDAR sensor 140, the accuracy of positioning can be further improved. Outside the field, the ECU 184 estimates the position of the work vehicle 100 using data output from the LiDAR sensor 140 or camera 120. For example, ECU 184 may estimate the position of the work vehicle 100 by matching data output from the LiDAR sensor 140 or camera 120 with an environmental map. During autonomous driving, ECU 184 performs calculations necessary for the work vehicle 100 to travel along a target path or local path based on the estimated position of the work vehicle 100. ECU 184 sends a command to ECU 181 to change speed and a command to ECU 182 to change steering angle. ECU 181 changes the speed of the work vehicle 100 by controlling the engine 102, transmission 103, or brakes in response to the command to change speed. ECU 182 changes the steering angle by controlling the steering device 106 in response to the command to change steering angle.

[0068] ECU184 also controls the remote operation of the work vehicle 100. In remote operation mode, ECU184 controls ECUs 181, 182, and 183 in response to signals received by the communication device 190 from the remote device 400. This allows the work vehicle 100 to perform operations such as speed control, steering control, raising and lowering of the work equipment 300, and turning the work equipment 300 on / off in response to remote commands from the user.

[0069] The ECU 185 sequentially generates local paths that can avoid obstacles while the work vehicle 100 is traveling along the target path. While the work vehicle 100 is traveling, the ECU 185 recognizes obstacles present around the work vehicle 100 based on data output from the camera 120, obstacle sensor 130, and LiDAR sensor 140. The ECU 185 generates local paths to avoid the recognized obstacles.

[0070] The ECU 185 may have a function to perform global route planning on behalf of the management device 600. In that case, the ECU 185 may determine the destination of the work vehicle 100 based on the work plan stored in the storage device 170 and determine the target route from the starting point to the destination point of the work vehicle 100's movement. Based on the environmental map containing road information stored in the storage device 170, the ECU 185 can create a target route that, for example, can reach the destination in the shortest time. Alternatively, the ECU 185 may generate a target route that prioritizes a specific type of road (for example, a road along a specific feature such as a farm road or waterway, or a road that can receive satellite signals from a GNSS satellite well) based on the attribute information of each road included in the environmental map.

[0071] The ECU 186 generates or edits a map of the environment in which the work vehicle 100 travels. In embodiments of this disclosure, an environment map generated by an external device such as a management device 600 is transmitted to the work vehicle 100 and recorded in the storage device 170, but the ECU 186 can also generate or edit the environment map instead. The operation when the ECU 186 generates an environment map is described below. The environment map may be generated based on sensor data output from the LiDAR sensor 140. When generating an environment map, the ECU 186 sequentially generates three-dimensional point cloud data based on the sensor data output from the LiDAR sensor 140 while the work vehicle 100 is traveling. The ECU 186 can generate an environment map by stitching together the sequentially generated point cloud data using an algorithm such as SLAM. The environment map thus generated is a high-precision three-dimensional map and can be used for self-localization by the ECU 184. Based on this three-dimensional map, a two-dimensional map can be generated for use in global path planning. In this specification, both the 3D map used for self-localization and the 2D map used for global path planning are referred to as "environmental maps." The ECU186 can also edit the map by adding various attribute information to the map, such as features recognized based on data output from the camera 120 or LiDAR sensor 140 (e.g., waterways, rivers, grass, trees, etc.), road types (e.g., whether or not it is a farm road), road surface conditions, or road passability.

[0072] These ECUs enable the control unit 180 to perform both automated and remotely controlled driving. During automated driving, the control unit 180 controls the drive unit 240 based on the measured or estimated position of the work vehicle 100 and the generated path. This allows the control unit 180 to drive the work vehicle 100 along a target path. During remotely controlled driving, the control unit 180 controls the movement of the work vehicle 100 based on signals transmitted from the remote device 400. In other words, the control unit 180 controls the drive unit 240 in response to user operations using the remote device 400. This allows the control unit 180 to drive the work vehicle 100 according to user instructions.

[0073] Multiple ECUs included in the control unit 180 can communicate with each other according to a vehicle bus standard such as CAN (Controller Area Network). Instead of CAN, a faster communication method such as Automotive Ethernet (registered trademark) may be used. In Figure 3, each of the ECUs 181 to 186 is shown as a separate block, but each of their functions may be implemented by multiple ECUs. An on-board computer integrating at least some of the functions of ECUs 181 to 186 may be provided. The control unit 180 may also include ECUs other than ECUs 181 to 186, and any number of ECUs can be provided depending on their function. Each ECU includes a processing circuit containing one or more processors.

[0074] The communication device 190 is a device that includes circuits for communicating with the work machine 300, the remote device 400, and the management device 600. The communication device 190 transmits sensing data output from the sensing device 250 to the remote device 400. The communication device 190 includes circuits for transmitting and receiving signals compliant with ISOBUS standards, such as ISOBUS-TIM, to and from the communication device 390 of the work machine 300. This allows the work machine 300 to perform desired operations or to acquire information from the work machine 300. The communication device 190 may further include antennas and communication circuits for transmitting and receiving signals via the network 80 to and from the respective communication devices of the remote device 400 and the management device 600. The network 80 may include, for example, a cellular mobile communication network such as 3G, 4G, or 5G and the internet. The communication device 190 may also have a function to communicate with a mobile terminal used by a monitor located near the work vehicle 100. Communication with such mobile devices may take place using any wireless communication standard, such as Wi-Fi®, 3G, 4G, or 5G cellular mobile communication, or Bluetooth®.

[0075] The operation terminal 200 is a terminal for the user to perform operations related to the movement of the work vehicle 100 and the operation of the work machine 300, and is also called a virtual terminal (VT). The operation terminal 200 may be equipped with a display device such as a touchscreen and / or one or more buttons. The display device may be a display such as a liquid crystal or organic light-emitting diode (OLED). By operating the operation terminal 200, the user can perform various operations such as switching the automatic driving mode on / off, switching the remote control mode on / off, recording or editing the environmental map, setting a target route, and switching the work machine 300 on / off. At least some of these operations can also be achieved by operating the operation switch group 210. The operation terminal 200 may be configured to be detachable from the work vehicle 100. A user located away from the work vehicle 100 may control the operation of the work vehicle 100 by operating the detached operation terminal 200. The user may control the operation of the work vehicle 100 by operating a computer with the necessary application software installed, such as a remote device 400, instead of the operating terminal 200.

[0076] Figure 5 shows an example of an operating terminal 200 and a group of operating switches 210 provided inside the cabin 105. Inside the cabin 105 is a group of operating switches 210, which includes multiple switches that can be operated by the user. The group of operating switches 210 may include, for example, a switch for selecting the gear of the main or sub-transmission, a switch for switching between automatic driving mode and manual driving mode, a switch for switching between forward and reverse, a switch for switching between four-wheel drive and two-wheel drive, a switch for releasing the coupling of the left and right brakes, and a switch for raising and lowering the work equipment 300. Note that if the work vehicle 100 only performs unmanned operation and does not have a function for manned operation, the work vehicle 100 does not need to be equipped with a group of operating switches 210.

[0077] At least some of the operations that can be performed by the operation terminal 200 or the operation switch group 210 can also be performed by remote operation using the remote device 400. Any of the above operations can be performed by the user performing a predetermined operation on the screen displayed on the remote device 400's display.

[0078] The drive unit 340 in the work machine 300 shown in Figure 3 performs the operations necessary for the work machine 300 to perform a predetermined operation. The drive unit 340 includes devices such as a hydraulic system, an electric motor, or a pump, depending on the application of the work machine 300. The control device 380 controls the operation of the drive unit 340. The control device 380 causes the drive unit 340 to perform various operations in response to signals transmitted from the work vehicle 100 via the communication device 390. It can also transmit signals corresponding to the status of the work machine 300 from the communication device 390 to the work vehicle 100.

[0079] Next, the configuration of the management device 600 and the remote device 400 will be described with reference to Figure 6. Figure 6 is a block diagram illustrating the schematic hardware configuration of the management device 600 and the remote device 400.

[0080] The management device 600 comprises a storage device 650, a processor 660, a ROM (Read Only Memory) 670, a RAM (Random Access Memory) 680, and a communication device 690. These components are connected to each other via a bus for communication. The management device 600 manages the schedule of agricultural work performed by the work vehicle 100 in the field and can function as a cloud server to support agriculture by utilizing the managed data. Users can input the information necessary to create a work plan using the remote device 400 and upload that information to the management device 600 via the network 80. Based on this information, the management device 600 can create a schedule for agricultural work, i.e., a work plan. The management device 600 can also generate or edit environmental maps and perform global route planning for the work vehicle 100. The environmental maps may be distributed from a computer outside the management device 600.

[0081] The communication device 690 is a communication module for communicating with the work vehicle 100 and the remote device 400 via the network 80. The communication device 690 can perform wired communication compliant with communication standards such as IEEE 1394 (registered trademark) or Ethernet (registered trademark). The communication device 690 may also perform wireless communication compliant with Bluetooth (registered trademark) or Wi-Fi standards, or cellular mobile communication such as 3G, 4G, or 5G.

[0082] The processor 660 may be, for example, a semiconductor integrated circuit including a central processing unit (CPU). The processor 660 may be implemented by a microprocessor or microcontroller. Alternatively, the processor 660 may be implemented by an FPGA (Field Programmable Gate Array) equipped with a CPU, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an ASSP (Application Specific Standard Product), or a combination of two or more circuits selected from these circuits. The processor 660 sequentially executes a computer program stored in the ROM 670, which describes a set of instructions for performing at least one operation, to achieve the desired operation.

[0083] ROM670 can be, for example, writable memory (e.g., PROM), rewritable memory (e.g., flash memory), or read-only memory. ROM670 stores programs that control the operation of processor 660. ROM670 does not have to be a single storage medium; it may be a collection of multiple storage media. Some of these storage media may be removable memory.

[0084] RAM680 provides a workspace for temporarily unpacking the control program stored in ROM670 during boot-up. RAM680 does not need to be a single storage medium; it may be a collection of multiple storage media.

[0085] The storage device 650 primarily functions as database storage. The storage device 650 may be, for example, a magnetic storage device or a semiconductor storage device. An example of a magnetic storage device is a hard disk drive (HDD). An example of a semiconductor storage device is a solid-state drive (SSD). The storage device 650 may be a device independent of the management device 600. For example, the storage device 650 may be a storage device connected to the management device 600 via the network 80, such as cloud storage.

[0086] The remote device 400 shown in Figure 6 comprises an input device 420, a display device (display) 430, a storage device 450, a processor 460, a ROM 470, a RAM 480, and a communication device 490. These components are connected to each other via a bus so as to be able to communicate with one another. The input device 420 is a device for converting user instructions into data and inputting it into the computer. The input device 420 may be, for example, a keyboard, a mouse, or a touch panel. The display device 430 may be, for example, a liquid crystal display or an organic EL display. The descriptions of the processor 460, ROM 470, RAM 480, storage device 450, and communication device 490 are the same as those described in the hardware configuration example of the management device 600, and are therefore omitted here.

[0087] In the example in Figure 6, the remote device 400 is a computer with a built-in display and input device, as shown in Figure 1A. The remote device 400 may also be a computer connected to the remote control unit 500 and display 430 in a remote monitoring center, as illustrated in Figure 1B.

[0088] [2. Operation] Next, the operation of the work vehicle 100, the remote control device 400, and the management device 600 will be explained.

[0089] [2-1. Automatic Driving Operation] First, an example of the operation of the work vehicle 100 in automatic driving mode will be described. In this embodiment of the disclosure, the work vehicle 100 can drive automatically both inside and outside the field. Inside the field, the work vehicle 100 drives along a pre-set target route and drives the work machine 300 to perform predetermined agricultural work. If the work vehicle 100 detects an obstacle by the obstacle sensor 130 while driving inside the field, it stops driving and performs actions such as emitting a warning sound from the buzzer 220 and transmitting a warning signal to the remote device 400. Inside the field, the positioning of the work vehicle 100 is mainly based on data output from the GNSS unit 110. On the other hand, outside the field, the work vehicle 100 drives automatically along a target route set on an agricultural road or public road outside the field. While the work vehicle 100 is traveling outside the field, it performs local path planning based on data acquired by the camera 120 or LiDAR sensor 140. Outside the field, if the work vehicle 100 detects an obstacle, it will either avoid the obstacle or stop in place. Outside the field, the position of the work vehicle 100 is estimated based on positioning data output from the GNSS unit 110, as well as data output from the LiDAR sensor 140 or camera 120.

[0090] The operation of the work vehicle 100 when it is automatically traveling within the field will be explained below. The operation of the work vehicle 100 when it is automatically traveling outside the field will be described later.

[0091] Figure 7 schematically shows an example of a work vehicle 100 that automatically travels within a field along a target path. In this example, the field includes a work area 72 where the work vehicle 100 performs work using a work implement 300, and a headland 74 located near the outer edge of the field. The user can pre-set which areas of the field on the map correspond to the work area 72 or the headland 74. The target path in this example includes a plurality of parallel main paths P1 and a plurality of turning paths P2 connecting the plurality of main paths P1. The main paths P1 are located within the work area 72, and the turning paths P2 are located within the headland 74. Although each main path P1 shown in Figure 7 is a straight path, each main path P1 may include a curved portion. The main path P1 can be automatically generated, for example, by the user specifying two points near the edge of the field (points A and B in Figure 7) while viewing a map of the field displayed on the operating terminal 200 or remote device 400. In this case, multiple main paths P1 are set parallel to the line segment connecting points A and B specified by the user, and the target path within the field is generated by connecting these main paths P1 with a turning path P2. The dashed line in Figure 7 represents the working width of the implement 300. The working width is pre-set and recorded in the storage device 170. The working width can be set and recorded by the user operating the operating terminal 200 or remote device 400. Alternatively, the working width may be automatically recognized and recorded when the implement 300 is connected to the work vehicle 100. The spacing between the multiple main paths P1 can be set according to the working width. The target path can be created based on user operations before automatic operation starts. The target route can be created, for example, to cover the entire work area 72 within the field. The work vehicle 100 automatically travels along the target route, as shown in Figure 7, repeatedly going back and forth from the start point to the end point of the work. Note that the target route shown in Figure 7 is merely an example, and the method of defining the target route is arbitrary.

[0092] Next, we will explain an example of control during automatic operation in a field using the control device 180.

[0093] Figure 8 is a flowchart illustrating an example of steering control operation during automatic driving performed by the control device 180. The control device 180 performs automatic steering while the work vehicle 100 is in motion by executing the operations from steps S121 to S125 shown in Figure 8. The speed is maintained, for example, at a preset speed. While the work vehicle 100 is in motion, the control device 180 acquires data indicating the position of the work vehicle 100 generated by the GNSS unit 110 (step S121). Next, the control device 180 calculates the deviation between the position of the work vehicle 100 and the target path (step S122). The deviation represents the distance between the position of the work vehicle 100 at that time and the target path. The control device 180 determines whether the calculated position deviation exceeds a preset threshold (step S123). If the deviation exceeds the threshold, the control device 180 changes the steering angle by changing the control parameters of the steering device included in the drive device 240 so that the deviation becomes smaller. If the deviation does not exceed the threshold in step S123, the operation in step S124 is omitted. In the following step S125, the control device 180 determines whether or not it has received a command to terminate the operation. A command to terminate the operation may be issued, for example, when a user remotely instructs the automatic operation to stop, or when the work vehicle 100 reaches its destination. If no command to terminate the operation has been issued, the process returns to step S121 and performs the same operation based on the newly measured position of the work vehicle 100. The control device 180 repeats the operations from steps S121 to S125 until a command to terminate the operation is issued. The above operations are performed by the ECUs 182 and 184 in the control device 180.

[0094] In the example shown in Figure 8, the control device 180 controls the drive unit 240 based only on the deviation between the position of the work vehicle 100 identified by the GNSS unit 110 and the target path, but it may also take into account the azimuth deviation. For example, if the azimuth deviation, which is the angular difference between the orientation of the work vehicle 100 identified by the GNSS unit 110 and the direction of the target path, exceeds a preset threshold, the control device 180 may change the control parameters of the steering device of the drive unit 240 (e.g., steering angle) according to that deviation.

[0095] Below, we will explain in more detail an example of steering control by the control device 180, referring to Figures 9A to 9D.

[0096] Figure 9A shows an example of a work vehicle 100 traveling along a target path P. Figure 9B shows an example of a work vehicle 100 positioned to the right of the target path P. Figure 9C shows an example of a work vehicle 100 positioned to the left of the target path P. Figure 9D shows an example of a work vehicle 100 facing inclined relative to the target path P. In these figures, the position and orientation of the work vehicle 100 as measured by the GNSS unit 110 are represented as r(x,y,θ). (x,y) are coordinates representing the position of the reference point of the work vehicle 100 in the XY coordinate system, which is a two-dimensional coordinate system fixed to the Earth. In the examples shown in Figures 9A to 9D, the reference point of the work vehicle 100 is located where the GNSS antenna is installed on the cabin, but the position of the reference point is arbitrary. θ is the angle representing the measured orientation of the work vehicle 100. In the illustrated examples, the target path P is parallel to the Y axis, but generally, the target path P is not necessarily parallel to the Y axis.

[0097] As shown in Figure 9A, if the position and orientation of the work vehicle 100 are not deviating from the target path P, the control device 180 maintains the steering angle and speed of the work vehicle 100 without changing them.

[0098] As shown in Figure 9B, if the position of the work vehicle 100 is shifted to the right of the target path P, the control device 180 changes the steering angle so that the direction of travel of the work vehicle 100 is tilted to the left and approaches path P. At this time, the speed may also be changed in addition to the steering angle. The magnitude of the steering angle can be adjusted, for example, according to the magnitude of the position deviation Δx.

[0099] As shown in Figure 9C, if the position of the work vehicle 100 is shifted to the left of the target path P, the control device 180 changes the steering angle so that the direction of travel of the work vehicle 100 is tilted to the right and approaches path P. In this case as well, the speed may be changed in addition to the steering angle. The amount of change in the steering angle can be adjusted, for example, according to the magnitude of the position deviation Δx.

[0100] As shown in Figure 9D, if the position of the work vehicle 100 is not significantly off the target path P, but its orientation differs from that of the target path P, the control device 180 changes the steering angle to reduce the azimuth deviation Δθ. In this case as well, the speed may be changed in addition to the steering angle. The magnitude of the steering angle can be adjusted, for example, according to the magnitudes of the position deviation Δx and the azimuth deviation Δθ. For example, the smaller the absolute value of the position deviation Δx, the larger the amount of change in the steering angle corresponding to the azimuth deviation Δθ may be. When the absolute value of the position deviation Δx is large, the steering angle will be changed significantly in order to return to path P, so the absolute value of the azimuth deviation Δθ will inevitably be large. Conversely, when the absolute value of the position deviation Δx is small, it is necessary to bring the azimuth deviation Δθ closer to zero. For this reason, it is appropriate to relatively increase the weight (i.e., control gain) of the azimuth deviation Δθ used to determine the steering angle.

[0101] Control technologies such as PID control or MPC control (model predictive control) can be applied to the steering and speed control of the work vehicle 100. By applying these control technologies, the control of the work vehicle 100 to approach the target path P can be made smoother.

[0102] Furthermore, if one or more obstacle sensors 130 detect an obstacle while the vehicle is in motion, the control device 180 will stop the work vehicle 100. At this time, the buzzer 220 may emit a warning sound or a warning signal may be transmitted to the remote device 400. If it is possible to avoid the obstacle, the control device 180 may control the drive unit 240 to avoid the obstacle.

[0103] In the embodiments of this disclosure, the work vehicle 100 is capable of autonomous driving not only within the field but also outside the field. Outside the field, the control device 180 can detect objects (e.g., other vehicles or pedestrians) located relatively far from the work vehicle 100 based on data output from the camera 120 or LiDAR sensor 140. The control device 180 can achieve autonomous driving on roads outside the field by generating a local path to avoid the detected object and performing speed control and steering control along the local path.

[0104] As described above, the work vehicle 100 in the embodiment of this disclosure can travel automatically inside and outside the field without a driver. Figure 10 is a schematic diagram showing an example of a situation in which multiple work vehicles 100 are automatically traveling inside a field 70 and on a road 76 outside the field 70. The storage device 170 records an environmental map and target route for an area including multiple fields and surrounding roads. The environmental map and target route can be generated by the management device 600 or the ECU 185. When the work vehicle 100 is traveling on a road, the work vehicle 100 travels along the target route with the work implement 300 raised, sensing the surroundings using sensing devices such as the camera 120 and the LiDAR sensor 140. While traveling, the control device 180 sequentially generates local routes and causes the work vehicle 100 to travel along the local routes. This enables automatic travel while avoiding obstacles. The target route may be changed during travel depending on the situation. Thus, the control device 180 in the embodiments of this disclosure can generate a target route for automatic driving within the field and on the roads surrounding the field. In automatic driving mode, the control device 180 causes the work vehicle 100 to automatically drive within the automatic driving area defined by the field and roads on which the target route has been generated.

[0105] [2-2. Remote Control] Next, we will explain the operation related to the remote control of the work vehicle 100.

[0106] When the work vehicle 100 is operating autonomously, the user can remotely monitor and control the work vehicle 100 using the remote control device 400. When the work vehicle 100 is operating autonomously, the control device 180 transmits images (e.g., moving images) captured by one or more cameras 120 mounted on the work vehicle 100 to the remote control device 400 via the communication device 190. The remote control device 400 displays the images on the display 430. The user can check the situation around the work vehicle 100 while viewing the displayed images and, if necessary, start remotely controlled driving.

[0107] Figure 11 shows an example of an image displayed on the display 430 of the remote device 400. The image in Figure 11 shows the field 70, the road 76, the sky 79, and the front of the work vehicle 100. This image was captured by a camera 120 that photographs the front of the work vehicle 100. Not limited to the camera 120 that photographs the front of the work vehicle 100, images captured by cameras 120 that photograph, for example, the rear, right, or left, may also be displayed on the display 430. The display 430 displays moving images with a frame rate of, for example, 3 fps or more (typically 30 fps or 60 fps, etc.). Multiple images captured by multiple cameras 120 may be displayed on multiple displays. For example, as shown in Figure 1B, multiple images may be displayed on multiple displays 430 in the remote monitoring center. In that case, the user (i.e., operator), who is the monitor, can check the situation around the work vehicle 100 in detail while viewing the multiple images displayed on the multiple displays 430. In addition to the image captured by camera 120, a map of the area including the work vehicle 100 may also be displayed on the display.

[0108] In the example shown in Figure 11, the displayed image includes a button 98 for instructing the start of remote control (remote control start button) and a button 99 for emergency stopping the work vehicle 100 (emergency stop button). The user can switch from automatic driving mode to remote control mode by touching or clicking the remote control start button 98. The user can also emergency stop the work vehicle 100 by touching or clicking the emergency stop button 99.

[0109] When the remote control start button 98 is pressed, remote control becomes possible. For example, in the example shown in Figure 1B, the operator can remotely control the work vehicle 100 using the remote control device 500. The control device 180 responds to the user's operation and causes the work vehicle 100 to perform the instructed action.

[0110] In the above example, images acquired by a camera 120 mounted on the work vehicle 100 (hereinafter also referred to as "camera images") are displayed on the display 430 of the remote device 400. In addition to camera images, images based on point cloud data or other sensing data acquired by, for example, a LiDAR sensor 140 may also be displayed on the display 430. Images based on point cloud data can be used for monitoring in the same way as camera images, as they show the distribution of features present around the work vehicle 100. Based on the camera images or images based on point cloud data, the operator can understand the situation around the work vehicle 100. In the following description, image data generated by the camera 120 and moving images or videos based on point cloud data generated by the LiDAR sensor 140 may be referred to as "time-series images".

[0111] [3. Video Display System] The image display system in the embodiment of this disclosure comprises an imaging device, a screen, and a control device. The imaging device is attached to an agricultural machine (work vehicle 100) to which a work machine 300 is connected, and generates time-series image data by taking photographs in the direction of travel of the work vehicle 100. The control device displays an image on the screen based on the time-series image data. The control device is configured to (1) display a composite image on the screen in which the trajectory of the work machine is superimposed on the image when the work machine is connected to the opposite side of the direction of travel of the work vehicle 100, or (2) display a composite image on the screen in which an image showing the work tracks after ground work in the direction of travel, which is expected to occur while the work vehicle 100 to which the work machine is connected is traveling, is superimposed on the image. For example, a camera 120 provided on the front of the work vehicle 100 can function as the imaging device. The display 430 of the remote device 400 illustrated in Figure 1B, or the touchscreen of the operation terminal 200 illustrated in Figure 5, can function as the screen of the image display system. The control device 180 of the work vehicle 100, the processor 460 of the remote device 400, or the processor of the operating terminal 200 may function as a control device for the video display system. Hereafter, the control device for the video display system will be simply referred to as the "control device."

[0112] In the embodiments of this disclosure, the implement 300 is attached to the rear of the work vehicle 100. The image acquired by the imaging device is an image of the front of the agricultural machinery. However, the implement 300 may be attached to the front of the work vehicle 100. In this case, the image acquired by the imaging device may be an image of the rear of the work vehicle 100. The control device can display the image of the front of the work vehicle 100 on a screen. The control device displays a composite image on the screen in which the track of the implement 300 and / or the work track after ground work are superimposed on the image.

[0113] According to the video display system of the embodiment of this disclosure, it is possible to display a composite image on a screen by superimposing an image showing at least one of the trajectory of the work machine 300 located behind the work vehicle 100 and the work track after ground work in the direction of travel of the work vehicle 100 onto a forward-facing image. This can improve, for example, the operability of remote operation. Depending on the type of work machine 300, the height and width of the work machine 300 may differ. Even in that case, for example, an operator operating a remote control device can easily grasp future states in the direction of travel of the work vehicle 100, such as the trajectory of the work machine or the work track after ground work, from the composite image. Therefore, when remotely operating the work vehicle 100, for example inside or outside a field, the operator can easily make decisions such as changing the target path of the work vehicle 100, starting or stopping the operation of the work machine 300, or stopping the movement of the work vehicle 100.

[0114] [3-1. First Implementation Example] Referring to Figures 12A to 21, the method for displaying the composite image according to the first implementation example will be explained.

[0115] Figure 12A schematically shows an example of a forward view of the work vehicle 100 displayed on screen S. Figure 12B schematically shows an example of a composite image in which the trajectory of the implement 300 in the direction of travel of the work vehicle 100 is superimposed on the image. The implement 300 can perform at least one of the following ground operations, including ridging, tilling, planting crops, and harvesting crops. Figures 12A and 12B each illustrate the implement 300 ridging in a field 70. As shown in Figure 12A, the conventional image includes several completed ridges 91 extending in the direction of travel of the work vehicle 100, and naturally, no ridges 91 have yet been formed on the ground in the direction of travel.

[0116] In the first implementation example, when the work machine 300 is connected to the opposite side of the direction of travel of the work vehicle 100, the control device displays a composite image on the screen S in which the track of the work machine 300 is superimposed on the image. As a result, the track of the work machine 300 can be seen on the ground in the direction of travel of the work vehicle 100 within the image displayed on the screen S. In the example shown in Figure 12B, the track of the work machine 300 is represented by a virtual dotted line display 60.

[0117] Figure 13 schematically shows the positional relationship between the reference point R1 in the local coordinate system of the work vehicle 100 and the positions of both ends of the work machine 300. The local coordinate system moves with the work vehicle 100 and the work machine 300 and is also called the mobile coordinate system. The reference point R1 in the local coordinate system can be set at any position on the work vehicle 100. In the example shown in Figure 13, the reference point R1 is set at the position of the camera 120.

[0118] In the local coordinate system of the embodiments of this disclosure, the longitudinal direction of the vehicle is the X direction and the lateral direction is the Y direction when the connected work vehicle 100 and work machine 300 are traveling in a straight line on flat ground. The direction from rear to front is the +X direction, and the direction from left to right is the +Y direction. ISO 11783 defines device geometry as follows: "The X axis is defined with the normal direction of travel as positive," and "The Y axis is defined with the right side of the device as positive with respect to the normal direction of travel." The X and Y directions in the local coordinate system can be defined based on that definition of device geometry. The units of the coordinate values ​​in the local coordinate system are arbitrary, for example, millimeters. The XY directions and units of the coordinate values ​​are defined in the same manner as above in the local coordinate system of the work vehicle 100 alone and the local coordinate system of the work machine 300 alone.

[0119] The control device acquires work equipment information relating to the work equipment 300 and posture information relating to the current posture of the work vehicle 100. The work equipment information includes size information of the work equipment. The work equipment information may further include unique information (e.g., model number) that can identify the model of the work equipment 300. The size information of the work equipment 300 includes the size of the entire work equipment 300 in the X and Y directions, as shown in Figure 13, and may further include height information of the work equipment 300. The work equipment information can be pre-stored in a storage device within the work equipment 300. The posture information in the embodiments of this disclosure is a pose indicating the position and orientation of the work vehicle 100 as measured by the GNSS unit 110, and is represented by r(x,y,θ) as described above. The work vehicle information includes size information of the work vehicle 100. The work vehicle information may further include unique information (e.g., model number) that can identify the model of the work vehicle 100. The size information of the work vehicle 100 may include the size of the entire work vehicle 100 in the X and Y directions, as shown in Figure 13.

[0120] As mentioned above, the work vehicle 100 and the work machine 300 can communicate, for example, in accordance with the ISOBUS standard. In this way, the control device can acquire work machine information, including the size of the work machine 300, by communicating with the work machine 300. Alternatively, the control device may acquire work machine information, including the size of the work machine 300, which is input by the user via an input device. The user can input work machine information, for example, via the input device 420 of the remote device 400.

[0121] As illustrated in Figure 13, the positions T1 and T2 at both ends of the work machine 300, located in the width direction, are located a distance LX behind the reference point R1. Position T1 is located a distance L1Y to the left of the reference point R1, and position T2 is located a distance L2Y to the right of the reference point R1. In the first implementation example, the control device calculates the length LX based on the respective sizes of the work vehicle 100 and the work machine 300 in the X direction, and calculates the lengths L1Y and L2Y based on the size of the work machine 300 in the Y direction. In this way, the control device can calculate the coordinates of positions T1 and T2 relative to the reference point R1 in the local coordinate system. The control device further transforms the coordinate points of positions T1 and T2 from the local coordinate system to the geographic coordinate system based on attitude information. The coordinates of positions T1 and T2 in the geographic coordinate system are represented, for example, by latitude and longitude.

[0122] An example of determining the positional relationship between a reference point in the local coordinate system of a work vehicle and the position of a specific part of the work machine is described in detail in Japanese Patent Application Publication No. 2022-101030, which is a patent application by the present applicant. All disclosures of Japanese Patent Application Publication No. 2022-101030 are incorporated herein by reference.

[0123] The attitude information r(θ) represents the azimuth angle in the direction of travel of the work vehicle 100, and the attitude information r(x,y) is represented, for example, by latitude and longitude. The azimuth angle is represented, for example, by a clockwise angle with true north as the reference direction. In the first implementation example, the control device predicts the trajectory of the work machine 300 based on the work machine information and attitude information. For example, the control device determines a virtual straight line that passes through the coordinate points in the geographic coordinate system of positions T1 and T2 at both ends of the work machine 300, which are obtained based on the work machine information and attitude information, and extends in the direction of the azimuth angle obtained from the current attitude information, and determines this straight line as the future trajectory of the work machine 300 in the direction of travel of the work vehicle 100.

[0124] The control device can predict the trajectory of the implement 300, including the predicted trajectory of at least one of its end portions. In the example shown in Figure 13, the control device predicts the trajectories of both end portions of the implement 300. The control device determines a hypothetical straight line 60L that passes through the coordinate point at position T1 in the geographic coordinate system and extends in the direction of true north (θ=0), and further determines a hypothetical straight line 60R that passes through the coordinate point at position T2 in the geographic coordinate system and extends in the direction of true north (θ=0). The control device determines the straight lines 60L and 60R as the trajectories of the left and right portions of both ends of the implement 300, respectively. In Figure 13, the straight lines 60L and 60R are shown as dotted lines. However, the representation of the implement 300's trajectory is not limited to line representation.

[0125] In the first implementation example, the control device performs a coordinate transformation on a point cloud that defines a virtual straight line in the geographic coordinate system, representing the trajectory of the work machine 300, to the pixel positions in the image coordinate system. Specifically, the control device uses the camera's external parameters for converting the geographic coordinate system to the local coordinate system to transform the coordinate points of the point cloud defining the virtual straight line from the geographic coordinate system to the local coordinate system. Furthermore, the control device uses the camera's internal parameters for converting the local coordinate system to the image coordinate system to transform the coordinate points of the point cloud defining the virtual straight line from the local coordinate system to the image coordinate system. This makes it possible to move a 3D point cloud located in the geographic coordinate system to a 2D plane in the image coordinate system. In this way, the control device can generate a composite image by superimposing the straight lines 60L and 60R, which represent the trajectories of the left and right ends of the work machine 300, onto the image. The control device can generate the composite image using, for example, real-time rendering (or real-time CG) technology.

[0126] In the example described above, the control device converts the coordinate points of positions T1 and T2 at both ends of the work machine 300 from the local coordinate system to the geographic coordinate system, and displays the trajectory calculated based on positions T1 and T2 in the geographic coordinate system on the screen S. However, the example of generating a composite image is not limited to this. For example, the control device may set straight lines 60L and 60R (see Figure 13) passing through both ends of the work machine 300 in the local coordinate system, and superimpose the set straight lines 60L and 60R onto the image acquired by the imaging device. The local coordinate system and the image coordinate system can be associated in advance. According to this association, the control device can generate a composite image in which the straight lines 60L and 60R are superimposed on the image by converting the coordinate points of the straight lines 60L and 60R from the local coordinate system to the image coordinate system.

[0127] The control device may predict the trajectory of the wheels 104 of the work vehicle 100 based on attitude information while the work vehicle 100 is in motion, and display a composite image on the screen S in which the trajectory of the wheels of the work vehicle 100 is superimposed on the video. The work vehicle information may include information on the coordinate points in the local coordinate system where each of the four wheels of the work vehicle 100 is located. In the first implementation example, the control device determines virtual straight lines 61L and 61R that pass through the coordinate points of the pair of rear wheels 104R in the local coordinate system and extend in the direction of the azimuth angle determined from the current attitude information, similar to the trajectory of the work machine 300. The control device determines the straight lines 61L and 61R as the trajectory of the pair of rear wheels 104R. In Figure 13, the straight lines 61L and 61R are shown as dashed lines. However, the display of the trajectory of the wheels 104 is not limited to line display.

[0128] Figure 14 schematically shows an example of a composite image in which the track of the work machine 300 and the tracks of the pair of rear wheels 104R are superimposed on the image. The control device can generate a composite image in which line displays showing the tracks of the pair of rear wheels 104R are further superimposed on the image by transforming the coordinate points of the lines 61L and 61R from the geographic coordinate system to the image coordinate system.

[0129] As another example of showing the trajectory of the work machine 300, the control device 180 of the work vehicle 100 can estimate the trajectory of the wheels in the image coordinate system in the video from work vehicle information and attitude information. Since the relative positional relationship of the wheels with respect to the camera 120 (reference point R1) of the work vehicle 100 is known, the control device can estimate the trajectories of both ends of the work machine 300 from the trajectories of the pair of rear wheels 104R in the image coordinate system based on this positional relationship.

[0130] Figure 15 schematically shows an example of a composite image in which the trajectory of the work machine 300 and a target line along the target path are superimposed on the image. The control device may display a composite image on the screen S in which a target line 62 along the target path set for the automatic driving of the work vehicle 100 is further superimposed on the image. In the example shown in Figure 15, the target line 62 is shown as a dashed line. However, the display of the target line 62 is not limited to a line display. Such a composite image can make it easier to visually grasp the deviation of the trajectory of the work machine 300 from the target line.

[0131] Figure 16 is a schematic diagram illustrating the deviation between the predicted trajectory of the implement 300 and the reference trajectory. In Figure 16, the predicted trajectory and the reference trajectory of the implement 300 are shown by lines 60 and 63, respectively. The control device can estimate the deviation of the predicted trajectory 60 of the implement 300 from the reference trajectory based on the predicted trajectory of the implement 300 and the reference trajectory predicted from the implement 300's past trajectory. For example, the control device obtains the regression line 63 from waypoints that define the implement 300's past trajectory using the least squares method. The control device can determine this regression line 63 as the reference trajectory. The deviation is expressed by the angle α between line 60 and the regression line 63.

[0132] The control device may display different indications on the screen S depending on whether the amount of deviation is below a threshold or above a threshold. If the amount of deviation is below a threshold, the control device may, for example, highlight the line 60 indicating the trajectory of the work machine 300 to inform the operator that automatic travel along the target path is proceeding smoothly.

[0133] Figure 17 schematically shows an example of a composite image including a warning display 81 that may be displayed on the screen S when the amount of deviation exceeds a threshold. The control device may cause the screen S to display a composite image including a warning display 81 that warns that the predicted trajectory of the implement 300 is deviating from a past trajectory when the amount of deviation exceeds a threshold. The warning display 81 may be displayed at the forefront of the image. In the example in Figure 17, the straight line 60 showing the trajectory of the implement 300 is shown to be in contact with the adjacent furrow 91. With such a composite image, for example, contact with the adjacent furrow can be avoided in advance by prompting the operator to change the target path.

[0134] Figure 18 schematically shows the positional relationship between the reference point R1 in the local coordinate system of the work vehicle 100 and the position of one side of the offset implement 300. Examples of offset implements include plows for deep plowing, inversion, and tillage, or mowers. In the example shown in Figure 18, the implement 300 is offset to the left with respect to the center of the work vehicle 100. As explained with reference to Figure 13, the control device calculates the coordinate point in the geographic coordinate system for the position of one side of the offset implement 300, T1, based on lengths Lx and L1Y. The control device determines a virtual straight line 60L that passes through the coordinate point of position T1 in the geographic coordinate system and extends in the true north direction (θ=0). The control device determines the straight line 60L as the trajectory of the left side of the implement 300.

[0135] Figure 19 schematically shows an example of a composite image in which the trajectory of an offset implement 300 is superimposed on the video. Figure 19 shows the grass cutting process performed by the offset implement 300. The control device may also display a composite image on the screen S in which a straight line 60L, representing the trajectory of the left portion of the offset implement 300, is superimposed on the video.

[0136] The video display system according to the embodiment of this disclosure may include a sensing device that acquires sensing data showing the distribution of features around the work vehicle 100. The sensing device may be a sensing device 250 of the work vehicle 100. If a feature exists in the direction of travel of the work vehicle 100, the control device may estimate the size of the feature based on the sensing data output from the sensing device. Depending on the comparison result between the size of the work machine 300 and the size of the feature, the control device may display a composite video on the screen S that includes a warning display 82 warning that the work machine 300 may collide with the feature.

[0137] Figure 20 schematically shows an example of a video showing an overpass 78 that crosses road 76 in the direction of travel of a work vehicle 100 traveling on a road 76 outside the field. The work vehicle 100 may travel on the road outside the field with the work implement 300 raised. Therefore, depending on the type of work implement 300, the work implement 300 may collide with an object located in the direction of travel of the work vehicle 100. For example, the control device can estimate the height of the overpass 78 based on sensing data output from the sensing device. The control device can also estimate the height of the top of the work implement 300 relative to the ground from the size of the work implement 300 in the height direction. If the estimated height of the top is greater than the estimated height of the overpass 78, the control device displays a composite video including a warning display 82 on the screen S. By displaying the warning display, collisions with objects can be avoided in advance.

[0138] An image display system according to the embodiments of this disclosure may include a light source controlled by a control device and an optical system that receives light emitted from the light source and forms a virtual image in front of a screen. In other words, the image display system may include a HUD. HUDs, which project information within a person's field of view, are used to assist driving by displaying information on the front windshield of a vehicle.

[0139] Figure 21 is a schematic diagram showing an example of a HUD unit configuration. One type of HUD is one that uses a virtual image optical system, which will be described below. However, the configuration of the HUD unit is not limited to the example shown in Figure 21.

[0140] The HUD unit 800 comprises a light source 810, a transmissive screen 820, a field lens 830, and a combiner 840. The optical system of the HUD unit 800 has the transmissive screen 820, the field lens 830, and the combiner 840, and may further include a MEME mirror, a movable lens, and the like. The HUD unit 800 is mounted, for example, on the ceiling surface of the roof inside the cabin of a work vehicle.

[0141] A light beam emitted from the light source 810 is focused by the transmissive screen 820 to form a real image. The transmissive screen 820 functions as a secondary light source, directing the focused light beam toward the combiner 840 so that its illumination area is approximately rectangular. The combiner 840 forms a virtual image based on the illuminated light beam. This allows the driver to view the image along with the scenery through the combiner 840.

[0142] The light source 810 is a device that displays images. The light source 810 is configured to emit display light toward a transmissive screen 820. For example, known methods for displaying images include DLP (Digital Light Processing) and laser projectors. The light source 810 may have a laser projector and a MEME mirror that scans the light beam emitted from the laser projector. An example of a laser projector is an RGB laser projector.

[0143] The transmissive screen 820 has a microlens array on the light-receiving surface side. The transmissive screen 820 functions to broaden the incident beam. The field lens 830 is positioned between the transmissive screen 820 and the combiner 840, and is located near the transmissive screen 820. The field lens 830 is formed, for example, from a convex lens and changes the direction of propagation of the light beam emitted from the transmissive screen 820. By using the field lens 830, the efficiency of light utilization can be further increased. However, the field lens 830 is not essential.

[0144] The combiner 840 typically uses a half-mirror, but a holographic element may also be used. The combiner 840 reflects the divergent light beam from the transmissive screen 820 to form a virtual image of light. The combiner 840 has the function of magnifying and displaying the image formed on the transmissive screen 820 at a distance, and also has the function of superimposing the image onto the scenery. This allows the driver to view the image along with the scenery through the combiner 840. In other words, the driver can view the image projected on the screen along with the scenery. Depending on the curvature of the combiner 840, the size of the virtual image or the position where the virtual image is formed can be changed. In the HUD unit 800, the combiner 840 functions as the screen of the video display system.

[0145] An example of the control device 850 is a processor. The control device 850 functions as a control device for a video display system.

[0146] Thus, the technology of this disclosure can also be applied to the display of images in a HUD unit.

[0147] [3-2. Second Implementation Example] Referring to Figures 22 to 28, the method for displaying the composite image according to the second implementation example will be explained.

[0148] Figure 22 schematically shows an example of a composite image in which the work track after ground work in the direction of travel is superimposed on the video. Figure 22 illustrates the work machine 300 performing ridging work in field 70. In the second implementation example, the control device displays a composite image on screen S, which is an image superimposed on the video showing the work track after ground work in the direction of travel that is predicted to occur while the work vehicle 100 to which the work machine 300 is attached is traveling. As a result, the display showing the work track after ground work in the direction of travel of the work vehicle 100 can be confirmed in the video displayed on screen S. In the illustrated example, the shape of the outer edge of the rib showing the virtual work track is indicated by line (dotted line) 65. By displaying the virtual work track on the video in front, an operator operating a remote control device, for example, can grasp the work track by the work machine 300 as if viewing the work vehicle 100 from the rear before actually performing the work. For example, by virtually displaying the predicted work path a few seconds later on screen S, the operator can easily grasp future states, such as the work path after ground work, from the synthesized image. This can contribute to improving the operability of remote control.

[0149] The work traces displayed on screen S after ground work include completed states that mimic the work performed by the implement 300. For example, in the case of ridging, the work traces are represented by a figure or image that mimics the completed state of ridging; in the case of tilling, they are represented by a figure or image that mimics the completed state of tilling; and in the case of transplanting, they are represented by a figure or image that mimics the completed state of transplanting.

[0150] Figure 23 schematically illustrates another example of a composite image in which the work traces after ground work in the direction of travel are superimposed on the video. Figure 23 illustrates the process of a work machine 300 planting seedlings along a furrow 91. In the illustrated example, a virtual crop row 65R is formed on the furrow 91 in the direction of travel of the work vehicle 100 as the work traces after ground work. In other words, the control device generates a composite image in which the virtual crop row 65R formed on the furrow 91 is superimposed on the video.

[0151] Figure 24 schematically shows an example of a composite image in which the work traces after ground work and the target line 62 are superimposed on the image. Similar to the first implementation example, the control device may display a composite image on the screen S in which the target line 62 along the target path is further superimposed on the image. In the illustrated example, the target line 62 is shown on the ridges indicating the work traces 65.

[0152] The control device in the second implementation example predicts the work track based on work equipment information regarding the work equipment 300 and attitude information regarding the current attitude of the work vehicle 100, similar to the first implementation example. For example, the control device predicts the trajectory of the work equipment 300, including the predicted trajectory of at least one of the end portions of the work equipment 300. The work equipment information includes information about the size of the work equipment and may further include information about the type of work equipment. As described in the first implementation example, the control device predicts the trajectory of the work equipment 300 based on the size information and attitude information of the work equipment 300 while the work vehicle 100 is in motion. The control device may display a composite image on the screen S in which the predicted trajectory of the work equipment 300 is further superimposed on the image.

[0153] The control device further predicts the work track based on information regarding the predicted trajectory of the implement 300 and the type of implement 300. For example, if the implement 300 performs tilling work, the information regarding the type of implement included in the implement information may include information such as tilling depth and tilling width. In the example shown in Figure 22, the implement 300 is a rotary tiller for making ribs, and the control device can draw a three-dimensional virtual rib image representing the work track after the rib-making work by, for example, performing real-time rendering based on information regarding the rib-making work, tilling depth, tilling width, and information regarding the predicted trajectory of the implement 300 (coordinate points in a geographic coordinate system). In the example shown in Figure 23, the implement 300 is a vegetable transplanter, and the control device can draw a three-dimensional virtual crop row image representing the work track after the crop planting work by, for example, performing real-time rendering based on information regarding the crop planting work, including the type of seedling, and information regarding the predicted trajectory of the implement 300. The control device can determine the color information and textures necessary for displaying the image based on the type of work being done on the ground or the type of seedling.

[0154] The data necessary for drawing a figure or image that mimics the state of the work traces described above can be pre-stored as a database in the storage device 650 of the management device 600, for example, in association with the type of work. The control device identifies the type of work by referring to information such as the type of implement and the width of the implement. The control device reads from the database the data necessary to mimic the state of the work traces associated with the identified type of work. The control device further refers to information such as tillage depth and tillage width, and based on the read data, can draw a figure or image that mimics the state of the work traces on the screen S.

[0155] The control device may display a composite image on screen S in which guidelines for the work are further superimposed on the video. The guidelines may be set based on a target line. In this example, the trajectory of the work machine 300 is shown by the work prediction line. The control device may display a composite image on screen S in which the work prediction line includes guidance displays that guide the user to align with the guidelines.

[0156] Figure 25 schematically shows an example of a composite video that includes guidance displays to guide the user to conform to guidelines. In Figure 25, the work track 65, the work prediction line 66, and the guidelines 67 are shown as dotted, dashed, and thick dashed lines, respectively. The automatic driving mode of the work vehicle 100 includes, for example, a guide mode that guides the user to conform to guidelines. Figure 25 shows the work vehicle 100 driving automatically in guide mode.

[0157] The control device may estimate the amount of deviation of the work prediction line 66 from the guideline 67 in a manner similar to that described with reference to Figure 16, for example. When the amount of deviation is less than a threshold, the work prediction line 66 is aligned with the guideline 67. When the amount of deviation is greater than or equal to the threshold, the work prediction line 66 is not aligned with the guideline 67. While the work prediction line 66 is aligned with the guideline 67, the control device may highlight the line display by, for example, making the work prediction line 66 thicker or changing the color of the work prediction line 66. While the work prediction line 66 is not aligned with the guideline 67, the control device may, for example, make the work prediction line 66 blink. By highlighting the work prediction line 66 according to the situation in this way, the operator can be guided.

[0158] Figure 26 schematically shows an example of a composite image including a warning display that warns that the work prediction line 66 is not aligned with the guideline 67. The control device may display a composite image on the screen S including a warning display 83 that warns that the work prediction line is not aligned with the guideline while the work prediction line 66 is not aligned with the guideline 67. Alternatively, while the work prediction line 66 is not aligned with the guideline 67, for example, the buzzer 220 of the work vehicle 100 may function as a notification device that notifies that the work prediction line 66 is not aligned with the guideline 67. Examples of notification devices are not limited to buzzers and may include optical devices such as LED (Light Emitting Diode) lamps or vibrating devices such as vibrators.

[0159] Figure 27 schematically shows an example of a composite image that includes guidance prompting the user to change the target route. If the deviation is greater than or equal to a threshold, that is, if the work prediction line 66 does not align with the guideline 67, the control device may display a composite image on the screen S that includes guidance prompting the user to change the target route set for automatic driving of the work vehicle 100.

[0160] Figure 28 schematically shows an example of a composite image that includes the display of a turning point located in the direction of travel of the work vehicle 100. The control device may predict, based on time-series image data, the points where the work machine 300 will start, stop, or raise / lower in the direction of travel of the work vehicle 100, and display a composite image including the display of those points on the screen S. The composite image illustrated in Figure 28 includes the display 68 of a turning point. For example, near a turning point, there is generally a field levee on the opposite side, separated by the headland of the field. The control device can detect the field levee from the time-series image using, for example, a machine learning or deep learning object detection method. The control device displays the display 68 at the location of the object identified in the image.

[0161] As illustrated in Figure 28, the control device may display a composite image on the screen S that includes a display of distance information 85 indicating the distance from the current position of the work vehicle 100 to the turning point. The control device may measure the distance to the turning point by performing image analysis of time-series images, or it may measure the distance to the turning point based on sensing data output from a sensing device. The composite image is not limited to the turning point and may include displays of any point where the work machine 300 will start, stop, or raise / lower.

[0162] The configurations and operations of the embodiments described above are illustrative only, and this disclosure is not limited to the embodiments described above. For example, the various embodiments described above may be combined as appropriate to form other embodiments.

[0163] In the above embodiment, the agricultural machinery operates automatically, but the agricultural machinery does not necessarily have to have an autonomous driving function. The technology of this disclosure can be broadly applied to remotely controlled agricultural machinery.

[0164] The systems or video display systems for controlling automatic and / or remotely controlled driving in the above embodiments can also be retrofitted to agricultural machinery that does not have these functions. Such systems can be manufactured and sold independently of the agricultural machinery. Computer programs used in such systems can also be manufactured and sold independently of the agricultural machinery. Computer programs can be provided, for example, stored in a computer-readable non-temporary storage medium. Computer programs can also be provided by download via telecommunications lines (e.g., the Internet).

[0165] As described above, this disclosure includes the video display system and work vehicles described in the following items.

[0166] [Item A1] An imaging device that is attached to a work vehicle to which a work machine is connected, and generates time-series image data by taking photographs in the direction of travel of the work vehicle, The screen and, A control device that displays video on the screen based on the data of the time-series images, Equipped with, When the work machine is connected to the opposite side of the direction of travel of the work vehicle, the control device causes the screen to display a composite image on which the trajectory of the work machine is superimposed on the image.

[0167] [Item A2] The aforementioned work machine is connected to the rear of the work vehicle, The aforementioned video shows the view from the front of the work vehicle. The control device is capable of displaying the video on the screen. The control device displays a composite image on the screen, which is an image obtained by superimposing the trajectory of the work machine onto the image, as described in item A1.

[0168] [Item A3] The control device is a video display system according to item A1 or A2, which predicts the trajectory of the work machine based on work machine information relating to the work machine and attitude information relating to the current attitude of the work vehicle.

[0169] [Item A4] The control device is While the work vehicle is in motion, the trajectory of the work vehicle's wheels is predicted based on the attitude information. The video display system according to item A2 or A3, which displays the composite image on the screen, in which the trajectory of the wheels of the work vehicle is further superimposed on the video.

[0170] [Item A5] The aforementioned work vehicle is capable of autonomous driving. The control device displays the composite image on the screen, which is obtained by further superimposing a target line along a target path set to automatically drive the work vehicle onto the image. This is the image display system according to item A3 or A4.

[0171] [Item A6] The control device is Based on the predicted trajectory of the work machine and the reference trajectory predicted from the past trajectory of the work machine, the amount of deviation of the predicted trajectory of the work machine from the reference trajectory is estimated. A video display system according to any one of items A3 to A5, which displays different displays on the screen depending on whether the amount of deviation is less than a threshold or whether the amount of deviation is greater than or equal to a threshold.

[0172] [Item A7] When the amount of deviation is greater than or equal to the threshold, the control device predicts the trajectory of the work machine. The video display system described in item A6, which displays the composite video on the screen, including a warning message that warns that the trajectory has deviated from its past path.

[0173] [Item A8] The control device is a video display system according to any one of items A3 to A7, which predicts the trajectory of the work machine, including the predicted trajectory of at least one of the end portions of the work machine, located in the width direction of the work machine.

[0174] [Item A9] The control device acquires information about the work machine, including its size, by communicating with the work machine, as described in any of items A3 to A8.

[0175] [Item A10] The control device is a video display system according to any one of items A3 to A8, which acquires work machine information, including the size of the work machine, input by the user via an input device.

[0176] [Item A11] The work vehicle is equipped with a sensing device that acquires sensing data showing the distribution of terrain features around it. The control device estimates the size of the terrain present in the direction of travel of the work vehicle based on the sensing data output from the sensing device. The video display system according to item A9 or A10, which displays the composite video on the screen, including a warning display that warns that the work machine may collide with the terrain, depending on the result of comparing the size of the work machine with the size of the terrain.

[0177] [Item A12] A light source controlled by the control device, An image display system according to any one of items A1 to A11, comprising: an optical system that receives light emitted from the light source and forms a virtual image in front of the screen.

[0178] [Item A13] Work vehicles and Work equipment and A video display system as described in any of items 1 through 12, Agricultural machinery equipped with [specific features / equipment].

[0179] [Item B1] An imaging device that is attached to a work vehicle to which a work machine is connected, and generates time-series image data by taking photographs in the direction of travel of the work vehicle, The screen and, A control device that displays video on the screen based on the data of the time-series images, Equipped with, The control device is a video display system that displays a composite image on the screen, which is an image superimposed on the video showing the work track after ground work in the direction of travel, as predicted while the work vehicle to which the work machine is attached is traveling.

[0180] [Item B2] The control device predicts the work track based on work machine information relating to the work machine and posture information relating to the current posture of the work vehicle, as described in item B1.

[0181] [Item B3] The aforementioned work equipment information includes information regarding the type and size of the work equipment, The control device predicts the trajectory of the work machine based on information regarding the size of the work machine and the attitude information while the work vehicle is in motion. The video display system described in item B2 predicts the work path based on information regarding the trajectory of the work machine and the type of the work machine.

[0182] [Item B4] The video display system according to item B3, wherein the control device displays the composite image on the screen, which is obtained by further superimposing the predicted trajectory of the work machine onto the video.

[0183] [Item B5] The trajectory of the aforementioned work machine is indicated by the work prediction line. The control device displays the composite image on the screen, which further superimposes work guidelines onto the image. The video display system according to item B4, which displays the composite video on the screen, including guidance displays that guide the user so that the work prediction line conforms to the guidelines.

[0184] [Item B6] The video display system according to item B5, wherein the control device causes the screen to display the composite video, which includes a warning display indicating that the work prediction line does not conform to the guideline, if the work prediction line does not conform to the guideline.

[0185] [Item B7] The video display system described in item B5, comprising a notification device that notifies that the work prediction line does not conform to the guideline when the work prediction line does not conform to the guideline.

[0186] [Item B8] The aforementioned work vehicle is capable of autonomous driving. The control device is The amount of deviation of the work prediction line from the aforementioned guidelines is estimated, A video display system according to any one of items B5 to B7, wherein if the amount of deviation is greater than or equal to a threshold, the system displays the composite video on the screen, which includes a guidance display prompting the user to change the target route set for the automatic operation of the work vehicle.

[0187] [Item B9] The video display system according to item B8, wherein the control device displays the composite image on the screen, which is obtained by further superimposing a target line along the target path onto the video.

[0188] [Item B10] The aforementioned guidelines are the video display system described in item B9, which is set based on the aforementioned target line.

[0189] [Item B11] The control device predicts the trajectory of the work machine, including the predicted trajectory of at least one of the end portions of the work machine, located in the width direction of the work machine, according to any one of items B3 to B10.

[0190] [Item B12] The control device predicts, based on the time-series image data, the points in the direction of travel of the work vehicle where the work machine will start, stop, or raise / lower, and displays the composite image including the display of the points on the screen, according to any one of items B2 to B11.

[0191] [Item B13] The control device displays the composite image, which includes distance information indicating the distance from the current position of the work vehicle to the point, on the screen, as described in item B12.

[0192] [Item B14] The control device acquires information about the work machine by communicating with the work machine, as described in any of items B2 to B13.

[0193] [Item B15] The control device is a video display system according to any one of items B2 to B13, which acquires the work machine information input by the user via an input device.

[0194] [Item B16] The aforementioned work machine performs at least one of the aforementioned ground operations, including ridging, tilling, planting crops, and harvesting crops. The control device is a video display system according to any one of items B1 to B15, which generates the image showing the work trace after the ground work.

[0195] [Item B17] A light source controlled by the control device, An image display system according to any one of items B1 to B16, comprising an optical system that receives light emitted from the light source and forms a virtual image in front of the screen.

[0196] [Item B18] Work vehicles and Work equipment and A video display system described in any of items B1 through B17, Agricultural machinery equipped with [specific features / equipment].

[0197] [Item B19] The aforementioned work machine is an agricultural machine as described in item B18, which is connected to the rear of the work vehicle. [Industrial applicability]

[0198] The technology disclosed herein can be applied to video display systems for autonomous agricultural machinery such as tractors, harvesters, rice transplanters, riding cultivators, vegetable transplanters, lawnmowers, seeders, fertilizer spreaders, or agricultural robots. [Explanation of Symbols]

[0199] 50: GNSS satellite, 60, 60L, 60R, 61, 61L, 61R, 63, 63L, 63R: line display, 60A: base station, 62: target line, 65: work trace, 65R: crop row, 66: work prediction line, 67: guideline, 68: display, 70: field, 72: work area, 74: headland, 76: road, 78: overpass, 79: sky, 80: network, 81-83 :Warning display, 84:Guidance display, 85:Display, 91:Furrow, 98:Remote control start button, 99:Emergency stop button, 100:Work vehicle, 101:Vehicle body, 102:Motor, 103:Transmission, 104:Wheels, 104F:Front wheels, 104R:Rear wheels, 105:Cabin, 106:Steering system, 107:Driver's seat, 108:Coupling device, 110:GNSS unit, 111:GNSS receiver, 112:RTK receiver, 116:Processing circuit, 120:Camera, 130:Obstacle sensor, 140:LiDAR sensor, 150:Sensor group, 152:Steering wheel sensor, 154:Steering angle sensor, 156:Axle sensor, 160:Driving control system, 170:Memory device, 180:Control device, 181~18 6: ECU, 190: Communication device, 200: Operating terminal, 210: Operating switch group, 220: Buzzer, 240: Drive device, 250: Sensing device, 300: Work machine, 340: Drive device, 380: Control device, 390: Communication device, 400: Remote device, 420: Input device, 430: Display device (display), 450: Storage device, 460: Processor, 470: ROM, 480: RAM, 490: Communication device, 500: Remote control device, 600: Management device, 650: Storage device, 660: Processor, 670: ROM, 680: RAM, 690: Communication device, 800: HUD unit, 810: Light source, 820: Transmissive screen, 830: Field lens, 840: Combiner, 850: Control device

Claims

1. An imaging device that is attached to a work vehicle to which a work machine is connected, and generates time-series image data by taking photographs in the direction of travel of the work vehicle, The screen and, A control device that displays video on the screen based on the data of the time-series images, Equipped with, The control device is a video display system that, when the work machine is connected to the opposite side of the direction of travel of the work vehicle, displays a composite image on the screen in which the trajectory of the work machine is superimposed on the video.

2. The aforementioned work machine is connected to the rear of the work vehicle, The aforementioned video shows the view from the front of the work vehicle. The control device is capable of displaying the video on the screen. The video display system according to claim 1, wherein the control device displays a composite image on the screen in which the trajectory of the work machine is superimposed on the image.

3. The video display system according to claim 1 or 2, wherein the control device predicts the trajectory of the work machine based on work machine information relating to the work machine and information indicating the current position and orientation of the work vehicle.

4. The control device is While the work vehicle is in motion, the trajectory of the work vehicle's wheels is predicted based on the position and orientation information. The video display system according to claim 2, wherein the composite image obtained by further superimposing the trajectory of the wheels of the work vehicle onto the video is displayed on the screen.

5. The aforementioned work vehicle is capable of autonomous driving. The video display system according to claim 3, wherein the control device displays the composite image on the screen, which is obtained by further superimposing a target line along a target path set to automatically drive the work vehicle onto the video.

6. The control device is Based on the predicted trajectory of the work machine and the reference trajectory predicted from the past trajectory of the work machine, the amount of deviation of the predicted trajectory of the work machine from the reference trajectory is estimated. The video display system according to claim 3, which displays different displays on the screen depending on whether the amount of deviation is less than a threshold or whether the amount of deviation is greater than or equal to a threshold.

7. The video display system according to claim 6, wherein the control device causes the screen to display the composite video, which includes a warning display that warns that the predicted trajectory of the work machine is deviating from a past trajectory when the amount of deviation is greater than or equal to the threshold.

8. The video display system according to claim 3, wherein the control device predicts the trajectory of the work machine, including the predicted trajectory of at least one of the end portions of the work machine located in the width direction of the work machine.

9. The video display system according to claim 3, wherein the control device acquires information about the work machine, including the size of the work machine, by communicating with the work machine.

10. The video display system according to claim 3, wherein the control device acquires work machine information, including the size of the work machine, which is input by the user via an input device.

11. The work vehicle is equipped with a sensing device that acquires sensing data showing the distribution of terrain features around it. The control device estimates the size of the terrain present in the direction of travel of the work vehicle based on the sensing data output from the sensing device. The video display system according to claim 9, which displays the composite video on the screen, including a warning display that warns that the work machine may collide with the terrain, depending on the result of comparing the size of the work machine with the size of the terrain.

12. A light source controlled by the control device, The image display system according to claim 1 or 2, further comprising: an optical system that receives light emitted from the light source and forms a virtual image in front of the screen.

13. Work vehicles and Work equipment and The video display system according to claim 1 or 2, Agricultural machinery equipped with [specific features / equipment].

Images