Work vehicle systems and programs

Abstract

We provide a work vehicle system that assists in the remote operation of work vehicles. [Solution] The work vehicle system S comprises a remotely operated work vehicle 2, an unmanned aerial vehicle 1 having an imaging device 13, a display unit, and a control unit. The control unit acquires images captured by the imaging device 13 from the unmanned aerial vehicle 1, determines the direction of movement D2 of the work vehicle 2 by analyzing the captured images, and controls the display unit to overlay and display an image indicating the direction of movement D2 with the captured image.

Description

Technical Field

[0001] The present invention relates to a work vehicle system and a program.

Background Art

[0002] Patent Document 1 discloses an excavator management system in which an overhead image captured by a camera is displayed in an image display area. In this excavator management system, the camera is installed on a flying object such as a drone. The overhead image is displayed in the image display area so that the operator can intuitively grasp the positional relationship between the excavator and the objects existing around the excavator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the work vehicle travels on a hard and flat ground, the actual moving direction of the work vehicle coincides with the steering direction (control direction) and the front direction of the vehicle body. However, when the work vehicle travels on a sloping ground or a flexible work area, the actual moving direction of the work vehicle may be different from the steering direction (control direction) and the front direction of the vehicle body.

[0005] An object of the present invention is to provide a work vehicle system that assists in the remote operation of a work vehicle.

Means for Solving the Problems

[0006] The characteristic configuration of the work vehicle system for achieving the above objectives comprises a remotely operated work vehicle, an unmanned aerial vehicle equipped with an imaging device, a display unit, and a control unit. The control unit acquires images captured by the imaging device from the unmanned aerial vehicle, determines the direction of movement of the work vehicle by analyzing the captured images, and controls the display unit to overlay and display an image indicating the direction of movement with the captured images.

[0007] According to the above configuration, the actual direction of movement of the work vehicle is determined by image analysis of the images captured by the unmanned aerial vehicle's imaging device. Then, an image showing the actual direction of movement of the work vehicle is superimposed on the captured image and displayed on the display unit. This makes it easier for the operator remotely controlling the work vehicle to understand its actual direction of movement. Consequently, remote control of the work vehicle becomes easier.

[0008] Another characteristic configuration of the work vehicle system according to the present invention is that the control device acquires the planned work direction and controls the display unit to overlay and display the image indicating the planned work direction with the captured image.

[0009] According to the above-described configuration, an image indicating the planned work direction is superimposed on the captured image on the display unit, making it easier to direct the work vehicle to perform its intended work.

[0010] Another characteristic configuration of the work vehicle system according to the present invention is that it includes a remote control terminal for remotely operating the work vehicle, and the control device acquires the operating direction of the work vehicle from the remote control terminal and controls the display unit to superimpose and display an image indicating the operating direction of the work vehicle with the captured image.

[0011] According to the above feature configuration, an image indicating the direction of operation is superimposed on the captured image on the display unit. This makes it easier for the operator to intuitively recognize the direction of operation of the work vehicle and the actual direction of movement, further simplifying the remote control of the work vehicle.

[0012] Another characteristic configuration of the work vehicle system according to the present invention is that the control device identifies the forward direction of the work vehicle by performing image analysis on the captured image, and controls the display unit to overlay and display the image indicating the forward direction of the vehicle with the captured image.

[0013] According to the above feature configuration, an image indicating the forward direction of the machine is superimposed on the captured image on the display unit. This makes it less likely for an operator located far from the work vehicle to misjudge the vehicle's forward and backward direction, further facilitating remote operation of the work vehicle.

[0014] The characteristic configuration of the program for achieving the above objective is that of an operation assistance program executed in a system that assists in the remote operation of a work vehicle, and which causes a computer to execute the following processes: acquiring captured images from an unmanned aerial vehicle's imaging device; determining the direction of movement of the work vehicle by analyzing the captured images; and controlling a display unit to overlay and display an image indicating the direction of movement with the captured images.

[0015] According to the above configuration, the actual direction of movement of the work vehicle is determined by image analysis of the images captured by the unmanned aerial vehicle's imaging device. Then, an image showing the actual direction of movement of the work vehicle is superimposed on the captured image and displayed on the display unit. This makes it easier for the operator remotely controlling the work vehicle to understand its actual direction of movement. Consequently, remote control of the work vehicle becomes easier. [Brief explanation of the drawing]

[0016] [Figure 1] This is a diagram showing a work vehicle system. [Figure 2] This is a block diagram of the work vehicle system. [Figure 3] This figure shows an example of a composite image displayed on the display unit. [Figure 4] This figure shows an example of a composite image displayed on the display unit. [Figure 5] This figure shows an example of identifying the forward direction of the aircraft. [Figure 6] It is a flowchart showing an example of a process of superimposing and displaying a moving direction image and a captured image. [Figure 7] It is a flowchart showing an example of a process of superimposing and displaying a planned work direction image and a captured image. [Figure 8] It is a flowchart showing an example of a process of superimposing and displaying a steering direction image and a captured image.

Mode for Carrying Out the Invention

[0017] Hereinafter, a work vehicle system and a program according to the present invention will be described based on the drawings. The work vehicle system S is a system that assists in remotely operating a work vehicle that travels and works in the area F. The program is an operation assistance program that causes a computer to execute processes in the work vehicle system S. In the present embodiment, the area F is a grassland with long grass. Also, the area F is a slope.

[0018] As shown in FIGS. 1 and 2, the work vehicle system S includes a drone 1 having an imaging device 13 and a satellite positioning device 14, a remotely operable lawn mower 2 (an example of a work vehicle), a remote operation terminal 3 for remotely operating the lawn mower 2, a control device 4 (corresponding to a control device), and a management device C. The work vehicle system S assists in remotely operating the lawn mower 2 by displaying a captured image IM1 and the like captured by the imaging device 13 of the drone 1 on the display unit 33 of the remote operation terminal 3.

[0019] Particularly when the lawn mower 2 travels and works in the sloped area F, the lawn mower 2 is likely to move in the direction of going down the slope due to gravity. Therefore, as shown in FIG. 1, the planned work direction D1 of the lawn mower 2 and the moving direction D2 of the lawn mower 2 may be different. The work vehicle system S assists in remotely operating the work vehicle by superimposing and displaying the captured image IM1 and an image indicating the planned work direction D1 or the moving direction D2 on the display unit 33.

[0020] 〔Drone〕 As shown in FIGS. 1 and 2, the unmanned aircraft 1 is an aircraft capable of autonomous flight. The unmanned aircraft 1 is an unmanned aerial vehicle that cannot be boarded by a human in terms of its structure. The unmanned aircraft 1 can transmit and receive information to and from the control device 4 via wireless communication. In the present embodiment, the unmanned aircraft 1 is a drone including a plurality of rotors 11. Without being limited thereto, the unmanned aircraft 1 may be an unmanned helicopter. The unmanned aircraft 1 is configured to automatically follow and fly after the lawn mower 2.

[0021] The unmanned aircraft 1 includes a battery 12 as a drive source, an inertial measurement unit 15, and a control device 16 that controls the unmanned aircraft 1.

[0022] The power of the battery 12 is supplied to motors provided on the rotors 11. The rotors 11 drive around an axis extending in the vertical direction. The unmanned aircraft 1 flies by driving a plurality of rotors 11. Without being limited thereto, the unmanned aircraft 1 may include an engine as a drive source, and the rotors 11 may be driven by the power of the engine.

[0023] The imaging device 13 can image the lawn mower 2 and the area F around the lawn mower 2. In the present embodiment, the imaging device 13 is a stereo camera. The imaging device 13 has two cameras, and measures the distance to the measurement target using the parallax between the two cameras. Without being limited thereto, the imaging device 13 may be a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera.

[0024] In the present embodiment, the captured image IM1 captured by the imaging device 13 is a still image. Without being limited thereto, the captured image IM1 may be a moving image or a video that continuously displays still images. The angle θ1 in FIG. 1 indicates the angle of view of the imaging device 13.

[0025] The satellite positioning device 14 receives positioning signals from artificial satellites used in GNSS (Global Navigation Satellite Systems, such as GPS, QZSS, Galileo, GLONASS, BeiDou, etc.). The positioning signals are output to the control device 16.

[0026] The inertial measurement unit 15 detects inertial information (yaw angle, pitch angle, roll angle, etc.) of the unmanned aerial vehicle 1. The inertial measurement unit 15 includes an acceleration sensor, a rotational angular acceleration sensor, a gyroscope sensor, a magnetic field sensor, and the like.

[0027] The control device 16 includes a storage unit and a CPU (Central Processing Unit) that executes programs. The storage unit is composed of, for example, an HDD, ROM, or non-volatile memory.

[0028] The control device 16 calculates the position of the unmanned aircraft 1 based on the positioning signal.

[0029] [Grass trimmer] As shown in Figures 1 and 2, the grass trimmer 2 can be remotely controlled by a remote control terminal 3. The grass trimmer 2 comprises a machine body, a grass cutting unit 22 as a work device, an engine 23, a cover member 24 that covers the machine body, an inertial measuring device 25, and an indicator lamp 26 that notifies information related to the work.

[0030] The front of the aircraft is supported by a pair of front wheels 21a, and the rear of the aircraft is supported by a pair of rear wheels 21b.

[0031] The grass cutting unit 22 comprises a cutting blade housing and a cutting blade provided inside the cutting blade housing. The cutting blade is rotatable around an axis extending in the vertical direction. The cutting blade is held in the cutting blade housing. The grass cutting unit 22 is provided between the front wheel 21a and the rear wheel 21b in the front-rear direction of the machine.

[0032] The engine 23 outputs driving force to the front wheels 21a, rear wheels 21b, and grass cutting unit 22 via a transmission or the like. However, the grass cutter 2 may also be equipped with an electric motor instead of the engine 23. In this case, the front wheels 21a, rear wheels 21b, and grass cutting unit 22 are driven by the electric motor.

[0033] A cover member 24 is provided on the upper part of the machine. The cover member 24 covers the machine, including the grass-cutting unit 22, from above. The color of the cover member 24 is different from the area F in which the grass-cutting machine 2 operates. For example, the color of area F is green, while the color of the cover member 24 is orange.

[0034] The inertial measurement unit 25 detects inertial information (yaw angle, pitch angle, roll angle, etc.) of the lawnmower 2. The inertial measurement unit 25 includes an acceleration sensor, a rotational angular acceleration sensor, a gyroscope sensor, a magnetic field sensor, and the like.

[0035] Multiple alarm lamps 26 are provided on the upper part of the aircraft. The multiple alarm lamps 26 are arranged in the front-to-back direction of the aircraft. In this embodiment, three alarm lamps 26 are arranged in a single line in the front-to-back direction of the aircraft.

[0036] [Remote control terminal] As shown in Figure 2, the remote control terminal 3 includes a gear shifting device 31 for adjusting forward and reverse speeds, a turning device 32 for selecting left and right turns, and a display unit 33. The remote control terminal 3 is a remote control for operating the lawnmower 2. The remote control terminal 3 can send and receive information with the lawnmower 2 or the control device 4 via wireless communication. Although not shown, the remote control terminal 3 is equipped with other control devices for operation (such as adjusting engine speed and starting / stopping the engine 23).

[0037] The gear shift control device 31 is swung in the forward and backward directions to change to the forward, neutral, or reverse position. When the gear shift control device 31 is tilted forward (forward position), the remote control terminal 3 outputs a forward signal to the grass trimmer 2. When the gear shift control device 31 is upright (neutral position), the remote control terminal 3 outputs a stop signal to the grass trimmer 2. When the gear shift control device 31 is tilted backward (reverse position), the remote control terminal 3 outputs a reverse signal to the grass trimmer 2.

[0038] The turning control device 32 is swung from side to side, changing to a left turn position, a straight-ahead position, or a right turn position. When the turning control device 32 is tilted to the left (left turn position), the remote control terminal 3 outputs a left turn signal to the grass trimmer 2. When the turning control device 32 is upright (straight-ahead position), the remote control terminal 3 outputs a straight-ahead signal to the grass trimmer 2. When the turning control device 32 is tilted to the right (right turn position), the remote control terminal 3 outputs a right turn signal to the grass trimmer 2. The gear shift control device 31 and the turning control device 32 may be composed of a single joystick.

[0039] [Display] The display unit 33 is a display capable of displaying characters, images, and the like. The display unit 33 can display captured images IM1 taken by the unmanned aerial vehicle 1.

[0040] [Management device] The control device C is a terminal located remotely from the lawnmower 2. The control device C has a storage unit and a CPU (Central Processing Unit) that executes programs. The storage unit is composed of, for example, an HDD, ROM, or non-volatile memory.

[0041] [Control device] As shown in Figure 2, the control device 4 includes a planned work direction acquisition unit 41, a steering direction acquisition unit 42, an imaging information acquisition unit 43, a self-position acquisition unit 44, an image acquisition unit 45, a movement direction identification unit 46, a forward direction identification unit 47, an image generation unit 48, a detection unit 49, and a display control unit 50.

[0042] The control device 4 is a terminal located outside the remote control terminal 3. The control device 4 has a storage unit and a CPU (Central Processing Unit) that executes programs. The storage unit is composed of, for example, an HDD, ROM, or non-volatile memory. In this embodiment, the control device 4 is a PC terminal or a mobile terminal. The operation assistance program in the work vehicle system S is executed by the PC terminal or mobile terminal.

[0043] The planned work direction acquisition unit 41 acquires the planned work direction D1. In this embodiment, the planned work direction acquisition unit 41 acquires the planned work direction D1 calculated based on the work path R. In Figure 3, the pre-set work path R of the brush cutter 2 is shown by a dashed line. In this embodiment, the planned work direction D1 is the direction in which the brush cutter 2 is driven along the work path R. When the brush cutter 2 is on the work path R, the planned work direction D1 is parallel to the work path R. When the brush cutter 2 is not on the work path R, the planned work direction D1 is the direction in which the brush cutter 2 is driven to approach the work path R. In the example in Figure 3, the planned work direction D1 is set to drive the brush cutter 2 to approach the work path R, and the line segment L1 extending along the planned work direction D1 intersects the work path R outside of Figure 3. The planned work direction acquisition unit 41 acquires the planned work direction D1 over time.

[0044] The steering direction acquisition unit 42 acquires the steering direction D3 of the brush cutter 2 from the remote control terminal 3. When a forward signal and a straight signal are output, the steering direction D3 indicates the forward direction of the machine. When a reverse signal and a straight signal are output, the steering direction D3 indicates the rear direction of the machine. When a forward signal and a left turn signal are output, the steering direction D3 indicates the left-diagonal forward direction of the machine. When a forward signal and a right turn signal are output, the steering direction D3 indicates the right-diagonal forward direction of the machine. When a reverse signal and a left turn signal are output, the steering direction D3 indicates the left-diagonal rear direction of the machine. When a reverse signal and a right turn signal are output, the steering direction D3 indicates the right-diagonal rear direction of the machine. In the example in Figure 4, the steering direction acquisition unit 42 acquires a steering direction D3 indicating the left-diagonal forward direction, and the steering direction D3 extends from the front of the machine to the left-diagonal forward direction of the machine.

[0045] The imaging information acquisition unit 43 acquires imaging information regarding the orientation and field of view of the imaging device 13. For example, the orientation of the imaging device 13 is calculated over time based on the inertial information of the unmanned aerial vehicle 1. The field of view of the imaging device 13 is stored in the control device 4 in advance.

[0046] The self-position acquisition unit 44 acquires the self-position calculated by the satellite positioning device 14 from the unmanned aerial vehicle 1. The self-position acquisition unit 44 acquires the self-position, which is the position coordinate of the unmanned aerial vehicle 1 in area F, over time. The image acquisition unit 45 acquires the image IM1 captured by the imaging device 13.

[0047] The movement direction identification unit 46 identifies the movement direction D2 of the brush cutter 2 by image analysis of the captured image IM1. In Figure 3, the brush cutter K in the captured image IM1 from one second ago is shown by a dashed line. The movement direction D2 is identified by the change in the position (absolute position) of the brush cutter 2. Specifically, when the brush cutter 2 is performing work, it is identified by the change between the position of the brush cutter K one second ago and the position of the brush cutter 2 after a predetermined time (for example, at the present time). In this embodiment, the movement direction D2 is the direction in which the movement changes over time from the position of the brush cutter K to the position of the brush cutter 2. The movement direction D2 indicates the direction in which the brush cutter 2 is actually moving.

[0048] The forward direction identification unit 47 identifies the forward direction D4 of the lawnmower 2 by performing image analysis on the captured image IM1. The forward direction D4 indicates the front of the lawnmower 2. Figure 5 shows the captured image IM1 and the converted image IM1' obtained by applying a binarization transformation to the captured image IM1. In other words, the image generation unit 48 generates the converted image IM1' by applying a binarization transformation to the captured image IM1. As shown in Figure 5, the converted image IM1' shows a region F shown in black and a cover member 24 of the lawnmower 2 shown in white.

[0049] Based on the converted image IM1', the forward direction determination unit 47 determines that the end of the cover member 24 with the larger width dimension is the front. As a result, the forward direction determination unit 47 determines the forward direction D4 of the aircraft.

[0050] Figure 3 shows a composite image IM6, which is formed by combining the captured image IM1 with the planned work direction image IM2, the movement direction image IM3, and the work path image IM7 showing the work path R. The image generation unit 48 generates the planned work direction image IM2, which shows the planned work direction D1. In Figure 3, the planned work direction image IM2 is indicated by a black arrow. The image generation unit 48 also generates the movement direction image IM3, which shows the movement direction D2. In Figure 3, the movement direction image IM3 is indicated by a white arrow.

[0051] Figure 4 shows a composite image IM6, which is formed by combining the captured image IM1 with the control direction image IM4. The image generation unit 48 generates the control direction image IM4, which indicates the control direction D3. In this embodiment, as shown in Figure 4, the control direction image IM4 is an arc-shaped arrow extending diagonally forward to the left of the aircraft.

[0052] The image generation unit 48 generates an aircraft forward-facing image IM5 that shows the aircraft forward direction D4. In Figure 5, the aircraft forward-facing image IM5 is indicated by a hatched arrow. The image generation unit 48 also generates a composite image IM6 by superimposing the aircraft forward-facing image IM5 onto the captured image IM1. The display control unit 50 controls the display unit 33 to display the aircraft forward-facing image IM5 showing the aircraft forward direction D4 and the captured image IM1 superimposed.

[0053] [Regarding the process of overlaying the movement direction image and the captured image] Next, an example of the process of overlaying the movement direction image IM3 and the captured image IM1 will be explained based on the flowchart shown in Figure 6. Note that the steps described below may be performed in any order, and multiple steps may be performed simultaneously, as long as no inconsistencies arise.

[0054] [1-1] Process to acquire captured images The image acquisition unit 45 acquires the captured image IM1 taken by the imaging device 13 from the unmanned aerial vehicle 1 (step S001 in Figure 6, the same applies hereafter). The image acquisition unit 45 acquires the captured image IM1 from the unmanned aerial vehicle 1 via wireless communication.

[0055] [1-2] Process to identify the lawnmower in the captured image The detection unit 49 analyzes the captured image IM1 to determine whether or not the grass trimmer 2 is shown in the captured image IM1 (step S002). Specifically, the detection unit 49 detects the grass trimmer 2 based on the difference in color between the cover member 24 of the grass trimmer 2 and region F. However, the detection unit 49 may also detect the grass trimmer 2 in the captured image IM1 using trained data such as a neural network trained by machine learning (deep learning).

[0056] [1-3] Process to determine the direction of movement of the work vehicle If the grass mower 2 is shown in the captured image IM1 (step S002: Yes), the detection unit 49 analyzes the captured image IM1 to determine whether or not the grass mower 2 is performing work (step S003). If the grass mower 2 is performing work (step S003: Yes), the movement direction determination unit 46 determines the movement direction D2 of the grass mower 2 by image analysis of the captured image IM1 (step S004). Whether or not the grass mower 2 is performing work is determined based on the change in the position of the grass mower 2 over time. The movement direction determination unit 46 also determines the movement direction D2 based on the change in the position of the grass mower 2 over time. In this embodiment, as shown in Figure 3, the movement direction D2 is diagonally downward from the plane of the paper.

[0057] [1-4] Process for generating composite images The image generation unit 48 generates a movement direction image IM3 that shows the movement direction D2. The image generation unit 48 also generates a composite image IM6 by superimposing the movement direction image IM3 onto the captured image IM1 (step S005).

[0058] [1-5] Process to display the composite image on the display unit. The display control unit 50 controls the display unit 33 to display the composite image IM6 on the display unit 33 (step S006), and the process is completed successfully. In other words, the display control unit 50 controls the display unit 33 to display the movement direction image IM3, which shows the movement direction D2, and the captured image IM1 superimposed on each other. If the grass trimmer 2 is not shown in the captured image IM1 (step S002: No) or if the grass trimmer 2 is not performing work (step S003: No), the display control unit 50 controls the display unit 33 to display the captured image IM1 as is on the display unit 33 (step S007), and the process is completed successfully.

[0059] [Regarding the process of overlaying the planned work direction image and the captured image] Next, an example of the process of overlaying the planned work direction image IM2 and the captured image IM1 will be explained based on the flowchart shown in Figure 7. Note that the steps described below may be performed in any order, and multiple steps may be performed simultaneously, as long as no inconsistencies arise. Furthermore, the steps described below may be performed simultaneously with the process of overlaying the movement direction image IM3 and the captured image IM1.

[0060] [2-1] Process for acquiring imaging information The imaging information acquisition unit 43 acquires imaging information relating to the orientation and field of view of the imaging device 13 (step S011 in Figure 7, the same applies hereafter). Specifically, the imaging information acquisition unit 43 acquires imaging information regarding the orientation of the imaging device 13 from the unmanned aerial vehicle 1 via wireless communication. The imaging information acquisition unit 43 also acquires imaging information regarding the field of view of the imaging device 13 that is pre-stored in the control device 4. However, the imaging information acquisition unit 43 may also acquire imaging information regarding the field of view of the imaging device 13 from the unmanned aerial vehicle 1 via wireless communication.

[0061] [2-2] Process for acquiring captured images and the position of the aircraft The control device 4 acquires the captured image IM1 taken by the imaging device 13 and the position of the unmanned aerial vehicle 1 calculated by the satellite positioning device 14 from the unmanned aerial vehicle 1 (step S012). Specifically, the self-position acquisition unit 44 acquires the self-position from the unmanned aerial vehicle 1 via wireless communication. In addition, the captured image acquisition unit 45 acquires the captured image IM1 from the unmanned aerial vehicle 1 via wireless communication.

[0062] [2-3] Process to identify the lawnmower in the captured image The detection unit 49 analyzes the captured image IM1 to determine whether or not the lawnmower 2 is shown in the captured image IM1 (step S013). Next, the detection unit 49 calculates the relative position of the lawnmower 2 with respect to its own position, which is its absolute position in region F. As a result, the detection unit 49 calculates the position coordinates of the lawnmower 2 in region F.

[0063] [2-4] Process to obtain the planned work direction If the lawnmower 2 is shown in the captured image IM1 (step S013: Yes), the planned work direction acquisition unit 41 acquires the planned work direction D1 (step S014). Specifically, the planned work direction acquisition unit 41 acquires the planned work direction D1 from the management device C via wireless communication. However, the planned work direction acquisition unit 41 may calculate the planned work direction D1 based on the position coordinates of the lawnmower 2 in area F and the work path R. In this case, the planned work direction acquisition unit 41 acquires the calculated planned work direction D1 via wired communication.

[0064] [2-5] Process for generating composite images The image generation unit 48 generates a planned work direction image IM2 that shows the planned work direction D1. The image generation unit 48 also generates a composite image IM6 by superimposing the planned work direction image IM2 onto the captured image IM1 (step S015).

[0065] [2-6] Process to display the composite image on the display unit The display control unit 50 controls the display unit 33 to display the composite image IM6 on the display unit 33 (step S016), and the process is completed successfully. In other words, the display control unit 50 controls the display unit 33 to overlay and display the planned work direction image IM2, which indicates the planned work direction D1, and the captured image IM1. If the grass trimmer 2 is not shown in the captured image IM1 (step S013: No), the display control unit 50 controls the display unit 33 to display the captured image IM1 as is on the display unit 33 (step S017), and the process is completed successfully.

[0066] In this embodiment, as shown in Figure 3, the display control unit 50 displays the planned work direction image IM2, the movement direction image IM3, the work path image IM7, and the captured image IM1 superimposed on each other.

[0067] [Regarding the process of overlaying the control direction image and the captured image] Next, an example of the process of overlaying the control direction image IM4 and the captured image IM1 will be explained based on the flowchart shown in Figure 8. Note that the steps described below may be performed in any order, and multiple steps may be performed simultaneously, as long as no inconsistencies arise. Furthermore, these steps may be performed simultaneously with the process described above.

[0068] Since S021 is the same process as S011 described above, its explanation will be omitted.

[0069] [3-1] Process to obtain steering direction If the remote control terminal 3 is being operated (step S022 in Figure 8: Yes, the same applies hereafter), the steering direction acquisition unit 42 acquires the steering direction D3 of the brush cutter 2 from the remote control terminal 3 (step S023). The steering direction acquisition unit 42 determines whether or not the remote control terminal 3 is being operated based on the signal output from the remote control terminal 3. The steering direction acquisition unit 42 acquires the steering direction D3 from the remote control terminal 3 via wireless communication. However, the control device 4 may also determine the steering direction D3 by acquiring a signal from the remote control terminal 3 to the brush cutter 2. In this case, the steering direction acquisition unit 42 acquires the steering direction D3 stored inside the control device 4 via wired communication.

[0070] [3-2] Process for generating a composite image The image generation unit 48 generates a steering direction image IM4 that shows the steering direction D3. The image generation unit 48 also generates a composite image IM6 by superimposing the steering direction image IM4 onto the captured image IM1 (step S024).

[0071] [3-3] Process to display the composite image on the display unit. The display control unit 50 controls the display unit 33 to display the composite image IM6 on the display unit 33 (step S025), and the process is completed successfully. In other words, the display control unit 50 controls the display unit 33 to display the steering direction image IM4, which indicates the steering direction D3, and the captured image IM1 superimposed on each other. If the remote control terminal 3 is not being operated (step S022: No), the display control unit 50 controls the display unit 33 to display the captured image IM1 as is on the display unit 33 (step S026), and the process is completed successfully.

[0072] [Another embodiment] The present invention is not limited to the embodiments described above. For example, it may be configured as in the following alternative embodiments. In the alternative embodiments described below, components identical to those in the embodiments are denoted by the same numbers and reference numerals as in the embodiments described above.

[0073] [1] In the above embodiment, the planned work direction D1 is the direction in which the brush cutter 2 is driven along the work path R. However, the planned work direction D1 may be set in advance based on the area F. For example, as shown in Figure 1, when the brush cutter 2 is traveling in a direction that crosses the area F (slope), the planned work direction D1 may be set in advance to indicate the direction of the contour lines of the area F (slope).

[0074] [2] In the above-described embodiment, the unmanned aerial vehicle 1 is configured to automatically follow the lawnmower 2. However, the unmanned aerial vehicle 1 may be remotely controlled by a remote control or the like.

[0075] [3] In the above-described embodiment, the control device 4 is a terminal located outside the remote control terminal 3. However, the control device 4 may be located on the remote control terminal 3 or the management device C. Furthermore, some functions of the control device 4 may be located on the unmanned aerial vehicle 1, the remote control terminal 3, or the management device C.

[0076] [4] In the above embodiment, the remote control terminal 3 is a remote control for remotely operating the lawnmower 2. However, the remote control terminal 3 may be a PC terminal or a mobile terminal.

[0077] [5] In the above-described embodiment, the remote control terminal 3 is equipped with a display unit 33. However, in addition to the display unit 33 of the remote control terminal 3, the control device 4 or another PC terminal or mobile terminal may be equipped with a display unit 33. Alternatively, instead of the display unit 33 of the remote control terminal 3, the control device 4 or another PC terminal or mobile terminal may be equipped with a display unit 33.

[0078] [6] The image generation unit 48 may generate a composite image IM6 by superimposing at least one image from among the planned work direction image IM2, the movement direction image IM3, the control direction image IM4, and the aircraft forward direction image IM5 onto the captured image IM1.

[0079] [7] In the above embodiment, the display control unit 50 displays the composite image IM6 generated by the image generation unit 48 on the display unit 33. However, the image generation unit 48 does not have to generate the composite image IM6. For example, when the display control unit 50 displays the planned work direction image IM2, the movement direction image IM3, and the captured image IM1 on the display unit 33, it may display these separate images superimposed on each other.

[0080] [8] In the above embodiment, the control direction image IM4 is an arc-shaped arrow extending diagonally forward to the left of the aircraft. However, the control direction image IM4 may also be a straight line arrow extending diagonally forward to the left of the aircraft.

[0081] [9] In the above-described embodiment, the planned work direction image IM2, the movement direction image IM3, the control direction image IM4, and the aircraft forward direction image IM5 are arrow-shaped. However, at least one of the planned work direction image IM2, the movement direction image IM3, the control direction image IM4, and the aircraft forward direction image IM5 may be in any other form. At least one of the planned work direction image IM2, the movement direction image IM3, the control direction image IM4, and the aircraft forward direction image IM5 may be, for example, triangular, polygonal, circular, or an image combining these with lines.

[0082]

[10] In the above-described embodiment, the work vehicle system S is equipped with a grass cutter 2 as a remotely operated work vehicle. Alternatively, the work vehicle system S may be equipped with a rice transplanter, fertilizer spreader, seed planter, tractor, combine harvester, snowplow, etc.

[0083]

[11] When the unmanned aerial vehicle 1 is following the lawnmower 2 at a constant distance, the movement direction determination unit 46 may determine the movement direction D2 based on the changes in the region F surrounding the lawnmower 2 in the continuous captured images IM1 along a time series. In this case, the movement direction determination unit 46 may determine the movement direction D2 to be in the opposite direction to the changes in the feature points of region F in the captured images IM1 along a time series.

[0084]

[12] In the above embodiment, the forward direction identification unit 47 identifies the forward direction D4 of the lawnmower 2 by performing image analysis on the captured image IM1. However, the forward direction identification unit 47 may also identify the forward direction D4 of the lawnmower 2 based on inertial information of the lawnmower 2 input from the lawnmower 2.

[0085]

[13] The configurations disclosed in the above-described embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with configurations disclosed in other embodiments, insofar as they do not cause a contradiction, and the embodiments disclosed herein are illustrative, and the embodiments of the present invention are not limited thereto and can be modified as appropriate without departing from the object of the present invention. [Industrial applicability]

[0086] This invention can be applied to work vehicle systems and programs. [Explanation of Symbols]

[0087] S: Work vehicle system 1: Unmanned aircraft 13: Imaging device 2: Lawn mower (work vehicle) 3: Remote control terminal 33:Display section 4: Control device D1: Planned work direction D2:Moving direction D3: Direction of steering D4: Forward direction of the aircraft IM1: Acquired image R: Work path

Claims

1. Remotely operated work vehicles, An unmanned aerial vehicle equipped with an imaging device, Display unit and A control device is provided, The control device is The image captured by the aforementioned imaging device is acquired from the unmanned aerial vehicle. By performing image analysis on the captured images, the direction of movement of the work vehicle is determined. A work vehicle system that controls the display unit to overlay and display an image indicating the direction of movement with the captured image.

2. The control device is Obtain the planned work direction, The work vehicle system according to claim 1, wherein the display unit is controlled to overlay and display an image indicating the planned work direction and the captured image.

3. The work vehicle is equipped with a remote control terminal for remotely operating the work vehicle, The control device is The direction of operation of the work vehicle is obtained from the remote control terminal, The work vehicle system according to claim 2, wherein the display unit is controlled to overlay and display an image indicating the steering direction of the work vehicle with the captured image.

4. The control device is By performing image analysis on the captured images, the forward direction of the work vehicle is determined. The work vehicle system according to claim 1, wherein the display unit is controlled to superimpose and display an image indicating the forward direction of the machine and the captured image.

5. An operation assistance program that runs on a system that assists in the remote operation of a work vehicle, A process for acquiring images captured by the imaging device of an unmanned aerial vehicle from the said unmanned aerial vehicle, The process involves determining the direction of movement of the work vehicle by performing image analysis on the captured image, A program that causes a computer to perform the following process: controlling the display unit to overlay and display an image indicating the direction of movement with the captured image.

Images