Work vehicle systems and programs

Abstract

To provide a work vehicle system that allows the operator to understand the surrounding environment of the work vehicle and to intuitively grasp the pre-set work area. [Solution] A work vehicle system comprising a remotely operated work vehicle 2, an unmanned aerial vehicle 1 having an imaging device 13 and a satellite positioning device, a display unit, and a control unit, wherein the control unit acquires work area information relating to the work area of ​​the work vehicle and imaging information relating to the orientation and field of view of the imaging device, acquires the image captured by the imaging device and the position of the unmanned aerial vehicle calculated by the satellite positioning device, identifies the position of the work area in the image based on the work area information, imaging information, image, and position of the unmanned aerial vehicle, and controls the display unit to overlay and display the image showing the work area and the image.

Description

Technical Field

[0001] The present invention relates to a work vehicle system and a program.

Background Art

[0002] Patent Document 1 discloses an excavator management system in which an aerial image captured by a camera is displayed in an image display area. In this excavator management system, the camera is installed on a flying object such as a drone. The aerial image is displayed in the image display area so that an operator can intuitively grasp the positional relationship between the excavator and the objects existing around the excavator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the aerial image of Patent Document 1, the range of the work area of the excavator planned in the work is not shown in the image display area. The excavator operator may have difficulty grasping the work area planned in the work.

[0005] An object of the present invention is to provide a work vehicle system that can grasp the situation around the work vehicle and assist in the remote operation of the work vehicle.

Means for Solving the Problems

[0006] The characteristic configuration of the work vehicle system for achieving the above objectives comprises a remotely operated work vehicle, an unmanned aerial vehicle having an imaging device and a satellite positioning device, a display unit, and a control unit. The control unit acquires work area information relating to the work area of ​​the work vehicle and imaging information relating to the orientation and field of view of the imaging device, acquires the image captured by the imaging device and the position of the unmanned aerial vehicle calculated by the satellite positioning device from the unmanned aerial vehicle, identifies the position of the work area in the image based on the work area information, the imaging information, the image, and the position of the unmanned vehicle, and controls the display unit to overlay and display the image showing the work area and the image.

[0007] According to the above configuration, the image captured by the unmanned aerial vehicle's imaging device and the image indicating the work area are superimposed and displayed on the display unit. This allows the operator remotely controlling the work vehicle to understand the situation around the vehicle and to intuitively grasp the pre-set work area. Therefore, remote operation of the work vehicle becomes easier.

[0008] Another characteristic configuration of the work vehicle system according to the present invention is that the control device uses the image information and the position of the vehicle to perform image analysis on the captured image to identify the previously worked area, which is the area where the work vehicle has traveled, and controls the display unit to overlay and display the image showing the previously worked area with the captured image.

[0009] According to the above configuration, the image captured by the unmanned aerial vehicle's imaging device and the image showing the completed work area are superimposed and displayed on the display unit. This allows the operator to understand the situation around the work vehicle and to intuitively grasp the area where work has already been completed. Therefore, remote control of the work vehicle becomes easier.

[0010] Another characteristic feature of the work vehicle system according to the present invention is that the control device detects a specific object within the captured image by performing image analysis on the captured image.

[0011] When remotely operating a work vehicle, if there is a specific object (such as an obstacle like a stone, rock, stake, or utility pole, or a person or animal) in a position that overlaps with the work vehicle from the operator's perspective, the operator cannot recognize the specific object around the work vehicle. With the above feature configuration, specific objects are automatically detected in the captured image, making it easier to remotely operate the work vehicle.

[0012] Another characteristic configuration of the work vehicle system according to the present invention is that it includes a notification unit, and the control device calculates the straight-line distance between the work vehicle and the specific object by analyzing the captured image, and when the straight-line distance between the work vehicle and the specific object falls below a threshold, the control device causes the notification unit to notify the system of that fact.

[0013] According to the above feature configuration, it is possible to accurately inform the operator that the work vehicle is approaching a specific object.

[0014] Another characteristic configuration of the work vehicle system according to the present invention is that the work vehicle comprises an aircraft body and an indicator provided on the upper part of the aircraft body, and the unmanned aerial vehicle is configured to follow the work vehicle with the indicator as its target.

[0015] According to the above characteristic configuration, the unmanned aerial vehicle follows the work vehicle with the indicator as its target, making it easier for the work vehicle and its surroundings to come into the imaging device's field of view.

[0016] The characteristic configuration of the program for achieving the above objective is an operation assistance program executed by a system that assists in the remote operation of a work vehicle, wherein the program causes a computer to execute the following: a process to acquire work area information relating to the work area of ​​the work vehicle which has been set in advance, and imaging information relating to the orientation and field of view of the imaging device; a process to acquire the image captured by the imaging device and the position of the unmanned aerial vehicle calculated by the satellite positioning device from the unmanned aerial vehicle; a process to identify the position of the work area in the image based on the work area information, the imaging information, the image, and the position of the unmanned aerial vehicle; and a process to control the display unit to display the image showing the work area and the image overlaid on top of each other.

[0017] According to the above characteristic configuration, an image captured by the imaging device of the unmanned aircraft and an image indicating the work area are overlapped and displayed on the display unit. As a result, the operator can grasp the situation around the work vehicle and can intuitively grasp the preset work area more easily. Therefore, it becomes easier to remotely control the work vehicle.

Brief Description of the Drawings

[0018] [Figure 1] It is a diagram showing a work vehicle system. [Figure 2] It is a block diagram showing a work vehicle system. [Figure 3] It is a diagram showing the follow-up flight of the unmanned aircraft to the lawn mower. [Figure 4] It is a plan view showing an example of a process of overlapping and displaying a work area image and an imaging image. [Figure 5] It is a front view showing an example of a composite image displayed on the display unit. [Figure 6] It is a flowchart showing an example of a process of overlapping and displaying a work area image and an imaging image. [Figure 7] It is a flowchart showing an example of a process of recording the position coordinates of the already-worked area image. [Figure 8] It is a flowchart showing an example of a process of overlapping and displaying an already-worked area image and an imaging image. [Figure 9] It is a plan view showing an example of a process of detecting a specific object. [Figure 10] It is a front view showing an example of a composite image displayed on the display unit. [Figure 11] It is a flowchart showing an example of a process of detecting a specific object.

Embodiments for Carrying Out the Invention

[0019] The work vehicle system and program according to the present invention will be described below with reference to the drawings. The work vehicle system S is a system that assists in the remote operation of a work vehicle that travels through area F. The program is an operation assistance program that causes a computer to execute the processing in the work vehicle system S. In this embodiment, area F is a grassland with long grass.

[0020] As shown in Figures 1 and 2, the work vehicle system S comprises an unmanned aerial vehicle 1 having an imaging device 13 and a satellite positioning device 14, a remotely operated lawnmower 2 (an example of a work vehicle), a remote control terminal 3 for remotely operating the lawnmower 2, a control device 4 (corresponding to a control device), and a management device C. The work vehicle system S assists in the remote operation of the lawnmower 2 by displaying images IM1 captured by the imaging device 13 of the unmanned aerial vehicle 1 on the display unit 33 (corresponding to a notification unit) of the remote control terminal 3.

[0021] [Unmanned aircraft] As shown in Figures 1 and 2, the unmanned aerial vehicle 1 is an autonomously flying aircraft. The unmanned aerial vehicle 1 is an unmanned aircraft that cannot carry a person due to its structure. The unmanned aerial vehicle 1 can send and receive information with the control device 4 via wireless communication. In this embodiment, the unmanned aerial vehicle 1 is a drone equipped with multiple rotors 11. However, it is not limited to this, and the unmanned aerial vehicle 1 may be an unmanned helicopter.

[0022] The unmanned aerial vehicle 1 comprises a battery 12 as a power source, an inertial measuring device 15, and a control device 16 for controlling the unmanned aerial vehicle 1.

[0023] The battery 12 supplies power to a motor located on the rotor blade 11. The rotor blade 11 is driven around an axis that extends vertically. The unmanned aerial vehicle 1 flies by driving multiple rotor blades 11. However, the unmanned aerial vehicle 1 may also be equipped with an engine as a power source, and the rotor blades 11 may be driven by the power of the engine.

[0024] The imaging device 13 is capable of imaging the lawnmower 2 and the surrounding area F. In this embodiment, the imaging device 13 is a stereo camera. The imaging device 13 has two cameras, and the parallax between these two cameras is used to detect the relative position of the object to be measured with respect to the imaging device 13. However, the imaging device 13 may also be a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera.

[0025] In this embodiment, the captured image IM1 captured by the imaging device 13 is a still image. However, the captured image IM1 may also be a video or moving image that displays still images in sequence. The angle θ1 in Figure 1 indicates the field of view of the imaging device 13.

[0026] The satellite positioning device 14 receives positioning signals from artificial satellites used in GNSS (Global Navigation Satellite Systems, such as GPS, QZSS, Galileo, GLONASS, BeiDou, etc.). The positioning signals are output to the control device 16.

[0027] The inertial measurement unit 15 detects inertial information (yaw angle, pitch angle, roll angle, etc.) of the unmanned aerial vehicle 1. The inertial measurement unit 15 includes an acceleration sensor, a rotational angular acceleration sensor, a gyroscope sensor, a magnetic field sensor, and the like.

[0028] The control device 16 includes a storage unit and a CPU (Central Processing Unit) that executes programs. The storage unit is composed of, for example, an HDD, ROM, or non-volatile memory.

[0029] The control device 16 calculates the self-position BP (see Figure 4) of the unmanned aircraft 1 based on the positioning signal.

[0030] [Grass trimmer] As shown in Figures 1 and 2, the grass trimmer 2 can be remotely controlled by a remote control terminal 3. The grass trimmer 2 comprises a machine body, a grass cutting unit 22 as a work device, an engine 23, a cover member 24 that covers the machine body, an inertial measuring device 25, an indicator lamp 26 (corresponding to an indicator unit) that notifies information related to the work, and an indicator 27 provided on the top of the machine body.

[0031] The front of the aircraft is supported by a pair of front wheels 21a, and the rear of the aircraft is supported by a pair of rear wheels 21b.

[0032] The grass cutting unit 22 comprises a cutting blade housing and a cutting blade provided inside the cutting blade housing. The cutting blade is rotatable around an axis extending in the vertical direction. The cutting blade is held in the cutting blade housing. The grass cutting unit 22 is provided between the front wheel 21a and the rear wheel 21b in the front-rear direction of the machine.

[0033] The engine 23 outputs driving force to the front wheels 21a, rear wheels 21b, and grass cutting unit 22 via a transmission or the like. However, the grass cutter 2 may also be equipped with an electric motor instead of the engine 23. In this case, the front wheels 21a, rear wheels 21b, and grass cutting unit 22 are driven by the electric motor.

[0034] A cover member 24 is provided on the upper part of the machine. The cover member 24 covers the machine, including the grass-cutting unit 22, from above. The color of the cover member 24 is different from the area F in which the grass-cutting machine 2 operates. For example, the color of area F is green, while the color of the cover member 24 is orange.

[0035] The inertial measurement unit 25 detects inertial information (yaw angle, pitch angle, roll angle, etc.) of the lawnmower 2. The inertial measurement unit 25 includes an acceleration sensor, a rotational angular acceleration sensor, a gyroscope sensor, a magnetic field sensor, and the like.

[0036] As shown in Figure 3, multiple alarm lamps 26 are provided on the upper part of the aircraft. The multiple alarm lamps 26 are arranged in the front-to-back direction of the aircraft. In this embodiment, three alarm lamps 26 are arranged in a single line in the front-to-back direction of the aircraft.

[0037] If a communication error occurs at the remote control terminal 3 and the machine stops, the rearmost of the three indicator lamps 26 will light up. Also, based on the inertial information of the grass cutter 2, if the tilt angle of the machine relative to the horizontal plane exceeds a set angle, the frontmost of the three indicator lamps 26 will light up. When the grass cutting unit 22 is in operation, the middle of the three indicator lamps 26 will blink. The three indicator lamps 26 will light up in different colors.

[0038] The indicator 27 is captured by the imaging device 13 of the unmanned aerial vehicle 1, allowing the unmanned aerial vehicle 1 to recognize the position of the lawnmower 2. In this embodiment, as shown in Figure 3, the indicator 27 is a code (for example, a QR code®) that can be read by the imaging device 13. However, the indicator 27 may be a circle ("〇") or other shapes or characters. The indicator 27 is located on the cover member 24 at the front of the aircraft. However, the indicator 27 may be located at any position detectable by the imaging device 13, such as the rear of the cover member 24. The unmanned aerial vehicle 1 is configured to automatically track the lawnmower 2 using the indicator 27 as a target.

[0039] [Remote control terminal] As shown in Figure 2, the remote control terminal 3 includes a gear shifting device 31 for adjusting forward and reverse speeds, a turning device 32 for selecting left and right turns, and a display unit 33. The remote control terminal 3 is a remote control for operating the lawnmower 2. The remote control terminal 3 can send and receive information with the lawnmower 2 or the control device 4 via wireless communication. Although not shown, the remote control terminal 3 is equipped with other control devices for operation (such as adjusting engine speed and starting / stopping the engine 23).

[0040] The gear shift control device 31 is swung in the forward and backward directions to change to the forward, neutral, or reverse position. When the gear shift control device 31 is tilted forward (forward position), the remote control terminal 3 outputs a forward signal to the grass trimmer 2. When the gear shift control device 31 is upright (neutral position), the remote control terminal 3 outputs a stop signal to the grass trimmer 2. When the gear shift control device 31 is tilted backward (reverse position), the remote control terminal 3 outputs a reverse signal to the grass trimmer 2.

[0041] The turning control device 32 is swung from side to side, changing to a left turn position, a straight-ahead position, or a right turn position. When the turning control device 32 is tilted to the left (left turn position), the remote control terminal 3 outputs a left turn signal to the grass trimmer 2. When the turning control device 32 is upright (straight-ahead position), the remote control terminal 3 outputs a straight-ahead signal to the grass trimmer 2. When the turning control device 32 is tilted to the right (right turn position), the remote control terminal 3 outputs a right turn signal to the grass trimmer 2. The gear shift control device 31 and the turning control device 32 may be composed of a single joystick.

[0042] [Display] The display unit 33 is a display capable of displaying characters, images, and the like. The display unit 33 can display captured images IM1 taken by the unmanned aerial vehicle 1.

[0043] [Management device] As shown in Figure 2, the management device C is a terminal located remotely from the lawnmower 2. The management device C has a storage unit and a CPU (Central Processing Unit) that executes programs. The storage unit is composed of, for example, an HDD, ROM, or non-volatile memory.

[0044] [Control device] As shown in Figure 2, the control device 4 includes a work area acquisition unit 41, an imaging information acquisition unit 42, a self-position acquisition unit 43, an image acquisition unit 44, a work area identification unit 45, a previously worked area identification unit 46, an image generation unit 47, a detection unit 48, a distance calculation unit 49, and a display control unit 50.

[0045] The control device 4 is a terminal located outside the remote control terminal 3. The control device 4 has a storage unit and a CPU (Central Processing Unit) that executes programs. The storage unit is composed of, for example, an HDD, ROM, or non-volatile memory. In this embodiment, the control device 4 is a PC terminal or a mobile terminal. The operation assistance program in the work vehicle system S is executed by the PC terminal or mobile terminal.

[0046] The work area acquisition unit 41 acquires work area information EP relating to the pre-set work area E1 of the brush cutter 2. The work area E1 is the area within area F that is artificially set as the target area for work performed by the brush cutter 2. In this embodiment, as shown in Figure 4, the work area E1 is rectangular. The work area information EP is the position coordinates indicating the absolute position of the work area E1 within area F.

[0047] In this embodiment, the work area information EP has the position coordinates of the four corner points (points P1, P2, P3, and P4) of the work area E1. The position of point P1 is indicated by x1 and y1. The position of point P2 is indicated by x2 and y2. The position of point P3 is indicated by x3 and y3. The position of P4 is indicated by x4 and y4.

[0048] The imaging information acquisition unit 42 acquires imaging information regarding the orientation and field of view of the imaging device 13. For example, the orientation of the imaging device 13 is calculated over time based on the inertial information of the unmanned aerial vehicle 1. The field of view of the imaging device 13 is stored in the control device 4 in advance.

[0049] The self-position acquisition unit 43 acquires the self-position BP calculated by the satellite positioning device 14 from the unmanned aerial vehicle 1. The self-position acquisition unit 43 acquires the self-position BP over time. The self-position BP is a position coordinate that indicates the absolute position of the unmanned aerial vehicle 1 in area F. As shown in Figure 4, the self-position BP is indicated by α and β. The image acquisition unit 44 acquires the image IM1 captured by the imaging device 13.

[0050] The work area identification unit 45 identifies the position of the work area E1 in the captured image IM1 based on the work area information EP, the imaging information, the captured image IM1, and the position of the local equipment BP. The orientation and field of view of the imaging device 13 included in the imaging information, along with the position of the local equipment BP, uniquely determine the area being photographed by the imaging device 13. Therefore, it is possible to establish a one-to-one correspondence between pixels (points in the captured image IM1) in the captured image IM1 and their absolute positions in the area F. Using this relationship, the work area identification unit 45 identifies the position of the work area E1 in the captured image IM1 and determines whether the work area E1 is shown in the captured image IM1.

[0051] In the examples in Figures 4 and 5, the working region E1 in the identified captured image IM1 is shown by the working region image IM2 (four points (points P'1, P'2, P'3, and P'4) and the lines connecting these points). The four points P1, P2, P3, and P4 in the working region E1 correspond to the four points P'1, P'2, P'3, and P'4 in the captured image IM1. For example, the image coordinates of point P1 are represented as (x'1, y'1) (the same applies to the other points). In other words, the working region identification unit 45 converts the coordinates of the working region information EP (e.g., (x1, y1)) to the image coordinates of the captured image IM1 (e.g., (x'1, y'1)).

[0052] The previously worked area identification unit 46 identifies the previously worked area E2, which is the area where the brush cutter 2 has worked, by performing image analysis on the captured image IM1 using the imaging information and the machine's position BP. Specifically, the previously worked area E2 is identified by the change in the position (absolute position) of the brush cutter 2. When the brush cutter 2 is working, the area between the position of the brush cutter 2 at a certain time and the position of the brush cutter 2 after a predetermined time (for example, 1 second later) is the area where the grass has been cut and the work has been completed. By accumulating these areas, the previously worked area E2 is identified. Note that the image coordinates in the captured image IM1 and the absolute position (position coordinates) in area F can be converted using the imaging information and the machine's position BP based on the above relationship. The identification of the previously worked area E2 is performed by converting the position (image coordinates) of the brush cutter 2 in the captured image IM1 at multiple time points into absolute position (position coordinates).

[0053] The image generation unit 47 generates a work area image IM2 that shows the work area E1. The work area image IM2 is shown in a different color from the color of area F.

[0054] Furthermore, the image generation unit 47 generates a previously worked area image IM3 that shows the previously worked area E2. The previously worked area image IM3 is shown in a different color from the colors of area F and the working area image IM2.

[0055] The detection unit 48 detects the lawnmower 2 and specific objects SP within the captured image IM1 by performing image analysis on the captured image IM1. Specific objects SP include obstacles such as stones, rocks, stakes, and utility poles within the region F, or people and animals.

[0056] [Regarding the process of overlaying the work area image and the captured image] Next, an example of the process of overlaying the work area image IM2 and the captured image IM1 will be explained based on the flowchart shown in Figure 6. Note that the steps described below may be performed in any order, and multiple steps may be performed simultaneously, as long as no inconsistencies arise.

[0057] [1-1] Process for acquiring work area information and imaging information The control device 4 acquires work area information EP relating to the pre-set work area E1 of the brush cutter 2, and imaging information relating to the orientation and field of view of the imaging device 13 (step S001 in Figure 6, the same applies hereafter). Specifically, the imaging information acquisition unit 42 acquires imaging information regarding the orientation of the imaging device 13 from the unmanned aerial vehicle 1 via wireless communication. The imaging information acquisition unit 42 also acquires imaging information regarding the field of view of the imaging device 13 that is pre-stored in the control device 4. However, the imaging information acquisition unit 42 may also acquire imaging information relating to the field of view from the unmanned aerial vehicle 1.

[0058] The work area acquisition unit 41 acquires work area information EP from the management device C via wireless communication. However, the imaging information acquisition unit 42 may acquire work area information EP that has been pre-stored in the control device 4.

[0059] [1-2] Process to acquire captured images and the position of the aircraft The control device 4 acquires the captured image IM1 taken by the imaging device 13 and the self-position BP of the unmanned aerial vehicle 1 calculated by the satellite positioning device 14 from the unmanned aerial vehicle 1 (step S002). Specifically, the self-position acquisition unit 43 acquires the self-position BP from the unmanned aerial vehicle 1 via wireless communication. In addition, the captured image acquisition unit 44 acquires the captured image IM1 from the unmanned aerial vehicle 1 via wireless communication.

[0060] [1-3] Process to identify the location of the work area in the captured image. The work area identification unit 45 analyzes the captured image IM1 to determine whether or not the work area E1 is shown in the captured image IM1 (step S003). In the example in Figure 4, the entire work area E1 is within the field of view of the imaging device 13, and the entire work area E1 is shown in the captured image IM1.

[0061] If the work area E1 is shown in the captured image IM1 (step S003: Yes), the work area identification unit 45 identifies the position of the work area E1 in the captured image IM1. In this embodiment, the work area identification unit 45 identifies points P'1, P'2, P'3, and P'4 on the captured image IM1 that correspond to points P1, P2, P3, and P4. Based on the above, the work area identification unit 45 identifies the position of the work area E1 in the captured image IM1 based on the work area information EP, the imaging information, the captured image IM1, and the position of the machine BP (step S004).

[0062] [1-4] Process for generating composite images The image generation unit 47 generates a work area image IM2 that represents the work area E1. The image generation unit 47 also superimposes the work area image IM2 onto the captured image IM1 at the position corresponding to the work area E1. As a result, the image generation unit 47 generates a composite image IM4 in which the work area image IM2 is superimposed on the captured image IM1 (step S005).

[0063] [1-5] Process to display the composite image on the display unit. The display control unit 50 controls the display unit 33 to display the composite image IM4 on the display unit 33 (step S006) and completes the process normally. In other words, the display control unit 50 controls the display unit 33 to overlay and display the work area image IM2, which shows the work area E1, and the captured image IM1. If the work area E1 is not shown in the captured image IM1 (step S003: No), the display control unit 50 controls the display unit 33 to display the captured image IM1 as is on the display unit 33 (step S007) and completes the process normally.

[0064] [Regarding the process of storing the position coordinates of the previously worked area] Next, an example of the process for storing the position coordinates of the previously worked area E2 will be explained based on the flowchart shown in Figure 7. Note that the steps described below may be performed in any order, and multiple steps may be performed simultaneously, as long as no inconsistencies arise. Furthermore, the steps described below may be performed simultaneously with the process described above.

[0065] Since S011 and S012 are the same as the processing content of S001 and S002 described above, their explanation will be omitted.

[0066] [2-1] Process to store the position coordinates of the lawnmower The work area identification unit 45 analyzes the captured images IM1 at multiple time points to determine whether or not there has been a change in the position of the lawnmower 2 (step S013). If there is no change in the position (absolute position) of the lawnmower 2 (step S013: No), the process is completed successfully.

[0067] If there is a change in the position (absolute position) of the brush cutter 2 (step S013: Yes), the previously worked area identification unit 46 identifies the absolute position (position coordinates) of the previously worked area E2 from the change in the position (change in absolute position) of the brush cutter 2 at multiple time points (step S014). Next, the previously worked area identification unit 46 stores the position coordinates of the previously worked area E2 in the control device 4, linked to the identified time (step S015).

[0068] [Regarding the process of overlaying the previously created work area image and the captured image] Next, an example of the process of overlaying the previously worked area image IM3 and the captured image IM1 will be explained based on the flowchart shown in Figure 8. Note that the steps described below may be performed in any order, and multiple steps may be performed simultaneously, as long as no inconsistencies arise. Furthermore, the steps described below may be performed simultaneously with the process described above.

[0069] [3-1] Process to acquire the previously worked area and imaging information The control device 4 acquires the completed work area E2 and imaging information (step S021 in Figure 8, the same applies hereafter). Specifically, the completed work area identification unit 46 acquires the position coordinates of the completed work area E2 for each time period stored in the control device 4. The imaging information acquisition unit 42 acquires imaging information about the orientation of the imaging device 13 from the unmanned aerial vehicle 1 via wireless communication. The imaging information acquisition unit 42 also acquires imaging information about the field of view of the imaging device 13 that is previously stored in the control device 4.

[0070] Step S022 is the same process as step S002 described above, so its explanation is omitted.

[0071] [3-2] Process to identify the location of the work area in the captured image. The previously worked area identification unit 46 analyzes the captured image IM1 to determine whether or not the previously worked area E2 is shown in the captured image IM1 (step S023). If the previously worked area E2 is shown in the captured image IM1 (step S023: Yes), the previously worked area identification unit 46 identifies the position (image coordinates) of the previously worked area E2 in the captured image IM1 that corresponds to the acquired position coordinates (absolute position) of the previously worked area E2 (step S024).

[0072] [3-3] Process for generating a composite image The image generation unit 47 generates a previously worked area image IM3 that shows the previously worked area E2. The image generation unit 47 also composites the previously worked area image IM3 onto the captured image IM1 at the position corresponding to the previously worked area E2. As a result, the image generation unit 47 generates a composite image IM4 in which the previously worked area image IM3 is superimposed on the captured image IM1 (step S025).

[0073] [3-4] Process to display the composite image on the display unit. The display control unit 50 controls the display unit 33 to display the composite image IM4 on the display unit 33. In other words, the display control unit 50 displays the previously worked area image IM3 and the captured image IM1 superimposed (step S026), and the process is completed successfully. If the previously worked area E2 is not shown in the captured image IM1 (step S023: No), the display control unit 50 controls the display unit 33 to display the captured image IM1 as is on the display unit 33 (step S027), and the process is completed successfully.

[0074] In this embodiment, as shown in Figure 5, the image generation unit 47 generates a composite image IM4 in which the work area image IM2 and the previously worked area image IM3 are superimposed on the captured image IM1. In other words, the display control unit 50 displays the work area image IM2, the previously worked area image IM3, and the captured image IM1 superimposed.

[0075] [Regarding the process for detecting specific objects] Next, an example of the process for detecting a specific object SP in the captured image IM1 will be explained based on the flowchart shown in Figure 11. Note that the steps described below may be performed in any order, and multiple steps may be performed simultaneously, as long as no inconsistencies arise.

[0076] Since S031 is the same process as S002 described above, its explanation will be omitted.

[0077] [4-1] Process for identifying specific objects in captured images As shown in Figures 9 and 10, the detection unit 48 analyzes the captured image IM1 to determine whether or not a specific object SP is shown in the captured image IM1 (step S032). The detection unit 48 detects the specific object SP in the captured image IM1 using trained data such as a neural network trained by machine learning (deep learning). However, the detection unit 48 may also detect the specific object SP based on the difference between the color of the specific object SP and the color of region F. The process in step S032 is repeated until it is determined that the specific object SP is shown in the captured image IM1 (step S032: No).

[0078] [4-2] Process for calculating the distance between the lawnmower and a specific object If it is determined that a specific object SP is shown in the captured image IM1 (step S032: Yes), the distance calculation unit 49 calculates the straight-line distance D between the lawnmower 2 and the specific object SP by performing image analysis on the captured image IM1, as shown in Figure 10 (step S033). Specifically, the distance calculation unit 49 calculates the straight-line distance D between the lawnmower 2 and the specific object SP in region F from the position coordinates of the lawnmower 2 and the specific object SP in the captured image IM1. However, as shown in Figure 9, the distance calculation unit 49 may also calculate the straight-line distance D between the lawnmower 2 and the specific object SP based on the distance L1 from the position BP to the lawnmower 2 and the distance L2 from the position BP to the specific object SP, without performing image analysis on the captured image IM1.

[0079] [4-3] Process to notify the notification unit Next, if the straight-line distance D between the lawnmower 2 and the specific object SP falls below a preset threshold (step S034: Yes), the lawnmower 2's notification lamp 26 or display unit 33 is activated to indicate this (step S035), and the process is completed successfully. For example, the threshold is approximately 0.5m. However, the threshold may be any value between approximately 0.5m and approximately 1m. In this embodiment, as shown in Figure 10, notification is provided by the illumination of the notification lamp 26 or the display of an exclamation mark image M(!) on the display unit 33. However, notification may also be provided by the display of characters on the display unit 33 or by a warning sound emitted from a speaker or the like.

[0080] [Another embodiment] The present invention is not limited to the embodiments described above. For example, it may be configured as in the following alternative embodiments. In the alternative embodiments described below, components identical to those in the embodiments are denoted by the same numbers and reference numerals as in the embodiments described above.

[0081] [1] In the above-described embodiment, the unmanned aerial vehicle 1 is configured to automatically follow the lawnmower 2 with the indicator 27 as the target. However, the unmanned aerial vehicle 1 may be remotely controlled by a remote control or the like. Also, the lawnmower 2 does not need to be equipped with the indicator 27.

[0082] [2] In the above-described embodiment, the control device 4 is a terminal located outside the remote control terminal 3. However, the control device 4 may be located on the remote control terminal 3 or the management device C. Furthermore, some functions of the control device 4 may be located on the unmanned aerial vehicle 1, the remote control terminal 3, or the management device C.

[0083] [3] In the above embodiment, the remote control terminal 3 is a remote control for remotely operating the lawnmower 2. However, the remote control terminal 3 may be a PC terminal or a mobile terminal.

[0084] [4] In the above-described embodiment, the remote control terminal 3 is equipped with a display unit 33. However, in addition to the display unit 33 of the remote control terminal 3, the control device 4, other PC terminals, or mobile terminals may also be equipped with a display unit 33. Alternatively, instead of the display unit 33 of the remote control terminal 3, the control device 4, other PC terminals, or mobile terminals may be equipped with a display unit 33.

[0085] [5] In the above embodiment, the image generation unit 47 generates a composite image IM4 in which both the work area image IM2 and the previously worked area image IM3 are superimposed on the captured image IM1. However, the image generation unit 47 may generate a composite image IM4 in which either the work area image IM2 or the previously worked area image IM3 is superimposed on the captured image IM1.

[0086] [6] In the above-described embodiment, the display control unit 50 displays the composite image IM4 generated by the image generation unit 47 on the display unit 33. However, the image generation unit 47 does not have to generate the composite image IM4. For example, when the display control unit 50 displays the work area image IM2, the previously worked area image IM3, and the captured image IM1 on the display unit 33, it may display these separate images superimposed on each other.

[0087] [7] In the above embodiment, the work area E1 is rectangular, and the work area image IM2 is the area enclosed by four points (points P'1, P'2, P'3, and P'4) and the lines connecting the points. However, the work area E1 is not limited to this and can be any shape. For example, the work area E1 may be polygonal or circular. In this case, the number of points defining the work area image IM2 may be less than four or more than four. Also, the lines connecting the points on the outer edge of the work area image IM2 may be curves.

[0088] [8] In the above-described embodiment, the work vehicle system S is equipped with a grass cutter 2 as a remotely operated work vehicle. Alternatively, the work vehicle system S may be equipped with a rice transplanter, fertilizer spreader, seed planter, tractor, combine harvester, snowplow, etc.

[0089] [9] The configurations disclosed in the above-described embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with configurations disclosed in other embodiments, insofar as they do not cause a contradiction, and the embodiments disclosed herein are illustrative, and the embodiments of the present invention are not limited thereto and can be modified as appropriate without departing from the object of the present invention. [Industrial applicability]

[0090] This invention can be applied to work vehicle systems and programs. [Explanation of symbols]

[0091] S: Work vehicle system 1: Unmanned aircraft 2: Lawn mower (work vehicle) 26: Notification lamp (Notification Department) 27: Indicator 4: Control device 13: Imaging device 14: Satellite positioning equipment 33:Display section BP: Player's position D: straight line distance E1 :Work area E2: Existing work area EP: Work area information IM1: Acquired image SP:Specific item θ1: Field of view

Claims

1. Remotely operated work vehicles, An unmanned aerial vehicle having an imaging device and a satellite positioning device, Display unit and A control device is provided, The control device is The system acquires work area information relating to the work area of ​​the work vehicle that has been set in advance, and imaging information relating to the orientation and field of view of the imaging device. The captured image taken by the aforementioned imaging device and the aircraft's position calculated by the aforementioned satellite positioning device are obtained from the unmanned aircraft. Based on the work area information, the imaging information, the captured image, and the position of the self-propelled aircraft, the position of the work area in the captured image is identified. A work vehicle system that controls the display unit to overlay and display an image showing the work area with the captured image.

2. The control device is By performing image analysis on the captured image using the aforementioned imaging information and the position of the vehicle, the previously worked area, which is the area where the work vehicle has traveled, is identified. The work vehicle system according to claim 1, wherein the display unit is controlled to overlay and display the image showing the completed work area and the captured image.

3. The control device detects a specific object in the captured image by performing image analysis on the captured image, as described in claim 1.

4. Equipped with a news department, The control device is By performing image analysis on the captured images, the straight-line distance between the work vehicle and the specific object is calculated. The work vehicle system according to claim 3, wherein the control device causes the notification unit to notify the user when the straight-line distance between the work vehicle and the specific object falls below a threshold.

5. The aforementioned work vehicle comprises a machine body and an indicator provided on the upper part of the machine body, The work vehicle system according to claim 1, wherein the unmanned aerial vehicle is configured to follow the work vehicle with the indicator as a target.

6. An operation assistance program that runs on a system that assists in the remote operation of a work vehicle, A process to acquire work area information relating to the work area of ​​the work vehicle which has been set in advance, and imaging information relating to the orientation and field of view of the imaging device, The process involves obtaining from the unmanned aerial vehicle the image captured by the aforementioned imaging device and the position of the unmanned aerial vehicle calculated by the satellite positioning device, A process to determine the position of the work area in the captured image based on the work area information, the imaging information, the captured image, and the position of the self-operated aircraft, A program that causes a computer to perform the following process: controlling the display unit to overlay and display an image representing the work area and the captured image.

Images