Agricultural robot control device

Abstract

To control a path of a robot for farm work by using only a video shot by an imaging device such as an unmanned flying body having a camera independently of a robot.SOLUTION: A robot control device for farm work converts images obtained by imaging a robot for farm work provided with a prescribed marker on an upper surface in a farm field and imaging the periphery of the robot by an imaging device independent of the robot into an image in a coordinate system of the robot for farm work with projective transformation using the marker in the image, and controls a path of the robot for farm work based on the image after the conversion.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control device for an agricultural robot, and more specifically to a control device for controlling the path of an agricultural robot using a camera independent of the robot. [Background technology]

[0002] There are conventional autonomous mobile robots as exemplified in Patent Documents 1 and 2, or Non-Patent Document 1. Many of the conventional autonomous mobile robots control the path of the robot by analyzing images captured by an equipped camera.

[0003] However, when moving over water in outdoor fields, the image may be whited out due to the reflection of sunlight on the water surface, and the camera installed on the robot may not be able to capture normal images. Also, a separate camera is required for reversing. Furthermore, when making a U-turn to avoid an obstacle, a camera that records the side view or a panning camera is required.

[0004] Furthermore, there have been conventional automated driving systems and agricultural work support systems that include route control of agricultural vehicles using a global navigation satellite system (GNSS) or a camera-equipped drone, as exemplified in Patent Documents 3 and 4. In these conventional technologies, the route of the agricultural vehicle is controlled using position information from the GNSS or GPS and camera photography.

[0005] For example, when controlling the path of a mobile robot by analyzing images from a camera installed independently of the robot, such as a drone, the images captured by the drone are taken in the drone's camera coordinates, and it is necessary to establish a relationship with the coordinates of the mobile robot whose path should be controlled. However, conventional automatic driving systems and agricultural work support systems do not pay attention to the relationship between the drone's camera coordinates and the mobile robot's coordinates, and do not disclose controlling the path of a mobile robot using only images captured by a drone. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2016-220654 [Patent Document 2] Patent Publication No. 2013-252087 [Patent Document 3] Patent Publication No. 2018-164424 [Patent Document 4] Patent Publication No. 2021-7386 [Non-patent literature]

[0007] [Non-Patent Document 1] Development of a small weeding robot for paddy fields (Aigamo robot) (7th report), Gifu Prefectural Information Technology Research Institute Research Report No. 17, 2015, pp. 48-51, published on July 11, 2016 Summary of the Invention [Problem to be solved by the invention]

[0008] The present invention aims to control the path of an agricultural work robot using only images captured by a camera independent of the agricultural work robot. [Means for solving the problem]

[0009] The present invention provides an agricultural robot control device that converts images of an agricultural robot having predetermined markers on its upper surface in a farm field and its surroundings taken by an imaging device independent of the robot into an image in the agricultural robot's coordinate system by projective transformation using the markers in the image, and controls the path of the agricultural robot based on the converted image. Effect of the Invention

[0010] According to the present invention, it is possible to control the path of at least one agricultural robot within a farm field using only images captured by an imaging device independent of the agricultural robot. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a basic configuration of an agricultural robot system including an agricultural robot control device according to one embodiment of the present invention. FIG. [Diagram 2] 1 is a block diagram showing a configuration of an agricultural robot control device according to an embodiment of the present invention; [Diagram 3] 1 is a block diagram showing a configuration of an agricultural robot according to an embodiment of the present invention. FIG. [Figure 4] 1 is a diagram showing an unmanned aerial vehicle according to one embodiment of the present invention. [Diagram 5] A block diagram showing the configuration of the main body of an unmanned aerial vehicle of one embodiment of the present invention. [Figure 6] FIG. 1 shows a marker according to one embodiment of the present invention. [Figure 7] FIG. 13 is a diagram for explaining projective transformation using a marker according to an embodiment of the present invention. [Figure 8] FIG. 13 is a diagram for explaining projective transformation using a marker according to an embodiment of the present invention. [Figure 9] 1 is a diagram showing a configuration of an agricultural robot system including an agricultural robot control device according to an embodiment of the present invention. FIG. [Figure 10] FIG. 13 is a diagram showing the configuration of an agricultural work robot system including an agricultural work robot control device according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] An embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the basic configuration of an agricultural robot system including an agricultural robot control device according to one embodiment of the present invention. The agricultural robot system includes an agricultural robot 2 and a photographing device 3 independent of the agricultural robot 2, in addition to an agricultural robot control device 1. The agricultural robot control device 1 can be configured as an independent device, or detachably attached to the agricultural robot 2 and configured as a part of or an associated device. The agricultural robot control device 1 can wirelessly communicate with the agricultural robot 2 and the photographing device 3 via a communication network 7. When attached to the agricultural robot 2, the agricultural robot control device 1 can be connected to the agricultural robot 2 by wire or by short-range wireless communication such as Bluetooth (registered trademark).

[0013] When the working robot control device 1 is configured as an independent device, it has a configuration similar to that of a so-called information processing terminal, and includes a central processing unit (CPU) 101, a storage device 102, and various I / Fs 104, which are connected to each other via a bus 103, as illustrated in FIG. 2 for example. The various I / Fs 104 are used as a general term including an input IF, an output I / F, an external storage I / F, an external communication I / F, etc., and each I / F is connected to a corresponding input / output means 105 such as a keyboard, a mouse, a communication adapter, etc., a display means 106 such as a CRT or an LCD, and an external storage device 107 such as a semiconductor memory or HDD connected via USB, etc. Examples of the embodiment include a notebook PC, a tablet PC, a smartphone, etc. When the working robot control device 1 is attached to an agricultural robot 2 and configured as a part of it or as an associated device, a single board computer such as Raspberry Pi (registered trademark) can be used.

[0014] The arithmetic processing device 101 executes a predetermined program stored in the storage device 102, transmits control commands to the image capture device 3 via the input / output means 105 for controlling camera capture, receives captured images from the image capture device 3 and stores them in the storage device 102, and then performs image processing including projective transformation, the details of which will be described later. The arithmetic processing device 101 generates control commands for controlling the path of the agricultural robot 2 based on the processed image, and transmits the control commands to the agricultural robot 2 via the input / output means 105. If the image capture device 3 is a device that can move in the air or on the ground, the arithmetic processing device 101 can transmit control commands to the image capture device 3 for controlling flight or running. The display device 106 can display camera images captured by the image capture device 3, images after image processing by the arithmetic processing device 101, and the like. While viewing images and the like on the display device 106, the operator can perform various remote operations by operating a mouse or keyboard via the input / output device 105, or by operating the touch panel if the display device 106 is configured with a touch panel.

[0015] The agricultural robot 2 is a self-propelled work vehicle (robot) for performing agricultural tasks such as transplanting seedlings, fertilizing, plowing, weeding, harvesting, and threshing in a field, which is farmland (rice field, field). FIG. 3 is a block diagram showing the configuration of the agricultural robot 2 according to an embodiment of the present invention. The agricultural robot 2 has at least one flat upper surface on which a marker, which will be described in detail later, can be placed. The agricultural robot 2 includes a traveling vehicle body 21 capable of traveling in the field, and a working device 22 for performing the various agricultural tasks described above. The traveling vehicle body 21 includes an engine or electric motor (not shown) as a power source, and a power transmission device (not shown) that transmits the power of the engine or electric motor to the drive wheels and the working device 22, and can travel freely on farm roads and inside the field. The engine is a heat engine such as a diesel engine or a gasoline engine. The electric power source is a storage battery.

[0016] The working device 22 may be, for example, a seedling planting device in a seedling transplanter, a fertilizer applicator in a fertilizer applicator, or a tilling rotary in a tractor. Also, for example, in a combine harvester, it may be a reaping section or a threshing section. The above-mentioned example is merely an example, and is not particularly limited as long as it is a working device for performing agricultural work in a field. Note that the working device 22 has a configuration similar to that of a conventional device, and details thereof will be omitted here.

[0017] The traveling vehicle body 21 can control each part by electronic control using a built-in control unit 210. The control unit 210 is connected to an ECU 211, which is a control device for an engine or an electric motor, an automatic steering device 212, and a communication control unit 213, and can control the power transmission to the drive wheels and the working device 22, and control the communication with the automatic steering device 212, the working robot control device 1, and the unmanned aerial vehicle 3. The automatic steering device 212 can perform automatic driving including changing the traveling course of the traveling vehicle body 21 by operating a control handle (not shown) or a crawler (not shown) provided on the traveling vehicle body 21 using a control signal from the control unit 210. Note that the traveling vehicle body 21 may be provided with an antenna (not shown) for GPS or GNSS to receive radio waves from a satellite and obtain position information at predetermined intervals at predetermined times. In addition, a camera may be provided and used for purposes other than changing the traveling course, such as checking the road surface condition underfoot.

[0018] As long as the photographing device 3 can photograph the agricultural robot 2 and the surrounding field surface from above, other photographing devices such as an unmanned aerial vehicle with a camera, a single fixed camera (fixed camera), a single camera (mobile camera) that can move on the ground, or multiple fixed cameras arranged in predetermined areas within the field can be used. In the following explanation, an unmanned aerial vehicle with a camera is used as an example of the photographing device 3, but the present invention is not limited to this.

[0019] The unmanned aerial vehicle 3 having a camera (hereinafter referred to as the unmanned aerial vehicle 3) can be configured as a so-called drone, for example, as illustrated in FIG. 4. However, the invention is not limited to this, and it is sufficient if the unmanned aerial vehicle can fly unmanned over a farm field. The unmanned aerial vehicle 3 has, for example, a rotor 32 provided at the tip of each of four arm sections 31b extending radially from a main body section 31a. The rotor 32 rotates by transmitting the drive of a motor (not shown) provided on each arm section 31b. The unmanned aerial vehicle 3 rises / descends according to the magnitude of the wind pressure generated by the rotation of the rotor 32. A camera 33 is provided at the bottom of the main body section 31a. The camera 33 is attached, for example, by two-point support so that its attitude can be freely changed.

[0020] A receiving antenna 35 is provided on the upper part of the main body 31a. The receiving antenna 35 is a GPS antenna, a GNSS antenna, or the like. The unmanned aerial vehicle 3 can receive signals related to its own position (latitude and longitude) via the receiving antenna 35 and recognize its position. Inside the main body 31a, as shown in the block diagram of FIG. 5, a control unit 36 ​​for flight control, a communication unit 37, a memory unit 38, a battery 39, and the like are provided. The control unit 36 ​​is connected to the receiving antenna 35, the camera 33, a motor, the communication unit 37, the memory unit 38, the battery 39, and the like. The motor constitutes a drive unit. The communication unit 37 transmits and receives various signals between the unmanned aerial vehicle 3 and the work robot control device 1 and the agricultural work robot 2 (traveling body 21).

[0021] Next, the path control of the agricultural work robot 2 by the working robot controller 1 in the agricultural work robot system 10 of one embodiment of the present invention will be described. Assume now that the agricultural work robot 2 is traveling or waiting to travel in a field. The working robot controller 1 sends a control command for flight control to the unmanned aerial vehicle 3, and moves the unmanned aerial vehicle 3 to the sky above the agricultural work robot 2. The agricultural work robot controller 1 sends a control signal for photography control to the unmanned aerial vehicle 3, causes the camera 34 to photograph the agricultural work robot 2 and its surroundings, and stores the photographed images in the memory unit 38 and transmits them to the agricultural work robot controller 1.

[0022] The agricultural work robot control device 1 stores the captured images received from the unmanned aerial vehicle 3 in the memory unit 102, and then performs image processing including projective transformation, which will be described below. A marker 24 as shown in FIG. 6 is attached to one top surface of the agricultural work robot, and this marker 24 is included in the captured images from the unmanned aerial vehicle 3. The marker 24 is a planar (two-dimensional) marker, and for example, a square planar marker of a black-and-white (binary) image can be used. The marker 24 has known coordinates (hereinafter referred to as marker coordinates or robot coordinates) on one top surface of the agricultural work robot 2.

[0023] Since the camera 34 of the unmanned aerial vehicle 3 captures images from an oblique direction above the agricultural work robot 2, the captured image of the agricultural work robot 2 including the marker 24 looks, for example, as shown in FIG. 7. In the experiment of FIG. 7, a small robot 2A simulating the agricultural work robot 2 is used. In FIG. 7, the image is taken in the coordinates of the camera 34 of the unmanned aerial vehicle 3 (hereinafter referred to as camera coordinates), so the marker 24 is not recognized as a square in the captured image. Therefore, with the image in the camera coordinates, it is difficult to control the travel path of the agricultural work robot 2 along a work route that is usually aligned in approximately one row in a farm field.

[0024] Therefore, a pseudo augmented reality space is created using projective transformation. That is, a pseudo image from another viewpoint (robot (marker) coordinates) is generated by performing projective transformation. Specifically, a homography matrix (projective transformation matrix: H-Matrix) for projectively transforming an image in the camera coordinates into an image in the robot (marker) coordinates, which is another viewpoint, is obtained to create an image after projective transformation. FIG. 8 shows an image after projective transformation of the captured image in FIG. 7. The image is transformed so that the marker 24 becomes a square in the image, and an image (front image) in the robot (marker) coordinates is obtained in a pseudo manner. By using the image after projective transformation exemplified in FIG. 8, the agricultural robot control device 1 can easily control the travel path of the agricultural robot 2 (small robot 2A) along a work path that is aligned in approximately one row. For example, the agricultural robot control device 1 can send a course control command such as forward, backward, or right turn to the agricultural robot 2 based on the image after projective transformation to control the travel of the agricultural robot 2.

[0025] Fig. 9 is a block diagram showing the configuration of an agricultural work robot system including an agricultural work robot control device of one embodiment of the present invention. In Fig. 9, the agricultural work robot control device 1 is an independent device (PC) that can wirelessly communicate with an agricultural work robot 2 traveling in a field and an unmanned aerial vehicle 3 via a communication network 7. Note that there may be two or more agricultural work robots 2.

[0026] Fig. 10 is a block diagram showing the configuration of an agricultural robot system including an agricultural robot control device according to another embodiment of the present invention. The agricultural robot control device 1 is attached to an agricultural robot 2, and can wirelessly communicate with an unmanned aerial vehicle 3 via a communication network 7, and can be connected to the agricultural robot 2 by wire or by short-range wireless communication such as Bluetooth (registered trademark). In the embodiment of Fig. 10, a single unmanned aerial vehicle 3 photographs multiple (e.g., three) different types of agricultural robots 2, making it possible to control the travel path of each robot.

[0027] The embodiments of the present invention have been described with reference to the drawings. However, the present invention is not limited to these embodiments. The present invention can be implemented in various forms with improvements, modifications, and variations based on the knowledge of those skilled in the art without departing from the spirit of the present invention. [Explanation of symbols]

[0028] 1. Agricultural robot control device 2. Agricultural robots 3 Unmanned aerial vehicles, drones 7. Communication Networks 21 Running vehicle 22 Work equipment 24 Marker 31a Main body 31b Arm section 32 Rotor 33 Camera 35 Receiving antenna 37 Communications Department 38 Memory section 39 Battery 36, 101, 210, 211 Control unit, arithmetic processing unit, ECU 102 Storage section 103 Bus 104 Various I / Fs 105 I / O Devices 106 Display device 107 External storage device 212 Automatic Steering Device 213 Communication Control Unit

Claims

1. An agricultural robot control device that converts an image of an automatically steerable agricultural robot having predetermined markers on its upper surface in a farm field and its surroundings taken from above by a camera independent of the robot into an image in the coordinate system of the agricultural robot by projective transformation using the markers in the image, and controls the path of the agricultural robot along an aligned work path based on the converted image and controls automatic travel along that path.

2. 2. The agricultural robot control device of claim 1, which is wirelessly connected to the imaging device and the agricultural robot, sends imaging control signals to the imaging device, receives the images captured from above from the imaging device, and sends the path and automatic driving control signals to the agricultural robot.

3. 2. The agricultural robot control device of claim 1, which is attached to the agricultural robot, is wirelessly connected to the imaging device, sends imaging control signals to the imaging device, receives the images captured from above from the imaging device, and generates control signals for the path and automatic driving of the agricultural robot.

4. The agricultural robot control device according to any one of claims 1 to 3, wherein the photographing device includes an unmanned aerial vehicle having a camera.

5. The agricultural robot control device according to any one of claims 1 to 4, wherein the marker includes a black and white flat marker.

6. 6. An agricultural robot system comprising the imaging device according to claim 1, the agricultural robot, and the agricultural robot control device.

Images