Control device

Abstract

To provide a technology for efficiently implementing self-pollination of plants.SOLUTION: A first input unit 410 receives a first type image in which unmanned flying objects 200 are captured, and an acquisition unit 412 acquires, based on the received first type image, respective positions of a plurality of unmanned flying objects 200. A control unit 414 uses a swarm intelligence algorithm for the acquired positions for deriving respective positions to which the plurality of unmanned flying objects 200 are to move, and moves the plurality of unmanned flying objects 200 to the derived positions. A second input unit 420 receives second type images in which the plurality of unmanned flying objects 200 are captured, respectively. A detection unit 422 detects a plant that can be pollinated based on the second type images. When the plant that can be pollinated is detected, the control unit 414 causes at least one of the plurality of unmanned flying objects 200 to stop using the swarm intelligence algorithm and move to a position of the detected plant.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to control technology, and more particularly to a control device for controlling the operation of an unmanned aerial vehicle. [Background technology]

[0002] To pollinate plants that produce fruit through self-pollination, pollinating insects such as honeybees and bumblebees are used, or the pollination can be done manually. For example, honeybees are used for pollination in plant factories that use only artificial light. A plant factory is a system in which the environmental conditions necessary for plant growth are artificially controlled to a high degree within the facility, allowing plants to be cultivated in a planned and stable manner throughout the year (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-226132 Summary of the Invention [Problem to be solved by the invention]

[0004] When plants are cultivated in a large area such as outdoors, it is desirable to efficiently carry out self-pollination of the plants.

[0005] The present invention has been made in view of the above circumstances, and its object is to provide a technique for efficiently carrying out self-pollination in plants. [Means for solving the problem]

[0006] In order to solve the above problem, a control device according to one embodiment of the present invention is a control device for controlling a plurality of unmanned aerial vehicles capable of flying within a cultivation area where plants are cultivated, and is equipped with a first input unit for receiving a first type of image captured by a first type of imaging device that captures an image of the cultivation area, an acquisition unit for acquiring the position of each of the plurality of unmanned aerial vehicles based on the first type of image received by the first input unit, a control unit for deriving a position to which each of the plurality of unmanned aerial vehicles should move by using a swarm intelligence algorithm on the positions acquired by the acquisition unit, and for moving the plurality of unmanned aerial vehicles to the derived positions, a second input unit for receiving a second type of image captured by a second type of imaging device mounted on each of the plurality of unmanned aerial vehicles, and a detection unit for detecting plants that can be pollinated based on the second type of image received by the second input unit. When the detection unit detects a plant that can be pollinated, the control unit selects one of the plurality of unmanned aerial vehicles that: The first unmanned aerial vehicle that captured the second type image containing the pollinable plant is caused to stay at the position where the second type image was captured, and a second unmanned aerial vehicle other than the first unmanned aerial vehicle among the plurality of unmanned aerial vehicles is caused to stay at the position where the second type image was captured. In response to this, the use of the swarm intelligence algorithm is stopped and the detection unit is moved to the location of the plant detected.

[0007] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0008] According to the present invention, self-pollination of plants can be carried out efficiently. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an outline of the operation of a pollination system according to an embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of the pollination system of FIG. 1. [Figure 3] FIG. 3 is a diagram showing the appearance of the unmanned aerial vehicle of FIG. 2. [Figure 4] 4(a)-(b) are diagrams showing an outline of the operation of the unmanned aerial vehicle of FIG. [Figure 5] 5(a) to 5(d) are diagrams showing the appearance of flowers included in the second type images. [Figure 6] FIG. 3 is a diagram illustrating the configuration of a detection unit in FIG. 2. [Figure 7] 3 is a flowchart showing a processing procedure performed by the control device of FIG. 2. DETAILED DESCRIPTION OF THE INVENTION

[0010] Before describing this embodiment in detail, an overview will be provided. This embodiment relates to a pollination system for pollinating self-pollinating plants, such as tomatoes, grown in a cultivation area. Self-pollination is the process by which pollen fertilizes the pistils of the same plant, achieved by bringing the stamens into contact with the pistils. The pollination system of this embodiment performs self-pollination by attaching a rod-shaped pollination rod to an unmanned aerial vehicle, such as a drone, and flying the unmanned aerial vehicle so that the pollination rod contacts flowers. Here, flowers can be in a state where pollination is not possible (hereinafter referred to as an "impossible state") or a state where pollination is possible (hereinafter referred to as a "possible state"). The pollination rod must be brought into contact with plants in a possible state. Efficient self-pollination of plants requires efficient execution of an operation for efficiently finding plants in a possible state (hereinafter referred to as a "search operation") and an operation for pollinating plants in a possible state (hereinafter referred to as a "pollination operation").

[0011] In the pollination system, viable plants are detected by applying machine learning such as a convolutional neural network to images captured by an imaging device mounted on an unmanned aerial vehicle. To streamline the search, multiple unmanned aerial vehicles are used, and a swarm intelligence algorithm is used to control the flight of the multiple unmanned aerial vehicles. This allows the multiple unmanned aerial vehicles to fly evenly throughout the cultivation area. Furthermore, when viable plants are detected, the use of the swarm intelligence algorithm for at least one unmanned aerial vehicle is stopped, and that unmanned aerial vehicle is moved toward the viable plants to streamline the pollination operation.

[0012] FIG. 1 shows an overview of the operation of the pollination system 1000. Multiple plants 100 are cultivated in a cultivation area 150. The plants 100 are self-pollinating plants, such as tomatoes, as described above. Flowers 110 are blooming on the plants 100, but there is a mixture of flowers 110 (plants 100) in a viable state and flowers 110 (plants 100) in an unviable state. The unmanned aerial vehicle 200 flying within the cultivation area 150 is, for example, a drone. The unmanned aerial vehicle 200 detects viable plants 100 through a search operation, and then performs pollination on the viable plants 100 through a pollination operation.

[0013] Figure 2 shows the configuration of the pollination system 1000. The pollination system 1000 includes a first unmanned aerial vehicle 200a to a third unmanned aerial vehicle 200c collectively referred to as unmanned aerial vehicles 200, a first infrared imaging device 300a to a third infrared imaging device 300c collectively referred to as infrared imaging devices 300, a control device 400, and a base station device 500. The control device 400 includes a first input unit 410, an acquisition unit 412, a control unit 414, an output unit 416, a communication unit 418, a second input unit 420, and a detection unit 422. The number of unmanned aerial vehicles 200 included in the pollination system 1000 is not limited to "3", and the number of infrared imaging devices 300 is not limited to "3".

[0014] The unmanned aerial vehicle 200 is equipped with a wireless communication function compatible with wireless communication systems such as a mobile phone system and a wireless LAN (Local Area Network), and can communicate with a base station device 500 via wireless communication. The unmanned aerial vehicle 200 receives a control signal from the control device 400 via the base station device 500 and flies within the cultivation area 150 in accordance with the instructions contained in the control signal. Since known techniques can be used to fly in accordance with the instructions, a description thereof will be omitted here. Figure 3 shows the appearance of the unmanned aerial vehicle 200. An identification number 210 is displayed on the top surface of the unmanned aerial vehicle 200. The identification number 210 is a number used to identify each unmanned aerial vehicle 200 and is different for each unmanned aerial vehicle 200. Hereinafter, both the number displayed on the top surface of the unmanned aerial vehicle 200 and the number used for processing in the pollination system 1000 will be referred to as the identification number 210. Return to Figure 2.

[0015] The infrared imaging devices 300 are imaging devices for visualizing infrared rays emitted from an object and perform motion capture. The first infrared imaging device 300a to the third infrared imaging device 300c are installed so that the imaging ranges of the first infrared imaging device 300a, the second infrared imaging device 300b, and the third infrared imaging device 300c are combined to cover the entire cultivation area 150. Therefore, it can be said that the infrared imaging devices 300 image the cultivation area 150. Depending on the shape of the cultivation area 150, the number of infrared imaging devices 300 required to cover the entire cultivation area 150 may be other than "3." The infrared imaging devices 300 are connected to the control device 400 via a communication cable or a wireless line and sequentially transmit images generated by motion capture (hereinafter referred to as "first type images") to the control device 400.

[0016] The control device 400 is a computer for controlling a plurality of unmanned aerial vehicles 200 that can fly within a cultivation area 150 where plants 100 are cultivated. The first input unit 410 receives a first type image from each of a plurality of infrared imaging devices 300. Identification information for identifying the infrared imaging device 300 that is the sender of the first type image is added to each first type image. Therefore, it is possible to identify which infrared imaging device 300 captured the first type image.

[0017] The acquisition unit 412 acquires the positions of each of the multiple unmanned aerial vehicles 200 based on the first type of image received by the first input unit 410. For example, the acquisition unit 412 performs image recognition processing on the first type of image to determine whether the first type of image includes an identification number 210. If the identification number 210 is included, the acquisition unit 412 identifies each unmanned aerial vehicle 200 by recognizing the value of the identification number 210.

[0018] Here, the first to third infrared imaging devices 300a to 300c are installed in fixed positions, and therefore the angle of view of the images captured by each is fixed. Therefore, when an unmanned aerial vehicle 200 is captured in a first-type image from the first infrared imaging device 300a, the position of the unmanned aerial vehicle 200 is identified based on the position and size of the unmanned aerial vehicle 200 in the first-type image, or the position and size of the identification number 210 in the first-type image. Since well-known techniques can be used to identify such positions, a detailed description will be omitted here. The position of each unmanned aerial vehicle 200 is expressed as position information indicated by the coordinates x, y, and z. The same applies to the first-type image from the second infrared imaging device 300b and the first-type image from the third infrared imaging device 300c. In addition, the position of the unmanned aerial vehicle 200 may be determined by combining the position and size of the unmanned aerial vehicle 200 etc. in the first type images captured by each of the first infrared imaging device 300a to the third infrared imaging device 300c.

[0019] As a search operation, the control unit 414 derives the position to which each of the multiple unmanned aerial vehicles 200 should move by using a swarm intelligence algorithm on the position of each unmanned aerial vehicle 200 acquired by the acquisition unit 412. Here, the swarm intelligence algorithm used is the ABC (Artificial Bee Colony) algorithm, which is inspired by the foraging behavior of honeybee swarms. An example of the search procedure of the ABC algorithm is shown as follows:

[0020] (1) Initialization The control unit 414 randomly sets each search point, i.e., the initial position of each unmanned aerial vehicle 200. At this time, one unmanned aerial vehicle 200 is selected as the best individual x best Set.

number

[0021] (2) Search by worker bees The control unit 414 executes the search as follows.

number

[0022] (3) Search by spectator bees The control unit 414 calculates the relative fitness of each individual, selects an update candidate individual by roulette strategy, and updates the position as follows:

number

[0023] c denotes the individual number selected by roulette wheel selection. ζ is a uniform random number in [0,1]. p i denotes the relative fitness in generation g, and P i f denotes the selection probability in the roulette strategy. c denotes the fitness in generation g.

[0024] (4) Search by scout bees The control unit 414 searches for a new position for an individual that has not been updated a certain number of times as follows.

number

[0025] (5) Updating the optimal value The control unit 414 compares the current generation with the best individual obtained up to the current generation and updates the current best individual. If the fitness of the best individual has converged to the optimal solution, the control unit 414 ends the process, but if not, returns to (2) search by worker bees.

number

[0026] The control unit 414 generates a signal (hereinafter referred to as a "control signal") that includes position information for each unmanned aerial vehicle 200. The control signal includes a combination of the identification number 210 and position information for each unmanned aerial vehicle 200. The output unit 416 outputs the control signal generated by the control unit 414 to the communication unit 418. The communication unit 418 is connected to the base station device 500 via a communication cable or wireless line, and transmits the control signal to the base station device 500. The base station device 500 has a wireless communication function that is compatible with the same wireless communication system as the wireless communication system of the unmanned aerial vehicle 200, and is able to communicate with the unmanned aerial vehicle 200 via wireless communication. The base station device 500 transmits the control signal to each unmanned aerial vehicle 200.

[0027] Each unmanned aerial vehicle 200 receives a control signal from the base station device 500. The unmanned aerial vehicle 200 extracts location information corresponding to its own identification number 210 from the control signal. The unmanned aerial vehicle 200 has a positioning function such as GNSS (Global Navigation Satellite System) and acquires its current location. The unmanned aerial vehicle 200 moves from its current location toward the extracted location information. In other words, the control unit 414 of the control device 400 moves multiple unmanned aerial vehicles 200 to the derived locations.

[0028] 4(a)-(b) show an overview of the operation of the unmanned aerial vehicle 200. FIG. 4(a) shows a side view of the unmanned aerial vehicle 200. An imaging device 220 is mounted on the bottom of the unmanned aerial vehicle 200. The mounting location of the imaging device 220 is not limited to the bottom of the unmanned aerial vehicle 200. The imaging device 220 captures images of the scenery from the unmanned aerial vehicle 200 and transmits a signal (hereinafter referred to as an "image signal") containing the captured image (hereinafter referred to as a "second-type image") to the control device 400 via the base station device 500. At this time, the image signal also contains the identification number 210 of the unmanned aerial vehicle 200 that is the sender. Furthermore, if the unmanned aerial vehicle 200 is located near a flower 110, the flower 110 will be included in the second-type image. When the infrared imaging device 300 is referred to as a first-type imaging device, the imaging device 220 may be referred to as a second-type imaging device. FIG. 4(b) will be described later, returning to FIG. 2.

[0029] The communication unit 418 of the control device 400 receives image signals from each unmanned aerial vehicle 200 via the base station device 500. The second input unit 420 accepts the second type image and identification number 210 contained in the image signal from the communication unit 418.

[0030] The detection unit 422 detects a pollinable plant 100 based on the second type image received by the second input unit 420. FIGS. 5(a)-(d) show the state of a flower 110 included in the second type image. FIG. 5(a) shows the state of a flower 110 in bud, and over time, the state of the flower 110 changes in the order of FIG. 5(b), FIG. 5(c), and FIG. 5(d). Here, FIGS. 5(a)-(c) correspond to an impossible state, and FIG. 5(d) corresponds to a possible state. In other words, the detection unit 422 is required to detect a plant 100 having a flower 110 in FIG. 5(d) as a pollinable plant 100. The detection unit 422 uses, for example, a convolutional neural network to detect a pollinable plant 100.

[0031] FIG. 6 shows the configuration of the detection unit 422. The detection unit 422 includes a feature extraction unit 430 and a classification unit 432. The feature extraction unit 430 includes a first convolutional layer 440a through an Nth convolutional layer 440n, collectively referred to as a convolutional layer 440, and a first pooling layer 442a through an Nth pooling layer 442n, collectively referred to as a pooling layer 442. The classification unit 432 includes a connection layer 450, a dropout layer 452, and an output layer 454. A second-type image is input to the detection unit 422. The convolutional layer 440 performs convolution of the features, and the pooling layer 442 reduces the pixel values. A convolutional neural network classifies the image into either an impossible state or a possible state. If the output layer 454 detects a plant 100 in a possible state, it outputs the identification number 210 of the unmanned aerial vehicle 200 that captured the second-type image to the control unit 414. Return to Figure 2.

[0032] When the control unit 414 receives an identification number 210 from the detection unit 422, that is, when the detection unit 422 detects a plant 100 that can be pollinated, the control unit 414 identifies the position that has already been instructed as a search operation for the unmanned aerial vehicle 200 (hereinafter referred to as the "discovering unmanned aerial vehicle 200") with the received identification number 210. The identified position corresponds to the position of the plant 100 detected by the detection unit 422 (hereinafter referred to as the "pollination position").

[0033] The control unit 414 selects other unmanned aerial vehicles 200 based on the pollination-possible location. For example, the control unit 414 selects other unmanned aerial vehicles 200 that are located within a certain range from the pollination-possible location. The control unit 414 may also select a certain number of other unmanned aerial vehicles 200 that are closest to the pollination-possible location. Alternatively, the control unit 414 may select all other unmanned aerial vehicles 200.

[0034] The control unit 414 stops the use of the swarm intelligence algorithm for the discovered unmanned aerial vehicle 200 and the other selected unmanned aerial vehicles 200, and generates a signal (hereinafter also referred to as a "control signal") that includes a possible pollination location. The control signal also includes the identification number 210 for each of the discovered unmanned aerial vehicle 200 and the other selected unmanned aerial vehicles 200. The control signal is transmitted to the discovered unmanned aerial vehicle 200 and the other selected unmanned aerial vehicles 200 via the output unit 416, the communication unit 418, and the base station device 500.

[0035] Upon receiving the control signal, the discovering unmanned aerial vehicle 200 and the selected other unmanned aerial vehicle 200 move to the pollination-enabled position included in the control signal. FIG. 4(b) is a side view showing the unmanned aerial vehicle 200 approaching the flower 110. A rod-shaped pollination wand 222 extending laterally is attached to the bottom of the unmanned aerial vehicle 200. The unmanned aerial vehicle 200 performs self-pollination by bringing the tip of the pollination wand 222 into contact with the flower 110. The unmanned aerial vehicle 200 may be equipped with a vibrator (not shown), which may vibrate the pollination wand 222. In other words, the control unit 414 causes the discovering unmanned aerial vehicle 200 and the selected other unmanned aerial vehicle 200 to perform the pollination operation.

[0036] Meanwhile, the control unit 414 derives the positions to which each of the other unmanned aerial vehicles 200 that were not selected should move by using the swarm intelligence algorithm in the same manner as before. Subsequent processing is performed in the same manner as before. In other words, the control unit 414 causes the other unmanned aerial vehicles 200 that were not selected to continue their search operations.

[0037] This configuration can be realized in hardware terms by the CPU, memory, and other LSIs of any computer, and in software terms by programs loaded into memory, but here we depict functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone or a combination of hardware and software.

[0038] The operation of the pollination system 1000 configured as described above will now be described. Figure 7 is a flowchart showing the processing procedure performed by the control device 400. The first input unit 410 accepts a first type of image (S10), and the acquisition unit 412 acquires the position of the unmanned aerial vehicle 200 (S12). If the detection unit 422 does not detect a receivable plant 100 based on the second type of image accepted by the second input unit 420 (N in S14), the control unit 414 derives the position using a swarm intelligence algorithm (S16). The output unit 416 instructs movement to the derived position (S18).

[0039] If the detection unit 422 detects a receivable plant 100 based on the second type image received by the second input unit 420 (Y of S14), the control unit 414 moves some of the unmanned aerial vehicles 200 toward the positions of the plants 100 (S20). The control unit 414 derives the positions of the remaining unmanned aerial vehicles 200 using a swarm intelligence algorithm (S22), and the output unit 416 instructs them to move to the derived positions (S24).

[0040] According to this embodiment, the destination position of each of the multiple unmanned aerial vehicles is derived by using a swarm intelligence algorithm, allowing for efficient search for pollinable plants. Furthermore, when a pollinable plant is detected, the use of the swarm intelligence algorithm is stopped for at least one unmanned aerial vehicle, and the unmanned aerial vehicle is moved to the location of the pollinable plant, allowing for efficient pollination. Furthermore, when a pollinable plant is detected, the destination position is derived by using a swarm intelligence algorithm for the remaining unmanned aerial vehicles, allowing for continued search for pollinable plants.

[0041] The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components, and that such modifications are also within the scope of the present invention.

[0042] In this embodiment, tomatoes are grown as the plants 100. However, the present invention is not limited to this, and plants other than tomatoes may be grown. According to this modification, the range of application of this embodiment can be expanded. [Explanation of symbols]

[0043] 100 plant, 110 flower, 150 cultivation area, 200 unmanned aerial vehicle, 210 identification number, 220 imaging device, 222 pollination rod, 300 infrared imaging device, 400 control device, 410 first input unit, 412 acquisition unit, 414 control unit, 416 output unit, 418 communication unit, 420 second input unit, 422 detection unit, 430 feature extraction unit, 432 identification unit, 440 convolution layer, 442 pooling layer, 450 combination layer, 452 dropout layer, 454 output layer, 500 base station device, 1000 pollination system.

Claims

[Claim 1] A control device for controlling a plurality of unmanned aerial vehicles capable of flying within a cultivation area where plants are cultivated, a first input unit that receives a first type image captured by a first type imaging device that captures an image of the cultivation area; an acquisition unit that acquires the positions of each of the plurality of unmanned aerial vehicles based on the first type image received by the first input unit; A control unit that derives positions to which each of the plurality of unmanned aerial vehicles should move by using a swarm intelligence algorithm on the positions acquired by the acquisition unit, and moves the plurality of unmanned aerial vehicles to the derived positions; a second input unit that receives second-type images captured by second-type imaging devices mounted on each of the plurality of unmanned aerial vehicles; a detection unit that detects a plant capable of pollination based on the second type image received by the second input unit, The control unit is a control device that, when the detection unit detects a pollinable plant, causes a first unmanned aerial vehicle among the plurality of unmanned aerial vehicles that captured the second type of image including the pollinable plant to remain at the position where the second type of image was captured, and stops using the swarm intelligence algorithm for a second unmanned aerial vehicle among the plurality of unmanned aerial vehicles other than the first unmanned aerial vehicle and moves it to the position of the plant detected by the detection unit.

Images