Agricultural work support device and agricultural work support program

Abstract

To provide an agricultural work support device that enables efficient farm work. [Solution] The agricultural work support device comprises a robot, a control device, and a terminal device. The robot is equipped with a camera and a movement device. The camera captures images and transmits them to a control device. The robot moves within the field and performs agricultural tasks in response to control information from the control device. The control device receives images from the robot, transmits the images to a terminal device, receives instruction information from the terminal device to instruct the robot's movements, and transmits control information to the robot based on the instruction information. The terminal device is equipped with a display device and an input device. It receives images from the control device and displays them on the display device, and transmits instruction information to the control device based on input information entered into the input device. The input device is equipped with a first input unit and a second input unit, and the first and second input units are associated with either a single input or the robot's movement and a single action input.

Description

Technical Field

[0001] The technology disclosed in this specification relates to an agricultural work support device and an agricultural work support program that utilize a robot moving in a field.

Background Art

[0002] Patent Document 1 discloses a harvesting robot system that has an identifier subjected to learning processing using teacher data associating an image of an object with the maturity of the object, and uses the identifier to estimate the amount of the object that can be harvested per unit harvesting area and determine a harvesting target area.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above technology, the harvesting robot automatically harvests the harvesting target objects in the area determined by the system or the area determined by the operator. When automatically harvesting the objects in the designated area, there is a risk of harvesting objects that are not suitable for harvesting. For example, immature objects may be harvested, resulting in food loss, or the efficiency may deteriorate by repeatedly trying to harvest objects that are difficult to harvest with the harvesting robot. As an example of agricultural work, it is preferable to individually determine whether to harvest each object, as it is less likely to cause waste. However, it is extremely difficult to leave this only to the judgment of the system.

[0005] This specification provides an agricultural work support technology that can perform agricultural work efficiently.

Means for Solving the Problems

[0006] A first aspect of the technology disclosed herein relates to an agricultural work support device. The agricultural support device comprises a robot, a control device capable of communicating with the robot, and a terminal device capable of communicating with the control device. The robot is equipped with a camera capable of taking pictures within the field and a mobile device. It transmits images taken by the camera to the control device, receives control information from the control device to control the robot's movements, and, in response to the control information, can move within the field using the mobile device and perform actions related to agricultural work. The control device receives images from the robot, transmits the images to the terminal device, receives instruction information from the terminal device to instruct the robot's movements, and transmits control information to the robot based on the instruction information. The terminal device comprises a display device and an input device. It receives images from the control device and displays them on the display device, and transmits instruction information to the control device based on input information input to the input device. The input device comprises a first input unit and a second input unit, and the first and second input units are associated with either a selective input or the input of robot movement and a single action.

[0007] In the above configuration, the operator operating the terminal device can decide whether or not to perform agricultural work for each object, enabling efficient agricultural work. The display specifications for the first and second input units may be determined by a program installed on the terminal device, or the control device may create input device display information and transmit it to the terminal device.

[0008] In the second embodiment, the alternative input in the first embodiment may be an input indicating whether or not to perform a predetermined agricultural task. With this configuration, for example, the first input unit could be an input unit for indicating whether to harvest crops, and the second input unit could be an input unit for indicating whether to not harvest crops. The reverse is also possible. Farming can be performed with simple, one-choice operations.

[0009] In a third embodiment, in the first embodiment described above, the input for a single action may be the selection of performing a predetermined agricultural task. With this configuration, users of the terminal device can perform agricultural work with simple operations.

[0010] In the fourth embodiment, the movement of the robot in the first embodiment may be a movement that changes the shooting range of the robot's shooting device. This configuration allows users of the terminal device to easily access the target of agricultural work. Changing the shooting range includes, for example, moving the shooting device mounted on the robot's arm in a predetermined direction, or moving the robot itself in a predetermined direction.

[0011] In a fifth aspect, in the first aspect, the robot further transmits a captured image indicating that it has performed a predetermined agricultural task as a work image to the control device, the control device acquires work result information which is the result of the robot performing the predetermined agricultural task, and receives the work image from the robot. The work images, work result information, and input information may be stored in a storage device. Information regarding agricultural work that was not performed (i.e., the user of the terminal device did not select "execute") may also be stored as work result information. The storage device may be integrated with the control device, installed in the field, or connected to the control device via the internet or the like for communication. With this configuration, it is also possible to use the work result information stored in the memory device to determine the screen to be displayed on the terminal device, the target of the agricultural work, the type of agricultural work, etc.

[0012] In the sixth embodiment, in the first embodiment described above, environmental sensors may be installed in the field, and the control device may receive output information from the environmental sensors. Furthermore, a prediction unit may be provided, which generates prediction information related to agricultural work based on the output information from the environmental sensors, and the terminal device may acquire the prediction information and display it on a display device. Environmental sensors may, for example, detect temperature, humidity, window open / closed status, and illuminance for each area. Predictive information related to agricultural work includes crop harvesting times, irrigation timing, appropriate temperature and humidity control, fertilizer application timing, pest and disease outbreaks, and pesticide application timing. Furthermore, information stored in a storage device, as described in the fifth embodiment above, may be used for prediction. In addition to being displayed on a display device, this information may also be reflected in control information. This configuration allows users to access predictive information, making it easier to plan and improving convenience.

[0013] In the seventh embodiment, the terminal device may be located at a location far from the field, as in the first embodiment described above. With this configuration, even from a remote location, the operator of the terminal device can remotely control the robot and perform agricultural tasks.

[0014] In the eighth embodiment, in the first embodiment described above, the agricultural work may be any of the following: harvesting crops, irrigation, temperature control, humidity control, fertilizer application, pesticide spraying, leaf removal, and bud removal. The system may be capable of performing multiple agricultural tasks, or the equipment may be selected based on predictions.

[0015] A ninth aspect of the technology disclosed herein relates to an agricultural work support program. The agricultural work support program is a program for a control device that receives images taken by the robot as it moves around the field, transmits the received images to a terminal device, receives instruction information from the terminal device based on input information selectively entered into the first and second input units of the terminal device based on the captured images, and transmits control information determined based on the received instruction information to the robot. "Based on the captured image" means, for example, that the captured image is displayed on a display device, and the user looks at the displayed image and makes inputs to each input unit. In the program described above, for example, a user operating a terminal device can easily decide whether or not to perform agricultural work for each object, enabling them to perform agricultural work efficiently.

[0016] The tenth aspect of the technology disclosed in this specification relates to an agricultural work support program. The agricultural work support program is a program for a control device that receives, from a robot, a captured image obtained by the robot moving and capturing images within a field, transmits the received captured image to a terminal device, and receives, from the terminal device, instruction information based on movement information of the robot associated with a first input unit of the terminal device and input information of a single operation of the robot associated with a second input unit, which are input based on the captured image, and transmits control information determined based on the received instruction information to the robot. In the above program, for example, a user operating the terminal device can easily determine whether to perform agricultural work for each object, and can perform agricultural work efficiently.

Brief Description of the Drawings

[0017] [Figure 1] Shows the configuration of the agricultural work support device of the first embodiment. [Figure 2] Shows a perspective view of the robot 10 of the first embodiment. [Figure 3] Shows a flowchart showing the operation of the control device 20 according to the first embodiment. [Figure 4] Shows a display example of the terminal device 30 according to the first embodiment. [Figure 5] Shows the configuration of the agricultural work support device of the second embodiment. [Figure 6] Shows a flowchart showing the operation of the control device 20a according to the second embodiment. [Figure 7] Shows a display example of the terminal device 30a according to the second embodiment.

Modes for Carrying Out the Invention

[0018] (First Embodiment) (Overall Configuration of Agricultural Work Support Device: Figure 1) Although various types of agricultural work that the robot 10 performs can be considered, in this embodiment, the case of harvesting cucumbers C in the field will be described. The agricultural work support device 1 shown in Figure 1 comprises a robot 10, a control device 20, and a terminal device 30. The robot 10 is equipped with a mobile device 120 (see Figure 2) for moving around in a field where crops such as cucumbers, eggplants, tomatoes, zucchini, bell peppers, and winter melons are cultivated, and a camera device 70. The robot 10 can transmit images of the crops captured by the camera device 70 to the control device 20. Furthermore, the robot 10 can receive control information from the control device 20 to control its movements. The robot 10 is configured to move around the field using the mobile device 120 in response to the control information and to harvest crops in the field.

[0019] The control device 20 is located on the main body 110 of the robot 10 (see Figure 2). The control device 20 receives images of crops in the field from the robot 10 and transmits them to a terminal device 30 located remotely. The display device 32 (see Figure 4) of the terminal device 30 displays the received captured image along with the first input key ("Take" key 311) and the second input key ("Don't Take" key 312). In this case, the control device 20 may be configured to transmit input device display information for display on the display device. The control device 20 receives instruction information from the terminal device 30 that instructs the robot 10 to move, and transmits control information to the robot 10 based on the instruction information and the captured image sent to the terminal device 30.

[0020] The terminal device 30 is, for example, a tablet terminal owned by the user and is equipped with a display device 32 (see Figure 4). The terminal device 30 receives images of crops that are ready for harvest from the control device 20 and displays them on the display device 32. Furthermore, the display device 32 displays a first input key ("take" key 311) and a second input key ("don't take" key 312) to constitute the input device 31 (see Figure 4). The display device 32 is a touch panel and transmits instruction information to the control device 20 based on the input information of the key touched by the operator, i.e., the alternative information of whether the "take" key 311 is tapped to harvest or the "don't take" key 312 is tapped to not harvest.

[0021] (Perspective view of robot 10: Figure 2) As shown in Figure 2, robot 10 is a device for harvesting fruits of fruit and vegetable plants. Robot 10 is suitable for harvesting fruit and vegetable plants where the fruit stalk is relatively short and the position of the fruit is relatively close to the main stem. The robot 10 comprises a main body 110, a mobile device 120, a container 130, a manipulator 140, and an end effector 150. The main body 110 has a control device 20 (see Figure 1) and controls the overall operation of the robot 10, including the operation of the mobile device 120, manipulator 140, and end effector 150. The mobile device 120 is located at the bottom of the main body 110. The mobile device 120 consists of multiple wheels 122 and a motor (not shown), and moves the robot 10. The mobile device 120 moves the harvesting robot 10 in the left-right direction on rails laid in the field. The container 130 is located below the end effector 150. The container 130 has a roughly rectangular parallelepiped shape with an open top and stores the crops harvested by the end effector 150. The manipulator 140 is attached to the main body 110. The manipulator 140 is configured to be able to move up and down vertically, and to swing left and right horizontally. The end effector 150 has a cutting section (not shown) and an imaging device 70 (see Figure 1). The end effector 150 is attached to the manipulator 140 via a drive motor and rotates around the drive motor as the axis of rotation. The cutting section has a pair of blades that grip and cut the fruit stalk attached to the crop. The control device 20 detects the position and size of the crop from the image captured by the imaging device 70. This determines the crop to be harvested. By equipping the robot 10 with an imaging device 70, the crop to be harvested can be easily recognized.

[0022] (Flowchart showing the operation of the control device 20: Figure 3) As shown in Figure 3, when the agricultural work support device 1 starts operating, the control device 20 determines in step S10 whether it has received an image of a cucumber C that is a candidate for harvest from the robot 10. If no image has been received (NO in step S10), it waits. If an image has been received (YES in step S10), it proceeds to step S11.

[0023] In step S11, the control device 20 transmits information about the captured image received from the robot 10 to the terminal device 30.

[0024] As a result, the terminal device 30 displays the images of cucumbers that are candidates for harvest, received from the control device 20, on the display device 32. The display device 32 also displays the "take" key 311 and the "don't take" key 312 as input devices 31. The display device 32 consists of a touch panel. When the operator touches the harvest key 311 or the do not harvest key 312, the terminal device 30 transmits instruction information (either a "take" instruction or a "do not take" instruction) based on the input information corresponding to the touched key to the control device 20.

[0025] In step S12, the control device 20 determines whether or not it has received instruction information from the terminal device 30. If it has not received instruction information (NO in step S12), it waits. If it has received instruction information (YES in step S12), it proceeds to step S13.

[0026] In step S13, the control device 20 determines whether the instruction is to "take" or "not take". If it determines that the instruction is not to take (NO in step S13), it proceeds to step S17. If it determines that the instruction is to take (YES in step S13), it proceeds to step S14.

[0027] If the process proceeds to step S17, the captured image, instruction information ("Do not take"), and a series of pieces of information including the date, time, location, and environment are stored in the database, and the process returns to step S10.

[0028] If the process proceeds to step S14, the control device 20 sends control information to the robot 10 instructing it to "take," and the process proceeds to step S15. This causes the robot 10 to drive the end effector 150 to cut the fruit stalk of the cucumber C that it has taken and to place it in the container 130. Once the harvesting is complete, the robot 10 sends work result information to the control device 20.

[0029] In step S15, the control device 20 receives work result information from the robot 10 and proceeds to step S16. The work result information includes at least whether the harvesting operation was successful or unsuccessful.

[0030] In step S16, the control device 20 stores a series of harvesting information, including captured images and work result information received from the robot 10, instruction information received from the terminal device 30, date and time, location, environment, etc., into a database, and then terminates.

[0031] With the agricultural work support device 1 of this embodiment, the operator can individually decide and instruct whether or not to harvest each individual cucumber, which is the target object, as an example of agricultural work. In reality, it is difficult for a device including the robot 10 to make autonomous decisions in terms of accuracy and speed. On the other hand, it is not practical for an operator who is remotely viewing images of the field taken by the robot 10 to give overly detailed instructions to the robot, as this would increase the difficulty of the work. With the agricultural work support device 1 of this embodiment, the operator only needs to look at the image of the cucumber C displayed on the terminal device 30 in their hand and make a simple and reliable decision of whether or not to harvest it. Even elderly people and others who are not comfortable operating machinery can easily harvest remotely.

[0032] (Second example) (Overall configuration of agricultural work support device 1a: Figure 5) In the first embodiment, the case in which the terminal device 30 displays an alternative key ("take" key 311 and "don't take" key 312) as an input device 31 on the screen of the terminal device 30 to indicate whether or not to harvest the cucumber C was described. As shown in Figure 5, the configuration of the agricultural work support device 1a according to the second embodiment is the same as that of the agricultural work support device 1 in the first embodiment. In this embodiment, the information displayed on the input device 31a is information for displaying a cross key for moving the shooting range of the imaging device 70a and a harvest key. In this embodiment, we will describe the case in which the screen of the terminal device 30a displays a cross key 311a (see Figure 7) for moving the shooting range of the imaging device 70a and a harvest key 312a as input device 31a.

[0033] The directional pad 311a is a key that moves the shooting range visible to the operator on the display device 32a. By tapping the top, bottom, left, or right parts of the directional pad 311a, the shooting range moves in the up, down, left, or right direction, respectively. Specifically, when the operator taps the top or bottom of the directional pad 311a, instruction information for moving the robot 10a's manipulator 140 in the up and down direction is sent from the terminal device 30a to the control device 20a. When the operator taps the left or right part of the directional pad 311a, instruction information for moving the manipulator 140 in the left and right direction, and for moving the robot 10a along the rail in the left and right direction, is sent from the terminal device 30a to the control device 20a. Furthermore, the operator can zoom in and out of the captured image displayed on the display device 32a by pinching (see Figure 7(B)). This allows the operator to arbitrarily change the shooting range of the imaging device 70a.

[0034] (Flowchart showing the operation of the control device 20a: Figures 6 and 7) As shown in Figure 6, when the agricultural work support device 1a starts operating, the control device 20a receives the captured image from the robot 10a in step S21. The received captured image is transmitted to the terminal device 30a in step S22.

[0035] The terminal device 30a displays the captured image received from the control device 20a on the display device 32a. The display device 32a also displays the directional keys 311a and the harvest keys 312a as input devices 31a. The display device 32a is composed of a touch panel. The terminal device 30a transmits instruction information corresponding to the directional keys 311a tapped by the operator to the control device 20a. Furthermore, when the harvest key 312a is tapped, instruction information to initiate harvesting is sent from the terminal device 30a to the control device 20a.

[0036] In step S22a, the control device 20a determines whether or not it has received instruction information from the terminal device 30a. If it has not received instruction information (NO in step S22a), it waits. If it has received instruction information (YES in step S22a), it proceeds to step S23.

[0037] In step S23, the control device 20a receives instruction information from the terminal device 30a and determines its content. If the terminal device 30a provides the instruction to "move", the process proceeds to step S27. In step S27, the captured image, instruction information regarding movement, and a series of pieces of information including date, time, location, and environment can be stored in the database. By recording the robot's actions before harvesting, it becomes possible to link work result information, such as the success or failure of the harvesting operation, with the robot's actions and approach methods leading up to the harvesting operation, and organize this information. Next, the process proceeds to step S20, where the shooting range is moved according to the content of the movement instruction.

[0038] When the operator views the captured image and finds a seedling bearing fruit, they can zoom in and enlarge the image to check the condition of cucumber C, as shown in Figure 7(B). When the operator taps the harvest key 312a, the terminal device 30a sends a "harvest" instruction to the control device 20a. If the terminal device 30a has a "harvest" instruction, the process proceeds to step S24.

[0039] In step S24, harvest control information is sent to the robot 10a, and the process proceeds to step S25. In step S25, the robot 10a provides work result information, and the process proceeds to step S26. In step S26, the captured images, instruction information, and work result information, along with supplementary information such as the date, are stored in the database, and the process ends. The work result information includes at least whether the harvesting operation was successful or unsuccessful.

[0040] According to the agricultural work support device of this embodiment, the robot 10a can be operated in a game-like manner using the directional pad 311a, allowing for efficient harvesting and making the harvesting work enjoyable.

[0041] (Correspondence) The "take" key 311 and "don't take" key 312 in Figure 4 are examples of either the "first input section" or the "second input section". The directional keys 311a and harvest key 312a in Figure 7 are examples of either the "first input section" or the "second input section". The harvesting of cucumber C in the first and second embodiments is an example of "farming work". A database is an example of a storage device.

[0042] Although embodiments of the present invention have been described in detail above, these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples illustrated above.

[0043] (1) The agricultural work support device 1 may be configured to communicate with a learning means using artificial intelligence (AI). The learning means can improve the robot's functions so that it can appropriately execute the operator's instructions by learning captured images, instruction information, work result information, and other information stored in a database in a timely manner.

[0044] (2) Various environmental sensors may be installed in the field. The control device 20 may receive output information from the environmental sensors. Furthermore, the agricultural work support device 1 may include a prediction unit. The prediction unit generates prediction information related to agricultural work based on output information from environmental sensors, etc. The terminal device 30 may acquire the prediction information and display it on the display device 32. Examples of predictive information include optimal harvest times, predictions of pest and disease outbreaks, and optimal timing and methods for pest and disease prevention measures. To make predictions, information stored in memory may be used. Additionally, weather information and market information (such as trends in market prices of crops to be harvested) may also be used.

[0045] (3) Using data from a database containing work result information, predictive information such as harvest time, prediction of pest and disease occurrence, and optimal timing and methods for pest and disease prevention measures may be created. Based on this predictive information, the terminal device may acquire the predictive information and display it on a display device. This allows users to see the captured images sent from the robot while also knowing the predictive information, which can be used to decide whether or not to harvest, and to plan agricultural work, thus improving convenience.

[0046] (4) Agricultural work includes harvesting crops, irrigation, temperature control, humidity control, fertilization, and pesticide application.

[0047] (5) The input device may be configured to allow the user to select between the first input unit and the second input unit as either an alternative input as shown in the first embodiment, or as an input of movement of the shooting range and a single action as shown in the second embodiment. The display device may display a means for selecting which of the two is to be selected, allowing the user to make an arbitrary selection. [Explanation of Symbols]

[0048] 1 Farming support equipment 10, 10a Robot 110 Main body 120 Mobile device 122 Wheels 130 containers 140 Manipulators 150 End Effectors 20, 20a Control device 30, 30a Terminal device 31, 31a Input device 311, 311a First input section 312, 312a Second input section 32, 32a Display device 70, 70a Imaging device

Claims

1. An agricultural work support device comprising a robot, a control device capable of communicating with the robot, and a terminal device capable of communicating with the control device, The aforementioned robot, Equipped with a camera capable of filming within the field and a mobile device, The image captured by the aforementioned imaging device is transmitted to the control device. The control device receives control information to control the operation of the robot, In response to the aforementioned control information, the mobile device can move within the field and perform actions related to agricultural work. The control device is The robot receives the captured image, The captured image is transmitted to the terminal device. The terminal device receives instruction information that instructs the robot to move, Based on the instruction information, the control information is transmitted to the robot. The aforementioned terminal device is Equipped with a display device and an input device, The captured image is received from the control device and displayed on the display device. Based on the input information input to the input device, the instruction information is transmitted to the control device. The aforementioned input device is It comprises a first input section and a second input section, The first input unit and the second input unit are, A choice input, or one that corresponds to the robot's movement and a single action input. Agricultural work support equipment.

2. The aforementioned alternative input is, The agricultural work support device according to claim 1, which is an input for whether or not to perform a predetermined agricultural work.

3. The input for the single operation is, The agricultural work support device according to claim 1, which is an option to perform a predetermined agricultural work.

4. The agricultural work support device according to claim 1, wherein the movement of the robot is a movement that changes the shooting range of the shooting device.

5. The robot further transmits the captured image indicating that it has performed a predetermined agricultural task to the control device as a work image. The control device acquires work result information which is the result of the robot performing a predetermined agricultural task, and receives the work image from the robot. The aforementioned work image, the aforementioned work result information, and the aforementioned input information are stored in a storage device. The agricultural work support device according to claim 1.

6. Environmental sensors are installed in the aforementioned field. The control device receives the output information from the environmental sensor, Furthermore, it is equipped with a prediction unit, The prediction unit generates prediction information related to agricultural work based on the output information of the environmental sensor. The agricultural work support device according to claim 1, wherein the terminal device acquires the prediction information and displays it on the display device.

7. The aforementioned terminal device is located at a location remote from the field, as described in claim 1.

8. The agricultural work support device according to claim 1, characterized in that the aforementioned agricultural work is any of the following: harvesting crops, irrigation, temperature control, humidity control, fertilizer application, pesticide spraying, leaf removal, and bud removal.

9. A program for a control device, The system receives images taken by the robot as it moves around the field. The received captured image is transmitted to the terminal device. The terminal device receives instruction information based on input information selectively input to the first input unit and the second input unit of the terminal device based on the captured image. A farming support program that transmits control information determined based on the received instruction information to the robot.

10. A program for a control device, The system receives images taken by the robot as it moves around the field. The received captured image is transmitted to the terminal device. The terminal device receives instruction information based on the robot's movement information associated with the first input unit of the terminal device and the single motion input information of the robot associated with the second input unit, which are input based on the captured image. A farming support program that transmits control information determined based on the received instruction information to a robot.

Images