Agricultural assistance system, agricultural assistance device, agricultural assistance method, and agricultural assistance program

Abstract

A system includes a control unit and an operation mechanism that performs a prescribed operation in contact with a plant, wherein the control unit includes a model generation unit that generates a three-dimensional model of the plant according to a combination of prescribed generation conditions, a learning unit that learns a first learned model that is generated by learning an image of the plant and a prescribed parameter in the three-dimensional model as learning data, an image acquisition unit that acquires an image of a plant that is a determination target, a model acquisition unit that acquires the first learned model, a parameter estimation unit that estimates the prescribed parameter of a prescribed part of the plant that is the determination target based on the first learned model and the image of the plant that is the determination target, and an operation instruction unit that causes the operation mechanism to perform the prescribed operation based on the estimated prescribed parameter.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an agricultural assistance system, an agricultural assistance device, an agricultural assistance method, and an agricultural assistance program. BACKGROUND

[0002] In agriculture, a technique of automatically performing pollination of fruits and the like is known.

[0003] In Patent Literature 1, a technique related to a drone control device for controlling a drone equipped with an attachment accessory that attaches pollen or a fruiting agent to a pistil is described.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent Application Publication No. 2021-45055 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] In the technique described in Patent Literature 1, a technique of determining a flowering state based on image data of a flower acquired from a drone and attaching pollen or a fruiting agent is described.

[0009] However, in the technique of Patent Literature 1, there is a concern that pollen or a fruiting agent cannot be properly attached in a case where a flower is oriented in a rotation direction.

[0010] Therefore, a technique capable of more appropriately performing various operations related to agriculture such as pollination is needed.

[0011] SOLUTION TO PROBLEM

[0012] According to one embodiment, an agricultural assistance system is provided, including a control unit and an operation mechanism that performs a prescribed operation in contact with a plant, wherein the control unit includes: a model generation unit that generates a three-dimensional model of the plant according to a combination of prescribed generation conditions; a learning unit that learns a first learned model generated by learning an image of the plant and a prescribed parameter in the three-dimensional model as learning data; an image acquisition unit that acquires an image of a plant that is a determination target; a model acquisition unit that acquires the first learned model; a parameter estimation unit that estimates a prescribed parameter of a prescribed part of the plant that is the determination target based on the first learned model and the image of the plant that is the determination target; and an operation instruction unit that causes the operation mechanism to perform the prescribed operation based on the estimated prescribed parameter.

[0013] EFFECT OF THE INVENTION

[0014] According to the present disclosure, various operations related to agriculture such as fertilization can be more appropriately performed. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a block diagram showing the structure of the system 1 as a whole.

[0016] Figure 2 is a diagram showing the functional structure of the terminal device 10.

[0017] Figure 3 is a diagram showing the functional structure of the server 20.

[0018] Figure 4 is a diagram showing the functional structure of the agricultural assistance device 30.

[0019] Figure 5 is a diagram showing an example of the appearance (perspective view) of the agricultural assistance device 30 according to the first embodiment.

[0020] Figure 6 is a diagram showing an example of the appearance (side view) of the agricultural assistance device 30 according to the first embodiment.

[0021] Figure 7 is a diagram showing the detailed structure of the main body portion 502 of the agricultural assistance device 30.

[0022] Figure 8 is a flowchart showing a series of processes performed by the agricultural assistance device 30 based on the acquired image of the plant and the learned model to perform a prescribed operation.

[0023] Figure 9 is a diagram showing the operation of the operation mechanism 302 of the agricultural assistance device 30 performing a prescribed operation based on a prescribed parameter with respect to a prescribed portion of a plant.

[0024] Figure 10 is a diagram showing an example of a screen displayed by the operation mechanism 302 of the agricultural assistance device 30 when the operation mechanism 302 detects a tendency of a disease of a plant and displays a notification to a user who manages a farm. DETAILED DESCRIPTION

[0025] Hereinafter, an embodiment of the present disclosure will be described with reference to the accompanying drawings. In the following description, the same reference numerals are assigned to the same components. Their names and functions are also the same. Thus, detailed description thereof will not be repeated.

[0026] <First Embodiment>

[0027] <Summary>

[0028] In the following embodiment, a technique of acquiring an image of a plant, estimating a prescribed parameter of the plant in three-dimensional space based on a learned model, and operating an operation mechanism provided at a front end of an arm of an agricultural assistance device based on the parameter is described.

[0029] As a comparative example of the embodiment, a structure described below is described. As a comparative example, a technique of acquiring an image of a plant and determining a position of a pistil of a flower, and performing automatic pollination by a device is assumed. For example, a learned model obtained by associating a learning image of a flower with a flowering state is used to determine a flowering state in an image of a flower taken, and a fitting is driven according to an orientation of the flower determined to be in a flowering state to pollinate the flower. However, in the comparative example, the orientation of the flower is only an orientation in a plane (two-dimensional) based on the taken image, and in a case where the flower is oriented to the inside, etc., the flowering determination and the pollination cannot be properly performed, and there is a concern that the pollination work is omitted and the harvest amount is damaged.

[0030] Therefore, in the system 1 described in the present embodiment, an agricultural assistance system is provided, which includes a control unit and an operation mechanism that performs a prescribed operation in contact with a plant, wherein the control unit includes: a model generation unit that generates a three-dimensional model of a plant according to a combination of prescribed generation conditions; a learning unit that learns a first learned model generated by learning an image of a plant and a prescribed parameter in a three-dimensional model as learning data; an image acquisition unit that acquires an image of a plant that is a determination target; a model acquisition unit that acquires the first learned model; a parameter estimation unit that estimates a prescribed parameter of a prescribed part of the plant that is the determination target based on the first learned model and the image of the plant that is the determination target; and an operation instruction unit that causes the operation mechanism to perform a prescribed operation based on the estimated prescribed parameter.

[0031] As described above, the system 1 provides a technique of more appropriately performing various operations related to agriculture such as pollination.

[0032] The system 1 can be used, for example, in a scene where pollination, harvesting, leaf picking, etc. are automatically performed in a plant factory, a greenhouse, or a plastic house. Thereby, various operations related to agriculture such as pollination can be more appropriately performed.

[0033] Hereinafter, the structure of the system 1 related to the first embodiment of the present disclosure will be described with reference to Figures 1-4 The system 1 is mainly a system that plans to produce plants such as a plant factory, a greenhouse, or a plastic house, and includes a device described below.

[0034] • A device that automatically performs pollination processing on a flower of a plant

[0035] • Device for automatically performing harvesting process on fruits of plants

[0036] • Device for automatically performing leaf picking, pruning, and weeding on leaves and stems of plants

[0037] • Device for automatically scattering fertilizer on plants

[0038] The above device is specifically a device for automatically performing pollination process on self-pollinated flowers whose pollen is attached to the stigma of the same flower, and a device for automatically performing pollination process on cross-pollinated flowers whose pollen is attached to the stigma of other flowers.

[0039] <1> Overall configuration of the system

[0040] Figure 1 The overall configuration of the system 1 in the first embodiment is shown.

[0041] As shown in Figure 1 , the system 1 includes a terminal device 10 (only the terminal device 10 is illustrated in Figure 1 , but can also be constituted by a plurality of terminal devices (10A, 10B, 10C, etc.), a server 20, and an agricultural assistance device 30. The terminal device 10, the server 20, and the agricultural assistance device 30 are communicatively connected via a network 80. The network 80 is constituted by a wired or wireless network.

[0042] The terminal device 10 is a device for each user to operate. The terminal device 10 is realized by a stationary PC (Personal Computer), a laptop PC, or the like. In addition to this, the terminal device 10 can also be, for example, a tablet, a smartphone, or the like portable terminal that supports a mobile body communication system. As shown in Figure 1 , the terminal device 10 is provided with a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The server 20 is provided with a communication IF 22, an input / output IF 23, a memory 25, a storage unit 26, and a processor 29. The agricultural assistance device 30 is provided with a communication IF 32, an input / output IF 33, a memory 35, a storage 36, and a processor 39.

[0043] The terminal device 10 is communicably connected with the server 20 and the agricultural assistance device 30 via the network 80. The terminal device 10 is connected with the network 80 by communicating with a communication device such as a wireless base station 81 supporting various communication standards (5G, LTE (Long Term Evolution), etc.), a wireless LAN router 82 supporting an IEEE (Institute of Electrical and Electronics Engineers) 802.11 or the like wireless LAN standard, and the like.

[0044] The communication IF 12 is an interface for inputting and outputting signals for communication between the terminal device 10 and an external device. The input device 13 is an input device (for example, a touch panel, a touch pad, a mouse or the like pointing device, a keyboard, and the like) for accepting an input operation from a user. The output device 14 is an output device (a display, a speaker, and the like) for presenting information to a user. The memory 15 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or the like for temporarily storing a program and data processed by the program, and the like. The storage section 16 is a storage device for storing data, for example, a flash memory, an HDD (Hard Disc Drive). The processor 19 is hardware for executing a command group described in a program, and is constituted by an arithmetic device, a register, a peripheral circuit, and the like.

[0045] The server 20 manages information and the like related to a learned model acquired by the agricultural assistance device. Details of the learned model will be described later.

[0046] In some cases, the server 20 can also manage various information of a user or the like who manages a farm. Specifically, for example, the server 20 can also manage the following information as information related to a user who manages a farm.

[0047] • a kind of vegetables and fruits cultivated by the user in the farm

[0048] • an area of the farm managed by the user

[0049] • a kind of tools and machines held by the user

[0050] • a tilt angle of the farm managed by the user

[0051] • a property of soil of the farm managed by the user

[0052] • a kind of fertilizers used by the user in the farm

[0053] • the region where the user-managed farm is located

[0054] • information on the climate (average temperature, rainfall, amount of sunlight, etc.) in the region where the user-managed farm is located

[0055] The communication IF 22 is an interface for inputting and outputting signals for communication between the server 20 and an external device. The input / output IF 23 functions as an interface between an input device for receiving an input operation from a user and an output device for presenting information to the user. The memory 25 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or the like, and is used to temporarily store programs and data processed by the programs and the like. The storage unit 26 is a storage device such as a flash memory or an HDD (Hard Disc Drive), and is used to store data. The processor 29 is hardware for executing a command group described in a program, and is constituted by an arithmetic device, a register, a peripheral circuit, and the like.

[0056] In the present embodiment, each device (terminal device, server, and the like) can also be understood as an information processing device. That is, a collection of each device can be understood as one "information processing device", and the system 1 can also be formed as a collection of a plurality of devices. As to a method of allocating a plurality of functions required for the system 1 related to the present embodiment to one or a plurality of hardware, the processing capacity of each hardware and / or the specifications required for the system 1, and the like can be appropriately decided.

[0057] The agricultural assistance device 30 is a device that performs a prescribed work in a farm based on an instruction from a user or a condition set in advance. The prescribed work specifically includes, for example, the following works in a user-managed farm performed by the agricultural assistance device 30.

[0058] • Pollination work (work of attaching pollen to a pistil)

[0059] • Harvest work

[0060] • Leaf picking work (work of removing overlapping leaves, unnecessary leaves)

[0061] • Fertilizer spreading work

[0062] • Work of removing diseased fruits, leaves, and the like

[0063] The communication IF 32 is an interface for inputting and outputting signals for communication between the agricultural assistance device 30 and an external device. The input / output IF 33 functions as an interface between an input device for receiving an input operation from a user and an output device for presenting information to the user. The memory 35 is a volatile memory such as a DRAM (Dynamic Random Access Memory) for temporarily storing programs and data processed by the programs and the like. The storage section 36 is a storage device such as a flash memory, an HDD (Hard Disc Drive) for storing data. The processor 39 is hardware for executing a command group described in a program, and is constituted by an arithmetic device, a register, a peripheral circuit, and the like.

[0064] <1.1 Configuration of terminal device 10>

[0065] Figure 2 is a block diagram showing a functional configuration of the terminal device 10 that constitutes the system 1 of Embodiment 1. As shown in Figure 2 , the terminal device 10 includes a plurality of antennas (antenna 111, antenna 112), a wireless communication section corresponding to each antenna (first wireless communication section 121, second wireless communication section 122), an operation receiving section 130 (including a keyboard 1301 and a mouse 1302), a sound processing section 140, a microphone 141, a speaker 142, a display 150, a position information sensor 160, a storage section 170, and a control section 180. The terminal device 10 also has functions and structures not particularly shown in Figure 2 , for example, a battery for holding electric power, a power supply circuit that controls supply of electric power from the battery to each circuit, and the like. As shown in Figure 2 , each block included in the terminal device 10 is electrically connected through a bus or the like.

[0066] The antenna 111 emits a signal transmitted from the terminal device 10 as an electric wave. In addition, the antenna 111 receives an electric wave from space and imparts a received signal to the first wireless communication section 121.

[0067] The antenna 112 emits a signal transmitted from the terminal device 10 as an electric wave. In addition, the antenna 112 receives an electric wave from space and imparts a received signal to the second wireless communication section 122.

[0068] The first wireless communication section 121 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 111 in order for the terminal device 10 to communicate with other wireless devices. The second wireless communication section 122 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 112 in order for the terminal device 10 to communicate with other wireless devices. The first wireless communication section 121 and the second wireless communication section 122 are communication modules including a tuner, an RSSI (Received Signal Strength Indicator) calculation circuit, a CRC (Cyclic Redundancy Check) calculation circuit, a high-frequency circuit, and the like. The first wireless communication section 121 and the second wireless communication section 122 perform modulation / demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10, and impart the received signals to the control section 180.

[0069] The operation receiving section 130 has a mechanism for receiving an input operation of a user. Specifically, the operation receiving section 130 includes a keyboard 1301 and a mouse 1302. In addition, the operation receiving section 130 can also be configured as a touch screen that detects a touch position of a user's contact with a touch panel by using an electrostatic capacitance method, for example.

[0070] The keyboard 1301 receives an input operation of a user of the terminal device 10. The keyboard 1301 is a device that performs character input, and outputs input character information as an input signal to the control section 180.

[0071] The mouse 1302 receives an input operation of a user of the terminal device 10. The mouse 1302 is a pointing device for selecting and the like of a display object displayed on the display 150, and outputs position information of a selection on a screen and information indicating that a button is pressed as an input signal to the control section 180.

[0072] The sound processing section 140 performs modulation / demodulation of a sound signal. The sound processing section 140 modulates a signal imparted from the microphone 141, and imparts the modulated signal to the control section 180. In addition, the sound processing section 140 imparts a sound signal to the speaker 142. The sound processing section 140 is realized by a processor for sound processing, for example. The microphone 141 receives a sound input, and imparts a sound signal corresponding to the sound input to the sound processing section 140. The speaker 142 converts a sound signal imparted from the sound processing section 140 to a sound, and outputs the sound to the outside of the terminal device 10.

[0073] The display 150 displays data such as images, videos, texts, and the like in accordance with the control of the control section 180. The display 150 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0074] The position information sensor 160 is a sensor for detecting the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals from at least three or four satellites are received, and the current position of the terminal device 10 on which the GPS module is mounted is detected on the basis of the received signals. For example, in the system 1, in a case where the position information of the user can be referred to, information relating to the position of the farm managed by the user and the like can be determined on the basis of the position information of the user, so that the climate conditions and the like in the farm can be determined. Further, in the system 1, in a case where the terminal device 10 and the like hold information relating to the user who manages the farm, the information of the user who manages the farm in the region can be determined on the basis of the position information. In addition, the position information sensor 160 can also be a transmitting and receiving device based on a communication standard used in a short distance communication system between information devices. Specifically, the position information sensor 160 is a Bluetooth (registered trademark) module or the like, uses a 2.4 GHz band, and receives a beacon signal from another information device on which a Bluetooth (registered trademark) module is mounted.

[0075] The storage section 170 is constituted by, for example, a flash memory or the like, and stores data and programs used by the terminal device 10. In some cases, the storage section 170 can also store, for example, the following information as information relating to the user who manages the farm.

[0076] • the kind of vegetables and fruits cultivated by the user in the farm

[0077] • the area of the farm managed by the user

[0078] • the kind of tools and machines held by the user

[0079] • the inclination angle of the farm managed by the user

[0080] • the property of the soil of the farm managed by the user

[0081] • the kind of fertilizers used by the user in the farm

[0082] • the region in which the farm managed by the user is located

[0083] • information on the climate in the region where the farm managed by the user is located (average temperature, rainfall, amount of sunlight, etc.)

[0084] The control section 180 controls the operation of the terminal device 10 by reading the program stored in the storage section 170 and executing the commands contained in the program. The control section 180 is, for example, an application processor. The control section 180 functions as an input operation accepting section 1801, a transmission and reception section 1802, a data processing section 1803, and a presentation control section 1804 by operating in accordance with the program.

[0085] The input operation accepting section 1801 performs processing of accepting the input operation of the user to the input device such as the keyboard 1301. Further, in a case where the input operation from the user is accepted through an input device such as a touch sensitive device (not shown) and the like, the input operation accepting section 1801 determines the type of operation included in the following based on information on the coordinates at which the user touches the touch sensitive device with a finger or the like.

[0086] • whether the operation of the user is a flick operation

[0087] • whether the operation of the user is a tap operation

[0088] • whether the operation of the user is a drag (swipe) operation

[0089] The transmission and reception section 1802 performs processing for transmitting and receiving data between the terminal device 10 and an external device such as the server 20 in accordance with a communication protocol.

[0090] The data processing section 1803 performs processing of operating on the data for which the input has been accepted by the terminal device 10 in accordance with the program and outputting the operation result to a storage or the like.

[0091] The presentation control section 1804 performs processing of presenting information to the user. The presentation control section 1804 performs processing of causing the display 150 to display a display image, processing of causing the speaker 142 to output sound, and the like.

[0092] <1.2 Functional structure of the server 20>

[0093] Figure 3 is a diagram showing the functional structure of the server 20. As shown in Figure 3 , the server 20 functions as a communication section 201, a storage section 202, and a control section 203.

[0094] The communication section 201 performs processing for communication between the server 20 and an external device.

[0095] The storage section 202 stores data and programs used by the server 20. The storage section 202 stores a learned model database 2021 and the like. In some cases, the storage section 202 can also store various information related to the user who manages the farm.

[0096] The learned model database 2021 is a database for managing information of a learned model obtained by associating a prescribed parameter of a prescribed part in a three-dimensional model of a plant with an image of the plant. Here, the association is not limited to an image of a plant. An association with a video of a plant or the like can also be made.

[0097] The reception control section 2031 controls processing in which the server 20 receives a signal from an external device in accordance with a communication protocol.

[0098] The transmission control section 2032 controls processing in which the server 20 transmits a signal to an external device in accordance with a communication protocol.

[0099] The model generation section 2033 controls processing in which a three-dimensional model of a plant is generated based on prescribed generation conditions. Specifically, for example, the model generation section 2033 generates a three-dimensional model of a plant with a combination of the elements included in the following as generation conditions for the three-dimensional model.

[0100] • Sizes of the leaves, stems, flowers (including pistils, stamens, and the like), and fruits of the plant

[0101] • Sizes of the leaves, stems, flowers (including pistils, stamens, and the like), and fruits of the plant

[0102] • Numbers of the leaves, stems, flowers (including pistils, stamens, and the like), and fruits of the plant

[0103] • Colors of the leaves, stems, flowers (including pistils, stamens, and the like), and fruits of the plant

[0104] • Information of the background

[0105] • Information of the lighting

[0106] • Orientations of the leaves, stems, flowers (including pistils, stamens, and the like), and fruits of the plant

[0107] • Positions of the leaves, stems, flowers (including pistils, stamens, and the like), and fruits of the plant

[0108] Here, the method of creating a three-dimensional model can use any existing three-dimensional drawing software.

[0109] The server 20 can also accumulate information of a three-dimensional model generated in accordance with a combination of the above-described elements in the storage section 202.

[0110] The learning unit 2034 controls the process of learning the learned model obtained by associating the generated three-dimensional model with the prescribed parameters of the plant and saving the learned model in the learned model database 2021 of the storage unit 202 of the server 20.

[0111] Here, the creation method of the learned model in the present disclosure will be exemplified.

[0112] For example, the learning unit 2034 acquires information of the three-dimensional model generated by the model generation unit 2033.

[0113] Next, the learning unit 2034 receives an image (or a video) of a plant associated with the three-dimensional model from the user who manages the farm, and associates various parameters of each part of the plant (position, orientation, positional relationship of pistil) with various parameters of each part of the three-dimensional model. For example, the learning unit 2034 receives an image of a plant in which the position of the pistil is being oriented toward a prescribed orientation. Next, the learning unit 2034 receives a model in which the position of the pistil in the three-dimensional model is being oriented toward the same orientation as the position of the pistil in the received image of the plant from the user who manages the farm, and associates the image of the plant with various parameters in the model.

[0114] When the server 20 receives an image of a plant as input, it determines the parameters (for example, orientation, positional relationship, etc.) of the prescribed parts (for example, flower, pistil, leaf, etc.) of the plant based on the learned model, and outputs the parameters as parameters for operating the agricultural assistance device. That is, the pistil is being oriented toward a prescribed direction (for example, a direction diagonally backward from the inside), so information for pollinating the pistil from the front direction is output. At this time, the output information can be a normal vector, and the agricultural assistance device can also operate the operation mechanism described later based on the output information of the normal vector.

[0115] Thus, the user who manages the farm can accurately and delicately cause the work in the farm to be performed automatically under various conditions.

[0116] In some cases, the server 20 can store information of a plurality of learned models in the storage unit in addition to the above-exemplified learned model (first learned model). Specifically, for example, the server 20 can store the following learned models.

[0117] • a learned model (second learned model) obtained by learning at least either of ventilation or sunlight in the farm and the amount of harvest of fruits at the harvest time

[0118] • a learned model (third learned model) obtained by associating an image of a fruit, a weight of the fruit, a sugar content of the fruit, and a quality of the fruit

[0119] • a learned model obtained by associating an image of a plant with a disease image of the plant (fourth learned model)

[0120] • a learned model obtained by associating an image of a plant with a nutrition condition of the plant (fifth learned model)

[0121] Here, a method of generating each learned model will be exemplified.

[0122] In some aspect, the server 20 can also store each learned model per plant. For example, the above learned models can be stored for each of plants cultivated in the farm, such as strawberries, tomatoes, melons, cucumbers, and the like. At this time, the server 20 can also determine the learned model to be acquired on the basis of determining which plant the acquired image is of.

[0123] Thus, even if the user managing the farm has cultivated a plurality of kinds of plants, the user can promptly change the learned model to be acquired, and thus can appropriately perform various operations.

[0124] A method of generating the second learned model will be exemplified. The server 20 receives information of ventilation in the farm (which can be approximated to the density of leaves of the plant or the like) or sunlight in the farm (the degree of sunlight at each place of the plant based on the density of leaves of the plant or the like) from the user managing the farm. Then, the server 20 receives information of the harvest amount of fruits of the plant at the harvest time from the user managing the farm, and associates the information with the information of the ventilation and the sunlight. The server 20 outputs the harvest amount prediction based on the second learned model when receiving an image of the plant as input.

[0125] In some aspect, the agricultural assistance device can estimate the positional relationship of leaves or the like that become the optimal harvest amount based on the output harvest amount prediction, and perform leaf picking or the like based on the result obtained by the estimation.

[0126] Thus, the user managing the farm can automatically adjust in a manner that the optimal harvest amount is obtained based on the information of the ventilation of the leaves or the like.

[0127] A method of generating the third learned model will be exemplified. The server 20 can also receive an image of a fruit, a weight of the fruit, a sugar degree of the fruit, and a quality of the fruit from the user managing the farm, and associate them. The server 20 outputs the quality of the fruit when receiving the image of the fruit, the weight, and the sugar degree as input.

[0128] Thus, the user managing the farm can quantitatively evaluate the quality of the fruit.

[0129] The generation method of the fourth learned model will be described. The server 20 receives information of a three-dimensional model of a plant created based on an image in which the plant is suffering from a disease from a user who manages a farm. At this time, the user who manages the farm can also create a disease model of the plant obtained by randomly setting the conditions exemplified below.

[0130] • the occurrence position of the disease

[0131] • the occurrence amount of the disease (area on leaves, flowers, etc.)

[0132] • the occurrence position of the disease in one plant

[0133] Next, the server 20 receives images in which the tendency of the occurrence of the disease in the image of the plant appears, and associates them with their three-dimensional models. That is, in a case where a prescribed disease (anthracnose, aphid, powdery mildew, etc., not limited to) appears on a part of a leaf or the like, information of the site in which the tendency of the occurrence of the disease appears is output.

[0134] In some cases, the agricultural assistance device can perform the removal, rejection, or the like of the site based on the output information of the site in which the tendency of the occurrence of the disease appears.

[0135] Thus, even in a case where the area of the farm is wide, the user who manages the farm can quickly detect the tendency of the disease, and can prevent damage caused by the disease at an early stage.

[0136] The generation method of the fifth learned model will be described. The server 20 receives information of a three-dimensional model of a plant created based on an image of the plant under various nutritional conditions (deficiency in nutrients, excess in nutrients, etc.) from a user who manages a farm. Next, the server 20 receives images in which the tendency of the nutritional condition exemplified above appears in the image of the plant, and associates them with their three-dimensional models. That is, in a case where a part of the plants in the farm appears to be deficient in nutrients, information of the site in which the tendency appears is output.

[0137] In some cases, the agricultural assistance device can adjust the amount, proportion, or the like of the fertilizer to be scattered based on the output information of the site in which the tendency of the deficiency in nutrients appears.

[0138] Thus, the user who manages the farm can grasp the imbalance in the scattering condition of the fertilizer, and can harvest fruits of uniform quality or the like.

[0139] The learning unit 2034 controls the process of saving the generated learned model in the learned model database 2021 of the storage unit 202 of the server 20. Specifically, for example, the learning unit 2034 can transmit the learned model to the agricultural assistance device 30 in response to an operation instruction received from the user who manages the farm, or can transmit the learned model corresponding to the plant in response to the reception of the image of the plant from the agricultural assistance device 30.

[0140] <1.3 Functional Configuration of Agricultural Assistance Device 30>

[0141] Figure 4 Fig. 1 is a diagram showing the functional configuration of the agricultural assistance device 30. As shown in Fig. 1, the agricultural assistance device 30 has a communication unit 301, an operation mechanism 302, a storage unit 303, a control unit 304, and a photographing mechanism 350. Figure 4

[0142] The communication unit 301 is used for the agricultural assistance device 30 to communicate with devices outside.

[0143] The operation mechanism 302 is used for the agricultural assistance device 30 to perform a prescribed operation on a plant. The prescribed operation includes, for example, the following work.

[0144] • Pollination (attaching pollen to a pistil) work

[0145] • Harvest work

[0146] • Leaf picking (removing overlapping leaves, unnecessary leaves) work

[0147] • Fertilizer scattering work

[0148] • Work of removing diseased fruits, leaves, and the like

[0149] The storage unit 303 stores data and programs used by the agricultural assistance device 30. In some cases, the storage unit 303 can also store various learned models and various information related to the user who manages the farm, and the like.

[0150] The control unit 304 controls the operation of the agricultural assistance device 30 by reading the program stored in the storage unit 303 and executing the commands included in the program. The control unit 304 is, for example, an application processor. The control unit 304 functions as an image acquisition unit 3041, a model acquisition unit 3042, a parameter estimation unit 3043, and an operation instruction unit 3044 by operating in accordance with the program.

[0151] ​The image acquisition section 3041 controls the operation of acquiring an image or a video obtained by the photographing by the photographing mechanism 350 described later. At this time, the format of the image or the video acquired by the image acquisition section 3041 is not limited. For example, the image acquisition section 3041 acquires an image or a video in a format including the following.

[0152] • JPEG (Joint Photographic Experts Group) format

[0153] • PNG (Portable Network Graphics) format

[0154] • GIF (Graphics Interchange Format) format

[0155] • TIFF (Tagged Image File Format) format

[0156] • Bitmap format

[0157] • MP4 (Moving Picture Experts Group) format

[0158] • MOV (QuickTime file format) format

[0159] • AVI (Audio Video still Images) format

[0160] • VOB (Video Object file) format

[0161] The model acquisition section 3042 controls the process of acquiring various learned models from the server 20. Specifically, for example, the model acquisition section 3042 acquires various learned models in response to the image acquisition section 3041 acquiring an image of a plant. At this time, the model acquisition section 3042 can also keep the acquired learned models in the storage section 303, thereby omitting the acquisition process of the learned models at the time of various operations on the same plant. In addition, the model acquisition section 3042 can temporarily keep the acquired learned models in a nonvolatile memory and acquire the corresponding learned models at each operation.

[0162] Thus, the user can perform various operations based on learned models even in a situation where the agricultural assistance device cannot ensure sufficient capacity of the storage section.

[0163] The parameter estimation section 3043 controls processing of estimating a prescribed parameter of a prescribed part of a plant based on an image of the plant acquired by the image acquisition section 3041 and the learned model acquired by the model acquisition section. Specifically, the prescribed part includes, for example, the following parts.

[0164] • a flower of the plant

[0165] • a pistil of the plant

[0166] • a stamen of the plant

[0167] • a leaf of the plant

[0168] • a fruit of the plant

[0169] • a stem of the plant

[0170] In addition, the prescribed parameter includes, for example, the following parameters.

[0171] • an orientation of the prescribed part of the plant described above

[0172] • a positional relationship of the prescribed part of the plant described above

[0173] At this time, the parameter estimation section 3043 can not calculate the parameters of upward, downward, and the like as the parameters of the orientation, but can calculate the parameters of the orientation as normal vectors in a three-dimensional space. The normal vectors in the three-dimensional space can be, for example, obtained by displaying various orientations in a 360° celestial sphere model as vectors.

[0174] Thus, regardless of the orientation of the flower, leaf, or the like of the plant, the user who manages the farm can accurately estimate the direction thereof and appropriately perform pollination, harvesting, leaf picking, or the like.

[0175] The operation instruction section 3044 controls processing of causing the operation mechanism 302 of the agricultural assistance device 30 to perform a prescribed operation. The prescribed operation includes, for example, the following work.

[0176] • pollination (attaching pollen to a pistil) work

[0177] • harvesting work

[0178] • leaf picking (removing overlapping leaves, unnecessary leaves) work

[0179] • work of scattering fertilizer

[0180] • work of removing diseased fruits, leaves, or the like

[0181] The imaging mechanism 350 acquires an image of a plant in a farm. At this time, the imaging mechanism 350 can also acquire a video of the plant.

[0182] The photographing mechanism 350 is not limited and can be any camera available. The photographing mechanism 350 can be either a camera using a silver halide film or a digital camera.

[0183] In some cases, the agricultural assisting device 30 can have a function other than the above depending on the use. For example, the agricultural assisting device can have a function including the following (none of which is illustrated).

[0184] • a water dispersing section for dispersing water

[0185] • a fertilizer dispersing section for dispersing fertilizer

[0186] • a chemical dispersing section for dispersing chemicals

[0187] As other functions, there are no limitations to the above functions and can be any function required for performing agriculture.

[0188] <1.3 Structure of the Agricultural Assisting Device 30>

[0189] Figure 5 and Figure 6 is a drawing showing an example of the appearance of the agricultural assisting device 30 according to the first embodiment. Figure 5 is a perspective view of the agricultural assisting device 30, Figure 6 is a side view of the agricultural assisting device 30.

[0190] As shown in Figure 5 and Figure 6 , the agricultural assisting device 30 has a tray setting support 501, a harvesting tray 5011, a main body 502, an information processing section 503, and a traveling section 504. At this time, the main body 502 can be provided on a guide rail that can move up and down.

[0191] The tray setting support 501 and the harvesting tray 5011 are mechanisms for storing fruits and the like harvested by the agricultural assisting device 30. Specifically, for example, the agricultural assisting device 30 sets the harvesting tray 5011 for storing the harvested fruits on the tray setting support 501. The agricultural assisting device 30 stores the harvested fruits (for example, strawberries) in the harvesting tray 5011. The tray setting support 501 is configured in a multi-layered manner, and in the case where the storage capacity of the harvesting tray 5011 reaches a predetermined value, the main body 502 is moved up and down so that the fruits are stored in the harvesting tray 5011 having a spare capacity.

[0192] Thus, the user managing the farm can efficiently perform the harvesting of fruits.

[0193] In some cases, the operation of setting the harvesting tray 5011 to the tray setting support 501 can be performed by the user who manages the farm or automatically. Specifically, the agricultural assistance device 30 is provided with a tray setting arm (not shown) that can move forward, backward, leftward, and rightward in the main body 502, and the harvesting tray 5011 is set to the prescribed position of the tray setting support 501 by the tray setting arm grasping and moving the harvesting tray 5011.

[0194] Thus, even if the number of layers of the tray setting support of the agricultural assistance device 30 increases, the labor of the user who manages the farm to manually set the support can be reduced.

[0195] In addition, in some cases, the tray setting support 501 and the harvesting tray 5011 can not be for harvesting fruits, but for housing seedlings of plants. In this case, the main body 502 can also perform the operation of sequentially taking out the seedlings from the harvesting tray 5011 and planting them in the farm.

[0196] The main body 502 is a mechanism for performing various operations (pollination, harvesting, leaf picking, and the like) in the farm. Details are described later.

[0197] The information processing section 503 is a mechanism for the agricultural assistance device 30 to communicate with a terminal outside, or a mechanism for operating and instructing the main body 502 and the like based on a program previously held in the information processing section 503. The information processing section 503 can be a computer provided with a processor, or a microcomputer (microprocessor).

[0198] The traveling section 504 is a mechanism for moving the agricultural assistance device 30 within the farm. Specifically, for example, the traveling section 504 is provided to the lower surface of the agricultural assistance device 30, and can be a wheel for traveling on the ground, or a wheel that can move on a guide rail. In addition, the traveling section 504 can also move the agricultural assistance device 30 by the mechanisms described below.

[0199] • a mechanism provided with a magnetic sensor and traveling the agricultural assistance device 30 by detecting a magnet laid on the road surface of the farm

[0200] • a mechanism provided with a line sensor and traveling the agricultural assistance device 30 by detecting a line drawn in the farm

[0201] In addition to this, the traveling section 504 can be provided with any mechanism used in automatic driving of a car or the like.

[0202] Figure 7 is a diagram showing the detailed structure of the main body 502 of the agricultural assistance device 30.

[0203] As Figure 7As shown, the agricultural auxiliary device 30 has a main body 502, an arm 701 disposed on the main body 502, an operating mechanism 302 disposed on the front end of the arm, a drive mechanism 751, and accessories 752.

[0204] The operating mechanism 302 can be mounted on the front end of the arm 701. The shooting mechanism 350 (not shown) can be mounted on other positions of the arm 701, in addition to the operating mechanism 302 (i.e., the front end of the arm 701).

[0205] In one scenario, arm 701 may also include a pollination imaging unit (not shown) for capturing the moment of pollination, and a harvest prediction unit (not shown) for predicting the harvest yield based on the image of the moment of pollination obtained from the imaging. Furthermore, the agricultural assistance device 30 can also present the predicted harvest yield to the user. Thus, the user managing the farm can appropriately determine whether pollination has been completed based on the information from the moment of pollination and the predicted harvest yield.

[0206] In addition, in a certain situation, the agricultural auxiliary device 30 can also perform pollination again if the harvest forecast result is less than the previous harvest forecast.

[0207] As a result, farm owners are able to harvest fruit consistently without experiencing insufficient pollination.

[0208] Operating mechanism 302 Figure 7 As shown, the operating mechanism 302, located at the front end of the movable arm 701, contacts the plant's flower, fruit, leaves, etc., to perform a prescribed operation. The operating mechanism 302 may also have different mechanisms at its front end depending on the type of operation. For example, in the case of pollination, a fibrous component may be provided at the front end for contact with the plant's flower. Furthermore, in the case of fruit harvesting, leaf removal, etc., an arm for pruning and harvesting fruit may be provided at the front end. The operating mechanism 302 performs prescribed operations on the plant through the various mechanisms provided at its front end. The operation during pollination will now be presented.

[0209] For example, the operation mechanism 302 has a driving mechanism 751 and a fitting 752. The operation mechanism 302 causes the driving mechanism 751 to drive the fitting 752 to contact a prescribed part of a plant for which an operation of applying pollen is to be performed (for example, a pistil of a flower) in accordance with the control of the control section 304. The control section 304 outputs the position of the pistil and the normal vector of the pistil in the image on the basis of the learned model and the image acquired from the imaging mechanism 350. The operation mechanism 302 causes the driving mechanism 751 to drive in the three-dimensional direction in accordance with the information of the normal vector output by the control section 304 and controls so that the fitting 752 contacts the pistil perpendicularly to the pistil (that is, from the front of the pistil) and pollen is attached to the pistil uniformly and comprehensively. In addition, it is also possible to cause the operation mechanism 302 to move in the up-and-down direction and the left-and-right direction while contacting the pistil of the flower.

[0210] <2 Action>

[0211] Next, a series of processes in which the agricultural assistance device 30 constituting the system 1 performs a prescribed operation on the basis of an acquired image of a plant and a learned model will be described.

[0212] Figure 8 is a flowchart showing a series of processes in which the agricultural assistance device 30 performs a prescribed operation on the basis of an acquired image of a plant and a learned model.

[0213] In step S811, the control section 180 of the terminal device 10 receives an operation input from a user who manages a farm. Specifically, for example, the control section 180 receives an operation input for a prescribed operation (pollination work, fruit harvesting work, or the like) in a farm from a user who manages the farm. At this time, the control section 180 can transmit the operation content input from the user to the agricultural assistance device 30 or can transmit information of an operation set in advance by the user who manages the farm to the agricultural assistance device 30.

[0214] In step S801, the control section 304 of the agricultural assistance device 30 acquires an image of a plant that is a determination target. Specifically, for example, the agricultural assistance device 30 moves in front of a plant that is a target by a guide rail provided to a lower surface of the device, and an image of the plant is imaged by the imaging mechanism 350. The control section 304 can also transmit the acquired image to the server 20. As for the imaging method, for example, specifically, it can be that the imaging mechanism 350 automatically discriminates a flower or the like to focus on the part and image while detecting a distance and an angle from the imaged flower or the like by a sensor such as an infrared sensor or a depth sensor, and further calculating the distance and the angle from the operation mechanism 302 to the flower or the like.

[0215] In step S851, the control section 203 of the server 20 transmits the first learned model to the agricultural assistance device 30. Specifically, for example, the control section 203 transmits the first learned model obtained by learning by the above-described method to the agricultural assistance device 30 in accordance with an instruction from the agricultural assistance device 30. At this time, the learned model to be transmitted is not limited to the first learned model. The control section 203 can also determine the learned model to be transmitted based on the image of the plant acquired by the agricultural assistance device 30. In addition, information of the learned model to be transmitted can also be accepted from a user who manages the farm.

[0216] In step S802, the control section 304 of the agricultural assistance device 30 acquires the first learned model from the server. Specifically, the model acquisition section 3042 constituting the control section 304 acquires various learned models in response to the image acquisition section 3041 acquiring the image of the plant. At this time, the model acquisition section 3042 can also hold the acquired learned models in the storage section 303, thereby omitting the acquisition process of the learned models at the time of various operations on the same plant. In addition, the model acquisition section 3042 can also temporarily hold the acquired learned models in a nonvolatile memory, and acquire the corresponding learned model at each operation.

[0217] In step S803, the control section 304 of the agricultural assistance device 30 estimates the prescribed parameter based on the first learned model and the image of the plant. Specifically, the parameter estimation section 3043 constituting the control section 304 controls the process of estimating the prescribed parameter of the prescribed part of the plant based on the image of the plant acquired by the image acquisition section 3041 and the learned model acquired by the model acquisition section. Specifically, the prescribed part includes, for example, the following parts.

[0218] • The flower of the plant

[0219] • The pistil of the plant

[0220] • The stamen of the plant

[0221] • The leaf of the plant

[0222] • The fruit of the plant

[0223] • The stem of the plant

[0224] In addition, the prescribed parameter includes, for example, the following parameters.

[0225] • The orientation of the prescribed part of the plant described above

[0226] • The positional relationship of the prescribed part of the plant described above

[0227] At this time, the parameter estimation section 3043 can also not calculate the parameters of upward, downward, and the like as the parameters of the orientation, but calculate the parameters of the orientation with the normal vector in the three-dimensional space. The normal vector in the three-dimensional space can be, for example, obtained by displaying various orientations in the 360° celestial sphere model as vectors.

[0228] In step S804, the control section 304 of the agricultural assistance device 30 causes the operation mechanism to perform a prescribed operation based on the estimated prescribed parameters. Specifically, the operation instruction section 3044 constituting the control section 304 controls processing of causing the operation mechanism 302 of the agricultural assistance device 30 to perform a prescribed operation. The prescribed operation includes, for example, the following work.

[0229] • Pollination (attaching pollen to pistil) work

[0230] • Harvest work

[0231] • Leaf removal (removing overlapping leaves, unnecessary leaves) work

[0232] • Fruit removal work

[0233] • Fertilizer scattering work

[0234] • Removal work of diseased fruits, leaves, and the like

[0235] Through the above processing, the user who manages the farm can appropriately perform various processing (pollination, harvesting, leaf removal, and the like) regardless of the direction and positional relationship of each part of the plant (flowers, stamens, leaves, stems, fruits, and the like). Therefore, it is possible to stably cultivate fruits with higher quality.

[0236] In some cases, the server 20 can also hold a result related to the quality of fruits when a self-pollinated plant is pollinated by cross-pollination. Based on the result, even if the plant is self-pollinated, the server 20 can present the user who manages the farm with the meaning that the quality of the fruits is expected to be improved when cross-pollination is performed.

[0237] Thus, the user who manages the farm can appropriately manage pollination and the like including quality improvement.

[0238] <3 Action Example>

[0239] Figures 9-10 is a diagram showing an action example and the like of a series of processing in which the system 1 disclosed in the present application operates the operation mechanism and the like of the agricultural assistance device based on an input for a prescribed operation in the farm received from the user who manages the farm.

[0240] Figure 9is a diagram showing an action of the operation mechanism 302 possessed by the agricultural assistance device 30 performing a prescribed operation on a prescribed part of a plant based on a prescribed parameter.

[0241] In Figure 9 , the agricultural assistance device 30 drives the arm 701 in accordance with a prescribed parameter of a prescribed part of a plant determined based on an image of the plant and a learned model. In Figure 9 , an example of an action when performing a pollination process on a pistil of a flower is exemplified. As Figure 9 indicated, the agricultural assistance device 30 moves the arm 701 and the driving mechanism 751 of the operation mechanism 302 in a prescribed direction based on the determined parameter, and drives so that the fitting 752 of the operation mechanism 302 provided at the front end of the arm 701 is perpendicular (directly in front) to the pistil of the flower. The prescribed direction is output, for example, by the learned model as a normal vector, and is determined by a combination of vectors indicated below.

[0242] • roll (rotation in the x-axis direction)

[0243] • pitch (rotation in the y-axis direction)

[0244] • yaw (rotation in the z-axis direction)

[0245] Thus, based on the information of the image (two-dimensional) of the plant, it is also possible to output a direction in a three-dimensional space based on the learned model. The agricultural assistance device 30 can make the fitting 752 face an arbitrary direction and come into contact with the plant by combining the movement of the arm 701 and the movement of the driving mechanism 751. Therefore, even if the pistil of the flower or the like faces the inside direction when viewed from the direction of the agricultural assistance device 30, for example, the user managing the farm can appropriately perform the pollination or the like.

[0246] The operation described in the above is not limited to pollination. Even for operations (leaf picking, pruning, fruit removal, and the like) that occur in a farm, the same mechanism can be used.

[0247] Figure 10 A screen example when the agricultural assistance device 30 possessed by the operation mechanism 302 displays a notification to the user managing the farm when a tendency of a disease of a plant is detected is shown.

[0248] In Figure 10 , the photographed image 1001, the disease warning 1002, and the poor growth warning 1003 are displayed in the display 150 of the terminal device 10 possessed by the user managing the farm. The photographed image 1001 is an image of a plant obtained by photographing by the photographing mechanism 350 possessed by the agricultural assistance device 30.

[0249] The disease warning 1002 indicates a warning displayed for a plant in which a tendency of a disease has occurred in the captured image of the plant. Specifically, the control section 304 of the agricultural assistance device 30 estimates whether or not the tendency that has occurred in the plant is a tendency of a disease on the basis of the acquired image of the plant and the fourth learned model described above. In a case where the result of the estimation is that the plant is infected with a disease, the disease warning 1002 is presented to the user who manages the farm. In some cases, the agricultural assistance device 30 can also present the disease warning 1002 to the user who manages the farm in a case where the plant is infected with a disease, and cause the operation mechanism 302 to perform an operation of removing the site infected with the disease.

[0250] Thus, the user who manages the farm can early discover a disease, and can prevent a loss caused by the disease.

[0251] The poor growth warning 1003 indicates a warning displayed for a plant in which a tendency of poor growth has occurred in the captured image of the plant. Specifically, the control section 304 of the agricultural assistance device 30 estimates whether or not the tendency that has occurred in the plant is a tendency based on poor growth on the basis of the acquired image of the plant and the fifth learned model described above. In a case where the result of the estimation is that the plant is growing poorly, the poor growth warning 1003 is presented to the user who manages the farm. In some cases, the agricultural assistance device 30 can also present the poor growth warning 1003 to the user who manages the farm in a case where the plant is growing poorly, and cause the fertilizer adjustment section to perform an operation of adjusting the fertilizer for the region of the plant in which the farm is cultivated, in which the plant is growing poorly.

[0252] Thus, the user who manages the farm can early discover poor growth that has occurred due to unevenness of fertilizer scattering, and can harvest fruits of which the quality is uniform.

[0253] In some cases, the agricultural assistance device can also make the same notification to the user who manages the farm in the result of the determination of the overlapping condition of the leaves.

[0254] In the embodiment in the present disclosure, as the agricultural assistance system, an example in which the server 20 and the agricultural assistance device 30 communicate information is illustrated, but the embodiment is not limited thereto. For example, a series of processes can be executed without communication with the server 20 on the basis of the learned model held in the storage section of the agricultural assistance device 30. In addition, it can be that the control section of the agricultural assistance device 30 has a function of generating a three-dimensional model and learning a learned model.

[0255] <4 Modification>

[0256] A modification of the present embodiment will be described. That is, the following configuration can also be adopted.

[0257] (1) An information processing apparatus, the program can be installed in advance, can be installed later, can also store such a program in an external non-transitory storage medium, and can also make the program act through cloud computing.

[0258] (2) A method of causing a computer to function as an information processing apparatus in which the program can be installed in advance, can be installed later, can also store such a program in an external non-transitory storage medium, and can also make the program act through cloud computing.

[0259] <Supplementary Note>

[0260] The following supplementary notes describe matters described in each of the above embodiments.

[0261] (Supplementary Note 1)

[0262] A system 1 is an agricultural assistance system 1 provided with a control section 304 and an operation mechanism 302 that performs a prescribed operation in contact with a plant, wherein the control section 304 includes: a model generation section 2033 that generates a three-dimensional model of a plant according to a combination of prescribed generation conditions; a learning section 2034 that learns a first learned model that is generated by learning an image of a plant and a prescribed parameter in a three-dimensional model as learning data; an image acquisition section 3041 that acquires an image of a plant that is a determination target; a model acquisition section 3042 that acquires the first learned model; a parameter estimation section 3043 that estimates a prescribed parameter of a prescribed part of the plant that is the determination target based on the first learned model and the image of the plant that is the determination target; and an operation instruction section 3044 that causes the operation mechanism 302 to perform a prescribed operation based on the estimated prescribed parameter.

[0263] (Supplementary Note 2)

[0264] The agricultural assistance system 1 according to Supplementary Note 1, wherein the prescribed parameter is at least one selected from the group consisting of: an orientation of the prescribed part of the plant in a three-dimensional space and a positional relationship of the prescribed part of the plant.

[0265] (Supplementary Note 3)

[0266] The agricultural assistance system 1 according to Supplementary Note 1 or 2, wherein the prescribed generation condition is at least one selected from the group consisting of: a size of the three-dimensional model, a number of petals in the three-dimensional model, a color of the three-dimensional model, a shape of the three-dimensional model, a background of a space in which the three-dimensional model is generated, a position of illumination in the space in which the three-dimensional model is generated, and an orientation of the three-dimensional model.

[0267] (Supplementary Note 4)

[0268] The agricultural assistance system 1 according to any one of appendices 1 to 3, wherein the prescribed site of the plant that is a determination target is at least one selected from the group consisting of: a position of a pistil of a flower, a position of a stamen of a flower, a position of a fruit, and a position of a leaf.

[0269] (appendix 5)

[0270] The agricultural assistance system 1 according to any one of appendices 1 to 4, wherein the prescribed operation is at least one selected from the group consisting of: a pollination operation of a flower, a harvesting operation of a fruit, a leaf picking operation, and a fruit culling operation.

[0271] (appendix 6)

[0272] The agricultural assistance system 1 according to any one of appendices 1 to 5, further comprising a photographing mechanism 350 that photographs an image of the plant.

[0273] (appendix 7)

[0274] The agricultural assistance system 1 according to any one of appendices 1 to 6, wherein the model acquisition section 3042 acquires a second learned model that is learned with respect to at least either of ventilation or sunlight in the farm, and a harvest amount of a fruit at a harvest time.

[0275] (appendix 8)

[0276] The agricultural assistance system 1 according to appendix 7, wherein the control section 304 further includes a position relationship estimation section that estimates an optimal position relationship of any one of a group including a leaf, a flower, and a fruit, on the basis of a harvest amount prediction output by the second learned model, and the operation instruction section 3044 causes the operation mechanism 302 to perform the prescribed operation on the basis of the optimal position relationship of any one of the group including the leaf, the flower, and the fruit estimated by the position relationship estimation section.

[0277] (appendix 9)

[0278] The agricultural assistance system 1 according to any one of appendices 6 to 8, wherein the control section 304 further includes: a pollination photographing section that causes the photographing mechanism 350 to photograph a moment of pollination; a harvest amount prediction section that performs a harvest amount prediction on the basis of an image of the moment of pollination obtained by the photographing; and an output section that presents the harvest amount prediction obtained by the prediction to a user.

[0279] (appendix 10)

[0280] The agricultural assistance system 1 according to any one of the above 1 to 9, further comprising: a weight sensor that measures a weight of the fruit; and a sugar content sensor that measures a sugar content of the fruit, the model acquisition unit 3042 acquires a third learned model obtained by associating the image of the fruit, the weight of the fruit, the sugar content of the fruit, and the quality of the fruit, and the output unit determines the quality of the fruit based on the image of the fruit, the weight, the sugar content, and the third learned model, and outputs the quality of the fruit.

[0281] (Note 11)

[0282] The agricultural assistance system 1 according to any one of the above 1 to 10, wherein the model acquisition unit 3042 acquires a fourth learned model obtained by associating the image of the plant and a disease image of the plant, the output unit, when the image of the plant is accepted as input, outputs whether the plant is infected with a disease based on the fourth learned model, and in a case where it is output that the plant is infected with a disease, the operation instruction unit 3044 causes the operation mechanism 302 to perform an operation of removing a site infected with the disease.

[0283] (Note 12)

[0284] The agricultural assistance system 1 according to any one of the above 1 to 11, wherein the control unit 304 further includes a fertilizer adjustment unit, the model acquisition unit 3042 acquires a fifth learned model obtained by associating the image of the plant and a nutritional condition of the plant, the output unit, when the image of the plant is accepted as input, determines and outputs excess or deficiency of the nutritional condition of the plant based on the fifth learned model, and the fertilizer adjustment unit adjusts a fertilizer scattered on a farm based on the nutritional condition of the plant.

[0285] (Note 13)

[0286] An agricultural assistance device 30 including a control unit 304 and an operation mechanism 302 that performs a prescribed operation in contact with a plant, wherein the control unit 304 includes: a model generation unit 2033 that generates a three-dimensional model of the plant according to a combination of prescribed generation conditions; a learning unit 2034 that learns a first learned model generated by learning an image of the plant and a prescribed parameter in the three-dimensional model as learning data; an image acquisition unit 3041 that acquires an image of a plant that is a determination target; a model acquisition unit 3042 that acquires the first learned model; a parameter estimation unit 3043 that estimates a prescribed parameter of a prescribed site of the plant that is the determination target based on the first learned model and the image of the plant that is the determination target; and an operation instruction unit 3044 that causes the operation mechanism 302 to perform a prescribed operation based on the estimated prescribed parameter.

[0287] (Note 14)

[0288] An agricultural assistance method using a control section 304 and an operation mechanism 302 that performs a prescribed operation in contact with a plant, wherein the control section 304 executes the following steps: a model generation step of generating a three-dimensional model of a plant according to a combination of prescribed generation conditions; a learning step of learning a first learned model that is generated by learning an image of a plant and a prescribed parameter in a three-dimensional model as learning data; an image acquisition step (S801) of acquiring an image of a plant that is a determination target; a model acquisition step (S802) of acquiring the first learned model; a parameter estimation step (S803) of estimating a prescribed parameter of a prescribed part of the plant that is the determination target based on the first learned model and the image of the plant that is the determination target; and an operation instruction step (S804) of causing the operation mechanism 302 to perform a prescribed operation based on the estimated prescribed parameter.

[0289] (Additional Note 15)

[0290] A program for causing a computer 20 to execute agricultural assistance in which a control section 304 and an operation mechanism 302 that performs a prescribed operation in contact with a plant are used, the program causing the control section 304 to execute the following steps: a model generation step of generating a three-dimensional model of a plant according to a combination of prescribed generation conditions; a learning step of learning a first learned model that is generated by learning an image of a plant and a prescribed parameter in a three-dimensional model as learning data; an image acquisition step (S801) of acquiring an image of a plant that is a determination target; a model acquisition step (S802) of acquiring the first learned model; a parameter estimation step (S803) of estimating a prescribed parameter of a prescribed part of the plant that is the determination target based on the first learned model and the image of the plant that is the determination target; and an operation instruction step (S804) of causing the operation mechanism 302 to perform a prescribed operation based on the estimated prescribed parameter.

[0291] Explanation of Reference Signs

[0292] 1: system; 10: terminal device; 12: communication interface; 13: input device; 14: output device; 15: memory; 16: storage; 19: processor; 20: server; 22: communication interface; 23: input / output interface; 25: memory; 26: storage unit; 29: processor; 30: agricultural auxiliary device; 32: communication interface; 33: input / output interface; 35: memory; 36: storage; 39: processor; 80: network; 170: storage; 180: control unit; 1801: input operation receiving unit; 1802: transmission / reception unit; 1803: data processing unit; 1804: notification control unit; 130: operation receiving unit; 1301: keyboard; 1302: mouse; 140: sound processing unit; 141: microphone; 142: speaker; 150: display; 160: position information sensor; 202: storage; 2021: learned model database; 203: control unit; 2031: reception control unit; 2032: transmission control unit; 2033: model generation unit; 2034: learning unit; 302: operation mechanism; 303: storage; 304: control unit; 3041: image acquisition unit; 3042: model acquisition unit; 3043: parameter estimation unit; 3044: operation instruction unit; 350: photographing mechanism.

Claims

1. An agricultural auxiliary system comprising a control unit and an operating mechanism that comes into contact with plants to perform prescribed operations. in, The control unit includes: The model generation unit uses 3D rendering software to generate multiple 3D plant models with different generation conditions based on a combination of specified generation conditions. The specified generation conditions are selected from a group including the following elements: the size of the 3D model, the number of petals in the 3D model, the color of the 3D model, the shape of the 3D model, the background of the space in which the 3D model is generated, the position of the lighting in the space in which the 3D model is generated, and the orientation of the 3D model. The learning department learns a first learned model, which is generated by learning from images of plants and specified parameters in the three-dimensional model as learning data. The image acquisition unit acquires images of the plant that is the object of judgment. The model acquisition unit acquires the first learned model; The parameter estimation unit estimates the specified parameters of a specified part of the plant as the determination object based on the first learned model and an image of the plant as the determination object; and An operation instruction unit, based on specified parameters obtained from the estimation, causes the operating mechanism to perform specified operations.

2. The agricultural auxiliary system according to claim 1, wherein, The specified parameters are selected from at least one of the following parameters: the orientation of the specified part of the plant in the three-dimensional space used to generate the three-dimensional model and the positional relationship of the specified part of the plant.

3. The agricultural auxiliary system according to claim 1 or 2, wherein, The specified part of the plant to be determined is selected from at least one of the following positions: the position of the pistil of the flower, the position of the stamen of the flower, the position of the fruit, and the position of the leaf.

4. The agricultural auxiliary system according to claim 1 or 2, wherein, The specified operation is selected from at least one of the following operations: pollination of flowers, harvesting of fruits, leaf removal, and fruit removal.

5. The agricultural auxiliary system according to claim 1 or 2, wherein, It also includes a camera mechanism that captures images of the plant.

6. The agricultural auxiliary system according to claim 1 or 2, wherein, The model acquisition unit acquires a second learned model, which is obtained by learning at least one of ventilation and sunlight in the farm, as well as the harvest amount of fruit at the harvest time.

7. The agricultural auxiliary system according to claim 6, wherein, The control unit further includes a positional relationship estimation unit, which estimates the optimal positional relationship of any one of the group including leaves, flowers, and fruits based on the harvest prediction output by the second learned model. The operation instruction unit causes the operating mechanism to perform the prescribed operation based on the optimal positional relationship of any one of the group including leaves, flowers, and fruits estimated by the positional relationship estimation unit.

8. The agricultural auxiliary system according to claim 5, wherein, The control unit also includes: The pollination imaging unit enables the imaging mechanism to capture the moment of pollination; The harvest yield prediction unit predicts the harvest yield based on the image of the moment of pollination obtained during the shooting; and The output unit presents the harvest forecast obtained from the aforementioned forecast to the user.

9. The agricultural auxiliary system according to claim 8, further comprising: A weight sensor that measures the weight of the fruit; as well as Sugar content sensor, which measures the sugar content of fruit. The model acquisition unit acquires a third learned model obtained by associating the fruit image, the fruit weight, the fruit sugar content, and the fruit quality. The output unit determines the quality of the fruit based on the fruit's image, weight, sugar content, and the third learned model, and outputs the quality of the fruit.

10. The agricultural auxiliary system according to claim 8 or 9, wherein, The model acquisition unit acquires a fourth learned model obtained by associating the image of the plant with the image of the plant's disease. When the output unit receives an image of the plant as input, it outputs whether the plant is infected with a disease based on the fourth learned model. When the plant is found to be infected with a disease, the operation instruction unit causes the operating mechanism to remove the infected parts.

11. The agricultural auxiliary system according to claim 8 or 9, wherein, The control unit also includes a fertilizer adjustment unit. The model acquisition unit acquires a fifth learned model obtained by associating the image of the plant with the nutritional status of the plant. When the output unit receives the image of the plant as input, it determines and outputs whether the plant's nutritional status is excessive or insufficient based on the fifth learned model. The fertilizer adjustment unit adjusts the fertilizer distributed on the farm based on the nutritional status of the plants.

12. An agricultural auxiliary device comprising a control unit and an operating mechanism that comes into contact with plants to perform prescribed operations. in, The control unit includes: The model generation unit uses 3D rendering software to generate multiple 3D plant models with different generation conditions based on a combination of specified generation conditions. The specified generation conditions are selected from a group including the following elements: the size of the 3D model, the number of petals in the 3D model, the color of the 3D model, the shape of the 3D model, the background of the space in which the 3D model is generated, the position of the lighting in the space in which the 3D model is generated, and the orientation of the 3D model. The learning department learns a first learned model, which is generated by learning from images of plants and specified parameters in the three-dimensional model as learning data. The image acquisition unit acquires images of the plant that is the object of judgment. The model acquisition unit acquires the first learned model; The parameter estimation unit estimates the specified parameters of a specified part of the plant as the determination object based on the first learned model and an image of the plant as the determination object; and An operation instruction unit, based on specified parameters obtained from the estimation, causes the operating mechanism to perform specified operations.

13. An agricultural assistance method using a control unit and an operating mechanism that comes into contact with plants to perform predetermined operations, wherein the control unit performs the following steps: The model generation process involves using 3D modeling software to generate multiple 3D plant models with different generation conditions based on a combination of specified conditions. The specified generation conditions are selected from the group including the following elements: the size of the 3D model, the number of petals in the 3D model, the color of the 3D model, the shape of the 3D model, the background of the space in which the 3D model is generated, the position of the lighting in the space in which the 3D model is generated, and the orientation of the 3D model; The learning steps involve learning a first learned model, which is generated by learning from images of plants and specified parameters in the 3D model as learning data. The image acquisition step involves acquiring an image of the plant that will be used as the judgment object. The model acquisition steps are as follows: acquire the first learned model; The parameter estimation step estimates the specified parameters of a specified part of the plant as the determination object based on the first learned model and the image of the plant as the determination object. as well as The operation instructions are based on the specified parameters obtained from the estimation, to cause the operating mechanism to perform the specified operation.

14. A computer program product comprising a computer program for causing a computer to perform agricultural assistance, wherein a control unit and an operating mechanism that contacts plants to perform prescribed operations are used in the agricultural assistance, the computer program causing the control unit to perform the following steps: The model generation process involves using 3D modeling software to generate multiple 3D plant models with different generation conditions based on a combination of specified conditions. The specified generation conditions are selected from the group including the following elements: the size of the 3D model, the number of petals in the 3D model, the color of the 3D model, the shape of the 3D model, the background of the space in which the 3D model is generated, the position of the lighting in the space in which the 3D model is generated, and the orientation of the 3D model; The learning steps involve learning a first learned model, which is generated by learning from images of plants and specified parameters in the 3D model as learning data. The image acquisition step involves acquiring an image of the plant that will be used as the judgment object. The model acquisition steps are as follows: acquire the first learned model; The parameter estimation step estimates the specified parameters of a specified part of the plant as the determination object based on the first learned model and the image of the plant as the determination object. as well as The operation instructions are based on the specified parameters obtained from the estimation, to cause the operating mechanism to perform the specified operation.

15. A computer-readable recording medium having a computer program recorded thereon, the computer program being used to cause a computer to perform agricultural assistance, wherein a control unit and an operating mechanism that comes into contact with plants to perform prescribed operations are used in the agricultural assistance, the computer program causing the control unit to perform the following steps: The model generation process involves using 3D modeling software to generate multiple 3D plant models with different generation conditions based on a combination of specified conditions. The specified generation conditions are selected from the group including the following elements: the size of the 3D model, the number of petals in the 3D model, the color of the 3D model, the shape of the 3D model, the background of the space in which the 3D model is generated, the position of the lighting in the space in which the 3D model is generated, and the orientation of the 3D model; The learning steps involve learning a first learned model, which is generated by learning from images of plants and specified parameters in the 3D model as learning data. The image acquisition step involves acquiring an image of the plant that will be used as the judgment object. The model acquisition steps are as follows: acquire the first learned model; The parameter estimation step estimates the specified parameters of a specified part of the plant as the determination object based on the first learned model and the image of the plant as the determination object. as well as The operation instructions are based on the specified parameters obtained from the estimation, to cause the operating mechanism to perform the specified operation.

Images