Fruit tree harvest support device, inference device, machine learning device, fruit tree harvest support method, inference method, and machine learning method

Abstract

To provide a fruit tree harvest support device enabling a producer to perform efficient personnel allocation and business activity.SOLUTION: The fruit tree harvest assist device includes the acquisition unit 600 that acquires the fruit tree state information including the state of the fruit tree at the flowering time of the fruit tree to be assisted, and the generation processing unit 601 that generates the harvest information on the fruit tree by inputting the fruit tree state information on the fruit tree to be assisted acquired by the acquisition unit 600 to the learning models 110 to 112 that have learned the correlation between the fruit tree state information including the state of the fruit tree at the flowering time of the fruit tree to be assisted and the harvest information including the state of the fruit at the time of harvest on the fruit tree.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a fruit tree harvesting support device, an inference device, a machine learning device, a fruit tree harvesting support method, an inference method, and a machine learning method. [Background technology]

[0002] In the past, when cultivating fruit trees in orchards, it was impossible to predict the yield of fruit, which prevented orchard producers from efficiently allocating personnel or conducting sales activities. However, in recent years, systems for predicting fruit yields based on various data have been developed and introduced. For example, Patent Document 1 discloses a harvest prediction system that uses an imaging camera to capture images of fruit grown in a greenhouse where the indoor environment is controlled, and predicts the number of fruits to be harvested and fruit size information from the images captured by the imaging camera over time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-054289 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology described in Patent Document 1 requires the time and effort of periodically taking images of the fruit, making it difficult to use the conventional technology on fruit trees such as mandarin oranges grown on large sloping land outdoors.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a fruit tree harvesting support device, an inference device, a machine learning device, a fruit tree harvesting support method, an inference method, and a machine learning method that make it possible to easily predict information regarding the condition of fruit at the time of harvest. [Means for solving the problem]

[0006] In order to achieve the above object, a fruit tree harvesting support device according to one aspect of the present invention comprises: an acquisition unit that acquires fruit tree condition information including the condition of the fruit tree to be supported at the time of flowering; a generation processing unit that generates the harvest information for the fruit tree by inputting the fruit tree condition information for the support target fruit tree acquired by the acquisition unit into a learning model that has been trained to learn a correlation between fruit tree condition information, which includes the condition of the fruit tree at the flowering time of the fruit tree, and harvest information, which includes the condition of fruit that grows on the fruit tree at the time of harvest; The fruit tree status information includes the following status information: image information of the fruit tree taken during the flowering period; and proportion information indicating the proportion of flowers and buds of the fruit tree during the flowering period; and The information includes at least one of content information indicating the content of endogenous plant hormones in the fruit tree at the flowering time. [Effects of the Invention]

[0007] According to one aspect of the present invention, a fruit tree harvesting support device generates harvest information, including the condition of fruit at harvest time, based on the condition of a target fruit tree at the flowering period. This is based on the new knowledge that the condition of a fruit tree at the flowering period is related to the condition of fruit at harvest time. This new knowledge is then introduced as a learning model, and information regarding the condition of fruit at harvest time can be predicted based on the condition of the fruit tree at the flowering period. This allows producers to efficiently allocate personnel and conduct sales activities based on the predictions.

[0008] Problems, configurations, and effects other than those described above will become apparent from the detailed description of the invention that follows. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an overall configuration showing an example of a fruit tree harvest support system 1 according to an embodiment. [Figure 2] FIG. 1 is a diagram showing the growth process of fruit tree T. [Figure 3] FIG. 9 is a hardware configuration diagram showing an example of a computer 900. [Figure 4] FIG. 2 is a data structure diagram showing an example of a fruit tree database 40. [Figure 5] 1 is an explanatory diagram showing an example of information on a specific fruit tree T based on the fruit tree database 40. FIG. [Figure 6] FIG. 2 is a block diagram showing an example of a machine learning device 5 according to the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a first learning model 110 and first learning data 120 when fruit tree condition information is image information. [Figure 8] FIG. 10 is a diagram showing an example of a second learning model 111 and second learning data 121 when fruit tree condition information is ratio information. [Figure 9] FIG. 10 is a diagram showing an example of a third learning model 112 and third learning data 122 when fruit tree condition information is content information. [Figure 10] 10 is a flowchart showing an example of a machine learning method performed by the machine learning device 5. [Figure 11] 1 is a block diagram showing an example of a fruit tree harvesting support device 6 according to an embodiment. [Figure 12] FIG. 2 is a functional explanatory diagram showing an example of a fruit tree harvesting support device 6 according to an embodiment. [Figure 13] 10 is a flowchart showing an example of a fruit tree harvesting support method by the fruit tree harvesting support device 6. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. The scope necessary for the explanation to achieve the object of the present invention will be schematically shown, and the scope necessary for explaining the relevant part of the present invention will be mainly explained, and the parts that are omitted from the explanation will be based on publicly known techniques.

[0011] (Embodiment) Fig. 1 is an overall configuration diagram showing an example of a fruit tree harvesting support system 1 according to an embodiment. Fig. 2 is a diagram showing the growing process of a fruit tree T. The fruit tree harvesting support system 1 is a system used to support producers of orchards that cultivate a large number of fruit trees T.

[0012] The fruit tree harvest support system 1 comprises, as its main components, a producer terminal device 2, a fruit tree condition acquisition device 3, a database device 4, a machine learning device 5, and a fruit tree harvest support device 6. Each of the devices 2 to 6 is configured, for example, as a general-purpose or dedicated computer (see FIG. 3 described below), and is connected to a wired or wireless network 7 so as to be able to transmit and receive various types of data to and from each other. Note that the number of the devices 2 to 6 and the connection configuration of the network 7 are not limited to the example in FIG. 1 and may be changed as appropriate.

[0013] The fruit tree T is, for example, a fruit tree of the Rutaceae family, Rosaceae family, Ebenaceae family, or the like, and the variety of the fruit tree T is not particularly limited. Note that multiple varieties of fruit trees T may be cultivated in an orchard. In this embodiment, the fruit tree T will be described using as an example a species of the genus Citrus belonging to the Rutaceae family.

[0014] As shown in Figure 2, fruit tree T generally flowers around May each year. Then, from the end of May to early July, it undergoes physiological drop. Physiological drop is a phenomenon in which fruit tree T loses its vigor due to an excess of fruit, causing it to naturally drop some of the fruit. After the period of physiological drop from early July to the end of September, the remaining fruit continues to swell. This period is called fruit swell. The fruit then reaches ripening around October. The ripening period is when the fruit begins to ripen. Once the fruit reaches ripening, producers harvest the fruit between October and December. Note that the timing of the growth process shown in Figure 2 varies depending on the type of fruit tree T.

[0015] In the fruit tree harvesting support system 1, the condition of each fruit tree T is managed, and harvesting information including the condition of the fruit on the fruit tree T at the time of harvesting is determined for each fruit tree T based primarily on fruit tree condition information including the condition of the fruit tree T at the time of flowering.

[0016] The fruit tree condition information includes at least one of image information of the fruit tree at the flowering period, ratio information indicating the ratio of flowers to buds on the fruit tree at the flowering period, and content information indicating the content of endogenous plant hormones in the fruit tree at the flowering period.

[0017] The harvest information includes at least one of the following conditions of the fruit on the fruit tree at the time of harvest: yield, fruit size, fruit color, skin roughness, amount of fruit dropping after flowering, fruit ripening period, fruit sugar content, fruit acidity, set rate (percentage of fruit that bears fruit), and use of the fruit.

[0018] The producer terminal device 2 is a terminal device used by orchard producers (farm workers, orchard managers, etc.). The producer terminal device 2 is configured as a stationary device or a portable device, and when configured as a portable device such as a smartphone, tablet terminal, or smart glasses with AR (or MR) functionality, the producer terminal device 2 is equipped with a camera 20 that can acquire image information as fruit tree condition information.

[0019] The producer terminal device 2 accepts various input operations via a display screen of an application program, a web browser, etc., and displays various information (e.g., fruit tree condition information, harvest information, etc.) via the display screen. For example, the producer terminal device 2 accepts input operations of percentage information as fruit tree condition information, receives and displays information regarding the condition of fruit that has borne on the fruit tree T at the time of harvest from the database device 4, and receives and displays information regarding the condition of fruit that is scheduled to borne on the fruit tree T at the time of harvest from the fruit tree harvest support device 6.

[0020] The fruit tree condition acquisition device 3 acquires fruit tree condition information and provides the acquired fruit tree condition information to the database device 4, fruit tree harvesting support device 6, etc. via the network 7 or a recording medium, etc. When the fruit tree condition information is image information, the fruit tree condition acquisition device 3 is composed of, for example, a fixed or portable camera 30, a drone 31 equipped with a camera, etc. Note that when the producer terminal device 2 is equipped with the camera 20 as described above, the producer terminal device 2 may also function as the fruit tree condition acquisition device 3. When the fruit tree condition information is content information, the fruit tree condition acquisition device 3 is composed of, for example, a component analyzer 32, etc.

[0021] The database device 4 includes a fruit tree database 40 capable of registering various pieces of information about each fruit tree T cultivated in the orchard. The database device 4 receives fruit tree condition information from the producer terminal device 2 and the fruit tree condition acquisition device 3 at any time and registers the information in the fruit tree database 40, whereby various pieces of information about each fruit tree T are accumulated together with date information.

[0022] The machine learning device 5 operates as the main actor in the learning phase of machine learning, and, for example, acquires a portion of the fruit tree database 40 from the database device 4 as learning data 12, and generates a learning model 11 used in the fruit tree harvesting support device 6 by machine learning. The trained learning model 11 is provided to the fruit tree harvesting support device 6 via the network 7, a recording medium, or the like. In this embodiment, a case will be described in which supervised learning is adopted as the machine learning method.

[0023] The fruit tree harvesting support device 6 operates as the main body of the inference phase of machine learning, and uses the trained learning model 11 generated by the machine learning device 5 to generate harvesting information for the fruit tree T to be supported, and provides the harvesting information to the producer terminal device 2, database device 4, etc.

[0024] (Computer 900) 3 is a hardware configuration diagram showing an example of the computer 900. Each of the devices 2 to 6 of the fruit tree harvesting support system 1 is configured by a general-purpose or dedicated computer 900.

[0025] 3, the computer 900 includes, as its main components, a bus 910, a processor 912, a memory 914, an input device 916, an output device 917, a display device 918, a storage device 920, a communication I / F (interface) unit 922, an external device I / F unit 924, an I / O (input / output) device I / F unit 926, and a media input / output unit 928. Note that the above components may be omitted as appropriate depending on the application of the computer 900.

[0026] The processor 912 is composed of one or more arithmetic processing devices (such as a central processing unit (CPU), a micro-processing unit (MPU), a digital signal processor (DSP), or a graphics processing unit (GPU)), and operates as a control unit that controls the entire computer 900. The memory 914 stores various data and programs 930, and is composed of, for example, a volatile memory (such as a DRAM or SRAM) that functions as a main memory, a non-volatile memory (ROM), a flash memory, etc.

[0027] The input device 916 is composed of, for example, a keyboard, a mouse, a numeric keypad, an electronic pen, etc., and functions as an input unit. The output device 917 is composed of, for example, a sound (audio) output device, a vibration device, etc., and functions as an output unit. The display device 918 is composed of, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, etc., and functions as an output unit. The input device 916 and the display device 918 may be integrated into one device, such as a touch panel display. The storage device 920 is composed of, for example, a hard disk drive (HDD), a solid state drive (SSD), etc., and functions as a storage unit. The storage device 920 stores various data necessary for executing the operating system and the program 930.

[0028] The communication I / F unit 922 is connected to a network 940 such as the Internet or an intranet (which may be the same as network 7 in FIG. 1) via a wired or wireless connection and functions as a communication unit that transmits and receives data to and from other computers in accordance with a predetermined communication protocol. The external device I / F unit 924 is connected to an external device 950 such as a camera, printer, scanner, or reader / writer via a wired or wireless connection and functions as a communication unit that transmits and receives data to and from the external device 950 in accordance with a predetermined communication protocol. The I / O device I / F unit 926 is connected to an I / O device 960 such as various sensors and actuators and functions as a communication unit that transmits and receives various signals and data, such as detection signals from sensors and control signals to actuators, to and from the I / O device 960. The media input / output unit 928 is formed by a drive device such as a DVD (Digital Versatile Disc) drive or a CD (Compact Disc) drive and reads and writes data from and to media (non-transitory storage media) 970 such as DVDs and CDs.

[0029] In the computer 900 having the above configuration, the processor 912 loads a program 930 stored in the storage device 920 into the memory 914, executes the program, and controls each unit of the computer 900 via the bus 910. The program 930 may be stored in the memory 914 instead of the storage device 920. The program 930 may be recorded on the medium 970 in an installable file format or an executable file format and provided to the computer 900 via the media input / output unit 928. The program 930 may be provided to the computer 900 by being downloaded via the communication I / F unit 922 over the network 940. Furthermore, the computer 900 may implement various functions realized by the processor 912 executing the program 930 using hardware such as an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit).

[0030] The computer 900 may be, for example, a desktop computer or a portable computer, and may be any type of electronic device. The computer 900 may be a client computer, a server computer, or a cloud computer. The computer 900 may also be applied to other devices.

[0031] (Fruit Tree Database 40) 4 is a data structure diagram showing an example of the fruit tree database 40. The fruit tree database 40 is a database for managing the state of each fruit tree T and the state of the fruit that grows on each fruit tree T at the time of harvest, based on fruit tree identification information (fruit tree ID) assigned to each fruit tree T. The fruit tree database 40 is composed of, for example, a basic information table 400, a state record table 401, and a harvest record table 402.

[0032] The basic information table 400 has multiple records for recording basic information about the fruit tree T, and each record stores the fruit tree ID, variety, planting date, and planting location. The planting location indicates the location of the fruit tree T planted within the orchard, and is information that enables the location of the fruit tree T to be displayed on a map showing the orchard site, for example.

[0033] The condition record table 401 has multiple records for recording fruit tree condition information, and each record registers a fruit tree ID, acquisition date, fruit tree condition type, and fruit tree condition information. The fruit tree condition information is registered with image information, ratio information, or content information, and the fruit tree condition type is registered with the type of registered fruit tree condition information.

[0034] The harvest record table 402 has multiple records for recording the condition of the fruit at the time of harvest on the fruit tree T as harvest results, and each record registers one of the following: fruit tree ID, harvest date, harvest yield, fruit size, fruit color, skin roughness, amount of fruit dropping after flowering, fruit ripening period, fruit sugar content, fruit acidity, set rate (percentage of fruit that bears fruit), and fruit use.

[0035] The harvest yield is information on the total mass or number of all fruits that can be harvested from the fruit tree T. While the number of fruits is shown in FIG. 4 as an example, the present invention is not limited to this.

[0036] The fruit size is information about the diameter or standard of each fruit at the time of harvest on the fruit tree T. As an example, FIG. 4 shows the average size of all fruits that can be harvested from the fruit tree T as a standard consisting of three types: large, medium, and small, but is not limited to this.

[0037] The fruit color is information about the color of the fruit at the time of harvest from the fruit tree T. As an example, in FIG. 4, the average color value of all the fruits that can be harvested from the fruit tree T is shown as three color names: orange, yellow, and green, but it may also be a hexadecimal color code or RGB, and is not limited to these.

[0038] The skin roughness is information about the roughness of the outer skin that covers the surface of fruit at harvest time on fruit tree T. As an example, in Figure 4, the average skin roughness of all fruits that can be harvested from fruit tree T is shown as an integer value in a predetermined range (e.g., 0 to 5) with the standard skin roughness as the intermediate value, but this is not limited to this.

[0039] The amount of fruit that falls physiologically after flowering is information about the total mass or number of fruits that fall physiologically from the fruit tree T after flowering. As an example, FIG. 4 shows the ratio of the number of fruits that fall physiologically from the fruit tree T after flowering to the number of buds on the fruit tree T at the flowering time, but is not limited to this.

[0040] The fruit ripening period is information about the ripening period of the fruit on the fruit tree T. In FIG. 4, as an example, dates are shown as the ripening period of the fruit on the fruit tree T, but the present invention is not limited to this.

[0041] The sugar content of fruit is information about the percentage of sugar contained in fruit at the time of harvest from fruit tree T. As an example, Figure 4 shows the average sugar content of all fruit that can be harvested from fruit tree T as a Brix value, but this is not limited to this.

[0042] The acidity of fruit is information about the concentration of acid contained in fruit at the time of harvest from fruit tree T. As an example, in Figure 4, the average acidity of all fruit that can be harvested from fruit tree T is shown as how many grams of acid components are contained in 100 mL of solution (%), but the amount of acid loss, which is the degree of decrease in acidity of fruit within a specified period of time, may also be shown as an integer value within a specified range (e.g., 0 to 5), and is not limited to these.

[0043] The set rate is information about the rate at which fruit is borne on the fruit tree T. While FIG. 4 shows, as an example, the ratio of the number of fruits borne on the fruit tree T at the end of physiological abscission to the number of flowers that have bloomed, the rate is not limited to this.

[0044] The use of the fruit is information about the use of the fruit at the time of harvest from the fruit tree T. While the number of fruits available for sale is shown as an example in FIG. 4, it is not limited to this and may also be the ratio of the number of fruits for sale to the number of fruits used in processed foods. Here, processed foods are foods made by processing or seasoning fruit, such as canned fruit, jam, soft drinks, etc.

[0045] In addition, the fruit size, fruit color, skin roughness, fruit ripeness, fruit sugar content, fruit acidity, and fruit use may be recorded for each fruit.

[0046] Fig. 5 is an explanatory diagram showing an example of information for a specific fruit tree T based on the fruit tree database 40. In the fruit tree database 40, the information in each table shown in Fig. 4 is associated with a fruit tree ID, thereby centrally managing the past and present state of each fruit tree T and the harvest record of each fruit tree T.

[0047] FIG. 5 illustrates information about fruit tree T identified by its fruit tree ID ("T0103") at the flowering time in 2024. The information for 2021 to 2023 is the harvest performance of fruit tree T for each year. Information from the planting date through 2020 is also actually registered in the fruit tree database 40, but is omitted from FIG. 5. The information for 2024 is a prediction of the condition of the fruit at harvest time on fruit tree T, and is registered as harvest information D4 generated by the fruit tree harvest support device 6 based on image information D3 ("T0103-P24") at the flowering time in 2024. The information registered in the fruit tree database 40 is displayed on the display screen of the producer terminal device 2 and can be edited, searched, counted, printed, and processed in other ways.

[0048] (Machine Learning Device 5) 6 is a block diagram showing an example of a machine learning device 5 according to an embodiment. The machine learning device 5 includes a control unit 50, a communication unit 51, a learning data storage unit 52, and a trained model storage unit 53.

[0049] The control unit 50 functions as a learning data acquisition unit 500 and a machine learning unit 501. The communication unit 51 is connected to external devices (e.g., devices 2 to 4, 6, etc.) via the network 7, and functions as a communication interface for transmitting and receiving various types of data.

[0050] The learning data acquisition unit 500 is connected to an external device via the communication unit 51 and the network 7, and acquires learning data 12 consisting of fruit tree condition information including the condition of the fruit tree T at the flowering time of the fruit tree T to be learned, and harvest information including the condition of the fruit on the fruit tree T at the time of harvest. In this embodiment, the learning data acquisition unit 500 mainly acquires the learning data 12 based on past fruit tree condition information and harvest records registered in the fruit tree database 40. Note that the learning data acquisition unit 500 may also acquire the learning data 12 by accepting an operation by a manager or an experienced agricultural producer to input fruit tree condition information and harvest records via the producer terminal device 2.

[0051] The learning data storage unit 52 is a database that stores multiple sets of learning data 12 acquired by the learning data acquisition unit 500. As described above, the learning data 12 is composed of fruit tree condition information as input data and harvest information as output data. The learning data 12 is data used as teacher data (training data), verification data, and test data in supervised learning. The harvest information is data used as a correct answer label in supervised learning. The specific configuration of the database that constitutes the learning data storage unit 52 may be designed as appropriate.

[0052] The machine learning unit 501 performs machine learning using multiple sets of learning data 12 stored in the learning data storage unit 52. That is, the machine learning unit 501 inputs multiple sets of learning data 12 to the learning model 11 and causes the learning model 11 to learn the correlation between the fruit tree condition information and harvest information contained in the learning data 12, thereby generating a trained learning model 11.

[0053] The trained model storage unit 53 is a database that stores the trained learning model 11 (specifically, the adjusted weight parameter group) generated by the machine learning unit 501. The trained learning model 11 stored in the trained model storage unit 53 is provided to an actual system (for example, the fruit tree harvesting support device 6) via the network 7, a recording medium, or the like. Note that although the training data storage unit 52 and the trained model storage unit 53 are shown as separate storage units in FIG. 5, they may also be configured as a single storage unit.

[0054] In this embodiment, the learning data 12 is composed of first to third learning data 120-122, each of which contains image information, ratio information, and content information of the fruit tree T to be learned at the flowering period as fruit tree condition information, and the machine learning unit 501 generates first to third learning models 110-112 from the first to third learning data 120-122, respectively.

[0055] 7 is a diagram showing an example of the first learning model 110 and the first learning data 120 when the fruit tree condition information is image information. The first learning data 120 used for machine learning of the first learning model 110 is composed of image information of the fruit tree T to be learned at the flowering period and harvest information including the condition of the fruit on the fruit tree T at the time of harvest.

[0056] The image information is an image of the fruit tree T in flowering season taken by the producer terminal device 2 (camera 20) or the fruit tree condition acquisition device 3 (camera 30, drone 31). The image information may be an image of the fruit tree T taken from a specific direction, or, for example, if the fruit tree condition acquisition device 3 is configured with a drone 31, an image of the fruit tree T taken from directly above. The image information may be a plurality of images, for example, a plurality of images taken from a plurality of directions, or a combination of an image of the entire fruit tree T and an image of a portion of the fruit tree T. The image information may be either a monochrome image or a color image, or either a two-dimensional image or a three-dimensional image.

[0057] The learning data acquisition unit 500 acquires the first learning data 120 by referring to the fruit tree database 40 or by receiving input operations from the administrator via the producer terminal device 2. For example, the learning data acquisition unit 500 searches the fruit tree database 40 using, as a learning data search condition, for example, whether the amount of change in the ratio of flowers to buds at the flowering time is within a specific range from the next year. At this time, because the flowering time of fruit tree T occurs in an annual cycle, it is determined for each year for each fruit tree T whether the year satisfies the learning data search condition.

[0058] Then, the learning data acquisition unit 500 identifies a fruit tree T that satisfies the learning data search conditions and the target year (which may be specified by the administrator), and acquires image information of the fruit tree T taken during the flowering period of the target year and the harvest record of the fruit tree T (some or all of the harvest record may be specified by the administrator) from the fruit tree database 40, thereby acquiring first learning data 120. For example, information D1 and D2 in the ranges surrounded by dashed lines in FIG. 5 each correspond to the first learning data 120. FIG. 7 illustrates information D1 shown in FIG. 5 as an example of the first learning data 120.

[0059] The first learning model 110 employs, for example, a convolutional neural network (CNN) structure and includes an input layer 1100, an intermediate layer 1101, and an output layer 1102. Synapses (not shown) that connect each neuron are laid between each layer, and each synapse is associated with a weight. A group of weight parameters consisting of the weights of each synapse is adjusted by machine learning.

[0060] The input layer 1100 has neurons whose number corresponds to the number of pixels in the image information as input data, and the pixel value of each pixel is input to each neuron. The intermediate layer 1101 is composed of, for example, a convolutional layer, a pooling layer, and a fully connected layer. The output layer 1102 has neurons whose number corresponds to the state of the fruit at the time of harvest contained in the harvest information as output data, and the judgment results (inference results) of the state of each fruit are output as output data.

[0061] The first learning model 110 is configured as a regression model when the harvest information is expressed numerically, such as the harvest yield, or as a classification model when the harvest information is expressed by color names, such as the color of the fruit.

[0062] 8 is a diagram showing an example of the second learning model 111 and the second learning data 121 when the fruit tree condition information is percentage information. The second learning data 121 used for machine learning of the second learning model 111 is composed of percentage information at the flowering time of the fruit tree T to be learned, and harvest information including the condition of the fruit at the time of harvest on the fruit tree T. The harvest information is similar to the first learning data 120, and therefore a description thereof will be omitted.

[0063] The ratio information is, for example, the result of a producer visually inspecting a fruit tree T during the flowering season and determining the ratio of flowers to buds, and is input via the producer terminal device 2. The ratio information is defined as the value input as the producer's determination result, normalized to a predetermined range (for example, 0 to 1), with the larger the value, the higher the ratio of flowers.

[0064] The learning data acquisition unit 500 acquires the second learning data 121 by referring to the fruit tree database 40 or by accepting an input operation of the administrator via the producer terminal device 2. For example, the learning data acquisition unit 500 identifies a fruit tree T that meets the learning data search conditions and a target year (which may be specified by the administrator), and acquires, from the fruit tree database 40, percentage information about the fruit tree T at the flowering time of the target year and the work content that has been performed on the fruit tree T until the flowering time of the next year (some or all of the work content may be specified by the administrator), thereby acquiring the second learning data 121.

[0065] The second learning model 111 employs, for example, a neural network structure, and includes an input layer 1110, an intermediate layer 1111, and an output layer 1112. Synapses (not shown) that connect each neuron are laid between each layer, and a group of weight parameters consisting of the weights of each synapse are adjusted by machine learning.

[0066] The input layer 1110 has neurons corresponding to the ratio information as input data, and values ​​indicated by the ratio information are input to the neurons. The intermediate layer 1111 is composed of, for example, multiple layers. The output layer 1112 has neurons the number of which corresponds to the condition of the fruit at the time of harvest, which is included in the harvest information as output data, and the judgment results (inference results) of each task content are output as output data.

[0067] 9 is a diagram showing an example of the third learning model 112 and the third learning data 122 when the fruit tree condition information is content information. The third learning data 122 used for machine learning of the third learning model 112 is composed of content information at the flowering time of the fruit tree T to be learned, and harvest information including the condition of the fruit at the time of harvest on the fruit tree T. The harvest information is similar to the first learning data 120, and therefore a description thereof will be omitted.

[0068] The content information is obtained by measuring endogenous plant hormones present in the fruit tree T at the flowering stage using a component analyzer 32 serving as the fruit tree condition acquisition device 3. Examples of endogenous plant hormones include gibberellins, cytokinins, and auxins, and the content information represents the content of at least one of these endogenous plant hormones. The content of the endogenous plant hormones described above affects the growth status of the fruit tree T, for example, affecting the ratio of flowers to buds at the flowering stage. The content information may be expressed directly as the content measured by the component analyzer 32, or may be the measured value normalized to a predetermined range (for example, 0 to 1).

[0069] The learning data acquisition unit 500 acquires the third learning data 122 by referring to the fruit tree database 40 or by accepting an input operation by the administrator via the producer terminal device 2. For example, the learning data acquisition unit 500 identifies a fruit tree T that meets the learning data search conditions and a target year (which may be specified by the administrator), and acquires, from the fruit tree database 40, content information of the fruit tree T at the flowering time of the target year and the harvest record of the fruit tree T (some or all of the harvest record may be specified by the administrator), thereby acquiring the third learning data 122.

[0070] The third learning model 112 employs, for example, a neural network structure, and includes an input layer 1120, an intermediate layer 1121, and an output layer 1122. Synapses (not shown) that connect each neuron are laid between each layer, and a group of weight parameters consisting of the weights of each synapse are adjusted by machine learning.

[0071] The input layer 1120 has neurons whose number corresponds to the content information as input data (two, in this embodiment, corresponding to the gibberellin and cytokinin contents), and for example, normalized content values ​​are input to each neuron. The intermediate layer 1121 is composed of, for example, multiple layers. The output layer 1122 has neurons whose number corresponds to the state of the fruit at the time of harvest contained in the harvest information as output data, and the judgment results (inference results) of the state of each fruit are output as output data.

[0072] (machine learning methods) 10 is a flowchart showing an example of a machine learning method performed by the machine learning device 5. In the following, a description will be given of generating a learning model 11 using multiple sets of learning data 12, but the method can also be applied to creating first to third learning models 110 to 112 using first to third learning data 120 to 122, respectively.

[0073] First, in step S100, the training data acquisition unit 500 acquires a desired number of training data 12 from the fruit tree database 40 or the like as a preliminary preparation for starting machine learning, and stores the acquired training data 12 in the training data storage unit 52. The number of training data 12 to be prepared here may be set in consideration of the inference accuracy required for the ultimately obtained training model 11.

[0074] Next, in step S110, in order to start machine learning, the machine learning unit 501 prepares a pre-learning learning model 11. The pre-learning learning model 11 prepared here is configured with the neural network models exemplified in Figures 7 to 9, and the weights of each synapse are set to initial values.

[0075] Next, in step S120, the machine learning unit 501 acquires, for example, one set of training data 12 at random from the multiple sets of training data 12 stored in the training data storage unit 52.

[0076] Next, in step S130, the machine learning unit 501 inputs image information (input data) as fruit tree condition information contained in a set of learning data 12 to the input layer of the prepared learning model 11 before (or during) learning. As a result, harvest information (output data) is output as an inference result from the output layer of the learning model 11, and this output data has been generated by the learning model 11 before (or during) learning. Therefore, in the state before (or during) learning, the output data output as an inference result indicates information different from the harvest information (correct label) contained in the learning data 12.

[0077] Next, in step S140, the machine learning unit 501 performs machine learning by comparing the harvest information (correct label) included in the set of learning data 12 acquired in step S120 with the harvest information (output data) output from the output layer as an inference result in step S130, and performing a process of adjusting the weight of each synapse (back propagation).In this way, the machine learning unit 501 causes the learning model 11 to learn the correlation between fruit tree condition information and harvest information.

[0078] Next, in step S150, the machine learning unit 501 determines whether a predetermined learning termination condition has been met, for example, based on the evaluation value of an error function based on the harvest information (correct label) included in the learning data and the harvest information (output data) output as an inference result, or the remaining number of unlearned learning data stored in the learning data storage unit 52.

[0079] In step S150, if the machine learning unit 501 determines that the learning termination condition is not satisfied and that machine learning should be continued (No in step S150), the process returns to step S120, and the processes of steps S120 to S140 are performed multiple times on the learning model 11 under training using unlearned training data 12. On the other hand, in step S150, if the machine learning unit 501 determines that the learning termination condition is satisfied and that machine learning should be terminated (Yes in step S150), the process proceeds to step S160.

[0080] Then, in step S160, the machine learning unit 501 stores the trained learning model 11 (adjusted weight parameter group) generated by adjusting the weights associated with each synapse in the trained model storage unit 53, thereby completing the series of machine learning methods shown in Fig. 10. In the machine learning method, step S100 corresponds to a learning data storage step, steps S110 to S150 correspond to a machine learning step, and step S160 corresponds to a trained model storage step.

[0081] As described above, the machine learning device 5 and machine learning method of this embodiment can provide a learning model 11 (first to third learning models 110-112) that can generate (infer) harvest information including the condition of the fruit at harvest time on the fruit tree T from fruit tree condition information including the condition of the fruit tree T to be supported at the flowering period.

[0082] (Fruit harvesting support device 6) Fig. 11 is a block diagram showing an example of a fruit tree harvesting support device 6 according to an embodiment. Fig. 12 is a functional explanatory diagram showing an example of a fruit tree harvesting support device 6 according to an embodiment. The fruit tree harvesting support device 6 includes a control unit 60, a communication unit 61, and a trained model storage unit 62.

[0083] The control unit 60 functions as an acquisition unit 600, a generation processing unit 601, and an output processing unit 602. The communication unit 61 is connected to external devices (e.g., devices 2 to 6) via the network 7, and functions as a communication interface for transmitting and receiving various types of data.

[0084] The acquisition unit 600 is connected to an external device via the communication unit 61 and the network 7, and acquires fruit tree condition information including the condition of the fruit tree T to be supported at the time of flowering of the fruit tree T. For example, when the acquisition unit 600 acquires image information as information representing the condition of the fruit tree T to be supported, the acquisition unit 600 receives image information of the fruit tree T to be supported taken by the producer terminal device 2 (camera 20) or the fruit tree condition acquisition device 3 (camera 30, drone 31) from the producer terminal device 2 or the fruit tree condition acquisition device 3, or refers to image information of the fruit tree T to be supported registered in the fruit tree database 40. When the acquisition unit 600 acquires percentage information as information representing the condition of the fruit tree T to be supported, the acquisition unit 600 receives percentage information of the fruit tree T to be supported input by the producer to the producer terminal device 2 from the producer terminal device 2, or refers to the percentage information of the fruit tree T to be supported registered in the fruit tree database 40. When the acquisition unit 600 acquires content information as information representing the condition of the fruit tree T to be supported, it receives content information of the fruit tree T to be supported measured by the component analyzer 32 serving as the fruit tree condition acquisition device 3 from the fruit tree condition acquisition device 3, receives content information of the fruit tree T to be supported from the producer terminal device 2 in which the measurement results of the component analyzer 32 have been input by the producer to the producer terminal device 2, or refers to the content information of the fruit tree T to be supported registered in the fruit tree database 40.

[0085] The generation processing unit 601 inputs the fruit tree condition information for the fruit tree T to be supported, acquired by the acquisition unit 600, into the learning model 11, thereby generating harvest information for the fruit tree T. Specifically, the generation processing unit 601 inputs the fruit tree condition information into one of the first to third learning models 110 to 112 depending on the type of fruit tree condition information.

[0086] The trained model storage unit 62 is a database that stores trained learning models 11 (specifically, first to third learning models 110-112) used by the generation processing unit 601. Note that the learning models 11 stored in the trained model storage unit 62 are not limited to those generated according to the type of fruit tree condition information. For example, multiple trained models with different conditions, such as machine learning techniques, types of data included in fruit tree condition information, and types of fruit conditions at the time of harvest included in harvest information, may be stored and selectively used. Furthermore, the trained model storage unit 62 may be substituted by a storage unit of an external computer (e.g., a server-type computer or a cloud-type computer). In this case, the generation processing unit 601 may generate the above-mentioned harvest information by accessing the external computer.

[0087] The output processing unit 602 performs output processing for outputting the harvest information generated by the generation processing unit 601. For example, the output processing unit 602 may transmit the harvest information to the producer terminal device 2 so that a display screen based on the harvest information is displayed on the producer terminal device 2, or may transmit the harvest information to the database device 4 so that the harvest information is registered in the fruit tree database 40.

[0088] (Fruit tree harvesting support method) 13 is a flowchart showing an example of a fruit tree harvesting support method using the fruit tree harvesting support device 6. The following describes an example of the operation of the fruit tree harvesting support device 6 when a producer photographs a fruit tree T to be supported using the camera 20 of the producer terminal device 2.

[0089] First, in step S200, the producer photographs the fruit tree T to be supported with the camera 20 of the producer terminal device 2 and inputs a fruit tree ID (for example, "T0103" shown in FIG. 5) that identifies the fruit tree T to be supported. The producer terminal device 2 then generates image information of the photographed fruit tree T to be supported and transmits the image information and the fruit tree ID to the fruit tree harvesting support device 6. The image information and fruit tree ID are also transmitted to the database device 4 and registered in the fruit tree database 40. That is, image information D3 ("T0103-P24") surrounded by a dashed line in FIG. 5 is registered in association with the fruit tree ID "T0103."

[0090] Next, in step S210, the acquisition unit 600 of the fruit tree harvesting support device 6 acquires fruit tree condition information for the fruit tree T to be supported by receiving the image information ("T0103-P24") and fruit tree ID ("T0103") transmitted in step S200, as shown in Figure 12.

[0091] Next, in step S220, the generation processing unit 601 inputs the fruit tree condition information for the fruit tree T to be supported, acquired in step S210, into the learning model 11, thereby generating harvest information for the fruit tree T. Specifically, as shown in FIG. 12 , the generation processing unit 601 inputs image information for the fruit tree T to be supported ("T0103-P24") into the first learning model 110, thereby generating harvest information for the fruit tree T.

[0092] Next, in step S230, the output processing unit 602 performs output processing to output the harvest information generated in step S220, and transmits the harvest information to the producer terminal device 2. The output processing unit 602 also transmits the harvest information together with the fruit tree ID acquired in step S210 to the database device 4, whereby the harvest information is registered in the fruit tree database 40. Since the harvest information here is generated at the timing when fruit tree condition information is acquired during the flowering period, it is registered, for example, in the harvest record table 402 as a prediction of the condition of the fruit at the time of harvest on the fruit tree T. That is, the harvest information D4 surrounded by a dashed line in FIG. 5 is registered in association with the fruit tree ID "T0103."

[0093] Next, in step S240, when the producer terminal device 2 receives the harvest information transmitted in step S230, it displays a display screen showing a prediction of the condition of the fruit that will grow on the fruit tree T that is the target of support at the time of harvest based on the harvest information, and the series of steps in the fruit tree harvest support method shown in Figure 13 is completed. In the fruit tree harvest support method, step S210 corresponds to the acquisition step, step S220 corresponds to the generation processing step, and step S230 corresponds to the output processing step.

[0094] 13 is described above as being executed during the flowering period, but the processes from step S210 onwards may be executed at any timing chosen by the producer. For example, the producer may have the fruit tree harvesting support device 6 receive a request from the producer terminal device 2 when the period of physiological drop ends, execute the processes from step S210 onwards and generate harvest information.

[0095] Also, in step S220, if the fruit tree condition information is percentage information, the generation processing unit 601 inputs the fruit tree condition information into the second learning model 111, and if the fruit tree condition information is content information, the generation processing unit 601 inputs the fruit tree condition information into the third learning model 112.

[0096] Furthermore, if the drone 31 serving as the fruit tree condition acquisition device 3 patrols the sky above the orchard and sequentially photographs the target fruit trees T and transmits the image information to the fruit tree harvesting support device 6, the fruit tree harvesting support device 6 may repeatedly execute the processes from step S210 onward each time it receives the image information. In this case, the photographing position is added to the image information, and the fruit tree ID may be identified from the photographing position.

[0097] As described above, according to the fruit tree harvesting support device 6 and fruit tree harvesting support method of this embodiment, fruit tree condition information, including the condition of the fruit tree T to be supported at the flowering stage, is input into the learning model 11, and harvest information, including the condition of the fruit on the fruit tree T at the time of harvest, is generated. This is based on the new finding that the condition of the fruit on the fruit tree T at the flowering stage is related to the condition of the fruit on the fruit tree at the time of harvest. By introducing this new finding into the learning model 11, it is possible to predict harvest information, including the condition of the fruit on the fruit tree T at the time of harvest, based on the condition of the fruit tree T at the flowering stage, and producers can therefore efficiently allocate personnel and conduct sales activities based on the predictions.

[0098] (Other embodiments) The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention, all of which are included in the technical concept of the present invention.

[0099] In the above embodiment, the database device 4, the machine learning device 5, and the fruit tree harvesting support device 6 are described as being configured as separate devices, but these three devices may be configured as a single device, or any two of these three devices may be configured as a single device. Furthermore, at least one of the machine learning device 5 and the fruit tree harvesting support device 6 may be incorporated into the producer terminal device 2.

[0100] In the above embodiment, the machine learning unit 501 of the machine learning device 5 is described as generating the first to third learning models 110-112, but the machine learning unit 501 may be configured to generate any of the first to third learning models 110-112, and in that case, the learning data necessary for machine learning from the first to third learning data 120-122 may be acquired by the learning data acquisition unit 500 and stored in the learning data storage unit 52. Also, in the above embodiment, the generation processing unit 601 of the fruit tree harvest support device 6 is described as generating harvest information using the first to third learning models 110-112, but the generation processing unit 601 may be configured to use any of the first to third learning models 110-112, and in that case, the fruit tree condition information necessary for generating harvest information may be acquired by the acquisition unit 600.

[0101] In the above embodiment, a case has been described in which a neural network is used as a learning model for realizing machine learning by the machine learning unit 501, but other machine learning models may also be used. Examples of other machine learning models include tree types such as decision trees and regression trees, ensemble learning such as bagging and boosting, neural network types (including deep learning) such as recurrent neural networks, convolutional neural networks and LSTM, clustering types such as hierarchical clustering, non-hierarchical clustering, k-nearest neighbors and k-means, multivariate analyses such as principal component analysis, factor analysis and logistic regression, and support vector machines.

[0102] In the above embodiment, the fruit tree harvesting support system 1 has been described mainly as being used in a single orchard, but the machine learning device 5 and the fruit tree harvesting support device 6 may be used in multiple orchards, or the machine learning device 5 and the fruit tree harvesting support device 6 may be used in different orchards.

[0103] (Inference device, inference method or inference program) The present invention can be provided not only in the form of the fruit tree harvesting support device 6 (fruit tree harvesting support method or fruit tree harvesting support program) according to the above embodiment, but also in the form of an inference device (inference method or inference program) used to support the cultivation of a fruit tree to be supported. In this case, the inference device (inference method or inference program) can include a memory and a processor, and the processor executes a series of processes. The series of processes includes an acquisition process (acquisition step) for acquiring fruit tree condition information indicating the condition of the fruit tree to be supported at the time of flowering, and an inference process (inference step) for inferring harvest information including the condition of fruit that will grow on the fruit tree at the time of harvest, once the fruit tree condition information for the fruit tree to be supported has been acquired in the acquisition process.

[0104] By providing it in the form of an inference device (inference method or inference program), it can be more easily applied to various devices than when it is implemented as a fruit tree harvesting support device 6. It will be obvious to those skilled in the art that when the inference device (inference method or inference program) infers harvest information, it may apply an inference method implemented by the generation processing unit using a trained learning model generated by the machine learning device 5 and machine learning method according to the above embodiment. [Explanation of symbols]

[0105] 1... Fruit tree harvesting support system, 2... Producer terminal device, 3... Fruit tree condition acquisition device, 4...Database device, 5...Machine learning device, 6... Fruit tree harvesting support device, 7... Network, 11, 11a-11c...Learning model, 12, 12a-12c...Learning data, 20...camera, 30...camera, 31...drone, 32...component analyzer, 40...Fruit tree database, 50...control unit, 51...communication unit, 52...learning data storage unit, 53...Trained model memory unit, 60...control unit, 61...communication unit, 62...trained model storage unit 400...Basic information table, 401...Status record table, 402...Harvest record table, 500...learning data acquisition unit, 501...machine learning unit, 600: Acquisition unit, 601: Generation processing unit, 602: Output processing unit, 900...Computer

Claims

1. an acquisition unit that acquires fruit tree condition information including the condition of the fruit tree to be supported at the time of flowering; a generation processing unit that generates the harvest information for the fruit tree by inputting the fruit tree condition information for the support target fruit tree acquired by the acquisition unit into a learning model that has been trained to learn a correlation between fruit tree condition information, which includes the condition of the fruit tree at the flowering time of the fruit tree, and harvest information, which includes the condition of fruit that grows on the fruit tree at the time of harvest; The fruit tree status information includes the following status information: image information of the fruit tree taken during the flowering period; and proportion information indicating the proportion of flowers and buds of the fruit tree during the flowering period; and content information indicating the content of endogenous plant hormones in the fruit tree at the flowering time, Fruit tree harvesting support device.

2. The harvest information includes, as the condition of the fruit, the yield of the fruit; the size of the fruit; the color of the fruit; Roughness of the peel, Physiological fruit drop after flowering, the ripening stage of the fruit; the sugar content of the fruit; the acidity of the fruit; The rate at which the fruit is produced; and The fruit includes at least one of the uses thereof. The fruit tree harvesting support device according to claim 1.

3. An inference device used to support the harvesting of a fruit tree to be supported, the inference device comprises a memory and a processor; The processor: an acquisition process for acquiring fruit tree condition information indicating the condition of the fruit tree to be supported at the time of flowering of the fruit tree; When the fruit tree condition information for the support target fruit tree is acquired in the acquisition process, an inference process is executed to infer harvest information including the condition of fruit that grows on the fruit tree at the time of harvest; The fruit tree status information includes the following status information: image information of the fruit tree taken during the flowering period; and proportion information indicating the proportion of flowers and buds of the fruit tree during the flowering period; and content information indicating the content of endogenous plant hormones in the fruit tree at the flowering time, Reasoning device.

4. a learning data storage unit that stores multiple sets of learning data consisting of fruit tree condition information including the condition of the fruit tree at the time of flowering of the fruit tree to be learned and harvest information including the condition of the fruit on the fruit tree at the time of harvest; a machine learning unit that inputs a plurality of sets of the learning data into a learning model to cause the learning model to learn a correlation between the fruit tree condition information and the harvest information; a learned model storage unit that stores the learned model in which the correlation is learned by the machine learning unit, The fruit tree status information includes the following status information: image information of the fruit tree taken during the flowering period; and proportion information indicating the proportion of flowers and buds of the fruit tree during the flowering period; and content information indicating the content of endogenous plant hormones in the fruit tree at the flowering time, Machine learning device.

5. an acquisition step of acquiring fruit tree condition information including the condition of the fruit tree to be supported at the time of flowering of the fruit tree; a generation processing step of generating the harvest information for the fruit tree by inputting the fruit tree condition information for the support target fruit tree acquired in the acquisition step into a learning model that has been trained to learn the correlation between fruit tree condition information, which includes the condition of the fruit tree at the flowering time of the fruit tree, and harvest information, which includes the condition of the fruit on the fruit tree at the time of harvest; The fruit tree status information includes the following status information: image information of the fruit tree taken during the flowering period; and proportion information indicating the proportion of flowers and buds of the fruit tree during the flowering period; and content information indicating the content of endogenous plant hormones in the fruit tree at the flowering time, How to support fruit harvesting.

6. An inference method used to support cultivation of a fruit tree to be supported, an acquisition step of acquiring fruit tree condition information indicating the condition of the fruit tree to be supported at the time of flowering; When the fruit tree condition information for the support target fruit tree is acquired in the acquisition step, an inference step is executed to infer harvest information including the condition of fruit that grows on the fruit tree at the time of harvest; The fruit tree status information includes the following status information: image information of the fruit tree taken during the flowering period; and proportion information indicating the proportion of flowers and buds of the fruit tree during the flowering period; and content information indicating the content of endogenous plant hormones in the fruit tree at the flowering time, Reasoning method.

7. a learning data storage step of storing in a learning data storage unit a plurality of sets of learning data, each set consisting of fruit tree condition information including the condition of the fruit tree at the time of flowering of the fruit tree to be learned and harvest information including the condition of the fruit on the fruit tree at the time of harvest; a machine learning process of inputting a plurality of sets of the learning data into a learning model to allow the learning model to learn a correlation between the fruit tree condition information and the harvest information; a learned model storage step of storing the learned model, which has learned the correlation through the machine learning step, in a learned model storage unit; The fruit tree status information includes the following status information: image information of the fruit tree taken during the flowering period; and proportion information indicating the proportion of flowers and buds of the fruit tree during the flowering period; and content information indicating the content of endogenous plant hormones in the fruit tree at the flowering time, Machine learning methods.

Images