Method for estimating the state of crops, program for estimating the state of crops, and apparatus for estimating the state of crops.

Abstract

To automatically estimate the number of new leaves of an agricultural product body.SOLUTION: An image acquisition unit acquires a plurality of images obtained by photographing an area where a plurality of agricultural product bodies is present at different timing, and a motion information extraction unit 34 determines an optical flow from dynamic images obtained by connecting the plurality of images to each other and extracts pieces of motion information in the area. A range identifying unit 36 identifies ranges corresponding to the agricultural product bodies in the area on the basis of the distribution of extracted pieces of motion information, and a cluster generation unit performs clustering processing on the pieces of motion information on the basis of the positions and directions of the pieces of motion information to generate clusters (FIG. 8). A new leaf number estimation unit estimates the number of the clusters for every range and the number of new leaves in each of the agricultural product bodies.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a method for estimating the state of a crop, a state estimation program, and a state estimation device.

Background Art

[0002] As a method for grasping the growth state of agricultural crops (for example, tomatoes), there is a method for grasping the number of expanded leaves. When grasping the number of expanded leaves, it is common to investigate the number of new leaves. However, since the investigation of the number of new leaves is performed visually, it is laborious.

[0003] Conventionally, based on a division criterion based on growth characteristics, an image is divided into a plurality of divided regions, a statistic is calculated using optical flow in each divided region, and a growth state is obtained based on the calculated statistic (see, for example, Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, even when using Patent Document 1 above, it is not possible to automatically investigate the number of new leaves or the number of expanded leaves.

[0006] An object of the present invention is to provide a method for estimating the state of a crop, a state estimation program, and a state estimation device capable of automatically estimating the number of new leaves of a crop.

Means for Solving the Problems

[0007] The present invention provides a method for estimating the state of a crop, which involves a computer performing the following steps: acquiring multiple images taken at different times in an area where multiple crops exist; determining optical flow from the multiple images; extracting motion information within the area; identifying a range corresponding to each crop within the area based on the distribution of the extracted motion information; generating clusters by clustering the motion information for each identified range based on the position and orientation of the motion information; and estimating the number of clusters in each range as the number of new leaves in each crop. [Effects of the Invention]

[0008] The crop condition estimation method, condition estimation program, and condition estimation device of the present invention have the effect of automatically estimating the number of new leaves on a crop. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the configuration of an agricultural system according to one embodiment. [Figure 2] This is a diagram showing an example of an image captured by a camera. [Figure 3] This figure shows the hardware configuration of the processing unit shown in Figure 1. [Figure 4] Figure 1 is a functional block diagram of the processing unit. [Figure 5] This figure shows an example of a table structure for a crop plant. [Figure 6] This diagram schematically shows the motion information extracted by the motion information extraction unit from the video in region X. [Figure 7] This is a diagram illustrating the processing of the range identification section. [Figure 8] Figures 8(a) and 8(b) are diagrams illustrating the processing of the cluster generation unit. [Figure 9] Figure 9(a) is a diagram illustrating the processing of the new leaf number estimation unit, and Figure 9(b) is a graph illustrating the processing of the unfolded leaf number estimation unit. [Figure 10]This is a flowchart showing the processing steps of the processing unit. [Modes for carrying out the invention]

[0010] The following describes in detail one embodiment of the agricultural system with reference to Figures 1 to 10. Figure 1 schematically shows the configuration of the agricultural system 100 according to one embodiment. The agricultural system 100 of this embodiment is, for example, a system that can be used by a producer (hereinafter referred to as a worker) who cultivates tomatoes.

[0011] As shown in Figure 1, the agricultural system 100 comprises a processing unit 10 as a state estimation device and multiple cameras 70 as imaging devices. The processing unit 10 and each camera 70 are connected to a network such as the Internet or a LAN (Local Area Network), and the processing unit 10 is capable of acquiring images captured by each camera 70.

[0012] Multiple cameras 70 are installed (fixed) at predetermined intervals on beam-like members 74 laid above tomato plants (crop bodies) being cultivated in a field (greenhouse, etc.). The field of view (shooting range) of each of the multiple cameras 70 is fixed, so that all the crop bodies being cultivated in the field can be photographed by all the cameras 70 in the field.

[0013] Each camera 70 takes a picture at predetermined intervals and transmits the captured image to the processing unit 10. For example, the image that one camera 70 can capture is an image like the one shown in Figure 2.

[0014] The processing unit 10 is an information processing device such as a PC or smartphone that can be used by the worker. It acquires images taken at predetermined time intervals by a camera 70 installed in the field, and performs information processing using the acquired images to estimate the number of new leaves and the number of unfolded leaves on the crops being cultivated in the field. The processing unit 10 also provides the worker with the estimated results by displaying them.

[0015] FIG. 3 schematically shows the hardware configuration of the processing device 10. As shown in FIG. 3, the processing device 10 includes a CPU (Central Processing Unit) 90, a ROM (Read Only Memory) 92, a RAM (Random access memory) 94, a storage unit (e.g., HDD (Hard Disk Drive), SSD (Solid State Drive), etc.) 96, a network interface 97, a display unit 93, an input unit 95, and a drive 99 for a portable storage medium, etc. The display unit 93 includes a liquid crystal display, etc., and the input unit 95 includes a keyboard, a mouse, a touch panel, etc. Each component of the processing device 10 is connected to a bus 98. In the processing device 10, the CPU 90 executes a program stored in the ROM 92 or the HDD 96 (including the crop state estimation program), or a program read by the drive 99 for a portable storage medium from the portable storage medium 91, whereby the functions of each part shown in FIG. 4 are realized. Note that the functions of each part in FIG. 4 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0016] FIG. 4 shows a functional block diagram of the processing device 10. In the processing device 10, when the CPU 90 executes a program, as shown in FIG. 4, an image acquisition unit 30 as an acquisition unit, a video generation unit 32, a motion information extraction unit 34 as an extraction unit, a range specification unit 36 as a specification unit, a cluster generation unit 38 as a generation unit, a new leaf number estimation unit 40 as an estimation unit, and a developed leaf number estimation unit 42 are realized. Note that FIG. 4 also shows an image DB 50 and a crop object table 52 stored in the storage unit 96, etc. Although details will be described later, the crop object table 52 has a table structure as shown in FIG. 5 as an example.

[0017] The image acquisition unit 30 acquires an image (image data) transmitted from each camera 70 and stores the image data in the image DB 50 in association with the identification information (camera ID) of each camera.

[0018] The video generation unit 32 connects a predetermined number of images (images captured within a predetermined time) captured by the same camera 70 to generate a video. The video generation unit 32 transmits the generated video to the motion information extraction unit 34. It can be said that the videos generated from the images captured by each camera are videos that capture different regions within the field.

[0019] The motion information extraction unit 34 obtains the optical flow from each of the videos (videos of different regions) generated by the video generation unit 32, and extracts the motion information within each region. The motion information extraction unit 34 transmits the extracted motion information to the range specification unit 36 and the cluster generation unit 38. Here, as a method for obtaining the optical flow, for example, the method described in 'B. D. Lucas and T. Kanade (1981), An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, pages 121-130' can be adopted.

[0020] FIG. 6 is a diagram schematically showing the motion information extracted by the motion information extraction unit 34 from the video of region X. As can be seen from the diagram showing an enlargement of the inside of the dashed circle in FIG. 6, in the present embodiment, the motion information is represented by an arrow (vector), the direction of the arrow indicates the direction of the motion, and the length of the arrow indicates the magnitude of the motion. Note that the motion information may be represented by other means than an arrow. For example, the motion information may be displayed with color separation, the type of color indicating the direction of the motion and the brightness indicating the magnitude of the motion.

[0021] Here, the inventors focused on the fact that new leaves (new leaves) on a tomato plant perform extremely complex three-dimensional movements such as rotational movements, and that leaves other than new leaves hardly move. Furthermore, the inventors focused on the fact that although each new leaf moves in the same way, each new leaf moves differently, and that the range of identical movements represents one new leaf. They decided to use this movement information to estimate the number of new leaves and the number of unfolded leaves. Note that in a tomato plant (crop body), multiple leaflets (for example, 7 leaflets) are grouped together to form one new leaf, but in Figure 6, for the sake of explanation and illustration, three leaflets are shown to form one new leaf.

[0022] Returning to Figure 4, the range identification unit 36 ​​identifies the range corresponding to each of the multiple crop bodies present within each region (each video) based on the motion information extracted by the motion information extraction unit 34. The range identification unit 36 ​​transmits the identified range information to the cluster generation unit 38. For example, in the case of the video of region X in Figure 6, the range identification unit 36 ​​identifies ranges 1 to 4 based on the distribution of motion information, as shown by the thick outline in Figure 7. The range identification unit 36 ​​then assumes that different crop bodies exist in these ranges 1 to 4 and assigns identification information (crop body ID) 1 to 4 to the crop bodies present in each range.

[0023] The cluster generation unit 38 generates clusters by clustering the motion information based on the position and orientation of each motion information for each range (for each crop) identified by the range identification unit 36.

[0024] Specifically, the cluster generation unit 38 removes motion information that is smaller than a predetermined size (minute motion information) from the motion information present within range 1 in Figure 7. The cluster generation unit 38 also obtains the XY coordinates of the starting point and the orientation Z (Angle) of each motion information in the XY coordinate system shown in Figure 8(a), and plots them in the XYZ coordinate system shown in Figure 8(b). Then, the cluster generation unit 38 performs clustering on each point in the XYZ coordinate system of Figure 8(b) to generate clusters as shown by the dashed circles. For clustering, for example, the k-means method can be used. The cluster generation unit 38 also identifies the number of generated clusters and transmits this information to the new leaf number estimation unit 40. For determining the number of clusters, for example, the elbow method can be used.

[0025] The new leaf number estimation unit 40 estimates the number of new leaves on each crop corresponding to each range based on the number of clusters within each range generated by the cluster generation unit 38. The new leaf number estimation unit 40 stores the estimated number of new leaves in the crop table 52. Here, as shown in Figure 5, the crop table 52 is a table that stores the estimated number of new leaves and the estimated number of unfolded leaves for each crop ID, based on the number of days elapsed since the start of cultivation. For example, if the number of clusters on day n in range 1 is 3, the new leaf number estimation unit 40 estimates that the number of new leaves on day n of the crop with crop ID=1 is 3, as shown in Figure 9(a), and stores this information in the crop table 52.

[0026] Returning to Figure 4, the leaf number estimation unit 42 refers to the change in the number of new leaves stored in the crop table 52 and estimates when and how many leaves unfolded. For example, if we graph the change in the number of new leaves of the crop with crop ID=1 in Figure 5, it will look like Figure 9(b). In this case, the number of new leaves increased by 1 between day n+3 and day n+4, so the leaf number estimation unit 42 stores the information that the number of unfolded leaves on day n+4 is +1 in the crop table 52 in Figure 5.

[0027] (Regarding the processing of the processing unit 10) Next, the processing of the processing unit 10 will be explained according to the flowchart in Figure 10.

[0028] When the process shown in Figure 10 begins, first, in step S10, the image acquisition unit 30 determines whether or not an image has been transmitted. If the determination in step S10 is positive, the process moves to step S12, where the image acquisition unit 30 stores the image (image data) in the image DB 50, linked to the camera ID. After that, the process moves to step S14. If the determination in step S10 is negative, the process moves directly to step S14 without going through step S12.

[0029] When the process moves to step S14, the video generation unit 32 determines whether a predetermined timing has been reached. The predetermined timing is, for example, a predetermined interval (every day, every half day, every hour, etc.). Preferably, the predetermined timing is one in which the change in the number of leaves can be confirmed (not too short, not too long). Therefore, when setting the predetermined timing, it is preferable to set an appropriate timing based on data showing the change in the number of leaves over time obtained in the past. If the determination in step S14 is negative, the process returns to step S10, and the image acquisition unit 30 repeats the process of storing the image in the image DB 50 each time an image is transmitted. On the other hand, if the determination in step S14 is positive, the process moves to step S16.

[0030] When the process moves to step S16, the video generation unit 32 generates a video from multiple images taken of the same area. For example, it generates a video by stitching together multiple images taken by each camera during a certain period of time immediately preceding the generation of the video.

[0031] Next, in step S18, the motion information extraction unit 34 selects one unselected area from the entire field area (the area captured by each camera 70).

[0032] Next, in step S20, the motion information extraction unit 34 extracts motion information from the video of the selected region. In this case, as described above, the motion information extraction unit 34 determines the optical flow in the video of the selected region and extracts motion information within each region (see Figure 6).

[0033] Next, in step S22, the range identification unit 36 ​​identifies the range corresponding to the crops based on the distribution of movement information. In the example in Figure 6, based on the distribution of movement information, it identifies ranges 1 to 4 as shown by the thick outline in Figure 7. The range identification unit 36 ​​also assigns crop IDs 1 to 4 to the crops present in each of the ranges 1 to 4.

[0034] Next, in step S24, the cluster generation unit 38 selects one unselected range from the ranges identified by the range identification unit 36. Here, as an example, the cluster generation unit 38 selects range 1.

[0035] Next, in step S26, the cluster generation unit 38 performs clustering processing on the motion information within the selected range (range 1) and generates clusters. As a result, if, for example, three clusters are generated as shown in Figure 9, the cluster generation unit 38 transmits cluster number = 3 to the new leaf number estimation unit 40.

[0036] Next, in step S28, the new leaf number estimation unit 40 estimates the number of clusters as the number of new leaves. That is, when the new leaf number estimation unit 40 receives information from the cluster generation unit 38 that the number of clusters in range 1 (crop ID=1) is 3, it estimates that the crop with crop ID=1 has 3 new leaves. The new leaf number estimation unit 40 stores the estimation result in the crop table 52 in Figure 5.

[0037] Next, in step S30, the leaf count estimation unit 42 estimates the number of unfolded leaves. For example, if the judgment in step S14 is affirmed every day, the unit compares the number of new leaves of crop body ID=1 on the previous day with the number of new leaves today, and if the number of new leaves has increased by m, it estimates that m new leaves have unfolded today. The leaf count estimation unit 42 stores the estimation result in the crop body table 52 in Figure 5. If the judgment in step S14 is affirmed every half day, the unit compares the number of new leaves of crop body ID=1 half a day ago with the number of new leaves today, and if the number of new leaves has increased by m, it estimates that m new leaves have unfolded during the half day. In this case, the crop body table 52 in Figure 5 stores the number of unfolded leaves for each half day. Furthermore, if the judgment in step S14 is affirmed every hour, the number of new leaves of crop ID=1 one hour ago is compared with the current number of new leaves, and if the number of new leaves has increased by m, it is estimated that m new leaves have unfolded in the past hour. In this case, the crop table 52 in Figure 5 stores the number of unfolded leaves each hour.

[0038] Next, in step S32, the cluster generation unit 38 determines whether all ranges have been selected. If even one of the four ranges shown in Figure 7 has not been selected, the determination in step S32 is rejected, and the process returns to step S24.

[0039] Subsequently, in step S24, ranges 2, 3, and 4 are selected sequentially, and the processes in steps S26 to S30 are repeatedly executed. This estimates the number of new leaves and unfolded leaves for crops 1 to 4, and stores them in the crop table 52. If the judgment in step S32 is affirmed, the process proceeds to step S34.

[0040] When the process moves to step S34, the motion information extraction unit 34 determines whether all areas have been selected. That is, it determines whether all areas (all cameras 70) in the field were selected in step S18. If the determination in step S34 is negative, the process returns to step S18, and the processing from step S18 onward is executed for the next area. In this case, crop IDs 5, 6, ... are assigned to the ranges included in the next area, and the number of new leaves and unfolded leaves of the crops with crop IDs 5, 6, ... is estimated.

[0041] Subsequently, if the judgment in step S34 is affirmed, it means that the estimation of the number of new leaves and the number of unfolded leaves for all crops in the field has been completed, so the process returns to step S10. From this point onward, the processes from step S10 onward described above are repeatedly executed.

[0042] As described in detail above, according to this embodiment, the image acquisition unit 30 acquires multiple images taken at different times in an area where multiple crops exist (S10), the motion information extraction unit 34 calculates the optical flow from the video created by stitching together the multiple images and extracts motion information within the area (S20). The range identification unit 36 ​​identifies the range corresponding to each crop within the area based on the distribution of the extracted motion information (S22), and the cluster generation unit 38 generates clusters for each range by clustering the motion information based on the position and orientation of each piece of motion information (S26). The new leaf number estimation unit 40 then estimates the number of clusters for each range as the number of new leaves for each crop (S28). Thus, in this embodiment, the range where each new leaf with different movements exists is identified by clustering the motion information, and the number of clusters is estimated as the number of new leaves, so the number of new leaves can be automatically estimated. By automatically estimating the number of new leaves in this way, the worker does not have to visually check the number of new leaves, thus reducing the workload on the worker.

[0043] Furthermore, in this embodiment, the leaf number estimation unit 42 estimates the number of unfolded leaves in each crop based on the change in the number of new leaves in each crop, so the number of unfolded leaves can be estimated automatically and accurately. Because the number of unfolded leaves can be estimated automatically and accurately, the operator can appropriately grasp the growth of the crop.

[0044] Furthermore, in this embodiment, the cluster generation unit 38 excludes motion information with a magnitude less than a predetermined value and performs clustering processing on motion information with a magnitude greater than or equal to the predetermined value. This allows for accurate clustering of motion information by removing noise, making it possible to accurately estimate the number of new leaves and the number of unfolded leaves.

[0045] In the above embodiment, the camera 70 was positioned above the crop, and the number of new leaves and unfolded leaves was estimated using images taken from above the crop using the camera 70. However, the invention is not limited to this. In other words, the shooting direction of the camera 70 is arbitrary as long as new leaves can be photographed. For example, the camera 70 may photograph the crop from the side, or from diagonally above. Alternatively, the camera 70 may photograph the crop from below.

[0046] In the above embodiment, a case in which multiple cameras 70 are installed (fixed) within the field has been described, but the invention is not limited to this. The cameras 70 may be moved within the field.

[0047] In the above embodiment, the case in which the camera 70 captures an image (still image) has been described, but the camera 70 is not limited to this and may also capture a video. In this case, the motion information extraction unit 34 may use the captured video as is and extract motion information from the video. Alternatively, the image acquisition unit 30 may acquire multiple frames from the video, the video generation unit 32 may combine the multiple frames to generate a video, and the motion information extraction unit 34 may extract motion information from the generated video.

[0048] In the above embodiment, the case where the crop photographed by the camera 70 is a tomato plant was described, but it is not limited to this, and for example, it may be a crop of the Solanaceae family other than a tomato. Furthermore, it is not limited to the Solanaceae family, but may be other crops or trees (for example, spinach, lettuce, tea, etc.) whose new leaves move in different ways.

[0049] In the above embodiment, the processing shown in Figure 10 was described in the case where the processing device 10 performs the processing shown in Figure 10. However, the invention is not limited to this, and an information processing device (such as a server) connected to the processing device 10 via a network may also perform the processing shown in Figure 10. Furthermore, multiple information processing devices may share the processing shown in Figure 10.

[0050] In the above embodiment, the case in which the camera 70 and the processing unit 10 are connected to a network has been described, but the camera 70 and the processing unit 10 are not limited to this case. In this case, the processing unit 10 reads the image captured by the camera 70 and stored in the portable storage medium 91 (for example, an SD card) from the portable storage medium drive 99 and performs the same processing as in the above embodiment (processing in Figure 10).

[0051] The above processing functions can be implemented by a computer. In this case, a program describing the processing content of the functions that the processing unit should have is provided. By executing this program on a computer, the above processing functions are implemented on the computer. The program describing the processing content can be recorded on a storage medium that can be read by a computer (except for carrier waves).

[0052] When distributing a program, it may be sold in the form of a portable storage medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. Alternatively, the program can be stored in the storage device of a server computer and transferred from the server computer to other computers via a network.

[0053] A computer executing a program stores programs, for example, those recorded on a portable storage medium or transferred from a server computer, in its own memory. The computer then reads the program from its memory and executes the processing according to the program. Alternatively, the computer can directly read the program from the portable storage medium and execute the processing according to that program. Furthermore, the computer can sequentially execute the processing according to the programs received as they are transferred from the server computer.

[0054] The embodiments described above are preferred examples of the present invention. However, the invention is not limited thereto, and various modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0055] 10 Processing unit (state estimation device) 30 Image acquisition unit (acquisition unit) 32 Video Generation Section 34 Motion Information Extraction Unit (Extraction Unit) 36. Range Identification Unit (Identification Unit) 38. Cluster Generation Unit (Generation Unit) 40. New leaf number estimation unit (estimation unit) 42. Number of unfolded leaves estimation unit 70. Camera (imaging device) 100 Agricultural Systems

Claims

1. By acquiring multiple images taken at different times in an area where multiple crops are present, Optical flow is determined from the aforementioned multiple images, and motion information within the region is extracted. Based on the distribution of the extracted movement information, the range corresponding to each of the crop bodies within the region is identified. For each of the identified ranges, the motion information is clustered based on the position and orientation of each piece of motion information to generate clusters. The number of clusters in each of the aforementioned ranges is estimated to be the number of new leaves in each of the crop plants. A method for estimating the state of a crop, characterized in that the processing is performed by a computer.

2. The method for estimating the state of a crop according to claim 1, characterized in that the computer performs a process of calculating the number of unfolded leaves in each crop based on the change in the number of new leaves in each crop.

3. The method for estimating the state of a crop according to claim 1, characterized in that, in the process of generating the clusters, motion information from the motion information whose magnitude of motion is greater than or equal to a predetermined value is subjected to clustering processing.

4. By acquiring multiple images taken at different times in an area where multiple crops are present, Optical flow is determined from the aforementioned multiple images, and motion information within the region is extracted. Based on the distribution of the extracted movement information, the range corresponding to each of the crop bodies within the region is identified. For each of the identified ranges, the motion information is clustered based on the position and orientation of each piece of motion information to generate clusters. The number of clusters in each of the aforementioned ranges is estimated to be the number of new leaves in each of the crop plants. A crop condition estimation program characterized by having a computer perform the processing.

5. An acquisition unit that acquires multiple images taken at different times in an area where multiple crops are present, An extraction unit that determines optical flow from the aforementioned multiple images and extracts motion information within the aforementioned region, A special unit that identifies the range corresponding to each of the crop bodies within the region based on the distribution of the extracted movement information, For each of the specified ranges, a generation unit generates clusters by clustering the motion information based on the position and orientation of each of the motion pieces, An estimation unit that estimates the number of clusters in each of the aforementioned ranges as the number of new leaves in each of the crop bodies, A crop condition estimation device equipped with the following features.

Images