State estimation method, state estimation program, and state estimation device for agricultural product body
A system using cameras and image processing automatically estimates the number of new leaves in crops by analyzing motion information, enhancing monitoring efficiency and accuracy.
Patent Information
- Application Number
- JP2023078772
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2043-05-11
AI Technical Summary
Existing methods for determining the number of new leaves in crops, such as tomatoes, are time-consuming and require manual visual inspection.
A method that utilizes a system of cameras and a processing device to acquire images, extract optical flow and motion information, cluster this information based on position and orientation, and automatically estimate the number of new leaves by calculating the number of clusters in the motion information.
Automatically estimates the number of new leaves, reducing the manual effort required and improving the accuracy and efficiency of crop growth monitoring.
Smart Images

Figure 2024162843000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a crop state estimation method, a state estimation program, and a state estimation device. [Background technology]
[0002] One method for understanding the growth status of agricultural crops (e.g. tomatoes) is to count the number of expanded leaves. When counting the number of expanded leaves, it is common to count the number of new leaves, but counting the number of new leaves is done visually, which is time-consuming.
[0003] A technology is known in the related art that divides an image into multiple regions using division criteria based on growth characteristics, calculates statistics in each region using optical flow, and determines the growth condition based on the calculated statistics (see, for example, Patent Document 1, etc.). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] International Publication No. 2016 / 017150 Summary of the Invention [Problem to be solved by the invention]
[0005] However, even if the above-mentioned Patent Document 1 is used, it is not possible to automatically investigate the number of new leaves or the number of expanded leaves.
[0006] An object of the present invention is to provide a crop body state estimation method, a state estimation program, and a state estimation device that are capable of automatically estimating the number of new leaves that a crop body has. [Means for solving the problem]
[0007] The crop body condition estimation method of the present invention is a method in which a computer executes the following processes: acquiring multiple images taken at different times of an area where multiple crop bodies are present, calculating optical flow from the multiple images, extracting movement information within the area, identifying ranges within the area corresponding to each of the crop bodies based on the distribution of the extracted movement information, clustering the movement information for each of the identified ranges based on the position and orientation of each piece of movement information to generate clusters, and estimating the number of clusters for each range as the number of new leaves in each of the crop bodies. Effect of the Invention
[0008] The crop body state estimating method, state estimation program, and state estimation device of the present invention have the effect of automatically estimating the number of new leaves in a crop body. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an agricultural system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an image captured by a camera. [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of the processing device in FIG. [Figure 4] FIG. 2 is a functional block diagram of the processing device of FIG. [Diagram 5] FIG. 13 is a diagram showing an example of a table structure of a crop body table. [Figure 6] 2 is a diagram showing a schematic diagram of motion information extracted from a moving image of an area X by a motion information extraction unit. FIG. [Figure 7] 11 is a diagram for explaining the process of a range specifying unit; FIG. [Figure 8] 8(a) and 8(b) are diagrams for explaining the processing of the cluster generating unit. [Figure 9] FIG. 9(a) is a diagram for explaining the processing of the new leaf number estimating unit, and FIG. 9(b) is a graph for explaining the processing of the expanded leaf number estimating unit. [Figure 10]13 is a flowchart showing the processing of the processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] An embodiment of an agricultural system will be described in detail below with reference to Fig. 1 to Fig. 10. Fig. 1 shows a schematic configuration of an agricultural system 100 according to an embodiment. The agricultural system 100 of this embodiment is a system that can be used by a producer (hereinafter referred to as a worker) who grows tomatoes, as an example.
[0011] 1, the agricultural system 100 includes a processing device 10 as a state estimation device and a plurality of cameras 70 as imaging devices. The processing device 10 and each camera 70 are connected to a network such as the Internet or a LAN (Local Area Network), and the processing device 10 is capable of acquiring images captured by each camera 70.
[0012] The multiple cameras 70 are installed (fixed) at a predetermined interval on a beam-like member 74 laid above tomato stalks (crops) grown in a farm field (greenhouse, etc.). The angle of view (captureable range) of each of the multiple cameras 70 is fixed, and all of the cameras 70 in the farm field can capture images of all of the crops grown in the farm field.
[0013] Each camera 70 takes an image at a predetermined time interval and transmits the taken image to the processing device 10. For example, an image that one camera 70 can take is an image as shown in FIG.
[0014] The processing device 10 is an information processing device such as a PC or a smartphone that can be used by an operator, and acquires images taken at a predetermined time interval by a camera 70 installed in the field, executes information processing using the acquired images, and estimates the number of new leaves and the number of expanded leaves of the crop plant cultivated in the field. The processing device 10 also provides the estimated results to the operator by displaying them.
[0015] FIG. 3 shows a schematic diagram of the hardware configuration of the processing device 10. As shown in FIG. 3, the processing device 10 includes a central processing unit (CPU) 90, a read only memory (ROM) 92, a random access memory (RAM) 94, a storage unit (e.g., a hard disk drive (HDD) or a solid state drive (SSD)) 96, a network interface 97, a display unit 93, an input unit 95, and a portable storage medium drive 99. The display unit 93 includes a liquid crystal display or the like, and the input unit 95 includes a keyboard, a mouse, a touch panel, and the like. These components of the processing device 10 are connected to a bus 98. In the processing device 10, the CPU 90 executes a program (including a crop condition estimation program) stored in the ROM 92 or the HDD 96, or a program read by the portable storage medium drive 99 from the portable storage medium 91, thereby realizing the functions of the components shown in FIG. 4. The functions of the various units in FIG. 4 may be realized by an integrated circuit such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA).
[0016] Fig. 4 shows a functional block diagram of the processing device 10. In the processing device 10, as shown in Fig. 4, the CPU 90 executes a program to realize functions of an image acquisition unit 30 as an acquisition unit, a video generation unit 32, a motion information extraction unit 34 as an extraction unit, a range identification unit 36 as an identification unit, a cluster generation unit 38 as a generation unit, a new leaf number estimation unit 40 as an estimation unit, and an expanded leaf number estimation unit 42 as an estimation unit. Fig. 4 also shows an image DB 50 and a crop body table 52 stored in the storage unit 96, etc. The crop body table 52 has a table structure as shown in Fig. 5, for example, and will be described in detail later.
[0017] The image acquisition unit 30 acquires images (image data) transmitted from each camera 70, and stores the image data in the image DB 50 in association with the identification information (camera ID) of each camera.
[0018] The video generating unit 32 generates a video by connecting a predetermined number of images (images taken within a predetermined time) taken by the same camera 70. The video generating unit 32 transmits the generated video to the movement information extracting unit 34. Note that the video generated from the images taken by each camera can be said to be a video of different areas in the field.
[0019] The motion information extraction unit 34 obtains an optical flow from each of the moving images (moving images of different regions) generated by the moving image generation unit 32, and extracts motion information within each region. The motion information extraction unit 34 transmits the extracted motion information to the range identification unit 36 and the cluster generation unit 38. Here, as a method for obtaining the optical flow, for example, the method described in 'BD Lucas and T. Kanade (1981), An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, pages 121-130' can be adopted.
[0020] Fig. 6 is a diagram showing a schematic of the motion information extracted by the motion information extraction unit 34 from the moving image of the area X. As can be seen from the enlarged view of the dashed circle in Fig. 6, in this embodiment, the motion information is represented by an arrow (vector), with the direction of the arrow indicating the direction of the motion and the length of the arrow indicating the magnitude of the motion. Note that the motion information may be represented by something other than an arrow. For example, the motion information may be displayed in different colors, with the type of color indicating the direction of the motion and the brightness indicating the magnitude of the motion.
[0021] Here, the inventors have focused on the fact that new leaves (new leaves) in tomato stalks have three-dimensional and extremely complicated movements such as turning movements, and that leaves other than the new leaves hardly move. The inventors have also focused on the fact that each new leaf moves in the same way, but each new leaf moves differently, so that an area of the same movement indicates one new leaf, and have decided to estimate the number of new leaves and the number of unfolded leaves using the movement information. In a tomato stalk (crop body), a single new leaf is formed by gathering multiple leaflets (for example, seven leaflets), but in FIG. 6, for the sake of convenience of explanation and illustration, three leaflets are assumed to form one new leaf.
[0022] Returning to Fig. 4, the range identifying unit 36 identifies a range corresponding to each of the multiple crop bodies present in each region, based on the motion information in each region (in each video) extracted by the motion information extracting unit 34. The range identifying unit 36 transmits information on the identified ranges to the cluster generating unit 38. For example, in the case of the video of region X in Fig. 6, the range identifying unit 36 identifies ranges 1 to 4 based on the distribution of motion information, as shown by the thick frame in Fig. 7. Then, the range identifying unit 36 considers that different crop bodies exist in these ranges 1 to 4, and assigns identification information (crop body IDs) 1 to 4 to the crop bodies present in each range.
[0023] The cluster generating unit 38 performs a clustering process on the movement information based on the position and orientation of each piece of movement information for each range (each crop body) specified by the range specifying unit 36, to generate a cluster.
[0024] Specifically, the cluster generating unit 38 removes, for example, motion information (minor motion information) whose magnitude is less than a predetermined value from among the motion information present within range 1 in FIG. 7. The cluster generating unit 38 also obtains the XY coordinates of the start point of each piece of motion information and the direction Z (Angle) of each piece of motion information in the XY coordinate system shown in FIG. 8(a), and plots them in an XYZ coordinate system as shown in FIG. 8(b). The cluster generating unit 38 then performs clustering processing on each point on the XYZ coordinate system in FIG. 8(b), and generates clusters as shown by the dashed circles. For example, the k-means method can be used as the clustering processing. The cluster generating unit 38 also specifies the number of generated clusters, and transmits the number of new leaf estimation units 40. For example, the elbow method can be used as a method for specifying the number of clusters.
[0025] The new leaf number estimation unit 40 estimates the number of clusters in each range generated by the cluster generation unit 38 as the number of new leaves possessed by the crop body corresponding to each range. The new leaf number estimation unit 40 stores the estimated number of new leaves in the crop body table 52. Here, the crop body table 52 is a table that stores the estimated result of the number of new leaves and the estimated result of the number of expanded leaves for each number of days elapsed since the start of cultivation for each crop body ID, as shown in FIG. 5. For example, when the number of clusters on the nth day in the range 1 is 3, the new leaf number estimation unit 40 estimates that the number of new leaves on the nth day of the crop body with the crop body ID=1 is 3 as shown in FIG. 9(a), and stores this in the crop body table 52.
[0026] Returning to Fig. 4, the expanded leaf number estimation unit 42 refers to the change in the number of new leaves stored in the crop body table 52, and estimates when and how many leaves have expanded. For example, the change in the number of new leaves of the crop body with crop body ID = 1 in Fig. 5 can be represented in a graph as shown in Fig. 9(b). In this case, since the number of new leaves increases by 1 between the n+3th day and the n+4th day, the expanded leaf number estimation unit 42 stores information that the number of expanded leaves on the n+4th day is +1 in the crop body table 52 in Fig. 5.
[0027] (Regarding the processing of the processing device 10) Next, the processing of the processing device 10 will be described with reference to the flowchart of FIG.
[0028] 10 is started, first, in step S10, the image acquiring unit 30 determines whether an image has been transmitted. If the determination in step S10 is positive, the process proceeds to step S12, where the image acquiring unit 30 associates the image (image data) with the camera ID and stores it in the image DB 50. Then, the process proceeds to step S14. However, if the determination in step S10 is negative, the process proceeds to step S14 without passing through step S12.
[0029] When the process proceeds to step S14, the video generating unit 32 judges whether or not a predetermined timing has arrived. The predetermined timing is, for example, a timing of every predetermined time (every day, every half day, every hour, etc.). The predetermined timing is preferably a timing at which a change in the number of leaves can be confirmed (a timing that is neither too short nor too long). Therefore, when setting the predetermined timing, it is preferable to set an appropriate timing based on data indicating a time change in the number of leaves obtained in the past. If the judgment in step S14 is negative, the process returns to step S10, and the image acquiring unit 30 repeats the process of storing the image in the image DB 50 every time an image is transmitted. On the other hand, if the judgment in step S14 is positive, the process proceeds to step S16.
[0030] In step S16, the video generator 32 generates a video from a plurality of images taken of the same area. For example, the video is generated by stitching together a plurality of images taken by each camera within a certain period of time immediately before.
[0031] Next, in step S18, the movement information extraction unit 34 selects one unselected area from among all areas in the farm field (areas photographed by each camera 70).
[0032] Next, in step S20, the motion information extraction unit 34 extracts motion information from the moving image of the selected region. In this case, the motion information extraction unit 34 obtains an optical flow in the moving image of the selected region as described above, and extracts motion information within each region (see FIG. 6).
[0033] Next, in step S22, the range identifying unit 36 identifies a range corresponding to the crop body from the distribution of the movement information. In the example of Fig. 6, ranges 1 to 4 as shown by thick frames in Fig. 7 are identified based on the distribution of the movement information. Furthermore, the range identifying unit 36 assigns crop body IDs = 1 to 4 to the crop bodies present in the ranges 1 to 4, respectively.
[0034] Next, in step S24, the cluster generating unit 38 selects one unselected range from among the ranges identified by the range identifying unit 36. Here, it is assumed that the cluster generating unit 38 selects range 1, as an example.
[0035] Next, in step S26, the cluster generating unit 38 performs a clustering process on the motion information within the selected range (range 1) to generate clusters. As a result, when three clusters are generated as shown in FIG. 9, the cluster generating unit 38 transmits the cluster number=3 to the new leaf number estimating unit 40.
[0036] Next, in step S28, the new leaf number estimation unit 40 estimates the number of clusters as the number of new leaves. That is, when the new leaf number estimation unit 40 receives information from the cluster generation unit 38 that the number of clusters in range 1 (crop body ID=1) is 3, the new leaf number estimation unit 40 estimates that the crop body with crop body ID=1 has three new leaves. The new leaf number estimation unit 40 stores the estimation result in the crop body table 52 in FIG. 5.
[0037] Next, in step S30, the expanded leaf number estimation unit 42 estimates the number of expanded leaves. For example, if the judgment in step S14 is positive for each day, the number of new leaves of the crop body with the crop body ID=1 on the previous day is compared with the number of new leaves today, and if the number of new leaves has increased by m, it is estimated that m new leaves have expanded today. The expanded leaf number estimation unit 42 stores the estimation result in the crop body table 52 of FIG. 5. Note that, if the judgment in step S14 is positive for each half day, the number of new leaves of the crop body with the crop body ID=1 half a day ago is compared with the current number of new leaves, and if the number of new leaves has increased by m, it is estimated that m new leaves have expanded during the half day. In this case, the number of expanded leaves for each half day is stored in the crop body table 52 of FIG. 5. Moreover, if the determination in step S14 is positive every hour, the number of new leaves of the crop body with the crop body ID=1 one hour ago is compared with the current number of new leaves, and if the number of new leaves has increased by m, it is estimated that m new leaves have unfolded during the hour. In this case, the number of unfolded leaves for each hour is stored in the crop body table 52 in FIG. 5.
[0038] Next, in step S32, the cluster generating unit 38 judges whether or not all ranges have been selected. If there is even one unselected range among the four ranges shown in Fig. 7, the judgment in step S32 is negative, and the process returns to step S24.
[0039] Thereafter, ranges 2, 3, and 4 are selected in sequence in step S24, and the processes of steps S26 to S30 are repeatedly executed. As a result, the numbers of new leaves and the numbers of expanded leaves of crop bodies 1 to 4 are estimated and stored in crop body table 52. Then, if the determination in step S32 is positive, the process proceeds to step S34.
[0040] When the process proceeds to step S34, the movement information extraction unit 34 judges whether all the regions have been selected. That is, it judges whether all the regions in the field (all the cameras 70) have been selected in step S18. If the judgment in step S34 is negative, the process returns to step S18, and the processes in and after step S18 are executed for the next region. In this case, crop body IDs = 5, 6, ... are assigned to the ranges included in the next region, and the number of new leaves and the number of expanded leaves of the crop bodies with crop body IDs = 5, 6, ... are estimated.
[0041] Thereafter, if the determination in step S34 is positive, this means that estimation of the number of new leaves and the number of expanded leaves for all crop bodies in the field has been completed, and the process returns to step S10. Then, from this point on, the above-mentioned processes from step S10 onwards are repeatedly executed.
[0042] As described above in detail, according to this embodiment, the image acquisition unit 30 acquires a plurality of images taken at different times in an area where a plurality of crop bodies exist (S10), and the motion information extraction unit 34 obtains an optical flow from a video in which a plurality of images are joined together, and extracts motion information within the area (S20). In addition, the range identification unit 36 identifies a range corresponding to each of the crop bodies within the area based on the distribution of the extracted motion information (S22), and the cluster generation unit 38 performs clustering processing on the motion information based on the position and orientation of each of the motion information for each range to generate clusters (S26). Then, the new leaf number estimation unit 40 estimates the number of clusters for each range as the number of new leaves in each of the crop bodies (S28). As a result, in this embodiment, the range in which each of the new leaves that move differently is present is identified by clustering processing on the motion information, and the number of clusters is estimated as the number of new leaves, so that the number of new leaves can be automatically estimated. By automatically estimating the number of new leaves in this way, the operator does not need to visually check the number of new leaves, and the operator's workload can be reduced.
[0043] In the present embodiment, the expanded leaf number estimating unit 42 estimates the number of expanded leaves in each crop body based on the change in the number of new leaves in each crop body, so that the number of expanded leaves can be estimated automatically and accurately. Since the number of expanded leaves can be estimated automatically and accurately, the worker can appropriately grasp the growth of the crop body.
[0044] In this embodiment, the cluster generation unit 38 excludes motion information whose magnitude of motion is less than a predetermined value from the motion information, and performs clustering processing on motion information whose magnitude of motion is equal to or greater than a predetermined value. This allows noise to be removed, and the motion information to be subjected to accurate clustering processing, thereby enabling accurate estimation of the number of new leaves and the number of expanded leaves.
[0045] In the above embodiment, the camera 70 is provided above the crop body, and the number of new leaves and the number of developed leaves are estimated using an image captured by the camera 70 from above the crop body, but the present invention is not limited to this. In other words, as long as the new leaves can be captured in an arbitrary direction by the camera 70. For example, the camera 70 may capture the image of the crop body from the side or from diagonally above. The camera 70 may also capture the image of the crop body from below.
[0046] In the above embodiment, a case has been described in which a plurality of cameras 70 are provided (fixed) in the field, but the present invention is not limited to this. The cameras 70 may be movable in the field.
[0047] In the above embodiment, the camera 70 captures an image (still image), but the present invention is not limited to this. The camera 70 may capture a moving image. In this case, the motion information extraction unit 34 may use the captured moving image as is to extract motion information from the moving image. Alternatively, the image acquisition unit 30 may acquire multiple frames from the moving image, the moving image generation unit 32 may connect the multiple frames to generate a moving image, and the motion information extraction unit 34 may extract motion information from the generated moving image.
[0048] In the above embodiment, the case where the crop body photographed by the camera 70 is a tomato stalk has been described, but the present invention is not limited to this, and may be, for example, a crop body of the Solanaceae family other than a tomato. Also, the present invention is not limited to the Solanaceae family, and may be, for example, a crop body of other agricultural crops or trees (for example, spinach, lettuce, tea, etc.) whose new leaves move in different directions.
[0049] In the above embodiment, the processing in Fig. 10 is performed by the processing device 10, but the present invention is not limited to this, and an information processing device (such as a server) connected to the processing device 10 through a network may perform the processing in Fig. 10. Also, a plurality of information processing devices may share the processing in Fig. 10.
[0050] In the above embodiment, the camera 70 and the processing device 10 are connected to a network, but the present invention is not limited to this, and the camera 70 and the processing device 10 do not have to be connected to a network. In this case, the processing device 10 reads an image captured by the camera 70 and stored in a portable storage medium 91 (e.g., an SD card, etc.) from the portable storage medium drive 99, and executes the same process as in the above embodiment (the process in FIG. 10).
[0051] The above processing functions can be realized by a computer. In that case, a program is provided that describes the processing contents of the functions that the processing device should have. The above processing functions are realized on the computer by executing the program on the computer. The program that describes the processing contents can be recorded on a computer-readable storage medium (excluding carrier waves).
[0052] When a program is distributed, it is sold in the form of a portable storage medium on which the program is recorded, such as a DVD (Digital Versatile Disc), a CD-ROM (Compact Disc Read Only Memory), etc. The program can also be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.
[0053] A computer that executes a program stores, for example, a program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing in accordance with the program. The computer can also read the program directly from a portable storage medium and execute processing in accordance with that program. The computer can also execute processing in accordance with the received program each time a program is transferred from the server computer.
[0054] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this embodiment and can be modified in various ways without departing from the scope of the present invention. [Explanation of symbols]
[0055] 10 Processing device (state estimation device) 30 Image acquisition unit (acquisition unit) 32 Video Generation Unit 34 Motion information extraction unit (extraction unit) 36 Range specification part (specific part) 38 Cluster Generation Unit (Generation Unit) 40 New leaf number estimation section (estimation section) 42 Estimation of number of expanded leaves 70 Camera (photographic equipment) 100 Agriculture System
Claims
1. Acquire multiple images taken at different times of an area where multiple crops are present, Obtaining an optical flow from the plurality of images and extracting motion information within the region; identifying a range within the region corresponding to each of the crop bodies based on a distribution of the extracted movement information; generating clusters for each of the identified ranges by performing a clustering process on the motion information based on the position and orientation of each of the motion information; The number of clusters for each range is estimated as the number of new leaves in each of the crop plants; A method for estimating a state of a crop body, the processing of which is executed by a computer.
2. The method for estimating a state of a crop body according to claim 1 , characterized in that the computer executes a process of calculating a number of expanded leaves in each of the crop bodies based on a change in the number of new leaves in each of the crop bodies.
3. 2. The method for estimating a state of a crop body according to claim 1, wherein, in the process of generating the clusters, the motion information having a magnitude of motion equal to or greater than a predetermined value is subjected to clustering processing.
4. Acquire multiple images taken at different times of an area where multiple crops are present, Obtaining an optical flow from the plurality of images and extracting motion information within the region; identifying a range within the region corresponding to each of the crop bodies based on a distribution of the extracted movement information; generating clusters for each of the identified ranges by performing a clustering process on the motion information based on the position and orientation of each of the motion information; The number of clusters for each range is estimated as the number of new leaves in each of the crop plants; A crop state estimation program that causes a computer to execute a process.
5. An acquisition unit that acquires a plurality of images captured at different times of an area where a plurality of crop bodies are present; an extraction unit that calculates an optical flow from the plurality of images and extracts motion information within the region; an identification unit that identifies a range corresponding to each of the crop bodies within the region based on a distribution of the extracted movement information; a generation unit that performs a clustering process on the motion information based on a position and an orientation of each of the motion information for each of the specified ranges to generate a cluster; An estimation unit that estimates the number of clusters for each range as the number of new leaves in each of the crop bodies; A crop state estimation device comprising:
Citation Information
Patent Citations
Leaf area calculation system of strawberry plant
JP2023034558A
Plant Detection and Display System
JP2023553443A
Method for monitoring growth of plants and generating a plant grow schedule
US20220007589A1
Method and apparatus of evaluating fitness-for-plucking of tea leaf, system of evaluating fitness-for-plucking of tea leaf, and computer-usable medium
WO2009116613A1
Growth state estimation device, plant growth observation system, growth state estimation method, growth index estimation method, and recording medium
WO2016017150A1
