Growth information estimation device, learning device, growth information estimation system, growth information estimation program, and growth information estimation method for estimating plant growth state

Abstract

To accurately estimate the growth state of a plant from an image of the ground surface on which the plant is growing.SOLUTION: An information processing device 10 comprises a memory 12 and a processor 11. The memory 12 stores an object detection model Md, the model being trained by associating images capturing a predetermined plant growing on a ground surface Fov with growth information. The processor 11 uses the object detection model Md to output growth information Gri of the predetermined plant growing on the ground surface Fov included in an image Fim captured by a flight imaging unit 20.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an apparatus, a program, a system, and a method for inferring the growth condition of a plant from an image of the ground surface on which the plant grows. [Background technology]

[0002] Conventionally, a configuration for inferring the growth state of a plant from an image of the ground surface on which the plant grows is known. For example, JP 2022-128401 A (Patent Document 1) discloses an information processing device having a first mode and a second mode for analyzing an aerial image of a field in which a plurality of plants of a predetermined plant are planted in ridges to identify plants that are growing poorly. In the first mode, a poorly grown area is identified in the aerial image, which is an area representing a poorly grown plant, according to the area of ​​each of the plurality of plants in the aerial image. In the second mode, a poorly grown area is identified in the aerial image according to the width of the area of ​​the plant in the width direction of the ridge. According to the information processing device, even if it is difficult to distinguish each plant in the aerial image of the field, it is possible to identify a poorly grown area of ​​the plant. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-128401 Summary of the Invention [Problem to be solved by the invention]

[0004] In the first mode of Patent Document 1, for example, a plant area where the ratio of overlapping plants is less than a threshold, a plant area where the plant area is sorted in descending order based on area and is equal to or less than a predetermined threshold, or a plant area where the area is smaller than a predetermined threshold is identified as a poor growth area. In the second mode, a plant area where the plant area is sorted in descending order based on width in the width direction of the ridge and is equal to or less than a predetermined threshold, or a plant area where the width of the plant area is smaller than a predetermined threshold is identified as a poor growth area.

[0005] In the rule-based determination method as in Patent Document 1, the design of the threshold and feature amount tends to be complicated, and therefore the threshold cannot be flexibly adapted to changes in the tendency or pattern of the image of the field. According to the configuration disclosed in Patent Document 1, the accuracy of determining whether or not an area included in the aerial image is a poor growth area can be significantly reduced due to such a mismatch between the aerial image of the field and the threshold.

[0006] The present disclosure has been made to solve the above-mentioned problems, and has an object to infer with high accuracy the growth state of plants from images of the ground surface on which the plants grow. [Means for solving the problem]

[0007] According to an aspect of the present disclosure, a growth information estimation device includes a storage unit and a processing unit. The storage unit stores an object detection model that is learned by associating an image of a specific plant growing on the ground surface with the growth information. The processing unit uses the object detection model to output the growth information of the specific plant growing on the ground surface included in the image.

[0008] A learning device according to another aspect of the present disclosure includes a storage unit and a processing unit. The storage unit stores an object detection model that detects a predetermined plant in a growth state corresponding to a correct answer label for machine learning from an image of the ground surface. The processing unit performs machine learning on the object detection model by using the learning image with the correct answer label.

[0009] A growth information estimation system according to another aspect of the present disclosure includes an aerial photography unit, a storage unit, and a processing unit. The aerial photography unit captures an image of the ground surface from the sky. The storage unit stores an object detection model that is learned by associating an image of a specific plant growing on the ground surface with the growth information. The processing unit uses the object detection model to output the growth information of the specific plant growing on the ground surface included in the image.

[0010] A growth information estimation program relating to another aspect of the present disclosure, when executed by a processor, causes the processor to acquire an image of the ground's surface, and output growth information of a specified plant growing on the ground surface contained in the image using an object detection model that has been trained to associate an image of a specified plant growing on the ground surface with the growth information.

[0011] A growth information estimation method according to another aspect of the present disclosure includes the steps of acquiring an image of the ground's surface, and outputting growth information of a specified plant growing on the ground surface contained in the image using an object detection model that has been trained to associate an image of a specified plant growing on the ground surface with the growth information. Effect of the Invention

[0012] According to the growth information estimation device, learning device, growth information estimation system, growth information estimation program, and growth information estimation method disclosed herein, the growth condition of a plant can be inferred with high accuracy from an image of the ground surface on which the plant is growing, by using an object detection model that is trained to associate images taken of a specific plant growing on the ground surface with growth information. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration of a growth condition inference system according to an embodiment. [Diagram 2] 2 is a diagram showing a learning image which is an example of an image included in the learning data of FIG. 1. FIG. [Diagram 3] FIG. 2 is a diagram showing an example of an aerial image of FIG. 1. [Figure 4]FIG. 4 is a diagram showing blue-chip stocks detected from the aerial image of FIG. 3 by the object detection model of FIG. 1 . [Diagram 5] This is a diagram showing, as straight lines, each of several rows derived from the positions of several excellent plants shown in Figure 4. [Figure 6] FIG. 6 shows the locations of missing plants identified based on the multiple rows shown in FIG. 5 and the arrangement information in FIG. 1. [Figure 7] 2 is a flowchart showing an example of a flow of processing performed by a processor that executes the growth state inference program of FIG. 1. [Figure 8] FIG. 1 is a block diagram showing a configuration of a growth condition inference system according to a first modified example of an embodiment. [Figure 9] FIG. 11 is a block diagram showing a configuration of an information processing device according to a second modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] FIG. 1 is a block diagram showing a configuration of a growth state inference system 1 according to an embodiment. The growth state inference system 1 estimates the growth state of open-field vegetables planted in a farm field Fov (ground surface) from an aerial image Fim of the farm field Fov. Examples of open-field vegetables include cabbage, radish, Chinese cabbage, lettuce, cucumber, tomato, eggplant, green pepper, pumpkin, carrot, watermelon, and melon. In addition, the plants whose yields are predicted by the growth state inference system 1 may be plants other than open-field vegetables. In the following, a case will be described in which cabbage (a plant species) having rosette leaves is planted in the farm field Fov.

[0015] As shown in Fig. 1, the growth condition inference system 1 includes an information processing device 10 (growth information estimation device) and a flight photography unit 20. The flight photography unit 20 captures an aerial image Fim of a farm field Fov from the sky. The aerial image Fim includes aerial photography information (e.g., the date of capture and coordinates). The flight photography unit 20 includes at least one of a drone 21, an airplane 22, and an artificial satellite 23. The drone 21, the airplane 22, and the artificial satellite 23 have higher captureable altitudes in this order, and therefore the range of the ground surface that can be captured becomes wider in this order.

[0016] The information processing device 10 analyzes the aerial photographed image Fim to calculate the harvest yield (or harvested stalk rate) and outputs the growth information Gri on the growth state of the cabbage in the field Fov. The growth information Gri includes the number of cabbage stalks with good growth state (superior stalks) and the number of cabbage stalks with poor growth state (missing stalks).

[0017] The information processing device 10 includes a processor 11 (processing unit), a memory 12 (storage unit), an input unit 13, an output unit 14, and a bus 15. The processor 11, the memory 12, the input unit 13, and the output unit 14 are communicatively connected to each other via the bus 15. The information processing device 10 includes, for example, a PC (Personal Computer) or a workstation.

[0018] The processor 11 includes, for example, a central processing unit (CPU). The processor 11 may include a graphics processing unit (GPU). The processor 11 executes programs stored in the memory 12 to realize each function of the information processing device 10.

[0019] The memory 12 includes a non-volatile memory (e.g., a hard disk) and a volatile memory (e.g., a RAM (Random Access Memory)). The memory 12 stores an object detection model Md, a learning image Lim, a machine learning program Pg1, a growth state inference program Pg2, and placement information Pmi. Although not shown in FIG. 1, the memory 12 may store an OS (Operating System), data required for executing each program, and the processing results of each program. Note that the memory 12 may store a plurality of object detection models corresponding to a plurality of plant species, respectively, and a user may appropriately select one of the plurality of object detection models.

[0020] The object detection model Md is a machine learning model that is trained by associating an image of a cabbage growing on the ground surface with growth information of the cabbage. The object detection model Md detects a cabbage stalk (a specified plant) in a good growth state corresponding to a correct answer label for machine learning from an aerial image Fim. The object detection model Md includes, for example, YOLO (You Look Only Once), SSD (Single Shot MultiBox Detector), or Faster RCNN (Regional Convolutional Neural Network). The machine learning program Pg1 is a program for performing machine learning (for example, supervised learning) on ​​the object detection model Md using training data Ld. The training data Ld includes multiple image data with correct answer labels.

[0021] The growth state inference program Pg2 is a program for outputting growth information Gri of cabbages growing on the ground surface Fov included in the aerial photographed image Fim using a trained object detection model Md. The arrangement information Pmi includes information on the arrangement of cabbages in the field Fov. The arrangement information Pmi includes at least one of the interval between two adjacent ridges in the aerial photographed image Fim when a plurality of cabbage stalks are planted in a line in each of a plurality of ridges to form one row, and the interval between two adjacent rows in the aerial photographed image Fim when a plurality of cabbage stalks are planted in a line in each of a plurality of ridges to form a plurality of rows. The arrangement information Pmi may include the interval between two adjacent cabbage stalks in each row.

[0022] The input unit 13 receives input from a user (for example, a GUI (Graphical User Interface) operation, a command input, or a voice input). The input unit 13 includes a mouse, a keyboard, a touch panel, and a microphone. The output unit 14 outputs the processing result of the processor 11 to the user. The output unit 14 includes, for example, a touch panel, a display, and a speaker.

[0023] Fig. 2 is a diagram showing a learning image Lim, which is an example of an image included in the learning data of Fig. 1. In Fig. 2, the X-axis, Y-axis, and Z-axis are mutually orthogonal, and the Z-axis direction indicates the direction of gravity. The same is true for Figs. 3 to 6.

[0024] As shown in FIG. 2, the learning image Lim is an image of a part of a cabbage field in which a plurality of ridges are formed along each other. In each of the plurality of ridges, a plurality of cabbage plants are planted in a straight line to form one row. That is, the learning image Lim is an image of one row planting. The number of rows formed in one ridge in the learning image Lim may be two or more. In FIG. 2, a cabbage plant in a good growth state (good plant) is surrounded by a dotted rectangle (correct label). The correct label may be attached manually or automatically. A good plant is a cabbage plant that has grown to a level that can be shipped, or a cabbage plant that is expected to grow to a level that can be shipped at the harvesting season. On the other hand, a missing plant is not surrounded by a rectangle. A missing plant is a cabbage plant that does not meet the quality standard and cannot be shipped, or a cabbage plant that is not expected to grow to a level that can be shipped at the harvesting season. For example, among cabbage plants Cb1, Cb2, and Cb3 that form rows in a furrow Grd, cabbage plants Cb1 and Cb3 are good plants, and cabbage plant Cb2 is a missing plant. An object detection model Md that has been trained by machine learning using a training image Lim can detect good plants from an aerial image Fim with high accuracy.

[0025] FIG. 3 is a diagram showing an example of the aerial image Fim in FIG. 1. As shown in FIG. 3, the aerial image Fim is a part of a cabbage field in which a plurality of ridges are formed along each other. A plurality of cabbage plants planted in each of the plurality of ridges form one row. That is, the aerial image Fim is an image of one row planting, similar to the learning image Lim in FIG. 2. Note that, in order to make the aerial image Fim data that can be input to the object detection model Md, preprocessing (e.g., compression processing and division processing) is performed on the aerial image Fim. Also, even if the size of the cabbage in the image included in the learning data Ld is different from the size of the cabbage in the image input to the trained object detection model Md, by improving the machine learning method, it is possible to detect cabbages of various sizes according to the growth state. Alternatively, it is possible to detect cabbages of various sizes according to the growth state by preparing a plurality of trained object detection models for each size of cabbage.

[0026] Fig. 4 is a diagram showing high-quality stocks detected from the aerial photographed image Fim of Fig. 3 by the object detection model Md of Fig. 1. As shown in Fig. 4, the positions of the high-quality stocks detected by the object detection model Md are marked with circles. The positions of the high-quality stocks are, for example, the center positions of the rectangular regions in the aerial photographed image Fim that are detected as high-quality stocks by the object detection model Md.

[0027] Fig. 5 is a diagram showing, as straight lines Ln, each of a plurality of lines derived from the positions of a plurality of high-quality stocks shown in Fig. 4. As shown in Fig. 5, the plurality of lines included in the aerial photograph image Fim are derived from the positions (points) of a plurality of high-quality stocks by, for example, a Hough transform.

[0028] FIG. 6 is a diagram showing the positions of missing stalks identified based on the multiple rows Ln shown in FIG. 5 and the arrangement information Pmi in FIG. 1. FIG. 6 is a diagram in which circles indicating the positions of missing stalks are added to FIG. 3. The circles indicating good stalks and the circles indicating missing stalks are hatched differently. Referring to FIG. 6, cabbage stalks are planted at equal intervals along each of the multiple rows. The interval between adjacent cabbage stalks in each row is included in the arrangement information Pmi. If no other good stalks are detected at adjacent positions separated by the interval from the position of a good stalk detected by the object detection model Md along the row passing through the good stalk, it means that a cabbage stalk is planted at the adjacent position, but the cabbage stalk at the adjacent position is not detected as a good stalk. Therefore, the adjacent position where no good stalk is detected is detected as the position of a missing stalk. In addition, the ratio of the number of good stalks to the total number of good stalks and the number of missing stalks (the total number of cabbage stalks planted on the ground surface included in the aerial image Fim) is calculated as the harvest yield.

[0029] Fig. 7 is a flow chart showing an example of the flow of processing performed by the processor 11 that executes the growth state inference program Pg2 in Fig. 1. In the following, steps will be simply abbreviated as S, and Figs. 3 to 6 will be referred to as necessary.

[0030] As shown in FIG. 7, in S101, the processor 11 acquires an aerial image Fim (FIG. 3), and proceeds to processing in S102. In S102, the processor 11 performs preprocessing on the aerial image Fim, and proceeds to processing in S103. The processor 11 inputs the aerial image Fim to the object detection model Md, detects high-quality plants from the aerial image Fim (FIG. 4), and proceeds to processing in S103. In S104, the processor 11 performs Hough transformation (FIG. 5) on the detection result in S103 to detect missing plants (FIG. 6), and proceeds to processing in S105. In S105, the processor 11 calculates growth information Gri, and proceeds to processing in S106. In S106, the processor 11 outputs the growth information Gri, and ends the processing. Note that images of cabbage stalks according to growth stages may be learned, and the growth state of the cabbage stalks may be inferred for each growth stage. Furthermore, each process included in the growth state inference program Pg2 may have a separate step or configuration.

[0031] In the embodiment, a configuration having both an inference function and a learning function has been described. The configuration according to the present disclosure may include either one of the inference function and the learning function. In the following, a configuration having an inference function but not a learning function (Modification 1) will be described with reference to FIG. 8, and a configuration having a learning function but not an inference function (Modification 2) will be described with reference to FIG. 9.

[0032] FIG. 8 is a block diagram showing a configuration of a growth state inference system 1A according to a first modified example of the embodiment. The configuration of the growth state inference system 1A is a configuration in which the information processing device 10 in the growth state inference system 1 of FIG. 1 is replaced with an information processing device 10A (growth information estimation device). The configuration of the information processing device 10A is a configuration in which the learning data Ld and the machine learning program Pg1 are removed from the information processing device 10 of FIG. 1, and the object detection model Md is replaced with an object detection model MdA. The object detection model MdA is previously trained by an external learning device and is updated as appropriate. In updating the object detection model MdA, the object detection model MdA is replaced with an object detection model trained by an external learning device. Other configurations are similar to those of the growth state inference system 1, and therefore description of similar configurations will not be repeated.

[0033] Fig. 9 is a block diagram showing a configuration of an information processing device 10B according to a second modified example of the embodiment. The configuration of the information processing device 10B (learning device) is a configuration in which the growth state inference program Pg2 and the arrangement information Pmi are removed from the information processing device 10 in Fig. 1. The configuration other than this is the same as that of the information processing device 10, and therefore the description of the similar configuration will not be repeated. The trained object detection model Md that has been trained by the information processing device 10B may be transmitted to another information processing device (growth information estimation device) via, for example, a cloud or a server.

[0034] As described above, according to the device, system, program, and method according to the embodiment and the first and second modifications, the growth state of a plant can be inferred with high accuracy from an image of the ground surface on which the plant grows.

[0035] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0036] 1,1A Growth condition inference system, 10,10A,10B Information processing device, 11 Processor, 12 Memory, 13 Input unit, 14 Output unit, 15 Bus, 20 Flight photography unit, 21 Drone, 22 Airplane, 23 Artificial satellite, Cb1~Cb3 Cabbage stalk, Fim Image, Fov Field, Grd Furrow, Gri Growth information, Ld Learning data, Lim Learning image, Ln Row, Md,MdA Object detection model, Pg1 Machine learning program, Pg2 Growth condition inference program, Pmi Placement information.

Claims

1. A storage unit that stores an object detection model that is trained by associating images of a specific plant growing on the ground surface with growth information; A growth information estimation device comprising: a processing unit that uses the object detection model to output growth information of the specified plant growing on the ground surface included in the image.

2. The growth information estimation device according to claim 1 , wherein the predetermined plant is a plant in a growth state corresponding to a correct answer label for machine learning.

3. The processing unit further uses information on the location of the predetermined plant on the ground surface included in the image to output the growth information; the predetermined plant is planted in a furrow formed on the ground surface included in the image; the arrangement information includes at least one of a distance between two adjacent ridges and a distance between two adjacent rows in the image; The growth information estimation device according to claim 2 , wherein the processing unit derives at least one row formed by a plurality of specified plants from an arrangement of the plurality of specified plants corresponding to the correct label detected from the image.

4. The growth information estimation device according to claim 2 , wherein the processing unit performs the machine learning on the object detection model by using the learning images to which the correct answer labels are attached.

5. A storage unit that stores an object detection model that detects a predetermined plant in a growth state corresponding to a correct answer label for machine learning from an image of the ground surface; A processing unit that performs the machine learning on the object detection model using the learning images to which the correct answer labels are attached.

6. a flying photography unit that takes images of the ground surface from the sky; A storage unit that stores an object detection model that is trained by associating images of a specific plant growing on the ground surface with growth information; A growth information estimation system comprising: a processing unit that uses the object detection model to output growth information of the specified plant growing on the ground surface included in the image.

7. When executed by a processor, the processor Acquire images of the Earth's surface, A growth information estimation program that uses an object detection model that has been trained to associate an image of a specific plant growing on the ground's surface with growth information, and outputs growth information of the specific plant growing on the ground's surface that is contained in the image.

8. acquiring an image of the Earth's surface; A growth information estimation method comprising the step of outputting growth information of a specified plant growing on the ground surface contained in an image using an object detection model that has been trained to correspond between an image of the specified plant growing on the ground surface and the growth information.

Images