Chlorophyll quantity learning and estimation system, chlorophyll quantity estimation model generation system, chlorophyll quantity estimation system, chlorophyll quantity estimation method, and computer program

Abstract

To allow a measurer to measure the amount of chlorophyll in the entire target plant without using a chlorophyll meter.SOLUTION: Based on chlorophyll amount and color information for each of multiple locations of a first individual that is a particular type of plant and used as a sample, a learning estimation server calculates a model representing the relationship between the chlorophyll amount and color information for the plant (#702). Then, based on an image of a second individual of the plant subject to estimation, and the calculated model, the server estimates the amount of chlorophyll in a part captured in each image element of the image (#704).SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] This invention relates to a technique for estimating the amount of chlorophyll in plants. [Background technology]

[0002] Plants that perform photosynthesis change the amount of chlorophyll in their leaves as they grow. This property is sometimes used to determine the timing of harvesting crops or flowers and other plants cultivated in a field, as well as the timing of fertilization.

[0003] Chlorophyll content is conventionally measured using a chlorophyll meter. For example, according to the chlorophyll meter described in Non-Patent Literature 1, a leaf is clamped with a clip-like head about the size of a thumb, and light is shone onto a portion of the clamped leaf. The SPAD (Soil & Plant Analyzer Development) value, which represents the chlorophyll content, is calculated based on the reflected light. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Chlorophyll meter "SPAD-502Plus", Konica Minolta website webpage: https: / / www.konicaminolta.jp / instruments / products / color / chlorophyll / index.html, accessed June 4, 2021. [Overview of the project] [Problems that the invention aims to solve]

[0005] As mentioned above, conventional chlorophyll meters can only measure the amount of chlorophyll in the parts of the plant that have been exposed to light. Therefore, measuring the amount of chlorophyll in plants cultivated in a large field traditionally required a great deal of time and manpower.

[0006] In view of such problems, an object of the present invention is to enable a measurer to measure the chlorophyll content of an entire target plant without using a chlorophyll meter.

Means for Solving the Problems

[0007] A chlorophyll content learning and estimation system according to one embodiment of the present invention includes the chlorophyll content and Includes the difference between the brightness of green and the brightness of blue. color information Training data consisting of at each of a plurality of positions of a first individual that is a specific type of plant and is used as a sample, and based on this, a model calculation means for calculating a model representing the relationship between the chlorophyll content and and includes the difference between the brightness of green and the brightness of blue. color Report of the plant, and an estimation means for estimating the chlorophyll content of the portion shown in each pixel of the image based on the image of a second individual that is the plant and is the object of estimation and the model.

Advantages of the Invention

[0008] According to the present invention, a measurer can measure the chlorophyll content of an entire target plant without using a chlorophyll meter.

Brief Description of the Drawings

[0009] [Figure 1] It is a diagram showing an example of the overall configuration of a chlorophyll content estimation system. [Figure 2] It is a diagram showing an example of the hardware configuration of a learning and estimation server. [Figure 3] It is a diagram showing an example of the functional configuration of a learning and estimation server. [Figure 4] It is a diagram showing an example of measurement value data. [Figure 5] It is a diagram showing an example of intermediate data. [Figure 6] It is a diagram showing an example of learning data. [Figure 7] It is a diagram showing an example of a linear model function. [Figure 8] It is a diagram showing an example of the state of photographing a field by a drone. [Figure 9] It is a diagram showing an example of a field image. [Figure 10] This figure shows an example of a partial image. [Figure 11] This figure shows an example of the positional relationship between multiple partial images. [Figure 12] This figure shows an example of a SPAD value distribution map. [Figure 13] This flowchart illustrates an example of the overall processing flow by a learning estimation program. [Figure 14] This is a flowchart illustrating an example of the learning process flow. [Figure 15] This is a flowchart illustrating an example of the estimation process flow. [Figure 16] The figure shows an example of a linear model function. [Modes for carrying out the invention]

[0010] Figure 1 shows an example of the overall configuration of the chlorophyll quantity estimation system 5. Figure 2 shows an example of the hardware configuration of the learning estimation server 1. Figure 3 shows an example of the functional configuration of the learning estimation server 1.

[0011] The chlorophyll quantity estimation system 5 is a system that estimates the chlorophyll quantity of plants cultivated in a field using AI (Artificial Intelligence) based on aerial photographs, and as shown in Figure 1, it consists of a learning estimation server 1, a first client 21, a second client 22, a drone 31, a chlorophyll meter 32, a colorimeter 33, and a communication line 4, etc.

[0012] The learning estimation server 1, the first client 21, and the second client 22 can communicate with each other via a communication line 4. The communication line 4 can be the internet, a public network, or a LAN (Local Area Network).

[0013] Drone 31 is a small UAV (Unmanned Aerial Vehicle) equipped with a digital camera and communication functions, used to photograph fields from above. A commercially available drone is used as Drone 31.

[0014] The chlorophyll meter 32 measures the amount of chlorophyll in plant leaves. A commercially available chlorophyll meter is used as the chlorophyll meter 32. The following explanation uses Konica Minolta's chlorophyll meter "SPAD-502 Plus" as an example. According to this chlorophyll meter, the SPAD (Soil & Plant Analyzer Development) value is calculated and output as a value representing the amount of chlorophyll.

[0015] The colorimeter 33 measures the color of plant leaves. A commercially available colorimeter 33 is used as the colorimeter 33. The following explanation uses the Konica Minolta colorimeter "CR-400" as an example. According to this colorimeter, the CIE (Commission Internationale de l'Eclairage) XYZ color system value (hereinafter referred to as "first color value (X,Y,Z)") is calculated and output as a value representing the color.

[0016] The first client 21 is a terminal device that causes the learning estimation server 1 to perform machine learning. The second client 22 is a terminal device that causes the learning estimation server 1 to perform the estimation of chlorophyll content in plants cultivated in the field. The first client 21 and the second client 22 are computers with communication capabilities, such as commercially available personal computers, tablet computers, or smartphones.

[0017] The learning estimation server 1 performs machine learning based on commands from the first client 21, or estimates chlorophyll quantity based on commands from the second client 22. A so-called server device or cloud server is used as the learning estimation server 1. The following explanation will use the case where a server device is used as the learning estimation server 1 as an example.

[0018] As shown in Figure 2, the learning estimation server 1 consists of a main unit 10, a display 11, a keyboard 12, and a pointing device 13, among other components. The main unit 10 consists of a processor 101, RAM (Random Access Memory) 102, ROM (Read Only Memory) 103, auxiliary storage device 104, a network controller 105, and an input / output interface 106, among other components.

[0019] The learning estimation program 14 (see Figure 3) is installed in the ROM 103 or auxiliary storage device 104. The learning estimation program 14 is a program for generating data for estimating chlorophyll content using machine learning and for estimating the chlorophyll content of plants in a field. The learning estimation program 14 is loaded into the RAM 102 and executed by the processor 101.

[0020] The network controller 105 is a communication device such as a NIC (Network Interface Card) and is used to communicate with the first client 21 and the second client 22, etc.

[0021] The input / output interface 106 is an input / output board compatible with standards such as USB (Universal Serial Bus), and a display 11, a keyboard 12, and a pointing device 13 are connected to it.

[0022] The display 11 shows a screen for inputting commands or information, and a screen showing the results of processing by the processor 101. The keyboard 12 and pointing device 13 are used for inputting commands or information.

[0023] By executing the learning estimation program 14 on the processor 101, the functions of the machine learning unit 15, the model data storage unit 16, and the estimation unit 17 shown in Figure 3 are realized. Below, each function will be explained using the example of estimating the amount of chlorophyll in cabbage, along with the usage of the first client 21, the second client 22, the drone 31, the chlorophyll meter 32, and the colorimeter 33.

[0024] [Machine Learning] Figure 4 shows an example of measured data 6A. Figure 5 shows an example of intermediate data 6C. Figure 6 shows an example of training data 6D. Figure 7 shows an example of the linear model function f2(P).

[0025] The worker prepares several cabbage plants. Ideally, these plants should have been grown in multiple fields with different soil conditions. For example, five cabbage plants should be prepared from each of six fields with different soil conditions.

[0026] The operator then measures the SPAD values ​​S and the first color values ​​(X, Y, Z) at multiple locations on the leaves of each of these plants using a chlorophyllometer 32 and a colorimeter 33. The SPAD values ​​S are transmitted from the chlorophyllometer 32 to the first client 21, and the first color values ​​(X, Y, Z) are transmitted from the colorimeter 33 to the first client 21.

[0027] As a result, the first client 21 receives the SPAD values ​​S and the first color values ​​(X, Y, Z) for each individual (cabbage) at multiple locations, as shown in Figure 4. The individual identifier is a code for identifying the individual, and the location identifier is a code for identifying the location where the SPAD values ​​S and the first color values ​​(X, Y, Z) were measured. Both identifiers are assigned for convenience and are not used during machine learning.

[0028] Hereinafter, data showing the SPAD value S and the first color values ​​(X, Y, Z) measured from the same location on the same individual using the chlorophyllometer 32 and the colorimeter 33 will be referred to as "measured value data 6A". If the SPAD value S and the first color values ​​(X, Y, Z) are measured at N2 locations from each of N1 individuals, the first client 21 will obtain (N1 × N2) pieces of measured value data 6A. A larger number of samples, i.e., the number of measured value data 6A, is preferable, but it is desirable to collect the measured value data 6A by measuring evenly from locations with varying concentrations.

[0029] Furthermore, the operator inputs the SPAD value S and the first color values ​​(X, Y, Z) of the plant from which the plant identification information (e.g., plant name) was measured. In this example, "cabbage" is entered.

[0030] The first client 21 then accesses a web page for machine learning via a web browser and sends these measurement data 6A, along with machine learning commands and learning command data 6B indicating the input identification information (plant name), to the learning estimation server 1.

[0031] In the learning estimation server 1, the machine learning unit 15 is composed of a measurement value receiving unit 151, a color space conversion unit 152, a feature calculation unit 153, and a regression analysis unit 154, as shown in Figure 3.

[0032] The measurement value receiving unit 151 receives measurement value data 6A and learning command data 6B from the first client 21 via a web page for machine learning.

[0033] The color space conversion unit 152 then converts the first color values ​​(X, Y, Z) shown in each measurement data 6A to RGB color system values ​​(hereinafter referred to as "second color values ​​(R, G, B)") according to a known conversion formula, thereby generating intermediate data 6C as shown in Figure 5. As a known conversion formula, for example, formulas (1) to (7) below are used.

[0034]

number

[0035] [Number]

[0036] [Number]

[0037] [Number]

[0038] R = 255 × f1(α(R)) … (5) G = 255 × f1(α(G)) … (6) B = 255 × f1(α(B)) … (7)

[0039] The feature quantity calculation unit 153 calculates the feature quantity P by substituting the second color representation values (R, G, B), that is, the brightness of red, green, and blue respectively, shown in each intermediate data 6C into the feature quantity calculation formula, and generates learning data 6D as shown in FIG. 6. For example, the following formula (8) is used as the feature quantity calculation formula for cabbage. P = (G - B) / (G + B) … (8)

[0040] For example, the feature quantity P of the first learning data 6D 0101 , 0101 , , 0101 , 0101 , , 0101 , 0101 ,

[0041] , , 0101 , 0101 , is obtained by substituting the second color representation values (R 0101 , B 0101 , G 0101 ) shown in the first intermediate data 6C into formula (8), P 0101 = (G 0101 - B 0101 ) / (G 0101 + B 0101 ) and is calculated as follows. <**********]] While this explanation uses cabbage as an example, the plant is not limited to agricultural crops such as cabbage and barley; it could also be an ornamental plant such as roses and tulips. Furthermore, a single feature calculation formula can be used for multiple types of plants. For example, formula (8) can be used not only for cabbage but also for onions, lettuce, leeks, chives, and citrus fruits.

[0042] However, onions and leeks have tubular leaves (tubular leaves), which do not transmit light easily. Therefore, when measuring chlorophyll content with the chlorophyll meter 32 during the collection of training data, these leaves must be split into thin pieces. However, in the inference phase described later, feature quantities P can be calculated based on images taken with a digital camera and the SPAD value S can be estimated without performing measurements with the chlorophyll meter 32, so there is no need to split these leaves. In addition, even for leaves other than tubular leaves, if they do not transmit light easily due to their thickness or other reasons, they can be split into thin pieces and the chlorophyll content measured. However, even in this case, the SPAD value S can be estimated without splitting these leaves in the inference phase described later.

[0043] The regression analysis unit 154 generates a linear model function that represents the correlation between the SPAD value S and the feature P by performing regression analysis based on the SPAD value S and feature P shown in each training data 6D. In this process, the SPAD value S is used as the dependent variable, and the feature P is used as the independent variable.

[0044] For regression analysis, known methods such as the least squares method are used. In this case, assuming that the correlation between the SPAD value S and the feature P is represented by a linear function, the regression analysis unit 154 generates a linear model function f2(P) as shown in Figure 7 by calculating the slope and intercept of the linear function using the least squares method.

[0045] The regression analysis unit 154 then stores the model data 6E, which represents the generated linear model function, in the model data storage unit 16, associating it with the identification information (in this example, the plant name "cabbage") shown in the learning command data 6B.

[0046] Through the work of the operators and the processing of each device, the model data 6E of the cabbage is generated and stored in the model data storage unit 16.

[0047] The learning estimation server 1 may also check whether the generated linear model function is significant using RMSE (Root Mean Square Error) or NRMSE (Normalized Mean Squared Logarithmic Error). In this case, if the RMSE or NRMSE is less than or equal to a predetermined value, it should be determined to be significant; otherwise, it should be determined to be not significant.

[0048] For example, the regression analysis unit 154 selects a portion of the training data 6D (e.g., 20%) for validation. Hereafter, the selected training data 6D will be referred to as "validation data". The validation data is not used to generate the model data 6E.

[0049] The regression analysis unit 154 converts the RGB values ​​(R, G, B) shown in one validation data into features P using a feature calculation formula. It calculates the SPAD value S by substituting each feature P into the linear model function shown in the model data 6E. It calculates the RMSE or NRMSE of the calculated SPAD value S and the SPAD value S shown in the validation data as the error between the two SPAD values ​​S. If the error is less than a predetermined value, it is judged as "pass," and if it is greater than or equal to the predetermined value, it is judged as "fail." The error is similarly calculated for the remaining validation data and judged as pass or fail. If the number of passes relative to the number of validation data is greater than or equal to a predetermined percentage (e.g., 95%), it is judged that the linear model is significant, and if it is less than the predetermined percentage, it is judged as not significant.

[0050] If the result is significant, the data representing that linear model function is stored in the model data storage unit 16 as training data 6D. If the result is not significant, the linear model function is discarded, and the linear model function can be regenerated by increasing the sample data, i.e., the measured value data 6A.

[0051] Model data 6E for plant types other than cabbage is generated by similar operations and processing, and is stored in the model data storage unit 16 in association with the identification information of that plant.

[0052] However, different feature calculation formulas are used for each plant. For example, the following formula (9) is used to calculate the features of hulless barley. P = GB … (9) [Estimated] Figure 8 shows an example of a drone 31 photographing field 800. Figure 9 shows an example of a field image 8A. Figure 10 shows an example of a partial image 8C. Figure 11 shows an example of the positional relationship between multiple partial images 8C. Figure 12 shows an example of a SPAD value distribution map 8F.

[0053] The user can have the chlorophyll content of plants being cultivated in the field estimated by the chlorophyll content estimation system 5. Below, the user's work and the processing of each part of each device will be explained using the example of estimating the total chlorophyll content of many individual cabbage plants being cultivated in field 800 shown in Figure 8.

[0054] The user places the color reference card 801 almost horizontally in the designated position in the field 800. The color reference card 801 is a piece of paper or board, such as pure white or 18 percent gray, for the white balance of an image, commonly referred to as a "white card" or "gray card."

[0055] The user flies the drone 31 above the farmland 800 and makes the drone 31 photograph the farmland 800 from a height h1. The height h1 is about 50 to 120 meters. Hereinafter, a case where a farmland image 8A, which is an overall image of the farmland 800 as shown in FIG. 9, is obtained by one shooting will be described as an example. The farmland image 8A includes an image of the color reference card 801 as a color reference image 8B. Due to the constraints of the patent drawings, FIG. 9 is a monochrome image as the farmland image 8A, but in reality it is a color image. The same applies to FIG. 10.

[0056] Furthermore, the user makes the drone 31 photograph the farmland 800 part by part from a height h2 lower than the height h1. h2 is about 6 meters. Thereby, a partial image 8C, which is an image of a part of the farmland 800 as shown in FIG. 10, is obtained. Since the partial image 8C is taken at a lower altitude than the farmland image 8A, the state of the farmland 800 is shown more clearly than in the farmland image 8A.

[0057] Hereinafter, a case where i×j partial images are obtained as the partial image 8C will be described as an example. Each partial image 8C is described as "partial image 8C 11 ", "partial image 8C 12 ", …, "partial image 8C ij " for distinction as shown in FIG. 11. Each position of the farmland 800 is shown in one of these partial images 8C. Note that the same part of the farmland 800 may be shown somewhat overlapping in adjacent partial images 8C.

[0058] The drone 31 transmits the image data of the farmland image 8A and the image data of each partial image 8C to the second client 22.

[0059] As a result, the second client 22 obtains image data of the entire field 800 and image data of each part of the field 800. Furthermore, the user inputs the plant name of the plant being cultivated in the field 800 into the second client 22 as identification information for that plant. In this example, the plant name "cabbage" is entered. Alternatively, the second client 22 may display a list of identification information (plant names) for the types of plants corresponding to each of the training data 6D stored in the model data storage unit 16 of the learning estimation server 1, and the user may select the identification information from the list.

[0060] The second client 22 then accesses the webpage for the estimation service and sends estimation command data 6F, which includes the estimation command and the input identification information (plant name), along with the image data of the field image 8A and the partial image 8C, to the learning estimation server 1.

[0061] In the learning estimation server 1, the estimation unit 17 is composed of an image data receiving unit 171, a brightness adjustment unit 172, a noise reduction unit 173, a SPAD value calculation unit 174, a distribution map generation unit 175, and an estimation result transmission unit 176, as shown in Figure 3.

[0062] The image data receiving unit 171 receives image data of the field image 8A, image data of the partial image 8C, and estimation command data 6F from the second client 22 via a web page for estimation services.

[0063] Then, the brightness adjustment unit 172 processes the white balance of the field image 8A as follows.

[0064] If the image data of field image 8A is non-RAW data such as JPEG (Joint Photographic Experts Group), it does not contain color temperature data, so the color reference image 8B included in field image 8A is used. Specifically, the brightness adjustment unit 172 corrects the entire field image 8A so that the color of color reference image 8B becomes the original color (the color of color reference card 801).

[0065] On the other hand, if the image data of field image 8A is RAW data, the brightness adjustment unit 172 corrects field image 8A to achieve a predetermined color temperature. In this case as well, as with non-RAW data, field image 8A may be corrected so that the colors of the color reference image 8B become the original colors.

[0066] Hereafter, the field image 8A corrected by the brightness adjustment unit 172 will be referred to as the "first corrected image 8D".

[0067] By the way, field 800 has pathways. Also, soil or weeds may be visible between adjacent cabbage plants. Therefore, field image 8A may include images of pathways, soil, and weeds. Furthermore, the cabbages may be damaged by insects. Pathways, soil, and weeds are not part of the cabbage, so they should be excluded from the estimation of the SPAD value. Also, areas damaged by insects should be excluded because they lack chlorophyll. Furthermore, shaded areas should also be excluded.

[0068] Therefore, the noise reduction unit 173 performs the following process to remove pixels from parts other than leaves, i.e., pixels representing pathways, soil, weeds, insect damage, or shadows, from the corrected field image 8A, i.e., the first corrected image 8D.

[0069] The noise reduction unit 173 identifies pixels representing leaves, pathways, soil, weeds, insect damage, and shadows from the first corrected image 8D. These pixels can be identified, for example, as follows: A large number of pixels representing leaves, pathways, soil, weeds, insect damage, and shadows are collected in advance, and a classifier (trained model) for classifying these is generated by machine learning. Then, this classifier is used to determine whether each pixel in the first corrected image 8D represents a leaf, pathway, soil, weed, insect damage, or shadow. In this case, a known algorithm such as MLC (Maximum Likelihood Classifier) ​​is used. ENVI5.5 from Harris Geospatial Solutions is a known software that applies MLC. This software may also be used for identification. Alternatively, a classifier created on a system other than the learning estimation server 1 may be used for identification.

[0070] Hereafter, pixels representing any of the following—pathways, soil, weeds, insect damage, and shadows—will be referred to as "noise pixels." The noise reduction unit 173 then removes the identified noise pixels from the first corrected image 8D.

[0071] Alternatively, the noise reduction unit 173 removes noise pixels from the first corrected image 8D as follows: It identifies noise images (in this example, pixels representing pathways, soil, weeds, insect damage, and shadows) from the partial images 8C of each part of the field 800. The method of identification is as described above.

[0072] Furthermore, the noise reduction unit 173 identifies a pixel in the first corrected image 8D that represents the same position as the noise pixel identified in the partial image 8C. This "position" is a position within the field 800. In other words, the identified noise pixel is located at the same position (L) within the field 800. x ,L y If the pixel represents ), the noise reduction unit 173 selects a pixel from the first corrected image 8D at the position (L x ,L yThe pixel representing ) is identified. This pixel can also be said to be a noise pixel. Then, the noise reduction unit 173 removes the identified noise pixel from the first corrected image 8D.

[0073] This method allows for more reliable removal of noise pixels than the method of directly identifying and removing noise pixels from the first corrected image 8D without using the partial image 8C. However, information on the correspondence between pixels in the partial image 8C and pixels in the first corrected image 8D is necessary. This information can be obtained by known methods, but for example, it can be obtained as follows.

[0074] When the drone 31 captures the field image 8A and the partial image 8C, it acquires location information such as GPS (Global Positioning System). The noise reduction unit 173 then calculates which pixel in the first corrected image 8D corresponds to each pixel in the partial image 8C based on the acquired location information.

[0075] Alternatively, the noise reduction unit 173 reduces the partial image 8C to match the scale (actual length per pixel) of the first corrected image 8D. Then, by matching the reduced partial image 8C with the first corrected image 8D, it identifies which pixel in the first corrected image 8D corresponds to each pixel in the reduced partial image 8C.

[0076] Hereafter, the first corrected image 8D, from which noise pixels have been removed, will be referred to as the "second corrected image 8E". Note that if the target of estimation is hulless barley, pixels from the ear portion may be added to the noise pixels. Alternatively, if the target is apples or oranges, pixels from the fruit, as well as pixels from the trunk and branches of the tree, may be added to the noise pixels.

[0077] The SPAD value calculation unit 174 calculates the SPAD value S for each pixel of the second corrected image 8E as follows, based on the linear model function shown in the model data 6E of the plant name shown in the estimated command data 6F.

[0078] The SPAD value calculation unit 174 calculates a feature P for each pixel of the second corrected image 8E using the feature calculation formula for the plant name. Then, it calculates the SPAD value S by substituting the calculated feature P into the linear model function of the plant.

[0079] In this example, since the plant is a cabbage, the SPAD value calculation unit 174 calculates the feature quantity P of each pixel using equation (8). Then, by substituting the feature quantity P of each pixel into the linear model function (see Figure 7) shown in the cabbage model data 6E, the SPAD value S of each pixel is calculated. For example, the RGB value of a certain pixel shown in the second corrected image 8E is (R a ,G a ,B a If this is the case, the feature quantity P of that pixel a (G a -B a ) / (G a +B a ) is calculated. Then, the SPAD value S of that pixel is calculated. a As, f(P a Calculate ).

[0080] The distribution map generation unit 175 generates a distribution map 8F, as shown in Figure 12, which represents the magnitude of the SPAD value S of each pixel in the second corrected image 8E calculated by the SPAD value calculation unit 174, in grayscale. For example, if the upper limit (maximum value) of the SPAD value S is S MAX The grayscale is 256 (0-255), and the number of pixels is U MAX ×V MAX When generating a distribution map as SPAD value distribution map 8F, the pixel value of the pixel at coordinate (u,v) in SPAD value distribution map 8F is (255 / S MAX )×S (u,v) , becomes. Note that the decimal part can be rounded up, rounded down, or rounded to the nearest integer. u=1,2,…,U MAX Therefore, v = 1, 2, ..., V MAX That is the case.

[0081] Alternatively, the distribution map generation unit 175 may generate a distribution map 8F that represents the magnitude of the SPAD value S of each pixel on a color scale. For example, the maximum value (i.e., 255) may be represented by 100% red brightness, the intermediate value (127) by 100% green brightness, and the minimum value (0) by 100% blue brightness.

[0082] The estimation result transmission unit 176 transmits estimation result data 6G, which shows the SPAD value S of each pixel of the second corrected image 8E, to the second client 22 along with the image data of the SPAD value distribution map 8F.

[0083] When the second client 22 receives the estimated result data 6G and the distribution map 8F image data from the learning estimation server 1, it displays the SPAD value of each pixel shown in the estimated result data 6G, or displays the distribution map 8F.

[0084] Figure 13 is a flowchart illustrating an example of the overall processing flow by the learning estimation program 14. Figure 14 is a flowchart illustrating an example of the learning process flow. Figure 15 is a flowchart illustrating an example of the estimation process flow.

[0085] Next, the overall processing flow of the learning estimation server 1 will be explained with reference to the flowchart. The learning estimation server 1 processes according to the learning estimation program 14, following the procedure shown in Figure 13.

[0086] When the learning estimation server 1 receives the learning command data 6B along with the measured value data 6A (Yes in #701 in Figure 13), it performs machine learning according to the procedure shown in Figure 14 (#702).

[0087] The learning estimation server 1 converts the XYZ values ​​(X,Y,Z) shown in each measurement data 6A into RGB values ​​(R,G,B) (#711 in Figure 14), and calculates the feature quantity P by substituting the RGB values ​​(R,G,B) into the plant feature quantity calculation formula for the identification information shown in the learning command data 6B (#712).

[0088] Then, the learning estimation server 1 generates a linear model function that shows the relationship between feature P and SPAD value S through regression analysis (#713), and saves the data showing the generated linear model function as model data 6E, associated with the identification information of the plant (#714).

[0089] Returning to Figure 13, when the learning estimation server 1 receives the estimation command data 6F along with the image data of the field image 8A (see Figure 9) and the partial image 8C (see Figure 10) (Yes in #703), it performs estimation (inference) processing according to the procedure shown in Figure 15 (#704).

[0090] The learning estimation server 1 generates a first corrected image 8D by adjusting the brightness of the field image 8A based on a reference (#721 in Figure 15). A second corrected image 8E is generated by removing noise pixels from the first corrected image 8D (#722). The feature quantity P of each pixel is calculated by substituting the pixel value (RGB value) of each pixel in the second corrected image 8E into the linear model function of the model data 6E corresponding to the identification information shown in the estimation command data 6F (#723). An image representing the magnitude of the SPAD value S of each pixel, i.e., a distribution map 8F (see Figure 12), is generated in grayscale (#724).

[0091] Then, the learning estimation server 1 sends back (transmits) the estimation result data 6G, which shows the feature quantities P of each pixel of the second corrected image 8E, to the source of the estimation command data 6F, along with the image data of the estimation command data 6F (#725).

[0092] Returning to Figure 13, while the learning estimation server 1 is providing the service for estimating chlorophyll quantity (SPAD value) (Yes in #705), it performs the process in step #702 each time it receives learning command data 6B and the process in step #704 each time it receives estimation command data 6F.

[0093] According to this embodiment, the user (measurer) can measure the total chlorophyll content (SPAD value S) of the target plant without using a chlorophyll meter. Moreover, since the SPAD value S is estimated based on the color temperature of the image of the target plant, it is less affected by conditions such as season, time of day, and weather, and can be estimated stably.

[0094] In this embodiment, the feature calculation unit 153 calculated (GB) / (G+B) as the feature quantity P for cabbage, as shown on the right side of equation (8). It also calculated GB as the feature quantity P for hulless barley, as shown on the right side of equation (9). Thus, both feature quantities P are calculated based on GB.

[0095] The characteristic quantity P of hulless barley may also be calculated by the following equation (10). P = 2G - RB … (10)

[0096] The right-hand side of equation (10) can be transformed into (GB) + (GR). In other words, even when using equation (10), the feature P is calculated based on GB.

[0097] Figure 16 shows an example of a linear model function f3(P). Depending on the plant species, features may be calculated using equations other than those in (8) to (10), and a linear model function may be generated. The feature calculation unit 153 and the regression analysis unit 154 generate, for example, a linear model function f3(P) for tomato leaves as follows.

[0098] The worker measures the SPAD value S and the first color values ​​(X, Y, Z) at multiple locations on the leaves of each of several tomato plants using a chlorophyll meter 32 and a colorimeter 33.

[0099] The feature calculation unit 153 calculates the feature P for each location based on the measured first color values ​​(X, Y, Z) using equation (11). P = xy … (11) However, x = X / (X+Y+Z) and y = Y / (X+Y+Z). In other words, the feature vector P is calculated based on the color values ​​x and y of the Yxy color system. This yields data corresponding to the training data 6D shown in Figure 6, which consists of combinations of location-specific feature vector P and SPAD values ​​S.

[0100] The regression analysis unit 154 generates a linear model function that represents the correlation between the SPAD value S and the feature P by performing regression analysis based on the SPAD value S and the feature P, similar to the case of cabbage. This yields a linear model function f3(P) as shown in equation (12) or Figure 16. f3(P) = aP + b … (12) a is a negative value, which in the example in Figure 16 is -809.33. b is a positive value, which in the example in Figure 16 is 150.84. Since a is a negative value, the larger the product of x and y, i.e., the feature P, the smaller the SPAD value S.

[0101] Then, the estimation unit 17 estimates the SPAD value S of the tomato based on the linear model function f3(P). Learning estimation server 1 is L * a * b * Feature vectors P may be calculated based on the values ​​of a color system. For example, L * Alternatively, you can calculate this as feature P. * This can also be calculated as a feature P.

[0102] In this embodiment, the case in which the machine learning unit 15, model data storage unit 16, and estimation unit 17 are integrated into the learning and estimation server 1 has been described as an example, but they may be distributed across multiple devices or systems. For example, the machine learning unit 15 may be implemented by a computer or system for machine learning, and the model data storage unit 16 and estimation unit 17 may be implemented by a computer or system for inference.

[0103] In this embodiment, the estimation unit 17 estimated the SPAD value S based on an image of the plant taken from above the field 800, but it may also be estimated based on an image of the plant taken from the side. For example, when estimating the SPAD value of the leaves in the middle section of a tomato plant, it is effective to use an image taken from the side. It is also effective when estimating the SPAD value of the leaves of plants cultivated in a plant factory or greenhouse.

[0104] In this embodiment, the learning estimation server 1 performed machine learning and estimation for each type, such as "cabbage" and "hulled barley," but machine learning and estimation may be performed according to more specific types. For example, machine learning and estimation may be performed for each variety, such as "winter cabbage," "spring cabbage," "green ball," and "red cabbage." Similarly, in the case of tomatoes, machine learning and estimation may be performed for each variety, such as "large tomato," "medium tomato," and "mini tomato."

[0105] Alternatively, machine learning and estimation may be performed by combining type with other attributes. For example, machine learning and estimation of cabbage may be performed for each soil type of the field where it is grown. That is, the learning and estimation server 1 generates cabbage model data 6E for each soil type of the field where it is grown. Then, using the learning data 6D corresponding to the soil type of the field to be estimated, it estimates the SPAD value S of the cabbage grown in that field.

[0106] As described above, this embodiment allows for stable estimation that is less affected by conditions such as season, time of day, and weather. However, for certain types or varieties of plants (e.g., citrus fruits), to further improve accuracy, one linear model function may be prepared and used for each condition. For example, a linear model function may be prepared and used for each weather condition, such as sunny days, cloudy days, and rainy days. Alternatively, a linear model function may be prepared and used for each time of day, such as around sunrise, around noon, and around sunset.

[0107] In this embodiment, the learning estimation server 1 represents the estimated SPAD values ​​S for each part (pixel) of the field 800 using a two-dimensional map such as the SPAD value distribution map 8F (see Figure 12), but it may also be represented using a three-dimensional map. For example, it may be represented using a three-dimensional bar graph. That is, the field 800 may be represented on a vertical × horizontal plane, and the SPAD values ​​S may be represented by bars of height corresponding to their size.

[0108] The learning estimation server 1 may also notify the second client 22 of the harvest time for the plants being cultivated in field 800, for example, as follows:

[0109] The user periodically (for example, daily or every few days) has the drone 31 photograph the entire field 800 and each part of it as described above, and has the second client 22 send the field image data 8A, the partial image data 8C of each part, and the estimation command data 6F to the learning estimation server 1. In addition to the functions shown in Figure 3, the learning estimation server 1 has a timing determination unit and a notification unit.

[0110] When the estimation unit 17 of the learning estimation server 1 receives this data, it generates a second corrected image 8E as described above and estimates the SPAD value S of each pixel in the second corrected image 8E. The timing determination unit determines whether the characteristics of these estimated SPAD values ​​S meet a predetermined standard corresponding to the plant. If it is determined that the predetermined standard has been met, the notification unit notifies the second client 22 that it is time for harvest. For example, a notification is made when the average value of these SPAD values ​​S exceeds a predetermined value. Alternatively, a notification is made when the ratio of pixels with SPAD values ​​S greater than a predetermined value to the total number of pixels exceeds a predetermined value.

[0111] The learning estimation server 1 may use a mechanism similar to the one used to notify the harvest time to determine the timing of fertilization and sowing using a timing determination unit and notify the timing using a notification unit. Alternatively, the timing determination unit may continuously record a value (e.g., the average value) that represents the SPAD value S of the entire field 800 and predict the timing of harvest, fertilization, or sowing. The notification unit may then notify the second client 22 of the predicted timing.

[0112] Alternatively, the timing determination unit may notify a party other than the second client 22. For example, it may notify the terminal device of the person in charge of field 800. The notification destination may be set according to the task. That is, notification destinations may be set for each task such as harvesting, fertilizing, and sowing, and notifications may be sent to the appropriate destination for each task.

[0113] In this embodiment, the learning estimation server 1 generates a second corrected image 8E by adjusting the color temperature of the field image 8A and then removing noise pixels. However, it may also be generated by removing noise pixels first and then adjusting the color temperature.

[0114] In this embodiment, the learning estimation server 1 generates a linear model function by linear regression analysis as the model for estimation (inference), but it may also generate a trained model by deep learning. The SPAD value S may then be estimated using the trained model.

[0115] In this embodiment, the learning estimation server 1 estimated the SPAD values ​​S of many individuals being cultivated in the field, but it may also estimate the SPAD values ​​S of one or a few individuals. In this case, instead of the drone 31 acquiring field images 8A, the user can acquire images of one or a few individuals by taking pictures of them from nearby with a digital camera. The learning estimation server 1 can then adjust the color temperature of these images, remove noise pixels, and then estimate the SPAD values ​​S.

[0116] Furthermore, the overall configuration or individual parts of the chlorophyll quantity estimation system 5, the learning estimation server 1, the first client 21, and the second client 22, as well as the processing content, processing order, and data structure, can be appropriately modified in accordance with the spirit of the present invention. [Explanation of Symbols]

[0117] 1. Learning Estimation Server (Chlorophyll Quantity Learning Estimation System) 15. Machine Learning Department (Model Generation System for Chlorophyll Quantity Estimation) 151 Measurement Value Receiving Unit (Acquisition Means) 152 Color space conversion unit (acquisition means) 153 Feature Calculation Unit (Model Calculation Means) 154 Regression Analysis Unit (Model Calculation Means) 16. Model data storage unit (storage means) 17. Estimation Unit (Chlorophyll Quantity Estimation System) 171 Image data receiving unit (acquisition means) 172 Brightness adjustment unit (corrected image generation means) 173 Noise reduction unit (corrected image generation means) 174 SPAD value calculation unit (estimation means) 6D training data (chlorophyll content, color information) 6E Model Data (Model) 8A Field image 8E Second corrected image (image, input image)

Claims

1. A model calculation means calculates a model representing the relationship between the chlorophyll content of a first individual plant of a specific species used as a sample and the color information including the difference between the brightness of green and the brightness of blue, based on training data consisting of the chlorophyll content and color information including the difference between the brightness of green and the brightness of blue at multiple locations of the first individual plant, and An estimation means for estimating the amount of chlorophyll in each pixel of an image based on an image of a second individual which is the plant being estimated and the model, A chlorophyll quantity learning and estimation system characterized by having the following features.

2. The aforementioned plants are cabbage, onion, or hulless barley. The training data consists of color information including the amount of chlorophyll at each of the multiple locations of the first individual and the difference between the lightness of green and the lightness of blue relative to the sum of the lightness of green and the lightness of blue. The chlorophyll quantity learning estimation system according to claim 1.

3. The model calculation means calculates the model based on the learning data of each of the plurality of first individuals. A chlorophyll quantity learning estimation system according to claim 1 or claim 2.

4. An acquisition means for acquiring learning data consisting of chlorophyll content at multiple locations on an individual of a specific type of plant, and color information including the difference between the brightness of green and the brightness of blue, A model calculation means calculates a model that represents the relationship between the amount of chlorophyll in the plant and color information including the difference between the brightness of green and the brightness of blue, based on the learning data acquired by the acquisition means. A model generation system for estimating chlorophyll content, characterized by having the following features.

5. The aforementioned plants are cabbage, onion, or hulless barley. The training data consists of color information including the amount of chlorophyll at each of the multiple locations of the first individual and the difference between the lightness of green and the lightness of blue relative to the sum of the lightness of green and the lightness of blue. A chlorophyll quantity estimation model generation system according to claim 4.

6. The model calculation means calculates the model based on the learning data of each of the plurality of first individuals. A chlorophyll quantity estimation model generation system according to claim 4 or claim 5.

7. An acquisition means for acquiring an input image which is an image of a specific type of plant and an individual that is the target of estimation, Estimation means for estimating the amount of chlorophyll in each pixel of the input image based on the input image acquired by the acquisition means and the model calculated by the chlorophyll amount estimation model generation system of claim 4, A chlorophyll quantity estimation system characterized by having the following features.

8. Correction image generation means for generating a corrected image by removing noise pixels which are pixels corresponding to objects other than leaves, It has, The acquisition means acquires an image taken from above of a field where multiple individuals are being cultivated as the input image. The corrected image generation means generates the corrected image by removing the noise pixels from the input image. The estimation means uses the corrected image as the input image to estimate the amount of chlorophyll in the portion captured by each pixel. The chlorophyll quantity estimation system according to claim 7.

9. The corrected image generation means generates an image as the corrected image, in which the color temperature has been adjusted to match the reference. The chlorophyll quantity estimation system according to claim 8.

10. A determination means for determining whether or not the time for work on the plant has come, based on the amount of chlorophyll estimated by the estimation means, When it is determined that the aforementioned period has arrived, a notification means for notifying that the aforementioned period has arrived, Having, A chlorophyll quantity estimation system according to any one of claims 7 to 9.

11. A prediction means for predicting the timing of work on the plant based on the amount of chlorophyll estimated by the estimation means, A notification means for notifying the predicted time, Having, A chlorophyll quantity estimation system according to any one of claims 7 to 9.

12. The aforementioned work is harvesting the plants, fertilizing the plants, or sowing the plants. The chlorophyll quantity estimation system according to claim 10.

13. Based on training data consisting of chlorophyll content and color information including the difference between green and blue lightness at multiple locations of a first individual plant of a specific species used as a sample, a computer calculates a model that represents the relationship between the chlorophyll content of the plant and the color information including the difference between green and blue lightness. Based on the image of the second individual plant that is the subject of estimation and the model, the computer estimates the amount of chlorophyll in the part of the image that is captured by each pixel. A method for estimating chlorophyll content, characterized by the features described above.

14. The computer is made to perform an acquisition process to obtain training data consisting of chlorophyll content at multiple locations on an individual plant of a specific species, as well as color information including the difference in brightness between green and blue. The computer is instructed to perform a calculation process to calculate a model that represents the relationship between the amount of chlorophyll in the plant and color information, including the difference between the brightness of green and the brightness of blue, based on the learning data acquired by the acquisition process. A computer program characterized by the following features.

15. The computer is instructed to perform an acquisition process to obtain an input image, which is an image of a specific type of plant and the target of estimation. Based on the input image acquired by the acquisition process and the model calculated by the chlorophyll amount estimation model generation system of claim 4, the computer is instructed to perform an estimation process to estimate the amount of chlorophyll in the portion of the input image that is captured by each pixel. A computer program characterized by the following features.

Images