Information processing device, information processing method, and program

Abstract

To provide a technology for predicting a number of pieces of fruits of crops, while restraining labor required for preparations.SOLUTION: An information processing device: obtains an actual measured value of a number of pieces of fruits of crops within a range corresponding to a learning image from a label of the learning image (Step S204); and recognizes an area encompassing the fruit for each number of pieces of the fruits of the crops from the learning image (Step S206). Then, the information processing device: derives an estimated value of the number of pieces of the fruits of the crops within the range corresponding to the learning image, from a count of the recognized area using the actual measured value (Step S208); and executes machine learning of a prediction model to predict the number of pieces of the fruits of the crops using the actual measured value and the estimated value (Step S210). Then, the information processing device applies the count of the area for each number of pieces of the fruits of the crops recognized from a prediction image to the prediction model, for deriving a predicted value of the number of pieces of the fruits of the crops within the range corresponding to the prediction image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to predicting crop fruit count using images of farm fields. [Background technology]

[0002] Various techniques have been proposed for predicting the yield of agricultural crops grown in fields. For example, Japanese Patent No. 6673442 (Patent Document 1) discloses a harvesting system in which a mobile work vehicle travels between multiple cultivation beds, photographs the fruits of plants with a camera, counts the total number of mature fruits with a fruit discrimination device, calculates a provisional harvest number according to the work capacity of each of multiple workers who harvest the mature fruits, and sets the work area of ​​each worker in a cultivation room based on the provisional harvest number.

[0003] In addition, JP 2020-24672 A (Patent Document 2) discloses an information processing device that trains a trained model that derives an estimate of the number of grape bunches to be harvested from an image taken in a field, uses the trained model to derive an estimate of the number of grape bunches to be harvested from the image of the field, and derives an estimate of the weight from the derived estimate of the number of bunches. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6673442 [Patent Document 2] JP 2020-24672 A Summary of the Invention [Problem to be solved by the invention]

[0005] As described in Patent Document 1, counting the total number of fruits from an image requires a great deal of skill from an operator. The technology described in Patent Document 2 is expected to reduce the counting operation in that it uses a machine learning model. However, while the number of fruits is more important than the weight as a yield for some agricultural crops, the technology disclosed in Patent Document 2 derives the number and weight of bunches as estimated values, but does not derive the number of fruits. If an attempt is made to combine the technology described in Patent Document 1 with a technology using a machine learning model, a great deal of effort is required in preparing learning data, as an annotation, to tag the area of ​​each fruit in the learning image.

[0006] The present disclosure has been devised in light of the above-mentioned circumstances, and has an object to provide a technique for predicting the number of fruits of an agricultural crop while reducing the labor required for preparation. [Means for solving the problem]

[0007] According to one aspect of the present disclosure, there is provided an information processing device including: an acquisition means for acquiring actual measurements of the number of crop fruits within a range corresponding to a training image; a recognition means for recognizing an area surrounding each number of crop fruits from the training image; a learning means for performing machine learning of a prediction model for predicting the number of crop fruits using the actual measurements and an estimate of the number of crop fruits derived using the number of areas recognized by the recognition means; and a prediction means for deriving a predicted value for the number of crop fruits within a range corresponding to the prediction image by applying to the prediction model the number of areas for each number of crop fruits recognized by the recognition means from the prediction image.

[0008] The learning means may further use values ​​related to the environment of the field of the crop in the machine learning of the prediction model. The prediction means may further apply the values ​​related to the environment of the field of the crop to the prediction model to derive a prediction value.

[0009] The environmental values ​​may include values ​​relating to the amount of solar radiation in the crop field. In machine learning, information indicating the area for each number of fruit of a crop that is added to a learning image may be used.

[0010] The information processing device may include a display means for displaying an area surrounding each of the fruits of the agricultural crop recognized from the prediction image for each number of fruits.

[0011] The information processing device may further include a second recognition means for recognizing areas surrounding each of the fruit of the crop from the learning image. The learning means may further utilize a second estimate of the number of fruit of the crop derived from the number of areas recognized by the second recognition means in machine learning of the prediction model. The prediction means may derive a prediction value by further applying the number of areas surrounding each of the fruit of the crop recognized by the second recognition means from the prediction image to the prediction model.

[0012] According to another aspect of the present disclosure, there is provided an information processing method executed by a computer, comprising the steps of obtaining actual measurements of the number of crop fruits in an area corresponding to a training image; recognizing from the training image an area surrounding each number of crop fruits; deriving an estimate of the number of crop fruits in an area corresponding to the training image from the number of areas recognized in the recognition step; performing machine learning of a predictive model for predicting the number of crop fruits using the actual measurements and the estimate; and deriving a predicted value of the number of crop fruits in an area corresponding to the prediction image by applying the number of areas for each number of crop fruits recognized from the prediction image to the predictive model.

[0013] The step of performing machine learning may further utilize values ​​related to the environment of the field of the crop in the machine learning of the prediction model. The step of deriving the predicted value may further apply the values ​​related to the environment of the field of the crop to the prediction model to derive the predicted value.

[0014] The environmental values ​​may include values ​​relating to the amount of solar radiation in the crop field. In machine learning, information indicating the area for each number of fruit of a crop that is added to a learning image may be used.

[0015] The method further includes a step of displaying an area surrounding each of the fruits of the crop recognized from the prediction image for each number of fruits.

[0016] The information processing method may further include a step of recognizing areas surrounding each of the crop fruits from the training image. The step of performing machine learning may include deriving a second estimate of the number of crop fruits using the number of areas surrounding each of the crop fruits recognized from the training image, and the second estimate may further be used in the machine learning of the predictive model. The step of deriving a predicted value may derive a predicted value by further applying the number of areas surrounding each of the crop fruits recognized from the prediction image to the predictive model.

[0017] According to yet another aspect of the present disclosure, there is provided a program which, when executed by a computer, causes the computer to execute the above-described information processing method. Effect of the Invention

[0018] According to an aspect of the present disclosure, in predicting the number of fruits of a crop using an image, information on an area assigned to each number of fruits of the crop is used. As a result, in annotation, two or more fruits are treated as one area, and the number of fruits of the crop is derived as a predicted value. Therefore, it is possible to predict the number of fruits of the crop while reducing the labor required for annotation work. [Brief description of the drawings]

[0019] [Figure 1] 1 is a schematic diagram illustrating an example of a hardware configuration of an information processing device 100. FIG. [Diagram 2] FIG. 2 is a diagram illustrating functional blocks of the information processing device 100. [Diagram 3] FIG. 2 is a diagram showing an example of a screen displaying a recognition result by a recognition unit 205. [Figure 4] 4 is a flowchart showing an example of a main routine executed by the information processing device 100. [Diagram 5] 5 is a flowchart of a subroutine of step S200 in FIG. 4. [Figure 6] 5 is a flowchart of a subroutine of step S400 in FIG. 4. [Figure 7] 1 is a diagram for explaining the accuracy of prediction by the information processing device 100. FIG. [Figure 8] FIG. 2 is a diagram showing a first modified example of the functional blocks of the information processing device 100. [Figure 9] 13 is a flowchart of a subroutine of a learning process in the modified example (1). [Figure 10] 13 is a flowchart of a subroutine of a prediction process in the modified example (1). [Figure 11] 1 is a diagram for explaining the accuracy of prediction by the information processing device 100. FIG. [Figure 12] FIG. 13 is a diagram showing a second modified example of the functional blocks of the information processing device 100. [Figure 13] 13 is a diagram showing an example of a screen displaying a recognition result by a first recognition unit 2051. FIG. [Figure 14] 13 is a flowchart of a subroutine of a learning process in the modified example (2). [Figure 15] 13 is a flowchart of a subroutine of a prediction process in the modified example (2). [Figure 16] 1 is a diagram for explaining the accuracy of prediction by the information processing device 100. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are given the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. Note that the embodiments and modifications described below may be appropriately and selectively combined.

[0021] [Hardware configuration of information processing device] 1 is a schematic diagram showing an example of a hardware configuration of an information processing device 100. The information processing device 100 recognizes areas corresponding to bunches of fruit of an agricultural product such as tomatoes from an image captured in a farm field. The information processing device 100 recognizes areas corresponding to each bunch for each number of fruits that are considered to compose the bunch. Then, the information processing device 100 derives a prediction result of the number of fruits in the range corresponding to the image by using the area recognition result.

[0022] The agricultural product that is the target of prediction by the information processing device 100 is not limited to tomatoes, and may be any agricultural product, such as apples or mandarin oranges, as long as the fruits are handled one by one.

[0023] 1, an information processing device 100 includes a control device 101, a read only memory (ROM) 102, a random access memory (RAM) 103, a communication interface 104, a display interface 105, an input interface 107, and a storage device 120. These components are connected to a bus 110.

[0024] The control device 101 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or a combination thereof.

[0025] The control device 101 controls the operation of the information processing device 100 by executing various programs such as a learning program 122 and a prediction program 124. In one implementation example, these programs are stored in the storage device 120. At least one of these programs may be stored in an external server, and the control device 101 may execute these programs using an API (Application Programming Interface).

[0026] A LAN (Local Area Network), an antenna, and the like are connected to the communication interface 104. The information processing device 100 exchanges data with external devices via the communication interface 104. The information processing device 100 may be configured to be able to download various programs from a server via the communication interface 104.

[0027] A display 106 is connected to the display interface 105. The display interface 105 sends an image signal for displaying an image to the display 106 in accordance with a command from the control device 101 or the like. The display 106 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, or other display device. The display 106 may be configured integrally with the information processing device 100, or may be configured separately from the information processing device 100.

[0028] An input device 108 is connected to the input interface 107. The input device 108 is, for example, a mouse, a keyboard, a touch panel, or other device capable of accepting a user's operation. The input device 108 may be configured integrally with the information processing device 100, or may be configured separately from the information processing device 100.

[0029] The storage device 120 is, for example, a storage medium such as a hard disk or a flash memory. The storage device 120 stores various programs and data used to execute the programs. The storage location of these programs and / or data is not limited to the storage device 120, and may be a storage area (e.g., cache memory, etc.) of the control device 101, the ROM 102, the RAM 103, or an external device (e.g., a server).

[0030] [Function block] Fig. 2 is a diagram showing functional blocks of the information processing device 100. As shown in Fig. 2, the information processing device 100 functions as an information acquisition unit 201, a learning unit 202, a parameter storage unit 203, a prediction unit 204, and an output unit 207. The prediction unit 204 includes a recognition unit 205 and a number derivation unit 206.

[0031] The information processing device 100 includes a prediction model, and executes machine learning of the prediction model by using the learning data 210. When a video 220 to be predicted is input, the information processing device 100 also uses the prediction model to output a predicted value of the number of tomatoes to be harvested two weeks after the video 220 to be predicted is shot, within a range corresponding to the video 220 to be predicted.

[0032] The video included in the learning data 210 is generated based on, for example, a video of a farm field captured by a camera mounted on a mobile vehicle (hereinafter also referred to as the "original video"). In one example, the original video is generated in avi format, and adjustments are made to the original video. The adjustments include format conversion to MP4 format, cutting out the time portion in which the farm field was captured, distortion correction, and trimming to remove parts other than the crops (such as the body of the mobile vehicle). The video included in the learning data 210 is generated by adding the above-mentioned labels (position of the area, number of fruits surrounded by each area, etc.) to each frame of the video after adjustment.

[0033] The video 220 to be predicted is also generated by performing the above adjustments on a video of a farm field captured by the camera, similar to the video included in the learning data 210.

[0034] The information acquisition unit 201 extracts the actual measurement value of the number of tomatoes from the labels of the learning data 210, and outputs it to the learning unit 202. The actual measurement value is the number of tomatoes that were actually harvested two weeks after the shooting of the video, within the range that the video included in the learning data 210 corresponds to.

[0035] The learning unit 202, for example, executes machine learning of a prediction model. The parameter storage unit 203 stores parameters of the prediction model that are set in the machine learning by the learning unit 202.

[0036] The prediction unit 204 receives an input of an image, and derives from the input image an estimate or prediction value of the number of tomatoes that will be harvested two weeks after the image was taken in a range corresponding to the image. The prediction unit 204 includes a prediction model. In this specification, the number of tomatoes derived by the prediction unit 204 in the learning stage of the prediction model is formally referred to as an "estimated value," and the number of tomatoes derived by the prediction unit 204 in the utilization stage of the prediction model is formally referred to as a "predicted value."

[0037] In the prediction unit 204, the recognition unit 205 recognizes areas surrounding tomatoes from the input image for each number of tomatoes surrounded. The recognition unit 205 recognizes the areas in the input image according to an object recognition algorithm such as YOLO (You Only Look Once). The recognition unit 205 outputs the area recognition results to the number derivation unit 206. The recognition results of the recognition unit 205 include information representing each of one or more areas recognized from the image and information representing the number of tomatoes surrounded by each area. The number derivation unit 206 calculates the above estimated value or predicted value by applying the recognition results output from the recognition unit 205 to a prediction model.

[0038] The prediction model may be a linear regression model such as that shown in the following equation (1). f(x) = αx … (1) In formula (1), α is a coefficient that is updated in machine learning. The value α updated in machine learning is stored in the parameter storage unit 203. In formula (1), x is an explanatory variable of the objective variable f(x), and is expressed by the following formula (2).

[0039] x = pA + qB + rC + … … (2) In formula (2), A, B, and C each represent the number of recognized regions. The regions corresponding to A, B, and C each have a different number of tomatoes that they enclose. For example, A is the number of regions that enclose one tomato fruit, B is the number of regions that enclose two tomato fruits, and C is the number of regions that enclose three tomato fruits. p, q, and r are the coefficients of A, B, and C, respectively. p, q, and r may each be a predetermined value, or may be a value that is updated in machine learning. When updated in machine learning, p, q, and r after the update in machine learning are stored in the parameter storage unit 203.

[0040] The number of terms on the right side of formula (2) represents the number of types of regions (each type has a different number of tomatoes surrounded by the region) that the predictive model recognizes. For example, if the predictive model recognizes regions with one fruit, regions with two fruits, and regions with three fruits, the number of terms on the right side of formula (2) is "3," as shown in the following formula (2X). In formula (2X), A represents the number of regions with one fruit. Additionally, B represents the number of regions with two fruits, and C represents the number of regions with three fruits.

[0041] x = pA + qB + rC … (2X) The output unit 207 outputs the predicted value derived by the prediction unit 204 by displaying it on the display 106, for example.

[0042] Each of the information acquisition unit 201, the learning unit 202, the prediction unit 204, and the output unit 207 may be realized by the control device 101 executing a given program. More specifically, as the prediction unit 204, the control device 101 may function as the recognition unit 205 by executing a program for object recognition such as YOLO, or may function as the number derivation unit 206 by executing the prediction program 124. In addition, the control device 101 may function as the learning unit 202 by executing the learning program 122. The learning program 122 and the prediction program 124 may be configured as one application program that consistently controls the processing from learning to using the model.

[0043] [Area Recognition] FIG. 3 is a diagram showing an example of a screen displaying the recognition result by the recognition unit 205. In the screen 300 shown in FIG. 3, a frame representing the recognized area is superimposed on an image captured in a farm field. The screen 300 includes a plurality of frames, and a numerical value (for example, "20.76", "30.45", etc.) representing the size of the frame is added to each of the plurality of frames. The number of tomato fruits surrounded by the frame may be one, two, or three. That is, the recognition unit 205 may recognize an area surrounding one fruit, an area surrounding two fruits, and an area surrounding three fruits, each of which may be distinguished from one another. The types of areas recognized by the recognition unit 205 are not limited to these three types. The types of areas recognized may be two or less, or four or more. In addition, the number of fruits included in the recognized area is not limited to one to three, and may be four or more.

[0044] [Flow of information processing device] <Main routine> Fig. 4 is a flowchart showing an example of a main routine executed by the information processing device 100. In the information processing device 100, the control device 101 may execute a given program to implement the processing shown in Fig. 4.

[0045] 4, in step S100, information processing device 100 determines whether or not an instruction to learn a prediction model has been input. A user can input the instruction to learn to information processing device 100 by operating input device 108, for example.

[0046] If information processing device 100 determines that an instruction to learn a prediction model has been input (YES in step S100), it proceeds to step S200, and if not (NO in step S100), it proceeds to step S300.

[0047] In step S200, the information processing device 100 executes learning of a prediction model. The process content of step S200 will be described later with reference to Fig. 5. After that, the information processing device 100 advances the control to step S300.

[0048] In step S300, the information processing device 100 determines whether or not a prediction instruction (of the harvest amount of tomatoes) has been input. The user can input a prediction instruction to the information processing device 100 by operating the input device 108, for example.

[0049] If information processing device 100 determines that a prediction instruction has been input (YES in step S300), it proceeds to step S400, and if not (NO in step S300), it returns control to step S100.

[0050] In step S400, the information processing device 100 executes prediction using a prediction model. The process content of step S400 will be described later with reference to Fig. 6. After that, the information processing device 100 returns control to step S100.

[0051] <Learning process> FIG. 5 is a flowchart of a subroutine of step S200 in FIG.

[0052] 5, in step S202, information processing device 100 reads learning data 210.

[0053] In step S204, the information processing device 100 extracts actual measurement values ​​from the learning data 210. The control in step S204 may be realized as a function of the information acquisition unit 201.

[0054] In step S206, the information processing device 100 executes image recognition on the images (one or more frame images constituting the video) included in the learning data 210. In the image recognition, an area surrounding the tomato is recognized for each number of fruits. The control of step S206 may be realized as a function of the recognition unit 205.

[0055] In step S208, the information processing device 100 applies the recognition result in step S206 to equation (2), and further applies the value x derived according to equation (2) to equation (1), thereby deriving the above-mentioned estimated value for the tomato harvest yield.

[0056] In step S210, the information processing device 100 executes machine learning of the prediction model using the actual measured value acquired in step S204 and the estimated value acquired in step S208, updates a parameter in the prediction model (for example, α in formula (1)) as a result of the machine learning, and stores the updated parameter in the parameter storage unit 203. After that, the information processing device 100 returns control to step S202 until the machine learning has completed a predetermined number of learning times. When the above-mentioned number of learning times is completed, the information processing device 100 advances control to step S300 in FIG. 4.

[0057] The process described above with reference to Fig. 5 is merely an example of a procedure for learning a prediction model. As long as the prediction model learning process is performed using the above-mentioned actual measurement values ​​and estimated values, the specific procedure is not limited to that shown in Fig. 5.

[0058] <Prediction processing> FIG. 6 is a flowchart of a subroutine of step S400 in FIG.

[0059] With reference to FIG. 6, in step S402, the information processing device 100 reads the video 220 to be predicted.

[0060] In step S404, the information processing device 100 executes image recognition on one or more frame images constituting the moving image loaded in step S402. In the image recognition in step S404, similar to the image recognition in step S206, an area surrounding the tomato is recognized for each number of fruits. The control of step S404 may be realized as a function of the recognition unit 205.

[0061] In step S406, the information processing device 100 applies the recognition result in step S404 to equation (2), and further applies the value x derived according to equation (2) to equation (1), thereby deriving the above-mentioned predicted value for the tomato harvest yield.

[0062] In step S408, the information processing device 100 outputs the derived predicted value, for example, by displaying it on the display 106. After that, the information processing device 100 returns control to step S100 in FIG.

[0063] In step S408, the information processing device 100 may further display on the display 106 a button for playing the video 220 to be predicted together with the recognition result in step S404. When an operation such as a click is performed on the button, the information processing device 100 may play the video 220 to be predicted together with the recognition result. The recognition result may be a frame superimposed on the image as described with reference to FIG. 3, and may be accompanied by a numerical value indicating the size of the frame. Furthermore, the displayed frames may be displayed in different modes according to the number of fruits that each frame surrounds. For example, a frame surrounding one fruit may be displayed in pink, a frame surrounding two fruits in orange, and a frame surrounding three fruits in yellow-green.

[0064] <Summary> In the present embodiment described above, the information processing device 100 acquires an actual measurement value of the number of fruits of the crop in the range corresponding to the learning image from the label of the learning image (step S204). Meanwhile, the information processing device 100 recognizes an area surrounding each number of fruits of the crop from the learning image (step S206). Then, the information processing device 100 uses the actual measurement value to derive an estimate of the number of fruits of the crop in the range corresponding to the learning image from the number of recognized areas (A, B, C, etc. described above) (step S208). The information processing device 100 uses the actual measurement value and the estimate value to perform machine learning of a prediction model for predicting the number of fruits of the crop (step S210).

[0065] Then, the information processing device 100 derives a predicted value of the number of crop fruits within the range corresponding to the prediction image by applying the number of areas for each number of crop fruits recognized from the prediction image (the video 220 to be predicted) to the prediction model (step S406). The information processing device 100 outputs the derived predicted value by displaying it on the display 106 or the like (step S408).

[0066] FIG. 7 is a diagram for explaining the accuracy of prediction by the information processing device 100. Graph G10 in FIG. 7 shows the predicted and actual yield values ​​calculated every 7 days for a certain field from April 2020 ("2020-04") to October 2020 ("2020-10"). The predicted values ​​are shown by a solid line (line L11), and the actual values ​​are shown by a dashed and dotted line (L12). Note that the predicted and actual values ​​shown in FIG. 7 are each shown as weights obtained by converting the number of predicted values ​​derived by the information processing device 100 and the number of actual values ​​acquired by the information processing device 100 using weights per unit number.

[0067] As can be seen from Figure 7, the predicted values ​​were relatively significantly higher than the actual measured values ​​in the second week of April 2020 and the second week of August 2020, and were relatively significantly lower than the actual measured values ​​in the first week of August 2020. However, the overall error showed a favorable result of approximately MAPE (Mean Absolute Percentage Error) = 38.27.

[0068] [Variation (1)] A first modification of the above embodiment will be described.

[0069] <Function block> Fig. 8 is a diagram showing a first modified example of the functional blocks of the information processing device 100. In the example of Fig. 8, compared to the example of Fig. 2, the information processing device 100 further functions as an environmental data storage unit 208. The environmental data storage unit 208 stores environmental data of the farm field.

[0070] An example of the environmental data is a value related to the amount of solar radiation, for example, the amount of global solar radiation. The information processing device 100 may obtain the amount of global solar radiation for the day and place (prefecture, city, town, village, etc.) corresponding to each of the learning data 210 and the video 220 to be predicted, for example, by accessing the website of the Japan Meteorological Agency. The environmental data may be an index representing temperature, wind speed, saturation deficit, and / or carbon dioxide concentration instead of or in addition to the amount of global solar radiation. In addition, when the agricultural crops are grown hydroponically, the environmental data may include an index representing the "amount of liquid supply / drainage" related to hydroponic cultivation, or an index representing the "leaf length and stem thickness" that are plant observation results.

[0071] Other examples of the environmental data may be temperature, maximum temperature, minimum temperature, the difference between the maximum and minimum temperatures, or average humidity.

[0072] An example of a prediction model used in this modification is shown below as equation (1-1). f(x)=α1x1+α2x2+α3x3+… …(1-1) x1 in equation (1-1), like x shown as equation (2) in the above explanation, is a function of the number (A, B, C, etc.) recognized for each of two or more types of areas, and is more specifically expressed by the following equation (2-1).

[0073] x1 = pA + qB + rC … (2-1) Each term from the second term on the right side of the formula (1-1) onward is formed by the product of a variable (x2, x3, etc.) representing environmental data and a coefficient (for example, α2, α3, etc.) assigned to the variable. The coefficient may be updated in machine learning. The updated coefficient (parameter) is stored in the parameter storage unit 203.

[0074] In the prediction model expressed by formula (1-1), the number of terms from the second item onwards is determined by the number of types of environmental data used. For example, when one type of variable is used as environmental data, the prediction model is expressed by the following formula (1-1-1), where the right-hand side includes two terms and the number of terms from the second item onwards is "1".

[0075] f(x) = α1x1 + α2x2 … (1-1-1) Furthermore, when three types of variables are used as environmental data, the predictive model is expressed by the following equation (1-1-2), where the right-hand side contains four terms and the number of terms from the second item onwards is "3".

[0076] f(x)=α1x1+α2x2+α3x3+α4x4…(1-1-2) <Processing flow> In the modified example (1), the main routine (FIG. 4) is unchanged, but changes are made to the learning process (FIG. 5) and the prediction process (FIG. 6). The following describes each process in the modified example (1), focusing on the changes.

[0077] <Process flow: Learning process> Fig. 9 is a flowchart of a subroutine of the learning process in the modified example (1). The process in Fig. 9 further includes control of step S207, as compared with the process in Fig. 5. In step S207, the information processing device 100 acquires environmental data of a location corresponding to the learning data 210 from the environmental data storage unit 208.

[0078] Then, in step S208, the information processing device 100 applies the environmental data acquired in step S207 together with the area recognition result (step S206) to the prediction model described with reference to equation (1-1) etc., to derive an estimated value.

[0079] That is, in the process of FIG. 9, environmental data of the location to which the learning data 210 corresponds is used as a variable of the prediction model in deriving an estimated value.

[0080] <Process flow: Prediction process> Fig. 10 is a flowchart of a subroutine of the prediction process in the modified example (1). Compared with the process in Fig. 6, the process in Fig. 10 further includes control of step S405. In step S405, the information processing device 100 acquires environmental data of a location corresponding to the video 220 to be predicted from the environmental data storage unit 208. The location corresponding to the video 220 to be predicted and the location corresponding to the learning data 210 may be the same or different.

[0081] Then, in step S406, the information processing device 100 applies the environmental data acquired in step S405 together with the area recognition result (step S404) to the prediction model described with reference to equation (1-1) etc., to derive a predicted value.

[0082] That is, in the process of FIG. 10, environmental data of the location to which the learning data 210 corresponds is used as a variable of the prediction model in deriving a predicted value.

[0083] <Summary> In the variant example (1) described above, the prediction model includes terms corresponding to environmental data, thereby allowing the influence of the field environment to be taken into account in the learning process of the prediction model and in the prediction of yield using the prediction model.

[0084] Fig. 11 is a diagram for explaining the accuracy of prediction by the information processing device 100. Graph G20 in Fig. 11 corresponds to the result when the above-mentioned formula (1-1-1) is adopted as the prediction model and the amount of global solar radiation is used as the variable (x2) of the environmental data.

[0085] Graph G20, like graph G10 in FIG. 7, shows the predicted and actual yield values ​​calculated every 7 days from April 2020 ("2020-04") to October 2020 ("2020-10"). The predicted values ​​are shown by a solid line (line L21), and the actual values ​​are shown by a dashed line (L22). The predicted and actual values ​​are each shown as a weight converted from the number of predicted and actual values ​​using the weight per unit number.

[0086] As can be seen from Figure 11, although the predicted values ​​were relatively higher than the actual measured values ​​in the second week of September 2020, the overall error showed favorable results of about MAPE (Mean Absolute Percentage Error) = 15.17. Note that the MAPE value for the example in Figure 11 is lower than the MAPE value (about 38.27) described above for the example in Figure 7, and this provides the insight that prediction accuracy can be improved by including a term of environmental data in the prediction model.

[0087] [Variation (2)] A second modification of the above embodiment will be described.

[0088] <Function block> FIG. 12 is a diagram showing a second modified example of the functional blocks of the information processing device 100. In the example of FIG. 12, compared to the example of FIG. 2, a first recognition unit 2051 and a second recognition unit 2052 are included instead of the recognition unit 205. The first recognition unit 2051 recognizes the fruits of the crop one by one in the image. The second recognition unit 2052 recognizes the area surrounding the fruits of the crop by number, similar to the recognition unit 205. Each of the first recognition unit 2051 and the second recognition unit 2052 may be realized by the control device 101 executing a program according to an algorithm for object recognition such as YOLO.

[0089] Fig. 13 is a diagram showing an example of a screen displaying the recognition result by the first recognition unit 2051. In the screen 1300 shown in Fig. 13, a frame representing the recognized fruit is superimposed on an image captured in a farm field. The screen 1300 includes a plurality of frames, and each of the plurality of frames is accompanied by a numerical value (for example, "0.48", "0.51", etc.) representing the size of the frame.

[0090] Returning to FIG. 12, the prediction model used by number deriving section 206 to derive the estimated value and predicted value of the actual number is expressed as the following equation (3).

[0091] f(x)=α1x one +α2x multi …(3) In formula (3), x one is a variable calculated as the number of fruits of the agricultural product using the recognition result of the first recognition unit 2051. For example, when the video included in the learning data 210 includes multiple frame images, x one is the total number of objects recognized in each of the plurality of frame images. In addition, when the video 220 to be predicted includes a plurality of frame images, x one is the total number of fruits recognized in each of the plurality of frame images.

[0092] x multi can be calculated according to the following formula (2A) in the same way that x is calculated according to formula (2) above.

[0093] x multi =pA+qB+rC+… …(2A) The variables A, B, C, etc. and the constants p, q, r, etc. in formula (2A) are the same variables and constants as those described for formula (2), respectively.

[0094] The example in Figure 12 differs from the example in Figure 2 in that the recognition results of recognizing each individual fruit of a crop are used as variables to be used in the predictive model. However, in other respects, i.e., the estimated values ​​and predicted values, can be used in the same way as the example in Figure 2, so duplicate explanations will not be repeated here.

[0095] <Processing flow> In the modified example (2), the main routine (FIG. 4) is unchanged, but changes are made to the learning process (FIG. 5) and the prediction process (FIG. 6). The following describes each process in the modified example (2), focusing on the changes.

[0096] <Process flow: Learning process> 14 is a flowchart of a subroutine of the learning process in the modified example (2). The process in FIG. 14 further includes control of step S205, as compared with the process in FIG. 5. In step S205, the information processing device 100 executes image recognition by the first recognition unit 2051 for images included in the learning data 210 (each of one or more frame images constituting a moving image). That is, the first recognition unit 2051 executes image recognition for searching for an area including one fruit in the image, and outputs the result to the number derivation unit 206.

[0097] In step S206, the information processing device 100 performs image recognition by the second recognition unit 2052 in the same manner as the recognition unit 205 in Fig. 2 recognizes an area for each number of fruits. The second recognition unit 2052 performs image recognition by searching for an area surrounding each number of fruits in the image, and outputs the result to the number derivation unit 206.

[0098] In step S208, the information processing apparatus 100 applies the recognition result from the first recognition unit 2051 and the recognition result from the second recognition unit 2052 to the prediction model (Equation (3)) to derive an estimated value.

[0099] <Process flow: Prediction process> 15 is a flowchart of a subroutine of the prediction process in the modified example (2). The process in FIG. 15 further includes control of step S403, as compared with the process in FIG. 6. In step S403, the information processing device 100 executes image recognition by the first recognition unit 2051 for images included in the video 220 to be predicted (each of one or more frame images constituting the video). That is, the first recognition unit 2051 executes image recognition for searching for an area including one fruit in the image, and outputs the result to the number derivation unit 206.

[0100] In step S404, the information processing device 100 performs image recognition by the second recognition unit 2052 in the same manner as the recognition unit 205 in Fig. 2 recognizes an area for each number of fruits. The second recognition unit 2052 performs image recognition by searching for an area surrounding each number of fruits in the image, and outputs the result to the number derivation unit 206.

[0101] In step S406, the information processing apparatus 100 applies the recognition result from the first recognition unit 2051 and the recognition result from the second recognition unit 2052 to the prediction model (Equation (3)) to derive a predicted value.

[0102] <Summary> In the above-described modified example (2), the prediction model uses the results of two types of image recognition: image recognition that recognizes an area for each fruit (first recognition unit 2051), and image recognition that recognizes an area for each number of fruits (second recognition unit 2052).

[0103] Fig. 16 is a diagram for explaining the accuracy of prediction by the information processing device 100. Graph G30 in Fig. 16 corresponds to the result when the above-mentioned formula (3) is adopted as the prediction model.

[0104] Graph G30, like graph G10 in FIG. 7, shows the predicted and actual yield values ​​calculated every 7 days from April 2020 ("2020-04") to October 2020 ("2020-10"). The predicted values ​​are shown by a solid line (line L31), and the actual values ​​are shown by a dashed line (L32). The predicted and actual values ​​are each shown as a weight converted from the number of predicted and actual values ​​using the weight per unit number.

[0105] The overall error of the predicted values ​​in Fig. 16 showed a favorable result of about MAPE (Mean Absolute Percentage Error) = 26.71. Note that the MAPE value for the example in Fig. 16 is lower than the MAPE value (about 38.27) described above for the example in Fig. 7. This indicates that the prediction model using the results of the above-mentioned two types of image recognition can improve prediction accuracy more than when using the results of only one type of image recognition (only the second recognition unit 2052). [Explanation of symbols]

[0106] 100 information processing device, 101 control device, 122 learning program, 124 prediction program, 201 information acquisition unit, 202 learning unit, 203 parameter storage unit, 204 prediction unit, 205 recognition unit, 206 number derivation unit, 207 output unit, 208 environmental data storage unit, 300, 1300 screen, 2051 first recognition unit, 2052 second recognition unit.

Claims

1. an acquisition means for acquiring an actual measurement value of the number of fruits of a crop that has actually been harvested a predetermined period after the capture of the learning image in a range corresponding to the learning image; A recognition means for recognizing an area surrounding each of the fruits of the agricultural crop for each number of the fruits from the learning image; a learning means for executing machine learning of a prediction model for predicting the number of fruits of the crop that will be harvested after the specified period from the time the image was captured, using an estimated value of the number of fruits of the crop that will be harvested after the specified period from the time the image was captured, the estimated value being derived using the actual measurement value and the number of areas recognized by the recognition means; and a prediction means for deriving a predicted value for the number of crop fruits to be harvested a specified period of time from the time the prediction image was captured within a range corresponding to the prediction image by applying the number of areas for each number of crop fruits recognized by the recognition means from the prediction image to the prediction model.

2. The learning means further uses values ​​related to the environment of a field of the crop in the machine learning of the prediction model, The information processing apparatus according to claim 1 , wherein the prediction means further applies a value related to an environment of a field of the agricultural crop to the prediction model to derive the predicted value.

3. The information processing device according to claim 2 , wherein the environmental value includes a value related to an amount of solar radiation in a field of agricultural crops.

4. The information processing device according to any one of claims 1 to 3, wherein the machine learning utilizes information indicating an area for each number of fruits of the crop that is added to the learning image.

5. 5. The information processing apparatus according to claim 1, further comprising a display means for displaying an area surrounding each of the fruits of the agricultural crop recognized from the prediction image for each number of the fruits.

6. The method further includes a second recognition means for recognizing an area surrounding each of the fruits of the agricultural crop from the learning image, the learning means further uses a second estimate of the number of fruits of the crop derived from the number of regions recognized by the second recognition means in the machine learning of the prediction model; The information processing device according to any one of claims 1 to 5, wherein the prediction means derives the predicted value by further applying the number of areas surrounding each of the crop fruits recognized from the prediction image by the second recognition means to the prediction model.

7. 1. A computer-implemented method comprising: acquiring an actual measurement value of the number of fruits of a crop that have actually been harvested a predetermined period of time after the capture of the learning image in a range corresponding to the learning image; Recognizing an area surrounding each of the fruits of the agricultural crop for each number of the fruits from the learning image; a step of deriving an estimate of the number of fruits of the crop that will be harvested after the predetermined period from the capture of the learning image in a range corresponding to the learning image, based on the number of areas recognized in the recognizing step; A step of executing machine learning of a prediction model for predicting the number of fruits of the agricultural crop to be harvested after the predetermined period from the time of capturing the image, using the actual measured value and the estimated value; and applying the number of areas for each number of crop fruits recognized from the prediction image to the prediction model to derive a predicted value for the number of crop fruits to be harvested a specified period from the time the prediction image was captured within a range corresponding to the prediction image.

8. The step of performing machine learning further uses values ​​related to the environment of a field of the crop in the machine learning of the prediction model; The information processing method according to claim 7 , wherein the step of deriving the predicted value further applies a value related to an environment of a field of the agricultural crop to the prediction model to derive the predicted value.

9. The information processing method according to claim 8 , wherein the environmental values ​​include values ​​related to an amount of solar radiation in a field of agricultural crops.

10. The information processing method according to any one of claims 7 to 9, wherein the machine learning utilizes information representing an area for each number of fruits of the crop that is added to the learning image.

11. The information processing method according to any one of claims 7 to 10, further comprising a step of displaying an area surrounding each of the fruits of the crop recognized from the prediction image for each number of the fruits.

12. The method further comprises a step of recognizing an area surrounding each of the fruits of the agricultural crop from the learning image, The step of performing machine learning includes: deriving a second estimate of the number of crop fruits using a number of regions surrounding each of the crop fruits recognized from the training images; further utilizing the second estimate in machine learning of the predictive model; The information processing method according to any one of claims 7 to 11, wherein the step of deriving the predicted value further applies the number of areas surrounding each of the crop fruits recognized from the prediction image to the prediction model, thereby deriving the predicted value.

13. A program that, when executed by a computer, causes the computer to execute the information processing method according to any one of claims 7 to 12.

Images