Information processing device, information processing method, and program

Abstract

To predict the yield of a crop in consideration of a state of an entire farm field without depending on the ability and skill of an inspector.SOLUTION: An information processing apparatus 100 comprises: an image acquiring unit 110 which acquires a plurality of aerial images obtained by capturing a crop in a farm field F from different positions above the farm field with a camera 12 mounted on an unmanned aircraft 10 at a plurality of times, via a relay device 20; a numerical surface layer model generation unit which generates a plurality of three-dimensional models representing a surface layer of the farm field from the plurality of aerial images captured at the plurality of times; an ortho-mosaic image generation unit which generates an ortho-mosaic image; and a yield prediction unit 140 which predicts the crop yield in the farm field on the basis of the plurality of three-dimensional models and the ortho-mosaic image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, a technique for predicting the yield of crops grown in a field is known. For example, Patent Document 1 discloses a technique for predicting the yield of crops based on data including the characteristics of the land to be predicted and the environment of the land.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Generally, in the prior art, for a field to be predicted, data is acquired using a ground survey such as a sampling survey, and the yield is predicted based on the acquired data. That is, the situation of the entire field is not always considered, and the prediction accuracy may be low. Furthermore, such a ground survey depends on the ability and proficiency of the investigator, such as the selection of an appropriate sampling survey position, and the prediction accuracy may not be stable.

[0005] The present invention has been made in consideration of such circumstances, and an object thereof is to provide an information processing apparatus, an information processing method, and a program that can predict the yield of crops considering the situation of the entire field without depending on the ability and proficiency of the investigator.

Means for Solving the Problems

[0006] An information processing device according to one aspect of the present invention includes: an acquisition unit that acquires a plurality of aerial images of crops in a field taken from different positions above the field by a camera mounted on an unmanned aerial vehicle at multiple time points; a generation unit that generates a plurality of three-dimensional models representing the surface of the field and an orthomosaic image representing the entire field from the plurality of aerial images taken at multiple time points; and a prediction unit that predicts the yield of crops in the field based on the plurality of three-dimensional models and the orthomosaic image.

[0007] The aforementioned multiple periods include a first period, a second period after the first period, and a third period after the second period. The generation unit generates a first three-dimensional model from the multiple aerial images captured in the first period, generates an orthomosaic image from the multiple aerial images captured in the second period, and generates a second three-dimensional model from the multiple aerial images captured in the third period. The prediction unit may predict the yield of crops in the field based on the first three-dimensional model, the orthomosaic image, and the second three-dimensional model.

[0008] The prediction unit may obtain the number of plants of the crop by performing image processing on the orthomosaic image, obtain the plant height and lodging area of ​​the crop by performing a vertical difference calculation between the plurality of three-dimensional models, and predict the crop yield of the field based on the number of plants, the plant height and the lodging area.

[0009] The prediction unit may, as part of the image processing, generate a vegetation index image from the orthomosaic image, perform a binarization process to classify each pixel of the vegetation index image into vegetated pixels and non-vegetated pixels, and obtain the number of plants of the crop by identifying spatially consecutive vegetated pixels as a single plant.

[0010] The prediction unit may acquire pixels corresponding to the plant height as the lodged region if the plant height is below a threshold. [Effects of the Invention]

[0011] According to the present invention, crop yields can be predicted by considering the overall conditions of the field, without depending on the abilities or skill level of the researcher. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of the operating environment and configuration of the information processing device 100. [Figure 2] This figure shows an example of multiple timings at which the image acquisition unit 110 acquires aerial images. [Figure 3] This diagram illustrates how the yield prediction unit 140 obtains the number of feed corn plants in the second period. [Figure 4] This figure illustrates the overview of the image processing performed by the yield prediction unit 140, which removes groups of vegetation pixels VP with low spatial continuity as noise. [Figure 5] This diagram illustrates how the yield prediction unit 140 obtains the plant height of feed corn in the third stage. [Figure 6] This diagram illustrates how the yield prediction unit 140 obtains the lodging area of ​​feed corn in the third stage. [Figure 7] This flowchart shows an example of how the yield prediction unit 140 obtains the number of feed corn plants in the second period. [Figure 8] This flowchart shows an example of how the yield prediction unit 140 obtains the plant height and lodging area of ​​feed corn in the third stage. [Figure 9] This flowchart shows an example of how the yield prediction unit 140 predicts the yield of feed corn. [Modes for carrying out the invention]

[0013] [overview] Hereinafter, referring to the drawings, the information processing apparatus 100 according to an embodiment of the present invention will be described. FIG. 1 is a diagram showing an example of the usage environment and configuration of the information processing apparatus 100. The information processing apparatus 100 operates in cooperation with, for example, the aircraft 10, the relay apparatus 20, and the terminal apparatus 30.

[0014] The aircraft 10 is, as an imaging device, for example, a drone (unmanned aerial vehicle) to which a camera 12 is attached. In the present embodiment, it is assumed that the camera 12 is an RGB camera or a multispectral camera. The aircraft 10 flies over the field F where the forage corn grows, and images the field F from different positions in the air with the camera 12. More specifically, the aircraft 10 flies over the field F so that a plurality of captured aerial images overlap with each other in the traveling direction and the lateral direction of the aircraft 10 and include the entire field F.

[0015] The aircraft 10 stores a plurality of captured aerial images captured by the camera 12 together with the camera parameters and position information such as GPS of the captured aerial images in a memory card mounted on the aircraft 10 or transmits them to the relay apparatus 20. The aircraft 10 is an example of an "unmanned aerial vehicle", and the forage corn is an example of a "crop". Hereinafter, in the present embodiment, an example in which the camera 12 of the aircraft 10 images the field F where the forage corn grows will be described, but the present invention is not limited to such an example. For example, the aircraft 10 may image the field F where crops with a plant height of a predetermined value or more, such as forage rice or sugarcane, grow.

[0016] The relay apparatus 20 is a terminal apparatus such as a tablet terminal equipped with an application for operating the aircraft 10 and receiving and displaying the captured aerial images transmitted from the aircraft 10. The relay apparatus 20 displays the captured aerial image as an RGB image or transmits the received captured aerial image to the information processing apparatus 100 via the network NW. The network NW is an arbitrary network such as, for example, a LAN, a WAN, or an Internet line, and may be wired or wireless.

[0017] The terminal device 30 is a computer device such as a personal computer, a smartphone, or a tablet terminal. The terminal device 30 communicates with the information processing device 100 via the network NW and displays the information received from the information providing unit 150 of the information processing device 100 described later.

[0018] The information processing device 100 is a server device such as a web server. The information processing device 100 receives a plurality of aerial images captured at a plurality of times from the relay device 20, and generates a plurality of three-dimensional models representing the surface layer of the farm field F and an orthomosaic image representing the entire farm field F from the received plurality of aerial images using the method described later. The information processing device 100 predicts the crop yield of the farm field F based on the generated plurality of three-dimensional models and orthomosaic images.

[0019] [Functional Configuration] The functions of the information processing device 100 will be described in more detail below. The information processing device 100 includes, for example, an image acquisition unit 110, a numerical surface model generation unit 120, an orthomosaic image generation unit 130, a yield prediction unit 140, and an information provision unit 150. Each of the image acquisition unit 110, the numerical surface model generation unit 120, the orthomosaic image generation unit 130, the yield prediction unit 140, and the information provision unit 150 is realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. The combination of the numerical surface model generation unit 120 and the orthomosaic image generation unit 130 is an example of a "generation unit".

[0020] The image acquisition unit 110 acquires multiple aerial images of field F captured by camera 12 at multiple time points from the relay device 20. Figure 2 shows an example of multiple time points at which the image acquisition unit 110 acquires aerial images. First, the image acquisition unit 110 acquires multiple aerial images of field F when it is bare ground after sowing feed corn. The time when field F is bare ground after sowing feed corn is an example of the "first time point". The multiple aerial images captured in the first time point are used for processing by the numerical surface model generation unit 120, which will be described later.

[0021] Next, the image acquisition unit 110 acquires multiple aerial images of field F taken at the 3-leaf or 4-leaf stage of feed corn. The 3-leaf or 4-leaf stage of feed corn is an example of the "second stage," which is later than the first stage. The multiple aerial images taken in the second stage are used for processing by the orthomosaic image generation unit 130, which will be described later.

[0022] Next, the image acquisition unit 110 acquires multiple aerial images of field F taken at a predetermined time before the harvest of feed corn, which is after the 3-leaf or 4-leaf stage of the feed corn. The predetermined time before the harvest of feed corn is an example of the "third period". The multiple aerial images taken at the third period are used for processing by the numerical surface model generation unit 120, which will be described later.

[0023] The digital surface model generation unit 120 applies known photogrammetry processing to multiple aerial images captured in the first period to generate a three-dimensional digital surface model (DSM) representing the surface of field F. As described above, each aerial image is associated with location information such as GPS, so the generated digital surface model retains location information. The digital surface model generated from multiple aerial images captured in the first period is an example of the "first three-dimensional model".

[0024] The numerical surface model generation unit 120 similarly applies known photogrammetry processing to multiple aerial images captured in the third period to generate a three-dimensional numerical surface model representing the surface of field F. The numerical surface model generated from multiple aerial images captured in the third period is an example of the "second three-dimensional model". Hereinafter, the numerical surface model generated from multiple aerial images captured in the first period will be referred to as the "first numerical surface model", and the numerical surface model generated from multiple aerial images captured in the third period will be referred to as the "second numerical surface model".

[0025] The orthomosaic image generation unit 130 applies known photogrammetry processing to multiple aerial images captured in the second period to generate an orthomosaic image representing the entire field F. As described above, the multiple aerial images of field F overlap with each other in terms of the direction of travel and lateral direction of the aircraft 10, and include the entire field F. Therefore, the orthomosaic image generation unit 130 can generate an orthomosaic image that corrects distortions caused by the tilt of the camera 12 and the shape of the captured objects with respect to the entire field F. Furthermore, similar to the numerical surface model, each aerial image is associated with location information such as GPS, so the generated orthomosaic image retains location information.

[0026] The yield prediction unit 140 predicts the yield of feed corn in field F based on the first and second numerical surface models generated by the numerical surface model generation unit 120 and the orthomosaic image generated by the orthomosaic image generation unit 130. The details of the prediction process performed by the yield prediction unit 140 will be described below.

[0027] [Prediction Procedure] The yield prediction unit 140 first obtains the number of feed corn plants in the second period based on the orthomosaic image generated by the orthomosaic image generation unit 130. Figure 3 is a diagram illustrating how the yield prediction unit 140 obtains the number of feed corn plants in the second period. Figure 3(a) shows an RGB image, which is an orthomosaic image generated by the orthomosaic image generation unit 130.

[0028] The yield prediction unit 140 first generates a vegetation index (VARI: Visible Atmospherically Resistant Index) image using the RGB color bands in the orthomosaic image. Next, the yield prediction unit 140 performs a binarization process to classify each pixel of the generated vegetation index image into vegetated pixels and non-vegetated pixels, thereby obtaining a binarized vegetation index image as shown in Figure 3(b). As an example, the white dots in Figure 3(b) represent vegetated pixels VP. The yield prediction unit 140 performs image processing to retain groups of vegetated pixels VP with high spatial continuity from the vegetated pixels VP obtained by the binarization process, while removing groups of vegetated pixels VP with low spatial continuity as noise. The image processing by which the yield prediction unit 140 removes groups of vegetated pixels VP with low spatial continuity as noise will be described below.

[0029] Figure 4 illustrates the overview of the image processing performed by the yield prediction unit 140 to remove groups of vegetation pixels VP with low spatial continuity as noise. In Figure 4, white pixels represent vegetation pixels, and black pixels represent non-vegetation pixels. For example, the yield prediction unit 140 determines whether the number of vegetation pixels adjacent to each pixel in the binarized vegetation index image is greater than or equal to a threshold (e.g., 5). In the case of the pixel distribution pattern shown in Figure 4(a), the number of adjacent vegetation pixels exceeds the threshold for any pixel in the sequence. Therefore, the yield prediction unit 140 converts the two non-vegetation pixels shown in Figure 4(a) into vegetation pixels and retains the whole as vegetation pixels. On the other hand, in the case of the pixel distribution pattern shown in Figure 4(b), the number of adjacent vegetation pixels is below the threshold for any pixel in the sequence. Therefore, the yield prediction unit 140 converts the two vegetation pixels shown in Figure 4(b) into non-vegetation pixels and removes them as noise. On the other hand, in the case of the pixel distribution pattern shown in Figure 4(c), among the non-vegetated pixels, only the non-vegetated pixels in the upper right corner have a number of adjacent vegetated pixels that exceeds the threshold. Therefore, the non-vegetated pixels in the upper right corner are converted to vegetated pixels to obtain a group of consecutive vegetated pixels.

[0030] The yield prediction unit 140 then performs a raster / vector conversion on the image obtained by performing the image processing shown in Figure 4, obtaining a group of spatially continuous vegetation pixels as a single plant (i.e., a plant). The yield prediction unit 140 generates a point at the center of the vector (polygon) obtained as a single plant, with each point indicating the location of the plant. Figure 3(c) shows the image obtained by the raster / vector conversion, where the code HP represents the generated plant point. The yield prediction unit 140 can obtain the number of plants by counting the plant points HP. As mentioned above, since the orthomosaic image has positional information, the plant points HP also have positional information.

[0031] The yield prediction unit 140 then obtains the plant height of feed corn in the third period based on the first and second numerical surface models generated by the numerical surface model generation unit 120. Figure 5 is a diagram illustrating how the yield prediction unit 140 obtains the plant height of feed corn in the third period. The yield prediction unit 140 performs a vertical difference calculation between the first and second numerical surface models using common GIS software such as ESRI ArcGIS or QGIS. Figure 5 shows the plant height diagram obtained as a result of the yield prediction unit 140 performing the vertical difference calculation. As described above, the first numerical surface model represents the ground level when field F is bare after sowing feed corn (first period), and the second numerical surface model represents the height of the feed corn at a predetermined time before harvest (third period). Therefore, the plant height of feed corn in the third period can be obtained by performing a vertical difference calculation.

[0032] As mentioned above, since each of the multiple aerial images is associated with location information, the first and second numerical surface models generated from the multiple aerial images also possess location information. Therefore, the yield prediction unit 140 can easily perform vertical difference calculations by superimposing these first and second numerical surface models on the GIS software.

[0033] The yield prediction unit 140 further extracts pixels from the plant height diagram shown in Figure 5 where the plant height is below a threshold as lodging regions (or regions where pest damage occurs). Figure 6 is a diagram illustrating how the yield prediction unit 140 obtains lodging regions of feed corn in the third stage. Figure 6(a) shows an RGB image of feed corn, and Figure 6(b) shows the lodging regions extracted by the yield prediction unit 140. When the yield prediction unit 140 extracts pixels where the plant height is below a threshold as lodging regions, it performs a raster / vector conversion on the extracted pixels to extract a single lodging region. In Figure 6(b), region FA represents the single lodging region that is finally extracted.

[0034] As mentioned above, since the first and second numerical surface models contain location information, the plant height maps generated from them also contain location information. Therefore, the plant height maps can be used to quantify the total area of ​​field F, the area of ​​lodged plants, and the overall ratio.

[0035] As described above, the yield prediction unit 140 obtains the number of feed corn plants, plant height, and lodging area, and then predicts the yield of feed corn based on these obtained values. More specifically, the yield prediction unit 140 predicts the yield by inputting the number of plants, plant height, and lodging area into the yield prediction formula, which is: Yield = Sum of estimated plant heights of all plants × Bulk density × Coefficient a. Here, bulk density represents the average weight per unit plant height of the entire above-ground part of the plant (i.e., feed corn) and is given as a predetermined value in advance. Furthermore, coefficient a is a value that includes the effects of the obtained number of plants, plant height, estimation error of the lodging area, variation in bulk density, and loss rate during harvesting.

[0036] The information provision unit 150 transmits the predicted yield to the terminal device 30 via the network NW, and the terminal device 30 displays the received predicted yield on its screen. This allows the manager of field F to understand the predicted yield of feed corn and efficiently formulate a harvesting and processing work plan and a feed supply plan. For example, the manager of field F can efficiently allocate resources such as personnel and machinery for harvesting and processing work based on the predicted yield they have obtained.

[0037] Next, with reference to Figures 7 to 9, the processing flow performed by the information processing device 100 according to this embodiment will be described. Figure 7 is a flowchart showing an example of the flow in which the yield prediction unit 140 acquires the number of feed corn plants in the second period.

[0038] First, the yield prediction unit 140 acquires the orthomosaic image for the second period, which is generated by the orthomosaic image generation unit 130 (step S100). Next, the yield prediction unit 140 generates a vegetation index image from the acquired orthomosaic image (step S102).

[0039] Next, the yield prediction unit 140 performs a binarization process to classify each pixel of the generated vegetation index image into vegetated pixels and non-vegetated pixels (step S104). Next, the yield prediction unit 140 performs image processing to remove noise from the binarized vegetation index image (step S106).

[0040] Next, the yield prediction unit 140 performs a raster / vector conversion on the vegetation index image from which spatially continuous groups of vegetation pixels have been extracted (step S108). Next, the yield prediction unit 140 generates points at the centers of the plant vectors on the image obtained by the raster / vector conversion (step S110). Next, the yield prediction unit 140 counts the number of generated points and obtains them as the number of plants (step S110). This completes the processing of this flowchart.

[0041] Figure 8 is a flowchart showing an example of the process by which the yield prediction unit 140 obtains the plant height and lodging area of ​​feed corn in the third stage.

[0042] First, the yield prediction unit 140 obtains the first and second numerical surface models generated by the numerical surface model generation unit 120 (step S200). Next, the yield prediction unit 140 performs a vertical difference calculation between the first and second numerical surface models to obtain a plant height diagram (step S202). In other words, this obtains the plant height of feed corn in the third period.

[0043] Next, the yield prediction unit 140 extracts pixels from the plant height diagram in which the plant height is below a threshold (step S204). Next, the yield prediction unit 140 performs a raster / vector conversion on the extracted pixels (step S206). The yield prediction unit 140 obtains a group of lodged areas from the image obtained by performing the raster / vector conversion (step S208). This completes the processing of this flowchart.

[0044] Figure 9 is a flowchart showing an example of the process by which the yield prediction unit 140 predicts the yield of feed corn.

[0045] First, the yield prediction unit 140 acquires the number of plants using the method shown in the flowchart in Figure 7, and acquires the plant height and lodging area using the method shown in the flowchart in Figure 8 (step S300). Next, the yield prediction unit 140 predicts the yield of feed corn using the prediction formula described above, based on the acquired number of plants, plant height, and lodging area (step S302). Next, the yield prediction unit 140 presents the predicted yield to the terminal device 30 (step S304). This completes the processing of this flowchart.

[0046] As described above, according to this embodiment, multiple aerial images of the crops in the field are acquired at different positions above the field by a camera mounted on an unmanned aerial vehicle at multiple time points. Multiple 3D models representing the surface of the field and an orthomosaic image representing the entire field are generated from the multiple aerial images taken at multiple time points. The yield of the crops in the field is predicted based on the multiple 3D models and the orthomosaic image. This makes it possible to predict the yield of the crops while considering the conditions of the entire field, without depending on the ability or skill level of the surveyor.

[0047] Furthermore, according to this embodiment, unlike ground surveys such as sampling surveys, a complete enumeration survey is performed, which improves the accuracy of yield prediction. Moreover, according to this embodiment, compared to methods that predict yield using environmental data or satellite images, the accuracy of yield prediction can be improved by using drone images with higher resolution.

[0048] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0049] 10 flying objects 12 cameras 20 Relay device 30 Terminal devices 100 Information Processing Devices 110 Image acquisition unit 120 Numerical Surface Model Generation Unit 130 Orthomosaic Image Generation Unit 140 Yield prediction section 150 Information Provision Department

Claims

1. An acquisition unit that acquires multiple aerial images of crops in a field from different positions above the field using a camera mounted on an unmanned aerial vehicle at multiple time points, A generation unit that generates multiple three-dimensional models representing the surface of the field and an orthomosaic image representing the entire field from the multiple aerial images captured at the multiple times mentioned above, The system includes a prediction unit that predicts the yield of crops in the field based on the plurality of three-dimensional models and the orthomosaic image, The prediction unit obtains the number of plants of the crop by performing image processing on the orthomosaic image, obtains the plant height and lodging area of ​​the crop by performing vertical difference calculations between the plurality of three-dimensional models, and predicts the crop yield of the field based on the number of plants, the plant height and the lodging area. Information processing device.

2. An acquisition unit that acquires multiple aerial images of crops in a field from different positions above the field using a camera mounted on an unmanned aerial vehicle at multiple time points, A generation unit that generates multiple three-dimensional models representing the surface of the field and an orthomosaic image representing the entire field from the multiple aerial images captured at the multiple times mentioned above, The system includes a prediction unit that predicts the yield of crops in the field based on the plurality of three-dimensional models and the orthomosaic image, The aforementioned multiple periods include a first period, a second period following the first period, and a third period following the second period. The generation unit generates a first three-dimensional model from the plurality of aerial images captured in the first period, generates the orthomosaic image from the plurality of aerial images captured in the second period, and generates a second three-dimensional model from the plurality of aerial images captured in the third period. The prediction unit predicts the crop yield of the field based on the first three-dimensional model, the orthomosaic image, and the second three-dimensional model. Information processing device.

3. The prediction unit, as part of the image processing, generates a vegetation index image from the orthomosaic image, performs a binarization process to classify each pixel of the vegetation index image into vegetated pixels and non-vegetated pixels, and obtains the number of plants of the crop by identifying spatially consecutive vegetation pixels as a single plant. The information processing apparatus according to claim 1.

4. The prediction unit acquires pixels corresponding to the plant height as the lodged region when the plant height is below a threshold. The information processing apparatus according to claim 1.

5. Computers At multiple times, multiple aerial images of the crops in the field are acquired from different positions above the field using cameras mounted on unmanned aerial vehicles. From the multiple aerial images captured at the aforementioned multiple time periods, multiple three-dimensional models representing the surface of the field and an orthomosaic image representing the entire field are generated. Based on the aforementioned three-dimensional models and the orthomosaic image, the yield of the crops in the field is predicted. The prediction involves obtaining the number of plants of the crop by applying image processing to the orthomosaic image, obtaining the plant height and lodging area of ​​the crop by performing vertical difference calculations between the multiple three-dimensional models, and predicting the crop yield in the field based on the number of plants, the plant height, and the lodging area. Information processing methods.

6. On the computer, At multiple times, cameras mounted on unmanned aerial vehicles were used to acquire multiple aerial images of the crops in the field from different positions above the field. From the multiple aerial images captured at the aforementioned multiple time periods, multiple three-dimensional models representing the surface of the field and an orthomosaic image representing the entire field are generated. Based on the aforementioned multiple three-dimensional models and the orthomosaic image, the yield of the crops in the field is predicted. The prediction involves obtaining the number of plants of the crop by applying image processing to the orthomosaic image, obtaining the plant height and lodging area of ​​the crop by performing vertical difference calculations between the multiple three-dimensional models, and predicting the crop yield in the field based on the number of plants, the plant height, and the lodging area. program.

Images