Information processing device, information processing method, and program
Projective transformation and row detection in aerial images enhance crop counting accuracy by aligning and scaling images uniformly, addressing inaccuracies in existing methods and providing precise crop density data.
Patent Information
- Application Number
- JP2021572986
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-01-21
- Filing Date
- 2020-11-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-11-27
AI Technical Summary
Existing methods for counting crops from aerial images are inaccurate due to variations in camera angle, altitude, and zoom, leading to significant errors in crop density estimation.
The use of projective transformation to correct image perspectives, combined with row detection and counting algorithms, ensures accurate crop counting by aligning rows parallel and uniform scaling, allowing for precise crop number calculation per unit area.
This approach significantly reduces errors in crop counting by ensuring accurate alignment and scaling of aerial images, providing reliable crop density data for farm management.
Smart Images

Figure 0007586100000001 
Figure 0007586100000002 
Figure 0007586100000003
Abstract
Description
[Technical field]
[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to a technology suitable for generating information regarding crop cultivation. [Background technology]
[0002] In recent years, as farms have become larger, automatic planting of crops has become common. In automatic planting, it is very important to check whether the crops have been planted correctly and whether they are growing as expected. If there are areas where the crops are not planted or growing as expected, replanting may be performed to optimize the final yield.
[0003] On the other hand, with the development of aerial imaging technology using cameras mounted on drones and other flying objects, attempts are being made to simplify and automate the work of manually inspecting farmland on large-scale farms. Detection of crop defects as mentioned above is also becoming automated using such aerial imaging technology. Patent Document 1 discloses a technique for capturing images of farm fields and performing remote sensing of vegetation conditions. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 5162890 Summary of the Invention [Problem to be solved by the invention]
[0005] Incidentally, in order to confirm the cultivation, it is sufficient to count the number of crops from an aerial photograph, but it is difficult to count the crops with high accuracy from an aerial photograph. Furthermore, when counting crops using image processing, counting accuracy can be improved by specifying rows (furrows or planting lines) in the image; however, this requires manual input of the field area, row direction, row position, etc., which can be very time-consuming when processing a large number of images. In addition, when taking images from the air, the camera angle and direction are not constant due to the shaking of the aircraft, and the area of the captured range is not necessarily uniform due to the shaking of the altitude and the zoom state, and the image is not necessarily parallel to the wax. For this reason, if crops are counted from the images and converted to the number per unit area, there may be a large error.
[0006] In view of this, the present disclosure proposes a technique for improving the accuracy of counting the number of crops by image processing of images taken from the air. [Means for solving the problem]
[0007] The information processing device according to the present technology includes a projection transformation unit that performs projection transformation on an image captured from above of a farm field to obtain a projection transformed image, and an output information calculation unit that uses the projection transformed image to count the number of crops and generate crop number information. That is, an image of a farm field is transformed by projective transformation into an image of the farm field as seen directly below from above, and the number of crops appearing on this projective transformed image is counted.
[0008] In the information processing device related to the above-mentioned present technology, the output information calculation unit may set a count area in the projectively transformed image in which the number of crops is to be counted, and calculate the number of crops per unit area based on the number of crops counted within the count area. That is, a certain area of an actual farm field is a pixel range that appears on the projectively transformed image, and the area in which the number of crops is counted is set as a count area. Then, for example, the number of crops in the count area in the projectively transformed image is counted. This number of crops is converted into the number of crops per unit area, such as one acre.
[0009] In the information processing device related to the above-mentioned present technology, the output information calculation unit may set a plurality of count areas as the count areas, each of which corresponds to an area of the same area within a field, obtain a representative value of the number of crops based on the number of crops counted in each count area, and use the representative value to calculate the number of crops per unit area. For example, the average number of crops counted in a plurality of counting areas is set as the representative value.
[0010] In the information processing device related to the above-mentioned present technology, the output information calculation unit may calculate the number of pixels on the projectively transformed image that corresponds to the area to be counted in the field, and set the count area as the image range of the calculated number of pixels. In other words, the count area is set to a range that covers a specific area of the field.
[0011] In the information processing device according to the present technology described above, it is considered that the count area is set as a range including one row, which is a line in which crops are planted. In other words, the count area is set to a range that includes one row.
[0012] In the information processing device related to the above-mentioned present technology, the projection transformation unit determines the attitude of the imaging device at the time of capturing the captured image, and generates a projection transformation image if the tilt is outside of an acceptable range, and the output information calculation unit is thought to count the number of crops using the captured image and generate crop number information if the projection transformation unit does not generate a projection transformation image. That is, a projectively transformed image is generated only when necessary.
[0013] The information processing device related to the above-mentioned technology is thought to include a row determination unit that performs crop detection within an image of a farm field and determines rows, which are lines along which crops are planted within the image, based on the results of the crop detection, and the output information calculation unit counts the number of crops using row information determined by the row determination unit. Crop portions are detected in an image of a field, and rows are determined in the image data based thereon.
[0014] In the information processing device related to the present technology described above, the row determination unit can use a projectively transformed image produced by the projective transformation unit as an image of a farm field, and determine rows, which are lines along which crops are planted, within the projectively transformed image based on the results of the crop detection. For example, a captured image to be processed is subjected to projective transformation, row determination is performed on the projectively transformed image, and then crop counting is performed.
[0015] In the information processing device related to the above-mentioned present technology, the row determination unit may detect row candidates based on the results of the crop detection, calculate the likelihood of each row candidate being a row based on the crop spacing, and determine the row based on the result of the likelihood calculation. By detecting areas that appear to be crops, rows of crops can be estimated, and these are used as row candidates. The likelihood of each row candidate is determined based on the spacing between the crops.
[0016] In the information processing device related to the above-mentioned present technology, the law determination unit may calculate a crop size from the size of a crop-like area located near the top of the law candidate, detect crop candidates on the law candidate from the calculated crop size, calculate the crop spacing on the image from the spacing of the detected crop candidates, determine the likelihood of the law candidate from the determined crop spacing, and perform a process to determine the law after excluding law candidates determined to be erroneous detections using the determined likelihood. By estimating the size of crops in areas that look like crops, the spacing between crops in the image can be determined appropriately. The likelihood of row candidates is determined from such spacing between crops.
[0017] The information processing method according to the present technology performs projective transformation on an image of a farm field captured from above to generate a projectively transformed image, and counts the number of crops using the projectively transformed image to generate crop number information. This improves the accuracy of calculation of crop information in the information processing device. A program according to the present technology causes an information processing device to execute the processing of the above-described method, thereby making it easy to realize a computer device that generates accurate output information without requiring the user to do anything. [Brief description of the drawings]
[0018] [Figure 1] FIG. 1 is an explanatory diagram of a sensing system according to an embodiment of the present technology. [Diagram 2] FIG. 2 is an explanatory diagram of rows and an imaging range in a farm field. [Diagram 3] FIG. 1 is a block diagram of a hardware configuration of an information processing apparatus according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a functional configuration of the information processing apparatus according to the embodiment; [Diagram 5] 4 is a flowchart of a processing example according to the first embodiment. [Figure 6] 11 is a flowchart of a crop number calculation process according to the first embodiment. [Figure 7] FIG. 11 is an explanatory diagram of an image showing crop-likeness according to the embodiment. [Figure 8] FIG. 4 is an explanatory diagram of a law candidate according to the embodiment; [Figure 9] 11 is an explanatory diagram of a straight line projection calculation of a row candidate according to an embodiment; [Figure 10] FIG. 13 is an explanatory diagram of detection of a crop location on a wax according to an embodiment. [Figure 11] 11A and 11B are explanatory diagrams of narrowing down Law candidates according to an embodiment; [Figure 12] FIG. [Figure 13] FIG. 2 is an explanatory diagram of projective transformation according to an embodiment. [Figure 14] FIG. 2 is an explanatory diagram of projective transformation according to an embodiment. [Figure 15] FIG. 4 is an explanatory diagram of coordinates of a projectively transformed image according to the embodiment; [Figure 16] FIG. 2 is an explanatory diagram of a count area according to the first embodiment. [Figure 17] 13 is a flowchart of a processing example according to the second embodiment. [Figure 18] FIG. 11 is an explanatory diagram of a count area according to the second embodiment. [Figure 19] 13 is a flowchart of a crop number calculation process according to a third embodiment. [Figure 20] 13 is a flowchart of a crop number calculation process according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] The embodiments will be described below in the following order. <1. Sensing system configuration> <2. Configuration of information processing device> <3. First embodiment> <4. Second embodiment> <5. Third embodiment> <6. Fourth embodiment> <7. Fifth embodiment> <8. Summary and variations>
[0020] <1. Sensing system configuration> First, a sensing system according to an embodiment will be described. 1 shows an information processing device 1 constituting a sensing system, and an imaging device 220 mounted on a small flying object 200 such as a drone. Also shown is a tractor (seeder) 270 that performs planting.
[0021] The flying object 200 can move above the field 300, for example, by radio control by an operator or by automatic control. An imaging device 220 is set on the flying object 200 so as to capture, for example, an image of a downward direction. When the flying object 200 moves above the farm field 300 along a predetermined route, the imaging device 220 captures still images, for example, periodically.
[0022] It should be noted that, by flying object 200 at a relatively low altitude (for example, at an altitude of about 10 to 20 m), only a portion of farm field 300 appears in one captured image. By capturing still images at short intervals, it is possible to stitch the captured images together to obtain a composite image of the entire field. However, in the case of sensing in this embodiment, this is not necessarily required. For example, it is sufficient if discrete areas in the field 300 are captured as sensing samples to obtain multiple image data. Hereinafter, image data for each captured still image will also be referred to as a "sample."
[0023] The imaging device 220 mounted on the flying object 200 may be a visible light image sensor (an image sensor that captures visible light of R (red), G (green), and B (blue)), a camera for capturing NIR (Near Infra Red) images, a multi-spectrum camera that captures images in multiple wavelength bands, a hyper-spectrum camera, a Fourier transform infrared spectrophotometer (FTIR), an infrared sensor, etc. Of course, multiple types of cameras (sensors) may be mounted on the flying object 200. A multispectral camera may be used that captures, for example, NIR and R (red) images and can calculate the NDVI (Normalized Difference Vegetation Index) from the images obtained. NDVI is a vegetation index that indicates the degree of vegetation, and can be used as an index showing the distribution and activity of vegetation. NDVI can be calculated from R and NIR images. That is, the NDVI value is: NDVI=(NIR-R) / (NIR+R) It is required as follows.
[0024] Tag information is added to the images captured by the imaging device 220. The tag information includes imaging date and time information, position information (latitude / longitude information) as GPS (Global Positioning System) data, information on the flying altitude of the flying object 200 at the time of capturing the image, imaging device information (camera individual identification information, model information, etc.), and information on each image data (image size, wavelength, imaging parameters, etc.).
[0025] Image data and tag information captured by the imaging device 220 attached to such an aircraft 200 are acquired by the information processing device 1. For example, image data and tag information are exchanged by wireless communication or network communication between the imaging device 220 and the information processing device 1. Examples of the network include the Internet, a home network, a LAN (Local Area Network), a satellite communication network, and various other networks. Alternatively, the image data and tag information are transferred to the information processing device 1 in such a manner that a recording medium (eg, a memory card) attached to the imaging device 220 is read by the information processing device 1 side.
[0026] The information processing device 1 uses the image data and tag information to generate analytical information for measuring the farm field 300. The information processing device 1 also performs processing to present the analysis results to the user as an image. Specifically, the crops captured in the image data are counted, and based on that, information that can be used to manage the field 300, such as the number of crops captured in the sample image, the number of crops per unit area, germination rate, predicted harvest yield, crop ratio, etc., can be generated and presented to the user.
[0027] The information processing device 1 is realized as, for example, a personal computer (PC) or a field-programmable gate array (FPGA), or a terminal device such as a smartphone or a tablet. In FIG. 1, the information processing device 1 is separate from the imaging device 220. However, for example, a calculation device (such as a microcomputer) serving as the information processing device 1 may be provided in a unit including the imaging device 220.
[0028] In the case of the sensing system of this embodiment, for example, the locations in the farm field 300 that are deemed to be "row" on the image data are automatically determined, and the number of planted crops is counted based on that determination. A row is a line along which crops are planted, and for example, the ridges formed for planting seeds in the field 300 are a type of row. Rows are not limited to raised soil such as ridges, but also include imaginary lines formed when seeds are sown on flat land. For example, the planting line formed when sowing seeds with the tractor 270 is called a row.
[0029] 2 shows a schematic diagram of rows 301 formed in a farm field 300. For example, as time passes after sowing, the seeds germinate and the rows of crop leaves form rows 301 as shown in the figure. In this embodiment, the image capturing device 220 captures images of the farm field 300 while the flying object 200 is moving, and images of various locations are captured at appropriate timing, for example, as in the image capturing ranges SP1 and SP2 in the figure. For example, image data of the image capturing range SP1 and image data of the image capturing range SP2 are obtained by the image capturing device 220 as a still image captured image of one frame, and these are each imported into the information processing device 1 as a sample.
[0030] 2 shows the imaging ranges SP1 and SP2 as rectangles of a certain size for ease of explanation, but the actual imaging range, i.e., the range imaged by the imager of the imaging device 220, differs depending on conditions such as the imaging direction, angle of view, and zoom magnification of the imaging device 220 mounted on the flying object 200, and further varies greatly due to the shaking of the flying object 200 during flight. Imaging is not necessarily performed with the imaging device 220 facing straight down and with its optical axis aligned with a vertical line from the ground. Therefore, the actual range of the farm field captured in the rectangular image as each sample is often not rectangular. The projective transformation described below is carried out taking these circumstances into consideration.
[0031] 2 shows linear rows 301, the rows 301 are not necessarily always straight. Straight rows 301 may bend at the edge of the field 300. Depending on the shape of the field 300, the route taken during sowing, obstacles, etc., the rows 301 may be partially curved, or may be formed in a spiral or concentric shape, for example.
[0032] The tractor 270 plants seeds and the like (seeds, seedlings, saplings, etc.) in the field 300. The tractor 270 automatically travels and plants based on planting instruction data provided by, for example, the information processing device 1. As a result, the above-mentioned rows 301 are formed in the field 300. A computer device is mounted on the tractor 270, and the computer device controls the traveling and planting operations based on instruction data. For example, by importing a data file including planting instruction data (rate map data including the number of planting instructions per unit area for each location) as shown in an instruction rate map described below, the computer device controls the traveling and planting operations of the tractor 270 based on the instruction data. Furthermore, the results of the planting work by the tractor 270 (the number of plants planted per unit area at each location) are provided to the information processing device 1 as result data.
[0033] <2. Configuration of information processing device> An information processing device 1 that acquires and processes image data from the imaging device 220 in the above sensing system will be described.
[0034] 3 shows a hardware configuration of the information processing device 1. The information processing device 1 includes a central processing unit (CPU) 51, a read only memory (ROM) 52, and a random access memory (RAM) 53. The CPU 51 executes various processes according to a program stored in the ROM 52 or a program loaded from the storage unit 59 to the RAM 53. The RAM 53 also stores data and the like necessary for the CPU 51 to execute various processes. The CPU 51, the ROM 52, and the RAM 53 are interconnected via a bus 54. An input / output interface 55 is also connected to this bus 54.
[0035] To the input / output interface 55, a display unit 56 consisting of a liquid crystal panel or an organic EL (Electroluminescence) panel, an input unit 57 consisting of a keyboard, a mouse, etc., a speaker 58, a storage unit 59, a communication unit 60, etc. can be connected.
[0036] The display unit 56 may be integrated with the information processing device 1 or may be a separate device. The display unit 56 displays captured images, various calculation results, and the like on the display screen based on instructions from the CPU 51. Furthermore, the display unit 56 displays various operation menus, icons, messages, and the like, that is, a GUI (Graphical User Interface), based on instructions from the CPU 51.
[0037] The input unit 57 refers to an input device used by a user of the information processing device 1. For example, the input unit 57 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 57, and a signal corresponding to the input operation is interpreted by the CPU 51.
[0038] The storage unit 59 is composed of a storage medium such as a hard disk drive (HDD) or a solid-state memory. The storage unit 59 stores various information such as detection data and analysis results received from an external device such as the imaging device 220. The storage unit 59 is also used to store program data for analysis processing and the like.
[0039] The communication unit 60 performs communication processing via networks including the Internet, and communication with various peripheral devices. The communication unit 60 may be a communication device that communicates with the image capture device 220, for example.
[0040] A drive 61 is connected to the input / output interface 55 as required, and a storage device 62 such as a memory card is attached to it, for writing and reading data. For example, a computer program read from the storage device 62 is installed in the memory unit 59 as necessary, and data processed by the CPU 51 is stored therein. Of course, the drive 61 may be a recording / reproducing drive for removable storage media such as a magnetic disk, an optical disk, or a magneto-optical disk. These magnetic disks, optical disks, magneto-optical disks, etc. are also aspects of the storage device 62.
[0041] The information processing device 1 according to the embodiment is not limited to being configured as a single information processing device (computer device) 1 having a hardware configuration as shown in Fig. 3, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN or the like, or may be located in a remote location using a VPN (Virtual Private Network) or the like using the Internet or the like. The multiple computer devices may include a computer device that can be used by a cloud computing service. 3 can be realized as a desktop or notebook personal computer, a tablet terminal, a smartphone, or other portable terminal. Furthermore, the information processing device 1 of this embodiment can be mounted in electronic devices having the function of the information processing device 1, such as a measuring device, a television device, a monitor device, an imaging device, and a facility management device.
[0042] For example, an information processing device 1 having such a hardware configuration has a calculation function by a CPU 51, a storage function by a ROM 52, a RAM 53, and a storage unit 59, a data acquisition function by a communication unit 60 and a drive 61, and an output function by a display unit 56, etc., and is provided with a functional configuration as shown in Figure 4 by the installed software functioning.
[0043] That is, as an example, the information processing device 1 is provided with an image acquisition unit 2, a projective transformation unit 3, a row determination unit 4, an output information calculation unit 5, a user interface control unit 6 (hereinafter, "user interface" will also be abbreviated as "UI"), and a recording control unit 7, as shown in FIG. The programs constituting the software that realizes these processing functions are downloaded from a network or read from a storage device 62 (e.g., a removable storage medium) and installed in the information processing device 1 in Fig. 3. Alternatively, the programs may be pre-stored in the storage unit 59 or the like. When the programs are started in the CPU 51, the functions of the above-mentioned units are realized. The calculation process and results of each function are stored, for example, in the memory area of the RAM 53 or the memory unit 59.
[0044] The image acquisition unit 2 is a function for acquiring image data to be processed and tag information associated with the image data. For example, image data captured by the imaging device 220 is stored in the storage unit 59, and the CPU 51 reads out specific image data and uses it as a target for calculation processing.
[0045] The projective transformation unit 3 performs projective transformation on a sample image, that is, a captured image of the farm field 300 captured from above the farm field 300, to obtain a projectively transformed image. The projectively transformed image is an image of the farm field 300 as viewed directly below from above the field 300, and is an image in which the scale within the image is made uniform. This will be described in more detail later.
[0046] The row determination unit 4 is a processing function that performs crop detection within the image data of the farm field 300, i.e., the image processed by the image acquisition unit 2 (the captured image before projective transformation or the projectively transformed image), and determines the rows 301 (i.e. the parts of the image in which the rows 301 are captured) within the image data based on the results of crop detection.
[0047] The output information calculation unit 5 has a function of performing calculations to generate information indicating the state of the farm field 300. For example, the output information calculation unit 5 counts the number of crops captured in the image data, and calculates the number of crops per unit area (such as 1 acre), predicted harvest amount, crop ratio, and the like based on the count number of crops. In particular, crop counting is performed from the projectively transformed image, and the counted number of crops is converted into the number of crops per unit area.
[0048] The UI control unit 6 has a function of performing, for example, display control of the display unit 56 and acquisition processing of operation information input by the input unit 57. That is, the UI control unit 6 presents images and information calculated by the output information calculation unit 5 and recognizes user operations.
[0049] The recording control unit 7 has a function of controlling the storage of information calculated by the output information calculation unit 5 and the like in the storage unit 59. The communication control unit 8 has a function of controlling the transmission of information calculated by the output information calculation unit 5 and the like to an external device via the communication unit 60.
[0050] <3. First embodiment> The processing of the information processing device 1 according to the first embodiment will be described. The information processing device 1 counts the number of crops using an image captured by the imaging device 220, but in this embodiment in particular, counting is performed using an image obtained by performing projective transformation on the captured image as a sample (projectively transformed image). In order to count, the system detects and determines the wax on the image, enabling highly accurate crop counting.
[0051] First, the circumstances that led to the development of the technology of this embodiment will be described. After sowing, farm managers need to count the number of crops per unit area and check the germination status. If this is done manually, it would look like this:
[0052] For example, in the United States, the unit of land management is the acre. One acre is approximately 4047 m 2 Since it is unrealistic to count the number of crops in an acre, it is generally calculated using 1 / 1000 acre, or 4.047 m 2 The number of crops per acre is calculated by counting the number of crops per acre and multiplying it by 1,000. Here, 4.047m 2 Since it is tedious for a person to surround the area in two dimensions, the row spacing W is used to create a 4.047m area, as shown by the shaded area in Figure 2. 2 A portion equivalent to a certain area is assumed, and the segment length L of one row 301 included therein is set. The row spacing W [m] is the spacing between the seed holes of the tractor 270 and is known. Then, the segment length L [m] of the row 301 to be obtained is expressed by the following equation. L=4.047 / W formula (1)
[0053] For example, if W=0.5 m, the section length L is about 8 m. Then, the calculated section length L [m] (for example, a section of about 8 m) is measured with a tape measure in the field 300, and the number of crops on the rows 301 within that range is visually counted. The count result is then multiplied by 1000 to determine the number of crops per acre. In this way, the number of crops on one row 301 is counted to estimate the number of crops per unit area. If the same procedure is performed using image processing as in this embodiment, a numerical value close to the intuition cultivated through the experience of the user, such as a farm manager or agricultural worker, can be provided. Rather than performing unnecessarily advanced calculations in order to achieve high accuracy, it is important to use efficient processing that places a low calculation burden while still being able to calculate values that are close to the intuition of farm managers / workers.
[0054] Therefore, in this embodiment, an area equivalent to, for example, 1 / 4000 acre of the actual farm field 300 is set on the image, and the crops on the rows 301 are counted. The count is then converted into the number of crops per unit area, such as 1 acre.
[0055] The following describes the crop counting process of the information processing device 1 with reference to Fig. 5 and Fig. 6. In the process of the embodiment, in order to count the crops, first the row is detected on the image, and then the number of crops on the row is counted. The process including the detection of the row will be described in detail.
[0056] Fig. 5 shows a series of processes in which the CPU 51 performs necessary arithmetic processing on the image data to be processed, and generates and outputs output information. This process is realized by the CPU 51 having the functions shown in Fig. 4. That is, step S101 is processed by the image acquisition unit 2, steps S102 to S108 are processed by the row determination unit 4, step S110 is processed by the output information calculation unit 5, and step S120 is processed by the UI control unit 6 (or the recording control unit 7, or the communication control unit 8).
[0057] In step S101, the CPU 51 identifies image data to be processed. That is, the CPU 51 identifies one of the image data samples taken from the air by the imaging device 220 mounted on the flying object 200 as the processing target. In this case, the image data may be color image data from an RGB (red, green, blue) camera, or may be image data as IR (near infrared) data and R (red) data. It may also be NDVI image data calculated from the IR (near infrared) data and R (red) data.
[0058] In step S102, the CPU 51 detects a region that resembles a crop within the image data to be processed. For example, G pixel detection will be described as a detection method when an image captured by an RGB camera is used. Crops are mainly composed of leaves, and leaves are generally green, so green areas are detected from the RGB color image. To detect green areas, for example, each RGB component of a pixel in the image may be compared, and only the pixel with the highest G (green) may be detected. Alternatively, RGB can be converted to a lightness, saturation, and hue space such as HSV, and pixels having a hue within the region can be detected by setting a region with a certain width in the green region of the hue value.
[0059] Next, threshold detection for NDVI images will be explained as a detection method when using images captured in near infrared and red. NDVI, an index showing the distribution of crops (plants), is calculated from near infrared and red light and has a range of ±1, with crops having high NDVI values and soil having low NDVI. Therefore, an NDVI image is created from IR data and R data, and pixels that resemble crops are detected by threshold processing. If an NDVI image already exists, the NDVI image is acquired in step S101 as described above, and pixels that are likely to be crops are detected by threshold processing in step S102.
[0060] It is possible to detect crop-like areas using only one of the above processes, but it is also possible to select one of the above two processes (G pixel detection and threshold detection for NDVI images) according to the type of information contained in the input image to detect crop-like areas.
[0061] For example, consider the case where an image as shown in FIG. 7A is input as a processing target. The image in FIG. 7A shows crops 310 planted in rows on a row, but there are some gaps (places where crops 310 should be but are not), and there are plants other than crops, such as weeds 315, in places other than the row.
[0062] The result of the detection of crop-like areas obtained by the above-mentioned G pixel detection or threshold detection of the NDVI image is expressed as a binary value of crop-like (white) or non-crop-like (black) for each pixel in an image of the same size as the input image, as shown in Fig. 7B. The white pixel part is a pixel area where plants are considered to be captured, but in this case, not only the crop 310 but also the weeds 315 are considered to be crop-like areas.
[0063] It should be noted that, in order to obtain such a result, a plurality of threshold values may be set, rather than just one. For example, two thresholds may be set for the NDVI value, and in addition to the two categories of "likely to be crops" and "not likely to be crops," three values may be added: "unclear whether crop or not."
[0064] Also, since there are no visible lines for the wax, it does not appear in an image like that of Figure 7A, but in this case, the line along which the crops 310 are lined up vertically in the figure is the wax. Of course, there are also cases where the wax can be physically seen due to ridges, etc. Thereafter, the CPU 51 performs processing to search for rows within the image.
[0065] In step S103, the CPU 51 detects law candidates. Since rows are generally straight lines, straight lines of the rows of crops 310 in the image are detected. For detecting straight lines, a Hough transform, a RANSAC (Random Sample Consensus), or the like is used. Straight lines passing through the crop-like pixels (white pixels in FIG. 7B) are detected as law candidates. Since the number and direction of the rows are unknown, all straight lines that appear to be wax in the image, such as dashed lines K1 to K7 in FIG. 8, are detected as law candidates.
[0066] Next, the CPU 51 calculates the crop size in step S104. First, for a certain line of a row candidate, we project the crop-like pixels near the line along the line. Specifically, we assume an arbitrary line parallel to the row candidate line, and count the number of crop-like pixels on that line. This makes it possible to obtain the distribution of crop-like pixels in the direction perpendicular to the row candidate line. Figure 9 shows an example of calculating the projection of a line of a row candidate indicated by the dashed line K1 on the left side of Figure 8. Since the line of the row candidate is detected to pass through pixels that are likely to be crops, this distribution is highest near the center and decreases as it moves away from the center. A certain threshold thS is then set for this distribution, and the width WS that is equal to or greater than the threshold thS is estimated as the crop size. The threshold thS can be set, for example, to half the median.
[0067] This process is performed on all straight lines of the row candidates (dashed lines K1 to K7) to calculate (estimate) the crop size. Since crops planted at the same time are considered to be approximately the same size, the average value, median value, or other value of the crop sizes of all the calculated row candidates is determined as the crop size SS. Alternatively, the projections of all crop-like pixels in the vicinity of a line of any row candidate may be summed and a unique crop size SS may be determined by thresholding.
[0068] Next, in step S105, the CPU 51 detects the location of the crop on the row. In this case, a reference area RA is first set on the row according to the calculated crop size. An example of the reference area RA is shown in FIG. 10A. The reference area RA may be, for example, a circular area with the crop size SS as a diameter, or a square area with the crop size SS as one side. Of course, other shapes are also acceptable. For example, the area shape may be set according to the type of crop (shape of the crop).
[0069] The reference area RA is moved along the straight line of the row candidate (dashed line K1 to dashed line K7 in FIG. 10C) to sequentially determine the crop location. For example, as shown in FIG. 10B, the reference area RA is moved along dashed line K1 to determine whether each location in the reference area RA is a crop location. For example, the number of pixels that look like crops in the reference area RA is counted, and if the count number is equal to or greater than a certain threshold value thC, the position of the reference area RA is determined to be the crop location CS. In this case, the threshold value thC may be set to, for example, half or 2 / 3 of the area of the reference area RA. FIG. 10C shows an example of detecting crop locations CS on row candidates.
[0070] Next, in step S106, the CPU 51 determines the spacing between crops on the rows. If crops are considered to be planted at equal intervals on any row in the input image to be processed, the spacing between crops on the image is determined. First, for any line of a row candidate, the distance to the adjacent crop point CS detected by the above process is calculated. For each crop point CS, the distance to the adjacent crop point CS is calculated, and the average or median of the distances is determined as the crop interval in the image to be processed.
[0071] Next, in step S107, the CPU 51 calculates the likelihood of each of the Law candidates (dashed lines K1 to K7). Since crops are evenly planted on the correct rows, the calculated crop spacing is used to count the number of sets of crops planted with the correct crop spacing. A "pair of crops" refers to two adjacent crops, which corresponds to two adjacent crop locations CS in the process. In other words, the number of pairs of two crop locations CS with the correct crop interval is counted. Whether the crop spacing is correct or not is determined based on whether the distance between the crops is within an error range for the above-mentioned crop spacing, which is preset to, for example, 20%.
[0072] For example, Fig. 11A shows four crop locations CS, and in this case, there are three sets GP1, GP2, and GP3 as "crop sets." If the crop spacing of each set GP1, GP2, and GP3 is distance BT1, BT2, and BT3, and the distances BT1 and BT2 are correct crop spacings and the distance BT3 is an incorrect crop spacing, then the number of crop sets with correct crop spacings is "2." In this way, the CPU 51 calculates the number of combinations of crops planted at the correct crop intervals for each Law candidate, and sets the calculated value as the likelihood value of the Law candidate. The larger the value, the more likely the Law candidate is defined as being Law.
[0073] Next, in step S108, the CPU 51 determines the correct row from among the candidate rows, for example by carrying out the following steps (P1) to (P3).
[0074] (P1) Since laws do not generally intersect, we look for two intersecting law candidates, and of the two law candidates, we delete the one with the lower probability. This is repeated until there are no more intersecting Law candidates.
[0075] (P2) The inclination of the line of each of the row candidates remaining in the process of (P1) above is calculated, and the average value or median value is calculated as the row inclination of the entire image. Since each row is arranged in parallel, row candidates whose inclination deviates from the calculated row inclination by more than an allowable range are deleted. The allowable range of deviation is preset, for example, an angle difference of 10 degrees or less. The allowable range may be set based on information such as the inclination and angle of view of the imaging device 220 at the time of imaging.
[0076] (P3) The Law interval is calculated from the average or median of the intervals of the narrowed-down Law candidates. Then, among the Law candidates remaining from the processes (P1) and (P2) above, Law candidates whose intervals are closer to the calculated Law interval (for example, 80% or less of the Law interval) are eliminated as Law candidates with low probability.
[0077] By this process, the law candidates indicated by the dashed lines K1, K2, K3, and K4 remain as likely law candidates, as shown in Fig. 11B. In this case, not only are the intersecting law candidates removed, but also the erroneous law candidates detected due to weeds, etc. are removed. The CPU 51 determines these remaining law candidates as the law in the image.
[0078] The CPU 51 calculates the number of crops in step S110. Regarding the number of crops, it is possible to count the number of crop locations CS on a straight line determined as a row, for example, and count the number of crops in the image to be processed. However, here we will describe an example of counting the number of crops within a range set as a count area in the image, and then calculating the number of crops per unit area (for example, 1 acre) based on that. In particular, since the wax is detected in the processing up to step S108, it is possible to distinguish between plants growing randomly outside the wax, such as weeds, and crops growing on the wax, and therefore the number of crops can be counted with high accuracy.
[0079] A specific example of the process in step S110 will be described with reference to FIG. In step S111, the CPU 51 performs projective transformation on the image to be processed. The reason for performing the projective transformation is as follows.
[0080] When counting the number of crops from an image, the problem is the relationship between the captured image and the actual ground surface. The multiple rows detected in the image captured from the air by an aircraft 200 such as a drone are sown at equal intervals by a tractor 270, so if the camera direction during capture is perpendicular to the ground, they will be lined up in a straight line that is approximately parallel to each other.
[0081] However, in reality, the imaging device 220 attached to the flying object 200 cannot always capture images perpendicular to the ground. Also, even within the captured range, the distance from the imaging device 220 to the area directly below the ground and the surrounding area is different. For this reason, the rows are not necessarily parallel on the image, and multiple rows detected on the image may be arranged radially, as shown in Fig. 12. In the figure, the rows are indicated by dashed lines RL, but in this example, they are radially extending upward in the drawing. The fact that parallel rows actually radiate out also means that the scale is not uniform across the image: for example, in the image of Figure 12, the lower part of the image, where the rows are closer together, actually captures a larger area.
[0082] In such a state, it is difficult to set with high accuracy on the screen the range that corresponds to the shaded area in Figure 2, for example, the range of 1 / 1000 acre. In terms of specific processing, it is difficult to set with high accuracy the length of the row to be counted for crops, i.e., the above-mentioned division length L.
[0083] Therefore, the inclination is corrected using information from an acceleration sensor mounted on the flying object 200 together with the imaging device 220. That is, the imaging angle is determined by the acceleration sensor, and the image is corrected by a process called projective transformation. By correcting the image by projective transformation, the rows on the corrected projective transformed image become approximately parallel. 13A is a schematic diagram of a captured image. The image is a rectangle captured by the imager of imaging device 220, but rows (dotted and dashed lines RL) are radial due to the angle of the imaging direction relative to the vertical. FIG. 13B shows a schematic of a projectively transformed image. In a projectively transformed image, the rows are parallel. Projectively transformed images also result in a uniform scale within the image.
[0084] Even if the rows become parallel by projective transformation, the scale differs depending on the altitude at which the image capturing device 220 captures the image, and therefore scale correction of the image according to the altitude is also required. The scale of the image is determined using information from a GPS sensor mounted on the flying object 200 together with the imaging device 220. The GPS sensor senses information on the sensor's latitude, longitude, and altitude.
[0085] In this way, by performing a projection transformation, including a scale transformation, on the captured image onto the ground surface using the imaging angle, latitude, longitude, and altitude information from the GPS sensor, and the angle of view of the imaging device 220, the image created due to the tilt of the imaging device 220 is corrected to an image captured facing directly downward, and a scale conversion coefficient is calculated to convert the actual measured length [m] to pixels [pixels].
[0086] 14A shows a coordinate system in which the ground is a plane defined by the x- and y-axes, and the vertical direction is the z-axis, and the image capture device 220 captures an image at a certain three-dimensional position pc with an attitude Rcam having an offset angle from the vertical direction. Vectors VC1, VC2, VC3, and VC4 are directional vectors from the lens focus of the image capture device 220 to each vertex of the rectangular captured image surface (image plane IP). Note that this example is just one model, but if we assume that the flight direction of the flying object 200 is in the x-axis direction, then the imaging direction of the imaging device 220 has a shift of angle θ in both the x-axis and y-axis directions.
[0087] Fig. 14B shows the state where the vectors VC1, VC2, VC3, and VC4 are extended to the ground surface, and the shape with the points where each of them reaches the ground surface as vertices is shown by the diagonal lines in Fig. 14C. The range of the ground surface corresponding to this shape appears as a projectively transformed image as shown in Fig. 13B.
[0088] Specifically, the CPU 51 obtains the position pc of the imaging device 220 relative to the ground surface from the latitude, longitude, and altitude information of the imaging device 220. Next, the CPU 51 obtains the angle of view of the imaging device 220 and directional vectors VC1, VC2, VC3, and VC4 from the lens focus to the four corners of the image plane IP of the imaging device 220. Next, the CPU 51 extends these four directional vectors VC1, VC2, VC3, and VC4 until they intersect with the ground surface.
[0089] The CPU 51 then determines the projection transformation parameters by considering these four intersecting points as the four corners of the projection transformation. The projection transformation for transforming a certain coordinate (x, y) on the original captured image as shown in FIG. 15A into a coordinate (x', y') on the projection transformation image as shown in FIG. 15B can be calculated by the following equation. x'=(ax+by+c) / (gx+hy+1) y'=(dx+ey+f) / (gx+hy+1)
[0090] Here, the eight parameters of the projective transformation can be uniquely determined from the eight numerical values of the x and y coordinates of the four corners (x0, y0, x1, y1, x2, y2, x3, y3). In addition, since this ground surface is a coordinate system based on actual measurements, when the projectively transformed coordinates are extracted as an image, a scale conversion coefficient, which is the relationship between the actual distance and the number of pixels, can be obtained.
[0091] After performing such projective transformation in step S111 of FIG. 6, the CPU 51 sets the division pixel count in step S112. That is, the segment length L is calculated by the above-mentioned formula (1) using the row interval W, and the number of pixels corresponding to the segment length L is calculated. The row interval W is determined by the tractor 270, so it may be input / stored in the information processing device 1 in advance.
[0092] In step S113, the CPU 51 sets a count area CA as shown in FIG. FIG. 16 shows a part of a projectively transformed image in which the rows have been rectified, and in it, a counting area CA that surrounds a range of pixels equivalent to the segment length L for one row is illustrated. The count area CA is an area to be counted, and is also a pixel range in which a certain area, such as 1 / 1000 acre, actually appears on the projectively transformed image. In this case, the area includes only one row. The certain area is preferably an area that can be easily converted to a unit area, such as 1 acre. By setting the width of each row to W / 2 on either side (row spacing direction) and the length of the section L, the count area CA is equivalent to the area of 1 / 1000 acre that contains one row.
[0093] In step S114, the CPU 51 counts the number of crops in the count area CA. That is, the CPU 51 counts the number of crop locations on the row previously determined in step S108. In step S115, the CPU 51 converts the count number within the count area CA into the number of crops per unit area. For example, if the count area CA is set to a range equivalent to 1 / 1000 of an acre as described above, the number of crops counted in the count area CA is multiplied by 1000 to calculate the number of crops per acre.
[0094] For example, after performing the process of FIG. 6 as described above, the CPU 51 performs a process of outputting the number of crops per unit area as output information in step S120 of FIG. For example, the CPU 51 performs processes such as displaying information on the number of crops per unit area on the display unit 56, recording the information in the memory unit 59 or the storage device 62, and transmitting the information to an external device via the communication unit 60. Furthermore, the CPU 51 may output, as auxiliary information, information required before calculating the number of crops, such as the number and positions of rows, and crop locations CS.
[0095] <4. Second embodiment> The second embodiment is an example in which projective transformation is performed before row determination. Fig. 17 shows an example of processing by the CPU 51. Note that the same processes as those in Fig. 5 are given the same step numbers to avoid duplicated explanations.
[0096] After the CPU 51 specifies the captured image to be processed in step S101, it performs projective transformation in step S150, which is the same process as step S111 in FIG. Thereafter, the CPU 51 performs the same process for determining the rows as described above on the projectively transformed image in steps S102 to S108, and then performs a process for calculating the number of crops in step S110A. The process of calculating the number of crops in step S110A is generally similar to that in FIG. 5, but since projective transformation has already been performed, it is sufficient to carry out the processes from step S112 to step S115 in FIG.
[0097] As in the second embodiment, by performing projective transformation before processing for determining the row, an image in which the rows have been parallelized can be used. This allows for a more accurate determination when determining the correct row in step S108 (for example, when parallelism is a condition as in (P2) above), thereby improving the accuracy of row determination.
[0098] However, when projective transformation is performed after law determination as in the first embodiment, there is an advantage that the size of the image to be processed is constant during the law determination process (that is, the processing time is constant). If projective transformation is performed first, a part of the image will be crushed if it is reduced, so it is necessary to enlarge it for the row determination process. Also, if rotation correction is applied, the circumscribing rectangle of the image will become larger. Also, if there is an error in the imaging angle or flight altitude of the imaging device 220, the parallel condition may not be satisfied, and if the parallel condition is applied as is, the accuracy may decrease.
[0099] Therefore, whether to perform projective transformation after law determination as in the first embodiment, or to perform projective transformation prior to law determination processing as in the second embodiment, may be selectable depending on the implementation situation, various measurement circumstances, flight plan, etc.
[0100] <5. Third embodiment> In the first embodiment, an example has been described in which one count area CA is set on a projectively transformed image, but it is of course possible to set a plurality of count areas CA on one projectively transformed image. FIG. 18 shows a state in which a plurality of count areas CA are set as count areas CA1, CA2, CA3, and CA4.
[0101] Then, the CPU 51 performs the process of FIG. 19 as the process of calculating the number of crops in step S110 in FIG. Steps S111 and S112 are the same as those in FIG.
[0102] In step S113A, the CPU 51 sets a plurality of (N) count areas CA on the projectively transformed image. In this case, the rows for which the count area CA is set may be selected from all rows detected that are equal to or greater than the number of segment pixels (i.e., rows that appear in the projectively transformed image with a length equal to or greater than the segment length L), or may be selected from a number of rows not exceeding a preset number. For example, as shown in Figure 18, each of the count areas CA1, CA2, CA3, and CA4 is an area of segment length L that contains only one row. Each of the count areas CA1, CA2, CA3, and CA4 is an area on the image corresponding to an area of L×W.
[0103] After step S130 in FIG. 19, the CPU 51 sequentially performs crop counting in each count area CA. That is, the CPU 51 sets the variable n=1 in step S130, and counts the number of crops in the nth count area CA in step S131. If the variable n has not reached the number of count areas N in step S132, the variable n is incremented in step S133, and the process returns to step S131 to count the number of crops in the nth count area CA. This is repeated until n=N in step S132, whereby counting of the number of crops is completed in each of the set count areas CA.
[0104] When the counting of the number of crops in all count areas CA is completed, the CPU 51 calculates a representative value in step S134. For example, when counting is performed in count areas CA1, CA2, CA3, and CA4, the average of these count numbers is set as the representative value. The representative value is not limited to the average value, but may be the median value, the center of gravity value, or the intermediate value between the maximum value and the minimum value.
[0105] In step S115, the representative value is used to calculate the number of crops per unit area. For example, if each count area CA is equivalent to 1 / 1000 of an acre, the representative value is multiplied by 1000 to obtain the number of crops per acre.
[0106] As described above, by setting multiple count areas CA and calculating the number of crops per unit area using their representative values, it is possible to avoid situations where the number of crops per unit area significantly differs when a single count area CA happens to have an exceptionally low (or high) number of crops, thereby improving the detection accuracy of the number of crops per unit area.
[0107] In addition, in the case of the process of calculating the number of crops in step S110A when projective transformation is performed before the row determination process as in FIG. 17, step S111 in FIG. 19 is unnecessary.
[0108] <6. Fourth embodiment> The fourth embodiment is an example in which projective transformation is not performed in some cases. FIG. 20 shows the process of calculating the number of crops performed in step S110 of FIG. In step S140, the CPU 51 determines the attitude of the imaging device 220 when capturing the captured image to be processed. That is, it checks whether the angle of deviation from a state facing directly below the imaging direction (optical axis) as the attitude of the imaging device 220 is within a predetermined allowable range.
[0109] If the deviation angle, ie, the inclination, is not within the allowable range, the CPU 51 proceeds from step S141 to step S111 to perform projective transformation processing. On the other hand, if the inclination is within the allowable range, the CPU 51 skips step S141 and proceeds to the process of step S112 and subsequent steps. The process of step S112 and subsequent steps is the same as that in FIG.
[0110] In other words, if an image is originally captured with little deviation in the vertical direction, the rows will be parallel even without projective transformation, and the uniformity of scale within the image will be maintained to a certain extent. Therefore, in such cases, the processing load can be reduced by not performing projective transformation processing.
[0111] For example, images are taken as discrete samples as shown in Figure 2 as imaging areas SP1 and SP2. On a large farm, several tens to several hundreds of sample images can be obtained in one flight, and crop counts are performed on each of them. The attitude of the flying object 200 during flight is unstable due to flight speed, turning conditions, wind conditions, etc. Therefore, even if image capturing is usually performed at an angle to the vertical direction, it is conceivable that the image capturing device 220 may face directly downward in some cases, in which case a captured image suitable for crop counting can be obtained without projective transformation. In anticipation of such a case, the fourth embodiment is configured to not perform projective transformation processing if it is unnecessary.
[0112] When projective transformation is performed before the row determination process as in FIG. 17, the process in steps S140 and S141 in FIG. 20 may be performed at that time to determine whether or not to perform the projective transformation process.
[0113] <7. Fifth embodiment> In the above embodiment, the number of rows and crops is detected based on one captured image. However, when the capturing altitude is low or the row length (section length L) calculated by formula (1) is long, one image may not cover the range of section length L in formula (1). In such a case, it may be possible to project a plurality of captured images onto the ground and stitch them together, and then perform row determination and crop counting from the stitched image.
[0114] <8. Summary and variations> According to the above embodiment, the following effects can be obtained. The information processing device 1 of the embodiment includes a projection transformation unit 3 that performs projection transformation on an image captured of the field 300 from above the field 300 to obtain a projection transformed image, and an output information calculation unit 5 that counts the number of crops using the projection transformed image and generates crop number information. Projective transformation makes it possible to obtain an image captured by the imaging device 220 without any misalignment during imaging. In other words, it is an image captured by the imaging device 220 facing directly downward. In such a projective transformation image, the rows appear in the image in a roughly parallel manner, just like the actual rows. In other words, they are not radial due to the misalignment of the camera orientation. In addition, the scale within the image is made uniform. In other words, the area per pixel is made uniform. By using such a projectively transformed image to perform crop counting, highly accurate crop counting is possible.
[0115] In the first to fifth embodiments, the output information calculation unit 5 sets a count area CA in the projectively transformed image in which the number of crops is to be counted, and calculates the number of crops per unit area based on the number of crops counted within the count area CA. For example, if you want to output the number of crops per acre, the area in the projectively transformed image whose actual area is 1 / 1000 of an acre is set as the count area CA. The number of crops measured in this count area CA is multiplied by 1000 to get the number of crops per acre. In such cases, even if the error in the counting numbers using the images is small, the impact of the error becomes significant when multiplied by 1000, and there is a risk that the farm manager will not be provided with appropriate information. For example, since the scale of the captured images is not necessarily uniform within the images, errors in crop counting are likely to occur due to circumstances such as the counting area selected in the images as a range equivalent to 1 / 1000 acre in actuality having a large error from 1 / 1000 acre. In contrast, by using a projectively transformed image, the count area can be set without error as a range in the image that is equivalent to 1 / 1000 of an acre in actual area, and as a result, the error in counting the number of crops is also small. Therefore, even if the number is converted to the number of crops per acre, the error is small. Therefore, this is desirable as information to present to the user.
[0116] In the third embodiment, the output information calculation unit 5 sets multiple counting areas CA (CA1, CA2, CA3, CA4), each corresponding to an area of the same size within the field 300, determines a representative value of the number of crops based on the number of crops counted in each counting area CA, and uses the representative value to calculate the number of crops per unit area. If only one count area CA is used in the projectively transformed image, the number of crops in that count area CA may happen to be significantly different from other areas, which may result in a relatively large error when converted to the number of crops per unit area. Therefore, by counting the number of crops in multiple count areas CA and finding a representative value, the number of crops per unit area can be found based on the representative value that reflects the number of crops over a wider range. This makes it possible to reduce the error in the number of crops per unit area. The representative value may be an average value, a centroid value, a median value, or the like.
[0117] In the first to fifth embodiments, an example was given in which the output information calculation unit 5 calculates the number of pixels on the projectively transformed image that corresponds to the area to be counted in the field 300, and sets the count area CA as the image range of the calculated number of pixels. For example, the area to be counted to find the number of crops in 1 acre is assumed to be 1 / 1000 acre. In this case, the number of pixels in the image range equivalent to 1 / 1000 acre in the projectively transformed image, that is, the image range capturing the range of 1 / 1000 acre, is found. By setting one count area as the range of this number of pixels, the number of crops counted from the projectively transformed image will be the number of crops in the actual range of 1 / 1000 acre. By setting the count area using the number of pixels in this manner, the number of crops within a specified area in the actual farm field 300 can be counted from the image.
[0118] In the first to fifth embodiments, the count area CA is set as a range including one row. This allows the number of crops to be counted by counting plants that are approximately equally spaced apart on a single row, making it a relatively simple process with high accuracy. In this case, the number of crops is counted in the range of the division length L in one row and converted into the number of crops per unit area. This is in line with the method that the farm manager / worker has been performing visually, and the resulting number of crops per unit area is very close to the feeling of the farm manager / worker. In other words, rather than performing unnecessarily advanced calculations in order to achieve high accuracy, the system uses efficient processing that places a low calculation burden while still being able to calculate values that are close to what farm managers / workers have traditionally perceived, thereby providing farm managers / workers with calculation results that are intuitively easy to understand.
[0119] The count area CA may be set to a range that includes multiple rows. For example, it is possible to define a range having a width of 2W, which is half the length of the segment length L in FIG. 16, as the count area CA, and to define the range including two rows as the count area CA.
[0120] In the fourth embodiment, the projective transformation unit 3 determines the orientation of the imaging device 220 at the time of capturing an image, and generates a projectively transformed image if the tilt is outside the allowable range. If the projectively transformed image is not generated by the projective transformation unit 3, the output information calculation unit 5 counts the number of crops using an image that has not been subjected to projective transformation, and generates crop number information. The imaging direction of the imaging device 220 mounted on the flying object 200 is not always constant. This is because the attitude of the flying object 200 itself fluctuates due to flight movements, the influence of wind, and the like. For this reason, projective transformation is performed assuming changes in the attitude of the imaging device 220. However, conversely, a captured image may be obtained in which the imaging device 220 happens to be facing straight down. In that case, counting with less error is possible even without using a projectively transformed image. Therefore, if the attitude of the imaging device 220 at the time of capturing an image is within an allowable range with respect to the vertical direction, counting is performed directly with the captured image without projective transformation. This reduces the processing load of projective transformation.
[0121] In the first to fifth embodiments, a row determination unit 4 is provided that performs crop detection within an image captured of a farm field 300 and determines rows, which are lines within the image along which crops are planted, based on the results of the crop detection, and an output information calculation unit 5 counts the number of crops using information about the rows determined by the row determination unit 4. This allows the row to be automatically determined from image data captured of the field 300. This eliminates the need for user input or data transfer to identify the row, making the system extremely easy to use, for example as a system for checking the condition of a field. By identifying the row, crops can be counted as plants on the row, and counting accuracy can be improved by eliminating weeds and the like that grow in places other than the row.
[0122] In the second embodiment, the row determination unit 4 uses a projectively transformed image produced by the projective transformation unit as an image of the field 300, and determines rows, which are lines along which crops are planted, within the projectively transformed image based on the results of crop detection. This improves the accuracy of row determination because the row determination uses an image in which row parallelism and scale errors have been eliminated, and also improves the accuracy of count area setting and crop counting.
[0123] In the embodiment, an example has been described in which the row determination unit 4 detects row candidates based on the results of crop detection, calculates the likelihood of each row candidate being a row based on the crop spacing, and determines a row based on the calculation result of the likelihood (see Figure 5). When determining rows, first of all, areas where crops (plants) are present can be candidates for rows. By using the presence of crops as a criterion and finding many row (row of crops) candidates, it is possible to eliminate missing a row. Then, by calculating the likelihood from among the row candidates, accurate row detection can be achieved. It is particularly easy to find the likelihood by using the spacing between crops as a criterion. Of course, accuracy can be further improved by taking into account the fact that rows are lined up in parallel. And by accurately detecting wax in an image, it will be possible to distinguish crops from randomly growing vegetation, such as weeds.
[0124] As described in detail in the first embodiment, the law determination unit 4 calculates the crop size SS from the size of the crop-like area near the top of the law candidate, detects crop candidates on the law candidate from the calculated crop size SS, calculates the crop spacing on the image data from the spacing of the detected crop candidates, determines the likelihood of the law candidate from the determined crop spacing, and uses the determined likelihood to exclude law candidates that are determined to be erroneously detected. In other words, it excludes law candidates that are erroneously detected. Then, it performs a process to determine the law. When determining the likelihood of a row candidate based on the spacing between crops, it is also important that the spacing between crops is highly accurate. In the process of the embodiment, crop candidates on a row candidate are detected from the estimated crop size. In other words, by detecting the spacing between crops while taking the growing size into account and determining the likelihood of the row candidate based on that, it becomes possible to determine the row with higher accuracy.
[0125] The field 300 described in the embodiment may be not only cultivated land in outdoor farmland, but also land used for hydroponic cultivation or greenhouse cultivation, and the technology of the embodiment can be used to present information about crops grown in various locations. Furthermore, the technology of the embodiment can be used to present sensing results relating to the growth of, for example, fruit trees, weeds, etc., other than crops. Therefore, it can also be applied to remote sensing of forests, vacant lots, etc. Furthermore, the technology of the embodiment can be applied not only to plants but also to sensing the quality of soil in the farm field 300 and the like.
[0126] 4 does not necessarily have to include all of the functions listed in the present technology. The information processing device of the present technology is required to include at least the projection transformation unit 3 and the output information calculation unit 5. By including other functions as appropriate, it is possible to obtain the effects that will be understood from the description of the embodiment.
[0127] In the embodiment, an example has been given in which the imaging device 220 is mounted on the flying object 200 that can be wirelessly controlled or automatically controlled. Examples of the flying object 200 that can be radio-controlled or automatically controlled include so-called drones, small radio-controlled fixed-wing airplanes, and small radio-controlled helicopters.
[0128] The program of the embodiment is a program that causes the information processing device 1 to execute a process of performing projective transformation of an image captured of the field 300 from above the field 300 to generate a projectively transformed image, and a process of counting the number of crops using the projectively transformed image to generate crop number information. That is, it is a program that causes the information processing device 1 to execute the processes described with reference to FIGS. 5, 6, 17, 19, 20, and so on.
[0129] Such a program makes it easy to realize the information processing device 1 of the present embodiment. Such a program can be stored in advance in a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU. Alternatively, the program can be temporarily or permanently stored (stored) in a removable recording medium such as a semiconductor memory, a memory card, an optical disk, a magneto-optical disk, or a magnetic disk. Such a removable recording medium can be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN or the Internet.
[0130] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.
[0131] The present technology can also be configured as follows. (1) a projection transformation unit that performs projection transformation on an image of the farm field captured from above to obtain a projection transformed image; an output information calculation unit that counts the number of crops using the projectively transformed image and generates crop number information; Information processing device. (2) The output information calculation unit is A count area for counting the number of crops is set on the projectively transformed image, and the number of crops per unit area is calculated based on the number of crops counted within the count area. The information processing device according to (1) above. (3) The output information calculation unit is As the counting areas, a plurality of counting areas each corresponding to an area of the same surface area within the farm field are set, and a representative value of the number of crops is calculated based on the number of crops counted within each counting area, and the representative value is used to calculate the number of crops per unit surface area. The information processing device according to (2) above. (4) The output information calculation unit is The number of pixels on the projectively transformed image that corresponds to the area of the field to be counted is calculated, and the count area is set as the image range of the calculated number of pixels. The information processing device according to (2) or (3) above. (5) The counting area is set as a range including one row, which is a line where crops are planted. An information processing device according to any one of (2) to (4) above. (6) the projective transformation unit determines an attitude of the imaging device at the time of capturing the captured image, and generates a projectively transformed image if the inclination is outside an allowable range; When the projective transformation unit does not generate a projectively transformed image, the output information calculation unit counts the number of crops using an image that has not been subjected to projective transformation, and generates crop number information. An information processing device according to any one of (1) to (5) above. (7) a row determination unit that performs crop detection within an image of a farm field and determines rows, which are lines along which crops are planted, within the image based on a result of the crop detection; The output information calculation unit counts the number of crops using information on the row determined by the row determination unit. An information processing device according to any one of (1) to (6) above. (8) The row determination unit A projectively transformed image obtained by the projective transformation unit is used as an image of a farm field, and rows, which are lines along which crops are planted, are determined in the projectively transformed image based on the result of the crop detection. The information processing device according to (7) above. (9) The row determination unit detecting row candidates based on the results of the crop detection; Calculate the likelihood of each row candidate as a row based on the crop interval of the row candidate; Determine the law based on the result of the likelihood calculation. The information processing device according to (7) or (8) above. (10) The row determination unit is The size of the crop is calculated from the size of the crop-like area near the top of the row candidate. Detecting crop candidates on the row candidates from the calculated crop size; Calculating the spacing between crops on the image from the spacing between the detected crop candidates; The likelihood of a row candidate is determined from the determined crop spacing. Using the determined likelihood, the process of determining the law is performed after eliminating law candidates determined to be false positives. An information processing device according to any one of (7) to (9) above. (11) A projective transformation is performed on an image of the farm field captured from above to generate a projective transformed image; The number of crops is counted using the projectively transformed image, and crop number information is generated. Information processing methods. (12) A process of performing projective transformation on an image of the farm field captured from above to generate a projectively transformed image; A process of counting the number of crops using the projectively transformed image and generating information on the number of crops; A program for causing an information processing device to execute the above. [Explanation of symbols]
[0132] 1. Information processing device 2. Image acquisition section 3 Projection transformation section 4 Law determination section 5. Output information calculation section 6 UI control section 7 Recording control section 8. Communication Control Unit 51 CPU 56 Display section 57 Input section 59 Memory section 60 Communications Department 200 Flying Objects 220 Imaging device 270 Tractor 300 fields
Claims
1. a projection transformation unit that performs projection transformation on an image of the farm field captured from above to obtain a projection transformed image; a row determination unit that detects crops in an image of a farm field and determines rows, which are lines along which crops are planted, in the image by using information on crop sizes obtained by the crop detection; an output information calculation unit that uses the projectively transformed image and the information on the law determined by the law determination unit to count the number of crops and generate information on the number of crops. Information processing device.
2. A projection transformation unit that performs projection transformation on an image of a farm field captured from above to obtain a projection transformed image; a row determination unit that performs crop detection within an image of a farm field and determines rows, which are lines along which crops are planted, within the image based on a result of the crop detection; an output information calculation unit that uses the projectively transformed image and the information on the law determined by the law determination unit to count the number of crops and generate information on the number of crops; Equipped with The row determination unit The size of the crop is calculated from the size of the crop-like area near the top of the row candidate. Detecting crop candidates on the row candidates from the calculated crop size; Calculating the spacing between crops on the image from the spacing between the detected crop candidates; The likelihood of a row candidate is determined from the determined crop spacing. Using the determined likelihood, the process of determining the law is performed after eliminating law candidates determined to be false positives. Information processing device.
3. The output information calculation unit is A count area for counting the number of crops is set on the projectively transformed image, and the number of crops per unit area is calculated based on the number of crops counted within the count area.
3. The information processing device according to claim 1 or 2.
4. The output information calculation unit is As the counting areas, a plurality of counting areas each corresponding to an area of the same surface area within the farm field are set, and a representative value of the number of crops is calculated based on the number of crops counted within each counting area, and the representative value is used to calculate the number of crops per unit surface area. The information processing device according to claim 3 .
5. The output information calculation unit is The number of pixels on the projectively transformed image that corresponds to the area of the field to be counted is calculated, and the count area is set as the image range of the calculated number of pixels. The information processing device according to claim 3 .
6. The counting area is set as a range including one row, which is a line where crops are planted. The information processing device according to claim 3 .
7. the projective transformation unit determines an attitude of the imaging device at the time of capturing the captured image, and generates a projectively transformed image if the inclination is outside an allowable range; When the projective transformation unit does not generate a projectively transformed image, the output information calculation unit counts the number of crops using the image that has not been subjected to projective transformation, and generates crop number information.
7. The information processing device according to claim 1.
8. The row determination unit A projectively transformed image obtained by the projective transformation unit is used as an image of a farm field, and rows, which are lines along which crops are planted, are determined in the projectively transformed image based on the result of the crop detection.
8. The information processing device according to claim 1.
9. An information processing device comprising: a projective transformation process for performing projective transformation on an image of the farm field captured from above to obtain a projectively transformed image; a row determination process for performing crop detection in an image of a farm field and determining rows, which are lines along which crops are planted, in the image by using information on crop size obtained by the crop detection; an output information calculation process for counting the number of crops using the projectively transformed image and the information on the rows determined in the row determination process, and generating information on the number of crops. Information processing methods.
10. An information processing device comprising: a projective transformation process for performing projective transformation on an image of the farm field captured from above to obtain a projectively transformed image; a row determination process for detecting crops in an image of a farm field and determining rows, which are lines along which crops are planted, in the image based on the results of the crop detection; an output information calculation process for counting the number of crops using the projectively transformed image and the information on the rows determined in the row determination process, and generating information on the number of crops; In the row determination process, The size of the crop is calculated from the size of the crop-like area near the top of the row candidate. Detecting crop candidates on the row candidates from the calculated crop size; Calculating the spacing between crops on the image from the spacing between the detected crop candidates; The likelihood of a row candidate is determined from the determined crop spacing. Using the determined likelihood, the process of determining the law is performed after eliminating law candidates determined to be false positives. Information processing methods.
11. A projective transformation process for performing projective transformation on an image of a farm field captured from above to obtain a projectively transformed image; a row determination process for performing crop detection in an image of a farm field and determining rows, which are lines along which crops are planted, in the image by using information on crop size obtained by the crop detection; and an output information calculation process for counting the number of crops using the projectively transformed image and the information on the rows determined in the row determination process, and generating information on the number of crops. program.
12. A projective transformation process for performing projective transformation on an image of a farm field captured from above to obtain a projectively transformed image; a row determination process for detecting crops in an image of a farm field and determining rows, which are lines along which crops are planted, in the image based on the results of the crop detection; an output information calculation process for counting the number of crops using the projectively transformed image and information on the rows determined in the row determination process, and generating crop number information; In the row determination process, The size of the crop is calculated from the size of the crop-like area near the top of the row candidate. Detecting crop candidates on the row candidates from the calculated crop size; Calculating the spacing between crops on the image from the spacing between the detected crop candidates; The likelihood of a row candidate is determined from the determined crop spacing. The information processing device is caused to execute a process of determining a law after excluding a law candidate determined to be a false positive using the determined certainty. program.
Citation Information
Patent Citations
Kaishitsuechirenkei kyojugotaino ratetsukusuno seizohoho
JP1976062890A
Device and method for predicting harvest
JP2003006612A
Crop growth level measuring system, crop growth level measuring method, crop growth level measuring program, and computer-readable recording medium recorded with the program
JP2003009664A
Image processor for working vehicle
JP2007043919A
Crop image processing program, crop image processing method and crop image processing system
JP2012198688A
