Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program that can acquire the average leaf inclination angle of crops included in a farm field with higher accuracy. [Solution] An information processing device of an embodiment includes an image information acquisition unit that acquires image information including an image of a crop present in a field photographed by a depth camera; a height acquisition unit that acquires information regarding the height of each pixel of the image based on the image information acquired by the image information acquisition unit; a tilt angle acquisition unit that corrects the height for any deviation in the angle of view of the image and acquires the tilt angle of the crop from the three-dimensional coordinates of an area consisting of vectors in two directions using a spatial vector that includes the difference in height between adjacent pixels included in the corrected image; and a derivation unit that derives the average tilt angle of the crop from multiple tilt angles obtained from the image, excluding predetermined outliers.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] The leaf inclination angle of a crop canopy is known to be an important trait related to crop growth, such as canopy photosynthesis. In this regard, there are known analytical devices that can measure a similar trait, the average leaf canopy inclination angle, along with leaf area, and techniques that capture leaf angle trends using distance data from a depth camera (see, for example, Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Meiwafosis Co., Ltd., "Plant Canopy Analyzer (LAI-2200C)", [online], [Retrieved April 1, 2025], Internet<https: / / meiwanet.co.jp / products / lai-2200c / > [Non-patent document 2] Uto et al, “Estimation of Leaf Angle Distribution Based on Statistical Properties of Leaf Shading Distribution.”, 2020 IEEE International Geoscience and Remote Sensing Symposium, pp. 5195-5198, October 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the method described in Non-Patent Document 1, if the crop has organs other than leaves, such as panicles, or if the leaf area is large, the accuracy of measuring the leaf area may be poor, which may similarly result in poor measurement accuracy for the average leaf canopy inclination angle, which is an additional measurement item. Furthermore, with the method described in Non-Patent Document 2, a method for deriving the inclination angle taking into account the camera's angle of view, etc., and a method for measuring the inclination angle for crops that have organs other than leaves, such as panicles, have not been established. As a result, there have been cases where the average leaf inclination angle of a crop canopy in a field could not be obtained with high accuracy.

[0005] The aspects of the present invention have been made in consideration of these circumstances, and one of their objectives is to provide an information processing device, an information processing method, and a program that can more accurately obtain the average leaf inclination angle of crops in a field. [Means for solving the problem]

[0006] The information processing device, the information processing method, and the program according to the present invention employ the following configuration. A first aspect of the information processing device of the present invention is an information processing device that includes an image information acquisition unit that acquires image information including an image of a crop present in a field photographed by a depth camera; a height acquisition unit that acquires information regarding the height of each pixel of the image based on the image information acquired by the image information acquisition unit; a tilt angle acquisition unit that corrects the height for any deviation in the angle of view of the image and acquires the tilt angle of the crop from the three-dimensional coordinates of an area consisting of vectors in two directions using a spatial vector that includes the difference in height between adjacent pixels included in the corrected image; and a derivation unit that derives the average of multiple tilt angles obtained from the image, excluding predetermined outliers, as the average leaf tilt angle of the crop.

[0007] The information processing device of a second aspect of the present invention further includes an exclusion unit that extracts edges from the image and excludes a portion of the image from the tilt angle acquisition unit's target for acquiring the tilt angle based on the extracted edges.

[0008] In the information processing device according to a third aspect of the present invention, the exclusion unit further excludes at least a portion of the ear of the crop included in the image from the target for obtaining the tilt angle.

[0009] The information processing device of a fourth aspect of the present invention further includes an exclusion unit that excludes at least the ground or water surface area of ​​the field contained in the image from the target for obtaining the inclination angle based on the information regarding the height.

[0010] In the information processing device according to a fifth aspect of the present invention, the image acquired by the image information acquisition unit is an image captured from a position a predetermined distance above the crop.

[0011] In the information processing device according to a sixth aspect of the present invention, the predetermined distance is a distance based on at least one of the type and lineage of the crop and the time period when the image is captured.

[0012] The information processing device according to a seventh aspect of the present invention further comprises a prediction unit that makes a prediction regarding the growth of the crop based on the average leaf inclination angle of the crop.

[0013] An information processing method according to an eighth aspect of the present invention is an information processing method in which a computer acquires image information including an image of a crop present in a field photographed by a depth camera, acquires information regarding the height of each pixel of the image based on the acquired image information, corrects the height for any deviation in the angle of view of the image, acquires the inclination angle of the crop from the three-dimensional coordinates of an area consisting of vectors in two directions using a spatial vector including the difference in height between adjacent pixels included in the corrected image, and derives the average of the inclination angles obtained from the image, excluding a predetermined number of outliers, as the average leaf inclination angle of the crop.

[0014] A ninth aspect of the present invention is a program that causes a computer to acquire image information including an image of a crop present in a field photographed by a depth camera, acquire information regarding the height of each pixel of the image based on the acquired image information, correct the height for any deviation in the angle of view of the image, acquire the inclination angle of the crop from the three-dimensional coordinates of an area consisting of vectors in two directions using a spatial vector including the difference in height between adjacent pixels included in the corrected image, and derive the average of the inclination angles obtained from the image, excluding a predetermined number of outliers, as the average leaf inclination angle of the crop. [Effects of the Invention]

[0015] According to the above-described aspects of the present invention, the average leaf inclination angle of the crops included in the field can be obtained with higher accuracy. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system 1 including an information processing apparatus 100 according to an embodiment. [Figure 2] FIG. 2 illustrates an example of the functional configuration of an acquisition unit 141. [Figure 3] FIG. 2 is a diagram showing an example of a visible image IM10. [Figure 4] FIG. 10 is a diagram showing an example of a depth image IM20. [Figure 5] FIG. 10 is a diagram for explaining acquisition of a tilt angle. [Figure 6] FIG. 10 is a diagram showing an example of an edge image IM30. [Figure 7] FIG. 10 is a diagram showing an example of rice leaves used as input. [Figure 8] 1 is a diagram showing the relationship between the angle (tilt angle) measured by the depth camera 10 and the actual tilt angle. [Figure 9] FIG. 1 is a diagram showing several rice plants of different lines (varieties). [Figure 10] FIG. 10 is a diagram showing the average leaf blade inclination angle of rice plant Rp derived at different times according to an embodiment. [Figure 11] FIG. 10 is a diagram showing an example of an image of a plurality of soybeans of different varieties photographed from above. [Figure 12] FIG. 12 is a diagram showing the leaf blade inclination angle of soybean when the above-described embodiment is applied to varieties A to F shown in FIG. [Figure 13] 4 is a flowchart showing an example of processing executed by the information processing device 100. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, with reference to the drawings, embodiments of an information processing device, an information processing method, and a program of the present invention will be described. Note that the following describes an embodiment in which the average leaf (leaf blade) inclination angle, etc. of a crop present in a field is derived. In the embodiment, a crop refers to a plant body having at least leaves. Examples of crops include, but are not limited to, rice, wheat, soybeans, etc. The following description will mainly focus on the case in which the crop is rice.

[0018] [Information Processing Systems] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system 1 including an information processing device 100 according to an embodiment. The information processing system 1 includes, for example, a depth camera 10 and the information processing device 100. The depth camera 10 and the information processing device 100 are communicably connected via, for example, a network NW. The network NW includes, for example, a Wi-Fi network, a cellular network, the Internet, a wide area network (WAN), a local area network (LAN), a provider device, a wireless base station, etc. The information processing system 1 may include multiple depth cameras 10 and / or multiple information processing devices 100.

[0019] The depth camera 10, for example, photographs one or more rice plants Rp (canopies) present in the farm field FA and acquires an image showing height data (depth) of the rice plants Rp. The height data (depth) is an example of information related to height, and may be rephrased as distance data from the photographing position of the depth camera 10 to the rice plants Rp. The depth camera 10 may also acquire a visible image (RGB image) of the rice plants Rp. In the embodiment, the depth camera 10 that acquires the image showing the height data (depth) may be provided separately from a digital camera capable of capturing visible images, or may be a digital camera with depth measurement capabilities. The depth camera 10 may be, for example, a RealSense (registered trademark) depth camera "D435" manufactured by Intel (registered trademark), but is not limited thereto.

[0020] The depth camera 10 captures an image of the rice plant Rp from a position a predetermined distance D1 above the top end U1 of the rice plant Rp toward the bottom (the −Z-axis direction in the figure). The predetermined distance D1 may be set based on, for example, at least one of the type of crop being photographed, its lineage (variety), and the time of day during which the image is captured. The predetermined distance D1 is adjusted, for example, so that a predetermined number of crops (or leaves) fit within the field of view. Furthermore, in the case of soybeans and other crops, the orientation of the leaves tends to change depending on the time of day (the direction of the sun). Therefore, adjusting the predetermined distance D1 depending on the time of day allows for a more accurate calculation of the average leaf inclination angle. When the crop is rice plant Rp, the predetermined distance D1 is preferably approximately 30 cm. When the crop is soybeans, the predetermined distance D1 is preferably approximately 50 cm, but is not limited thereto. When actually taking photographs using the depth camera 10, the predetermined distance D1 and the photographing height H from the ground (or water surface) may be measured, for example, using other measuring instruments, or may be adjusted based on the height (depth) obtained by the depth camera 10.

[0021] In addition, in the embodiment, for example, a worker may hold the depth camera 10 in his / her hand and photograph the rice plants Rp from above, or the depth camera 10 may be mounted on an air vehicle to photograph the rice plants Rp from above. The air vehicle may be, for example, a drone or a small unmanned aerial vehicle (UAV). The flight of the air vehicle is controlled by instruction information, etc., obtained from, for example, the information processing device 100 or another external device (e.g., a terminal device used by a worker near the field FA) connected via a network NW. Alternatively, the air vehicle may fly autonomously according to a program stored in an internal memory. The depth camera 10 mounted on the air vehicle photographs the field FA including one or more rice plants Rp from above the field FA, for example, every time a certain period of time or every time the air vehicle travels a certain distance. In addition, in the embodiment, a frame (frame body) for moving the depth camera 10 over the field FA may be installed, and the rice plants Rp may be photographed while the depth camera 10 is automatically or manually moved through the installed frame. In addition, when moving automatically, for example, a moving device capable of running on the frame is also installed.

[0022] The images (depth images, visible images) captured by the depth camera 10 are transmitted to the information processing device 100 via a network NW by a communication unit (not shown) included in the depth camera 10. At this time, the depth camera 10 may transmit image information including position information of the depth camera 10 to the information processing device 100. The position information is acquired, for example, by a positioning device (not shown) included in the depth camera 10. The positioning device measures the position based on signals received from satellites constituting a Global Navigation Satellite System (GNSS), such as a Global Positioning System (GPS). The position information includes information on the latitude and longitude of the location of the depth camera 10, and may also include altitude information. The image information may also include information on the date and time when the image was captured.

[0023] The information processing device 100 acquires image information captured by the depth camera 10 and derives the average tilt angle of the rice plants Rp based on the acquired image information. The information processing device 100 is, for example, a general-purpose PC (Personal Computer) or a server device. The information processing device 100 may also be a cloud computing system realized by a server device or a storage device. The information processing device 100 may also be a communication terminal (terminal device) such as a smartphone or a tablet terminal.

[0024] The functional configuration of the information processing device 100 will be specifically described below. The information processing device 100 includes, for example, a communication unit 110, an input unit 120, an output unit 130, a processing unit 140, and a storage unit 150. The processing unit 140 is realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of the components of the processing unit 140, which will be described later, may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory of the information processing device 100, or may be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the HDD or flash memory of the information processing device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0025] The storage unit 150 is realized by, for example, the above-mentioned storage device, an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory). The storage unit 150 stores, for example, image data 152, processing result data 154, programs, and various other information. The image data 152 is data in which image information (including not only images but also location information and date and time information) transmitted by the depth camera 10 is accumulated. The processing result data 154 is data processed by the processing unit 140 using the image data 152, etc.

[0026] The communication unit 110 communicates with the depth camera 10 and other external devices, for example, via a network NW. For example, the communication unit 110 receives image information including images captured by the depth camera 10. The communication unit 110 may also acquire various information from the external devices described above via the network NW. The communication unit 110 may also receive predetermined data (e.g., weather data associated with location information of the farm field FA) from the external devices via the network NW based on instructions from the processing unit 140. The communication unit 110 may also transmit processing result data 154, which is the result of processing by the processing unit 140, to the external devices via the network NW. If the depth camera 10 is mounted on an air vehicle, the communication unit 110 may also transmit information regarding the flight path, flight speed, altitude, and shooting location of the air vehicle to the air vehicle.

[0027] The input unit 120 includes various input devices such as a keyboard, a mouse, a touch panel, etc., and receives input information (such as instruction information) from a user of the information processing device 100. The input device may be, for example, a microphone, a switch, a lever, a button, etc.

[0028] The output unit 130 has, for example, a display unit and a speaker, and outputs predetermined information to at least one of them. The display unit is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The display unit displays a screen showing various information (text and images) in the embodiment. The speaker outputs predetermined sound. Note that the output unit 130 may be configured integrally with the input unit 120 as a touch panel. The output unit 130 outputs, for example, images showing the image data 152 and the processing result data 154 to the display unit, or outputs sound corresponding to the image displayed on the display unit to the speaker.

[0029] The processing unit 140 executes various processes, such as a process for deriving the average leaf inclination angle of rice plants Rp included in a predetermined region of the farm field FA using the image data 152, and a prediction process using the average leaf inclination angle. The predetermined region may be a region corresponding to the angle of view range captured by the depth camera 10, a partial region within the angle of view range, or the entire region of the farm field FA using multiple images. The average leaf inclination angle may more specifically be the average inclination angle of the leaf blade.

[0030] The processing unit 140 includes, for example, an acquisition unit 141, an exclusion unit 142, a derivation unit 143, a prediction unit 144, and a provision unit 145. Fig. 2 is a diagram showing an example of the functional configuration of the acquisition unit 141. The acquisition unit 141 includes, for example, an image information acquisition unit 141A, a height acquisition unit 141B, a relative three-dimensional coordinate acquisition unit 141C, and a tilt angle acquisition unit 141D.

[0031] The image information acquisition unit 141A acquires, for example, image information (e.g., a visible image of the rice plants Rp in the field photographed by the depth camera 10, a depth image, location information, date and time information), etc., received by the communication unit 110. FIG. 3 is a diagram showing an example of a visible image IM10. The visible image IM10 in FIG. 3 shows an RGB image (color image) of multiple rice plants Rp in the field FA photographed from above. FIG. 4 is a diagram showing an example of a depth image IM20. The depth image IM20 shown in FIG. 4 is a depth image of multiple rice plants Rp in the field FA photographed from above, and the darker the color (shade), the closer the distance from the depth camera 10. Note that the visible image IM10 and the depth image IM20 have the same angle of view (the photographed area and the distance (height) from the rice plants Rp). However, if they differ, the image information acquisition unit 141A may adjust the angle of view of the other image so that it becomes the same angle of view as that of the other image. 3 and 4, the angle of view is 640 pixels (px) wide and 360 pixels (px) high, with the same height (shooting height). These images may be stored in the storage unit 150 as image data 152.

[0032] The height acquisition unit 141B acquires height (depth) data (information about height) for each pixel of the depth image IM20 based on the color (shade) information of the depth image IM20. This makes it possible to acquire data indicating where the rice plant Rp is located within a rectangular parallelepiped of 640 pixels wide x 360 pixels high x shooting height H [m]. However, the depth image IM20 (including the visible image IM10) has deviations (e.g., distortions) due to the angle of view, and therefore does not become highly accurate three-dimensional data in real space as it is.

[0033] Therefore, the relative three-dimensional coordinate acquisition unit 141C corrects the deviation due to the angle of view and acquires the relative three-dimensional coordinates for each pixel. For example, the relative three-dimensional coordinate acquisition unit 141C acquires the relative three-dimensional coordinates for each pixel after correcting the deviation due to the angle of view using a correction function published by the manufacturer of the depth camera 10. The relative three-dimensional coordinate acquisition unit 141C may also acquire the relative three-dimensional coordinates by performing an orthogonal transformation process so that each of the multiple rice plants Rp included in the angle of view of the depth image IM20 is displayed with the correct size and position without distortion, as if viewed from directly above. Some of the functions of the relative three-dimensional coordinate acquisition unit 141C may also be included in the tilt angle acquisition unit 141D.

[0034] The tilt angle acquisition unit 141D acquires the tilt angle of each rice grain Rp included in the angle of view using the relative three-dimensional coordinates of each pixel acquired by the relative three-dimensional coordinate acquisition unit 141C. FIG. 5 is a diagram for explaining how tilt angles are acquired. For example, as shown in FIG. 5, the tilt angle acquisition unit 141D acquires a three-dimensional space vector obtained from the coordinate difference (height direction difference) between adjacent pixels in two directions, the X-axis direction (horizontal direction in the figure) and the Y-axis direction (vertical direction in the figure). The tilt angle acquisition unit 141D then determines the area (cross product) of a parallelogram (including cases where it is a rectangle) formed by the two vectors x and y in the X-axis direction and the Y-axis direction as the area of ​​that pixel S(i, j), and acquires the angle that the parallelogram makes with the vertical direction as the tilt angle θ(i, j) of the rice grain Rp. This tilt angle θ(i, j) is the tilt angle when, for example, the vertical direction (Z-axis direction) is 0 degrees (horizontal is 90 degrees).

[0035] For example, if the coordinate difference between adjacent pixels is less than a threshold, the tilt angle acquisition unit 141D groups the images of those pixels as images of the same organ (for example, leaves, stems, flowers, etc.), and performs a similar comparison with adjacent pixels to acquire the area S(i,j) of the same organ, and then acquires the tilt angle θ(i,j) of that organ from the overall tilt of that area (area). Note that the above variable i indicates the position in the X-axis direction, and the variable j indicates the position in the Y-axis direction. Furthermore, the variables i and j for the area S and tilt angle θ may be the position of the grouped area (for example, the center position of the area).

[0036] Furthermore, when acquiring the inclination angle θ for each rice plant Rp, the inclination angle acquisition unit 141D may acquire the inclination angle θ using information from which unnecessary information has been removed based on the processing result by the removal unit 142.

[0037] The exclusion unit 142 excludes a partial region of the image from the target for acquisition of the inclination angle θ by the inclination angle acquisition unit 141D. For example, the exclusion unit 142 excludes information unnecessary for acquiring the inclination angle θ of the rice plant Rp from the image. The unnecessary information is, for example, information other than the leaves (more specifically, the leaf blades), and includes at least the ground (or water surface) part of the field FA or the ears of the rice plant Rp. For example, the exclusion unit 142 excludes information on the relative three-dimensional coordinates of the ground (or water surface) and the relative three-dimensional coordinates of parts other than the leaves of the rice plant Rp (for example, the ears) from the relative three-dimensional coordinates for each pixel acquired by the relative three-dimensional coordinate acquisition unit 141C.

[0038] For example, the exclusion unit 142 can delete information about parts (three-dimensional coordinates) other than the leaves of the rice plant Rp by excluding relative three-dimensional coordinates (or corresponding pixel parts) having coordinates (height or distance) close to the ground (or water surface) based on the shooting height H. The exclusion unit 142 can also set a range of layer thickness (e.g., upper and lower limits of height (depth)) for acquiring the inclination angle θ of the rice plant Rp and exclude information about coordinates outside the set range. This allows only information about the relative three-dimensional coordinates of spatial regions with many leaves (few panicles or flowers) to be extracted, thereby enabling the inclination angle acquisition unit 141D to acquire the leaf inclination angle θ with higher accuracy. Furthermore, by setting the range of upper and lower limits of height, the leaf inclination angle θ can be acquired, for example, at both the upper and lower parts of the rice plant Rp, thereby acquiring a more precise inclination angle.

[0039] Alternatively (or in addition) to the above-described method, the exclusion unit 142 may extract an edge image based on the visible image IM10 and, based on the extracted edge image, exclude information about unnecessary portions (e.g., portions other than leaves, such as spikelets). FIG. 6 is a diagram showing an example of an edge image IM30. The exclusion unit 142 performs edge extraction processing on the visible image IM10 using a known algorithm, such as the Canny algorithm, to extract the edge image IM30. The edge extraction processing may involve, for example, smoothing the visible image IM10 to reduce noise, obtaining first-order differentials in the horizontal and vertical directions using a Sobel filter or the like, and obtaining the edge gradient (image brightness gradient) and direction from these two differential images, performing processing to suppress non-maximum values, or performing threshold processing using hysteresis.

[0040] The exclusion unit 142 then extracts the ear portion (area) from the edge image IM30 and excludes the extracted area from the acquisition target for the tilt angle θ. For example, in the case of rice Rp, the ear is formed by a cluster of small grains, so in order to exclude the grain portion, if the area formed by the edge (the area on the top and bottom (vertical direction of the image) and the left and right (horizontal direction of the image)) is less than a predetermined number of pixels, the information of that area is excluded from the acquisition target for the tilt angle θ. The predetermined number of pixels is, for example, a pixel (e.g., about 2 pixels) that is set in advance for the size of the grain, but may be adjusted depending on the type of crop, growth status, and shooting height H [m]. This process makes it possible to exclude information on pixels (relative three-dimensional coordinates) that capture the ear from the acquisition target for the tilt angle.

[0041] Furthermore, instead of (or in addition to) the above-described method, the exclusion unit 142 may acquire color information of each pixel in the visible image IM10, and if the acquired color is a predetermined color (for example, the color of the ground, the color of the water surface, the color of the ears of flowers, or the color of the flowers), exclude the portion corresponding to that pixel from the objects to be acquired for the tilt angle. Furthermore, the exclusion unit 142 may exclude data of relative three-dimensional coordinates of areas corresponding to flowers and stems in the image from the objects to be acquired for the tilt angle, based on preset shape information of the flowers and stems.

[0042] The tilt angle acquisition unit 141D can acquire the leaf tilt angle of the rice plant Rp with higher accuracy by acquiring the tilt angle using the relative three-dimensional coordinates from which unnecessary information has been removed by the above-mentioned method using the exclusion unit 142. Information related to the tilt angle θ acquired by the tilt angle acquisition unit 141D may be stored in the storage unit 150 as processing result data 154.

[0043] The derivation unit 143 derives the average leaf tilt angle for the depth image by excluding predetermined outliers from the leaf tilt angles obtained from the entire depth image. For example, the derivation unit 143 excludes the tilt angles θ in the top 10% and bottom 10% of the leaf tilt angles acquired by the tilt angle acquisition unit 141D as outliers. This makes it possible to exclude, for example, tilt angles of leaves bent due to external factors or tilt angles acquired as abnormal values ​​by processing. After excluding the outliers, the derivation unit 143 uses the remaining information to derive the mean leaf tilt angle (Mean Leaf Angle) using the following formula (1) (weighted average) or the like. Average leaf inclination angle=Σ(S(i,j)×θ(i,j)) / ΣS(i,j) … (1) This allows the average leaf inclination angle to be quantified and obtained.

[0044] The prediction unit 144 makes a prediction regarding the growth of the rice plants Rp based on the average leaf inclination angle derived by the derivation unit 143. For example, the prediction unit 144 may predict a subsequent change in the average leaf inclination angle based on a time-series change in the average leaf inclination angle derived at different times at the same position (field FA). The prediction unit 144 may also predict the overall average leaf inclination angle of the field FA based on the average leaf inclination angle for multiple depth images captured at multiple positions in the field FA. The prediction unit 144 may also predict the yield of the rice plants Rp for the field FA based on the predicted growth status.

[0045] The prediction unit 144 may also predict the relationship between leaf inclination and photosynthesis based on the average leaf inclination angle of the rice plants Rp. For example, the prediction unit 144 predicts the amount of photosynthesis of the rice plants Rp based on the average leaf inclination angle and date and time information contained in the image, weather information obtained from an external source, leaf size obtained from the image, and the like. Note that the weather information is not limited to actual weather, and average information based on the time (season) or statistical results may also be used. Furthermore, the leaf size may be measured by measuring the actual size from the image, or the average leaf size according to the growing season of the rice plants Rp may also be used. The prediction unit 144 may also predict the current or future growth status of the rice plants Rp based on the amount of photosynthesis.

[0046] The data predicted by the prediction unit 144 may be stored in the storage unit 150 as processing result data 154 .

[0047] The providing unit 145 generates images, sounds, etc. that indicate the image data 152 and the processing result data 154, and outputs the generated data to the output unit 130 or provides the generated data to an external device via the communication unit 110 over the network NW.

[0048] [Comparison of tilt angle in embodiment and actual measured value] Next, a comparison between the inclination angle and the actual measured value in this embodiment will be described using the drawings. FIG. 7 is a diagram showing an example of a rice leaf used as input. In the example of FIG. 7, an image IM40 of one leaf of rice plant Rp is shown for reference. FIG. 8 is a diagram showing the relationship between the angle (inclination angle) measured by the depth camera 10 and the actual inclination angle. In the example of FIG. 8, the inclination angle of the leaf shown in FIG. 7 is changed multiple times (six times in the example of FIG. 8), and the relationship between the leaf inclination angle acquired from the image (depth image) captured by the depth camera 10 and the actual leaf inclination angle (actual measured value) measured by the inclinometer is shown. As shown in FIG. 8, the inclination angle obtained from the depth image captured using the depth camera 10 of this embodiment is highly correlated with the actual measured value (in other words, the leaf inclination angle obtained from the depth image is close to the actual measured inclination angle (with a small error)). (In the example of FIG. 8, the regression equation is y = 0.957x + 2.0863, and the coefficient of determination R 2 =0.9763).

[0049] [Differences in leaf (leaf blade) inclination angle depending on lineage (variety)] Fig. 9 is a diagram showing a plurality of rice plants of different lines (varieties). In the example of Fig. 9, images of five rice plants Rp (lines a to e) of different lines (varieties) are shown. In the example of Fig. 9, images taken from the side are shown to clearly show the differences between the lines (varieties), but in the embodiment, as described above, the rice plants Rp are photographed from above using the depth camera 10.

[0050] FIG. 10 is a diagram showing the average leaf blade inclination angle of rice plants Rp calculated at multiple different times according to an embodiment. The example in FIG. 10 shows the average leaf blade inclination angles at four different times for five types of rice plants Rp, lines a to e. In the example in FIG. 10, time T1 is the earliest, followed by times T2, T3, and T4, in that order. By acquiring the average leaf blade inclination angle using the method described in the above-described embodiment, it is possible to grasp the trend of leaf blade inclination angle over time for each line, as shown in FIG. 10, and to grasp the differences in the trends among the lines. Therefore, according to the embodiment, it is possible to make more accurate predictions regarding growth for each line (variety) and analyze differences in growth conditions among lines (varieties).

[0051] [About crops other than rice] Crops other than rice may be applicable to the embodiments. FIG. 11 is a diagram showing an example of an image of a plurality of soybean varieties photographed from above. The example of FIG. 11 shows images of leaves of six different varieties of soybean (varieties A to F) photographed from above as an example of a crop. FIG. 12 is a diagram showing the leaf blade inclination angle of soybean when the above-described embodiment is applied to varieties A to F shown in FIG. 11. The angles shown in FIG. 12 indicate the inclination angle when vertical is 0 degrees (horizontal is 90 degrees). The example of FIG. 12 also shows an analysis of variance (ANOVA) for varieties D, E, and F. By applying the embodiment, as shown in FIG. 12, it was possible to numerically determine that the average leaf blade inclination angle tends to be closer to vertical (more vertical than the reference varieties A and D) for varieties B, C, E, and F, which were previously confirmed by observation to have leaves that tend to be closer to vertical.

[0052] As described above, according to the embodiment, the average inclination angle of the leaves (leaf blades) for each crop and each line (variety) can be more accurately determined with a simple configuration. Furthermore, by quantifying the average inclination angle of the leaf blades, the growth status of the crops can be more accurately determined.

[0053] [Processing flowchart] Next, an example of processing executed by the information processing device 100 will be described using a flowchart. FIG. 13 is a flowchart illustrating an example of processing executed by the information processing device 100 according to the embodiment. In the example of FIG. 13, the image information acquisition unit 141A acquires image information of a crop (such as rice) captured by the depth camera 10 (step S100). Next, the height acquisition unit 141B acquires height data for each pixel of the acquired image information (e.g., a depth image) (step S110). Next, the relative three-dimensional coordinate acquisition unit 141C performs misalignment correction on the height data of the image and acquires relative three-dimensional coordinates for each pixel included in the misalignment-corrected image (step S120). Next, the exclusion unit 142 removes coordinate information of unnecessary parts (e.g., parts other than leaves) from the relative three-dimensional coordinates (step S130). Next, the inclination angle acquisition unit 141D acquires the leaf (leaf blade) inclination angle based on the information of the relative three-dimensional coordinates from which the unnecessary parts have been removed (step S140).

[0054] Next, the derivation unit 143 excludes a preset outlier from the plurality of leaf tilt angles included in the image region (step S150). Next, the derivation unit 143 derives an average leaf tilt angle in the image region using the remaining leaf tilt angles after excluding the outliers (step S160).

[0055] Next, the prediction unit 144 predicts information about growth based on the average tilt angle (step S170). Next, the provision unit 145 outputs the results of various processes executed by the information processing device to the output unit 130 or the like (step S180). This ends the processing of this flowchart.

[0056] [Variations] In this embodiment, the depth camera 10 and the information processing device 100 may be integrated. In addition, in this embodiment, the average ear inclination angle may be derived instead of (or in addition to) the average leaf inclination angle. This allows for more accurate prediction of the crop yield, etc., of the entire farm field FA from the average ear inclination angle. In addition, in this embodiment, the depth camera 10 may not acquire visible information, but may acquire only images showing crop height data (depth). In this case, the exclusion unit 142 may not perform the exclusion process using the visible image IM10 (e.g., the exclusion process using an edge image) among the various exclusion processes described above. This allows for the average leaf inclination angle of crops to be derived using the above-described method even with a less expensive depth camera 10 that does not acquire the visible image IM10.

[0057] As described above, according to the information processing device 100 of the embodiment, it is possible to obtain the average leaf inclination angle of the crops included in the field more accurately by including an image information acquisition unit 141A that acquires image information including images of crops present in the field photographed by the depth camera 10, a height acquisition unit 141B that acquires information regarding the height of each pixel of the image based on the image information acquired by the image information acquisition unit 141A, a tilt angle acquisition unit 141D that corrects the height for deviations in the angle of view of the image and acquires the inclination angle of the crop from the three-dimensional coordinates of an area consisting of vectors in two directions using a spatial vector that includes the difference in height between adjacent pixels included in the corrected image, and a derivation unit 143 that derives the average of the inclination angles obtained from the image, excluding predetermined outliers, as the average leaf inclination angle of the crop.

[0058] For example, according to an embodiment, the average leaf inclination angle of a crop canopy can be quickly measured in an outdoor field using an inexpensive depth camera. Specifically, according to an embodiment, the crop inclination angle is derived from the area (cross product) of a parallelogram formed by a spatial vector of three-dimensional coordinate data including height (depth) captured by the depth camera. Furthermore, by combining a technique for removing pixel areas based on edges detected by the Canny algorithm and pixel areas of the water surface (ground) based on height data during derivation, the average leaf inclination angle of a crop canopy can be more accurately derived using an inexpensive camera (configuration) that can be used in outdoor fields.

[0059] Furthermore, according to the embodiment, the leaf inclination angle is obtained based on depth data captured from above the crop, so that the angle of the upper leaves, which are important for photosynthesis, can be accurately obtained even when the amount of leaves is large. Furthermore, according to the embodiment, organs other than leaves, such as panicles (unnecessary parts), can be removed by image analysis (edge ​​detection, etc.). Furthermore, according to the embodiment, the absolute value of the angle can be calculated based on the internal information of the depth camera by vector calculation using a function that calculates the three-dimensional coordinates of each pixel, which is a known technology.

[0060] Furthermore, according to the embodiment, the average leaf inclination angle can be quantified and acquired, allowing for more detailed analysis of differences and trends in inclination angle for each crop type and strain. Furthermore, according to the embodiment, it is expected that the relationship between leaf inclination and photosynthesis can be investigated and that it can be used for crop growth diagnosis. Therefore, this embodiment can be directly transferred to, for example, fields related to measurement equipment in the life sciences. Furthermore, according to the embodiment, technology transfer is possible with just an information processing device that can be brought into the field and a commercially available depth camera, making direct technology transfer to researchers who perform measurements possible.

[0061] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0062] 1...information processing system, 10...depth camera, 100...information processing device, 110...communication unit, 120...input unit, 130...output unit, 140...processing unit, 141...acquisition unit, 141A...image information acquisition unit, 141B...height acquisition unit, 141C...relative three-dimensional coordinate acquisition unit, 141D...tilt angle acquisition unit, 142...exclusion unit, 143...derivation unit, 144...prediction unit, 145...providing unit, 150...storage unit

Claims

1. an image information acquisition unit that acquires image information including an image of a crop present in a field photographed by a depth camera; a height acquisition unit that acquires information about heights of each pixel of the image based on the image information acquired by the image information acquisition unit; an inclination angle acquisition unit that corrects the height for a deviation in the angle of view of the image, and acquires an inclination angle of the crop from three-dimensional coordinates of an area formed by vectors in two directions using a spatial vector including a difference in height between adjacent pixels included in the corrected image; a derivation unit that derives an average of the inclination angles obtained from the image, excluding predetermined outliers, as an average leaf inclination angle of the crop; An information processing device comprising:

2. an exclusion unit that extracts an edge from the image and excludes a partial area of ​​the image from a target for obtaining the tilt angle by the tilt angle obtaining unit based on the extracted edge; The information processing device according to claim 1 .

3. the exclusion unit excludes at least a portion of the ear of the crop included in the image from a target for obtaining the tilt angle. The information processing device according to claim 2 .

4. an exclusion unit that excludes at least a ground or water surface area of ​​the field included in the image from a target for obtaining the inclination angle based on the information about the height; The information processing device according to claim 1 .

5. the image acquired by the image information acquisition unit is an image captured from a position a predetermined distance above the crop. The information processing device according to claim 1 .

6. The predetermined distance is a distance based on at least one of the type of the crop, the lineage, and the time period when the image is captured. The information processing device according to claim 5 .

7. Further, a prediction unit is provided that predicts the growth of the crop based on the average leaf inclination angle of the crop. The information processing device according to claim 1 .

8. The computer Acquire image information including an image of the crops present in the field photographed by the depth camera; acquiring information about the height of each pixel of the image based on the acquired image information; correcting the height for a deviation in the angle of view of the image, and using a spatial vector including the difference in height between adjacent pixels included in the corrected image, obtaining the inclination angle of the crop from the three-dimensional coordinates of an area consisting of vectors in two directions; deriving an average of the inclination angles obtained from the image, excluding predetermined outliers, as an average leaf inclination angle of the crop; Information processing methods.

9. On the computer, Acquiring image information including an image of a crop present in a field photographed by a depth camera; acquiring information about the height of each pixel of the image based on the acquired image information; correcting the height for a deviation in the angle of view of the image, and obtaining the inclination angle of the crop from the three-dimensional coordinates of an area formed by vectors in two directions using a spatial vector including a difference in height between adjacent pixels included in the corrected image; deriving an average of the inclination angles obtained from the image, excluding predetermined outliers, as the average leaf inclination angle of the crop; program.

Images