Yield estimation system, yield estimation device, and yield estimation method

Abstract

To provide a yield estimation system, device, and method which are capable of estimating yield by using three-dimensional image information before harvesting each fruit from plants and trees.SOLUTION: A yield estimation system comprises: an imaging apparatus which images fruiting plants and trees; and a yield estimation device which acquires image information DS from the imaging apparatus, where the image information includes three-dimensional image information DSb. The yield estimation device comprises: a fruit specifying part which specifies a fruit from each object expressed by image information; a volume estimation part which estimates the volume of the fruit by using the three-dimensional image information; a weight estimation part which estimates the weight of the fruit from the volume of the fruit; and a yield calculation part which calculates a numerical value obtained by totalizing the weight of a plurality of fruits.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to yield estimation System, yield estimation device, and yield estimation method and pertains to it.

Background Art

[0002] Conventionally, technologies for grasping (estimating) various information regarding fruits before harvesting have been proposed. For example, in the technology of Patent Document 1, the ripeness of the fruit can be estimated in advance from the image information of the fruit before harvesting.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, the image information is two-dimensional image information and does not include three-dimensional image information. With the above prior art, the weight of each fruit before harvesting cannot be accurately estimated. Therefore, there was a situation where the fruit yield could not be accurately estimated before harvesting. Considering the above situation, an object of the present invention is to enable accurate estimation of the fruit yield before harvesting.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides a yield estimation system comprising a photographing device for photographing fruit-bearing plants and a yield estimation device for acquiring image information from the photographing device, wherein the image information includes three-dimensional image information, and the yield estimation device comprises a fruit identification unit for identifying fruits from each object represented by the image information, a volume estimation unit for estimating the volume of a fruit using the three-dimensional image information, a weight estimation unit for estimating the weight of a fruit from its volume, and a yield calculation unit for calculating a numerical value obtained by summing the weights of multiple fruits. [Effects of the Invention]

[0006] According to the present invention, it becomes possible to estimate the yield of each fruit using three-dimensional image information before harvesting them from plants. [Brief explanation of the drawing]

[0007] [Figure 1] This is a hardware configuration diagram of the yield estimation system. [Figure 2] This is a functional block diagram of the yield estimation system. [Figure 3] This is a diagram illustrating specific examples of how to photograph fruit. [Figure 4] This is a diagram illustrating specific examples of the position in which to photograph fruit. [Figure 5] This is a diagram to illustrate specific examples of fruit images. [Figure 6] This is a diagram illustrating the configuration for estimating partial volumes. [Figure 7] This diagram illustrates specific examples of coefficients based on fruit type. [Figure 8] This is a flowchart for the yield estimation process. [Modes for carrying out the invention]

[0008] <First Embodiment> Figure 1 is a hardware configuration diagram of the yield estimation system 1. As shown in Figure 1, the yield estimation system 1 includes a computer 10, a conveying device 20, a camera 30, and a wire rope 40. According to the yield estimation system 1 of this embodiment, the yield of each fruit F can be estimated before harvesting from each tree T. Figure 1 shows an excerpt of one of the trees T in an orchard.

[0009] Figure 1 shows a specific example of estimating the harvest yield of mangoes among fruits F. However, this embodiment is configured to also estimate the harvest yield of fruits F other than mangoes. In this invention, "plants" is a concept that includes both "grasses" and "trees." Therefore, in this invention, "fruits" is a concept that includes both "fruits of woody plants" and "fruits of herbaceous plants." Furthermore, in this invention, "fruits" is a concept that includes not only "fruits" such as mangoes, but also "vegetables" such as tomatoes.

[0010] However, the present invention includes a configuration that estimates both the yield of "fruits of woody plants" and the yield of "fruits of herbaceous plants," as well as a configuration that estimates only the yield of "fruits of woody plants," and a configuration that estimates only the yield of "fruits of herbaceous plants." Similarly, the present invention includes a configuration that estimates both the yield of "fruits" and the yield of "vegetables," as well as a configuration that estimates only the yield of "fruits," and a configuration that estimates only the yield of "vegetables."

[0011] Computer 10 comprises a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). In this embodiment, computer 10 is a portable computer (for example, a tablet device). However, a desktop PC or a notebook PC may also be used as computer 10.

[0012] The HDD of computer 10 stores various data, including the yield estimation program PG. The CPU of computer 10 executes the yield estimation program PG to realize various functions described later (such as the yield calculation unit 105). The RAM of computer 10 temporarily stores various information that the CPU references when executing programs. The ROM of computer 10 also stores various information non-volatilely. Note that the yield estimation program PG may be stored in a location other than the HDD.

[0013] As shown in Figure 1, the conveying device 20 can have a camera 30 attached to its lower end. The conveying device 20 is also configured to be suspended from a wire rope 40. The conveying device 20 is equipped with rollers that are rotated by a motor, and when the conveying device 20 is suspended from the wire rope 40, the rollers come into contact with the wire rope 40. With this configuration, the conveying device 20 moves along the wire rope 40 by rotating the rollers.

[0014] As will be described in detail later, the transport device 20 transports the camera 30 to the shooting position (directly below M in Figure 3). Note that the transport device 20 only needs to be capable of transporting the camera 30 to the shooting position and is not limited to the above example. Furthermore, the transport device 20 and the camera 30 may be configured as an integrated unit (the transport device 20 possesses the functions of the camera 30).

[0015] Camera 30 captures images of trees T (including fruits F) and generates image information DS. Specifically, camera 30 consists of a color (RGB) camera 30a and a depth camera 30b. The color camera 30a generates two-dimensional image information DSa, which represents a two-dimensional color image. On the other hand, the depth camera 30b generates three-dimensional image (distance image) information DSb, which includes depth information indicating the distance to the subject. For example, the three-dimensional image information DSb is expected to represent information showing a point cloud image captured using LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) technology.

[0016] In addition, the camera 30 is provided with an inclination sensor. The above inclination sensor detects the magnitude of the inclination of the shooting direction with respect to the vertical direction (pitch angle θp and roll angle θr described later; refer to FIGS. 4(b) and 4(c)). The camera 30 generates "inclination information" indicating the magnitude of the inclination detected by the inclination sensor during shooting. The above inclination information is used when adjusting (correcting) the inclination of the fruit image (Gf in FIG. 5(b) described later) showing the fruit F.

[0017] In this embodiment, both the color camera 30a and the depth camera 30b shoot the tree T substantially simultaneously (from the same position). When the tree T is shot, the two-dimensional image information DSa and the three-dimensional image information DSb indicating the tree T are associated to generate the image information DS. Further, the image information DS includes the inclination information generated during shooting. When the camera 30 generates the image information DS, it transmits the image information DS to the computer 10. Specifically, the camera 30 shoots each tree T from a plurality of shooting positions, and generates the image information DS every time it shoots. In the above configuration, a plurality of pieces of image information DS generated at different shooting positions are transmitted to the computer 10.

[0018] Although it will be described in detail later, in this embodiment, the harvest amount of the fruit F is estimated using the image information DS. Specifically, using the image information DS, the volume V of each fruit F is estimated. Further, the computer 10 stores the density E of the fruit F in advance, multiplies the estimated volume V by the density E, and calculates the weight W of the fruit F. The sum of the weights W of each fruit F is calculated as the harvest amount Y.

[0019] As shown in FIG. 1, the camera 30 transmits the image information DS to the computer 10 by wireless communication. However, a configuration in which the image information DS is transmitted by wired communication may also be used. Further, a removable memory may be provided in the camera 30, and the image information DS may be stored in the memory. In the above configuration, by attaching the memory to the computer 10, the computer 10 can take in the image information DS.

[0020] As shown in Figure 1, the wire rope 40 is stretched at a predetermined height above the ground. Specifically, as shown in Figure 1, poles P1 and P2 are installed in the orchard. One end of the wire rope 40 is attached to pole P1 and the other end is attached to pole P2, and it is stretched approximately parallel to the ground.

[0021] However, the position of the wire rope 40 only needs to be such that each tree T (fruit F) is photographed by the camera 30, and may be changed as appropriate depending on the position of the trees T. Also, in this embodiment, the wire rope 40 is used as the "track section" of the present invention, but the "track section" is not limited to the above example. For example, a rail may be used as the "track section". Similar to the wire rope 40, the rail is supported at a position that allows the camera 100 to be transported to a position where each tree T can be photographed. In the above configuration, a transport device 20 that travels (moves) on the rail may be used.

[0022] Figure 2 is a functional block diagram of the yield estimation device 100. The yield estimation device 100 includes an image information acquisition unit 101, a fruit identification unit 102, a volume estimation unit 103, a weight estimation unit 104, and a yield calculation unit 105. For example, the above-mentioned computer 10 functions as the yield estimation device 100 by executing the weight estimation program PG. In addition to the yield estimation device 100, the yield estimation system 1 includes a transport unit 200, a photography device 300, and a track unit 400 (see Figure 3 below). For example, the above-mentioned transport device 20 functions as the transport unit 200, the camera 30 functions as the photography device 300, and the wire rope 40 functions as the track unit 400.

[0023] The image information acquisition unit 101 of the yield estimation device 100 acquires (receives) image information DS (DSa, DSb) from the imaging device 300. For example, when the imaging device 300 takes an image, the image information DS is automatically transmitted to the yield estimation device 100. Alternatively, the yield estimation device 100 may be configured to transmit a request signal, and the imaging device 300 may transmit the image information DS upon receiving this request signal.

[0024] The fruit identification unit 102 identifies fruit F from each object (fruit, branch, leaf, etc.) shown in the image information DS. Specifically, the image shown in the image information DS displays not only fruit F but also branches and leaves of tree T. The fruit identification unit 102 identifies fruit F from each object in the 3D image shown in the 3D image information DSb.

[0025] In this embodiment, the type of fruit F (for example, mango) for which the yield Y is calculated is pre-inputted to the yield estimation device 100. The yield estimation device 100 (fruit identification unit 102) segments (divides) each object included in the 2D image shown by the 2D image information DSa before identifying the fruit F in the 3D image shown by the 3D image information DSb. Artificial intelligence (AI) technology may be employed for the above segmentation.

[0026] For example, each object in a 2D image is segmented using a trained FCN (Fully Convolutional Network) (for example, the technology described in Japanese Patent Publication No. 2022-29169 can be employed). Furthermore, each object (fruit, branch, leaf, etc.) in the 2D image shown by the 2D image information DSa is classified. The yield estimation device 100 identifies the fruit F for which the yield Y is calculated from each classified object. Specifically, if the 2D image contains multiple fruits F, all fruits F are identified. Note that the method for identifying fruit F from the 2D image shown by the 2D image information is not limited to the above example.

[0027] The yield estimation device 100 divides the 3D image shown by the 3D image information DSb into individual objects. Furthermore, the yield estimation device 100 associates each object in the 2D image with each object in the 3D image based on the position of each object in the 2D image shown by the 2D image information Dsa and the position of each object in the 3D image shown by the 3D image information DSb.

[0028] As described above, the object representing fruit F is identified from among the objects in the 2D image shown by the 2D image information DSa. The yield estimation device 100 identifies the object in the 3D image corresponding to the object in the 2D image identified as fruit F as fruit F. Hereafter, for the sake of explanation, fruit F (object) in the 3D image may be referred to as "fruit image Gf".

[0029] The volume estimation unit 103 estimates the volume V of the fruit F using the 3D image information DSb. Specifically, it estimates the volume V of the fruit F using the fruit image Gf from the 3D image shown by the 3D image information DSb. The above configuration will be described in detail later using Figures 6(a) to (d).

[0030] The weight estimation unit 104 estimates the weight W of fruit F from the volume V of fruit F. Specifically, the yield estimation device 100 estimates the volume V of fruit F from the 3D image information DSb as described above. The yield estimation device 100 also pre-stores the density E for each fruit F for which the yield Y can be estimated. The density E stored by the yield estimation device 100 is obtained, for example, using a sample fruit F. The yield estimation device 100 stores the result of multiplying the estimated volume V of fruit F by the density E as the weight W of fruit F (W = V × E).

[0031] The yield calculation unit 105 calculates a value by summing the weights W of multiple fruits F. Specifically, the yield estimation device 100 photographs each tree T in the orchard and estimates the weight W of all fruits F on that tree T. The yield estimation device 100 also sums up all the estimated weights W and stores the total result as the yield Y. The yield estimation device 100 may also be equipped with an image display device to display the yield Y. Alternatively, instead of (or in addition to) calculating the yield Y of fruits F on all photographed trees T, the device may be configured to calculate the yield Y for each tree T (photographing location).

[0032] Figure 3 is a diagram illustrating a specific example of a method for photographing fruit F. Figure 3 assumes a view of the orchard from directly above. Figure 3 shows each tree T(1~n) in the orchard. Figure 3 also shows the components of the yield estimation system 1, namely the transport unit 200, the photography device 300, and the wire rope (track unit) 400.

[0033] As shown in Figure 3, each tree T is arranged in a single row. The wire rope 400 is attached to each pole P(1, 2) at both ends and stretched in the air along each tree T. Specifically, the wire rope 400 is positioned diagonally above each tree T (see Figure 4(b) below). The position of the wire rope 400 can be changed as needed, as long as it allows for the photography of each fruit F of each tree T. For example, the wire rope 400 may be positioned directly above the trees T.

[0034] In the following explanation, the direction opposite to the direction of gravity may be referred to as the "Z-axis direction." Also, the direction perpendicular to the Z-axis direction may be referred to as the "X-axis direction." The above X-axis direction is also perpendicular to the shooting direction Sc of the shooting device 300 (the direction in front of the shooting device 300). In this embodiment, for the sake of simplicity, we assume that the wire rope 400 is stretched in the X-axis direction. The above X-axis direction is approximately parallel to the direction in which the transport unit 200 (shooting device 300) moves (the direction of the white arrow in Figure 3). Furthermore, the direction perpendicular to both the X-axis direction and the Z-axis direction may be referred to as the "Y-axis direction." As shown in Figure 3, each tree T is located on the Y-axis side as viewed from the shooting device 300. Note that Figure 3 shows the shooting direction Sc of the shooting device 300.

[0035] As shown in Figure 3, each mark section M(1~n) is provided on the wire rope 400. Each mark section M is configured to be distinguishable from other parts of the wire rope 400. Specifically, each mark section M has a different color from other parts of the wire rope 400. Also, as shown in Figure 3, each mark section M is provided corresponding to each tree T. For example, mark section M1 corresponds to tree T1, mark section M2 corresponds to tree T2, mark section M3 corresponds to tree T3, and so on, with mark section Mn corresponding to tree Tn.

[0036] When the transport unit 200 is positioned at the mark unit M, each mark unit M is positioned so that each fruit F of the tree T corresponding to the mark unit M can be photographed by the photographing device 300. For example, in the specific example shown in Figure 3, we assume that the transport unit 200 is positioned at the mark unit M1. In this case, each fruit F of the tree T1 can be photographed by the photographing device 300 (each fruit F is located inside the photographing range). However, depending on the tree T, it may not be possible to photograph all the fruits F at once (from one direction). In this case, multiple mark units M may be provided corresponding to one tree T, and the tree T may be photographed from multiple photographing positions. Alternatively, if the fruits F of multiple trees T can be photographed at once, a single mark unit M may be provided corresponding to the multiple trees T.

[0037] The transport unit 200 of this embodiment can be operated remotely using a remote control R. For example, by operating the remote control R as appropriate, the transport unit 200 can be moved along the wire rope 400. The user operates the remote control R so that the transport unit 200 is positioned at the markings M. For example, the user operates the remote control R sequentially so that the transport unit 200 stops in the order of markings M1, M2...Mn. However, the transport unit 200 may be configured to automatically stop and move repeatedly. For example, a configuration in which a sensor for detecting the markings M is provided on the transport unit 200 is conceivable. In the above configuration, when a marking M is detected, the transport unit 200 automatically stops for the time required for shooting, and then the transport unit 200 automatically starts moving to the next marking M.

[0038] The imaging device 300 of this embodiment can be operated remotely. For example, when the remote control R described above is operated appropriately, the imaging device 300 captures an image (2D image, 3D image), and image information DS indicating the image is generated. With the above configuration, it becomes possible to remotely instruct the imaging device 300 to photograph the tree T while the transport unit 200 is stopped at the mark unit M. However, the means for remotely operating the imaging device 300 and the means for remotely operating the transport unit 200 may be separate.

[0039] In this embodiment, first, the transport unit 200 is stopped at the mark unit M1 to capture an image of the first tree T1. Then, by stopping the transport unit 200 in the order of mark unit M2, mark unit M3... mark unit Mn, images of trees T2 through Tn can be captured. In addition, each time a tree T(1~n) is captured, image information DS(DSa, DSb) is transmitted from the imaging device 300 to the harvest yield estimation device 100, and the harvest yield estimation device 100 calculates (estimates) the harvest yield Y. The harvest yield Y is the amount of fruit F harvested from multiple trees T from tree T1 to tree Tn.

[0040] As described above, in this embodiment, by transporting the imaging device 300, multiple trees T can be photographed with a single imaging device 300. Let's assume a proportional arrangement where the imaging device 300 is fixed at a predetermined position. In this proportional arrangement, one imaging device 300 is required to photograph one tree T. Therefore, when estimating the yield of fruit F harvested from multiple trees T, multiple imaging devices 300 are required. In this proportional arrangement, there is a disadvantage that the cost of the imaging devices 300 becomes excessive when there are many trees T. In this embodiment, since multiple trees T can be photographed with a single imaging device 300, there is an advantage in that the cost of the imaging devices 300 can be suppressed compared to the proportional arrangement described above.

[0041] Incidentally, let's assume a configuration in which a remotely controlled drone is used as the transport unit 200 (hereinafter referred to as "proportionality"). In the above proportionality, a camera device 300 is attached to the drone, and the camera device 300 is transported to the shooting position. However, depending on the drone operator (especially a beginner), there may be a problem in that the camera device 300 cannot be transported to a suitable shooting position. This problem tends to become apparent when calculating the harvest yield Y repeatedly (for example, every day) before harvesting the fruit F.

[0042] In this embodiment, the imaging device 300 can be transported to the imaging position by a wire rope 400 that is supported (stretched) at a predetermined position. Therefore, the imaging device 300 can be easily transported repeatedly to a suitable imaging position, and there is an advantage in that the disadvantages described above are suppressed compared to, for example, the proportional method described above. However, the present invention does not exclude the proportional method described above.

[0043] Figures 4(a) and 4(b) illustrate specific examples of the shooting position of the tree T. Figures 4(a) and 4(b) assume that the transport unit 200 is located at the marked area M. Furthermore, Figures 4(a) and 4(b) show only the tree T corresponding to the marked area M where the transport unit 200 is located, omitting the other trees T. In other words, only the subject tree T is shown.

[0044] Figure 4(a) is a view of the imaging device 300 (transport unit 200) from the tree T side (Y-axis direction side). As shown in Figure 4(a), when the transport unit 200 is located at the mark M, the position of the tree T on the X-axis corresponding to the mark M and the position of the imaging device 300 on the X-axis are approximately the same. That is, the position of the tree T being photographed on the X-axis and the position of the imaging device 300 on the X-axis are approximately the same. However, it is sufficient that the fruit F on the tree T is photographed, and the position of the tree T being photographed on the X-axis and the position of the imaging device 300 on the X-axis do not necessarily have to be approximately the same.

[0045] As shown in Figure 4(a), the shooting position for each tree T is above the tree T (towards the Z-axis direction). Specifically, if the height of the tree T from the ground is "height Ht" and the height from the ground to the shooting position (shooting device 300) is "height Hc", then "height Hc" is higher than "height Ht" (Hc > Ht). In this embodiment, regardless of which tree T is being photographed, the shooting position is above the tree T. With the above configuration, each fruit F of each tree T is photographed from above the fruit F.

[0046] Figure 4(b) is a view of the imaging device 300 in the X-axis direction. The specific example in Figure 4(b) assumes that the imaging device 300 is in the imaging position, similar to the specific example in Figure 4(a) described above. Figure 4(b) also shows the imaging direction Sc. In this embodiment, the imaging device 300 is fixed to the transport unit 200, and the imaging direction Sc as seen from the transport unit 200 does not change. However, the imaging device 300 may be configured to be rotatable relative to the transport unit 200. Alternatively, the imaging device 300 may be configured to be rotatable remotely relative to the transport unit 200. In these configurations, the imaging direction Sc can be changed remotely.

[0047] As described above, the imaging device 300 is equipped with a tilt sensor that detects the tilt of the imaging direction Sc. Specifically, the pitch angle θp and low angle θr of the imaging direction Sc with respect to the direction of gravity (Z axis) are detected by the tilt sensor. Figure 4(b) shows the pitch angle θp, one of the pitch angle θp and low angle θr. As can be seen from Figure 4(b), the pitch angle θp is the tilt of the imaging direction Sc around the X axis with respect to the direction of gravity.

[0048] Figure 4(c) is another view of the imaging device 300 from the Y-axis direction. Figure 4(c) shows the low angle θr, one of the low angles θr and pitch angle θp detected by the tilt sensor. As can be seen from Figure 4(c), the low angle θr is the tilt of the imaging direction Sc around the Y-axis with respect to the direction of gravity. When the imaging device 300 photographs a tree T, it detects the pitch angle θp and the low angle θr and generates tilt information that identifies the detected pitch angle θp and low angle θr.

[0049] As described above, the image information DS includes 3D image information DSb, which represents a 3D image. The yield estimation device 100 uses the tilt information (pitch angle θp, row angle θr) contained in the image information DS to correct the tilt of the 3D image. Through this tilt correction, the 3D image is corrected to a 3D image taken from the Z-axis direction (directly above).

[0050] Specifically, if the coordinates of an arbitrary point in a 3D image are (xs, ys, zs), the coordinates of that point are corrected to (xw, yw, zw) by Equation 1 below. In Equation 1, Rp represents the rotation matrix around the X-axis (pitch axis). Also, Rr in Equation 1 represents the rotation matrix around the Y-axis (row axis). Note that well-known rotation matrices Rp and Rr can be used as appropriate.

[0051]

number

[0052] Figures 5(a) to 5(c) illustrate specific examples of fruit images Gf captured by the imaging device 300 (depth camera 30b). As described above, the imaging device 300 generates three-dimensional image information DSb showing a three-dimensional image of the tree T (including the fruit F). The yield estimation device 100 identifies a fruit image Gf showing the fruit F from the three-dimensional image of the tree T. The following describes in detail a specific example of the fruit F (subject) shown in the fruit image Gf, prior to a specific example of the fruit image Gf (see Figure 5(b)).

[0053] Figure 5(a) is a diagram illustrating a specific example (mango) of fruit F from which a 3D image is taken. For the purposes of the following explanation, the portion of fruit F that has the largest cross-sectional area when cut parallel to the XY plane may be referred to as the "central portion Fm". Also, as shown in Figure 5(a), the upper part (Z-axis direction) of fruit F including the central portion Fm may be referred to as the "upper portion F1". Similarly, the portion of fruit F below the central portion Fm may be referred to as the "lower portion F2". As can be seen from Figure 5(a), the shape of the upper portion F1 and the shape of the lower portion F2 of fruit F in this embodiment are different.

[0054] As will be explained in detail later using Figures 6(a) to 6(d), the yield estimation device 100 first estimates the volume of the upper part F1 of the fruit F (hereinafter referred to as "partial volume Vp") using the fruit image Gf when estimating the weight W of the fruit F. The yield estimation device 100 then uses the partial volume Vp and the fruit-specific coefficient k (see Figure 7) described later to determine the total volume V of the fruit F, which is the sum of the upper part F1 and the lower part F2. As described above, the weight W of the fruit F is calculated by multiplying the volume V by the density E.

[0055] Figure 5(a) shows the shooting direction Sc of the shooting device 300. As described above, in this embodiment, the shooting device 300 is located above the tree T. Therefore, the fruit F is photographed from above (towards the Z-axis direction) as viewed from the fruit F. In the specific example in Figure 5(a), we assume that the fruit F is located towards the Y-axis direction from the shooting device 300, and that the fruit F is photographed from an oblique upward direction.

[0056] Incidentally, when photographing a fruit F with the camera 300, typically only the front side of the fruit F (towards the camera 300) is photographed, while the back side of the fruit F is not. In Figure 5(a), the boundary L between the area of ​​the fruit F that is photographed and the area that is not photographed is shown by a dashed line. In the specific example in Figure 5(a), only the side of the fruit F that is on the camera 300 side of boundary L is photographed.

[0057] Specifically, since the fruit F is photographed from above as viewed from the fruit F, the upper end of the fruit F (Ft in Figure 5(a)) is photographed almost entirely. On the other hand, for example, the central part Fm of the fruit F is photographed on the opposite side of the Y-axis (towards the viewer) as viewed from the boundary L, while the side in the Y-axis direction (towards the viewer) is not photographed. Also, the lower end of the fruit F (Fu in Figure 5(a)) is not photographed even if it is on the side of the imaging device 300. As can be understood from the above explanation, the upper part F1 of the fruit F is more likely to be photographed over a wider area than the lower part F2.

[0058] Figure 5(b) shows a specific example of a fruit image Gf. The fruit image Gf in Figure 5(b) shows the 3D image immediately after capture with the tilt correction described above applied. The z-axis in the 3D space where the fruit image Gf is displayed corresponds to the Z-axis parallel to the direction of gravity in real space. Similarly, the x-axis in the 3D space corresponds to the X-axis in real space, and the y-axis in the 3D space corresponds to the Y-axis in real space.

[0059] The fruit image Gf in Figure 5(b) shows the fruit F captured in the specific example shown in Figure 5(a) above. That is, Figure 5(b) assumes a fruit image Gf in which a portion of the fruit F (behind the boundary L in Figure 5(a)) is not captured. The above fruit image Gf is a 3D image representing a portion of the fruit F (a portion of the fruit F is missing). Note that in Figure 5(b), the region corresponding to the portion of the fruit F that was not captured is indicated by a dashed line.

[0060] As described above, the lower end Fu of the fruit F is not captured (see Figure 5(a)). In this embodiment, the region corresponding to the lower end Fu of the fruit F in the three-dimensional space where the fruit image Gf is located may be referred to as "region Rb". When the fruit F is photographed from above, region Rb is usually located below the portion of the fruit image Gf that represents the central part Fm of the fruit F (the part with the largest cross-sectional area).

[0061] Figure 5(c) is a diagram illustrating the cross-sectional view of the fruit image Gf. Note that Figure 5(c) assumes that each cross-section of the fruit image Gf is viewed from the Z-axis direction (from above).

[0062] Figure 5(c) shows the AA and BB cross-sections in Figure 5(b) described above. Of the cross-sections in Figure 5(c), the AA cross-section represents the upper end Ft (see Figure 5(a)) of the fruit image Gf (see Figure 5(b)). As described above, the upper end Ft of the fruit F is almost entirely captured. Therefore, the AA cross-section of the fruit image Gf is roughly annular, as shown in Figure 5(c). On the other hand, of the cross-sections in Figure 5(c), the BB cross-section represents the central part Fm (see Figure 5(a)) of the fruit image Gf (see Figure 5(b)). As described above, the central part Fm of the fruit F is captured on the side facing the imaging device 300, but not on the opposite side. Therefore, the BB cross-section of the fruit image Gf has a shape in which a part of the ellipse has been cut off, as shown in Figure 5(c).

[0063] As described above, when estimating the weight W of fruit F, the partial volume Vp of the upper portion F1 of fruit F is used. If a 3D image showing the entire upper portion F1 is taken, the partial volume Vp can be calculated from that 3D image. However, as explained using Figure 5(b), if a portion of the upper portion F1 is not photographed, only a portion of the upper portion F will be represented in the fruit image Gf (the other portion will be missing). Taking these circumstances into consideration, this embodiment employs a configuration that allows estimation of the partial volume Vp of the upper portion F1 from a fruit image Gf in which a portion of the upper portion F1 is missing.

[0064] Figures 6(a) to 6(d) illustrate the configuration for estimating the partial volume Vp. As will be explained in detail below, when estimating the partial volume Vp, the shape of the cross-section obtained by cutting the fruit image Gf in the xy plane is used.

[0065] Figure 6(a) is a diagram illustrating the various cross-sections s(1, 2, 3…n, n+1…) of the fruit image Gf. As with Figure 5(b) above, Figure 6(a) shows dashed lines indicating regions corresponding to parts of the fruit F that were not captured. Furthermore, Figure 6(a) shows the region Rb corresponding to the lower end Fu of the fruit F (see Figure 5(b)). In the specific example of Figure 6(a), as with Figure 5(b) above, we assume a case where the lower end Fu is not captured.

[0066] Each cross-section s shown in Figure 6(a) is obtained by cutting the fruit image Gf parallel to the xy-plane while shifting its position along the z-axis. Specifically, adjacent cross-sections s are separated by a height Δz in the z-axis direction. The height Δz is sufficiently small compared to the size of the fruit image Gf. In this embodiment, the shape of the missing portion of the fruit F in the fruit image Gf is estimated using each cross-section s of the fruit image Gf.

[0067] As shown in Figure 6(a), there is no cross-section s in the region Rb corresponding to the lower end Fu of the fruit F. Therefore, estimating the shape of the lower end Fu is difficult (including cases where it is practically impossible). However, in this embodiment, only the portion of the fruit image Gf representing the upper part F1 (excluding the lower end Fu), whose shape can be estimated with relatively high accuracy, is used to estimate the volume V (partial volume Vp). Therefore, there is little need to estimate the shape of the lower end Fu with high accuracy (details will be described later).

[0068] Figure 6(b) illustrates a specific example of a configuration for estimating the shape of the missing fruit F in the fruit image Gf. The cross-section of the fruit F parallel to the xy-plane has an outer edge that is approximately elliptical (circular). Considering these circumstances, in this embodiment, it is assumed that the cross-section s of the fruit image Gf is part of an ellipse, and the shape of the missing portion of the fruit F in the fruit image Gf is estimated.

[0069] Specifically, the cross-section s (fruit image Gf) is a point cloud image. As shown in Figure 6(b), the yield estimation device 100 finds an ellipse that minimizes the squared error of the point cloud of cross-section s (elliptic approximation). Hereafter, for explanatory purposes, the elliptical image generated from cross-section s may be referred to as the "corrected cross-section p". The corrected cross-section p is parallel to the xy-plane. The above corrected cross-section p is estimated to represent the shape of the cross-section of the fruit F. Therefore, by stacking each corrected cross-section p generated from each cross-section s (1, 2, 3...n...) in the z-axis direction, a 3D image representing the fruit F with the missing parts in the fruit image Gf filled in (see Gx in Figure 6(d)) can be formed.

[0070] However, as mentioned above, the fruit F is photographed from above as viewed from the fruit F. Therefore, it is conceivable that there may be cases where the fruit image Gf (cross-section s) does not exist in the region Rb (see Figure 6(a)) corresponding to the lower end Fu of the fruit F (see Figure 5(a)) (for example, the specific example in Figure 6(a)). In such cases, the shape of the lower part F2 including the lower end Fu of the fruit F cannot be estimated with high accuracy.

[0071] Considering the above circumstances, this embodiment employs a configuration in which the volume V is estimated using the cross-section s of the portion representing the upper part F1 of the fruit image Gf. Specifically, a configuration is adopted in which the partial volume Vp, which is the volume of the upper part F1, is calculated from the cross-section s of the portion representing the upper part F1. The volume V of the entire fruit F is calculated (estimated) by multiplying the partial volume Vp by a fruit-specific coefficient k (this will be explained in detail using Figure 7).

[0072] Figure 6(c) is a diagram illustrating a specific example of a configuration for calculating the partial volume Vp. As described above, each corrected cross section p is generated from each cross section s of the fruit image Gf. When the yield estimation device 100 calculates the partial volume Vp, it calculates the area A of each corrected cross section p. Figure 6(c) shows a specific example of the area A for each corrected cross section p. The specific example in Figure 6(c) assumes that the area A of corrected cross section p1 generated from cross section s1 is "A1". Similarly, it assumes that the area A of corrected cross section p2 is "A2", the area A of corrected cross section p3 is "A3", and so on.

[0073] The yield estimation device 100 calculates the area A of each correction cross-section p and identifies the correction cross-section p with the largest area A. In the specific example in Figure 6(c), we assume that the area A(An) of the nth correction cross-section pn from the top is the largest (n is a natural number). In this case, the yield estimation device 100 identifies the correction cross-section pn. Hereafter, for explanatory purposes, the correction cross-section p with the largest area A among the various correction cross-sections p may be referred to as the "central cross-section p". For example, in the specific example in Figure 6(c), the nth correction cross-section pn from the top is the central cross-section p.

[0074] The central section p typically represents the cross-section of the central part Fm of the fruit F (the part with the largest cross-sectional area; see Figure 5(a) above). In other words, the central section p can also be described as the lower edge (bottom surface) of the 3D image representing the upper part F1 of the fruit F. As described above, the 3D image obtained by stacking the central section p and each corrected section p located above the central section p (hereinafter sometimes referred to as "upper section p") represents the upper part F1.

[0075] As can be understood from the above explanation, the volume of the 3D image obtained by stacking each upper cross-section p is the partial volume Vp of the upper part F1 of the fruit F. Also, as mentioned above, the height Δz, which is the distance between each cross-section s in the z-axis direction, is sufficiently small. In this case, the partial volume Vp is calculated (approximated) by the following equation 2. Note that "n" in equation 2 represents the number of upper cross-sections p.

[0076]

number

[0077] Incidentally, each fruit F on tree T has a nearly identical fruiting direction. Specifically, each fruit F on tree T usually has its upper portion F1 facing upwards (the upper portion F1 is located above the lower portion F2). Therefore, the portion of the fruit image Gf above the central cross-section p is usually a three-dimensional image representing the upper portion F1.

[0078] Considering the above circumstances, in this embodiment, the partial volume Vp is calculated assuming that the area above the central cross-section p of the fruit image Gf is a three-dimensional image of the upper portion F1. Specifically, the partial volume Vp is calculated assuming that the area above the central cross-section p of the fruit image Gf is a three-dimensional image of the upper portion F1, without any special processing (e.g., pattern matching) required to identify the portion representing the upper portion F1 from the fruit image Gf.

[0079] The above configuration reduces the processing load on the yield estimation device 100 compared to a configuration in which, for example, special processing is performed to identify the portion representing the upper portion F1 from the fruit image Gf. However, the present invention does not exclude a configuration in which special processing is performed to identify the portion representing the upper portion F1 from the fruit image Gf.

[0080] Furthermore, according to this embodiment, since all fruits F can be photographed from above, a wider area of ​​the upper portion F1 of the fruit F can be easily photographed compared to, for example, when the fruit F is photographed from the side or below. This configuration can also be rephrased as making it easier to photograph a wider area of ​​the upper portion F1 used for estimating the volume V of the fruit F. Therefore, there is an advantage in that the volume V of the fruit F can be estimated with high accuracy. However, the present invention does not exclude configurations in which the fruit F is photographed from the side or below.

[0081] Figure 7 is a diagram illustrating a specific example of the fruit-specific coefficient k. As described above, the yield estimation device 100 calculates the total volume V of the fruit F by multiplying the partial volume Vp of the fruit F by the fruit-specific coefficient k (V = Vp × k). The yield estimation device 100 of this embodiment stores multiple types of fruit-specific coefficients k in advance. Furthermore, it is configured so that the fruit-specific coefficient k can be changed according to the type of fruit F for which the yield Y is calculated. For example, the user can change the fruit-specific coefficient k by operating the yield estimation device 100 as appropriate. However, it is also possible to configure it so that the fruit-specific coefficient k cannot be changed. That is, the number of types of fruit F for which the yield Y can be estimated may be limited to one type.

[0082] Let's assume a roughly spherical fruit F (for example, a kumquat). In this fruit F, the shape of the upper part F1 and the shape of the lower part F2 are both roughly hemispherical. Therefore, the total volume V of the fruit F can be calculated by doubling the partial volume Vp of the upper part F1 (V = 2Vp).

[0083] On the other hand, depending on the fruit F (for example, a mango), the shape of the upper part F1 and the shape of the lower part F2 may differ. However, if the fruits F are of the same type (for example, if they are all mangoes), even if the volume (size) of each fruit F differs, the shapes of each fruit F will be approximately similar. Therefore, the proportion (=Vp / V) of the partial volume Vp of the upper part F1 to the total volume V of the fruit F is common to all fruits F of the same type. In this embodiment, the proportion of the partial volume Vp to the volume V is determined in advance using a sample fruit F, and the reciprocal of this proportion is stored as the fruit-specific coefficient k. Note that the method for determining the fruit-specific coefficient k is not limited to the above example.

[0084] The yield estimation device 100 stores a combination of fruit-specific coefficient k and density E for each fruit F. When the type of fruit F for which the yield Y is to be estimated is specified, the yield estimation device 100 sets the fruit-specific coefficient k and density E for that fruit F. The yield estimation device 100 also calculates the weight W of the fruit F by multiplying the volume V of the fruit F by the set fruit-specific coefficient k and density E. The weight W of all photographed fruit F is also calculated, and the sum of the weights W of all fruit F is stored as the yield Y. As described above, the weight W of each fruit is calculated using 3D image information DSb. According to this embodiment, for example, the yield Y can be calculated with higher accuracy compared to a configuration in which the yield Y is calculated without using 3D image information DSb.

[0085] Figure 8 is a flowchart of the yield estimation process. The yield estimation device 100 executes the yield estimation process at appropriate triggers. For example, after the camera 300 has photographed all the trees T, the yield estimation process is executed when the yield estimation device 100 is operated appropriately. As shown in Figure 8, when the yield estimation process is started, the yield estimation device 100 sets the target image (S101). Specifically, in this embodiment, as described above, multiple image information DS are generated. In step S101 described above, one of the multiple image information DS is set as the target image.

[0086] After setting the target image, the yield estimation device 100 performs fruit identification processing (S102). In fruit identification processing, using AI technology, fruits F are identified from each object indicated by the 2D image information DSa of the image information DS set as the target image. Furthermore, using the identification results, fruits F are identified from each object indicated by the 3D image information DSb of the image information DS. If each object indicated by the 3D image information DSb contains multiple fruits F, all fruits F are identified.

[0087] After performing the fruit identification process, the yield estimation device 100 determines the target fruit (S103). Specifically, the yield estimation device 100 sets one of the fruits F identified in the preceding fruit identification process as the target fruit. After setting the target fruit, the yield estimation device 100 performs the volume estimation process (S104). In the volume estimation process, the volume V of the target fruit set in the preceding step S103 is estimated. Specifically, as explained in Figures 6(a) to 6(d) above, the partial volume Vp of the target fruit is calculated. Also, as explained in Figure 7, the volume V of the target fruit is calculated by multiplying the partial volume Vp by the fruit-specific coefficient k.

[0088] After performing the volume estimation process, the yield estimation device 100 performs the weight estimation process (S105). In the weight estimation process, the volume V of the target fruit estimated in the preceding volume estimation process is multiplied by the density E to calculate the weight W of the target fruit (W = V × E). The yield estimation device 100 stores the calculated weight W.

[0089] After the weight estimation process is performed, it is determined whether all fruits F identified in the preceding fruit identification process have been set as target fruits (S106). If any of the fruits F shown in the image information DS have been set as target fruits before (S106: No), then steps S103 to S106 are repeatedly executed while changing the target fruit in step S103. In step S103, fruits F that have never been set as target fruits are set as target fruits.

[0090] If all fruits F shown in the image information DS have already been set as target fruits (S106: Yes), the yield estimation device 100 determines whether all of the image information DS has been set as target images (S107). If the image information DS acquired from the imaging device 300 includes any that have never been set as target images (S107: No), steps S101 to S107 are repeatedly executed while changing the target image in step S101. In step S101, the image information DS that has never been set as a target image is set as a target image.

[0091] If all image information DS has been set to the target image (S106: Yes), the yield estimation device 100 executes the yield calculation process (S108). In the yield calculation process, the sum of the weights W of all fruits F calculated in step S105 is calculated, and the calculation result is stored as the yield Y. Alternatively, the yield Y may be automatically displayed when the yield calculation process is executed. After executing the yield calculation process, the yield estimation device 100 terminates the yield estimation process.

[0092] <Variation> Each of the above forms can be modified in various ways. Specific examples of modifications are given below. Two or more forms can be arbitrarily selected from the following examples and combined as appropriate.

[0093] (1) In each configuration, multiple imaging devices 300 may be provided. For example, if multiple rows of trees T are provided in an orchard, an imaging device 300 may be provided for each row of trees T. Specifically, a wire rope 400 is provided along each row of trees T. In addition, one imaging device 300 is provided for each wire rope 400. In the above modified configuration, image information DS is transmitted from each imaging device 300 to the yield estimation device 100, and the yield Y of the fruit F is estimated.

[0094] (2) In each configuration, the type of fruit F used to calculate the yield Y may be automatically determined. Specifically, the type of fruit F shown in the image indicated by the image information DS (2D image information DSa) is identified, for example, by AI technology. When the yield estimation device 100 identifies the fruit F shown in the image of the image information DS, it makes it possible to calculate the yield Y of that fruit F. For example, the fruit-specific coefficient k and density E of the fruit F are automatically set.

[0095] (3) In each configuration, the system may be configured to calculate (estimate) other information in addition to the harvest yield Y from the image information DS. For example, as described above, each fruit F is identified from each object represented by the image information DS. In the above configuration, the number of objects identified as fruit F may be calculated as the number of fruit F in the tree T. Also, in the above configuration, the weight W is calculated for each fruit F. In the above configuration, the weight W for each fruit F may be stored separately from the harvest yield Y. Also, the system may be configured to add information indicating the shooting location (mark part M) to each of the above pieces of information. For example, the system may be configured to store a combination of the number of fruit F identified from the image information DS, the weight W of the fruit F, the shooting location (type of mark part M), and the date and time the photo was taken. The system may be configured to display each of the above pieces of information separately.

[0096] (4) Depending on the type of fruit F, the flowers may be thinned from the tree T before the fruit F develops. In this case, the method of thinning (such as the number of flowers thinned) for each tree T may be input into the yield estimation device 100. Alternatively, the device may be configured to display the combination of the yield Y for each tree T and the method of thinning for that tree T. Alternatively, the device may be configured to display the combination of the yield Y of the fruit F for all trees T and the method of thinning for each tree T. The above configuration has the advantage of enabling research into the optimal method of thinning (for example, the method of thinning that maximizes the yield Y).

[0097] <Summary of the operation and effects of the embodiment> <First aspect> The yield estimation system (1) of this embodiment comprises a photographing device (300) for photographing plants (T) bearing fruit (F), and a yield estimation device (100) for acquiring image information (DS) from the photographing device, wherein the image information includes three-dimensional image information (DSb), and the yield estimation device comprises a fruit identification unit (102) for identifying fruit from each object represented by the image information, a volume estimation unit (103) for estimating the volume of fruit using the three-dimensional image information, a weight estimation unit (104) for estimating the weight of the fruit from its volume, and a yield calculation unit (105) for calculating a numerical value (yield Y) which is the sum of the weights of multiple fruits. In this embodiment, for example, compared to a configuration that estimates the yield Y without using three-dimensional image information, the yield Y can be calculated with high accuracy.

[0098] <Second and Third Embodiments> The yield estimation system (1) of this embodiment comprises a track section (wire rope 400) supported at a predetermined position and on which a camera device is transported, and a transport section (200) that transports the camera device by the track section to a shooting position where vegetation can be photographed, and the shooting position is above the vegetation (see Figure 4(a)).

[0099] Incidentally, the direction of fruiting is generally the same for all fruits in plants, and the upper portion F1 is usually located on the upper side. According to this embodiment, since all fruits can be photographed from above, a wider area of ​​the upper portion F1 of the fruit F can be photographed, compared to, for example, when the fruit F is photographed from the side or below.

[0100] <Third aspect> The yield estimation system (1) of this embodiment has a volume estimation unit that estimates the partial volume (Vp), which is the volume of the upper part (upper part F1) of the fruit as shown by the 3D image information, and estimates the total volume of the fruit using the partial volume and a fruit-specific coefficient (k) according to the type of fruit.According to this embodiment, by adopting a configuration that makes it easy to photograph a wide area of ​​the upper part F1 of the fruit F (the part used to estimate the volume of the fruit F) (for example, the second embodiment described above), the volume V of the fruit F can be estimated with high accuracy.

[0101] <Fourth aspect> The yield estimation device (100) of this embodiment is a yield estimation device that acquires image information (DS) from a photographing device (300) that photographs plants (T) bearing fruit (F), the image information includes three-dimensional image information (DSb), and comprises a fruit identification unit (102) that identifies fruit from each object represented by the image information, a volume estimation unit (103) that estimates the volume of fruit using the three-dimensional image information, a weight estimation unit (104) that estimates the weight of the fruit from the volume of the fruit, and a yield calculation unit (105) that calculates a numerical value (yield Y) by summing the weights of multiple fruits. According to this embodiment, the same effects as the first embodiment described above can be achieved.

[0102] <Fifth aspect> The harvest yield estimation method of this embodiment is a method for estimating the harvest yield of fruit using a photographing device (300) that photographs plants (T) bearing fruit (F) and a harvest yield estimation device (100) that acquires image information (DS) from the photographing device. The harvest yield estimation device identifies the fruit from each object represented by the image information (DS) (S102 in Figure 8), estimates the volume of the fruit using the three-dimensional image information (DSb) contained in the image information (S104 in Figure 8), estimates the weight of the fruit from its volume (S105 in Figure 8), and calculates a numerical value by summing the weights of multiple fruits (S108 in Figure 8). According to this embodiment, the same effects as the first embodiment described above can be achieved. [Explanation of Symbols]

[0103] 100...Yield estimation device, 101...Image information acquisition unit, 102...Fruit identification unit, 103...Volume estimation unit, 104...Weight estimation unit, 105...Yield calculation unit, 200...Conveying unit, 300...Photography device, 400...Training track unit.

Claims

1. A camera for photographing plants that bear fruit, A harvest yield estimation device that acquires image information from the aforementioned imaging device. A yield estimation system comprising the following: The aforementioned image information includes three-dimensional image information, The harvest yield estimation device is A fruit identification unit that identifies the fruit from each object represented by the image information, A volume estimation unit that estimates the volume of the fruit using the three-dimensional image information, A weight estimation unit that estimates the weight of the fruit from the volume of the fruit, It includes a harvest yield calculation unit that calculates a value obtained by summing the weights of multiple fruits, The volume estimation unit is, From the shape of the upper part of the fruit as shown by the three-dimensional image information, the partial volume, which is the volume of the upper part, is estimated. The total volume of the fruit is estimated using the fruit-specific coefficient corresponding to the type of fruit and the partial volume. Yield estimation system.

2. A track section supported at a predetermined position, through which the imaging device is transported, The system comprises a transport unit that transports the photographic device by the track unit to a photographic position where the aforementioned plants and trees can be photographed, The aforementioned shooting position is above the aforementioned vegetation. The yield estimation system according to claim 1.

3. A harvest yield estimation device that acquires image information from a camera that photographs fruit-bearing plants, The aforementioned image information includes three-dimensional image information, A fruit identification unit that identifies the fruit from each object represented by the aforementioned image information, A volume estimation unit that estimates the volume of the fruit using the three-dimensional image information, A weight estimation unit that estimates the weight of the fruit from the volume of the fruit, It includes a harvest yield calculation unit that calculates a value obtained by summing the weights of multiple fruits, The volume estimation unit is, From the shape of the upper part of the fruit as shown by the three-dimensional image information, the partial volume, which is the volume of the upper part, is estimated. The total volume of the fruit is estimated using the fruit-specific coefficient corresponding to the type of fruit and the partial volume. Yield estimation device.

4. A method for estimating the yield of fruit, using a photographing device for photographing fruit-bearing plants and a yield estimation device for acquiring image information from the photographing device, The harvest yield estimation device identifies the fruit from each object represented by the image information, The harvest yield estimation device estimates the partial volume, which is the volume of the upper part, from the shape of the upper part of the fruit shown by the three-dimensional image information included in the aforementioned image information. The harvest yield estimation device estimates the total volume of the fruit using the fruit-specific coefficient corresponding to the type of fruit and the partial volume. The harvest yield estimation device estimates the weight of the fruit from the volume of the fruit. The harvest yield estimation device calculates a value by summing the weights of multiple fruits. Methods for estimating yield.

Images