ESTIMATION DEVICE, CONTROL METHOD FOR ESTIMATION DEVICE, AND ESTIMATION PROGRAM

Abstract

To achieve an estimation device capable of efficiently identifying overlapping fruits on a picked-up image as individual fruits with high accuracy.SOLUTION: An estimation device (1) includes a detection part (121) for detecting an inclusion area, an identifying part (122) for identifying an object area, a clustering part (123) for clustering each pixel in the object area by unsupervised learning using depth information, and an estimation part (124) for estimating a boundary between fruits in the object area on the basis of a result of the clustering.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an estimation device, a control method for an estimation device, and an estimation program. [Background technology]

[0002] In order to ensure stable production of fruit vegetables and fruits such as strawberries and tomatoes in greenhouses, technologies have been developed to predict the harvest time and yield of fruit based on factors such as the accumulated temperature of the fruit after flowering.

[0003] Previously, the accumulated temperature was calculated based on the air temperature inside a greenhouse, which resulted in many errors during fruit harvesting.

[0004] Therefore, in order to predict fruit harvest times and yields with high accuracy, there is a demand for technology that can measure the temperature and color of individual fruits with high accuracy based on captured images of the fruits without destroying the fruits, and monitor the growth of each individual fruit.

[0005] However, in greenhouses for horticulture, when fruits are photographed, fruits that are close to each other tend to overlap on the captured image, and in two-dimensional image processing, overlapping fruits on the captured image are often recognized as one fruit. Therefore, technology has been developed to identify overlapping fruits on the captured image as individual fruits.

[0006] Japanese Patent Laid-Open Publication No. 62-9210 discloses a technique for separating and extracting overlapping fruit regions in a captured image based on color information, and estimating boundaries between individual fruits in the captured image based on shading information.

[0007] Japanese Patent Application Laid-Open No. 4-184573 discloses a technology in which light is projected onto the objects, such as fruit at harvest time, and the positional relationship of overlapping objects is estimated based on color information and brightness information of an image captured of the light reflected from the objects.

[0008] JP 2015-162073 A discloses a technology for detecting boundaries between pixel regions below a certain depth from a depth image of a harvest object such as fruit, and pattern matching the geometric shape of the specified harvest object against the boundaries to estimate the boundaries between the harvest objects. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 62-9210 [Patent Document 2] Japanese Patent Application Publication No. 4-184573 [Patent Document 3] JP 2015-162073 A Summary of the Invention [Problem to be solved by the invention]

[0010] However, rule-based techniques such as those described in Patent Documents 1 and 2 that estimate overlapping fruit objects in a captured image as individual fruits based on color and brightness information cannot accurately identify the fruits as individual fruits.

[0011] A technology such as that described in Patent Document 3, which estimates the boundaries between harvest objects by pattern matching the geometric shape of a specified harvest object against the boundaries between pixel regions below a specified depth detected from a depth image, cannot efficiently identify overlapping fruits in a captured image as individual fruits.

[0012] Therefore, in order to predict fruit harvest times and yields with high accuracy, there is a demand for technology that can accurately and efficiently identify overlapping fruits in an image as individual fruits, so that the temperature and color of each fruit can be measured with high accuracy without destroying the fruit.

[0013] An object of one aspect of the present invention is to provide an estimation device and related techniques that can accurately and efficiently identify overlapping fruits in a captured image as individual fruits. [Means for solving the problem]

[0014] In order to solve the above problems, an estimation device according to one embodiment of the present invention includes a detection unit that detects an encapsulation region that contains each fruit in an image of a plurality of fruits, an identification unit that identifies a target region including an area where the encapsulation regions overlap each other, a clustering unit that clusters each pixel in the target region by unsupervised learning using depth information corresponding to the pixel, and an estimation unit that estimates boundaries between fruits in the target region based on a result of the clustering.

[0015] A control method for an estimation device according to one aspect of the present invention includes a detection step in which the estimation device detects an encapsulation region that contains each fruit in an image capturing a plurality of fruits; an identification step in which the estimation device identifies a target region including an area where the encapsulation regions overlap; a clustering step in which the estimation device clusters each pixel in the target region by unsupervised learning using depth information corresponding to the pixel; and an estimation step in which the estimation device estimates boundaries between fruits in the target region based on a result of the clustering.

[0016] The estimation device according to each aspect of the present invention may be realized by a computer. In this case, the estimation program of the estimation device that causes the computer to operate as each part (software element) of the estimation device to realize the estimation device, and the computer-readable recording medium on which the estimation program is recorded, also fall within the scope of the present invention. Effect of the Invention

[0017] According to one aspect of the present invention, it is possible to realize an estimation device and related techniques that can accurately and efficiently identify overlapping fruits on a captured image as individual fruits. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing an example of a configuration of an estimation device according to a first embodiment. [Diagram 2] 4 is a flowchart showing an example of a control method of the estimation device according to the first embodiment. FIG. [Diagram 3] FIG. 4 is a diagram showing an example of a captured image. [Figure 4] FIG. 13 is a diagram illustrating an example of a depth image. [Diagram 5] FIG. 2 is a diagram illustrating an example of a superpixel. [Figure 6] FIG. 2 illustrates an example of boundaries between fruits in a target region. [Figure 7] FIG. 11 is a block diagram showing an example of the configuration of an estimation device according to a second embodiment. [Figure 8] FIG. 11 is a flowchart showing an example of a control method of the estimation device according to the second embodiment. [Figure 9] FIG. 4 is a diagram showing an example of a captured image. [Figure 10] FIG. 11 is a block diagram showing an example of the configuration of an estimation device according to a third embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of a control method of the estimation device according to the third embodiment. [Figure 12] FIG. 13 is a diagram for explaining separation between fruits in a target region. [Figure 13] FIG. 11 is a block diagram showing an example of the configuration of an estimation device according to a fourth embodiment. [Figure 14] FIG. 11 is a flowchart showing an example of a control method of the estimation device according to the fourth embodiment. [Figure 15] FIG. 13 is a diagram showing an example of a thermal image. [Figure 16] FIG. 11 is a diagram for explaining an example of a measurement process. [Figure 17] FIG. 13 is a block diagram showing an example of the configuration of an estimation device according to a fifth embodiment. [Figure 18] FIG. 13 is a flowchart showing an example of a control method of the estimation device according to the fifth embodiment. [Figure 19] FIG. 4 is a diagram showing an example of a captured image. [Figure 20] FIG. 13 is a diagram illustrating an example of a depth image. [Figure 21] FIG. 13 illustrates an example of a boundary between leaves in a second region of interest. [Figure 22] FIG. 13 is a diagram for explaining separation between leaves in a second target region. [Figure 23] FIG. 13 is a diagram for explaining restoration of an entire image of a leaf. [Figure 24] FIG. 13 is a block diagram showing an example of the configuration of an estimation device according to a sixth embodiment. [Diagram 25] FIG. 13 is a flowchart showing an example of a control method of the estimation device according to the sixth embodiment. [Figure 26] FIG. 13 is a diagram for explaining restoration of an entire image of a fruit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] <Embodiment 1> The first embodiment will be described with reference to Figures 1 to 5. Figure 1 is a block diagram showing an example of the configuration of an estimation device 1 according to the first embodiment.

[0020] [Estimation device 1] The estimation device 1 estimates boundaries between fruits in a captured image (captured image information) of a plurality of fruits. As shown in Fig. 1, the estimation device 1 includes an acquisition unit 11, a control unit 12, an output unit 13, and a storage unit 14. The estimation device 1 may be, for example, a computer.

[0021] [Acquisition part 11] The acquisition unit 11 acquires various images (various image information) such as captured images of a plurality of fruits and depth images (depth image information).

[0022] [Control unit 12] The control unit 12 controls the estimation device 1. As shown in Fig. 1, the control unit 12 includes a detection unit 121, a specification unit 122, a clustering unit 123, and an estimation unit 124.

[0023] (Detection unit 121) The detection unit 121 detects an inclusion region that contains each fruit in a captured image in which a plurality of fruits are captured.

[0024] (Specific Section 122) The identifying unit 122 identifies a target region that includes a region where the connotative regions overlap each other.

[0025] (Clustering section 123) The clustering unit 123 clusters each pixel in the target region identified by the identification unit 122 through unsupervised learning using the depth information corresponding to the pixel.

[0026] (Estimation part 124) The estimation unit 124 estimates boundaries between fruits in the target region based on the clustering result by the clustering unit 123.

[0027] [Output section 13] The output unit 13 displays various images such as the captured image and the depth image, as well as the estimation result, such as the boundary between fruits in the target region, on the display unit 131. As shown in FIG.

[0028] (Display section 131) The display unit 131 displays various images such as captured images and depth images, as well as estimation results, etc. Examples of the display unit 131 include a computer display.

[0029] [Storage section 14] The storage unit 14 stores various information. As shown in Fig. 1, the storage unit 14 includes a trained model 141. The trained model 141 will be described later.

[0030] [Control method S1 of estimation device 1] Next, a control method S1 of the estimation device 1 according to the first embodiment will be described with reference to Figures 2 to 6. Figure 2 is a flow chart showing an example of the control method S1 of the estimation device 1 according to the first embodiment.

[0031] As shown in FIG. 2, the control method S1 of the estimation device 1 includes an acquisition step S11, a detection step S12, an identification step S13, a clustering step S14, an estimation step S15, and an output step S16.

[0032] [Acquisition step S11] In the acquisition step S11, the acquisition unit 11 of the estimation device 1 acquires various images such as captured images of a plurality of fruits and depth images corresponding to the captured images.

[0033] (An example of the acquisition step S11) An example of the obtaining step S11 will be described below with reference to Figures 3 and 4. Figure 3 is a diagram showing an example of a captured image I1. Figure 4 is a diagram showing an example of a depth image I2.

[0034] The acquisition unit 11 functions as an imaging unit such as an image camera, and captures a plurality of fruits C1 to C6 to acquire a captured image I1 in which the encapsulated regions A1 to A6 are not detected and the target region B1 is not specified. The acquisition unit 11 also functions as an imaging unit such as a depth camera, and captures a plurality of fruits C1 to C6 to acquire a depth image I2 in which the target region B5 is not mapped.

[0035] The acquisition unit 11 may acquire a captured image I1 in which multiple fruits C1 to C6 are captured, the encapsulated regions A1 to A6 are not detected, and the target region B1 is not specified, from an image camera or the like external to the estimation device 1. The acquisition unit 11 may also acquire a depth image I2 in which multiple fruits C1 to C6 are captured, and the target region B5 is not mapped, from a depth camera or the like external to the estimation device 1.

[0036] [Detection step S12] In the detection step S12, the detection unit 121 of the estimation device 1 detects an inclusion region that includes each fruit in a captured image of a plurality of fruits.

[0037] (An example of the detection step S12) An example of the detection step S12 will be described below with reference to FIG.

[0038] 3, the captured image I1 capturing a plurality of fruits C1 to C6 may be an RGB image (RGB image information). This allows the clustering unit 123 of the estimation device 1 to perform clustering in a clustering step S14 described later by further using at least one of color information and luminance information corresponding to each pixel in the captured image I1. As a result, the estimation device 1 can more efficiently identify overlapping fruits C1 and C2 on the captured image I1 as individual fruits C1 and C2.

[0039] As shown in Fig. 3, the fruits C1 to C6 may be fruits of fruit vegetables or fruit trees. Examples of fruits of fruit vegetables include strawberries, tomatoes, paprika, green peppers, eggplants, and fruits of the same family (e.g., strawberry-based fruits). Examples of fruits of fruit trees include mandarins, blueberries, and fruits of the same family (e.g., mandarin orange-based fruits). Even if small and close fruits C1 and C2 such as fruits of fruit vegetables or fruit trees overlap on the captured image I1, the estimation device 1 can accurately and efficiently identify the fruits as individual fruits C1 and C2.

[0040] The detection unit 121 may use the trained model 141 to detect the encapsulated areas A1 to A6 in the captured image I1.

[0041] The trained model 141 is generated by machine learning using a set of training images (training image information) of multiple fruits and encapsulated regions (encapsulated region information) that encapsulate each fruit in the training images as training data. The encapsulated regions are, for example, manually specified. The encapsulated regions are associated with position information such as coordinates on the training images.

[0042] The detection unit 121 detects the encapsulation regions A1-A6 output from the trained model 141 by inputting the captured image I1 to the trained model 141. That is, the detection unit 121 detects position information of each pixel corresponding to the fruits C1-C6 contained in the encapsulation regions A1-A6. In this way, the detection unit 121 can detect the encapsulation regions A1-A6 efficiently with high accuracy by using the trained model 141.

[0043] In the above example, the shape of the encapsulation regions A1 to A6 is rectangular, but in this embodiment, the shape of the encapsulation region is not particularly limited as long as it encapsulates each fruit, and may be, for example, circular. However, since the encapsulation region has a rectangular shape, it is easy to encapsulate each fruit, and the target region can be easily identified by the identification unit 122 of the estimation device 1.

[0044] [Specific process S13] In the identification step S13, the identification unit 122 of the estimation device 1 identifies a target region including a region where the connotative regions overlap each other.

[0045] (An example of the identification step S13) An example of the identifying step S13 will be described below with reference to FIG.

[0046] First, for each of the connotative regions, identification unit 122 determines whether or not there is another connotative region that overlaps with the connotative region, and if it determines that the other connotative region exists, it identifies the other connotative region.

[0047] For example, identification unit 122 determines whether or not there is another connotative region that overlaps with each of connotative regions A1 to A6 in Fig. 3, as shown in the overlapping relationship between the connotative regions in Table 1. If identification unit 122 determines that the other connotative region exists, it identifies the other connotative region.

[0048] [Table 1]

[0049] Next, for each of the connotative regions, the identification unit 122 refers to other connotative regions that overlap with the connotative region, and identifies a target region that includes an area where the connotative regions overlap each other.

[0050] For example, the identification unit 122 refers to the overlapping relationship between the encapsulation regions shown in Table 1 and identifies a target region B1 including an area where the encapsulation regions A1 and A2 overlap. Similarly, the identification unit 122 identifies a target region B2 including an area where the encapsulation regions A4 and A5 overlap. In the example shown in FIG. 3, the encapsulation region A5 is contained in the encapsulation region A4, so the target region B2 is the same as the encapsulation region A5. The identification unit 122 identifies a target region B3 including an area where the encapsulation regions A4 and A6 overlap. In the example shown in FIG. 3, the encapsulation region A6 is contained in the encapsulation region A4, so the target region B3 is the same as the encapsulation region A6. The identification unit 122 identifies a target region B4 including an area where the encapsulation regions A4 to A6 overlap.

[0051] That is, the identifying unit 122 identifies position information of each pixel corresponding to the target regions B1 to B4. As shown in Fig. 3, the identifying unit 122 may identify regions where the encapsulating regions overlap each other as the target regions B1 to B4.

[0052] Thus, the identification unit 122 can allow the user to recognize the overlapping relationship between the encapsulation regions including the fruits, i.e., the overlapping relationship between the fruits on the captured image, even if the fruits are small in size and close to each other, such as strawberries, and overlap on the captured image. Furthermore, the identification unit 122 can identify only the encapsulation regions that overlap with each encapsulation region, i.e., the fruits that overlap with each fruit on the captured image. In particular, the identification unit 122 can effectively identify small fruits such as strawberries that overlap with multiple fruits on the captured image and are hidden behind the multiple fruits.

[0053] Furthermore, in a clustering step S14 described later, the clustering unit 123 of the estimation device 1 can cluster only the target regions B1 to B4. As a result, the estimation unit 124 of the estimation device 1 can more efficiently identify the individual fruits C1, C2, and C4 to C6, compared to the case where the clustering unit 123 performs clustering including non-overlapping contained regions.

[0054] As shown in the above example, the identification unit 122 may identify an area where two encapsulated areas overlap as one target area B1, B2, and B3, or may identify an area where three or more encapsulated areas overlap as one target area B4. In this case, the identification unit 122 can identify the target area by a similar method.

[0055] The identification unit 122 may also identify the degree of overlap between the encapsulating regions. For example, the identification unit 122 may identify an intersection over union (IoU), which is an index indicating the degree of overlap between the encapsulating regions A1 to A6.

[0056] Identification unit 122 may identify the IoU of each of two or more included regions overlapping a certain included region, even if two or more included regions overlap the certain included region. Furthermore, identification unit 122 may identify the IoU as 1 or 100% even when one or more included regions are included in a certain included region. This allows the user to recognize from the IoU value that the one or more included regions are located before the certain included region.

[0057] For example, when the inclusion region A5 containing the fruit C5 and the inclusion region A6 containing the fruit C6 are contained in the inclusion region A4 containing the fruit C4, the identification unit 122 may identify the IoU of the inclusion regions A5 and A6 as 1 or 100%. In this way, the identification unit 122 can not only allow the user to recognize that the inclusion region A4 contains the two inclusion regions A5 and A6, but also that the inclusion regions A5 and A6 are located before the inclusion region A4.

[0058] [Clustering step S14] In the clustering step S14, the clustering unit 123 of the estimation device 1 clusters each pixel in the target region identified in the identification step S13 by unsupervised learning using the depth information corresponding to the pixel.

[0059] (An example of the clustering step S14) An example of the clustering step S14 will be described below with reference to FIGS.

[0060] The clustering unit 123 may perform clustering on each pixel in the target regions B1 to B4 using depth information corresponding to the pixel and an algorithm while performing machine learning. Hereinafter, an example in which the clustering unit 123 clusters each pixel in the target region B1 will be described.

[0061] In this case, the clustering unit 123 first maps the position information of the target area B1 of the captured image I1 identified by the identification unit 122 to the depth image I2 as a target area B5. Next, the clustering unit 123 may obtain depth information from the target area B5 of the depth image I2 that corresponds to the target area B1 of the captured image I1, and perform clustering using the depth information. This allows the clustering unit 123 to perform clustering efficiently.

[0062] In the depth image I2, black is represented as 0 gradation and white is represented as 255 gradation, and the deeper the depth, which is the distance from an imaging unit such as a depth camera that captures the depth image I2 to the fruit, the closer to white the fruit becomes. In the example shown in Fig. 4, the area corresponding to fruit C2 is whiter than the area corresponding to fruit C1, so the area corresponding to fruit C2 is deeper than the area corresponding to fruit C1.

[0063] The clustering unit 123 clusters each pixel in the target region B1 into a region corresponding to fruit C1 and a region corresponding to fruit C2, which is shown whiter than the region of fruit C1 in Fig. 4. That is, the clustering unit 123 classifies each pixel in the target region B1 into two regions: a region corresponding to fruit C1 and a region corresponding to fruit C2, which is deeper than fruit C1 and located behind fruit C1. This allows the clustering unit 123 to efficiently cluster each pixel in the target region B1.

[0064] The clustering unit 123 may further use at least one of color information and luminance information corresponding to each pixel to perform clustering. In this case, if the captured image I1 shown in Fig. 4 is an RGB image, the clustering unit 123 may obtain at least one of color information and luminance information corresponding to each pixel in the target region B1 from the RGB image.

[0065] In this way, the clustering unit 123 can cluster the pixels in the target region B1 with higher accuracy by further using at least one of the color information and the luminance information corresponding to each pixel. As a result, the estimation unit 124 can identify the overlapping fruits C1 and C2 on the captured image I1 as individual fruits C1 and C2 with higher accuracy.

[0066] The above-mentioned color information may be color information corresponding to a superpixel, that is, the clustering unit 123 may use the color information corresponding to the superpixel for clustering.

[0067] An example in which the clustering unit 123 uses color information corresponding to a superpixel for clustering will be described below with reference to Fig. 5. Fig. 5 is a diagram showing an example of a superpixel. Fig. 5 is merely a diagram for explanation purposes and is not usually displayed on the display unit 131.

[0068] 5 are small regions formed by grouping pixels in the target region B1 that have similar colors or textures. The clustering unit 123 further refers to the superpixels C1α, C1β, C1γ, C1δ, C2α, C2β, C2γ, and C2δ to separate the region into one corresponding to fruit C1 and one corresponding to fruit C2.

[0069] For example, the clustering unit 123 separates the regions by determining the boundary of one of these superpixels, indicated by the dashed line in Fig. 5, as the boundary between the region corresponding to fruit C1 and the region corresponding to fruit C2. As a result, the clustering unit 123 can cluster the pixels in the target region B1 with higher accuracy.

[0070] [Estimation process S15] In the estimation step S15, the estimation unit 124 of the estimation device 1 estimates boundaries between fruits in the target region based on the result of the clustering in the clustering step S14.

[0071] (An example of the estimation step S15) An example of the estimation step S15 will be described below with reference to Fig. 6. Fig. 6 is a diagram showing an example of a boundary D1 between fruits C1 and C2 in a target region B1.

[0072] The estimation unit 124 estimates a boundary D1 between fruits C1 and C2 in the target region B1 based on the result of clustering by the clustering unit 123. For example, the estimation unit 124 estimates that the boundary between the region corresponding to fruit C1 and the region corresponding to fruit C2 clustered by the clustering unit 123 is the boundary D1 between fruits C1 and C2. That is, the estimation unit 124 estimates position information such as the coordinates of the boundary D1.

[0073] Here, a rule-based technique such as that of Patent Document 1, which separates overlapping fruit regions in a captured image based only on color information and estimates boundaries between individual fruits in the captured image, cannot accurately identify fruits of the same color, such as strawberries and tomatoes, as individual fruits. Furthermore, a technique such as that of Patent Document 1 cannot accurately identify overlapping fruits in a captured image as individual fruits in a complex environment such as greenhouse horticulture where fruits are overlapping on the captured image.

[0074] A technology such as that in Patent Document 2, which estimates the positional relationship of overlapping objects based only on color and brightness information of an image obtained by projecting light onto objects such as fruits at harvest time and capturing the light reflected from the objects, estimates the positional relationship based on the positions of the centers of the objects. Therefore, a technology such as that in Patent Document 2 cannot estimate the boundary between objects with high accuracy, and cannot identify fruits that are hidden by overlapping with other fruits in the captured image as individual fruits with high accuracy.

[0075] Furthermore, a technology such as that of Patent Document 2, which is intended to estimate the positional relationship of only objects such as fruits at harvest time, cannot accurately identify overlapping fruits in a captured image, such as strawberries, which are small in size, have various shapes, and are closely packed together. Therefore, a technology such as that of Patent Document 2 cannot measure the harvest time and yield of fruits with high accuracy by measuring the temperature and color of each fruit when the fruit is small and monitoring each individual fruit.

[0076] Technology such as that described in Patent Document 3, which estimates boundaries between harvest objects by pattern matching the geometric shape of harvest objects such as fruits to boundaries between pixel regions below a certain depth detected from a depth image, cannot efficiently identify overlapping fruits in a captured image as individual fruits.

[0077] In addition, because the technology in Patent Document 3 estimates boundaries between harvest objects based on feature points, if the harvest objects are small fruits, it will estimate the boundaries as boundaries between fruits if there is a difference in depth, even if the boundaries are not between fruits. Therefore, the technology in Patent Document 3 cannot estimate fruits in the captured image as individual fruits with high accuracy.

[0078] In response to this, the estimation unit 124 can estimate the boundary D1 between fruits C1 and C2 based on the clustering results, thereby identifying overlapping fruits C1 and C2 on the captured image I1 as individual fruits C1 and C2 with high accuracy and efficiency.

[0079] In addition, the estimation unit 124 can accurately and efficiently identify overlapping fruits C1 and C2 on the captured image I1 as individual fruits C1 and C2, even in the case of fruits such as strawberries that are small in size, have various shapes, and tend to be close to each other.

[0080] [Output process S16] In the output step S16, the output unit 13 of the estimation device 1 causes the display unit 131 to display the boundaries between fruits in the target region, which are the estimation results in the estimation step S15.

[0081] (An example of the output process S16) An example of the output step S16 will be described below with reference to Fig. 6. As shown in Fig. 6, the output unit 13 causes the display unit 131 to display a target area B1 in which a boundary D1 between fruits C1 and C2 is visualized by a distinguishable line such as a dashed line, a dotted line, or a chain line.

[0082] <Embodiment 2> The estimation device according to an aspect of the present invention may specify, as the target region, a region obtained by combining overlapping connotative regions, like the estimation device 1α according to the second embodiment.

[0083] Hereinafter, the second embodiment will be described with reference to Figures 7 to 9. For convenience of explanation, the members having the same functions as those explained in the above embodiment are denoted by the same reference numerals, and the explanation thereof will be omitted.

[0084] [Estimation device 1α] Fig. 7 is a block diagram showing an example of the configuration of an estimation device 1α according to embodiment 2. As shown in Fig. 7, the estimation device 1α includes a control unit 12α instead of the control unit 12 in embodiment 1. Except for this, the estimation device 1α has the same configuration as the estimation device 1 according to embodiment 1.

[0085] [Control unit 12α] The control unit 12α includes an identifying unit 122α and a clustering unit 123α instead of the identifying unit 122 and the clustering unit 123 in the first embodiment. Except for this, the control unit 12α has the same configuration as the control unit 12 in the first embodiment.

[0086] (Specific part 122α) The identification unit 122α identifies a region formed by combining the overlapping encapsulating regions as a target region.

[0087] (Clustering section 123α) The clustering unit 123α clusters each pixel in the target region identified by the identification unit 122α through unsupervised learning using the depth information corresponding to the pixel.

[0088] [Control method S2 of estimation device 1α] Next, a control method S2 of the estimation device 1α according to the second embodiment will be described with reference to Figures 8 and 9. Figure 8 is a flow chart showing an example of the control method S2 of the estimation device 1α according to the second embodiment.

[0089] As shown in FIG. 8, the control method S2 of the estimation device 1α includes an acquisition step S21, a detection step S22, a specification step S23, a clustering step S24, an estimation step S25, and an output step S26.

[0090] The acquiring step S21, the detecting step S22, the estimating step S25, and the outputting step S26 are similar to the acquiring step S11, the detecting step S12, the estimating step S15, and the outputting step S16 in the first embodiment, respectively.

[0091] [Specific process S23] In the identification step S23, the identification unit 122α of the estimation device 1α identifies a region obtained by combining the overlapping connotative regions as a target region.

[0092] (An example of the identification step S23) An example of the specifying step S23 will be described below with reference to Fig. 9. Fig. 9 is a diagram showing an example of the captured image I1α.

[0093] The identification unit 122α identifies a region formed by combining the overlapping encapsulated regions A1 and A2 as a target region B1α, and also identifies a region formed by combining the overlapping encapsulated regions A4 to A6 as a target region B4α.

[0094] [Clustering step S24] In the clustering step S24, the clustering unit 123α of the estimation device 1α clusters each pixel in the target region identified in the identification step S23 by unsupervised learning using the depth information corresponding to the pixel.

[0095] (An example of the clustering step S24) An example of the clustering step S24 will be described below with reference to Fig. 9. The clustering unit 123α clusters the target regions B1α and B4α that combine the overlapping encapsulated regions A1 and A2. As a result, the estimation unit 124 of the estimation device 1α can identify the individual fruits C1, C2, and C4 to C6 with higher accuracy than when the clustering unit 123α clusters only the target regions B1 to B4 where the encapsulated regions A1 and A2 overlap.

[0096] The clustering unit 123α may further use the non-overlapping encapsulated regions A3 for clustering. This allows the clustering unit 123α to perform clustering with higher accuracy using more information. As a result, the estimation unit 124 can identify the individual fruits C1, C2, and C4 to C6 with higher accuracy.

[0097] <Embodiment 3> The estimation device according to an aspect of the present invention may separate a target region into regions corresponding to individual fruits based on boundaries between fruits, like the estimation device 1β according to embodiment 3. Furthermore, the estimation device according to an aspect of the present invention may predict at least one of the harvest time and yield of each fruit based on color information corresponding to each pixel in the region corresponding to each fruit, like the estimation device 1β according to embodiment 3.

[0098] Hereinafter, the third embodiment will be described with reference to Figures 10 to 12. For convenience of explanation, the members having the same functions as those explained in the above embodiments are denoted by the same reference numerals, and the explanation thereof will be omitted.

[0099] [Estimation device 1β] Fig. 10 is a block diagram showing an example of the configuration of an estimation device 1β according to embodiment 3. As shown in Fig. 10, the estimation device 1β includes a control unit 12β and an output unit 13β instead of the control unit 12 and the output unit 13 in embodiment 1. Except for this, the estimation device 1β has the same configuration as the estimation device 1 according to embodiment 1.

[0100] [Control unit 12β] The control unit 12β further includes a separation unit 125 and a prediction unit 126. Except for this, the control unit 12β has the same configuration as the control unit 12 in the first embodiment.

[0101] (separation part 125) The separation unit 125 separates the target region into regions corresponding to individual fruits based on the boundaries between the fruits estimated by the estimation unit 124.

[0102] (Prediction Section 126) The prediction unit 126 predicts at least one of the harvest time and the yield of each fruit, based on color information corresponding to each pixel in the area corresponding to each fruit.

[0103] [Output section 13β] The output unit 13β further outputs to the display unit 131β the prediction result predicted by the prediction unit 126. Except for this, the output unit 13β has the same configuration as the output unit 13 in the first embodiment.

[0104] (Display section 131β) The display unit 131β further displays the prediction result predicted by the prediction unit 126. Except for this, the display unit 131β has the same configuration as the display unit 131 in the first embodiment.

[0105] [Control method S3 of estimation device 1β] Next, a control method S3 of the estimation device 1β according to the third embodiment will be described with reference to Figures 11 and 12. Figure 11 is a flow chart showing an example of the control method S3 of the estimation device 1β according to the third embodiment.

[0106] As shown in FIG. 11, the control method S3 of the estimation device 1β includes an acquisition step S31, a detection step S32, an identification step S33, a clustering step S34, an estimation step S35, a separation step S36, a prediction step S37, and an output step S38.

[0107] The acquiring step S31, the detecting step S32, the identifying step S33, the clustering step S34, and the estimating step S35 are similar to the acquiring step S11, the detecting step S12, the identifying step S13, the clustering step S14, and the estimating step S15 in the first embodiment, respectively.

[0108] [Separation process S36] In the separation step S36, the separation unit 125 of the estimation device 1β separates the target region into regions corresponding to individual fruits based on the boundaries between the fruits estimated by the estimation unit 124 of the estimation device 1β in the estimation step S35.

[0109] (An example of the separation step S36) An example of the separation step S36 will be described below with reference to Fig. 12. Fig. 12 is a diagram for explaining an example of the separation step S36.

[0110] The separation unit 125 separates the target region B1 into a region corresponding to the fruit C1 and a region corresponding to the fruit C2, using a region division method such as graph cut, random walker division, and trimming, based on the boundary D1 between the fruits C1 and C2. This allows the overlapping fruits on the captured image I1 to be separated into regions corresponding to individual fruits without being destroyed.

[0111] [Prediction step S37] In the prediction step S37, the prediction unit 126 of the estimation device 1β predicts at least one of the harvest time and yield of each fruit, based on color information corresponding to each pixel in the area corresponding to each fruit.

[0112] (An example of the prediction process S37) An example of the prediction step S37 will be described below with reference to FIG.

[0113] First, the prediction unit 126 acquires color information corresponding to each pixel in an area B1X corresponding to fruit C1 (area corresponding to fruit) and color information corresponding to each pixel in an area B1Y corresponding to fruit C2 (area corresponding to fruit) from the captured image I1, which is an RGB image.

[0114] Next, the prediction unit 126 evaluates the ripeness of fruits C1 and C2 based on color information corresponding to each pixel in an area B1X corresponding to fruit C1 and color information corresponding to each pixel in an area B1Y corresponding to fruit C2.

[0115] For example, the prediction unit 126 assumes that the ripeness of a white fruit is 0 and that of a bright red fruit is 5, and evaluates the ripeness of fruit C1 as 3 and that of fruit C2 as 3.2 based on the proportion, intensity, and uniformity of red color occupied by each pixel in the area corresponding to each fruit. Regarding the uniformity of red color, the prediction unit 126 evaluates that the smaller the number of superpixels in the area corresponding to each fruit, the higher the uniformity of red color and the higher the ripeness of the fruit.

[0116] Next, the prediction unit 126 predicts at least one of the harvest time and the yield of each of the fruits C1 and C2 based on the ripeness of the fruits C1 and C2. In this case, the prediction unit 126 predicts that the higher the ripeness of the fruit, the earlier the harvest time. For example, when the ripeness of the fruit C1 is 3 and the ripeness of the fruit C2 is 3.2, the prediction unit 126 predicts that the harvest time of the fruit C1 will be the day after tomorrow and the harvest time of the fruit C2 will be tomorrow. In this way, the prediction unit 126 can predict the harvest time and the yield of each of the fruits with high accuracy by referring to the color information and ripeness of the fruits C1 and C2.

[0117] [Output process S38] In the output step S38, the output unit 13γ further outputs the prediction result predicted by the prediction unit 126 in the prediction step S37 to the display unit 131β.

[0118] (An example of output process S38) The output unit 13γ causes the display unit 131β to display the predicted harvest times, ie, the harvest time of the fruit C1 is the day after tomorrow and the harvest time of the fruit C2 is tomorrow. Instead of or in addition to outputting the harvest times of the fruits C1 and C2 to the display unit 131β, the output unit 13γ may output the harvest times of the fruits C1 and C2 by voice.

[0119] <Embodiment 4> An estimation device according to an aspect of the present invention may predict at least one of the harvest time and yield of each fruit further based on temperature information, like the estimation device 1γ according to embodiment 4.

[0120] Hereinafter, the fourth embodiment will be described with reference to Figures 13 to 16. For convenience of explanation, the members having the same functions as those explained in the above embodiments are denoted by the same reference numerals, and the explanation thereof will be omitted.

[0121] [Estimation device 1γ] Fig. 13 is a block diagram showing an example of the configuration of an estimation device 1γ according to embodiment 4. As shown in Fig. 13, the estimation device 1γ includes an acquisition unit 11γ and a control unit 12γ instead of the acquisition unit 11 and the control unit 12β in embodiment 3. Except for this, the estimation device 1γ has the same configuration as the estimation device 1β according to embodiment 3.

[0122] [Acquisition unit 11γ] The acquisition unit 11γ further acquires a thermal image corresponding to the captured image.

[0123] [Control unit 12γ] The control unit 12γ further includes a measurement unit 127, and includes a prediction unit 126γ instead of the prediction unit 126. Except for this, the control unit 12γ has the same configuration as the control unit 12β in the third embodiment.

[0124] (Measurement part 127) The measurement unit 127 measures the temperature of each fruit based on the boundaries between the fruits and the thermal image.

[0125] (Prediction unit 126γ) The prediction unit 126γ predicts at least one of the harvest time and yield of each fruit, further based on temperature information indicated by the temperature of each fruit measured by the measurement unit 127. [Control method S4 of estimation device 1γ] Next, a control method S4 of the estimation device 1γ according to the fourth embodiment will be described with reference to Figures 14 and 15. Figure 14 is a flow chart showing an example of the control method S4 of the estimation device 1γ according to the fourth embodiment.

[0126] As shown in FIG. 14, the control method S4 of the estimation device 1γ includes an acquisition step S41, a detection step S42, an identification step S43, a clustering step S44, an estimation step S45, a separation step S46, a measurement step S47, a prediction step S48, and an output step S49.

[0127] The detection step S42, the identification step S43, the clustering step S44, the estimation step S45, the separation step S46, and the output step S49 are similar to the detection step S32, the identification step S33, the clustering step S34, the estimation step S35, the separation step S36, and the output step S38 in the third embodiment, respectively.

[0128] [Acquisition step S41] In the acquisition step S41, the acquisition unit 11γ of the estimation device 1γ further acquires a thermal image corresponding to the captured image.

[0129] (An example of the acquisition step S41) An example of the obtaining step S41 will be described below with reference to Fig. 15. Fig. 15 is a diagram showing an example of a thermal image I3.

[0130] The acquisition unit 11γ also functions as an imaging unit such as a thermal camera that uses infrared rays, and acquires a thermal image I3 in which a plurality of fruits C1 to C6 are captured.

[0131] The thermal camera may be located adjacent to an image camera that captures the captured image I1 and a depth camera that captures the depth image I2, or a camera that captures both the captured image I1 and the depth image I2.

[0132] This makes it easier for the thermal image I3 to correspond to the captured image I1 and the depth image I2, for example, by making the coordinates of the fruits C1 and C2 in the thermal image I3 and the coordinates of the fruits C1 and C2 in the captured image I1 and the depth image I2 approximately coincident with each other. As a result, it becomes easier for the measurement unit 127 to map the position information of the captured image I1 and the depth image I2 and the boundary D1 between the fruits C1 and C2 onto the thermal image I3, and makes it easier to measure the temperatures of the fruits C1 and C2 in the thermal image I3.

[0133] The acquisition unit 11γ may acquire a thermal image I3 in which a plurality of fruits C1 to C6 are captured, from a thermal camera or the like external to the estimation device 1γ.

[0134] (Measurement process S47) In the measurement step S47, the measurement unit 127 of the estimation device 1γ measures the temperature of each fruit based on the boundaries between the fruits and the thermal image.

[0135] (An example of the measurement process S47) An example of the measuring step S47 will be described below with reference to Figures 15 and 16. Figure 16 is a diagram for explaining an example of the measuring step S47.

[0136] First, the measurement unit 127 maps the positional information of the captured image I1 and the depth image I2, as well as the boundary D1 between fruits C1 and C2, onto the thermal image I3 shown in FIG. 15 as the target area B1γ in the thermal image I3 shown in FIG. 16 and the boundary D1γ between fruits C1 and C2.

[0137] Next, the measurement unit 127 measures the temperature of a region B1γX (region corresponding to the fruit) corresponding to the fruit C1 and a region B1γY (region corresponding to the fruit) corresponding to the fruit C2 in the target region B1γ of the thermal image I3.

[0138] For example, the measurement unit 127 measures the temperatures of fruits C1 and C2 based on the gradation of an area B1γX corresponding to fruit C1 and an area B1γY corresponding to fruit C2, respectively. In the example shown in Fig. 16, fruit C1 is closer to white and has a larger gradation value, so the temperature of fruit C1 measured by the measurement unit 127 is higher than the temperature of fruit C2, e.g., the temperature of fruit C1 is 21.2°C and the temperature of fruit C2 is 20.2°C.

[0139] [Prediction step S48] In the prediction step S48, the prediction unit 126γ of the estimation device 1γ predicts at least one of the harvest time and yield of each fruit, further based on temperature information indicated by the temperature of each fruit measured by the measurement unit 127 in the measurement step S47.

[0140] (An example of the prediction step S48) An example of the prediction step S48 will be described below with reference to FIG.

[0141] The prediction unit 126γ predicts the harvest time of the fruits C1 and C2 based on temperature information indicated by the temperatures of the fruits C1 and C2 measured by the measurement unit 127 in the measurement step S47.

[0142] Here, it is assumed that fruits C1 and C2 are continuously monitored by measuring unit 127 measuring the temperature of fruits C1 and C2 after flowering once a day, and when the accumulated temperature of either fruit reaches 400°C, the fruit with the accumulated temperature of 400°C is harvested.

[0143] In this case, if the temperature of fruit C1 measured by the measurement unit 127 in the measurement step S47 on the day is 21.2°C and the accumulated temperature of fruit C1 is 378.8°C, the prediction unit 126γ predicts that the planned harvest time of fruit C1 is tomorrow. Also, if the temperature of fruit C2 measured by the measurement unit 127 in the measurement step S47 on the day is 20.2°C and the accumulated temperature of fruit C2 is 377.8°C, the prediction unit 126γ predicts that the planned harvest time of fruit C2 is the day after tomorrow.

[0144] In this way, the measurement unit 127 can continuously monitor the fruits C1 and C2 by measuring the temperatures of the fruits C1 and C2 from when the fruits C1 and C2 are small after flowering, and calculate the accumulated temperature of each of the fruits C1 and C2. Furthermore, the measurement unit 127 can measure the temperature of each of the fruits C1 and C2 even if the fruits C1 and C2 overlap. Therefore, the prediction unit 126γ can predict the harvest time and yield of each fruit with higher accuracy by further referring to the temperatures of the fruits C1 and C2 measured by the measurement unit 127.

[0145] As is clear from the above example, the prediction unit 126γ may predict at least one of the harvest time and yield of each fruit based only on the temperature information indicated by the temperature of each fruit without referring to the color information of each fruit. However, the prediction unit 126γ can predict at least one of the harvest time and yield of each fruit with higher accuracy by predicting at least one of the harvest time and yield of each fruit based on both the color information and the temperature information of each fruit.

[0146] <Embodiment 5> The estimation device according to an aspect of the present invention may identify a second object region in which second included regions including each leaf in the captured image overlap, and estimate a boundary between leaves in the second object region, like the estimation device 1δ according to embodiment 5. The estimation device according to an aspect of the present invention may separate the second object region into individual leaves based on the boundaries between the leaves, reconstruct an entire image of the leaf from the region corresponding to each leaf, and predict fruit yield based on the area of ​​the entire leaf, like the estimation device 1δ according to embodiment 5.

[0147] Hereinafter, the fifth embodiment will be described with reference to Figures 17 to 23. For convenience of explanation, the members having the same functions as those explained in the above embodiments are denoted by the same reference numerals, and the explanation thereof will be omitted.

[0148] [Estimation device 1δ] Fig. 17 is a block diagram showing an example of the configuration of an estimation device 1δ according to embodiment 5. As shown in Fig. 17, the estimation device 1δ includes an acquisition unit 11δ, a control unit 12δ, an output unit 13δ, and a storage unit 14δ, instead of the acquisition unit 11, the control unit 12β, the output unit 13β, and the storage unit 14 in embodiment 3.

[0149] [Acquisition unit 11δ] The acquiring unit 11δ acquires a captured image including a plurality of leaves. Except for this, the acquiring unit 11δ has the same configuration as the acquiring unit 11 in the third embodiment.

[0150] [Control unit 12δ] The control unit 12δ includes a detection unit 121δ, a specification unit 122δ, a clustering unit 123δ, an estimation unit 124δ, a separation unit 125δ, and a prediction unit 126δ, instead of the detection unit 121, the specification unit 122, the clustering unit 123, the estimation unit 124, the separation unit 125, and the prediction unit 126 in the embodiment 3. The control unit 12δ further includes a restoration unit 128.

[0151] (Detection unit 121δ) The detector 121δ detects the second included regions that include each leaf in the captured image acquired by the acquirer 11δ. Except for this, the detector 121δ has the same configuration as the detector 121 in the third embodiment.

[0152] (Specific part 122δ) The identification unit 122δ further identifies a second target region including a region where the second encapsulated regions overlap each other. Except for this, the identification unit 122δ has the same configuration as the identification unit 122 in the third embodiment.

[0153] (Clustering part 123δ) The clustering unit 123δ further clusters each pixel in the second target region by unsupervised learning using the depth information corresponding to the pixel. Except for this, the clustering unit 123δ has the same configuration as the clustering unit 123 in the third embodiment.

[0154] (Estimation part 124δ) The estimation unit 124δ further estimates boundaries between leaves in the second target region based on the result of the clustering by the clustering unit 123δ. Except for this, the estimation unit 124δ has the same configuration as the estimation unit 124 in the third embodiment.

[0155] (separation part 125δ) Separation unit 125δ further separates the second target region into individual leaves based on the boundaries between the leaves estimated by estimation unit 124δ. Except for this, separation unit 125δ has the same configuration as separation unit 125 in the third embodiment.

[0156] (Restore section 128) The restoration unit 128 restores the entire image of each leaf from the area corresponding to the individual leaf based on the captured image.

[0157] (Prediction section 126δ) The prediction unit 126δ predicts the fruit yield based on the area of ​​the entire leaf restored by the restoration unit 128. Except for this, the prediction unit 126δ has the same configuration as the prediction unit 126 in the third embodiment.

[0158] [Output section 13δ] The output unit 13δ further outputs the prediction result predicted by the prediction unit 126δ to a display unit 131δ. Except for this, the output unit 13δ has the same configuration as the output unit 13β in the third embodiment.

[0159] (Display section 131δ) The display unit 131δ further displays the prediction result predicted by the prediction unit 126δ. Except for this, the display unit 131δ has the same configuration as the display unit 131β in the third embodiment.

[0160] [Storage unit 14δ] The storage unit 14δ further includes a second trained model 142 and a third trained model 143. Except for this, the storage unit 14δ has the same configuration as the storage unit 14 in the third embodiment. The second trained model 142 and the third trained model 143 will be described later.

[0161] [Control method S5 of estimation device 1δ] 18 to 21, the control method S5 of the estimation device 1δ according to the fifth embodiment will be described. Fig. 18 is a flow chart showing an example of the control method S5 of the estimation device 1δ according to the fifth embodiment.

[0162] As shown in FIG. 18, the control method S5 of the estimation device 1δ includes an acquisition step S51, a detection step S52, an identification step S53, a clustering step S54, an estimation step S55, a separation step S56, a restoration step S57, a prediction step S58, and an output step S59.

[0163] [Acquisition step S51] In an acquisition step S51, an acquisition unit 11δ of an estimation device 1δ acquires a captured image including a plurality of leaves.

[0164] (An example of the acquisition step S51) An example of the acquiring step S51 will be described below with reference to Fig. 19. Fig. 19 is a diagram showing an example of the captured image I1δ.

[0165] The acquisition unit 11δ functions as an imaging unit such as an image camera, and captures a plurality of fruits C1 to C6 and a plurality of leaves C7 and C8 to acquire a captured image I1δ in a state in which the encapsulated regions A1 to A6 and the second encapsulated regions A7 and A8 are not detected and the target regions B1 to B4 and B6 are not specified. The acquisition unit 11δ also functions as an imaging unit such as a depth camera, and captures a plurality of fruits C1 to C6 and a plurality of leaves C7 and C8 to acquire a depth image I2δ in a state in which the target regions B5 and B7 are not mapped.

[0166] The acquisition unit 11δ may acquire an image I1δ in a state where a plurality of fruits C1 to C6 and a plurality of leaves C7 and C8 are captured, the encapsulated regions A1 to A6 and the second encapsulated regions A7 and A8 are not detected, and the target regions B1 and B6 are not specified, from an image camera or the like external to the estimation device 1δ. The acquisition unit 11δ may also acquire a depth image I2δ in a state where a plurality of fruits C1 to C6 and a plurality of leaves C7 and C8 are captured, and the target regions B5 and B7 are not mapped, from a depth camera or the like external to the estimation device 1δ.

[0167] 19, the acquiring unit 11δ does not need to acquire an image in which the fruits C1 to C6 are captured separately from the captured image I1δ. However, if the captured images are of the same fruit vegetables or fruit trees, the acquiring unit 11δ may acquire an image in which a plurality of leaves are captured and an image in which a plurality of fruits are captured separately.

[0168] [Detection step S52] In the detection step S52, the detection unit 121δ of the estimation device 1δ detects second included regions including each leaf in the captured image acquired by the acquisition unit 11δ.

[0169] (An example of the detection step S52) An example of the detection step S52 will be described below with reference to FIG.

[0170] The detection unit 121δ may use the second trained model 142 to detect the second encapsulated regions A7 and A8 in the captured image I1δ.

[0171] The second trained model 142 was generated by machine learning using as training data a pair of a second training image of multiple leaves and a second contained area in the second training image that contains each leaf.

[0172] In addition to detecting the encapsulation regions A1 to A6 containing the fruits C1 to C6, the detection unit 121δ inputs the captured image I1δ to the second trained model 142 to detect the second encapsulation regions A7 and A8 output from the second trained model 142. That is, the detection unit 121δ detects position information of each pixel corresponding to the leaves C7 and C8 contained in the second encapsulation regions A7 and A8. In this way, the detection unit 121δ can detect the second encapsulation regions A7 and A8 with high accuracy and efficiency by using the second trained model 142.

[0173] When the captured image includes flowers, the detection unit 121δ may detect a third encapsulation region that encapsulates each flower in addition to the encapsulation region that encapsulates each fruit and a second encapsulation region that encapsulates each leaf in a manner similar to the manner in which the detection unit 121δ detects these encapsulation regions.

[0174] [Specific process S53] In the identifying step S53, the identifying unit 122δ of the estimating device 1δ further identifies a second target region including a region where the second connotative regions overlap each other.

[0175] (An example of the identification step S53) An example of the specifying step S53 will be described below with reference to FIG.

[0176] The identification unit 122δ further identifies a second target region B6 including an area where the second encapsulated regions A7 and A8 overlap with each other, in the same manner as the target region B1. That is, the identification unit 122δ identifies position information of each pixel corresponding to the target region B6. As shown in FIG. 19, the identification unit 122δ may identify the area where the second encapsulated regions A7 and A8 overlap with each other as the target region B6. The identification unit 122δ may identify the area where the overlapping second encapsulated regions A7 and A8 are combined with each other as the target region.

[0177] [Clustering step S54] In a clustering step S54, the clustering unit 123δ of the estimation device 1δ further clusters each pixel in the second target region identified in the identification step S53 by unsupervised learning using the depth information corresponding to the pixel.

[0178] (An example of the clustering step S54) An example of the clustering step S54 will be described below with reference to Figures 19 and 20. Figure 20 is a diagram showing an example of a depth image I2δ.

[0179] Clustering section 123δ may perform clustering on each pixel in second target region B6 using the depth information corresponding to that pixel.

[0180] In this case, the clustering unit 123δ first maps the position information of the target region B6 of the captured image I1δ identified by the identification unit 122δ to the depth image I2δ as a target region B7. Next, the clustering unit 123δ acquires depth information from the target region B7 of the depth image I2δ that corresponds to the target region B6 of the captured image I1δ, and performs clustering using the depth information.

[0181] In the depth image I2δ, the region corresponding to the leaf C8 is whiter than the region corresponding to the leaf C7, and therefore the region corresponding to the leaf C8 has a deeper depth than the region corresponding to the leaf C7. Therefore, the clustering unit 123δ clusters each pixel in the target region B6 into a region corresponding to the leaf C7 and a region corresponding to the leaf C8.

[0182] [Estimation process S55] In the estimation step S55, the estimation unit 124δ of the estimation device 1δ estimates boundaries between leaves in the target region based on the result of the clustering in the clustering step S54.

[0183] (An example of the estimation process S55) An example of the estimation step S55 will be described below with reference to Fig. 21. Fig. 21 is a diagram showing an example of the boundary between leaves C7 and C8 in the second target region B6.

[0184] The estimation unit 124δ estimates the boundary D2 between the leaves C7 and C8 in the target region B6 based on the result of the clustering by the clustering unit 123δ. For example, the estimation unit 124δ estimates that the boundary between the region corresponding to the leaves C7 and the region corresponding to the leaves C8 clustered by the clustering unit 123δ is the boundary D2 between the leaves C7 and C8. That is, the estimation unit 124δ estimates position information such as the coordinates of the boundary D2. This allows the estimation unit 124δ to accurately and efficiently identify the overlapping leaves C7 and C8 on the captured image I1δ as individual leaves C7 and C8.

[0185] [Separation process S56] In a separation step S56, a separation section 125δ of the estimation device 1δ separates the second target region into regions corresponding to individual fruits based on the boundaries between leaves estimated by the estimation section 124δ in the estimation step S55.

[0186] (An example of the separation step S56) An example of the separation step S56 will be described below with reference to Fig. 22. Fig. 22 is a diagram for explaining separation between leaves C7 and C8 in the second target region B6.

[0187] Based on the boundary D1 between leaves C7 and C8, the separator 125δ separates the second target region B6 into a region B6X corresponding to leaf C7 and a region B6Y corresponding to leaf C8, using the region division method described above.

[0188] [Restoration process S57] In the restoration step S57, the restoration unit 128 of the estimation device 1δ restores an entire image of each leaf from an area corresponding to the individual leaf, based on the captured image.

[0189] (An example of the restoration process S57) An example of the restoration step S57 will be described below with reference to Figures 19 and 23. Figure 23 is a diagram for explaining the restoration of the entire image of a leaf C8 (the entire image of the leaf) C8δ.

[0190] First, the restoration unit 128 extracts a region corresponding to the leaf C8 from the second encapsulated region A8, excluding the region of pixels overlapping and hidden by the leaf C7, based on the color information and brightness information of the pixels of the captured image I1δ.

[0191] Next, the restoration unit 128 restores the entire image C8δ of the leaf C8 from the region corresponding to the leaf C8 excluding the region of pixels overlapping and hidden by the leaf C7, using the third trained model 143 stored in the storage unit 14δ. The third trained model 143 is generated by machine learning using as training data a pair of a region corresponding to a leaf that is partially overlapped and hidden by other leaves and a region corresponding to a leaf that is not hidden.

[0192] The restoration unit 128 obtains an overall image C8δ of leaf C8 output from the third trained model 143 by inputting the area corresponding to leaf C8, excluding the area of ​​pixels overlapping and hidden by leaf C7, into the third trained model 143.

[0193] In the above example, the third trained model 143 is stored in the storage unit 14δ, but in this embodiment, the storage unit 14δ may store a plurality of types of predetermined leaf shape information instead of or in addition to the third trained model 143. In addition, the restoration unit 128 may use a template matching technique to restore the entire image C8δ of the leaf C8 from the area corresponding to the leaf C8 excluding the area of ​​pixels overlapping and hidden by the leaf C7.

[0194] In this case, the restoration unit 128 selects, from among multiple types of specified leaf shape information stored in the memory unit 14δ, the leaf shape information whose shape and contour are closest to the area corresponding to the leaf C8, excluding the area of ​​pixels overlapping and hidden by the leaf C7.

[0195] The shape information of the plurality of types of predetermined leaves may be 3D data that can be rotated 360°. This allows the restoration unit 128 to rotate the shape information of the plurality of types of predetermined leaves according to the angle at which the leaf C8 is captured, and to superimpose it on the region corresponding to the leaf C8, excluding the region of pixels that overlap with and are hidden by the leaf C7. As a result, the restoration unit 128 can select the shape information of the predetermined leaf whose shape and contour are closest to the region corresponding to the leaf C8 with high accuracy.

[0196] The restoration unit 128 restores the entire image C8δ of the leaf C8 by complementing the area of ​​pixels that are overlapped and hidden by the leaf C7 based on the shape information of the selected specific leaf and the color information and brightness information of the pixels corresponding to the leaf C8 excluding the area of ​​pixels that are overlapped and hidden by the leaf C7.

[0197] In this way, the restoration section 128 can restore the entire image C8δ of the leaf C8 with high accuracy from the region corresponding to the leaf C8 by referring to the captured image I1δ.

[0198] [Prediction step S57] In the prediction step S57, the prediction unit 126δ of the estimation device 1δ predicts the fruit yield based on the total leaf area restored by the restoration unit 128.

[0199] (An example of the prediction process S57) An example of the prediction step S57 will be described below with reference to FIGS.

[0200] First, for leaf C7 that is not hidden by overlapping with other leaves, the prediction unit 126δ calculates the area of ​​leaf C7 based on position information of the pixel area of ​​leaf C7 in the captured image I1δ. For leaf C8 that is partially hidden by overlapping with leaf C7, the prediction unit 126δ calculates the area of ​​leaf C8 based on position information of the pixel area of ​​leaf C8 in the captured image I1δ and the pixel area of ​​the whole image C8δ of leaf C8 restored by the restoration unit 128.

[0201] Next, the prediction unit 126δ predicts the total leaf area of ​​the tree based on the total area of ​​the leaves C7 and C8 and the number of leaves in one fruit vegetable or fruit tree. In this case, the prediction unit 126δ may use the number of leaves in the tree that is stored in advance in the storage unit 14δ, or may use the number of leaves input from the user. The prediction unit 126δ may also obtain the number of leaves in the tree from the number of leaf-containing regions in the captured image of the entire tree detected by the detection unit 121δ.

[0202] Next, the prediction unit 126δ predicts the area of ​​the pixel area of ​​the leaf C8 that is overlapped and hidden by the leaf C7 of the tree, based on the area of ​​the pixel area of ​​the leaf C8 that is overlapped and hidden by the leaf C7 of the tree, and the number of leaves of the tree. The prediction unit 126δ also predicts the total area of ​​the leaf area capable of photosynthesis of the tree by subtracting the area of ​​the leaf area of ​​the tree that is overlapped and hidden by the leaf C7 of the tree from the total area of ​​the leaves of the tree.

[0203] The prediction unit 126δ predicts the fruit yield of a tree based on the total area of ​​the leaf regions capable of photosynthesis in the tree. In this way, the prediction unit 126δ can predict the fruit yield with high accuracy by referring to the area of ​​the entire image of the restored leaves.

[0204] [Output process S58] In the output step S58, the output unit 13δ further outputs the prediction result predicted by the prediction unit 126δ in the prediction step S57 to the display unit 131δ.

[0205] (An example of output process S58) The output unit 13δ may display the yield of fruit from one fruit vegetable or fruit tree on the display unit 131δ. Also, the output unit 13δ may output the yield of the fruit by voice instead of or in addition to outputting the yield of the fruit to the display unit 131δ.

[0206] <Embodiment 6> The estimation device according to an aspect of the present invention may reconstruct an entire image of each fruit from a region corresponding to each fruit based on a captured image, like the estimation device 1ε according to the sixth embodiment.

[0207] Hereinafter, the sixth embodiment will be described with reference to Figures 24 to 26. For convenience of explanation, the members having the same functions as those explained in the above-mentioned embodiments are denoted by the same reference numerals, and the explanation thereof will be omitted.

[0208] [Estimation device 1ε] Fig. 24 is a block diagram showing an example of the configuration of an estimation device 1ε according to embodiment 6. As shown in Fig. 24, the estimation device 1ε includes a control unit 12ε, an output unit 13ε, and a storage unit 14ε, instead of the control unit 12β, the output unit 13β, and the storage unit 14 in embodiment 3. Except for this, the estimation device 1ε has the same configuration as the estimation device 1β according to embodiment 3.

[0209] [Control unit 12ε] The control unit 12ε further includes a second restoration unit 128ε, and includes a prediction unit 126ε instead of the prediction unit 126β in the third embodiment. Except for this, the control unit 12ε has the same configuration as the control unit 12β in the third embodiment.

[0210] (Second restoration portion 128ε) The second restoration section 128ε restores an entire image of each fruit from the regions corresponding to the individual fruits based on the captured image.

[0211] (Prediction section 126ε) The prediction unit further predicts the quality of the fruit based on the overall image of the fruit.

[0212] [Output section 13ε] The output unit 13ε further outputs the prediction result predicted by the prediction unit 126ε to a display unit 131ε. Except for this, the output unit 13ε has the same configuration as the output unit 13β in the third embodiment.

[0213] (Display section 131ε) The display unit 131ε further displays the prediction result predicted by the prediction unit 126ε. Except for this, the display unit 131ε has the same configuration as the display unit 131β in the third embodiment.

[0214] [Storage section 14ε] The storage unit 14ε further includes a fourth trained model 144 and shape information of a plurality of types of predetermined fruits. Except for this, the storage unit 14ε has the same configuration as the storage unit 14 in the third embodiment. The fourth trained model 144 and the shape information of a plurality of types of predetermined fruits will be described later.

[0215] [Control method S6 of estimation device 1ε] A control method S6 for the estimation device 1ε according to the sixth embodiment will be described with reference to Figures 25 and 26. Figure 25 is a flow chart showing an example of the control method S6 for the estimation device 1ε according to the sixth embodiment.

[0216] As shown in FIG. 25, the control method S6 of the estimation device 1ε includes an acquisition step S61, a detection step S62, an identification step S63, a clustering step S64, an estimation step S65, a separation step S66, a second restoration step S67, a prediction step S68, and an output step S69.

[0217] The acquisition step S61, detection step S62, identification step S63, clustering step S64, estimation step S65 and separation step S66 are similar to the acquisition step S31, detection step S32, identification step S33, clustering step S34, estimation step S35 and separation step S36 in embodiment 3, respectively.

[0218] [Second restoration step S67] In the second reconstruction step S67, the second reconstruction unit 128ε of the estimation device 1ε reconstructs an entire image of each fruit from the regions corresponding to the individual fruits based on the captured image.

[0219] (An example of the second restoration step S67) An example of the second restoration step S67 will be described below with reference to Figures 3 and 26. Figure 26 is a diagram for explaining the restoration of the entire image of the fruit C2 (the entire image of the fruit) C2ε.

[0220] First, the second restoration unit 128ε extracts a region corresponding to the fruit C2 from the encapsulated region A2, excluding the region of pixels overlapping and hidden by the fruit C1, based on color information and brightness information of pixels in the captured image I1 of FIG.

[0221] Next, the second restoration unit 128ε restores an entire image C2ε of the fruit C2 from the area corresponding to the fruit C2 excluding the area of ​​pixels overlapping and hidden by the fruit C1, using the fourth trained model 144 stored in the storage unit 14ε. The fourth trained model 144 is generated by machine learning using as training data a pair of an area corresponding to a fruit partially overlapping and hidden by other fruits and an area corresponding to a fruit that is not hidden.

[0222] The second restoration unit 128ε obtains an overall image C2ε of fruit C2 output from the fourth trained model 144 by inputting the area corresponding to fruit C2 excluding the area of ​​pixels overlapping with fruit C1 into the fourth trained model 144.

[0223] In the above example, the fourth trained model 144 is stored in the storage unit 14ε, but in this embodiment, the fourth trained model 144 does not have to be stored in the storage unit 14ε. In addition, the second restoration unit 128ε may use a template matching technique to restore the entire image C2ε of the fruit C2 from the area corresponding to the fruit C2 excluding the area of ​​pixels overlapping and hidden by the fruit C1.

[0224] In this case, the second restoration unit 128ε selects, from the shape information of multiple types of specified fruits stored in the memory unit 14ε, the fruit shape information whose shape and contour are closest to the shape of the area corresponding to fruit C2 excluding the area of ​​pixels overlapping with fruit C1.

[0225] The shape information of the plurality of kinds of predetermined fruits may be 3D data that can be rotated 360°. This allows the second restoration unit 128ε to rotate the shape information of the plurality of kinds of predetermined fruits according to the angle at which the fruit C2 is captured, and superimpose it on the area corresponding to the fruit C2, excluding the area of ​​pixels that overlap with and are hidden by the fruit C1. As a result, the second restoration unit 128ε can select the shape information of the predetermined fruit whose shape and contour are closest to the area corresponding to the fruit C2 with high accuracy.

[0226] The second restoration unit 128ε restores the entire image C2ε of fruit C2 by complementing the area of ​​pixels that are overlapped and hidden by fruit C1 based on the shape information of a selected specific fruit and the color information and brightness information of pixels corresponding to fruit C2 excluding the area of ​​pixels that are overlapped and hidden by fruit C1.

[0227] In this way, the second restoration section 128ε can restore the entire image C2ε of the fruit C2 with high accuracy from the region corresponding to the fruit C2 by referring to the captured image I1.

[0228] [Prediction step S68] In a prediction step S68, the prediction unit 126ε of the estimation device 1ε further predicts the quality of the fruit based on the overall image of the fruit reconstructed by the second reconstruction unit 128ε in the second reconstruction step S67.

[0229] (An example of the prediction step S68) An example of the prediction step S68 will be described below with reference to FIGS.

[0230] The prediction unit 126ε selects, from among the shape information of a plurality of types of predetermined fruits, shape information of a predetermined fruit whose contour is closest to that of the overall image C2ε of the fruit C2 shown in FIG.

[0231] The prediction unit 126ε predicts the quality of the fruit C2 based on the shape information of the selected predetermined fruit. The prediction unit 126ε may predict the quality of the fruit C2 based on color information and brightness information of pixels of the overall image C2ε of the fruit C2. In this case, the prediction unit 126ε may evaluate the quality of the fruit C2 in a plurality of stages, such as excellent, excellent, good, fair, and none, in descending order of evaluation.

[0232] For example, the more regular, such as symmetrical, the more the shape information of the selected predetermined fruit is, the higher the grade of the fruit C2 is evaluated by the prediction unit 126ε. The higher the percentage, intensity, and uniformity of red in the color information of the overall image C2ε of the fruit C2 is, the higher the grade of the fruit C2 is evaluated by the prediction unit 126ε. The higher the brightness and gloss indicated by the brightness information of the overall image C2ε of the fruit C2 is, the higher the grade of the fruit C2 is evaluated by the prediction unit 126ε. This allows the prediction unit 126ε to predict the quality of the fruit C2 with high accuracy, even if the fruit C2 is partially hidden by overlapping with the fruit C1.

[0233] The prediction unit 126ε may also predict the grade of each fruit based on the overall image of each fruit restored based on each captured image. This allows the prediction unit 126ε to predict the number of fruits corresponding to each grade on a single fruit vegetable or fruit tree. Since the price of fruits differs depending on the grade, the prediction unit 126ε can ultimately predict the sales amount when each fruit on the tree is sold.

[0234] [Output process S69] In the output step S69, the output unit 13ε further outputs the prediction result predicted by the prediction unit 126ε in the prediction step S68 to the display unit 131ε.

[0235] (An example of output process S69) The output unit 13ε may display the quality of the fruit C2, such as the grade of the fruit C2, on the display unit 131ε. Also, the output unit 13ε may output the quality of the fruit C2 by voice instead of or in addition to outputting the quality of the fruit C2 to the display unit 131ε.

[0236] <Example of software implementation> The functions of the estimation devices 1, 1α, 1β, 1γ, 1δ and 1ε (hereinafter referred to as the "devices") can be realized by an estimation program for causing a computer to function as the device, and by an estimation program for causing a computer to function as each control block of the device (in particular, each part included in control units 12, 12α, 12β, 12γ, 12δ and 12ε).

[0237] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the estimation program. The control device and the storage device execute the estimation program to realize each function described in each of the above embodiments.

[0238] The estimation program may be non-transiently recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the estimation program may be provided to the device via any wired or wireless transmission medium.

[0239] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0240] <Additional Notes> The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0241] In addition, the diagnostic device according to each embodiment can improve productivity, ensure a sustainable food production system, and practice resilient agriculture, thereby contributing to the achievement of Goal 2 of the Sustainable Development Goals (SDGs), "Zero Hunger." [Industrial Applicability]

[0242] INDUSTRIAL APPLICABILITY The present invention can be used for greenhouse horticultural crops in which the problem of overlapping fruits is likely to occur in captured images used to predict the harvest time and yield of fruits. [Explanation of symbols]

[0243] 1, 1α, 1β, 1γ, 1δ, 1ε estimator 11, 11γ, 11δ acquisition part C7, C8 leaves 121, 121δ Detector 122, 122α, 122δ specific part 123, 123α, 123δ Clustering Section 124, 124δ estimation part 125, 125δ Separation part 126, 126β, 126γ, 126δ, 126ε prediction section 127 Measuring section 128 Restoration Department 128ε Second reconstruction part A1, A2, A3, A4, A5, A6 inclusion area A7, A8 Second Containment Region B1, B1α, B1γ, B2, B3, B4, B5 Target area B1X, B1Y, B1γX, B1γY regions (regions corresponding to fruit) B6, B7 Second target area C1, C2, C3, C4, C5, C6 fruit C2ε Overall image (overall image of the fruit) C8δ Overall image (overall image of leaf) D1, D1γ, D2 boundaries I1, I1α, I1δ images I2, I2δ depth image I3 Thermal Imaging S1, S2, S3, S4, S5, S6 control method S11, S21, S31, S41, S51, S61 acquisition process S12, S22, S32, S42, S52, S62 Detection process S13, S23, S33, S43, S53, S63 Specific process S14, S24, S34, S44, S54, S64 Clustering process S15, S25, S35, S45, S55, S65 Estimation process

Claims

1. A detection unit that detects an inclusion region that includes each of the fruits in a captured image of a plurality of fruits; An identification unit that identifies a target region including an area where the encapsulation regions overlap each other; A clustering unit that clusters each pixel in the target region by unsupervised learning using depth information corresponding to the pixel; and an estimation unit that estimates boundaries between fruits in the target region based on a result of the clustering.

2. The estimation device according to claim 1 , wherein the identification unit is configured to identify, as the target region, a region in which the connotative regions overlap each other.

3. The estimation device according to claim 1 , wherein the identification unit is configured to identify, as the target region, a region obtained by combining the overlapping connotative regions.

4. The estimation device according to any one of claims 1 to 3, characterized in that the identification unit determines whether or not there is another encapsulation area that overlaps the encapsulation area for each of the encapsulation areas, and identifies the target area when it is determined that the other encapsulation area exists.

5. The estimation device according to claim 4 , wherein the specification unit specifies a degree of overlap between the encapsulated regions.

6. 6. The estimation device according to claim 1, wherein the depth information is obtained from a depth image corresponding to the captured image.

7. 7. The estimation device according to claim 1, wherein the clustering unit further uses at least one of color information and luminance information corresponding to each of the pixels in the unsupervised learning.

8. The estimation device according to claim 7 , wherein the captured image is an RGB image.

9. 9. The estimation device according to claim 1, wherein the fruit is a fruit vegetable or a fruit tree.

10. The estimation device according to any one of claims 1 to 9, further comprising a separation unit that separates the target region into regions corresponding to individual fruits based on the boundary.

11. The estimation device according to claim 10, further comprising a prediction unit that predicts at least one of a harvest time and a yield of each of the individual fruits based on color information corresponding to each pixel in the area corresponding to each of the individual fruits.

12. an acquisition unit that acquires a thermal image corresponding to the captured image; A measurement unit that measures the temperature of each of the fruits based on the boundary and the thermal image, The estimation device according to claim 11 , wherein the prediction unit predicts at least one of a harvest time and a yield of each of the fruits further based on the temperature.

13. The captured image includes a plurality of leaves, The detection unit detects a second included region including each leaf in the captured image, The identification unit identifies a second target region including an area where the second encapsulated regions overlap each other, The clustering unit clusters each pixel in the second target region by unsupervised learning using depth information corresponding to the pixel; The estimation device according to claim 11 , wherein the estimation unit estimates boundaries between leaves in the second target region based on a result of the clustering.

14. A restoration unit that restores an entire image of the leaf is further provided, The separator separates the second region of interest into individual lobes based on boundaries between the lobes; the restoration unit restores an entire image of each leaf from an area corresponding to the individual leaf based on the captured image; The estimation device according to claim 13 , wherein the prediction unit predicts the yield of the fruit based on an area of ​​an entire image of a leaf restored by the restoration unit.

15. a second restoration unit that restores an entire image of each of the fruits from the areas corresponding to the individual fruits based on the captured image, The estimation device according to any one of claims 11 to 14, wherein the prediction unit further predicts a quality of the fruit based on an overall image of the fruit.

16. a detection step in which the estimation device detects an inclusion region that includes each of the fruits in a captured image of the plurality of fruits; a step of identifying a target region including an area where the encapsulated regions overlap each other by the estimation device; a clustering step in which the estimation device clusters each pixel in the target region by unsupervised learning using depth information corresponding to the pixel; an estimation step of estimating a boundary between fruits in the target region based on a result of the clustering, by the estimation device.

17. 2. An estimation program for causing a computer to function as the estimation device according to claim 1, the estimation program causing a computer to function as the detection unit, the identification unit, the clustering unit, and the estimation unit.

Images