Information processing device, information processing method, and program

Abstract

The present invention measures the size of fruit with high accuracy. An information processing device according to the present invention comprises: an acquisition unit that acquires a two-dimensional image of a fruit imaged by an imaging unit that images a fruit of a plant, and distance information indicating the distance between the fruit and the imaging unit; an extraction unit that extracts an outline of the fruit in the two-dimensional image; a calculation unit that fits two ellipses along each of mutually different portions of the outline of the fruit and calculates the size and position of two spheres on the basis of the distance information and the two ellipses; and an estimation unit that estimates, on the basis of the size and position of the two spheres, the three-dimensional shape of the fruit so as to follow at least a portion of the surface of each of the two spheres.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] Patent Literature 1 describes an estimation system that uses an input image containing a plant to estimate fruit information including at least one of the position, number, and presence or absence of fruit on the plant. Patent Literature 2 describes a method for determining the fruit-bearing state of a plant.

[0003] JP 2022-94331 A JP 2022-144240 A

[0004] For example, in agricultural fields, especially large-scale farms, there is a demand for highly accurate measurement of fruit size. While it is possible to measure fruit size with high accuracy by taking three-dimensional photographs, it takes a long time to measure the fruit size. Therefore, there is a demand for highly accurate measurement of fruit size based on two-dimensional images of the fruit.

[0005] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide a technique that makes it possible to measure the size of fruit with high accuracy.

[0006] According to one aspect of the present invention, an information processing device includes: an acquisition unit that acquires a two-dimensional image of a plant fruit captured by an image capturing unit that captures the fruit and distance information indicating the distance between the fruit and the image capturing unit; an extraction unit that extracts the outline of the fruit from the two-dimensional image; a calculation unit that fits two ellipses to different portions of the outline of the fruit and calculates sizes and positions of two spheres based on the distance information and the two ellipses; and an estimation unit that estimates a three-dimensional shape of the fruit based on the sizes and positions of the two spheres so as to fit the three-dimensional shape of the fruit to at least a portion of the surface of each of the two spheres. The information processing device enables the size of the fruit to be measured with high accuracy.

[0007] In the information processing device, the extraction unit extracts characteristic portions from the two-dimensional image of the fruit, and the calculation unit fits the two ellipses so that one of the different portions of the outline of the fruit is closer to the characteristic portion than the other of the different portions of the outline of the fruit. In the information processing device, the characteristic portion is a portion of the fruit in the two-dimensional image that is on the opposite side to a portion that is connected to the main body of the plant and that is visually different from other portions of the plant in the two-dimensional image. In the information processing device, the characteristic portion is a fruit apex, a fruit base, a calyx, or a flower scar. In the information processing device, the extraction unit extracts a characteristic portion from the two-dimensional image of the real object, and the calculation unit designates one of the two spheres as a first sphere and the other of the two spheres as a second sphere, and calculates, based on the distance information, a size and position of the first sphere having an image on an imaging surface that is an ellipse fitted so that one of the different portions of the outline of the real object is closer to the characteristic portion than the other of the different portions of the outline of the real object, and calculates a position of the second sphere having the same size as the first sphere as the image on the imaging surface of the other of the two ellipses. The information processing device includes a calculation unit that calculates the volume of the fruit based on the distance between the center of the first sphere and the center of the second sphere plus the radius of the first sphere and the radius of the second sphere, and the diameter of the first sphere or the diameter of the second sphere. The information processing device includes an addition unit that, when a portion of the fruit is hidden in the two-dimensional image, adds an outline of the hidden portion of the fruit to the two-dimensional image.

[0008] The present invention can also be understood as an information processing method including at least a part of the above-described processing, a program for causing a computer to execute at least a part of the above-described processing, or a computer-readable recording medium on which such a program is non-temporarily recorded. The above-described configurations and processing can be combined with each other to constitute the present invention as long as no technical contradiction occurs.

[0009] According to the present invention, it is possible to provide a technique that can measure the size of fruit with high accuracy.

[0010] FIG. 1 is a diagram illustrating the overall configuration of a processing system. FIG. 2 is a block diagram illustrating the configuration of an external device. FIG. 3 is a flowchart illustrating the overall operation of the external device. FIG. 4 is a diagram illustrating an example of creating a circumscribing rectangle within a two-dimensional image. FIG. 5 is a diagram illustrating an example of a two-dimensional image to which a circumscribing rectangle enclosing a fruit has been added. FIG. 6 is a flowchart illustrating an example (first example) of the processing of steps S3 to S5. FIG. 7 is a diagram illustrating an example (second example) of the processing of steps S3 to S5. FIG. 10 is a flowchart illustrating an example (third example) of the processing of steps S3 to S5. FIG. 11 is a flowchart illustrating an example (third example) of the processing of steps S6 and S7. FIG. 12 is a flowchart illustrating an example (third example) of the processing of steps S8 and S9. FIG. 13 is an explanatory diagram illustrating fitting two ellipses to the outline of a fruit in a cut-out image. FIG. 14 is a diagram illustrating a camera coordinate system. Fig. 15 is a diagram showing an example of calculation of fruit volume V. Fig. 16 is a flowchart showing an example of the processing of steps S52 and S53. Figs. 17A, 17B, and 17C are explanatory diagrams for explaining the processing of steps S52 and S53. Fig. 18 is a diagram showing an example of the structure of data used for yield prediction.

[0011] Application examples and embodiments will be described below with reference to the drawings. The application examples and embodiments described below are aspects of the present application and do not limit the scope of the rights of the present application.

[0012] <Application Example> Fig. 1 is a diagram showing the overall configuration of a processing system to which the present invention is applied. The processing system according to the embodiment includes an autonomous mobile robot 1, an external device 2, and an external device 3. The autonomous mobile robot 1 is a device (self-propelled mobile device) that functions as a self-propelled unmanned mobile vehicle, or a device (self-propelled transport device) that functions as a self-propelled unmanned guided vehicle. The external device 2 is, for example, configured as an information processing device such as a personal computer or a server computer. The external device 2 may be located on a cloud. The external device 2 may be configured as a plurality of information processing devices. For example, the external device 2 may be realized by cooperation between a plurality of information processing devices located in different locations on a network. The external device 3 is, for example, configured as an information processing terminal such as a tablet terminal or a smartphone.

[0013] The autonomous mobile robot 1 has a camera (photography device) 4 and a lighting device 5. The autonomous mobile robot 1 patrols a farm or the like and photographs the fruit of a plant 6 on the farm or the like using the camera 4. The plant 6 is a kiwi, but the object photographed by the camera 4 may be other plants, such as persimmons, peaches, tomatoes, pears, apples, melons, long gourds, and Saccharin. The image (photographed image) photographed by the camera 4 is a two-dimensional image (two-dimensional wide-area image) photographed from below or obliquely from the fruit of the plant 6. The two-dimensional image photographed by the camera 4 is sent from the autonomous mobile robot 1 to an external device 2. The lighting device 5 is installed near the camera 4. For example, when photographing at night, the plant 6 is always photographed with a light shining from below or obliquely. When photographing during the day, the plant 6 is generally photographed without a light. However, due to backlighting, the plant 6 may be illuminated when the plant's leaves are lush.

[0014] Data and information are transmitted and received between the autonomous mobile robot 1 and external device 2. Data and information are transmitted and received between the external device 2 and external device 3. The external device 2 acquires a two-dimensional image from the autonomous mobile robot 1, measures the size of the fruit in the two-dimensional image, predicts the yield, and sends the fruit size, yield, etc. to the external device 3. The external device 3 has a display unit such as a monitor. The external device 3 displays the fruit size, yield, etc. on the display unit. The autonomous mobile robot 1 and the external device 2 may be integrated. For example, the autonomous mobile robot 1 may have the functions of the external device 2. Furthermore, the fruit of the plant 6 may be photographed using a handheld or fixed camera as an imaging unit, and the two-dimensional image taken by the handheld or fixed camera may be sent to the external device 2.

[0015] <Embodiment> FIG. 2 is a block diagram showing the configuration of the external device 2. The external device 2 includes a control unit 10, a storage unit 20, and a communication unit 30. The control unit 10 controls each operation of the external device 2. The control unit 10 includes a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), etc., and controls each unit of the control unit 10 and performs various processes. The storage unit 20 stores programs executed by the control unit 10 and various data used in the processes executed by the control unit 10. For example, the storage unit 20 may be an auxiliary storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 20 may be realized by a removable storage medium. The communication unit 30 is a communication interface that executes communication between the autonomous mobile robot 1 and the external device 3. The communication unit 30 includes at least one of a wired communication module and a wireless communication module.

[0016] The control unit 10 is configured to include an acquisition unit 110, a detection unit 120, a determination unit 130, an addition unit 140, an extraction unit 150, a calculation unit 160, an estimation unit 170, a calculation unit 180, and an output unit 190. The acquisition unit 110 acquires a two-dimensional image of a plant fruit photographed by a camera 4 serving as an imaging unit, and distance information indicating the distance between the fruit and the camera 4. The detection unit 120 detects the fruit in the two-dimensional image. The determination unit 130 determines whether the fruit in the two-dimensional image is hidden. The addition unit 140 adds the outline of the hidden part of the fruit to the two-dimensional image. The extraction unit 150 extracts the outline of the fruit in the two-dimensional image. The calculation unit 160 fits two ellipses to each of different parts of the outline of the fruit, and calculates the size and position of two spheres based on the distance information and the two ellipses. The estimation unit 170 estimates the three-dimensional shape of the fruit based on the sizes and positions of the two spheres so that the shape fits at least a portion of the surface of each of the two spheres. The calculation unit 180 calculates the size of the fruit (diameter, length, volume, etc.) based on the three-dimensional shape of the fruit. The output unit 190 outputs the values ​​calculated by the calculation unit 180, as well as various information and data. Not all of the components of the control unit 10 shown in FIG. 2 are essential, and components of the control unit 10 may be added or deleted as appropriate. Furthermore, at least some of the functions of the control unit 10 may be implemented by a cloud-based computer.

[0017] The overall flow of operation of the external device 2 will be described with reference to Fig. 3. Fig. 3 is a flowchart for describing the overall flow of operation of the external device 2. In step S1, the control unit 10 acquires a two-dimensional image (a photographed image of the fruit) from the autonomous mobile robot 1. In step S2, the control unit 10 uses deep learning to detect the fruit from the two-dimensional image, and outputs a two-dimensional image to which a circumscribing square (circumscribing rectangle) surrounding the visible part of the fruit has been added.

[0018] In step S3, the control unit 10 determines whether the entire fruit is visible for each of the fruits in the two-dimensional image to which the circumscribing rectangle has been added. If the entire fruit is visible (step S3; YES), the process proceeds to step S4. If the entire fruit is not visible and only a portion of the fruit is hidden (step S3; NO), the process proceeds to step S5.

[0019] In step S4, the control unit 10 outputs a cropped image (2D image) of a fruit (whole portion) in the 2D image. A fruit (whole portion) refers to a fruit whose entirety is visible. In step S5, the control unit 10 outputs a cropped image of a fruit (part) in the 2D image. A fruit (part) refers to a fruit whose part is hidden. In step S6, the control unit 10 estimates the outline of the fruit in the cropped image of the fruit (part). In step S7, the control unit 10 adds the outline of the fruit in the cropped image of the fruit (part) and outputs the cropped image with the outline of the fruit added. In step S8, the control unit 10 estimates the three-dimensional shape of the fruit from the cropped image of the fruit (whole portion) and the cropped image with the outline of the fruit added. In step S9, the control unit 10 calculates the volume of the fruit based on the three-dimensional shape of the fruit and outputs the volume of the fruit.

[0020] The processing of step S2 will be described in detail. The control unit 10 detects the fruit from the two-dimensional image using a trained deep learning model. The trained deep learning model may be stored in the memory unit 20. To create the trained deep learning model, a user, such as a user, creates a square frame that is tangent to the outline of the fruit in a two-dimensional image of the fruit captured by the camera 4 of the autonomous mobile robot 1. A user may also create a square frame that is tangent to the outline of the fruit in a two-dimensional image captured by a handheld camera or a fixed camera.

[0021] FIG. 4 is a diagram showing an example of creating a rectangular frame (circumscribing rectangle) that is tangent to the outline of a fruit in a two-dimensional image. Because the accuracy of the rectangular frame and the method of deep learning affect the accuracy of fruit detection, it is preferable to create an accurate rectangular frame and perform extensive deep learning. The two-dimensional image with the circumscribing rectangle created is stored in a database in the storage unit 20 and used as training data. A trained deep learning model is created using the training data. For each fruit detected from the two-dimensional image, the control unit 10 adds a circumscribing rectangle that surrounds the visible portion of the fruit to the two-dimensional image, and outputs the two-dimensional image with the circumscribing rectangle added. FIG. 5 is a diagram showing an example of a two-dimensional image with a circumscribing rectangle that surrounds the fruit added. For example, if more than half of the fruit is visible, the fruit may be surrounded by a circumscribing rectangle.

[0022] The processing of steps S3 to S5 will be described in detail. FIG. 6 is a flowchart showing an example (first example) of the processing of steps S3 to S5. In step S11, the control unit 10 searches for overlaps between circumscribed rectangles in the two-dimensional image. The purpose of searching for overlaps between circumscribed rectangles in the two-dimensional image is to determine the degree of overlap between objects in the two-dimensional image. If objects in the two-dimensional image do not overlap, the circumscribed rectangles in the two-dimensional image also do not overlap. On the other hand, if objects in the two-dimensional image overlap, the circumscribed rectangles in the two-dimensional image also overlap.

[0023] In step S12, the control unit 10 calculates the intersection over union (IoU). IoU is an index representing the degree of overlap between circumscribing rectangles in a two-dimensional image, and is calculated, for example, by the following formula 1: IoU = area of ​​the union of two circumscribing rectangles / area of ​​the intersection of two circumscribing rectangles (Formula 1)

[0024] In step S13, the control unit 10 compares the IoU with a threshold ε and determines whether the IoU is greater than the threshold ε. The threshold ε may be determined experimentally. A value automatically calculated by simulation, machine learning, or the like may also be used as the threshold ε. The user may measure the fruit size in the two-dimensional image, and the median or average value of the fruit size may also be used as the threshold ε. The threshold ε is preferably set so as not to remove small fruits or fruits with small visible portions. If the IoU is greater than the threshold ε (step S13; YES), the process proceeds to step S14. If the IoU is equal to or less than the threshold ε (step S13; NO), the process proceeds to step S17.

[0025] In step S14, to investigate whether a part of a fruit in the two-dimensional image is hidden by another fruit, the control unit 10 measures (calculates) the area of ​​the circumscribing rectangle and determines whether the area of ​​the circumscribing rectangle is smaller than a threshold value T1. The threshold value T1 may be determined by simulation, machine learning, or the like. If the area of ​​the circumscribing rectangle is smaller than the threshold value T1 (step S14; YES), the process proceeds to step S15. If the area of ​​the circumscribing rectangle is equal to or greater than the threshold value T1 (step S14; NO), the process proceeds to step S16.

[0026] In step S15, the control unit 10 determines that a portion of the fruit is hidden in the two-dimensional image, and outputs a cropped image of the fruit (portion). In step S16, the control unit 10 determines that the entire fruit is visible in the two-dimensional image, and outputs a cropped image of the fruit (entire portion). In step S17, to detect overlap between the fruit and an obstructing object in the two-dimensional image, the control unit 10 measures at least one of the RGB color values ​​and brightness values ​​of all pixels in the two-dimensional image. The obstructing object may be a leaf, trunk, stem, branch, or the like other than the fruit.

[0027] In step S18, the control unit 10 divides the circumscribing rectangle into four regions and calculates the most frequent value of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region. FIG. 7 is a diagram showing an example of dividing the circumscribing rectangle into four regions. In the example shown in FIG. 7, the circumscribing rectangle is divided into first to fourth regions. FIG. 8 is a diagram showing an example of dividing the circumscribing rectangle into four regions. In the example shown in FIG. 8, fruit and leaves overlap in at least one of the four regions. Therefore, the RGB color values ​​(or luminance values) of all pixels in one of the four regions are different from the RGB color values ​​(or luminance values) of all pixels in three of the four regions.

[0028] In step S19, the control unit 10 determines whether the most frequent value of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region is smaller than a threshold value T2. The threshold value T2 may be determined by simulation, machine learning, or the like. Because the fruit is symmetrical, the most frequent values ​​of the RGB color values ​​and luminance values ​​of each region after dividing the circumscribing rectangle into four equal parts for a completely visible fruit are the same. Therefore, the control unit 10 compares the most frequent value of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region with the threshold value T2. If the most frequent value of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region is smaller than the threshold value T2 (step S19; YES), the process proceeds to step S15. If the most frequent value of the RGB color and / or luminance values ​​of all pixels in each region is equal to or greater than the threshold value T2 (step S19; NO), the process proceeds to step S16.

[0029] Although the above uses feature quantities such as RGB color values ​​and luminance values ​​of pixels, other feature quantities, such as histograms of RGB color values ​​and luminance values ​​of pixels, may also be used. In step S18, the control unit 10 may calculate a histogram of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region. In step S19, the control unit 10 may determine the degree of agreement or similarity between the shape of the histogram of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region and the shape of a predetermined histogram. The shape of the predetermined histogram may be determined by simulation, machine learning, or the like. If the shape of the histogram of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region does not match or resemble the shape of the predetermined histogram, the process proceeds to step S15. If the shape of the histogram of at least one of the RGB color values ​​and luminance values ​​of all pixels in each region matches or resembles the shape of the predetermined histogram, the process proceeds to step S16.

[0030] 9 is a flowchart showing an example (second example) of the processing of steps S3 to S5. Steps S21 to S26 are similar to steps S11 to S16, and therefore detailed description thereof will be omitted. In step S27, in order to detect overlap between fruit and an obstructing object in the two-dimensional image, the control unit 10 extracts the outline of each fruit in the two-dimensional image and outputs the outline of each fruit in the two-dimensional image. The obstructing object may be leaves, trunks, stems, branches, etc. other than the fruit.

[0031] In step S28, the control unit 10 calculates the circularity of the real outline. Circularity is a scale for quantitatively measuring circularity. The closer the circularity of the real outline is to 1.0, the closer the real outline is to a circle. The circularity is calculated, for example, by the following formula 2. π is the constant of the circumference of a circle (pi). Circularity = 4 × (area of ​​real) / π × (major axis of real) ^2 (Formula 2)

[0032] In step S29, the control unit 10 determines whether the circularity of the outline of the real object is smaller than the threshold value T3. The circularity of the outline of a real object whose outline is completely visible is large, while the circularity of the outline of an obstructing object is small. Therefore, when the real object overlaps with the obstructing object, the circularity of the outline of the real object is small. The threshold value T3 is set to, for example, a value that allows the obstructing object and the outline of the real object to be distinguished. The threshold value T3 may be determined by simulation, machine learning, or the like. If the circularity of the outline of the real object is smaller than the threshold value T3 (step S29; YES), the process proceeds to step S25. If the circularity of the outline of the real object is equal to or greater than the threshold value T3 (step S29; NO), the process proceeds to step S26.

[0033] 10 is a flowchart showing an example (third example) of the processing of steps S3 to S5. In step S31, in order to detect overlapping of fruits in the two-dimensional image, the control unit 10 extracts the outline of each fruit in the two-dimensional image and outputs the outline of each fruit in the two-dimensional image. In step S32, the control unit 10 measures the area within the outline of the fruit.

[0034] In step S33, the control unit 10 determines whether the area within the outline of the fruit is smaller than a threshold T4. The threshold T4 may be determined experimentally. A value automatically calculated by simulation, machine learning, or the like may also be used as the threshold T4. The user may measure the fruit size in the 2D image and use the median or average of the fruit size. The threshold T4 is preferably set so as not to remove small fruits or fruits with a small visible portion. For example, a kiwifruit shelf has little vertical variation, and the kiwifruit are photographed from below the shelf. Therefore, fruits that are closely packed in the 2D image often belong to the same branch. From a biological perspective, since neighboring fruits are roughly the same size, if the area within the outline of a fruit is small, it can be said that part of the fruit is hidden and the visible portion of the fruit is small.

[0035] If the area within the outline of the fruit is smaller than threshold T4 (step S33; YES), the process proceeds to step S34. If the area within the outline of the fruit is equal to or greater than threshold T4 (step S33; NO), the process proceeds to step S35. In step S34, the control unit 10 determines that a portion of the fruit is hidden in the two-dimensional image, and outputs a cropped image of the fruit (portion). In step S35, the control unit 10 determines that the entire fruit is visible in the two-dimensional image, and outputs a cropped image of the fruit (entire portion).

[0036] The processing of steps S6 and S7 will be described in detail. FIG. 11 is a flowchart showing an example of the processing of steps S6 and S7. In step S41, the control unit 10 acquires a cut-out image of a fruit (a portion). In step S42, the control unit 10 matches the cut-out image with a two-dimensional image in the database through image analysis in order to determine the outline of the fruit in the cut-out image. Matching in image analysis refers to a process of searching for a portion of the input image that is most similar to a template image. The control unit 10 may match the cut-out image with the two-dimensional image in the database by comparing the cut-out image with the captured image in the database in a brute-force manner.

[0037] The database stores images of the entire fruit. The outline of the fruit is determined by the size of the fruit, the inclination of the fruit, and the relative position between the fruit and the camera 4. Therefore, for example, images captured by varying the following (Condition 1) to (Condition 3) within an expected range may be prepared in advance in the database. A user may prepare images of the fruit and store two-dimensional images of the fruit in the database. The following (Condition 1) to (Condition 3) are examples, and the values ​​of the following (Condition 1) to (Condition 3) may be changed, or other conditions may be used. (Condition 1) Fruit size (three levels: small, medium, large) (Condition 2) Fruit inclination (360-degree rotation at predetermined angle intervals) (Condition 3) Relative position between the fruit and the camera 4 (two levels: first predetermined distance range, second predetermined distance range, first predetermined distance range < second predetermined distance range)

[0038] In step S43, the control unit 10 extracts from the two-dimensional image in the database the outline of the fruit in the cutout image that has the highest degree of match between the outline of the fruit in the two-dimensional image in the database. In step S44, the control unit 10 estimates the outline of the hidden part of the fruit in the cutout image of the fruit (portion) based on the extracted outline. In step S45, the control unit 10 adds the outline of the hidden part of the fruit to the cutout image of the fruit (portion) and outputs the cutout image with the outline of the fruit added. The control unit 10 may output the cutout image with the outline of the fruit added as the cutout image of the fruit (whole portion).

[0039] The processing of steps S8 and S9 will be described in detail. The processing of steps S8 and S9 is processing of modeling the fruit and estimating the three-dimensional shape of the fruit in order to measure the size of the fruit. FIG. 12 is a flowchart showing an example of the processing of steps S8 and S9. In step S51, the control unit 10 acquires a cut-out image (whole portion) and a cut-out image to which the outline of the fruit has been added. The control unit 10 may acquire either the cut-out image (whole portion) or the cut-out image to which the outline of the fruit has been added.

[0040] In step S52, the control unit 10 fits two ellipses to the outline of the fruit in the cropped image. Specifically, the control unit 10 fits two ellipses to fit different portions of the outline of the fruit in the cropped image. Fitting involves approximating a curve that best fits experimentally obtained data using a mathematical formula such as a polynomial. For example, the control unit 10 fits one ellipse so that a portion of the outline of the fruit overlaps with a portion of the first ellipse, and fits the other ellipse so that another portion of the outline of the fruit overlaps with a portion of the second ellipse. FIG. 13 is an explanatory diagram illustrating fitting two ellipses (a first ellipse and a second ellipse) to the outline of the fruit in the cropped image. In the example shown in FIG. 13, the imaging surface is assumed to be on the xy plane. As shown in FIG. 13, images of the two ellipses (the first ellipse and the second ellipse) are arranged on the imaging surface. The control unit 10 extracts characteristic features from the cropped image of the fruit. The fruit in the cropped image has characteristic parts (feature points). The characteristic parts of the fruit are located on the opposite side of the part of the fruit that is connected to the main body of the plant and are visually distinct from other parts of the plant. Characteristic parts of the fruit include, but are not limited to, the apex, base, calyx, and flower scar. For example, kiwi, persimmon, peach, and tomato have an apex, pears have a base, apples have a calyx, and melons have flower scars. The control unit 10 fits two ellipses to the outline of the fruit so that one of the different parts of the outline of the fruit is closer to the characteristic part of the fruit than the other of the different parts of the outline of the fruit. In the example shown in FIG. 13 , two ellipses are fitted to the outline of the fruit so that one of the two ellipses (the first ellipse) is closer to the characteristic part of the fruit in the cropped image than the other of the two ellipses (the second ellipse). In step S53, the control unit 10 calculates and outputs the centers, major axis a, and minor axis b of the two ellipses. In step S54, the control unit 10 places images of the two ellipses (first ellipse and second ellipse) on a plane (image capture plane) parallel to the xy plane of the camera coordinate system. Fig. 14 is a diagram showing the camera coordinate system.

[0041] In step S55, the control unit 10 creates two cones circumscribing the two ellipses, respectively. Specifically, the control unit 10 calculates a first cone having its vertex at the origin C (0,0,0) of the camera coordinate system and circumscribing the first ellipse, and calculates a second cone having its vertex at the origin C (0,0,0) of the camera coordinate system and circumscribing the second ellipse. In step S56, the control unit 10 identifies a real predetermined position point A (x,y,z) in the camera coordinate system based on the relative position between the camera 4 and the real object. The control unit 10 may also identify the real predetermined position point A (x,y,z) in the camera coordinate system based on the elevation angle β, the azimuth angle θ, and the linear distance between the camera 4 and the real object. The elevation angle β and the azimuth angle θ can be obtained from the captured image. The linear distance between the camera 4 and the real object is distance information indicating the distance between the real object and the camera 4. The camera 4 may have a depth sensor, and the linear distance between the camera 4 and the object may be measured from depth information measured by the depth sensor. The camera 4 may have a distance measurement sensor, and the linear distance between the camera 4 and the object may be measured from distance information measured by the distance measurement sensor. The linear distance between the camera 4 and the object may be the shortest distance between the camera 4 and the object. In step S1 or step S55, the control unit 10 may acquire the linear distance between the camera 4 and the object.

[0042] In step S57, the control unit 10 calculates and creates a first sphere that passes through the predetermined position point A and is inscribed in the first cone. For example, the control unit 10 calculates the size (e.g., diameter d) and position of the first sphere and outputs the size and position of the first sphere. In step S58, the control unit 10 calculates and creates a second sphere that has diameter d and is inscribed in the second cone. For example, the control unit 10 calculates the size (diameter d) and position of the second sphere and outputs the size and position of the second sphere. In this manner, the control unit 10 creates two spheres (first sphere, second sphere) that are inscribed in each of the two cones (first cone, second cone) and calculates the size and position of each of the two spheres. The first sphere and the second sphere have the same size and diameter d. In the above description, the cone corresponding to the sphere on which the real characteristic portion is located is defined as the first cone, but this is not limiting.

[0043] In step S59, the control unit 10 calculates the distance obtained by adding the center-to-center distance between the two spheres to the radii of the two spheres. Specifically, the control unit 10 calculates the distance L by adding the radius (d / 2) of the first sphere and the radius (d / 2) of the second sphere to the distance between the centers of the first sphere and the second sphere, and outputs the distance L. In step S60, the control unit 10 calculates the actual volume based on the actual diameter d (the diameter d of the first sphere or the second sphere) and the distance L, and outputs the actual volume. As shown in FIG. 15 , the actual volume V is calculated based on the actual diameter d and the distance L. FIG. 15 is a diagram illustrating an example of the calculation of the actual volume V. In this way, the control unit 10 calculates the size and position of the two spheres (the first sphere and the second sphere) based on the distance information and the two spheres (the first sphere and the second sphere). For example, based on the distance information, the control unit 10 calculates the size and position of a first sphere whose image on the imaging surface is an ellipse (first ellipse) fitted so that one of the different portions of the outline of the fruit is closer to the characteristic part of the fruit than the other of the different portions of the outline of the fruit. The control unit 10 calculates the position of a second sphere whose image on the imaging surface is the other of the two ellipses (second ellipse). Then, based on the sizes and positions of the two spheres, the control unit 10 estimates the three-dimensional shape of the fruit so that it follows at least a portion of the surface of each of the two spheres. Furthermore, based on the three-dimensional shape of the fruit, the control unit 10 calculates and outputs the diameter d, length (distance L), and volume V of the fruit. This makes it possible to measure the size of the fruit with high accuracy.

[0044] The details of the processing of steps S52 and S53 will be described. FIG. 16 is a flowchart illustrating an example of the processing of steps S52 and S53. FIGS. 17A, 17B, and 17C are explanatory diagrams for explaining the processing of steps S52 and S53. In step S61, in order to identify a first ellipse that fits to the outline of the fruit in the cut-out image, the control unit 10 detects characteristic parts of the fruit in the cut-out image and outputs the positions of the characteristic parts. In the example shown in FIG. 17A, the characteristic parts of the fruit in the cut-out image are indicated by point P1. The characteristic parts may be learned by deep learning. The control unit 10 may detect the characteristic parts of the fruit in the cut-out image using a trained model that has learned the characteristic parts. Alternatively, the control unit 10 may detect the characteristic parts by determining the color and brightness of the fruit.

[0045] In step S62, the control unit 10 creates an ellipse (first ellipse) whose minor axis is the shorter side of the circumscribed rectangle and which fits to the outline of the object closer to the characteristic portion, and calculates and outputs the center, major axis a, and minor axis b of the first ellipse. In the example shown in Fig. 17B, the center of the first ellipse is indicated by point P2, the major axis a of the first ellipse is indicated by axis a1, and the minor axis b of the first ellipse is indicated by axis b1.

[0046] In step S63, the control unit 10 creates an ellipse (second ellipse) that has a center on an extension of the real characteristic portion and the center of the first ellipse and fits to a portion of the outline farther from the real characteristic portion, and calculates and outputs the center, major axis a, and minor axis b of the second ellipse. In the example shown in Fig. 17C, the center of the second ellipse is indicated by point P3, the major axis a of the second ellipse is indicated by axis a2, and the minor axis b of the second ellipse is indicated by axis b2.

[0047] The measurement time required to measure fruit size based on a three-dimensional image is several times longer than the measurement time required to measure fruit size based on a two-dimensional image. In this embodiment, the three-dimensional shape of the fruit is estimated based on the two-dimensional image of the fruit, and the fruit size is measured based on the three-dimensional shape of the fruit, thereby shortening the time required to measure the fruit size. Therefore, according to this embodiment, it is possible to measure the fruit size with high accuracy and shorten the time required to measure the fruit size.

[0048] In this embodiment, when a part of a fruit is hidden in a two-dimensional image, the outline of the hidden part of the fruit is added to the two-dimensional image, the three-dimensional shape of the fruit is estimated based on the two-dimensional image to which the outline of the hidden part of the fruit has been added, and the size of the fruit is measured based on the three-dimensional shape of the fruit. Since it is possible to measure the size of the fruit even when a part of the fruit is hidden in the two-dimensional image, the accuracy of measuring the total number of fruits and the accuracy of measuring the size of each fruit is improved.

[0049] In the logistics field, objects such as cardboard boxes are measured using a contact method, but in the agricultural field, a non-contact method is preferable because contact with fruit may damage the fruit. Therefore, in the logistics field, the weight of objects such as cardboard boxes can be measured directly, but the weight of fruit cannot be measured directly until it is harvested. In the agricultural field, there is a demand for measuring the weight of fruit before harvest. According to this embodiment, it is possible to measure the size of fruit without contact, and it is possible to measure the weight of fruit (yield prediction) before harvest.

[0050] FIG. 18 is a diagram illustrating an example of the structure of data used for yield prediction. The weight calculation formula shown in FIG. 18 is a formula for calculating fruit weight from fruit diameter and is created by the user. For example, the formula may be created using the user's knowledge. The formula may be stored in the memory unit 20. The user inputs values ​​for the number of measurements (pieces), diameter (mm), and percentage (%), and the control unit 10 calculates the total weight. The total weight calculation may be performed daily or monthly. By accumulating the total weight, the fruit weight (yield) before harvest is predicted and displayed in the yield prediction column shown in FIG. 18. According to this embodiment, fruit size can be measured with high accuracy, thereby improving the accuracy of yield prediction. The data shown in FIG. 18 is sent from the external device 2 to the external device 3 and displayed on the display unit of the external device 3. The user can understand the yield prediction by visually checking the data displayed on the display unit of the external device 3.

[0051] <<Computer-readable recording medium>> A program that causes an information processing device or other machine or device (hereinafter, referred to as a computer, etc.) to realize any of the above functions can be recorded on a computer-readable recording medium. Then, the computer, etc. can provide the function by reading and executing the program from the recording medium.

[0052] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs electrically, magnetically, optically, mechanically, or chemically and that can be read by a computer, etc. Among such recording media, those that can be removed from a computer, etc. include, for example, flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, Blu-ray disks, flash memories, etc. Furthermore, recording media that are fixed to a computer, etc. include hard disks and ROMs, etc.

[0053] <Supplementary Note 1> An information processing device (2) comprising: an acquisition unit (110) that acquires a two-dimensional image of a plant fruit captured by an image capturing unit (4) that captures the fruit, and distance information indicating the distance between the fruit and the image capturing unit (4); an extraction unit (150) that extracts an outline of the fruit in the two-dimensional image; a calculation unit (160) that fits two ellipses to fit different parts of the outline of the fruit, and calculates sizes and positions of two spheres based on the distance information and the two ellipses; and an estimation unit (170) that estimates a three-dimensional shape of the fruit to fit at least a part of the surface of each of the two spheres, based on the sizes and positions of the two spheres. <Supplementary Note 2> The information processing device (2) according to Supplementary Note 1, wherein the extraction unit (150) extracts a characteristic portion from the two-dimensional image of the fruit, and the calculation unit (160) fits the two ellipses so that one of the different portions of the outline of the fruit is closer to the characteristic portion than the other of the different portions of the outline of the fruit. <Supplementary Note 3> The information processing device (2) according to Supplementary Note 2, wherein the characteristic portion is a portion of the fruit in the two-dimensional image that is on the opposite side to a portion that is connected to the main body of the plant, and is visually different from other portions of the plant in the two-dimensional image. <Supplementary Note 4> The information processing device (2) according to Supplementary Note 2 or 3, wherein the characteristic portion is a fruit apex, a fruit base, a calyx, or a flower scar. <Supplementary Note 5> The information processing device (2) according to any one of Supplementary Notes 1 to 4, wherein the extraction unit (150) extracts a characteristic portion from the two-dimensional image of the real, and the calculation unit (160): defines one of the two spheres as a first sphere and the other of the two spheres as a second sphere; calculates, based on the distance information, a size and position of the first sphere having an image on an imaging surface that is an ellipse fitted so that one of the different portions of the outline of the real is closer to the characteristic portion than the other of the different portions of the outline of the real; and calculates a position of the second sphere having the same size as the first sphere as an image on the imaging surface that is the other of the two ellipses.<Supplementary Note 6> The information processing device according to Supplementary Note 5, wherein the calculation unit (160) calculates a first cone circumscribing one of the two ellipses, calculates the first sphere inscribing the first cone, calculates a second cone circumscribing the other of the two ellipses, and calculates the second sphere inscribing the second cone. <Supplementary Note 7> The information processing device (2) according to Supplementary Note 6, further comprising: a calculation unit (180) that calculates the volume of the fruit based on a distance between the center of the first sphere and the center of the second sphere plus a radius of the first sphere and a radius of the second sphere, and a diameter of the first sphere or a diameter of the second sphere. <Supplementary Note 8> The information processing device (2) according to any one of Supplements 1 to 7, further comprising: an addition unit (140) that, when a portion of the fruit is hidden in the two-dimensional image, adds an outline of the hidden portion of the fruit to the two-dimensional image. <Supplementary Note 9> An information processing method executed by a computer, comprising: an acquisition step of acquiring a two-dimensional image of a plant fruit captured by an imaging unit (4) that captures the fruit, and distance information indicating the distance between the fruit and the imaging unit (4); an extraction step of extracting an outline of the fruit in the two-dimensional image; a calculation step of fitting two ellipses to fit different parts of the outline of the fruit, and calculating sizes and positions of two spheres based on the distance information and the two ellipses; and an estimation step of estimating a three-dimensional shape of the fruit to fit at least a part of the surface of each of the two spheres, based on the sizes and positions of the two spheres.<Supplementary Note 10> A program for causing a computer to execute the following steps: an acquisition step of acquiring a two-dimensional image of a plant fruit photographed by a photographing unit (4) that photographs the fruit, and distance information indicating the distance between the fruit and the photographing unit (4); an extraction step of extracting an outline of the fruit in the two-dimensional image; a calculation step of fitting two ellipses to follow different parts of the outline of the fruit, and calculating sizes and positions of two spheres based on the distance information and the two ellipses; and an estimation step of estimating a three-dimensional shape of the fruit based on the sizes and positions of the two spheres, so as to follow at least a part of the surface of each of the two spheres.

[0054] 1: Autonomous traveling robot 2, 3: External device 4: Camera 5: Lighting device 6: Plant 10: Control unit 20: Memory unit 30: Communication unit 110: Acquisition unit 120: Detection unit 130: Determination unit 140: Addition unit 150: Extraction unit 150: Calculation unit 170: Estimation unit 180: Calculation unit 190: Output unit

Claims

1. An information processing apparatus comprising: an acquisition unit that acquires a two-dimensional image of the fruit captured by an imaging unit that images the fruit of a plant and distance information indicating a distance between the fruit and the imaging unit; an extraction unit that extracts an outer contour line of the fruit in the two-dimensional image; a calculation unit that fits two ellipses along respective different portions of the outer contour line of the fruit, and calculates the sizes and positions of two spheres based on the distance information and the two ellipses; and an estimation unit that estimates a three-dimensional shape of the fruit so as to be along at least a part of the surface of each of the two spheres based on the sizes and positions of the two spheres.

2. The information processing apparatus according to claim 1, wherein the extraction unit extracts a feature part from the two-dimensional image of the fruit, and the calculation unit fits the two ellipses such that one of the different portions of the outer contour line of the fruit is closer to the feature part than the other of the different portions of the outer contour line of the fruit.

3. The information processing apparatus according to claim 2, wherein the feature part is a part on the side opposite to a part connected to the main body of the plant in the fruit in the two-dimensional image and is a part visually different from other parts of the plant in the two-dimensional image.

4. The information processing apparatus according to claim 2, wherein the feature part is a fruit apex, a fruit base, a calyx, or a flower scar.

5. The information processing apparatus according to claim 1, wherein the extraction unit extracts a feature part from the two-dimensional image of the fruit, and the calculation unit designates one of the two spheres as a first sphere and the other of the two spheres as a second sphere, calculates the size and position of the first sphere with an ellipse fitted such that one of the different portions of the outer contour line of the fruit is closer to the feature part than the other of the different portions of the outer contour line of the fruit based on the distance information as an image on the imaging surface, calculates the position of the second sphere with the other ellipse of the two ellipses as an image on the imaging surface and having the same size as the first sphere.

6. The information processing apparatus according to claim 5, wherein the calculation unit calculates a first cone circumscribing one of the two ellipses, calculates the first sphere inscribed in the first cone, calculates a second cone circumscribing the other of the two ellipses, and calculates the second sphere inscribed in the second cone.

7. The information processing apparatus according to claim 6, further comprising a calculation unit that calculates the actual volume based on the distance between the center of the first sphere and the center of the second sphere, the distance obtained by adding the radius of the first sphere and the radius of the second sphere, and the diameter of the first sphere or the diameter of the second sphere.

8. The information processing apparatus according to any one of claims 1 to 7, further comprising an adding unit that adds an outer contour line of the hidden portion of the actual object to the two-dimensional image when a part of the actual object in the two-dimensional image is hidden.

9. An information processing method executed by a computer, the method including: an acquisition step of acquiring a two-dimensional image of an actual object photographed by a photographing unit that photographs the actual object of a plant and distance information indicating a distance between the actual object and the photographing unit; an extraction step of extracting an outer contour line of the actual object in the two-dimensional image; a calculation step of fitting two ellipses along respective different portions of the outer contour line of the actual object and calculating sizes and positions of two spheres based on the distance information and the two ellipses; and an estimation step of estimating a three-dimensional shape of the actual object along at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres.

10. A program for causing a computer to execute: an acquisition step of acquiring a two-dimensional image of an actual object photographed by a photographing unit that photographs the actual object of a plant and distance information indicating a distance between the actual object and the photographing unit; an extraction step of extracting an outer contour line of the actual object in the two-dimensional image; a calculation step of fitting two ellipses along respective different portions of the outer contour line of the actual object and calculating sizes and positions of two spheres based on the distance information and the two ellipses; and an estimation step of estimating a three-dimensional shape of the actual object along at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres.

Images