Information processing apparatus, information processing method, and program

Abstract

To measure a size of a fruit with high accuracy.SOLUTION: An information processing apparatus according to the present invention has an acquiring unit for acquiring an actual two-dimensional image captured by an imaging unit for imaging a fruit of plant and distance information indicating a distance between the fruit and the imaging unit, an extracting unit for extracting a contour line of the fruit in the two-dimensional image, a calculation unit for fitting two ovals to the fruit so as to match the two with different portions of the fruit contour line to thereby calculate sizes and positions of two spheres based on the distance information and the two ovals, and an estimation unit for estimating a three-dimensional shape of the fruit based on the sizes and positions of the two spheres so that the three-dimensional shape should be along at least a part of a surface of each of the two spheres.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Patent Document 1 describes an estimation system that estimates fruit information including at least one item of the position, number, and presence or absence of fruits using an input image including plants. Patent Document 2 describes a method for grasping the fruit setting state of stocks.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, in agricultural fields, especially large-scale farms, higher accuracy in measuring the size of fruits is required. By performing three-dimensional photography, it is possible to measure the size of fruits with high accuracy, but the time required to measure the size of fruits becomes long. Therefore, it is required to measure the size of fruits with high accuracy based on a two-dimensional image of the fruits.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technology capable of measuring the size of fruits with high accuracy.

Means for Solving the Problems

[0006] An information processing apparatus according to an aspect of the present invention includes an acquisition unit that acquires a two-dimensional image of the fruit captured by an imaging unit that captures the fruit of a plant, and distance information indicating a distance between the fruit and the imaging unit, an extraction unit that extracts an outer contour line of the fruit in the two-dimensional image, a calculation unit that fits two ellipses along different portions of the outer contour line of the fruit, and calculates the sizes and positions of two spheres based on the distance information and the two ellipses, and an estimation unit that estimates a three-dimensional shape of the fruit along at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres. According to the above information processing apparatus, it is possible to accurately measure the size of the fruit.

[0007] In the above information processing apparatus, the extraction unit extracts a feature part from the two-dimensional image of the fruit, and the calculation unit fits the two ellipses so that one of the different parts of the outer contour line of the fruit is closer to the feature part than the other of the different parts of the outer contour line of the fruit. In the above information processing apparatus, the feature part is a part on the side opposite to the part connected to the main body of the plant in the fruit in the two-dimensional image, and is a part that is visually different from other parts of the plant in the two-dimensional image. In the above information processing apparatus, the feature part is the fruit apex, the fruit base, the calyx, or the flower scar. In the above information processing apparatus, the extraction unit extracts a feature part from the two-dimensional image of the fruit, and the calculation unit designates one of the two spheres as the first sphere and the other of the two spheres as the second sphere, and based on the distance information, calculates the size and position of the first sphere with the ellipse fitted so that one of the different parts of the outer contour line of the fruit is closer to the feature part than the other of the different parts of the outer contour line of the fruit as an image on the imaging surface, calculates the position of the second sphere with the other ellipse of the two ellipses as an image on the imaging surface and having the same size as the first sphere. In the above information processing apparatus, the calculation unit... Calculate a first cone circumscribing one of the elliptical cylinders, calculate the first sphere inscribed in the first cone, calculate a second cone circumscribing the other of the two elliptical cylinders, and calculate the second sphere inscribed in the second cone. The information processing apparatus includes a calculation unit that calculates the actual volume based on the distance between the centers of the first sphere and the second sphere plus the radii of the first sphere and the second sphere, and the diameter of the first sphere or the diameter of the second sphere. The information processing apparatus includes an adding unit that adds an outer contour line of the hidden part of the actual object to the two-dimensional image when a part of the actual object in the two-dimensional image is hidden.

[0008] Note that the present invention can also be regarded as an information processing method including at least a part of the above processing, a program for causing a computer to execute at least a part of the above processing, or a computer-readable recording medium in which such a program is non-temporarily recorded. Each of the above configurations and processes can be combined with each other to constitute the present invention as long as no technical contradiction occurs.

Advantages of the Invention

[0009] According to the present invention, it is possible to provide a technique capable of measuring the size of an actual object with high accuracy.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

MODE FOR CARRYING OUT THE INVENTION

[0011] Hereinafter, application examples and embodiments will be described with reference to the drawings. The application examples and embodiments shown below are one aspect of the present application and do not limit the scope of rights of the present application.

[0012] <Application Example> FIG. 1 is a diagram showing the overall configuration of a processing system to which the present invention is applied. The processing system according to the embodiment includes an autonomous mobile robot 1, an external device 2, and an external device 3. The autonomous mobile robot 1 is a device having a function as a self-propelled unmanned vehicle (self-propelled traveling device) or a device having a function as a self-propelled unmanned transport vehicle (self-propelled transport device). The external device 2 is composed of an information processing device such as a personal computer or a server computer, for example. The external device 2 may be arranged on the cloud. The external device 2 may be composed of a plurality of information processing devices. For example, the external device 2 may be realized by cooperation of a plurality of information processing devices located at different places on the network. The external device 3 is composed of an information processing terminal such as a tablet terminal or a smartphone, for example.

[0013] The autonomous mobile robot 1 has a camera (imaging device) 4 and a lighting device 5. The autonomous mobile robot 1 tours a farm or the like and images the fruits of the plants 6 in the farm or the like with the camera 4. The plant 6 is a kiwi, but the object imaged by the camera 4 may be other plants such as persimmons, peaches, tomatoes, pears, apples, melons, cucumbers, or Japanese cucumbers. The image (captured image) captured by the camera 4 is a two-dimensional image (two-dimensional wide-area image) captured from below or obliquely from under the fruit of the plant 6. The two-dimensional image captured by the camera 4 is sent from the autonomous mobile robot 1 to the external device 2. The lighting device 5 is installed in the vicinity of the camera 4. For example, in night-time imaging, imaging is performed while illuminating the plant 6 from below or obliquely. In daytime imaging, basically, imaging is performed without illuminating the plant 6, but since it is backlit, the plant 6 may be illuminated when the leaves of the plant 6 are thick.

[0014] Data and information are transmitted and received between the autonomous mobile robot 1 and the external device 2. Data and information are transmitted and received between the external device 2 and the external device 3. The external device 2 acquires a two-dimensional image from the autonomous mobile robot 1, measures the actual size within the two-dimensional image, predicts the yield, and sends the actual size, yield, etc. to the external device 3. The external device 3 has a display unit such as a display. The external device 3 displays the actual size, yield, etc. on the display unit. The autonomous mobile robot 1 and the external device 2 may be integrated. For example, the autonomous mobile robot 1 may have the functions of the external device 2. Also, the fruits of the plant 6 may be photographed by a hand-held camera or a fixed camera as an imaging unit, and the two-dimensional image photographed by the hand-held camera or the fixed camera may be sent to the external device 2.

[0015] <Embodiment> Figure 2 is a block diagram showing the configuration of the external device 2. The external device 2 has a control unit 10, a storage unit 20, and a communication unit 30. The control unit 10 controls each operation of the external device 2. The control unit 10 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a RO M (Read Only Memory), etc., and performs control of each part of the control unit 10 and various processes. The storage unit 20 stores programs executed by the control unit 10 and various data used in the processes executed in the control unit 10. For example, the storage unit 20 may be an auxiliary storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage unit 20 may be realized by a removable storage medium. The communication unit 30 is a communication interface that executes communication between the autonomous mobile robot 1 and the external device 3. The communication unit 30 includes at least one of a wired communication module and a wireless communication module.

[0016] The control unit 10 is configured to include an acquisition unit 110, a detection unit 120, a determination unit 130, an addition unit 140, an extraction unit 150, a calculation unit 160, an estimation unit 170, a computation unit 180, and an output unit 190. The acquisition unit 110 acquires a two-dimensional image of the fruit of a plant captured by the camera 4 as a photographing unit, and distance information indicating the distance between the fruit and the camera 4. The detection unit 120 detects the fruit within the two- dimensional image. The determination unit 130 determines whether the fruit in the two-dimensional image is hidden. The addition unit 140 adds an outer contour line of the hidden part of the fruit to the two-dimensional image. The extraction unit 150 extracts the outer contour line of the fruit in the two-dimensional image. The calculation unit 160 fits two ellipses along each of different parts on the outer contour line of the fruit, and calculates the sizes and positions of two spheres based on the distance information and the two ellipses. The estimation unit 170 estimates the three-dimensional shape of the fruit so as to be along at least a part of the surface of each of the two spheres based on the sizes and positions of the two spheres. The computation unit 180 calculates the size (diameter, length, volume, etc.) of the fruit based on the three-dimensional shape of the fruit. The output unit 190 outputs the value calculated by the computation unit 180, various kinds of information, and data. Not all of the components of the control unit 10 shown in FIG. 2 are essential, and components of the control unit 10 may be added or deleted as appropriate. Also, at least a part of the functions of the control unit 10 may be realized by a computer on the cloud.

[0017] With reference to FIG. 3, the overall flow of the operation of the external device 2 will be described. FIG. 3 is a flowchart for explaining the overall flow of the operation of the external device 2. In step S1, the control unit 10 acquires a two-dimensional image (a photographed image of the fruit) from the autonomous mobile robot 1. In step S2, the control unit 10 uses deep learning to detect the fruit from the two-dimensional image, and outputs a two-dimensional image with an outer circumscribed rectangle (outer circumscribed rectangle) surrounding the visible part of the fruit added thereto.

[0018] In step S3, the control unit 10 determines whether the entire part of the fruit can be seen for each fruit in the two-dimensional image to which the circumscribed quadrilateral is added. If the entire part of the fruit can be seen (step S3; YES), the process proceeds to step S4. If the entire part of the fruit cannot be seen and a part of the fruit is hidden (step S3; NO), the process proceeds to step S5.

[0019] In step S4, the control unit 10 outputs a cut-out image (two-dimensional image) of the fruit (entire part) in the two-dimensional image. The fruit (entire part) refers to the fruit whose entire part can be seen. In step S5, the control unit 10 outputs a cut-out image of the fruit (a part) in the two-dimensional image. The fruit (a part) refers to the fruit with a part hidden. In step S6, the control unit 10 estimates the outer contour line of the fruit in the cut-out image of the fruit (a part). In step S7, the control unit 10 adds the outer contour line of the fruit in the cut-out image of the fruit (a part) and outputs the cut-out image with the outer contour line of the fruit added. In step S8, the control unit 10 estimates the three-dimensional shape of the fruit from the cut-out image of the fruit (entire part) and the cut-out image with the outer contour line of the fruit added. In step S9, the control unit 10 calculates the volume of the fruit based on the three-dimensional shape of the fruit and outputs the volume of the fruit.

[0020] Details of the process in step S2 will be described. The control unit 10 detects the fruit from the two-dimensional image using a trained model of deep learning. The trained model of deep learning may be stored in the storage unit 20. To create a trained model of deep learning, for the two-dimensional image of the fruit captured by the camera 4 of the autonomous mobile robot 1, a user such as the operator creates a square frame so as to contact the outer shape of the fruit in the two-dimensional image. For the two-dimensional image captured by a hand-held camera or a fixed camera, the user may create a square frame so as to contact the outer shape of the fruit in the two-dimensional image.

[0021] FIG. 4 is a diagram showing an example of creating a square frame (circumscribed rectangle) so as to be in contact with the outer shape of the fruit in the two-dimensional image. Since the accuracy of the square frame and the method of deep learning affect the fruit detection accuracy, it is preferable to accurately create the square frame and perform a lot of deep learning. The two-dimensional image with the circumscribed rectangle created is stored in the database in the storage unit 20 and used as learning data. Using the learning data, a learned model of deep learning is created. The control unit 10 adds the circumscribed rectangle surrounding the visible part of the fruit detected from the two-dimensional image to the two-dimensional image and outputs the two-dimensional image with the circumscribed rectangle added. FIG. 5 is a diagram showing an example of a two-dimensional image with a circumscribed rectangle surrounding the fruit added. For example, when more than half of the fruit is visible, the fruit may be surrounded by the circumscribed rectangle. and outputs the two-dimensional image with the circumscribed rectangle added. FIG. 5 is a diagram showing an example of a two-dimensional image with a circumscribed rectangle surrounding the fruit added. For example, when more than half of the fruit is visible, the fruit may be surrounded by the circumscribed rectangle.

[0022] Details of the processing in steps S3 to S5 will be described. FIG. 6 is a flowchart showing an example (first example) of the processing in steps S3 to S5. In step S11, the control unit 10 searches for the overlap between the circumscribed rectangles in the two-dimensional image. The purpose of searching for the overlap between the circumscribed rectangles in the two-dimensional image is to determine the degree of overlap between the fruits in the two-dimensional image. If the fruits in the two-dimensional image do not overlap, the circumscribed rectangles in the two-dimensional image do not overlap either. On the other hand, if the fruits in the two-dimensional image overlap, the circumscribed rectangles in the two-dimensional image also overlap.

[0023] In step S12, the control unit 10 calculates the IoU (Intersection over Union). IoU is an index representing the degree of overlap between the circumscribed rectangles in the two-dimensional image and is calculated, for example, by the following formula 1. IoU = area of the union of two circumscribed rectangles / area of the intersection of two circumscribed rectangles (Formula 1)

[0024] In step S13, the control unit 10 compares the IoU with the threshold value ε and determines whether the IoU is greater than the threshold value ε. The threshold value ε may be obtained through experiments. As the threshold value ε, a value automatically calculated by simulation, machine learning, or the like may be used. As the threshold value ε, the user may measure the actual size in the two-dimensional image and use the median value, average value, or the like of the actual size. The threshold value ε is preferably set so that small fruits or fruits with small visible parts are not removed. If the IoU is greater than the threshold value ε (step S13; YES), the process proceeds to step S14. If the IoU is less than or equal to the threshold value ε (step S13; NO), the process proceeds to step S17.

[0025] In step S14, in order to investigate whether a part of the fruit in the two-dimensional image is hidden by other fruits, the control unit 10 measures (calculates) the area of the circumscribed rectangle and determines whether the area of the circumscribed rectangle is smaller than the threshold value T1. The threshold value T1 may be obtained through simulation, machine learning, or the like. If the area of the circumscribed rectangle is smaller than the threshold value T1 (step S14; YES), the process proceeds to step S15. If the area of the circumscribed rectangle is greater than or equal to the threshold value T1 (step S14; NO), the process proceeds to step S16.

[0026] In step S15, the control unit 10 determines that a part of the fruit in the two-dimensional image is hidden and outputs a cut-out image of the fruit (part). In step S16, the control unit 10 determines that the entire part of the fruit in the two-dimensional image is visible and outputs a cut-out image of the fruit (entire part). In step S17, in order to detect the overlap between the fruit and the shielding object in the two-dimensional image, the control unit 10 measures at least one of the RGB color values and luminance values of all the pixels in the two-dimensional image. The shielding object is leaves, trunks, stems, branches, etc. other than the fruit.

[0027] In step S18, the control unit 10 divides the circumscribed quadrilateral into four regions, and calculates the mode value of at least one of the RGB color values and the luminance values of all the pixels in each region. FIG. 7 is a diagram showing an example when the circumscribed quadrilateral is divided into four regions. In the example shown in FIG. 7, the circumscribed quadrilateral is divided into the first to fourth regions. FIG. 8 is a diagram showing an example when the circumscribed quadrilateral is divided into four regions. In the example shown in FIG. 8, at least one of the four regions has the fruit and the leaf overlapping. Therefore, the RGB color value (or luminance value) of all the pixels in one of the four regions is different from the RGB color values (or luminance values) of all the pixels in three of the four regions.

[0028] In step S19, the control unit 10 determines whether the mode value of at least one of the RGB color values and the luminance values of all the pixels in each region is smaller than the threshold value T2. The threshold value T2 may be obtained by simulation, machine learning, or the like. Since the shape of the fruit is symmetric, for a completely visible fruit, the mode values of the RGB color values and the luminance values of each region after dividing the circumscribed quadrilateral into four equal parts are the same. Therefore, the control unit 10 compares the mode value of at least one of the RGB color values and the luminance values of all the pixels in each region with the threshold value T2. When the mode value of at least one of the RGB color values and the luminance values of all the pixels in each region is smaller than the threshold value T2 (step S19; YES), the process proceeds to step S15. When the mode value of the RGB color and / or luminance of all the pixels in each region is equal to or greater than the threshold value T2 (step S19; NO), the process proceeds to step S16.

[0029] In the above, feature values such as the RGB color values and luminance values of pixels are used, but other feature values, for example, histograms of the RGB color values and luminance values of pixels may also be used. In step S18, the control unit 10 may calculate a histogram of at least one of the RGB color values and luminance values of all the pixels in each region. In step S19, the control unit 10 may determine the degree of coincidence or similarity between the shape of a histogram of at least one of the RGB color values and luminance values of all the pixels in each region and the shape of a predetermined histogram. The shape of the predetermined histogram may be obtained by simulation, machine learning, or the like. When the shape of a histogram of at least one of the RGB color values and luminance values of all the pixels in each region does not match or is not similar to the shape of the predetermined histogram, the process proceeds to step S15. When the shape of a histogram of at least one of the RGB color values and luminance values of all the pixels in each region matches or is similar to the shape of the predetermined histogram, the process proceeds to step S16.

[0030] Figure 9 is a flowchart showing an example (second example) of the processing of steps S3 to S5. Since steps S21 to S26 are the same as steps S11 to 16, detailed description thereof is omitted. In step S27, in order to detect the overlap between the fruits and the shielding objects in the two-dimensional image, the control unit 10 extracts the outer contour lines of each fruit in the two-dimensional image and outputs the outer contour lines of each fruit in the two-dimensional image. The shielding objects are leaves, trunks, stems, branches, etc. other than the fruits.

[0031] In step S28, the control unit 10 calculates the circularity of the outer contour line of the fruit. Circularity is a measure for quantitatively measuring circularity. The closer the circularity of the outer contour line of the fruit is to 1.0, the closer the outer contour line of the fruit is to a circle. Circularity is calculated, for example, by the following formula 2. π is the ratio of the circumference of a circle to its diameter (pi). Circularity = 4×(area of the fruit) / π×(major axis of the fruit)^2 (Formula 2)

[0032] In step S29, the control unit 10 judges whether the circularity of the outline of the real object is smaller than the threshold value T3. The circularity of the outline of the real object is large when the outline is completely visible, and the circularity of the outline of the obstructing object is small when the real object overlaps with the obstructing object. The threshold value T3 is set to a value that allows the obstructing object and the outline of the real object to be distinguished from each other. The threshold value T3 may be obtained by simulation, machine learning, or the like. If the circularity of the outline of the real object is smaller than the threshold value T3 (step S29; YES), the process proceeds to step S25. If the most frequent value of the RGB color and / or brightness of all pixels in each region is equal to or greater than the threshold value T2 (step S29; NO), the process proceeds to step S26.

[0033] 10 is a flowchart showing an example (third example) of the process of steps S3 to S5. In step S31, in order to detect overlapping of fruits in the two-dimensional image, the control unit 10 extracts the outline of each fruit in the two-dimensional image and outputs the outline of each fruit in the two-dimensional image. In step S32, the control unit 10 measures the area within the outline of the fruit.

[0034] In step S33, the control unit 10 judges whether the area inside the actual outline is smaller than a threshold value T4. The threshold value T4 may be obtained by an experiment. A value calculated automatically by computer simulation and machine learning may be used as the threshold T4. The user may measure the size of the fruit in the two-dimensional image, and the median and average values ​​of the fruit size may be used as the threshold T4. It is preferable that the threshold T4 is set so that small fruits and fruits with small visible parts are not removed. For example, the kiwi shelf has a small variation in the vertical direction, and the kiwi fruits are photographed from under the kiwi shelf. Therefore, fruits that are densely packed in the two-dimensional image are often on the same branch. From the viewpoint of biology, the size of neighboring fruits is roughly the same, so if the area within the outline of the fruit is small, it can be said that part of the fruit is hidden and the visible part of the fruit is small.

[0035] When the area within the outer contour of the fruit is smaller than the threshold value T4 (step S33; YES), the process proceeds to step S34. When the area within the outer contour of the fruit is greater than or equal to the threshold value T4 (step S33; NO), the process proceeds to step S35. In step S34, the control unit 10 determines that a part of the fruit in the two-dimensional image is hidden and outputs a cut-out image of the fruit (part). In step S35, the control unit 10 determines that the entire part of the fruit in the two-dimensional image is visible and outputs a cut-out image of the fruit (entire part).

[0036] The details of the processes in steps S6 and S7 will be described. FIG. 11 is a flowchart showing an example of the processes in steps S6 and S7. In step S41, the control unit 10 acquires a cut-out image of the fruit (part). In step S42, in order to obtain the outer contour of the fruit in the cut-out image, the control unit 10 matches the cut-out image with the two-dimensional image in the database by image analysis. Matching in image analysis refers to a process of searching for the location in the input image that is most similar to the template image. The control unit 10 may match the cut-out image with the two-dimensional image in the database by comparing the cut-out image with the captured images in the database one by one.

[0037] The database stores images in which the entire part of the fruit is photographed. The outer contour of the fruit is determined by the size of the fruit, the inclination of the fruit, and the relative position between the fruit and the camera 4. Therefore, the database may be prepared in advance with images captured by varying, for example, the following (Condition 1) to (Condition 3) within the assumed range. The user may prepare an image of the fruit and store the two-dimensional image of the fruit in the database. The following (Condition 1) to (Condition 3) are examples, and the numerical values of the following (Condition 1) to (Condition 3) may be changed, or other conditions may be used. (Condition 1) The size of the fruit (three levels: small, medium, large) (Condition 2) The inclination of the fruit (rotated 360 degrees at predetermined angles) (Condition 3) The relative position between the fruit and the camera 4 (two levels: the first predetermined distance range, the second predetermined distance range, the first predetermined distance range < the second predetermined distance range)

[0038] In step S43, the control unit 10 extracts the outline with the highest degree of coincidence between the actual outer outline in the cut-out image and the actual outer outline in the two-dimensional image in the database from the two-dimensional image in the database. In step S44, the control unit 10 estimates the outline of the hidden part of the actual object in the cut-out image of the actual object (a part) based on the extracted outline. In step S45, the control unit 10 adds the outline of the hidden part of the actual object to the cut-out image of the actual object (a part), and outputs the cut-out image with the outline of the actual object added. The control unit 10 may output the cut-out image with the outline of the actual object added as the cut-out image of the actual object (the whole part).

[0039] The details of the processes in steps S8 and S9 will be described. The processes in steps S8 and S9 are processes of modeling the actual object to estimate the three-dimensional shape of the actual object in order to measure the size of the actual object. FIG. 12 is a flowchart showing an example of the processes in steps S8 and S9. In step S51, the control unit 10 acquires the cut-out image (the whole part) and the cut-out image with the outline of the actual object added. The control unit 10 may acquire either the cut-out image (the whole part) or the cut-out image with the outline of the actual object added.

[0040] In step S52, the control unit 10 fits two ellipses to the actual outer contour line in the cut-out image. Specifically, the control unit 10 fits two ellipses along each of different portions of the actual outer contour line in the cut-out image. Fitting means obtaining, by approximating with a mathematical formula such as a polynomial, a curve that best fits experimentally obtained data. For example, the control unit 10 fits one ellipse so that a part of the actual outer contour line overlaps with a part of one ellipse, and fits the other ellipse so that another part of the actual outer contour line overlaps with a part of the other ellipse. FIG. 13 is an explanatory diagram of the case where two ellipses (first ellipse, second ellipse) are fitted to the actual outer contour line in the cut-out image. In the example shown in FIG. 13, it is assumed that the imaging plane is on the xy plane. As shown in FIG. 13, images of two ellipses (first ellipse, second ellipse) are arranged on the imaging plane. The control unit 10 extracts feature portions from the actual cut-out image. The actual object in the cut-out image has feature portions (feature points). The feature portion of the actual object is a portion on the side opposite to the portion connected to the main body of the plant in the actual object, and is a portion that is visually different from other portions in the plant. The feature portion of the actual object is, for example, the fruit apex, fruit bottom, calyx, flower scar, etc., but is not limited thereto. For example, kiwi, persimmon, peach, and tomato have a fruit apex, pear has a fruit bottom, apple has a calyx, and melon has a flower scar. The control unit 10 fits two ellipses to the actual outer contour line such that one of different portions of the actual outer contour line is closer to the feature portion of the actual object than the other of different portions of the actual outer contour line. In the example shown in FIG. 13, the two ellipses are fitted to the actual outer contour line such that one of the two ellipses (first ellipse) is closer to the feature portion of the actual object in the cut-out image than the other of the two ellipses (second ellipse). In step S53, the control unit 10 calculates and outputs the centers, major axes a, and minor axes b of the two ellipses. In step S54, the control unit 10 arranges images of two ellipses (first ellipse, second ellipse) on a plane (imaging plane) parallel to the xy plane of the camera coordinate system. FIG. 14 is a diagram showing the camera coordinate system.

[0041] In step S55, the control unit 10 creates two cones that circumscribe each of the two ellipses. Specifically, the control unit 10 calculates a first cone with the origin C(0, 0, 0) of the camera coordinate system as the apex and circumscribing the first ellipse, and calculates a second cone with the origin C(0, 0, 0) of the camera coordinate system as the apex and circumscribing the second ellipse. In step S56, the control unit 10 identifies a predetermined position point A(x, y, z) of the object in the camera coordinate system based on the relative position between the camera 4 and the object. The control unit 10 may identify the predetermined position point A(x, y, z) of the object in the camera coordinate system based on the elevation angle β, the azimuth angle θ, and the straight-line distance between the camera 4 and the object. The elevation angle β and the azimuth angle θ can be obtained from the captured image. The straight-line distance between the camera 4 and the object is distance information indicating the distance between the object and the camera 4. The camera 4 may have a depth sensor, and the straight-line distance between the camera 4 and the object may be measured from the depth information measured by the depth sensor. The camera 4 may have a distance measurement sensor, and the straight-line distance between the camera 4 and the object may be measured from the distance information measured by the distance measurement sensor. The straight-line distance between the camera 4 and the object may be the shortest distance between the camera 4 and the object. In step S1 or step S55, the control unit 10 may acquire the straight-line distance between the camera 4 and the object.

[0042] In step S57, the control unit 10 calculates and creates a first sphere that passes through the predetermined position point A and is inscribed in the first cone. For example, the control unit 10 calculates the size (e.g., diameter d) and position of the first sphere, and outputs the size and position of the first sphere. In step S58, the control unit 10 calculates and creates a second sphere that has the diameter d and is inscribed in the second cone. For example, the control unit 10 calculates the size (diameter d) and position of the second sphere, and outputs the size and position of the second sphere. In this way, the control unit 10 creates two spheres (the first sphere, the second sphere) that are inscribed in each of the two cones (the first cone, the second cone), and calculates the size and position of each of the two spheres. The first sphere and the second sphere have the same size and the same diameter d. In the above description, among the two cones, the cone corresponding to the sphere on the side where the characteristic part of the object is located is taken as the first cone, but it is not limited to this. Each of their sizes and positions is calculated. The first sphere and the second sphere have the same size and have the same diameter d. In the above, among the two cones, the cone corresponding to the sphere on the side where the characteristic part of the object is located is taken as the first cone, but it is not limited to this.

[0043] In step S59, the control unit 10 calculates the distance obtained by adding the respective radii of the two spheres to the distance between the centers of the two spheres. Specifically, the control unit 10 calculates a distance L obtained by adding the radius (d / 2) of the first sphere and the radius (d / 2) of the second sphere to the distance between the center of the first sphere and the center of the second sphere, and outputs the distance L. In step S60, the control unit 10 calculates the actual volume based on the actual diameter d (the diameter d of the first sphere or the second sphere) and the distance L, and outputs the actual volume. As shown in FIG. 15, the actual volume V is calculated based on the actual diameter d and the distance L. FIG. 15 is a diagram showing an example of the calculation of the actual volume V. Thus, the control unit 10 calculates the sizes and positions of the two spheres based on the distance information and the two spheres (the first sphere, the second sphere). For example, the control unit 10 calculates the size and position of the first sphere that uses, as an image on the imaging surface, an ellipse (the first ellipse) fitted so that one of the different portions on the actual outer contour is closer to the actual feature portion than the other of the different portions on the actual outer contour based on the distance information. The control unit 10 calculates the position of the second sphere that uses the other ellipse (the second ellipse) of the two ellipses as an image on the imaging surface. Then, the control unit 10 estimates the actual three-dimensional shape so as to follow at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres. Further, the control unit 10 calculates and outputs the actual diameter d, the length (distance L), and the volume V based on the actual three-dimensional shape. Thereby, it becomes possible to measure the actual size with high accuracy.

[0044] Details of the processes in steps S52 and S53 will be described. FIG. 16 is a flowchart showing an example of the processes in steps S52 and S53. FIGS. 17A, 17B, and 17C are explanatory diagrams for explaining the processes in steps S52 and S53. In step S61, in order to identify the first ellipse that fits the outer contour line of the fruit in the cut-out image, the control unit 10 detects the characteristic part of the fruit in the cut-out image and outputs the position of the characteristic part. In the example shown in FIG. 17A, the characteristic part of the fruit in the cut-out image is indicated by point P1. The characteristic part may be learned by deep learning. The control unit 10 may detect the characteristic part of the fruit in the cut-out image using the learned model that has learned the characteristic part. Also, the control unit 10 may detect the characteristic part by discriminating the color and luminance of the fruit.

[0045] In step S62, the control unit 10 creates an ellipse (first ellipse) that has the short side of the circumscribed rectangle as the minor axis and fits the outer contour line on the side closer to the characteristic part of the fruit, calculates the center, major axis a, and minor axis b of the first ellipse, and outputs them. In the example shown in FIG. 17B, the center of the first ellipse is indicated by point P2, the major axis a of the first ellipse is indicated by axis a1, and the minor axis b of the first ellipse is indicated by axis b1.

[0046] In step S63, the control unit 10 creates an ellipse (second ellipse) that has the center on the extension line of the characteristic part of the fruit and the center of the first ellipse and fits a part of the outer contour line on the side farther from the characteristic part of the fruit, calculates the center, major axis a, and minor axis b of the second ellipse, and outputs them. In the example shown in FIG. 17C, the center of the second ellipse is indicated by point P3, the major axis a of the second ellipse is indicated by axis a2, and the minor axis b of the second ellipse is indicated by axis b2.

[0047] When measuring the actual size based on a three-dimensional image, the measurement time is several times that when measuring the actual size based on a two-dimensional image. In the present embodiment, based on the actual two-dimensional image, the actual three-dimensional shape is estimated, and based on the actual three-dimensional shape, the actual size is measured, whereby it is possible to shorten the time required for measuring the actual size. Therefore, according to the present embodiment, it is possible to measure the actual size with high accuracy and to shorten the time required for measuring the actual size.

[0048] In the present embodiment, when a part of the actual object in the two-dimensional image is hidden, the outer contour line of the hidden part of the actual object is added to the two-dimensional image, and based on the two-dimensional image to which the outer contour line of the hidden part of the actual object is added, the actual three-dimensional shape is estimated, and based on the actual three-dimensional shape, the actual size is measured. Even when a part of the actual object in the two-dimensional image is hidden, it is possible to measure the actual size, so that the accuracy of measuring the total number of actual objects and the accuracy of measuring the size of each actual object are improved.

[0049] In the logistics field, objects such as cardboard boxes are measured in a contact manner, but in the agricultural field, a non-contact method is desirable because the actual object may be damaged if contacted. Therefore, in the logistics field, the weight of an object such as a cardboard box can be directly measured, but the weight of the actual object cannot be directly measured unless it is harvested. In the agricultural field, measurement of the weight of the actual object before harvesting is required. According to the present embodiment, it is possible to measure the size of the actual object non-contact, and it is possible to measure the weight of the actual object (yield prediction) before harvesting.

[0050] FIG. 18 is a diagram showing an example of the structure of data used for yield prediction. The calculation formula for calculating the weight shown in FIG. 18 is a formula for obtaining the actual weight from the actual diameter and is created by the user. For example, the calculation formula may be created using the user's knowledge. The calculation formula may be stored in the storage unit 20. When the user inputs respective values for the number of measurements (pieces), diameter (mm), and ratio (%), the total weight is calculated by the control unit 10. The calculation of the total weight may be performed daily or monthly. By accumulating the total weight, the weight (yield) of the fruits before harvest is predicted and displayed in the yield prediction column shown in FIG. 18. According to the present embodiment, since the size of the fruits can be measured with high accuracy, the accuracy of yield prediction can be improved. The data shown in FIG. 18 is sent from the external device 2 to the external device 3 and displayed on the display unit of the external device 3. The user can grasp the yield prediction by visually recognizing the content of the data displayed on the display unit of the external device 3.

[0051] 《Computer-readable recording medium》 A program for causing a computer or other machine or device (hereinafter referred to as a computer or the like) to realize any of the above functions can be recorded on a computer-readable recording medium. Then, by causing the computer or the like to read and execute the program of this recording medium, the function can be provided.

[0052] Here, a computer-readable recording medium refers to a recording medium that accumulates information such as data and programs by an electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer or the like. Examples of such removable recording media from a computer or the like include flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, Blu-ray disks, flash memories, and the like. Also, hard disks, ROMs, and the like are examples of recording media fixed to a computer or the like.

[0053] <Appendix 1> An acquisition unit (110) that acquires a two-dimensional image of the fruit captured by an imaging unit (4) that images the fruit of the plant, and distance information indicating the distance between the fruit and the imaging unit (4); An extraction unit (150) that extracts the outer contour line of the fruit in the two-dimensional image; A calculation unit (160) that fits two ellipses along different portions of the outer contour line of the fruit, and calculates the sizes and positions of two spheres based on the distance information and the two ellipses; An estimation unit (170) that estimates the three-dimensional shape of the fruit so as to follow at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres; An information processing apparatus (2) comprising the above. <Appendix 2> The extraction unit (150) extracts a feature part from the two-dimensional image of the fruit; The calculation unit (160) fits the two ellipses such that one of the different portions of the outer contour line of the fruit is closer to the feature part than the other of the different portions of the outer contour line of the fruit; The information processing apparatus (2) according to Appendix 1. <Appendix 3> The feature part is a part on the side opposite to the part connected to the main body of the plant in the fruit in the two-dimensional image, and is a part that is visually different from other parts of the plant in the two-dimensional image; The information processing apparatus (2) according to Appendix 2. <Appendix 4> The feature part is the fruit apex, fruit base, calyx or flower scar; The information processing apparatus (2) according to Appendix 2 or 3. <Appendix 5> The extraction unit (150) extracts a feature part from the two-dimensional image of the fruit; The calculation unit (160) Designates one of the two spheres as the first sphere and the other of the two spheres as the second sphere; Based on the distance information, calculate the size and position of the first sphere with an ellipse fitted such that one of the different parts on the outer contour of the fruit is closer to the characteristic part than the other of the different parts on the outer contour of the fruit as an image on the imaging surface. Calculate the position of the second sphere with the other ellipse of the two ellipses as an image on the imaging surface and having the same size as the first sphere. The information processing apparatus (2) according to any one of Appendices 1 to 4. <Appendix 6> The calculation unit (160) Calculate a first cone circumscribing one of the two ellipses. Calculate the first sphere inscribed in the first cone. Calculate a second cone circumscribing the other of the two ellipses. Calculate the second sphere inscribed in the second cone. The information processing apparatus according to Appendix 5. <Appendix 7> Comprising a calculation unit (180) for calculating the volume of the fruit based on the distance between the centers of the first sphere and the second sphere plus the radii of the first sphere and the second sphere, and the diameter of the first sphere or the diameter of the second sphere. The information processing apparatus (2) according to Appendix 6. <Appendix 8> When a part of the fruit in the two-dimensional image is hidden, comprising an adding unit (140) for adding the outer contour of the hidden part of the fruit to the two-dimensional image. The information processing apparatus (2) according to any one of Appendices 1 to 7. <Appendix 9> An information processing method executed by a computer, comprising: An acquisition step of acquiring the two-dimensional image of the fruit photographed by a photographing unit (4) and distance information indicating the distance between the fruit and the photographing unit (4); An extraction step of extracting the outer contour of the fruit in the two-dimensional image; A calculating step of fitting two ellipses along each of different portions in the outer contour of the fruit, and calculating the sizes and positions of two spheres based on the distance information and the two ellipses; An estimating step of estimating the three-dimensional shape of the fruit along at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres; An information processing method having the above. <Appendix 10> Causing a computer to An acquisition step of acquiring a two-dimensional image of the fruit photographed by a photographing unit (4) for photographing the fruit of a plant, and distance information indicating the distance between the fruit and the photographing unit (4); An extraction step of extracting the outer contour of the fruit in the two-dimensional image; A calculating step of fitting two ellipses along each of different portions in the outer contour of the fruit, and calculating the sizes and positions of two spheres based on the distance information and the two ellipses; An estimating step of estimating the three-dimensional shape of the fruit along at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres; A program for causing the above to be executed.

Explanation of Signs

[0054] 1: Autonomous driving robot 2, 3: External devices 4: Camera 5: Lighting device 6: Plant 10: Control unit 20: Storage unit 30: Communication unit 110: Acquisition unit 120: Detection unit 130: Judgment unit 140: Addition unit 150: Extraction unit 150: Calculation unit 170: Estimation unit 180: Calculation unit 190: Output unit

Claims

1. An acquisition unit that acquires a two-dimensional image of the fruit captured by an imaging unit that images the fruit of a plant, and distance information indicating the distance between the fruit and the imaging unit; An extraction unit that extracts an outer contour line of the fruit in the two-dimensional image; A calculation unit that fits two ellipses along respective different portions of the outer contour line of the fruit, and calculates the sizes and positions of two spheres based on the distance information and the two ellipses; An estimation unit that estimates the three-dimensional shape of the fruit so as to follow at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres; An information processing apparatus comprising the above.

2. The extraction unit extracts a feature part from the two-dimensional image of the fruit, The calculation unit fits the two ellipses such that one of the different portions of the outer contour line of the fruit is closer to the feature part than the other of the different portions of the outer contour line of the fruit. The information processing apparatus according to Claim 1.

3. The feature part is a part on the side opposite to the part connected to the main body of the plant in the fruit in the two-dimensional image, and is a part that is visually different from other parts of the plant in the two-dimensional image. The information processing apparatus according to Claim 2.

4. The feature part is the fruit apex, fruit base, calyx or flower scar. The information processing apparatus according to Claim 2.

5. The extraction unit extracts a feature part from the two-dimensional image of the fruit, The calculation unit designates one of the two spheres as the first sphere and the other of the two spheres as the second sphere, calculates the size and position of the first sphere with the ellipse fitted such that one of the different portions of the outer contour line of the fruit is closer to the feature part than the other of the different portions of the outer contour line of the fruit based on the distance information as an image on the imaging surface, calculates the position of the second sphere with the other ellipse of the two ellipses as an image on the imaging surface and having the same size as the first sphere. The information processing apparatus according to Claim 1.

6. The calculation unit calculates a first cone circumscribing one of the two ellipses, calculates the first sphere inscribed in the first cone, calculates a second cone circumscribing the other of the two ellipses, calculates the second sphere inscribed in the second cone. The information processing apparatus according to Claim 5.

7. A calculation unit that calculates the actual volume based on the distance between the centers of the first sphere and the second sphere plus the radii of the first sphere and the second sphere, and the diameter of the first sphere or the diameter of the second sphere. The information processing apparatus according to claim 6.

8. When a part of the actual object in the two-dimensional image is hidden, an adding unit that adds an outer contour line of the hidden part of the actual object to the two-dimensional image. The information processing apparatus according to any one of claims 1 to 7.

9. An information processing method executed by a computer, An acquisition step of acquiring a two-dimensional image of the actual object photographed by a photographing unit that photographs the actual object of the plant, and distance information indicating the distance between the actual object and the photographing unit; An extraction step of extracting an outer contour line of the actual object in the two-dimensional image; A calculation step of fitting two ellipses along respective different portions of the outer contour line of the actual object, and calculating the sizes and positions of two spheres based on the distance information and the two ellipses; An estimation step of estimating a three-dimensional shape of the actual object so as to follow at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres; An information processing method having the above steps.

10. Causing a computer to An acquisition step of acquiring a two-dimensional image of the actual object photographed by a photographing unit that photographs the actual object of the plant, and distance information indicating the distance between the actual object and the photographing unit; An extraction step of extracting an outer contour line of the actual object in the two-dimensional image; A calculation step of fitting two ellipses along respective different portions of the outer contour line of the actual object, and calculating the sizes and positions of two spheres based on the distance information and the two ellipses; An estimation step of estimating a three-dimensional shape of the actual object so as to follow at least a part of the surfaces of the two spheres based on the sizes and positions of the two spheres; A program for causing the above steps to be executed.

Images