Information processing apparatus, information processing method, and recording medium

Abstract

The present invention is to solve the problem of predicting the production amount of an object with higher accuracy. An information processing apparatus for predicting the production amount of an object includes: a region extraction unit that extracts an object region in which an object exists in a captured image of the object captured by a capturing device; a modification unit that modifies the size of the object region based on a distance from the capturing device to the object detected by a distance sensor; and a prediction unit that predicts the production amount of the object based on the size of the object region modified by the modification unit.

Description

TECHNICAL FIELD

[0001] The present application relates to an information processing apparatus, an information processing method, and a recording medium. BACKGROUND

[0002] It is important to know the harvest amount of a crop in advance before harvesting. By knowing the harvest amount in advance before harvesting, it is possible to efficiently prepare the required number of workers, working time, materials, equipment, and the like at the time of harvesting, and to make a shipment according to a plan.

[0003] As a technique for predicting the harvest amount of a crop, Patent Literature 1 discloses a technique of calculating a first predicted harvest amount of a crop based on a cultivation plan of the crop, calculating a modification value using a neural network, and calculating a predicted harvest amount during a prediction period based on the modification value.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent Application Laid-Open No. 2002-136223 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] However, in the technique of Patent Literature 1 described above, there is a problem that the prediction accuracy is not necessarily high.

[0009] The present application has been made in view of such a problem, and aims to provide a technique of predicting the production amount of an object with higher accuracy.

[0010] MEANS FOR SOLVING THE PROBLEM

[0011] To achieve the above object, the present application is an information processing apparatus of predicting a production amount of an object, including: a region extraction section that extracts an object region in which the object exists in a captured image of the object captured by a capturing apparatus; a modification section that modifies a size of the object region based on a distance from the capturing apparatus to the object detected by a distance sensor; and a prediction section that predicts the production amount of the object based on the size of the object region modified by the modification section.

[0012] Further, another aspect is an information processing method executed by an information processing apparatus that predicts a production amount of an object, including: an extraction step of extracting an object region in which the object exists in a captured image of the object captured by a capturing apparatus; a modification step of modifying a size of the object region based on a distance from the capturing apparatus to the object detected by a distance sensor; and a prediction step of predicting the production amount of the object based on the size of the object region modified by the modification step.

[0013] Further, another aspect is a program for causing a computer that predicts a production amount of an object to function as: an object region extraction section that extracts an object region in which the object exists in a captured image of the object captured by a capturing apparatus; a modification section that modifies a size of the object region based on a distance from the capturing apparatus to the object detected by a distance sensor; and a prediction section that predicts the production amount of the object based on the size of the object region modified by the modification section.

[0014] Effects of Invention

[0015] According to the present application, it is possible to predict a production amount of an object with higher precision. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a whole view of an information processing system.

[0017] Figure 2 is a schematic view that represents processing of a capturing unit.

[0018] Figure 3 is a hardware structure view of an information processing apparatus and a capturing unit.

[0019] Figure 4 is an explanatory view of a capturing position and a capturing direction of a capturing unit.

[0020] Figure 5 is a functional structure view of an information processing apparatus.

[0021] Figure 6 is a view that represents a data structure example of an image table.

[0022] Figure 7 is a flowchart that represents a prediction process.

[0023] Figure 8 is an explanatory view of a regression model.

[0024] Figure 9 is an explanatory view of an estimation model.

[0025] Figure 10 is an explanatory diagram of a presumption model.

[0026] Figure 11 is a flowchart representing a prediction process related to the second embodiment.

[0027] Figure 12 is an explanatory diagram of an object image related to the second embodiment.

[0028] Figure 13 is an explanatory diagram of a second modification of the second embodiment. DETAILED DESCRIPTION

[0029] (First Embodiment)

[0030] Figure 1 is a whole diagram of an information processing system 10 related to the first embodiment. The information processing system 10 is provided with an information processing apparatus 100, a photographing unit 200, and a cart 210. The information processing apparatus 100 and the photographing unit 200 are able to communicate through wire or wirelessly via a network. The information processing apparatus 100 predicts a harvest amount of grapes in an entire farm based on a photographed image of the grapes photographed in the farm. Here, the farm is one example of a prediction area that becomes an object of prediction of a production amount. In addition, the grapes are one example of an object that becomes a prediction object. In addition, the harvest amount is one example of a production amount.

[0031] The photographing unit 200 is mounted on the cart 210 and photographs images of the grapes from different positions in conjunction with movement of the cart 210. Further, the photographing unit 200 photographs images of the grapes and acquires a distance image in a photographed range corresponding to the photographed images and position information of the photographing unit 200 at the time of photographing the photographed images. Also, the photographing unit 200 transmits these pieces of information to the information processing apparatus 100.

[0032] Figure 2 is a schematic diagram representing a process of the photographing unit 200. In the present embodiment, a plurality of rows B are provided in a farm A. In the present embodiment, a length direction of the row B is set as an x axis, and a direction perpendicular to the length direction of the row B is set as a y axis. In addition, a plumb direction (a height direction of the farm A) is set as a z axis. In addition, a right direction of a paper surface of Figure 2 is set as a positive direction of the x axis, a top direction of the paper surface of Figure 2 is set as a positive direction of the y axis, and a front direction of the paper surface of Figure 2 is set as a positive direction of the z axis.

[0033] The imaging unit 200 images the ridge B while moving along the side of the ridge B, with a direction substantially parallel to the positive direction of the y-axis as the imaging direction. One captured image C includes only a part of the ridge B, and the imaging unit 200 captures a plurality of captured images to image the entire farm A. Further, the dolly 210 controls the moving direction and the moving speed in accordance with the user's operation.

[0034] Figure 3 Fig. 1 is a hardware configuration diagram of the information processing apparatus 100 and the imaging unit 200. The information processing apparatus 100 has a CPU 101, a ROM 102, a RAM 103, a HDD 104, an operation section 105, a display section 106, and a communication section 107. The CPU 101 reads a control program stored in the ROM 102 and executes various processes. The RAM 103 is used as a main memory, a work area, or the like of the CPU 101. The HDD 104 stores various data, various programs, and the like. The operation section 105 has a keyboard, a mouse, and the like, and accepts various operations by the user. The display section 106 displays various information. The communication section 107 performs communication with external apparatuses such as the imaging unit 200 via a network.

[0035] The imaging unit 200 has a communication section 201, an imaging section 202, a distance sensor 203, and a position sensor 204. The communication section 201 performs communication with external apparatuses such as the information processing apparatus 100 via a network. The imaging section 202 images an image. In detail, the imaging section 202 images a captured image (an RGB image) by visible light. The distance sensor 203 has a plurality of LEDs that project light, a plurality of light-receiving elements, and acquires a two-dimensional distance image that represents a distance between the distance sensor 203 and an object. As the distance sensor 203, for example, a Time-of-flight camera is given. The position sensor 204 is a tracking camera that has an acceleration sensor and a gyro sensor, and detects a relative position and a direction of the imaging unit 200 with a predetermined position and orientation as a reference. Further, the position sensor 204 can have other devices that enable V-SLAM (Visual-Simultaneous Localization and Mapping) such as a stereo camera.

[0036] The imaging unit 200 acquires the captured image, the distance image, and the position information at the same timing. Hereinafter, this timing is referred to as the imaging timing. Here, the position information includes a relative position and a direction. The imaging unit 200 acquires the captured image, the distance image, and the position information at regular intervals, and transmits them to the information processing apparatus 100 via the communication unit 201 at the timing of acquisition of the captured image, the distance image, and the position information. Further, as another example, the imaging unit 200 can transmit a plurality of captured images and the like collectively to the information processing apparatus 100 after the imaging of the entire farm A is completed.

[0037] Figure 4 is an explanatory view of the imaging position and the imaging direction of the imaging unit 200. As described above, the imaging unit 200 acquires the captured image, the distance image, and the position information while moving along the ridge B, i.e., along the x-axis. However, since there are obstacles such as unevenness of the ground, stones, and the like on the movement path, the imaging unit 200 actually moves along the direction of the x-axis while deviating in the positive or negative direction of the y-axis. Therefore, as shown in Figure 4 , the y-axis direction becomes the imaging direction in principle, but the imaging direction deviates from the y-axis direction depending on the moving direction and the orientation of the imaging unit 200. The position sensor 204 of the present embodiment acquires the imaging direction as an angle θ3 with the x-axis. As such, the imaging position and the imaging direction are included in the position information, and thus it is possible to correctly determine the imaging range of the captured image and the distance image, and the distance in the distance image. Further, the distance image at the imaging position P12 is shown as d2, which indicates the distance along the y-axis, and the distance image at the imaging position P13 is shown as d3, which indicates the distance in the direction of the angle θ3 with the x-axis.

[0038] In addition, the imaging unit 200 performs a plurality of imaging at the timing at which the imaging ranges overlap partially. Further, the control of such timing is achieved by controlling the moving speed of the cart 210 and the imaging speed (frame rate). As such, the captured image is obtained at a frequency at which the imaging ranges overlap partially, and thus it is possible to prevent the area that is not imaged from remaining in the farm A, i.e., in the prediction area.

[0039] Further, in the present embodiment, the adjustment is made so that the entire area of the ridge in the height direction, i.e., the z-axis direction, is included in the imaging range of one captured image.

[0040] Figure 5is a functional configuration diagram of the information processing apparatus 100. The information processing apparatus 100 has, as a functional configuration, an acquisition section 121, an image extraction section 122, a region extraction section 123, a modification section 124, a prediction section 125, an output processing section 126, and a presumption model learning section 127. Further, these functions are realized by the CPU 101 reading a program stored in the ROM 102 or the HDD 104 and executing the program. That is, the processing described as the processing performed by each section is processing performed by the CPU 101. In addition, as another example, the CPU 101 can read a program stored in a recording medium such as an SD card instead of the ROM 102 or the like and execute the program, thereby realizing the above-described functions and processing.

[0041] The acquisition section 121 acquires a captured image, a distance image, and position information from the photographing unit 200 and stores these pieces of information in association with each other in an image table 131 of a storage section 130. The storage section 130 is the HDD 104 or the like. Figure 6 is a diagram showing an example of a data structure of the image table 131 of the storage section 130. In the image table 131, a captured image, a distance image, and position information obtained at the same timing are stored in association as one record. Further, in the image table 131, a plurality of records corresponding to a plurality of distance images and the like captured by the photographing unit 200 are stored. Thus, the CPU 101 can determine the distance in the depth direction in the captured image and the photographing position.

[0042] The image extraction section 122 extracts a captured image that is a processing target from the storage section 130. Hereinafter, the captured image that is a processing target will be referred to as an object image. The region extraction section 123 extracts a region of a grape that is an object in each object image. Hereinafter, the region of the grape will be referred to as an object region. The modification section 124 modifies the size of the object region. The prediction section 125 predicts the harvest amount of the grape in the entire farm A, that is, the production amount of the object in the region, on the basis of the modified size of the object region. The output processing section 126 displays various kinds of information on the display section 106. Further, detailed processing of each of the above-described functional sections will be described with reference to the flowchart shown in Figure 6 The presumption model learning section 127 learns (generates) a presumption model referred to when the object region is extracted by the region extraction section 123. The processing of the presumption model learning section 127 will be described later.

[0043] Figure 7This is a flowchart illustrating the predictive processing of the information processing device 100. This processing is performed after the imaging unit 200 has finished photographing the entire farm A and multiple photographed images are stored in the storage unit 130. First, in step S100, the image extraction unit 122 extracts multiple object images from the storage unit 130 in the x-axis direction, ensuring that the shooting range is non-overlapping and that there are no un-photographed areas in the farm. That is, the image extraction unit 122 extracts object images where there are no gaps between the shooting ranges and no overlap within the shooting ranges. Thus, an image of the entire farm without overlap is extracted as the object image.

[0044] For example, in Figure 4 In the example shown, the image captured at shooting position P11 corresponding to shooting range R11 is extracted as a single object image. In this case, the image captured corresponding to shooting range R12, which is adjacent to shooting range R11, is extracted as the next object image. Furthermore, the image extraction unit 122 determines the shooting range based on the shooting position and shooting direction indicated by the position information associated with the captured image. Similarly, the image captured corresponding to shooting range R13, which is adjacent to shooting range R12, is extracted as the next object image. Shooting ranges R12 and R13 are determined by shooting position P12, shooting position P13, and shooting direction, respectively.

[0045] Furthermore, the image extraction unit 122 extracts target images as images within the shooting range along the y-axis direction, where there is no overlap or gap between the shooting ranges. In determining whether there is overlap or gap in the shooting range along the y-axis direction, the image extraction unit 122 refers not only to positional information but also to distance images.

[0046] Return to the instructions Figure 7 Following the processing in step S100, in step S102, the region extraction unit 123 extracts object regions from all object images extracted in step S100. The region extraction unit 123 utilizes an estimation model for estimating the object region during object region extraction. The estimation model is a deep learning model learned using actual captured images of grapes. In this embodiment, the estimation model is learned by the estimation model learning unit 127 and stored in the storage unit 130 before the prediction processing begins.

[0047] Next, in step S104, the modification section 124 modifies the size of each object region obtained in step S102. Even if the same object, the farther the position in the depth direction of the photographing, the smaller the object region. The modification section 124 modifies the difference in the size of the object due to the position in the depth direction (photographing direction). The modification section 124, for example, sets the position closest to the photographing unit 200 in the ridge B as a reference. Then, the modification section 124 modifies the size of each object region to the size in the case where the object region exists at the position closest to the photographing unit 200, according to the difference between the reference position and the position in the photographing direction of each object region. Further, the relationship information indicating the relationship between the distance from the reference position in the photographing direction and the coefficient for the modification is stored in advance in the storage section 130. The modification section 124 modifies the object region based on the relationship information and the distance of the object region shown by the distance image included in the position information associated with the photographing image of the extraction source of the object region.

[0048] Next, in step S106, the prediction section 125 predicts the harvest amount of the entire farm A, that is, the prediction range, based on the size of all the object regions modified in step S104. Specifically, first, the prediction section 125 calculates the total of the sizes of all the object regions modified in step S104. Then, the prediction section 125 converts the total of the sizes into the weight by referring to the regression model. Here, the obtained weight becomes the predicted value of the harvest amount. The regression model is a relationship formula that determines the relationship between the size and the weight.

[0049] Reference Figure 8 The regression model will be described. Figure 8 The horizontal axis of the graph shown in the drawing indicates the size of the object region at the reference position, and the vertical axis indicates the actual weight of the object. As such, the size of the object region is related to the weight (production amount). The regression model is a relationship formula obtained based on such measured values. Further, according to Figure 8 The relationship formula F is obtained. The regression model is stored in the storage section 130. The regression model of the present embodiment is one example of the first regression model.

[0050] Next, in step S108, the output processing section 126 outputs the harvest amount obtained in step S106. Specifically, the output processing section 126 displays the harvest amount on the display section 106. Thus, the prediction processing ends.

[0051] Next, the processing of the learning prediction model by the learning prediction model section 127 will be described with reference to Figure 9 and Figure 10 The processing of the learning prediction model by the learning prediction model section 127 will be described. In the photographing image of the object, labeling is performed by a person's hand when learning the prediction model. In this case, the person who performs the labeling is different from the person who performs the labeling in the photographing image of the object. In this case, the person who performs the labeling in the photographing image of the object is different from the person who performs the labeling in the photographing image of the object.Figure 9 The area indicated by the dashed line is designated as the object area; others will... Figure 9 The areas marked with solid lines are designated as object regions, and the way these regions are labeled varies from person to person. Such deviations may reduce the accuracy of the learning model. To address this, the estimation model learning unit 127 of this embodiment generates a learning model through the following processing.

[0052] That is, the estimation model learning unit 127 inputs (applies) labeled captured images to three learning models used for testing (learning). Then, if the inference result based on the learning model satisfies either condition 1 or condition 2 below, the estimation model learning unit 127 performs automatic corrections as shown by correction 1 and correction 2 respectively.

[0053] Condition 1: "The labeled region AND the region that neither learning model has inferred as an object."

[0054] Correction 1: "Corrected to areas that should not be marked".

[0055] Condition 2: "Unlabeled regions AND regions that are inferred as objects by any learning model"

[0056] Correction 2: "Corrected to areas that should be labeled"

[0057] By integrating multiple models instead of a single model, the probability of errors occurring during automatic correction can be reduced.

[0058] Furthermore, the estimation model learning unit 127 inputs the captured image of the labeled region, which has been corrected through the above processing, back into the three learning models used for testing, and repeats this process until the correction amount becomes below a constant value. Thus, labeled learning data can be obtained under equivalent benchmarks. The estimation model learning unit 127 uses this obtained learning data to learn the estimation model. Therefore, an estimation model with higher estimation accuracy can be obtained.

[0059] Furthermore, random erasing is employed in the learning of the estimation model in this embodiment. Therefore, the correct shape, taking into account occlusion (hidden surfaces), can be predicted using the estimation model. For example, as... Figure 10 As shown, even in cases where part of an object is hidden in the backlight of the support 30, the size is not estimated to be smaller than the actual size, as shown by the dashed line, and the correct shape is predicted taking into account the occlusion, as shown by the solid line.

[0060] As above, the information processing apparatus 100 of the present embodiment determines the object region based on the plurality of captured images that do not overlap and include the entire range of the prediction region, and thus can determine the object region without overlap. Further, the information processing apparatus 100 modifies the size of the object region based on the distance image, and thus can determine the size of the object region more accurately. Further, the information processing apparatus 100 converts the size of the object region into the weight using the regression model, and thus can predict the production amount. As such, the information processing apparatus 100 can predict the harvest amount (production amount) of the object with higher accuracy than before.

[0061] As a first modification of the first embodiment, the extraction of the captured image without overlap can be performed at the time of capturing by the capturing unit 200. In this case, the information processing apparatus 100 receives the captured image, the distance image, and the position information at the time of capturing by the capturing unit 200. Then, when the capturing image is acquired by the acquisition unit 121, the image extraction unit 122 determines whether the acquired captured image is a captured image that does not overlap and is consistent with the boundary with the previously selected captured image as the object image. In the case where the acquired captured image does not overlap and is consistent with the boundary with the object image, the image extraction unit 122 extracts the captured image as the object image. In the case of overlap with the object image, the captured image is deleted from the storage unit 130. As such, the object image is extracted in real time at the time of capturing, and the images other than the object image are deleted from the storage unit 130, and thus the memory of the storage unit 130 can be effectively utilized.

[0062] In addition, as a second modification, in the first modification, in the case where the captured image does not overlap and has a gap with the object image, it can be determined that the capturing position is too far forward, and a capturing instruction to capture the capturing range that returns to the predetermined distance is displayed on the display unit 106. In this case, the capturer performs capturing again on the basis of changing the capturing condition, such as moving the dolly 220 in the negative direction of the x-axis, moving the capturing direction toward the source, and the like by a remote controller or the like, and thus a captured image that does not overlap and is consistent with the boundary with the object image can be obtained.

[0063] In addition, as a third modification, the dolly 210 is a mobile system that can autonomously travel and can communicate with the information processing apparatus 100. In this case, the information processing apparatus 100 predicts the capturing position of the captured image without overlap, and instructs the predicted capturing position to the dolly 210, and thus the captured image without overlap is captured.

[0064] (Second Embodiment)

[0065] Next, the information processing system 10 according to the second embodiment will be described mainly on the basis of the difference from the information processing system 10 according to the first embodiment.Figure 11 is a flowchart representing the prediction processing of the information processing apparatus 100 according to the second embodiment. First, in step S200, the image extraction section 122 extracts a plurality of object images in which the photographed range in the x-axis direction repeats by a predetermined length amount. In the present embodiment, the image extraction section 122 extracts a plurality of object images in which the photographed range in the x-axis direction repeats by a half length amount in the photographed range R21.

[0066] Figure 12 is an explanatory diagram of the object images in the second embodiment. In the example shown in Figure 12 , the photographed image corresponding to the photographed range R21 is extracted as one object image. In this case, the photographed image corresponding to the photographed range R22 in which the photographed range repeats by a half length amount in the x-axis direction from the photographed range R21 is extracted as the next object image. Further, the photographed image corresponding to the photographed range R23 in which the photographed range repeats by a half length amount in the x-axis direction from the photographed range R22 is extracted as the next object image. Furthermore, the photographed ranges R22 and R23 are determined by the photographed position P22, the photographed range P23, and the photographed direction, respectively.

[0067] The explanation will return to Figure 11 . After the processing in step S200, in step S202, the region extraction section 123 extracts the object regions from all the object images extracted in step S200. Next, in step S204, the modification section 124 modifies the size of the object regions. The processing in step S202 and step S204 are the same as the processing in step S102 and step S104, respectively, explained with reference to Figure 7

[0068] Next, in step S206, the prediction section 125 confirms whether a plurality of object regions corresponding to the same object are included in the plurality of object regions extracted in step S202. Further, in the present processing, the prediction section 125 determines whether the objects are the same based on the position information (photographed position and photographed direction) associated with each of the object regions. In a case where a plurality of object regions corresponding to the same object are included (YES in step S206), the prediction section 125 causes the processing to proceed to step S208. In a case where a plurality of object regions corresponding to the same object are not included in the plurality of object regions (NO in step S206), the prediction section 125 causes the processing to proceed to step S210.

[0069] ​In step S208, the prediction unit 125 selects the largest object region among the plurality of object regions corresponding to the same object as the object region corresponding to the object. Further, for example, there are a plurality of object regions corresponding to the object Tl existing at a position Xl, and in addition, there are a plurality of object regions corresponding to the object T2 existing at another position X2. In this case, the prediction unit 125 selects the largest object region for each of the objects Tl, T2. In this way, the prediction unit 125 selects the largest object region for each object in a case where a plurality of the same objects exist. For example, in a captured image, in a case where a part of an object is hidden by a leaf, a branch, or the like, a range smaller than the size of the actual object is extracted as an object region. Therefore, in a case where the sizes of a plurality of object regions for the same object differ, the largest object region has the least hidden part and is considered to be close to the actual object.

[0070] Next, in step S210, the prediction unit 125 converts the sizes of the modified entire object regions into weights by using the regression model, thereby predicting the harvest amount of the entire farm. This process is the same as the process of step S106( Figure 7 ). Further, for the objects for which a plurality of object regions are extracted, the object region selected in step S208 is used. Next, in step S212, the output processing unit 126 displays the harvest amount on the display unit 106. This process is the same as the process of step S108( Figure 7 ). Further, the structure and the process of the information processing system 10 according to the second embodiment other than this are the same as the structure and the process of the information processing system 10 according to the first embodiment.

[0071] As described above, the information processing apparatus 100 according to the second embodiment predicts the production amount using a captured image in which an object region is larger in a case where the same object is included in different captured images in which the shooting ranges overlap. Thereby, the production amount can be predicted with higher accuracy.

[0072] A first modification example of the second embodiment will be described. In the present embodiment, a plurality of captured images in which the shooting ranges overlap by half have been extracted, but the degree of overlap is not limited to half. As another example, the information processing apparatus 100 can extract a plurality of captured images in which the shooting ranges overlap by more than half in the x-axis direction. Thereby, the likelihood that the same object is included in three or more captured images becomes higher. The information processing apparatus 100 selects the largest object region among the plurality of captured images, thereby being able to estimate the size of the object with higher accuracy.

[0073] A second modification example will be described. The imaging unit 200 can acquire a first captured image of the same region of the ridge B imaged from the outside of the first boundary surface of the ridge B in the first boundary surface direction and a second captured image of the same region of the ridge B imaged from the outside of the second boundary surface of the ridge B in the second boundary surface direction. For example, the imaging unit 200 images the first boundary surface S1 (set as the main surface) with the positive direction of the y-axis as the imaging direction with respect to the ridge B shown in FIG. 9. Thereafter, the imaging unit 200 images the second boundary surface S2 which becomes the back surface with respect to the first boundary surface S1 in the ridge B with the negative direction of the y-axis as the imaging direction. Thus, the imaging unit 200 can obtain the above-described first captured image and the second captured image. The information processing apparatus 100 extracts the first captured image and the second captured image whose imaging ranges overlap in the x-axis direction. Then, in a case where the first captured image and the second captured image include an object region corresponding to the same object, the information processing apparatus 100 selects the larger object region. Thus, it is possible to perform the yield prediction using the object region which is less likely to be occluded. Figure 13

[0074] A third modification example will be described. As described in the second modification example, the imaging unit 200 acquires the first captured image and the second captured image. Then, the information processing apparatus 100 extracts an object region as a processing target from a range of a first distance of the first boundary surface S1 of the ridge B included in the first captured image in the direction of the y-axis which is the depth direction until the half H of the ridge B. Similarly, the information processing apparatus 100 extracts an object region as a processing target from a range of a second distance of the second boundary surface S2 of the ridge B included in the second captured image in the depth direction until the half H of the ridge B. Thus, the information processing apparatus 100 can predict the yield using the object region which is less likely to be occluded. Further, the sum of the range of the first distance and the range of the second distance becomes the distance from the first boundary surface S1 to the second boundary surface S2.

[0075] The above-described embodiments are one example for implementing the present application, and various other embodiments can be employed. For example, a modification example of a certain embodiment can be applied to another embodiment, and the like, and various modifications and changes can be made within the scope of the gist of the present application described in the scope of claims.

[0076] ​As a first modification of the above-described embodiment, a part of the functions of the information processing apparatus 100 can be a system implemented by a plurality of apparatuses. In addition, a part of the structure of the above-described embodiment can be omitted, and the order of processing can be changed or omitted. In the present embodiment, for the photographing unit 200, the photographing section 202, the distance sensor 203, and the position sensor 204 are provided in one apparatus (the photographing unit 200), but at least one of these can be another apparatus. In this case, each apparatus is provided with a communication section, and each apparatus transmits the detection results obtained synchronously to the information processing apparatus 100 via a network.

[0077] In addition, as a second modification, the prediction region can be a region set in advance, and is not limited to the entire of a certain farm. For example, in a farm, in a case where a plurality of crops are cultivated, a certain kind of crop can be taken as the object, and a region in which the crop is cultivated can be taken as the prediction region.

[0078] As a third modification, the object is not limited to grapes. It can be other crops such as fruits and vegetables, trees in a forest, and the like.

[0079] As a fourth modification, in the present embodiment, the photographing unit 200 moves in a straight line along the ridge B formed in a straight line as the prediction region, and thereby performs photographing. However, the shape of the prediction region is not limited to the embodiment. In addition, the moving track of the photographing unit 200 is not limited to the embodiment. The photographing unit 200 can move along the boundary surface of the prediction region while photographing the photographing image of the prediction region.

[0080] As a fifth modification, the position sensor 204 of the photographing unit 200 detects the relative position and the orientation, but as to the position, instead of the relative position, an absolute position measured by a GNSS (global navigation satellite system) can be detected.

[0081] As a sixth modification, the output method of the harvest amount is not limited to the embodiment. As another example, it can be transmitted to an external apparatus via a network.

[0082] As a seventh modification, the information processing apparatus 100 can not be provided with the estimation model learning section 127. In this case, the estimation model is learned in another apparatus, and is transmitted to the information processing apparatus 100 via a network, and is stored in the storage section 130.

[0083] As an eighth modification, the information processing apparatus 100 can also utilize a reference Figure 11The algorithm of the image extraction in the step S200 is explained. Thus, the captured image of the object taken from various angles can be acquired as learning data.

[0084] As a ninth modification, in the regression model, the variable used to derive the weight is not limited to the size of the object region. As another example, the information processing apparatus 100 can determine a regression model for the weight on the basis of the size of the object region, using at least one of the four variables of the barycentric coordinates of the object region, the inclination of the object, the shooting position (the position of the shooting apparatus), and the shooting direction of the shooting apparatus. Here, the barycentric coordinates of the object region and the inclination of the object can be determined through the distance image, the position information, and the image recognition in the captured image.

[0085] In addition, as another example, the information processing apparatus 100 can calculate the volume of the object from the object region, and predict the production amount using a regression model representing the relationship between the volume and the weight. In this case, the information processing apparatus 100 calculates the three-dimensional voxel coordinates of the surface of the object (house), and maps the voxel coordinates on the 3D map. At this time, the information processing apparatus 100 connects the measured voxel coordinates smoothly by appropriate interpolation with respect to the missing house surface voxels that cannot be measured due to occlusion. Then, the information processing apparatus 100 determines the number of voxels in the inside of the 3D graph represented by the voxel coordinates of the house surface as the volume. The regression model of this example is one example of the second regression model.

[0086] As a tenth modification, in the present embodiment, the entire range in the height direction of the prediction region can be captured in one captured image. However, in a case where the entire range of the prediction region cannot be captured in the height direction (z-axis direction), the shooting unit 200 captures in the same manner as in the x-axis direction so as to cover the entire range in the height direction as well. Also, the information processing apparatus 100 of the first embodiment extracts captured images that do not overlap in the x-axis direction as well as in the height direction. In addition, the information processing apparatus 100 of the second embodiment extracts captured images that overlap in the x-axis direction as well as in the height direction. In this way, the information processing apparatus 100 can perform the above processing as a processing response to captured images taken not only one-dimensionally but also two-dimensionally.

[0087] As an eleventh modification, the information processing apparatus 100 can predict the production amount of the object included in the captured image in one captured image.

[0088] Further, the device, the program executed by the computer, the method described above exist as a case where it is realized as a single device, and a case where it is realized in a plurality of devices using shared components, including various modes. In addition, it is possible to appropriately change, for example, a part is software, a part is hardware, and the like. Further, the invention also stands as a recording medium of a program of a control device. Of course, the recording medium of the program can be a magnetic recording medium, or a semiconductor memory, and in all recording media developed in the future, it is possible to consider that it is completely the same.

[0089] Explanation of reference signs

[0090] 10: information processing system; 100: information processing device; 101: CPU; 102: ROM;

[0091] 103: RAM; 104: HDD; 105: operation part; 106: display part; 107: communication part;

[0092] 121: acquisition part; 122: image extraction part; 123: region extraction part; 124: modification part;

[0093] 125: prediction part; 126: output processing part; 127: estimation model learning part; 130: storage part;

[0094] 200: photographing unit; 201: communication part; 202: photographing part; 203: distance sensor;

[0095] 204: position sensor; 210: trolley.

Claims

1. An information processing apparatus that predicts a production amount of an object, characterized by comprising: a region extraction section that extracts an object region in which the object exists in a captured image of the object captured by a capturing apparatus; a modification section that modifies a size of the object region based on a distance from the capturing apparatus to the object detected by a distance sensor; and a prediction section that predicts the production amount of the object based on the size of the object region modified by the modification section, wherein the region extraction section takes, as a processing target, a plurality of the captured images in which at least a part of a captured range repeats, the plurality of the captured images being captured by the capturing apparatus moving in a predetermined direction, wherein in a case where images including the same object are included in at least two of the plurality of the captured images as the processing target, the prediction section selects, as the object region of the same object, a largest one of a plurality of the object regions corresponding to the same object, and predicts the production amount based on the size of the selected object region. 2.The information processing apparatus according to claim 1, wherein the region extraction section takes, as a processing target, a plurality of the captured images corresponding to different captured ranges within a prediction region that is an object of prediction of the production amount, the plurality of the captured images including all ranges within the prediction region, and extracts the object region in the captured images as the processing target, and the prediction section predicts the production amount of the object in the prediction region based on the size of each of the plurality of the object regions modified by the modification section. 3.The information processing apparatus according to claim 2, wherein the plurality of the captured images are images captured under a condition where at least one of a capturing position and a capturing direction is different, and the information processing apparatus further comprises an image extraction section that determines a captured range corresponding to each of the captured images based on the capturing position and the capturing direction of each of the captured images captured by the capturing apparatus, and extracts the captured images as the processing target based on the captured range. 4.The information processing apparatus according to claim 1 or 2, wherein the region extraction section extracts the object region from the captured image using an estimation model of the object, and the estimation model is generated using the learning data in which the region labeled as the object is corrected based on a result of estimation of each of the learning models and the labeled region, by applying the learning data including a plurality of captured images for learning in which the region labeled as the object is included, to a plurality of learning models, based on a result of estimation of each of the learning models, and the labeled region. 5.The information processing apparatus according to claim 3, wherein the image extraction section extracts a first captured image captured from an outside of a first boundary surface of the prediction region, and a second captured image captured from an outside of a second boundary surface inside the first boundary surface. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The region extraction section extracts the object region of the object existing within a first distance from the first boundary surface toward the second boundary surface, and further extracts the object region existing within a second distance from the second boundary surface toward the first boundary surface, The sum of the first distance and the second distance is the distance between the first boundary surface and the second boundary surface.

6. The information processing apparatus according to claim 3, wherein The image extraction section extracts a first captured image captured from an outside of a first boundary surface of the predicted region, and a second captured image captured from an outside of a second boundary surface of an inside of the first boundary surface, In a case where the same object is included in the first captured image and the second captured image, the prediction section selects a larger one of the two object regions corresponding to the same object as the object region of the same object, and predicts the production amount based on the size of the selected object region.

7. The information processing apparatus according to claim 1 or 2, wherein The prediction section predicts the production amount using a first regression model obtained from data of the size of the object and the production amount.

8. The information processing apparatus according to claim 7, wherein The first regression model represents a relationship between at least one of a center-of-gravity coordinate of the object, a tilt of the object, a position of the imaging device, an imaging direction of the imaging device, and the production amount, The prediction section further predicts the production amount based on the at least one and the first regression model.

9. The information processing apparatus according to claim 1 or 2, wherein The prediction section calculates a volume of the object region based on the size of the object region and a three-dimensional position of the object region obtained from a distance image, and predicts the production amount using a second regression model obtained from data of the volume of the object and the production amount.

10. The information processing apparatus according to claim 1, wherein A plurality of the captured images are repeated in a range of more than half of the imaging range in the predetermined direction.

11. An information processing method, which is an information processing method performed by an information processing apparatus that predicts a production amount of an object, characterized by comprising: a region extraction step of extracting an object region of the object existing in a captured image of the object captured by an imaging device; a modification step of modifying a size of the object region based on a distance from the imaging device to the object detected by a distance sensor; and a prediction step of predicting the production amount of the object based on the size of the object region modified by the modification step. ​ In the region extraction step, as processing targets, a plurality of the captured images in which at least a part of a captured range is repeated are used, the plurality of the captured images being captured by a capturing device moving in a predetermined direction, In the prediction step, in a case where images including the same object are included in at least two of the plurality of the captured images as the processing targets, the largest of a plurality of the object regions corresponding to the same object is selected as the object region of the same object, and the production amount is predicted based on the size of the selected object region.

12. A recording medium recording a program, characterized by comprising: The program is for causing a computer that predicts a production amount of an object to function as: a region extraction section that extracts an object region in which the object exists in a captured image of the object captured by a capturing device; a modification section that modifies the size of the object region based on a distance from the capturing device to the object detected by a distance sensor; and a prediction section that predicts the production amount of the object based on the size of the object region modified by the modification section, the region extraction section uses, as processing targets, a plurality of the captured images in which at least a part of a captured range is repeated, the plurality of the captured images being captured by a capturing device moving in a predetermined direction, in a case where images including the same object are included in at least two of the plurality of the captured images as the processing targets, the prediction section selects the largest of a plurality of the object regions corresponding to the same object as the object region of the same object, and predicts the production amount based on the size of the selected object region.

Images