Information processing device, information processing method, and program

Abstract

To provide an information processor, an information processing method, and a program in which a flower or a bunch can be accurately sensed in a certain kind of crop plant.SOLUTION: A information processor comprises: an acquisition part which acquires an image group of a plant captured at different time and date; a generation part which generates three-dimensional models of the plant for the different time and date; an extraction part which extracts a trunk and / or a main branch of the plant from the image group corresponding to each of the three-dimensional models of the plant for the different time and date; a detection part which detects a flower or a bunch of the plant from the image group corresponding to each of the three-dimensional models of the plant for the different time and date; and a reflection part which reflects position of the flower or bunch for the three-dimensional model of the plant for the different time and date to which common coordinates are set.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] For example, in large-scale grape cultivation, producers perform cultivation tasks such as shaping flower spikes and thinning the fruit before harvesting, but these tasks require a large amount of employed labor. Cultivation tasks for crops such as fruit trees must be carried out at the optimum time for each crop. The optimum time for the task is limited depending on the growth state of the crop, and delaying the optimum time for the task can result in a decline in the quality of the fruit. Therefore, to improve work efficiency, sensing systems that determine the optimum time for the task and robots that can perform various tasks on behalf of humans are needed. Efficient operation of robots requires the ability to accurately sense the position and shape of fruit, branches, etc. In addition to improving work efficiency through the use of robots, another goal is to increase profits by improving cultivation techniques, and for this reason, it is desirable to be able to accurately sense the growth status of crops.

[0003] Conventionally, there are known technologies such as fruit sensing technology for harvesting robots applied to greenhouse horticulture and technology for grasping the shape of forest plantations. Patent Document 1 describes a growth degree detection device that detects the growth degree of plants. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-033394 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 is based on the premise that the feature points within the detection object change (the feature points to be determined for the detection object are selected from the branching points of branches, the positions where leaves, flowers, and fruits are attached to branches, and the positions of the tips of branches and leaves), and there are cases where it is not possible to accurately sense flowers or clusters.

[0006] An object of the present invention is to provide an information processing device, an information processing method, and a program that are capable of accurately sensing flowers and clusters of certain types of crops. [Means for solving the problem]

[0007] The information processing device, the information processing method, and the program according to the present invention employ the following configuration. An information processing device according to one embodiment of the present invention is an information processing device comprising: an acquisition unit that acquires a group of images of a plant photographed at different dates and times; a generation unit that generates a three-dimensional model of the plant for each of the different dates and times based on the group of images; an extraction unit that extracts the main trunk and / or main branches of the plant from the group of images associated with each of the three-dimensional models of the plant for each of the different dates and times; a setting unit that sets common coordinates for the three-dimensional models of the plant for each of the different dates and times based on the main trunk and / or main branches; a detection unit that detects flowers or clusters of the plant from the group of images associated with each of the three-dimensional models of the plant for each of the different dates and times; and a reflection unit that reflects the position of the flowers or clusters in the three-dimensional models of the plant for each of the different dates and times for which the common coordinates have been set.

[0008] Another aspect of the present invention is an information processing method in which a computer acquires a group of images of a plant photographed at different dates and times, generates three-dimensional models of the plant for each of the different dates and times based on the group of images, extracts the main trunk and / or main branches of the plant from the group of images associated with each of the three-dimensional models of the plant for each of the different dates and times, sets common coordinates for the three-dimensional models of the plant for each of the different dates and times based on the main trunk and / or main branches, detects flowers or clusters of the plant from the group of images associated with each of the three-dimensional models of the plant for each of the different dates and times, and reflects the positions of the flowers or clusters in the three-dimensional models of the plant for each of the different dates and times for which the common coordinates have been set.

[0009] Another aspect of the present invention is a program that causes a computer to acquire a group of images of a plant taken at different dates and times, generate three-dimensional models of the plant for each of the different dates and times based on the group of images, extract the main trunk and / or main branches of the plant from the group of images associated with each of the three-dimensional models of the plant for each of the different dates and times, set common coordinates for the three-dimensional models of the plant for each of the different dates and times based on the main trunk and / or main branches, detect flowers or clusters of the plant from the group of images associated with each of the three-dimensional models of the plant for each of the different dates and times, and reflect the positions of the flowers or clusters in the three-dimensional models of the plant for each of the different dates and times for which the common coordinates have been set. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that are capable of accurately sensing flowers and clusters of certain types of crops. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 10 is a diagram showing the state of photographing and data collection in the field and the provision of information. [Figure 2] FIG. 10 is a diagram showing another example of the imaging device C. [Figure 3]FIG. 2 is a diagram showing the overall processing flow by the information processing device 100. [Figure 4] FIG. 10 is a diagram for explaining three-dimensional model generation. [Figure 5] FIG. 10 is a diagram illustrating the extraction of the main trunk and main branches. [Figure 6] FIG. 10 is a diagram showing an example of a main branch B1 extracted from an image. [Figure 7] FIG. 10 is a diagram showing main branches B2, B3, and B4 extracted from another image. [Figure 8] This figure shows images of the main trunk and main branch regions before and after normalization. [Figure 9] FIG. 10 is a diagram for explaining matching using BoVW (Bag of Visual Word). [Figure 10] FIG. 10 is a diagram illustrating an example of a matching result. [Figure 11] FIG. 10 is a diagram for explaining conversion to a common coordinate system. [Figure 12] FIG. 10 is a diagram showing an example of detecting fruit F1 from a photographed image using Faster R-CNN. [Figure 13] FIG. 10 is a diagram showing another example in which fruits F2 and F3 are detected from a photographed image by an SSD (Single Shot Multibox Detector). [Figure 14] 10A and 10B are diagrams for explaining a process of reflecting the position of a flower or a cluster in a three-dimensional model. [Figure 15] FIG. 10 is a diagram showing an example of a tuft detected by the detection unit 170 from a captured image using deep learning (Faster R-CNN). [Figure 16] FIG. 10 is a diagram showing an example of the result of reflecting the position of the tassel on a three-dimensional model at a certain date and time. [Figure 17] FIG. 10 is a diagram for explaining the tracking of flowers or clusters. [Figure 18] FIG. 10 is a diagram showing an example of a primitive representation of a tracking model of a flower or cluster. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an information processing device, an information processing method, and a program according to an embodiment of the present invention will be described with reference to the drawings.

[0013] [Overall configuration] FIG. 1 is a diagram showing the photographing and data collection in a farm field and the provision of information. A vehicle M is equipped with an imaging device C, such as a digital camera, and photographs plants, i.e., crops, being cultivated in a farm field F. The traveling of the vehicle M is controlled, for example, by a terminal device T1. Alternatively, the vehicle M may travel autonomously according to a program stored in an internal memory.

[0014] The imaging device C captures images, for example, at regular intervals or each time the vehicle M travels a certain distance. As a result, the imaging device C captures a group of images of the field F including plants. The vehicle M transmits the captured group of images to the terminal device T1. Furthermore, the vehicle M may be equipped with a positioning device (not shown) that determines the position of the vehicle M itself based on signals received from satellites that make up a Global Navigation Satellite System (GNSS), such as a Global Positioning System (GPS). The imaging device C associates the captured group of images of the field F with information indicating the position of the vehicle M (hereinafter referred to as "GPS information") and transmits the associated images to the terminal device T1. The GPS information includes, for example, information such as the latitude and longitude of the vehicle M's location. The vehicle M may also be a hand-pushed cart.

[0015] FIG. 2 is a diagram showing another example of the imaging device C. When capturing a group of images of plants in a field F at different dates and times, the configuration in FIG. 1 uses a vehicle M, but the means for capturing images is not limited to this. For example, a handheld imaging device C having a grip H as shown in FIG. 2 may be used. The imaging device C may also be mounted on an aerial vehicle (drone). Alternatively, the imaging device C may be mounted on a harvesting robot (not shown) that operates in the field F.

[0016] The terminal device T1 controls, for example, the movement of the vehicle M and the capture of a group of images by the imaging device C. The terminal device T1 is, for example, a computer device such as a personal computer, a smartphone, or a tablet terminal. An application for controlling the movement of the vehicle M and the capture of images by the imaging device C is executed on the terminal device T1. The application transmits information for controlling the movement of the vehicle M and the capture of images by the imaging device C to the vehicle M and the imaging device C in response to an operation by a user P1 using the information processing device 100. The application displays, on a display device provided in the terminal device T1, an image including operation buttons for controlling the movement of the vehicle M and the capture of images by the imaging device C, and an image showing the current capture range transmitted by the imaging device C.

[0017] When the user P1 operates the terminal device T1 to have the imaging device C photograph the field F, the user P1 faces the plants in the field F and photographs the plants multiple times while moving the vehicle M along the horizontal extension direction of the plants. The imaging device C may associate GPS information with each group of captured images and transmit them to the terminal device T1. The terminal device T1 transmits the group of images transmitted from the imaging device C to the information processing device 100. In this embodiment, such photographing is performed at different dates and times in order to sense the growth state of the plants in the field F.

[0018] The terminal device T1 transmits the image group to the information processing device 100 via, for example, a network NW. The network NW includes, for example, the Internet, a wide area network (WAN), a local area network (LAN), a provider device, a wireless base station, etc. For example, the terminal device T1 may temporarily store the image group transmitted by the imaging device C in a portable memory, and then transfer the image group to the information processing device 100 by attaching the portable memory to the information processing device 100.

[0019] The information processing device 100 includes, for example, a communication unit 110, an image acquisition unit 120, a storage unit 130, a generation unit 140, an extraction unit 150, a setting unit 160, a detection unit 170, and a reflection unit 180. The information processing device 100 may further communicate with a terminal device T2 separate from the terminal device T1 via a network NW. The terminal device T2 is, for example, a terminal device operated by a user P2 who uses the information processing device 100 in order to check the sensing results of the information processing device 100 connected via the network NW. Like the terminal device T1, the terminal device T2 is also, for example, a computer device such as a personal computer, a smartphone, or a tablet terminal.

[0020] The components of the information processing device 100, excluding the communication unit 110 and the storage unit 130, are implemented by a hardware processor, such as a CPU (Central Processing Unit), executing a program (software). Some or all of the functions of these components may be implemented by hardware (including circuitry), such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be implemented by a combination of software and hardware. Some or all of the functions of these components may be implemented by a dedicated LSI. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium), such as an HDD (Hard Disk Drive) or flash memory, included in the information processing device 100, or may be stored in a removable storage medium (a non-transitory storage medium), such as a DVD or CD-ROM, and installed in the storage device included in the information processing device 100 by inserting the storage medium into a drive device included in the information processing device 100. The information processing device 100 may be realized, for example, as a computer device such as a personal computer or a storage device. The information processing device 100 may also be realized as a server device or a storage device incorporated in a cloud computing system. In this case, the functions of the information processing device 100 may be realized by multiple server devices and storage devices in the cloud computing system.

[0021] The communication unit 110 is a communication interface such as a network card for connecting to the network NW. The communication unit 110 communicates with the terminal device T1 via the network NW. The image acquisition unit 120 acquires a group of images of plants in the field F taken at different dates and times by the imaging device C from the terminal device T1 that communicates with the communication unit 110, and stores the acquired group of images in the memory unit 130. The memory unit 130 stores data and information used when the components of the information processing device 100 perform processing. The memory unit 130 is, for example, a storage device such as an HDD or flash memory.

[0022] The generation unit 140 generates 3D models of plants for different dates and times based on a group of images captured at different dates and times. The extraction unit 150 extracts the main trunk and / or main branches of the plant from the group of images associated with each of the 3D models of the plant for different dates and times. The setting unit 160 sets common coordinates for the 3D models of the plant for different dates and times based on the extracted main trunk and / or main branches. The setting unit 160 may set the common coordinates by performing at least one of translation, rotation, and scaling on any of the 3D models of the plant for different dates and times. The detection unit 170 detects flowers or clusters of the plant from the group of images associated with each of the 3D models of the plant for different dates and times. The reflection unit 180 reflects the positions of the flowers or clusters detected by the detection unit 170 in the 3D models of the plant for different dates and times for which the common coordinates have been set by the setting unit 160.

[0023] FIG. 3 is a diagram showing the overall processing flow by the information processing device 100. The information processing device 100 executes processing from step S1 to step S7 on input sequential images I. In this processing, a first processing pass from generating a 3D model (step S1) to converting it to a common coordinate system (step S4) and a second processing pass from detecting flowers or clusters (step S5) to reflecting their positions in the 3D model (step S6) are executed in parallel. After the first and second processing passes are executed, step S7 is executed to track the flowers or clusters in the common coordinate system. Note that "in parallel" does not necessarily mean "simultaneously" in the temporal sense, but rather means that the processes can be executed independently without depending on each other. Each of the processes in steps S1 to S7 will be described in detail below.

[0024] (3D model generation: Step S1) FIG. 4 is a diagram for explaining the generation of a three-dimensional model. In step S1, the generation unit 140 generates a three-dimensional model from a group of images (input continuous images I) of plants taken while moving within a farm field F. For example, SfM (Structure from Motion) is used to generate the three-dimensional model. In SfM, the three-dimensional positions of feature points and the shooting positions are simultaneously estimated from multiple images, and the relative relationship between the shooting positions of each image can be obtained. In FIG. 4, IPos1 to IPos3 represent images (two-dimensional real-life images) taken at, for example, three shooting positions, point cloud Pos represents the shooting positions, and M represents the three-dimensional model.

[0025] (Extraction of main trunk and / or main branch: Step S2) FIG. 5 is a diagram for explaining extraction of the main trunk and main branches. In step S2, the extraction unit 150 extracts the main trunk and / or main branches of the plant from the image using, for example, deep learning. Specifically, a trained model (DNN) is used that has been trained to output the main branches and / or main branches when an image of a plant is input, using the image as learning data and area designation information (annotations) in the image as training data. The trained model has parameters trained by, for example, backpropagation. The extraction unit 150 extracts the main branches and / or main branches by inputting the image acquired by the image acquisition unit 120 into the trained model.

[0026] For this purpose, Instance Segmentation (Mask R-CNN) may be used, which identifies individual objects while extracting specific labels from an image. FIG. 6 shows an example of a main branch B1 extracted from an image. FIG. 7 shows main branches B2, B3, and B4 extracted from another image. The extraction unit 150 may extract the main trunk and main branches of a plant, or may extract only either the main trunk or the main branches. How to determine the extraction target is determined based on the accuracy and computational load of matching of three-dimensional models, which will be described next, but objects that are least affected by changes over time (temporal changes) should be selected.

[0027] (Data correspondence of the main trunk and / or main branch: Step S3) FIG. 8 is a diagram illustrating normalization of the main trunk and / or main branches, showing an image of the main trunk and / or main branch region before normalization (left) and an image of the main trunk and / or main branch region after normalization (right). In step S3, the extraction unit 150 normalizes the image region including the main trunk and / or main branch extracted in step S2. This normalization process is performed as preprocessing for the data matching described below, and includes edge enhancement to emphasize the main trunk and / or main branch in the image and brightness range adjustment. As can be seen from the normalized image (right), the leaf portion L is overexposed compared to before normalization, while the main trunk and / or main branch portion B is emphasized. This normalization process reduces time-dependent information, such as leaves, or reduces the influence of image changes due to wind (such as rustling leaves). The extraction unit 150 may normalize (emphasize) the main trunk and main branches of the plant, or may emphasize only either the main trunk or the main branches.

[0028] FIG. 9 is a diagram for explaining matching using BoVW (Bag of Visual Word). Following the normalization process, the extraction unit 150 performs a process of associating data of main trunks and / or main branches between images (three-dimensional models) captured at different dates and times, i.e., matching. This process may be, for example, matching using BoVW (Bag of Visual Word). Specifically, the loop closing technique used in SLAM (Simultaneous Localization and Mapping) may be applied to the matching process. This technique is a map creation technique used in autonomous driving, etc.

[0029] Furthermore, when SLAM is employed (as well as when Structure from Motion is employed), the extraction unit 150 simultaneously estimates landmark positions and self-location. While SLAM has the advantage of enabling real-time processing, accumulated errors in the estimation often prevent loop closure, resulting in distorted maps. By employing loop closing, the extraction unit 150 can retroactively correct the movement trajectory and map. To search for matching images between one image set and another, the extraction unit 150 learns a matching index (BoVW) for the entire image set. The matching index trained model is trained using the image set as training data and the matching index value (index) as training data. When an image at a certain date and time is input, the system outputs the image that matches it in the image set from another date and time. Here, the system takes advantage of the property that architectural structures and other objects in captured images are less susceptible to changes over time.

[0030] As shown in Fig. 9, it is assumed that an image set (image group) IG1 for "month / date" and an image set (image group) IG2 for "month / date" which is a different date and time are acquired. The matching index (BoVW) derivation unit 151 derives one matching index from the image set IG2 for "month / date". The matching processing unit 152 executes the above matching process using the matching index derived by the matching index (BoVW) derivation unit 151.

[0031] Fig. 10 is a diagram showing an example of a matching result. As shown in Fig. 10, for example, image IMG3 of "month / date" matches image IMG4 of "month / date." In this way, by emphasizing the main trunk and / or main branches that change little over time, it is possible to robustly match main trunks and / or main branches photographed at different dates and times.

[0032] (Transformation to a common coordinate system: Step S4) FIG. 11 is a diagram for explaining conversion to a common coordinate system. A coordinate system common to three-dimensional models (image groups) at different dates and times is called a "common coordinate system." According to the processing of step S3 above, as shown in FIG. 11, a main trunk and / or main branch Bp1-1 at a certain date and time corresponds (matches) with a main trunk and / or main branch Bp2-1 at another date and time. Similarly, a main trunk and / or main branch Bp1-2 corresponds with a main trunk and / or main branch Bp2-2, a main trunk and / or main branch Bp1-3 corresponds with a main trunk and / or main branch Bp2-3, and a main trunk and / or main branch Bp1-4 corresponds with a main trunk and / or main branch Bp2-4. Based on these correspondences, the setting unit 160 sets common coordinates for the three-dimensional models at least at one of the dates and times.

[0033] In the example of FIG. 11 , the setting unit 160 sets common coordinates for the three-dimensional model on the left. Specifically, the setting unit 160 sets common coordinates by performing at least one of translation, rotation, and scaling on the three-dimensional model. After such conversion to a common coordinate system, when three-dimensional models from different dates and times are superimposed, there will be image regions where the two models overlap and image regions where they do not, as shown in the lower part of FIG. 11 . The former includes the main trunk and / or main branches and can be used to derive parameters such as the translation and rotation described above. The latter includes the stems, leaves, flowers, and clusters of the plant and can be used to track the flowers or clusters, as described below.

[0034] (Detection of flowers or clusters: Step S5) In step S5, the detection unit 170 uses deep learning to detect flowers or clusters of plants from a group of images associated with each of the three-dimensional models for different dates and times. Specifically, a trained model (DNN) is used that is trained to output flowers or clusters when a photographed image of a plant is input, using the images as training data and area designation information (annotations) in the images as training data. The trained model has parameters trained using backpropagation or the like. The detection unit 170 detects flowers or clusters by inputting images acquired by the image acquisition unit 120 to the trained model. Examples of DNNs include Faster R-CNN and SSD. FIG. 12 shows an example in which fruit F1 is detected from a photographed image using Faster R-CNN. FIG. 13 shows another example in which fruits F2 and F3 are detected from a photographed image using SSD (Single Shot Multibox Detector). The method for detecting flowers or clusters from an image is not limited to these, and various methods can be applied.

[0035] (Reflection of the position of the flower or cluster: Step S6) 14 is a diagram illustrating the process of reflecting the position of a flower or a cluster onto a three-dimensional model. In step S6, the reflecting unit 180 reflects the position of the flower or cluster detected from each of the images by the detecting unit 170 in step S5 onto the three-dimensional model. This makes it possible to track the flower or cluster across time series of three-dimensional models in a common coordinate system.

[0036] (Flower or cluster tracking: step S7) FIG. 15 shows an example of a bunch detected by the detection unit 170 using Faster R-CNN from a captured image. FIG. 16 shows an example of the result of reflecting the position of the bunch on a 3D model at a certain date and time. In FIG. 16, the hatched portion reflects the position of the bunch detected from the captured image. In a display example of the 3D model, the reflected position of the bunch may be displayed, for example, using a monochrome graphic element. FIG. 17 is a diagram for explaining tracking of a flower or bunch. In step S7, the tracking unit (not shown in FIG. 3) 190 may track changes in the position of the flower or bunch based on a 3D model such as that shown in FIG. 16 in which the position of the flower or bunch is reflected. The tracking unit 190 tracks flowers or bunches that are close in three-dimensional position on multiple 3D models at different dates and times, for example, by applying one of the algorithm plans described below.

[0037] (Algorithm Plan 1) When algorithm plan 1 is adopted, the tracking unit 190 performs a simple search. Specifically, in consideration of the fact that the number of flowers or clusters in each time series of the three-dimensional model is limited, the tracking unit 190 determines, as correspondence candidates, flowers or clusters at a different date and time that are located near a flower or cluster at a certain date and time (see FIG. 17). The tracking unit 190 also identifies a tracking target from each correspondence candidate so that the entire time series is consistent. As shown in FIG. 17, global optimization is performed so that all corresponding points of flowers or clusters plotted on the three-dimensional model at different dates and times in a common coordinate system are consistent. This global optimization process includes bundle adjustment.

[0038] (Algorithm Plan 2) When algorithm plan 2 is adopted, the tracking unit 190 performs an advanced search. Specifically, the tracking unit 190 models the growth of stems and the like using an L-system. The tracking unit 190 also associates the positions of flowers or clusters using a predicted three-dimensional model after growth and a three-dimensional model after the passage of real time. Note that a Kalman filter may also be used in algorithm plan 2. Specifically, the tracking unit 190 uses an L-system in the prediction step of the Kalman filter.

[0039] (Primitive representation and data structures) Fig. 18 is a diagram showing an example of a primitive display of a tracking model of a flower or cluster. As shown in Fig. 18, the main trunk and / or main branches may be fitted with cylinders (Prm1, Prm2, Prm3), and the flowers or clusters may be displayed in a simplified manner by fitting a sphere Prm4 to their centers of gravity. This simplifies the display of the three-dimensional model 3DM, making it easier to view.

[0040] (Time series optimization) Another embodiment of the information processing device 100 optimizes object detection results based on a time series of images. The information processing device 100 configuration shown in FIG. 1 may further include an optimization unit that detects and corrects errors where a flower or cluster of a plant was not detected at a past date and time by detecting a flower or cluster of a plant at a future date and time, thereby optimizing the 3D model of the plant for each different date and time. The optimization unit can rescue flowers or clusters that were false negatives in the past by using future detection results. For example, suppose that a past date and time is determined to be negative with a 70% probability, i.e., not a flower or cluster, and a future date and time is determined to be positive with a 90% probability, i.e., a flower or cluster is determined with a 90% probability. The optimization unit compares the probability with a predetermined threshold and corrects the determination result for a certain date and time in the time series based on the result. Therefore, by correcting errors where flowers or clusters that should have existed in the past did not exist—in other words, errors where flowers or clusters were not detected at a past date and time—the entire sensing data time series can be optimized.

[0041] According to the embodiment described above, the information processing device 100 includes an image acquisition unit 120 that acquires a group of plant images I captured at different dates and times; a generation unit 140 that generates 3D models of the plant for each of the different dates and times; an extraction unit 150 that extracts the plant's main stem and / or main branches from the group of images I associated with each of the 3D models of the plant for each of the different dates and times; a detection unit 170 that detects plant flowers or clusters from the group of images I associated with each of the 3D models of the plant for each of the different dates and times; and an reflection unit 180 that reflects the positions of the flowers or clusters on the 3D models of the plant for each of the different dates and times, for which common coordinates have been set. This enables accurate sensing of flowers and clusters in certain crops. Specifically, it is possible to track changes over time, such as the elongation of plant leaves and stems, and the shifting positions of flowers and clusters. This embodiment is suitable for counting and tracking the positions of grapevine flowers and clusters, and can also be used to map the timing of gibberellin treatment, manage shoots, or track the results of pinching.

[0042] The above-described embodiment can be expressed as follows. a storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Obtain a group of plant images taken at different times and dates; generating a three-dimensional model of the plant for each of the different dates and times based on the group of images; extracting a main trunk and / or a main branch of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; setting common coordinates of the three-dimensional model of the plant for each of the different dates and times based on the main trunk and / or the main branch; detecting a flower or a cluster of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; reflecting the positions of the flowers or clusters in the three-dimensional models of the plant for the different dates and times for which the common coordinates have been set; The information processing device is configured as follows.

[0043] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0044] 100...Information processing device 110…Communications Department 120...Image acquisition unit 130...Storage section 140...Generation section 150...Extraction part 160...Settings section 170...Detection unit 180…Reflection part

Claims

1. an acquisition unit that acquires a group of images of plants taken at different dates and times; a generation unit that generates a three-dimensional model of the plant for each of the different dates and times based on the group of images; an extraction unit that extracts a main trunk and / or a main branch of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; a setting unit that sets common coordinates of the three-dimensional model of the plant for each of the different dates and times based on the main trunk and / or the main branch; a detection unit that detects flowers or clusters of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; a reflection unit that reflects the positions of the flowers or clusters in the three-dimensional models of the plant for the different dates and times for which the common coordinates are set; An information processing device comprising:

2. the setting unit sets the common coordinates by performing at least one of translation, rotation, and scaling on any of the three-dimensional models of the plant for each of the different dates and times. The information processing device according to claim 1 .

3. a tracking unit that tracks changes in the position of the flower or the cluster based on a three-dimensional model of the plant in which the position of the flower or the cluster is reflected; 3. The information processing device according to claim 1 or 2.

4. and an optimization unit that optimizes the 3D model of the plant for each different date and time by detecting and correcting an error in which a flower or a cluster of the plant was not detected at a past date and time based on the detection of a flower or a cluster of the plant at a future date and time. The information processing device according to claim 1 .

5. The computer Obtain a group of plant images taken at different times and dates; generating a three-dimensional model of the plant for each of the different dates and times based on the group of images; extracting a main trunk and / or a main branch of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; setting common coordinates of the three-dimensional model of the plant for each of the different dates and times based on the main trunk and / or the main branch; detecting a flower or a cluster of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; reflecting the positions of the flowers or clusters in the three-dimensional models of the plant for the different dates and times for which the common coordinates have been set; Information processing methods.

6. On the computer, Acquire a group of images of plants taken at different dates and times; generating a three-dimensional model of the plant for each of the different dates and times based on the group of images; extracting a main trunk and / or a main branch of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; setting common coordinates of the three-dimensional model of the plant for each of the different dates and times based on the main trunk and / or the main branch; detecting flowers or clusters of the plant from the group of images associated with each of the three-dimensional models of the plant for the different dates and times; reflecting the positions of the flowers or clusters in the three-dimensional models of the plant for the different dates and times for which the common coordinates have been set; program.

Images