Information processing device, information processing method, data structure and program

Abstract

To provide a technology that can correlate and store various data to track the process of crop growth.SOLUTION: An information processing device (1) includes: a storage unit (16) that stores plant identification information, which is information for identifying plants in a field, as information in layer 1, shape information indicating the shape of the plant at a certain point in time as information in layer 2, organ position information indicating the positions of the plant organs included in the shape information as information in layer 3, and organ identification information, which is information for identifying the plant organs, as information in layer 4: and an identification unit (12) that identifies the organ position information of a specific organ at a certain point in time by referring to information on a specific organ of a specific plant that is correlated across multiple layers.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a data structure, and a program. [Background technology]

[0002] In order to address the labor shortage issue and improve efficiency in the agricultural sector, mechanization of cultivation management of crops (plants) grown in fields is progressing. For example, in the cultivation management of fruit trees, meticulous work is required for each type of fruit tree and the growth stage of the fruit spike. Furthermore, because each individual fruit tree is at a different growth stage, cultivation management for each individual tree is required rather than a uniform cultivation management. Therefore, as the number of fruit trees increases in a large-scale field, it becomes difficult to perform sufficient cultivation management by having workers patrol the trees.

[0003] For example, Patent Document 1 discloses an information processing device that stores position information of crops grown in a field, position information of the fruit, and information on work performed on the fruit by layer, and identifies the correspondence between layers from a work image based on the position information and 3D model information. This information processing device discloses a method for identifying the correspondence between the position of the fruit of the crop and the content of work performed on the fruit using an image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-095197 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 identifies the correspondence between the position of a crop and the work performed on the crop. However, it is not intended to track the growth process of the crop. Furthermore, the prior art does not necessarily enable many types of data to be correlated and utilized.

[0006] An object of one aspect of the present invention is to provide a technology that can correlate and store various data and track the process of crop growth. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a memory unit that stores plant identification information, which is information that identifies a plant in a field, as information in layer 1, stores shape information that indicates the shape of the plant at a certain point in time as information in layer 2, stores organ position information that indicates the position of the organs of the plant included in the shape information as information in layer 3, and stores organ identification information, which is information that identifies the organs of the plant, as information in layer 4, and an identification unit that identifies the organ position information of the specified organ at a certain point in time by referring to information regarding a specified organ of a specified plant that is related to each other across multiple layers.

[0008] An information processing method according to one aspect of the present invention includes a storage process of storing plant identification information, which is information for identifying a plant in a field, as information in layer 1, storing shape information indicating the shape of the plant at a certain point in time as information in layer 2, storing organ position information indicating the position of an organ of the plant included in the shape information as information in layer 3, and storing organ identification information, which is information for identifying an organ of the plant, as information in layer 4, and an identification process of identifying the organ position information of the predetermined organ at a certain point in time by referring to information on a predetermined organ of a predetermined plant that is related to each other across multiple layers. Includes.

[0009] A data structure according to one embodiment of the present invention comprises a layer 1 that stores plant identification information, which is information that identifies plants in a field; a layer 2 that stores shape information that indicates the shape of the plant at a certain point in time; a layer 3 that stores organ position information that indicates the position of the organs of the plant included in the shape information; and a layer 4 that stores organ identification information, which is information that identifies the organs of the plant, wherein the plant identification information and the shape information are related to each other, the shape information and the organ position information are related to each other, and the organ position information and the organ identification information are related to each other.

[0010] The information processing device according to each aspect of the present invention may be realized by a computer. In this case, the control program that causes the computer to operate as each part (software element) of the information processing device to realize the information processing device on the computer, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present invention. [Effects of the Invention]

[0011] According to one aspect of the present invention, it is possible to provide a technology that can associate and store various data and track the process of crop growth. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an information processing device 1 according to an embodiment. [Figure 2] 2 is a schematic diagram showing an example of the contents and relationships of data stored in layers 1 to 4 according to the embodiment. FIG. [Figure 3] FIG. 10 is a schematic diagram showing a specific example of the relationship between the "Trees" table in Layer 1 and the "3D Models" table in Layer 2. [Figure 4] FIG. 10 is a schematic diagram showing a specific example of the relationship between the "3D model" table and "2D image" table of layer 2 and the "organ position in 3D model" table and "organ position in 2D image" table of layer 3. [Figure 5]This is a schematic diagram that also includes the relationship between the "organ position in 3D model" table in layer 3 and the "organ" table in layer 4. [Figure 6] This is a schematic diagram that also includes the relationship between the "organ positions in 2D images" table in Layer 3 and the "organ" table in Layer 4. [Figure 7] FIG. 10 is a schematic diagram showing a specific example of relationships in tables included in Layer 2. [Figure 8] 1 is a schematic diagram showing specific examples of table data included in Layer 1 and Layer 2 and link relationships. FIG. [Figure 9] 10 is a schematic diagram showing specific examples of data in tables included in Layer 2 and Layer 3 and link relationships. FIG. [Figure 10] 10 is a schematic diagram showing specific examples of table data included in Layer 3 and Layer 4 and link relationships. FIG. [Figure 11] 10 is an example of a display image (display screen) of a farm field. [Figure 12] 10 is an example of a display image displaying an image of a tree or a three-dimensional model thereof. [Figure 13] 10 is an example of a display image showing the positions of organs included in a three-dimensional model. [Figure 14] 1 is an example of a display image showing individual organs. [Figure 15] 10 is an example of a display image that displays images of organs having the same organ ID at different times. [Figure 16] 1 is a schematic diagram showing an information processing device 1 according to the present embodiment and examples of various data acquired by the information processing device 1. FIG. [Figure 17] 10 is a flowchart showing the flow of an information processing method S1. DETAILED DESCRIPTION OF THE INVENTION

[0013] [Embodiment 1] An embodiment of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this embodiment. The information processing device 1 is a device for tracking the growth status of crops such as fruit trees grown in a farm field, for each organ of the crop.

[0014] (Information processing device 1) As shown in FIG. 1, the information processing device 1 includes an acquisition unit 11, an identification unit 12, an editing unit 13, a presentation unit 14, an analysis unit 15, a storage unit 16, a communication unit 17, and a control unit 20. The information processing device 1 may be connected to a display device 30, or the information processing device 1 may include a display unit (not shown). The information processing device 1 may be connected to an imaging device 60 such as a camera or a depth camera, a 3D lidar 70, or the like (hereinafter, the imaging device 60 or the 3D lidar 70, or the like, may also be referred to as an "imaging device, or the like") via the Internet (or an intranet) 50 so as to be able to communicate information. The data acquired by the imaging device 60 or the 3D lidar 70, or the like, is analyzed by the analysis unit 15 to determine which organ is being imaged.

[0015] The acquisition unit 11 may acquire crop data from an external storage device (not shown) via the communication unit 17 based on a user operation and store the data in the storage unit 16. For example, the acquisition unit 11 may acquire images, position data, or point cloud data acquired by an operator patrolling the field with an imaging device or the like via the Internet or intranet and the communication unit 17, or may acquire the data from an external storage device via the communication unit 17, which can be wireless or wired. Instead of having an operator patrolling the field to acquire data, the acquisition unit 11 may acquire data via the communication unit 17 from an imaging device or the like mounted on a mobile object that moves on a track installed within the field. Fixed-point measurements using an imaging device or the like mounted on a mobile object facilitate the identification of individual crops (e.g., individual trees, individual stalks, etc.) or organs in the images. Each individual crop is assigned an individual ID as plant identification information, which will be described later.

[0016] The identification unit 12 identifies organ position information of a predetermined organ at a certain point in time by referring to information related to a predetermined organ of a predetermined plant across multiple layers. Organs include the trunk, branches, leaves, inflorescences, buds, and fruit of a crop (plant). In particular, organs are parts of a crop that are subject to work performed before harvesting. For example, grapes require work such as pruning and training of branches, pinching, shaping of fruit spikes, gibberellin treatment, fruit thinning, and bagging. Therefore, tracking of predetermined organs such as branches, inflorescences, and fruit is necessary. Organ position information is two-dimensional or three-dimensional position information of the predetermined organ in image data, point cloud data, etc.

[0017] Furthermore, the identification unit 12 may identify different individuals from image data or the like and assign an individual ID to each individual. Alternatively, the individual ID may be assigned by the user.

[0018] The editing unit 13 edits the multiple organ position information identified by the identification unit 12 so as to arrange each predetermined organ chronologically, with reference to the time information included in the shape information. The shape information is information indicating the shape of the plant, and is information indicating the skeletal structure of the plant, including, for example, the trunk, branches, etc. The shape information can be acquired as an image or point cloud data using an imaging device, etc. The shape information includes time information (date and time data) when the image data or point cloud data, etc. was acquired. Therefore, the editing unit 13 can arrange the predetermined organs chronologically. "Chronologically" means in chronological order, and the organs can be arranged chronologically with reference to the time information. Furthermore, "chronologically" also includes selecting the most recent one.

[0019] Each organ is assigned an organ ID. The identification unit 12 may identify different organs from image data, etc., and assign organ IDs to them. Alternatively, the organ IDs may be assigned by the user. The editing unit 13 selects image data, point cloud data, etc. having the same organ ID and arranges them chronologically. The editing unit 13 may also select and arrange the most recent information for at least one individual or organ from the data stored in the memory unit 16. This makes it possible to track the growth stage of a certain organ over time. Furthermore, since different crops require different tasks depending on their growth stage, it becomes easier to manage cultivation by determining which individual is in which growth stage at a given time and when and what tasks should be performed on that individual.

[0020] In response to a user's selection operation, the presentation unit 14 presents, over time, shape information corresponding to organ position information for each predetermined organ edited by the editing unit 13. Presenting means, for example, outputting to an external display device 30. The presentation unit 14 outputs (presents) to the display device 30 an image in which images of a certain organ are arranged over time, edited by the editing unit 13.

[0021] Furthermore, the presentation unit 14 may present the plant identification information and arrangement information indicating the arrangement of the plant in the field in response to a selection operation by the user. Furthermore, the presentation unit 14 may present the plant identification information and shape information in response to a selection operation by the user. Furthermore, the presentation unit 14 may present the plant identification information and organ position information in response to a selection operation by the user.

[0022] The analysis unit 15 analyzes the positions of plant organs from the shape information. The analysis unit 15 may include, for example, an image analysis unit. The analysis unit 15 analyzes image data, which is shape data, to detect organs contained in the image data. Since the image data contains information about the individual contained therein, the identification unit 12 or the analysis unit 15 can identify organs contained in the individual in multiple image data images of the same individual. Furthermore, when the morphology of an organ changes, the identification unit 12 or the analysis unit 15 may determine whether organs of different shapes are the same organ based on the temporal change in the position of the organ. For example, even if the morphology changes from a flower to a fruit, if the position of the organ relative to the tree skeleton is approximately the same, it can be determined that the same organ has undergone a morphological change. In addition to the analysis performed by the analysis unit 15, the information processing device 1 may acquire the results of image analysis using an external image analysis device and identify the organ by referring to the analysis results.

[0023] For example, in grapevines, the position of new shoots can move significantly in a short period of time due to the softness of the shoots or due to the worker's training, making it difficult to identify the organ. In such cases, for example, the image acquisition interval can be shortened to acquire images before the organ moves too much, making it possible to identify the organ position. Furthermore, if the position or shape of an organ changes significantly due to pruning, training, pinching, etc. performed by the worker, it is possible to track the changes in the position and shape of the organ and identify the specific organ by referring to cultivation management information and image data, as described below.

[0024] Specifically, for example, a 3D model including plant structures such as main branches and new shoots can be created from images taken with cameras at different viewpoints. From the 3D model, point clouds of main branches and new shoots can be extracted using methods such as the spatial density of the point cloud and color extraction using the RGB vegetation index. Even after training, the connection points between main branches and new shoots in the point cloud do not change, so the new shoots can be tracked and their positions identified from the connection points.

[0025] The memory unit 16 stores data on crops grown in the field. The type of crop is not limited, but it is assumed to be fruit trees, etc., for which predetermined tasks are required depending on the growth stage. The memory unit 16 may store data in a table format. The memory unit 16 has a hierarchical data structure. Specifically, the memory unit 16 has at least a layer 1, a layer 2, a layer 3, and a layer 4. The data stored in the memory unit 16 includes two-dimensional or three-dimensional image data, two-dimensional or three-dimensional position data, point cloud data, etc., related to the crops. These data are acquired by the acquisition unit 11 and stored in different layers depending on the type of data. The acquisition unit 11 may successively acquire new data and record it in the memory unit 16 in response to a user instruction.

[0026] Layer 1 is a layer that stores plant identification information, which is information that identifies plants in a field. There are many individual fruit trees and other plants in the field. Layer 1 is a layer that stores an individual ID assigned to each individual plant. The plant identification information is associated with shape information, which will be described later. Layer 1 may also store a field map, as will be described later.

[0027] Layer 2 is a layer that stores shape information that indicates the shape of a plant at a certain point in time. Shape information is information that indicates the skeletal structure of a plant, such as the trunk and branches of an individual. Shape information may be, for example, a three-dimensional model or two-dimensional image data. These three-dimensional models may be associated with two-dimensional images. A three-dimensional model is a model that indicates the three-dimensional shape of an individual. For example, the analysis unit 15 can generate a three-dimensional model using SfM (Structure from Motion). SfM is a method of generating a three-dimensional model (three-dimensional shape) of an individual from multiple image data of the same individual.

[0028] The two-dimensional image data is, for example, image data including a part of an individual plant, such as a branch, or the entire individual plant. The shape information can be acquired using an imaging device 60 or the like. The shape information in layer 2 is associated with the plant identification information in layer 1, and is also associated with organ position information, which will be described later.

[0029] The shape information may include both a 3D model and a 2D image of the same individual. This shape information can be used to identify organs contained in the individual. The shape information includes date and time information regarding the date and time when the shape information was acquired. The date and time information is an example of time information, which will be described later.

[0030] Layer 3 is a layer that stores organ position information indicating the positions of plant organs included in the shape information. The shape information may include information about organs included in the individual plant. Furthermore, the shape information may include two-dimensional or three-dimensional position data. The position data can be acquired using a ranging device or an imaging device. For example, using a depth camera, it is possible to associate an organ in an image with its three-dimensional coordinate data. Furthermore, using a two-dimensional camera, it is possible to associate an organ in an image with its two-dimensional coordinate data. Three-dimensional coordinate data of an organ acquired using a depth camera or Lidar, etc., and / or two-dimensional coordinate data of an organ acquired using a camera are examples of organ position information.

[0031] Layer 4 is a layer that stores organ identification information, which is information that identifies plant organs. Organs in images captured by an imaging device or the like can be identified using image analysis or the like by the analysis unit 15, which will be described later. Each identified organ is assigned an organ ID. The organ ID is an example of organ identification information. The organ identification information is associated with the individual ID of the individual to which the organ belongs. In addition, the organ identification information is associated with the organ position information in Layer 3.

[0032] Furthermore, the organ identification information stored in Layer 4 is associated with organ-associated information, which is information that may cause the movement or morphological change of the organ. The organ-associated information may be, for example, information about the cultivation of the organ (cultivation management information) and / or information about the growth of the organ (growth information), and may be recorded in Layer 4. The cultivation management information may be information about the work performed before harvesting the crop and the date and time the work was performed. The growth information may be information about the growth stage of the crop. For example, when a worker prunes the branches or vines of the crop, or trains and fixes them to a support material, the positions of organs such as flowers and fruits move. Furthermore, the positions of flowers and fruits change as the branches and vines extend with growth. Furthermore, as the plant grows, its shape changes from flowers to buds, and from buds to fruit, and as the fruit grows, its size and color change. In this way, cultivation management information, growth information, and the like that may cause the movement or morphological change of the organ are referred to as organ-associated information.

[0033] Each piece of information is related to the other. For example, plant identification information in layer 1 is related to shape information in layer 2. Shape information in layer 2 is also related to organ position information in layer 3. Organ position information in layer 3 is also related to organ identification information in layer 4. A more specific example of a data structure showing the relationship between the data in each layer will be described later.

[0034] The communication unit 17 is an interface for the information processing device 1 to communicate with the outside. The communication unit 17 may be, for example, a short-range wireless communication interface such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or may be a USB or the like.

[0035] The control unit 20 performs overall control of the information processing device 1. The control unit 20 includes at least one processor 21 and at least one memory 22. The processor 21 can be configured using a general-purpose processor such as at least one MPU (Micro Processing Unit) or CPU (Central Processing Unit). The memory 22 may include multiple types of memory such as ROM (Read Only Memory) and RAM (Random Access Memory). As an example, the processor 21 implements the functions of each unit by loading various control programs recorded in the ROM of the memory 22 into the RAM and executing them. The processor 21 may also include a dedicated processor configured using an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), PLD (Programmable Logic Device), or the like.

[0036] (Data Structure) Next, an example of the data structure of the storage unit 16 will be described in detail. Figures 2 to 7 are schematic diagrams showing an example of the types of data stored in each layer and their relationships. As mentioned above, the storage unit 16 has at least four layers, Layer 1 to Layer 4.

[0037] Figure 2 is a schematic diagram showing an example of the contents and relationships of data stored in Layer 1 (L1) to Layer 4 (L4). In the illustrated example, one table is shown in one box. The underlined name in the box is the name of the table. Tables connected by a line indicate that the data in the tables is related. Being related means that a link has been established. Furthermore, a line with a black circle at one end indicates that the data in the two tables connected by the line is related in a "one-to-many" relationship. Here, the table without a black circle on the line is "1," and the table with a black circle is "many." A "#" in the table indicates a PK (primary key).

[0038] Layer 1 stores plant identification information. In the illustrated example, the tree IDs (individual IDs) of multiple trees are stored as plant identification information in a single "tree" table. Layer 1 also stores "field map" data, and tree IDs are associated with map data. In other words, the positions of multiple tree IDs are stored in a single piece of map data, and when a tree ID is specified, the position of that individual tree is displayed on the map. The field map can be created by SLAM (Simultaneous Localization and Mapping) using point cloud data from a Lidar 70 mounted on a drone, for example. The field map may also be created by the analysis unit 15.

[0039] Layer 2 (L2) stores shape information of individuals. In the illustrated example, Layer 2 includes a "3D model" table and a "2D image" table, each of which is related to a "relationship between 3D models and 2D images" table. The data contained in the "3D model" table is 3D model data (shape data) including tree trunks and branches, acquired by a distance measuring device or the like. The data contained in the "2D image" table is 2D image data acquired by an imaging device. Each piece of data may include data such as the acquisition (photography) date and time, acquisition (photography) location, and acquisition (photography) angle. The "relationship between 3D model and 2D image" indicates 3D model data and 2D image data representing the same individual. For example, data for the same individual is related to each other by listing them in the same column.

[0040] As shown in the figure, the "Tree" table in Layer 1 is related to the "3D Model" table in Layer 2. In other words, multiple 3D models and 2D images are related to one tree. However, the 2D images are related to the "Tree" table in Layer 1 via the 3D models.

[0041] Layer 3 (L3) stores organ position information. As shown in the figure, Layer 3 stores an "organ position in 3D model" table and an "organ position in 2D image" table. The "organ position in 3D model" table is associated with the "3D model" table of Layer 2, and the "organ position in 2D image" table is associated with the "2D image" table of Layer 2. The "organ position in 3D model" table stores three-dimensional coordinate data of organs included in one tree in the "3D model" table. This table may also store the size of the organs. The "organ position in 2D image" table stores two-dimensional coordinate data of organs included in one tree in the "2D image" table. This table may also store the size of the organs.

[0042] Layer 4 (L4) stores organ identification information. As shown in the figure, Layer 4 stores an "organ" table. For example, the "organ" table stores cultivation management information, growth information, etc. for each organ ID. Specific examples of cultivation management information and growth information are as described above. The "organ" table is related to the "organ position in 3D model" table and the "organ position in 2D image" table in Layer 3.

[0043] Figure 3 is a schematic diagram showing a specific example of the relationship between the "Tree" table in Layer 1 and the "3D Model" table in Layer 2. A dotted frame indicates one layer and is assigned a layer number. A solid frame within a dotted frame represents a group of data. An arrow connecting two solid frames indicates that the two frames are related. As shown in the figure, multiple pieces of 3D model data with different photo dates and directions are related to one tree ID.

[0044] 4 is a schematic diagram showing a specific example of the relationship between the "3D model" table and "2D image" table of layer 2 and the "organ position in 3D model" table and "organ position in 2D image" table of layer 3. As shown in the figure, for example, one piece of 3D model data is related to the 3D coordinate data of an organ included in that model data (a flower in the illustrated example). Also, for example, one piece of 2D image data is related to the 2D coordinate data of an organ included in that image (a flower in the illustrated example). These relationships are one (layer 2) to many (layer 3) relationships.

[0045] Figure 5 is a schematic diagram that includes the relationships shown in Figure 4 as well as the relationship between the "Organ position in 3D model" table in Layer 3 and the "Organ" table in Layer 4. In this example, the data for "Flower ID #1" in the "Organ" table is related to the data for "ID #1, Model ID #1" and "ID #3, Model ID #4" in the "Organ position in 3D model" table. This indicates that this is the 3D position data for a flower after it has been moved by work in the cultivation management information included in, for example, "Cultivation management information 1, growth stage 2."

[0046] Fig. 6 is similar to Fig. 5, but is a schematic diagram that also includes the relationship between the "organ position in 2D image" table in Layer 3 and the "organ" table in Layer 4. In other words, it shows the 2D position data of the flower after it has been moved by the work of the cultivation management information included in "cultivation management information 1 growth stage 2."

[0047] FIG. 7 is a schematic diagram showing a specific example of relationships in tables included in layer 2. As described above, the "3D Model" table and the "2D Image" table included in layer 2 are respectively related to each other as in the "Relationship between 3D Models and 2D Images" table. For example, the topmost box (column) of the "Relationship between 3D Models and 2D Images" table has a relationship ID of #1, indicating that the data of the 3D model with "Model ID #1" and the data of the 2D image with "Image ID #1" are related to each other. Similarly, the second box (column) of the "Relationship between 3D Models and 2D Images" table has a relationship ID of #2, indicating that the data of the 3D model with "Model ID #1" and the data of the 2D image with "Image ID #2" are related to each other. The same applies below.

[0048] 8 to 10 are schematic diagrams showing specific examples of data stored in each table and the link relationships. As in FIG. 2, a line segment with a black circle at one end indicates that the data in the two tables connected by the line segment is related in a one-to-many relationship (the side with the black circle indicates the many). FIG. 8 is a schematic diagram showing specific examples of data in tables included in Layer 1 and Layer 2 and the link relationships. For example, the "tree" table (denoted as "tree") in Layer 1 stores "tree_id" (tree ID), "tree_num" (tree number), "gps_x" (X coordinate in the GPS: Global Positioning System), "gps_y" (Y coordinate in the GPS), "style_of_tree" (tree shape to be cultivated, e.g., short-shoot cultivation H-type, short-shoot cultivation straight-type, long-shoot cultivation, open-core natural type, etc.), "year of cultivation" (date of cultivation or transplanting), "variety" (variety), and "field_id" (field ID). In addition, the "Field Map" table (denoted as "field") in Layer 1 stores the following data: "field_id" (field ID), "field_gps_x" (X coordinate of the field using GPS), "field_gps_y" (Y coordinate of the field using GPS), "owner", "area" (field area), and "kind_of_plat" (type of tree).

[0049] The "3D model" table (denoted as 3D model) in layer 2 stores the "3D model_id" (3D model ID), the "view" of the 3D model (camera viewpoint, especially the vertical angle, e.g., 45° upwards), "model_fname" (model file name), "model_path" (model storage location), "model_size" (model data size), "sfm" (whether or not 3D point cloud processing is performed), "dense" (whether or not dense point cloud processing is performed), "cord" (whether or not a coordinate system is performed), "tree_id" (tree ID), "date" (date and time), "n_of_points" (number of points in 3D point cloud processing), "n_of_dense" (number of points in dense point cloud processing), "chunk" (data operation), "side" (approximate azimuth, e.g., facing west), and "link" (location where the model's images are stored). The "3D model to 2D image relationship" table (denoted as "3D model to 2D image") in Layer 2 stores the "3D model to 2D image id" (the relationship ID between the 3D model and the 2D image), as well as the "3D model_id" (3D model ID) and "2D image_id" (2D image ID) for the same tree. The "2D image" table (denoted as "2D image") in Layer 2 stores the "2D image_id" (2D image ID), the 2D image's "type" (image format, e.g., JPG, TIFF, PNG), "path" (image storage location), "resolution_x" (image resolution X), and "resolution_y" (image resolution Y). The lines connecting tables indicate the primary key (PK, a field designated to uniquely identify a record within a table) data and the linked foreign key (FK, a constraint that restricts a column in one table from entering fields contained in a specific column in another table). For example, the tree ID (primary key) in the "Trees" table in Layer 1 is linked to the tree ID (foreign key) in the "3D Model" table in Layer 2. Also, the field ID (primary key) in the "Field Map" table in Layer 1 is linked to the field ID (foreign key) in the "Trees" table.

[0050] Figure 9 is a schematic diagram showing specific examples of data and link relationships in tables included in layer 2 and layer 3. The data in the tables included in layer 2 is as explained using Figure 8. The "organ position in 3D model" table in layer 3 (shown as 3D model to plant organ) stores "3D model to plant organ_id" (ID of the organ included in the 3D model), "3D model_id" (3D model ID), "plant organ_id" (organ ID), "x" (X coordinate of the organ), "y" (Y coordinate of the organ), "z" (Z coordinate of the organ), and "range" (radius of the organ in 3D space). Furthermore, the "organ position in 2D image" table (denoted as "2D image to plant organ") in layer 3 stores the "2D image to plant organ_id" (ID of the organ included in the 2D image), "2D image_id" (ID of the 2D image), "plant organ_id" (organ ID), "class" (type of organ, e.g., shoot, flower, cluster, etc.), "x" (X coordinate of the organ), "y" (Y coordinate of the organ), "width" (horizontal length of the organ), and "height" (vertical length of the organ). The meaning of the lines connecting tables is as explained using Figure 8. Furthermore, tables connected by lines and indicated by thick frames indicate that they are linked in a many-to-many relationship.

[0051] FIG. 10 is a schematic diagram showing specific examples of data in tables included in layers 3 and 4 and link relationships. The data in the tables included in layer 3 is as explained using FIG. 9. In addition to the "organ" table (indicated as "plant organ"), layer 4 in FIG. 10 also includes a "cultivation management information" table (indicated as "cultivation") and a "growth information" table (indicated as "growth record"). The "organ" table stores "plant organ_id" (organ ID), "growth record_id" (growth information ID), and "cultivation_id" (cultivation management information ID). The "cultivation management information" table stores "cultivation_id" (cultivation management information ID), "user" (worker), "content" (work content), and "date" (date and time). The "growth information" table stores "growth record_id" (growth information ID), "user" (worker), "full flowing" (whether the flower is in full bloom), and "date" (date and time).

[0052] 11 to 15 are schematic diagrams showing examples of display images that the presentation unit 14 presents (outputs) to the external display device 30. FIG. 11 is an example of a display image (display screen) of a farm field. In this example display image, the user can select the type of image to display, such as a map or Lidar map. The user can also select which farm field to display. In this example, the screen is configured to display, along with the image of the farm field, information about the farm field being displayed and information about trees cultivated in the field, etc., as selected by the user. The image of the farm field shown in the figure is, for example, a Lidar map acquired by Lidar from the sky.

[0053] 12 is an example of a display image displaying an image of a tree or its three-dimensional model. In this example display image, the user can select the angle range of the image captured. Furthermore, depending on the user's selection, related information such as the tree number, cultivation method, variety, cultivation record (cultivation management information), and yield data of the displayed tree may be displayed, or a link to related information may be displayed. Also, a list may be displayed from which the user can select other images with the same tree ID.

[0054] 13 is an example of a display image that displays the positions of organs (flowers in the illustrated example) included in a 3D model. The display image may also display a list of organs included in the displayed 3D model. Furthermore, the positions of the organs may be marked, and when the user places the cursor over them, the organ ID, position data, etc. may be displayed.

[0055] FIG. 14 is an example of a display image displaying individual organs. Since organs are the subject of cultivation management, cultivation management information may be displayed within the same display image. For example, the system may be configured so that cultivation management information is displayed when the user places the cursor on the location of an organ. Furthermore, since there is a high possibility that the organs are at the same growth stage in terms of time, the system may be configured so that image information of nearby organs or individuals is displayed upon selection by the user. The cultivation management information may be obtained by SQL (sequence) search and displayed as a list.

[0056] FIG. 15 is an example of a display image that displays images of organs with the same organ ID taken at different times. While only one image is displayed in FIG. 15, multiple images taken at different times may be displayed on a single display screen by reducing the size of the image. A list listing the image capture dates and times, capture locations, etc., of images taken from different positions or directions, and images taken at different times that include organs with the same organ ID may also be displayed. Furthermore, a configuration may be adopted in which images that include organs with the same organ ID are arranged in chronological order, as selected by the user. Furthermore, a configuration may be adopted in which the most recent image, in terms of time, of images that include organs with the same organ ID is displayed, as selected by the user.

[0057] FIG. 16 is a schematic diagram showing an example of an information processing device 1 according to this embodiment and various types of data acquired by the information processing device 1. The information processing device 1 is shown in the box on the left side of FIG. 16. The "analysis data" in the box on the left side refers to data analyzed by a data analyst via a user screen and data after analysis. The "analysis data" is data obtained through analytical processes such as image processing, deep learning, and 3D processing, including the location and tracking of plant organs and the determination of work timing. The "cultivation management data" is manually entered by a cultivation manager via a data management screen, and includes, for example, data recording cultivation management information such as pesticide application dates and harvest dates. The "analysis data" and "cultivation management data" form a database that comprehensively manages data acquired from a single orchard, and can be called an orchard platform integrated database.

[0058] The right side of Fig. 16 is a schematic diagram showing an example of the types of data that can be processed by the information processing device 1 and the relationship between them. As mentioned above, there are many types of data, such as image data, distance data (depth data), point cloud data, position data (GPS data), two-dimensional data, and three-dimensional data, but with conventional technology, it has been difficult to manage all different types of data in a unified manner. However, by using the information processing device 1 according to this embodiment, different types of data can be managed in a unified manner.

[0059] As described above, in the information processing device 1, related data is stored in layers in the storage unit 16, and data between multiple layers is related by links. The identification unit 12 can then identify organ position information for a specific organ at a given point in time by referring to information related to a specific organ of a specific plant across multiple layers. This configuration makes it possible to extract and organize multiple types of data for any target (e.g., by organ). For example, data on a specific organ of a specific crop can be identified to track the growth process of that crop.

[0060] (Information processing method S1) Next, the information processing method S1 executed by the information processing device 1 will be described with reference to the drawings. Fig. 17 is a flowchart showing the flow of the information processing method S1. As shown in the figure, the information processing method S1 includes steps S11 to S16. It is not necessary to execute all of the steps S11 to S16. Furthermore, the order of the steps may be changed as appropriate, or the steps may be executed in parallel.

[0061] Step S11 is a storage step in which plant identification information, which is information that identifies plants in a field, is stored in the storage device as information of layer 1, shape information that indicates the shape of the plant at a certain point in time is stored in the storage device as information of layer 2, organ position information that indicates the positions of the plant organs included in the shape information is stored in the storage device as information of layer 3, and organ identification information, which is information that identifies the plant organs, is stored in the storage device as information of layer 4. The storage step is executed by the acquisition unit 11 of the information processing device 1 based on an input operation by the user.

[0062] Step S12 is an association step in which plant identification information is associated with shape information, shape information is associated with organ position information, and organ position information is associated with organ identification information. In this embodiment, since the storage unit 16 has a layer structure set by the user, predetermined data is stored in a predetermined table of the layer, and thus the predetermined association defined in the table is executed.

[0063] Step S13 is a detection step of detecting a predetermined organ included in the shape information by image analysis. The detection step is executed by the analysis unit 15 of the information processing device 1 based on a user operation.

[0064] Step S14 is an identification step in which organ position information of a predetermined organ at a certain point in time is identified by referring to information about a predetermined organ of a predetermined plant that is correlated across multiple layers. The identification step is executed by the identification unit 12 of the information processing device 1 based on a user operation. Note that steps S13 and S14 may be reversed in order or may be executed in parallel.

[0065] Step S15 is an editing step in which the identified organ position information is edited so as to be arranged over time for each predetermined organ by referring to the time information included in the shape information. The editing step is executed by the editing unit 13 of the information processing device 1.

[0066] Step S16 is a presentation step of presenting shape information corresponding to the edited organ position information for each predetermined organ over time. The presentation step is executed by the presentation unit 14 of the information processing device 1. The presentation unit 14 arranges the predetermined organs over time based on a user operation and outputs the arranged information to the display device 30 or the like.

[0067] According to the information processing method S1 described above, it is possible to detect a specific organ from data related by links between multiple layers in the storage unit 16, identify its position, and arrange it over time by referring to time information. Then, by displaying image data of the organ on a display device, a user can recognize the growth status of crops such as fruits over time and track the process of crop growth. This makes it easy to perform various tasks at the appropriate time during the growth stages of the crop.

[0068] [Software implementation example] The functions of the information processing device 1 (hereinafter referred to as the "device") can be realized by an information processing program that causes a computer to function as the device, and that causes a computer to function as each part executed by the control unit 20 of the device.

[0069] In this case, the device includes a computer having at least one control device (e.g., processor 21) and at least one storage device (e.g., memory 22) as hardware for executing the information processing program. The control device and storage device execute the information processing program, thereby realizing each function described in the above embodiment.

[0070] The information processing program may be stored non-transitory on one or more computer-readable storage media. The storage media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0071] Furthermore, some or all of the functions of the control unit 20 can be realized by a logic circuit. For example, the scope of the present invention also includes an integrated circuit in which a logic circuit that functions as the control unit 20 is formed. In addition, the functions of the control unit 20 can also be realized by, for example, a quantum computer.

[0072] (Additional notes) An information processing device according to aspect 1 of the present invention comprises a memory unit that stores plant identification information, which is information for identifying a plant in a field, as information in layer 1, shape information indicating the shape of the plant at a certain point in time as information in layer 2, organ position information indicating the position of the organ of the plant included in the shape information as information in layer 3, and organ identification information, which is information for identifying the organ of the plant, as information in layer 4, and an identification unit that identifies the organ position information of the specified organ at a certain point in time by referring to information regarding a specified organ of a specified plant that is related to each other across multiple layers.

[0073] An information processing device according to aspect 2 of the present invention is, in aspect 1, further provided with an editing unit that edits the multiple organ position information identified by the identification unit so that they are arranged over time for each specified organ by referring to time information included in the shape information, and a presentation unit that presents over time the shape information corresponding to the organ position information for each specified organ edited by the editing unit.

[0074] An information processing device according to aspect 3 of the present invention is the information processing device according to aspect 1 or 2, The plant identification information and the shape information are associated with each other, the shape information and the organ position information are associated with each other, and the organ position information and the organ identification information are associated with each other.

[0075] An information processing device according to a fourth aspect of the present invention is any one of the first to third aspects, wherein the shape information is represented by a three-dimensional model or a two-dimensional image, and the three-dimensional model and the two-dimensional image are associated with each other.

[0076] An information processing device according to aspect 5 of the present invention is any one of aspects 1 to 4, in which the organ identification information is associated with organ-associated information, which is information that may be the cause of movement or change in morphology of the plant organ.

[0077] An information processing device according to a sixth aspect of the present invention is in accordance with the fifth aspect, wherein the organ-associated information is at least one of information relating to the growth of the organ and information relating to the cultivation of the organ.

[0078] The information processing device according to a seventh aspect of the present invention is the information processing device of any one of the first to sixth aspects, further comprising an analysis unit that analyzes the position of the plant organ from the shape information.

[0079] An information processing device according to aspect 8 of the present invention is any one of aspects 1 to 7, wherein the presentation unit further presents the plant identification information and placement information indicating the placement of the plant in the field in response to a user's selection operation.

[0080] An information processing device according to a ninth aspect of the present invention is any one of the first to eighth aspects, wherein the presentation unit further presents the plant identifying information and the shape information in response to a selection operation by a user.

[0081] An information processing device according to a tenth aspect of the present invention is any one of the first to ninth aspects, wherein the presentation unit presents the plant identification information and the organ position information in response to a selection operation by a user.

[0082] An information processing method according to aspect 11 of the present invention includes a storage process that stores plant identification information, which is information that identifies a plant in a field, as information in layer 1, shape information indicating the shape of the plant at a certain point in time as information in layer 2, organ position information indicating the position of the organ of the plant included in the shape information as information in layer 3, and organ identification information, which is information that identifies the organ of the plant, as information in layer 4, and an identification process that identifies the organ position information of the specified organ at a certain point in time by referring to information regarding a specified organ of a specified plant that is related to each other across multiple layers.

[0083] The data structure of aspect 12 of the present invention comprises a layer 1 that stores plant identification information, which is information that identifies plants in a field; a layer 2 that stores shape information that indicates the shape of the plant at a certain point in time; a layer 3 that stores organ position information that indicates the position of the organs of the plant included in the shape information; and a layer 4 that stores organ identification information, which is information that identifies the organs of the plant, wherein the plant identification information and the shape information are related to each other, the shape information and the organ position information are related to each other, and the organ position information and the organ identification information are related to each other.

[0084] A program according to a thirteenth aspect of the present invention includes the steps of: The system executes a storage process in which plant identification information, which is information that identifies plants in a field, is stored as information in layer 1, shape information that indicates the shape of the plant at a certain point in time is stored as information in layer 2, organ position information that indicates the position of the organs of the plant included in the shape information is stored as information in layer 3, and organ identification information, which is information that identifies the organs of the plant, is stored as information in layer 4, and an identification process in which the organ position information of the specified organ at a certain point in time is identified by referring to information about a specified organ of a specified plant that is related to each other across multiple layers.

[0085] A non-transitory recording medium according to a fourteenth aspect of the present invention is a computer-readable non-transitory recording medium having the information processing program according to the thirteenth aspect recorded thereon.

[0086] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0087] 1...Information processing device 11…Acquisition part 12…Specific part 13...Editorial Department 14…Presentation part 15…Analysis department 16...Storage section 17…Communications Department 20...Control unit 21...Processor 22...Memory 30…Display device 50…Internet 60...imaging device 70...Lidar

Claims

1. a storage unit that stores plant identification information, which is information for identifying plants in a field, as information in layer 1, stores shape information indicating the shape of the plant at a certain point in time as information in layer 2, stores organ position information, which is included in the shape information and indicates the positions of the organs of the plant, as information in layer 3, and stores organ identification information, which is information for identifying the organs of the plant, as information in layer 4; an identification unit that identifies the organ position information of a predetermined organ at a certain point in time by referring to information about a predetermined organ of a predetermined plant that is correlated between multiple layers; An information processing device comprising:

2. an editing unit that edits the plurality of pieces of organ position information identified by the identifying unit so as to arrange the pieces of organ position information identified by the identifying unit in a chronological order for each predetermined organ by referring to time information included in the shape information; a presentation unit that presents shape information corresponding to the organ position information for each of the predetermined organs edited by the editing unit over time, The information processing device according to claim 1 .

3. the plant identification information and the shape information are associated with each other, the shape information and the organ position information are associated with each other, and the organ position information and the organ identification information are associated with each other; The information processing device according to claim 1 .

4. the shape information is represented by a three-dimensional model or a two-dimensional image; The three-dimensional model and the two-dimensional image are related to each other. The information processing device according to claim 3 .

5. The organ identification information is associated with organ-associated information, which is information that may cause movement or morphological change of the plant organ. The information processing device according to claim 3 .

6. The organ-associated information is at least one of information on the growth of the organ and information on the cultivation of the organ. The information processing device according to claim 5 .

7. further comprising an analysis unit that analyzes the position of the plant organ from the shape information; The information processing device according to claim 1 .

8. the presentation unit further presents the plant identification information and arrangement information indicating an arrangement of the plant in the field in response to a selection operation by a user. The information processing device according to claim 2 .

9. the presentation unit further presents the plant identification information and the shape information in response to a selection operation by a user. The information processing device according to claim 2 .

10. the presentation unit presents the plant identification information and the organ position information in response to a selection operation by a user. The information processing device according to claim 2 .

11. a storage process in which plant identification information, which is information for identifying plants in a field, is stored as layer 1 information, shape information indicating the shape of the plant at a certain point in time is stored as layer 2 information, organ position information indicating the positions of the organs of the plant included in the shape information is stored as layer 3 information, and organ identification information, which is information for identifying the organs of the plant, is stored as layer 4 information; an identification process for identifying the organ position information of a predetermined organ at a certain point in time by referring to information about the predetermined organ of a predetermined plant that is correlated between multiple layers; An information processing method, including:

12. a layer 1 that stores plant identification information that identifies plants in a field; Layer 2 stores shape information indicating the shape of the plant at a certain point in time; a layer 3 that stores organ position information indicating the positions of the plant organs included in the shape information; a layer 4 that stores organ identification information that identifies the organs of the plant; the plant identification information and the shape information are associated with each other, the shape information and the organ position information are associated with each other, and the organ position information and the organ identification information are associated with each other; Data structure.

13. On the computer, a storage process in which plant identification information, which is information for identifying plants in a field, is stored as layer 1 information, shape information indicating the shape of the plant at a certain point in time is stored as layer 2 information, organ position information indicating the positions of the organs of the plant included in the shape information is stored as layer 3 information, and organ identification information, which is information for identifying the organs of the plant, is stored as layer 4 information; an identification process for identifying the organ position information of a predetermined organ at a certain point in time by referring to information about the predetermined organ of a predetermined plant that is correlated between multiple layers; A program that executes.

Images