Agricultural product information processing device, agricultural product information processing method, and agricultural product information processing program

Abstract

To provide an agricultural products information processing device, an agricultural products information processing method, and an agricultural products information processing program capable of obtaining a processing technique of agricultural information effective for improving efficiency of agricultural work.SOLUTION: A method for managing information on a field stores, in a database, a position of a crop in a field as layer 1 information, a position of a fruit of a crop as information of a layer 2, content of work performed to the fruit as information of a layer 3 to specify correspondence relation between the information of the layer 2 and the information of the layer 3 based on an image obtained by photographing the work.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for processing information on agricultural products. [Background technology]

[0002] The use of various sensing technologies and robots is being explored with the aim of improving the efficiency of agricultural work (see, for example, Patent Document 1). For example, in the case of fruit tree cultivation, tasks such as pruning, shaping, and chemical treatment must be carried out at appropriate times depending on the stage of growth. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-532666 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, agricultural process management relies on the experience and intuition of workers. Meanwhile, while producers are aging and the number of agricultural producers is declining, there is a demand for larger-scale, more efficient agricultural production. Given this situation, the present invention aims to provide an agricultural information processing technology that is effective in improving the efficiency of agricultural work. [Means for solving the problem]

[0005] The present invention provides a system including a storage unit that stores the positions of crops in a field as information on Layer 1, the positions of the fruits of the crops as information on Layer 2, and the details of work performed on the fruits as information on Layer 3, and a correspondence relationship identification unit that identifies the correspondence relationship between the information on Layer 2 and the information on Layer 3 based on an image of the work, The layer 2 includes image data of the captured image of the fruit used to identify the position of the fruit, and the identification of the correspondence is performed based on objects that are commonly shown in the captured image included in the layer 2 and the image of the work. It is an agricultural product information processing device.

[0007] The present invention provides a storage unit that stores the position of a crop in a field as information on Layer 1, the position of a fruit of the crop as information on Layer 2, and the content of work performed on the fruit as information on Layer 3; and a correspondence relationship identification unit that identifies the correspondence relationship between the information on Layer 2 and the information on Layer 3 based on an image of the work,The layer 2 includes photographed images of the field, and the identification of the correspondence is performed by recognizing the fruit that appears in the image of the work from among the photographed images of the field included in the layer 2. Farming It is a product information processing device.

[0008] The present invention provides a storage unit that stores the position of a crop in a field as information on Layer 1, the position of a fruit of the crop as information on Layer 2, and the content of work performed on the fruit as information on Layer 3; and a correspondence relationship identification unit that identifies the correspondence relationship between the information on Layer 2 and the information on Layer 3 based on an image of the work, The layer 2 includes data of the actual 3D model, and the identification of the correspondence is performed based on the correspondence between the data of the 3D model included in the layer 2 and a 3D model created based on the image of the work. Farming It is a product information processing device.

[0009] In the present invention, the means for carrying out the work may include an identification mark for indicating the content of the work, and the content of the work may be identified based on an image obtained by capturing the identification mark.

[0010] The present invention provides a storage unit that stores the position of a crop in a field as information on Layer 1, the position of a fruit of the crop as information on Layer 2, and the content of work performed on the fruit as information on Layer 3; and a correspondence relationship identification unit that identifies the correspondence relationship between the information on Layer 2 and the information on Layer 3 based on an image of the work, The layer 1 includes image data for specifying the position of the crop in the field, and the layer 2 includes image data for specifying the position of the fruit, the position information of the layer 1 is described in an absolute coordinate system, and the position information of the layer 2 is described in a local coordinate system, and a positional relationship specifying unit is provided that determines the relationship between the absolute coordinate system and the local coordinate system based on the correspondence between the image data of the layer 1 and the image data of the layer 2. Farming It is a product information processing device.

[0011] The present invention stores the position of a crop in a field as information on Layer 1, the position of the fruit of the crop as information on Layer 2, and the details of the work performed on the fruit as information on Layer 3, and identifies the correspondence between the information on Layer 2 and the information on Layer 3 based on images of the work, The layer 2 includes image data of the captured image of the fruit used to identify the position of the fruit, and the identification of the correspondence is performed by a computer based on objects that are commonly shown in the captured image included in the layer 2 and the image of the work. A method for processing agricultural product information.

[0012] The present invention is a program that is read and executed by a computer, and causes the computer to operate as a storage unit that stores the positions of crops in a field as information on Layer 1, stores the positions of the fruits of the crops as information on Layer 2, and stores the details of work performed on the fruits as information on Layer 3, and a correspondence relationship identification unit that identifies the correspondence relationship between the information on Layer 2 and the information on Layer 3 based on images of the work, The layer 2 includes image data of the captured image of the fruit used to identify the location of the fruit, The correspondence is identified based on the objects that appear in common in the photographed image included in the layer 2 and the image of the work. This is a program for processing agricultural product information. [Effects of the Invention]

[0013] According to the present invention, an agricultural information processing technique that is effective in improving the efficiency of agricultural work can be obtained. [Brief explanation of the drawings]

[0014] [Figure 1] This is a schematic diagram of the field. [Figure 2] FIG. 2 is a conceptual diagram illustrating a data structure. [Figure 3] FIG. 2 is a block diagram of an agricultural product information processing device. [Figure 4] 10 is a flowchart illustrating an example of a processing procedure. [Figure 5] 10 is a flowchart illustrating an example of a processing procedure. [Figure 6] 10 is a flowchart illustrating an example of a processing procedure. [Figure 7] This is a conceptual diagram of a work robot seen from the side (A) and from the front (B). DETAILED DESCRIPTION OF THE INVENTION

[0015] 1. Overview Figure 1 shows an example of a vineyard 100 as an example of a farm field. In this example, an outdoor field is shown, but the field is not limited to being outdoors and may be a field inside an agricultural greenhouse. Furthermore, the plants are not limited to grapes and may be, for example, apples, pears, or other plants.

[0016] Vineyard 100 is planted with main trunks 110, 120, 130, and 140. Main branches extend from each main trunk in an H-shape when viewed from above. The main branches extend along wires (not shown) and support materials (not shown).

[0017] Figure 2 shows the data structure of the database that manages the information on the field in Figure 1. This database stores a 3D model map of vineyard 100. The data structure is hierarchical, with Layer 1, Layer 2, and Layer 3.

[0018] Layer 1 is an overall map that describes the location of the main trunks (individual trunks) in the field. Layer 2 is a map of the location of the fruit on each main trunk. Layer 3 is data that stores details of each fruit and data related to work, linked to Layer 2.

[0019] A number of reference points are set in the field, and reference point targets (e.g., reference number 160) are installed at these points. The reference points are those determined by surveying. The reference points are those in an absolute coordinate system. The reference point targets can be individually identified, and by detecting them in an image, the position of each point in the absolute coordinate system can be determined. The absolute coordinate system is a coordinate system used in GNSS and maps, and coordinates are described using latitude, longitude, and altitude. Once the position of the identification target in the absolute coordinate system has been determined, it functions as a position indicator that displays the position of each point in the field. Note that a local coordinate system can also be used instead of the absolute coordinate system.

[0020] A plurality of identification targets (e.g., symbol 150) are placed in various locations in the field. The identification targets are recognized from the captured image using image recognition technology and individually identified. Coded targets, two-dimensional barcodes, etc. are used as identification targets. This technology is described, for example, in JP 2021-139749 A.

[0021] Examples of locations where the identification targets can be placed include the ground, trunks, branches, cluster stalks, fruits, poles, support wires, other supporting materials, in the air (hanging from a wire, supported by a pole, etc.), the walls and ceilings of agricultural greenhouses, etc. The location where the identification targets can be placed is not particularly limited as long as they can be placed anywhere.

[0022] Identification targets can be in the form of flat plate structures, cylindrical shapes wrapped around branches or posts, sticker-like structures, displays directly on posts, or barcode displays on wires or poles.

[0023] It is also possible to arrange targets or patterns (dot patterns, etc.) to identify the correspondence between the two captured images.

[0024] The identification targets are positioned so that multiple targets are visible when photographing any location in the field. In particular, it is desirable to have three or more identification targets visible when photographing the buds (fruits).

[0025] A configuration that does not use an identification target is also possible. In this case, berries or clusters are individually identified using image recognition (e.g., AI image recognition). For example, suppose a large number of images are obtained by photographing various locations in a field. Then, suppose a specific cluster is photographed in the same field at a later time. In this case, image recognition technology is used to search for the cluster photographed later among the many former images. In this way, the correspondence between the first and second photographed images of the same object is identified.

[0026] In this embodiment, since information related to grape clusters is managed, care is taken to place the identification target near the position of the bud (cluster). Here, "nearby" means a position that appears in the captured image when the bud (cluster) is photographed, specifically within 50 cm, preferably within 30 cm, of the position of the bud (cluster). It is also possible to install additional identification targets during the growth process.

[0027] The identification targets in the captured images function as markers to identify the subject. They are also used as common points (corresponding points) in the process of determining correspondence between images. The same is true for the reference point targets.

[0028] 2. Layer details (Layer 1) Layer 1 is based on map information of the field described in an absolute coordinate system (general map information) and stores the position of the trunks of each crop in the field. Each crop is assigned an ID, and its position is stored in association with the ID. Layer 1 also stores a 3D model of the field in an absolute coordinate system created based on the principles of stereophotography. This 3D model digitizes the position of the main trunk in the field, as well as the three-dimensional positions of the main trunk and main branches (their state of development in three-dimensional space).

[0029] Layer 1 data also stores the positions in the absolute coordinate system of the identification targets placed in the field. In other words, the identification targets included in Layer 1 have known positions in the absolute coordinate system, so they function similarly to reference point targets. Layer 1 also stores the data of the captured images that form the basis of the Layer 1 data.

[0030] Some crops do not have a trunk (for example, crops that bear fruit on vines). In such cases, Layer 1 data is created for the crop row. Of course, Layer 1 data can also be created for crop rows that have a trunk.

[0031] (Layer 2) Layer 2 stores information about the position of the fruit on each main trunk. For example, in the case of grapes, the position of the bunch is stored. Each bunch is assigned an ID, and its position is stored in association with the ID. This is the same for fruits and vegetables other than grapes. Layer 2 also stores 3D data of the fruit. Layer 2 also stores data from the photographed images that form the basis of the data in Layer 2.

[0032] Layer 2 data includes data obtained at any position within the field, and is stored as data written in a local coordinate system. Note that a method for converting between each local coordinate system and an absolute coordinate system is determined using the method described below, and it is possible to convert Layer 2 data to an absolute coordinate system as needed.

[0033] By converting the data in layer 2 into an absolute coordinate system, the data in layer 2 can be associated with the data in layer 1. For example, the 3D model of a trunk in layer 1 and the 3D model of a bunch of trees at a specific part of a specific branch on that trunk are associated with each other by position information.

[0034] It is also possible to store data converted into data in an absolute coordinate system as data of layer 2. Of course, it is also possible to acquire data of layer 2 in an absolute coordinate system.

[0035] (Layer 3) Layer 3 stores information about the grapes whose position information is stored in Layer 2, as well as various work information. In this example, Layer 3 stores the size of the grapes in a grape cluster, the number of grapes, the axis length, work information, etc. Layer 3 also stores a 3D model of the grapes obtained from a photographed image. Work information includes information related to pruning, disinfection, pollination, shaping of branches and grapes, gibberellin treatment, bagging of grapes, grape thinning, application of growth regulators, etc.

[0036] The data in Layer 3 is linked to the data in Layer 2. That is, the number of berries, stalk length, details of the bunch, images, work information, etc. for each bunch whose position is stored in Layer 2 are stored as data in Layer 3. While the above example is of grapes, it is also possible to store information about fruit in general, such as a human subjective assessment of the size, shape, color, and growth status of the fruit, or the state recognized by the robot. Furthermore, by recording the position information of surrounding fruits, it is possible to make it easier for humans and robots to recognize the positional relationship between the next fruit to be worked on and fruit that has already been worked on.

[0037] (Data collection method) Data for Layer 1, Layer 2, and Layer 3 are created based on images captured by a camera. The camera is attached to a work robot or worker and continuously captures images as the work robot or worker moves or works in the field. The captured images are basically still images, but it is also possible to obtain still images from video recording. It is also possible to mount a camera on a UAV and capture images from there. In addition to cameras, it is also possible to collect 3D data (laser scan data) using LiDAR.

[0038] Here, we will explain the case where data is collected using a work robot. Figure 7 shows an example of a work robot. Work robot 200 is equipped with multiple cameras 201. Each camera is positioned so that it can capture as wide an area as possible. The posture and capture range of adjacent cameras are adjusted so that they capture as much overlapping area as possible. Furthermore, the relationship between the separation distance and posture of adjacent cameras is acquired in advance and is known so that they can be used as a stereo camera.

[0039] In the case of workers, multiple cameras are attached to their helmets and work vests, and the camera settings are the same as for the work robots.

[0040] 3. Hardware Configuration 3 is a block diagram of the agricultural produce information processing device 300. The agricultural produce information processing device 300 is configured using a PC (personal computer). The PC that constitutes the agricultural produce information processing device 300 includes a CPU, a storage device, a user interface, and a communication interface, and has the functions of a normal PC.

[0041] The agricultural product information processing device 300 includes an image data acquisition unit 301, an image recognition unit 302, a 3D model creation unit 303, an identification display detection unit 304, a work content identification unit 305, a work object identification unit 306, a first correspondence relationship identification unit 307, a layer data (update) creation unit 308, a data storage unit 309, a communication device 310, a positional relationship identification unit 311, a coordinate conversion unit 312, a real data acquisition unit 313, and a second correspondence relationship identification unit 314.

[0042] The data storage unit 309 and communication device 310 utilize hardware provided in the PC. An external storage device can also be used as the data storage unit. Communication is performed via a wireless LAN or a mobile phone communication line. Wired communication is also possible.

[0043] The other functional units are realized as software on the PC by installing application software on the PC.

[0044] Some or all of these functional units may be configured with dedicated hardware. Some or all of the processing of the agricultural produce information processing device 300 may also be performed by a processing server. In this case, some or all of the agricultural produce information processing device 300 is configured by the processing server.

[0045] The image data acquisition unit 301 acquires image data of the captured image required to create Layer 1, Layer 2, and Layer 3. For example, the image data is transmitted from the camera that captured the image using a wireless LAN, and is received by the communication device 310, and the received image data is acquired by the image data acquisition unit 301.

[0046] The image recognition unit 302 detects specific objects from the captured image. Detection targets include the entire crop, each part of the crop (trunk, branches, buds, clusters, fruit, leaves, etc.), various targets, supports and beams, gardening tools, building materials that make up agricultural greenhouses, etc. Detection is performed using known image recognition technology. Methods for detecting specific objects from the captured image include a method that compares them with a reference image and a method that uses AI image recognition technology that uses deep learning.

[0047] The 3D model creation unit 303 creates a three-dimensional model (3D model) of the subject based on the principle of stereophotography. The technology for creating 3D models is described in WO2011-70927, JP2012-230594A, and JP2014-15598A. It is also possible to create a 3D model using Lidar (laser scanning).

[0048] The identification display detection unit 304 detects the identification target. This technology is described in, for example, Japanese Patent Application Laid-Open No. 2021-139749. The work content identification unit 305 identifies the content of various work on the crop (for example, the work of shaping flower spikes) using a method described below. The work target identification unit 306 identifies the target of the above work (for example, the cluster that is the target of the work) using a method described below.

[0049] The first correspondence identification unit 307 identifies correspondences between images and between 3D models. The correspondences are identified using various matching techniques, such as template matching. Techniques for identifying correspondences are described in, for example, Japanese Patent Application Laid-Open Nos. 2013-186816, 2013-178656, WO2012-141235, 2014-35702, 2015-46128, and 2017-15598. AI image recognition technology can also be used to identify correspondences between images.

[0050] The layer data (update) creation unit 308 updates and creates data for Layer 1, Layer 2, and Layer 3. The details of this processing will be described later.

[0051] The data storage unit 309 stores various data necessary for the operation of the agricultural produce information processing device 300, operation programs, and data obtained as a result of the operations. Data for layers 1, 2, and 3 are also stored in the data storage unit 309. An external storage device may be used as part or all of the data storage unit 309.

[0052] The communication device 310 communicates with external devices. The communication is performed using, for example, a wireless LAN or a mobile phone line. Wired communication is also possible.

[0053] The positional relationship specification unit 311 specifies the correspondence (positional relationship) between the data in the absolute coordinate system of layer 1 and the data in the local coordinate system of layer 2. By specifying this correspondence, the data in the local coordinate system of layer 2 can be converted to the absolute coordinate system, and the data in layer 2 can be associated with the data in layer 1.

[0054] The coordinate conversion unit 312 performs coordinate conversion between the data of layer 1 and layer 2. For example, it converts the position data of layer 2 described in a local coordinate system into an absolute coordinate system. Because the data of layer 1 uses an absolute coordinate system, the above coordinate conversion allows the data of layer 2 to be handled in the same coordinate system (absolute coordinate system) as the data of layer 1.

[0055] The fruit data acquisition unit 313 acquires data about the fruit, in this case, the number of grapes in a bunch, the length of the stem, and other details of the bunch (color, etc.) Methods for acquiring fruit data include analyzing a 3D model based on a photographed image of the fruit, and analyzing the photographed image.

[0056] The second correspondence identification unit 314 identifies the correspondence between the object identified as the target of the work and the data in layer 2. For example, suppose a worker takes a picture of a bunch of grapes using a camera worn on his head while working on it. The second correspondence identification unit 314 performs a process to identify the correspondence between the bunch shown in the captured image and the data on the bunch managed in layer 2.

[0057] 4. Example of processing (Creating Layer 1 data) The following explanation will be given for a vineyard as the field. Note that the vineyard is just an example, and the principles are the same for other crops.

[0058] Figure 1 shows the state of the field at the time when the buds appear (usually around March in the Kanto region). At this stage, the plant is just a trunk and branches, with no leaves and, of course, no fruit.

[0059] An example of the processing procedure is shown in Figure 4. This processing is executed in the agricultural produce information processing device 300 of Figure 3. The program that executes this processing is stored in the data storage unit 309 and executed by the CPU provided in the agricultural produce information processing device 300. The program can also be stored in an appropriate storage medium or data storage server and downloaded from there for use. This is also the case in Figures 5 and 6.

[0060] First, image data is acquired at the stage shown in Fig. 1 (the stage before sprouts appear) (step S101). Here, a work robot (see Fig. 7) is used to acquire the image data. The work robot continuously takes pictures while moving using multiple cameras mounted on it.

[0061] The imaging range and direction of each camera are set so that the imaging ranges of adjacent cameras partially overlap. Furthermore, when a camera is moving, the imaging interval and the work robot's movement speed are adjusted so that the imaging range captured at one time partially overlaps with the imaging range at the next time. The image data of the images captured by each camera is recorded in association with the time of capture. Since there are no leaves at this stage, the image data can also be obtained from the air using a UAV.

[0062] When capturing images while the work robot is moving, multiple reference points are included in at least some of the captured images. Then, a 3D model of the subject is created using the SFM (Structure from Motion) principle (step S102). This technology is described in, for example, Japanese Patent Application Laid-Open No. 2013-186816, WO2011-70927, Japanese Patent Application Laid-Open No. 2012-230594, and Japanese Patent Application Laid-Open No. 2014-35702.

[0063] Below is a brief explanation of the principles of SFM. As a simple example, consider a stereo camera consisting of two cameras with a known baseline length (the distance between the two cameras). Using the stereo images taken by these stereo cameras, the relative positional relationship between the two cameras that make up the stereo camera and the object being photographed can be determined. This is called relative orientation.

[0064] In this example, the baseline length of the stereo cameras (the distance between the two cameras) is known, so a scale is applied to the above relative positional relationship. In this way, a 3D model of the object in the local coordinate system is obtained. Note that if there are two points in the captured image with a known distance between them, a scale is similarly applied to the above relative positional relationship, and a 3D model of the object in the local coordinate system is obtained.

[0065] This example is explained using a stereo camera, but if a scale is involved, SFM processing is possible with one camera as long as there are two or more viewpoints, and this processing can be performed with one camera. In this case, stereo images are obtained by using one camera to capture overlapping areas from two different viewpoints.

[0066] Now, consider the case where the positions of the two cameras are slightly shifted. Here, the two cameras are moved so that the new stereo image overlaps with the previous stereo image, but are slightly shifted. Then, the relative positional relationship between the two cameras and the photographed object at the new viewpoint (stereo camera position) is calculated using the method described above. This results in a 3D model (second 3D model) of the object as seen from a new viewpoint slightly shifted from the previous one. This 3D model is a 3D model as seen from a viewpoint slightly shifted from the previous one.

[0067] Because the first and second stereo images overlap, the first and second 3D models can be integrated by finding the correspondence between the images. By repeating this process for the third, fourth, and so on, a 3D model of the captured area is constructed.

[0068] If absolute coordinate points are given in the captured image at some stage, the 3D model is obtained in the absolute coordinate system. If absolute coordinate points are not given, the 3D model is constructed in the local coordinate system. This is the basic principle of SFM.

[0069] In this embodiment, an identification target is placed. In the above SFM, coordinate values ​​in the absolute coordinate system are assigned to the 3D model using the identification target placed at the reference point. In addition, by obtaining a 3D model of the subject on the absolute coordinate system, the position on the absolute coordinate system of an identification target that is not placed at the reference point can be determined.

[0070] Based on the above principle, a work robot patrols the field shown in Figure 1, and a 3D model of the field (vineyard) is created using SFM.

[0071] The update frequency of Layer 1 data is determined by the growth rate of the target crop's trunk. For crops with fast tree growth, more frequent updates are required to keep up with changes in the tree. On the other hand, for crops with slow trunk growth or trunks that barely change, Layer 1 data is updated frequently enough to keep up with changes in the trunk.

[0072] (Creating and updating data in Layer 2) Layer 2 data is created and updated based on photographs taken by work robots and workers patrolling the field and by the robots and workers performing various tasks. Layer 2 data acquisition also includes partial photography of the field. Therefore, as data acquisition may be performed locally, a local coordinate system is used as the coordinate system for the primary data. Of course, this does not exclude the acquisition of data in an absolute coordinate system.

[0073] In the case of grapes, the data in Layer 2 is the position of the cluster (flower spike). The position of this cluster is recognized as the position of the bud (the position of the base of the cluster axis). Note that the position of the cluster may change due to factors such as the growth of the branches. If a change in the cluster's position is detected, the cluster's position data is corrected.

[0074] An example of the processing procedure will be described below. The processing procedure is shown in Figure 5. Here, the sprouts emerge and then the buds or the buds are photographed as they grow into clusters, and the image data is acquired (step S201). Next, the position of the cluster is calculated using stereo images taken from different viewpoints (step S202).

[0075] The specific procedure of step S202 will be described. First, the sprouts and the bases of the bunches (the bases of the bunch stems) are detected from the stereoscopically photographed images. This process is performed by the image recognition unit 302. Next, the positions of the sprouts and the bases of the bunches are calculated using the principles of stereo photogrammetry.

[0076] In this case, the baseline lengths of the two cameras (three or more cameras are acceptable) that took the stereo photographs are known. Therefore, the shape and size of the triangle with the positions of the "root of the bud or bunch" - "first camera" - "second camera" as vertices are determined, and the position of the "root of the bud or bunch" in a local coordinate system based on the positions of the cameras at the time of photographing can be found. In this way, the position of the bunch in the local coordinate system can be found. This method is the very principle of forward intersection.

[0077] Next, a method for converting the position of the bunch obtained in step S202 into a position in the absolute coordinate system is obtained (step S203). By performing this coordinate conversion, the data in layer 2 can be associated with the data in layer 1. In this example, the data in layer 2 is managed and saved as data in the local coordinate system, and coordinate conversion is performed as necessary to link it with the data in layer 1.

[0078] The above coordinate transformation will be explained below. To perform this coordinate transformation, first, the position and orientation in the absolute coordinate system of each camera (stereo camera) that captured the stereo images used in step S202 are determined. Next, based on the position and orientation of the stereo camera, the position of the tuft in the absolute coordinate system is determined by the forward intersection method.

[0079] The position and orientation in the absolute coordinate system of each camera that captured the stereo images are calculated using the identification targets as control points. In this case, the stereo images used in step S202 are those that contain three or more identification targets. The positions of these three or more identification targets in the absolute coordinate system are known in the data of layer 1.

[0080] In this case, three or more points are determined in the stereo images in step S202 on the absolute coordinate system, and the positions and orientations of the two cameras that captured the stereo images in the absolute coordinate system can be determined by resection. By determining the positions and orientations of the cameras in the absolute coordinate system, the relationship between the local coordinate system and the absolute coordinate system used here can be determined, enabling coordinate conversion between the two. In other words, the coordinate data of layer 2 can be associated with the coordinate data of layer 1.

[0081] For example, let the positions of the stereo camera in the local coordinate system be (x1, y1, z1) and (x2, y2, z2). Let the positions of the two cameras that captured the stereo images in the absolute coordinate system obtained by the resection method be (X1, Y1, zZ) and (X2, Y2, Z2).

[0082] By comparing (x1, y1, z1) with (X1, Y1, Z1), (x2, y2, z2) with (X2, Y2, Z2), and (x3, y3, z3) with (X3, Y3, Z3), the correspondence between (xi, yi, zi) and (Xi, Yi, Zi) (i=1, 2, 3) can be determined, and the coordinate transformation formula can be found.

[0083] Photographing to obtain the image data that forms the basis of Layer 2 data is done repeatedly. Photographing is done by patrolling the field and performing various tasks in the field. By taking photographs more frequently, the frequency of updating the Layer 2 data can be increased. Photographing can also be done by a mobile robot. Of course, it is also possible to install a fixed camera and take photographs.

[0084] It is also possible to perform the processing of step S202 using image recognition technology. In this case, the captured image used to create Layer 1 and the captured image used to create Layer 2 must both contain images of the same subject.

[0085] In this case, the correspondence between the photographed image used to create Layer 1 and the created image used to create Layer 2 is searched for to find the corresponding image of the bunch. Since the position of the object in the image related to Layer 1 is known, if there is a corresponding image related to Layer 2, the position of the object in the absolute coordinate system shown in that image can be calculated from the information in Layer 1.

[0086] The processing in step S202 can also be performed using a 3D model. This method is described below. In this case, a 3D model of the object to be photographed is created using SFM based on the photography performed to acquire data for Layer 2. If no reference point is used, this 3D model is described in a local coordinate system.

[0087] On the other hand, Layer 1 contains data of a 3D model of the field. Therefore, we identify the correspondence between the two and determine which part of the 3D model in Layer 1 corresponds to the 3D model obtained by SFM.

[0088] Here, the 3D model of layer 1 is described in an absolute coordinate system. Therefore, based on the above correspondence, the absolute coordinate values ​​of the 3D model created by the SFM can be obtained. In other words, the 3D model can be described in an absolute coordinate system. In addition, the relationship with the data of layer 1 becomes clear.

[0089] Furthermore, when updating the data in Layer 2, it is necessary to identify the correspondence between the previously obtained data in Layer 2 and the currently obtained data in Layer 2. For example, if there is no change in the position of the bunch, it can be determined that it is in the same position and the correspondence can be determined.

[0090] If a different position of the tuft is calculated, the following steps are taken: 1. The tuft closest to the position at the time of the previous data acquisition, but whose position could not be identified this time, is recognized as the corresponding tuft; 2. The correspondence of the 3D model is used to determine the identity of the object.

[0091] (Creating and updating data in Layer 3) Layer 3 data is created and updated based on the patrols of work robots and workers in the field and on photographs taken by the work robots and workers during various tasks. Therefore, data for Layer 2 and Layer 3 may be acquired simultaneously.

[0092] In this example, the size of the grapes in a bunch, the number of grapes, the stem length, work information, etc. are obtained from the captured image, and this data is associated and managed as layer 3 data.

[0093] Workers wear multiple cameras on their heads or work vests, some of which are positioned to capture what is happening at their hands while they are working. For example, workers wear stereo cameras on their heads. These stereo cameras are attached to the worker's head so that they face in the same direction.

[0094] An example of the processing procedure is shown in Figure 6. First, captured image data is obtained (step S301). Next, a 3D model of the captured object is created based on the acquired image data (step S302). Next, actual data, in this case, grain size, grain number, and axial length, are obtained based on the 3D model.

[0095] Next, work information is obtained (step S304). Work information is obtained as follows. Here, an example of shaping will be explained. It is assumed that the shaping is performed by a worker. In this case, a work identification marker such as a barcode display is placed on the worker's hand or on the tool used for the work, in order to identify the work content and the work target from the captured image.

[0096] For example, if the work content is "shaping grape flower spikes" and the work object is "a bunch of grapes with flower spikes formed," a code display that can identify this is displayed on the work identification marker.

[0097] For example, a worker wears a wristband with a barcode that serves as a task identification marker. Alternatively, the barcode is displayed on the tool used for the task. By detecting this task identification marker in the captured image, the task content and the task target can be identified.

[0098] For example, if the work involves gibberellin treatment, the barcode (work identification marker) is displayed on a container containing a chemical solution for gibberellin treatment (a bunch of grapes is immersed in this container).By performing image recognition on this display, it is possible to recognize that the bunch of grapes shown in the photograph has been treated with gibberellin.

[0099] Next, the work object is identified in the image in which the work identification marker was detected (step S305). That is, individual objects such as branches and bunches are recognized in the captured image by AI image recognition, and which of these objects is the work object read from the work identification marker is identified.

[0100] For example, the processing of step S305 will be described using a bunch of grapes as an example. First, assume that a bunch is identified as the object of work in step S304. In this case, an image of the bunch is detected from the target image. This detection is performed using AI image recognition. The detected bunch is recognized as the object of work identified in step S304.

[0101] Next, the relationship between the bunch for which the work has been specified and the content of Layer 2 is determined (Step S306). Here, the bunch for which the work content has been specified is identified in the image, but at this stage it is not linked to the data of Layer 2. Therefore, it is determined which bunch in Data 2 the bunch that is the target of the work corresponds to. Through this process, the correspondence between the object identified as the target of the work and the data of Layer 2 is determined.

[0102] Specifically, the process is as follows: First, using the method described for Layer 2, the position of the bunch in the absolute coordinate system is calculated based on the image of the work being performed. This position information is compared with the data in Layer 2 to identify the correspondence between the bunch and the data in Layer 2. In this way, the relationship between the work content and the data in Layer 2 is identified. Then, the work content and its date and time (date and time of the photo) are recorded in Layer 3. In this way, the relationship between Layer 2 and Layer 3 is identified.

[0103] Another method for identifying the correspondence between the object identified as the work target and the data in Layer 2 is to find and use the correspondence between the captured images. In this case, the image data stored in Layer 2 is compared with the image data of the image captured during the work to search for corresponding image data. By finding the corresponding image, the correspondence between the object identified as the work target and the data in Layer 2 can be identified.

[0104] A 3D model can also be used as a method for identifying the correspondence between the object identified as the target of the work and the data in layer 2. In this case, the correspondence between the 3D model of the bunch in layer 2 and the 3D model of the bunch obtained from the image taken during the work is found, thereby identifying the correspondence between the bunch identified as the target of the work and the bunch in the data in layer 2.

[0105] In this way, data in layer 3 is created or updated. Each time various operations are performed, data in layer 3 associated with data in layer 2 is created or updated using the method described above.

[0106] (Conclusion) The data in Layer 1 and Layer 2 are associated with each other through coordinate transformation. As mentioned above, the data in Layer 3 is associated with the data in Layer 2. This realizes the data structure shown in Figure 2.

[0107] (Other ways to get work information) The following method can also be used to acquire work information: For example, the equipment used for work has a work content notification switch, and the worker turns this switch ON when starting work and OFF when finishing.

[0108] While this switch is ON, a signal indicating that the task is being performed is sent to the agricultural produce information processing device 300 via wireless LAN. Upon receiving this signal, the agricultural produce information processing device 300 stores the start and end times of the reception. This allows the agricultural produce information processing device 300 to acquire the content and target of the task, as well as the start and end times of the task.

[0109] Then, the time of the image capturing the work is compared with the start time and end time of the work, thereby identifying the image capturing the work.

[0110] It is also possible for the worker to carry a work notification device equipped with the switch. In this case, an identification signal that identifies the work content is sent to the agricultural produce information processing device 300 while the worker is working. In the case of a work robot, the work robot recognizes the work content, and therefore sends a notification signal to the agricultural produce information processing device 300 indicating the work content and that the work is currently being performed.

[0111] When multiple workers are working simultaneously in a field, the procedure is as follows. First, the IDs of the cameras and alarm devices carried by each worker are obtained in advance. The image data captured by the camera is recorded in association with the ID assigned to the camera. An alarm signal related to the work content is also transmitted from the alarm device along with the ID assigned to the alarm device. This allows the captured images related to each worker and the alarm signal informing them of the work content to be identified and the two are associated with each other.

[0112] (others) In the above explanation, an example was given in which images captured by a camera are used as the primary data for Layer 1 and Layer 2. LiDAR can also be used as the primary data. In this case, it is necessary to use identification targets and work identification markers that can be identified by LiDAR. It is also possible to use both image data and LiDAR data. [Explanation of symbols]

[0113] 100...field, 110...main trunk of grapevine, 120...main trunk of grapevine, 130...main trunk of grapevine, 140...main trunk of grapevine, 150...identification target, 160...reference target, 200...working robot, 210...multiple cameras.

Claims

1. a storage unit that stores the position of a crop in a field as information on layer 1, the position of a fruit of the crop as information on layer 2, and the content of work performed on the fruit as information on layer 3; a correspondence relationship specifying unit that specifies a correspondence relationship between the information on the layer 2 and the information on the layer 3 based on an image of the work; Equipped with The layer 2 includes image data of the captured image of the fruit used to identify the position of the fruit, The agricultural product information processing device identifies the correspondence based on objects that appear in common in the photographed image included in layer 2 and the image of the work.

2. An identification target is disposed near the object; The agricultural product information processing apparatus according to claim 1 , wherein the correspondence is determined based on an image of the identification target that appears in both the photographed image included in the layer 2 and the image of the work.

3. A memory unit that stores the position of a crop in a field as information on layer 1, the position of the fruit of said crop as information on layer 2, and the details of the work performed on said fruit as information on layer 3; a correspondence relationship specifying unit that specifies a correspondence relationship between the information on the layer 2 and the information on the layer 3 based on an image of the work; Equipped with The layer 2 includes a photographed image of the field, The identification of the correspondence relationship is An agricultural product information processing device that recognizes the fruit captured in the image of the work from the captured images of the field included in layer 2.

4. A memory unit that stores the position of a crop in a field as information on layer 1, the position of the fruit of said crop as information on layer 2, and the details of the work performed on said fruit as information on layer 3; a correspondence relationship specifying unit that specifies a correspondence relationship between the information on the layer 2 and the information on the layer 3 based on an image of the work; Equipped with Layer 2 contains data of the real 3D model, The agricultural product information processing device identifies the correspondence based on the correspondence between the 3D model data included in layer 2 and a 3D model created based on the image of the work.

5. The work-related means is provided with an identification display that displays the content of the work, The agricultural product information processing device according to any one of claims 1 to 4, wherein the content of the work is identified based on an image of the identification mark.

6. A memory unit that stores the position of a crop in a field as information on layer 1, the position of the fruit of said crop as information on layer 2, and the details of the work performed on said fruit as information on layer 3; a correspondence relationship specifying unit that specifies a correspondence relationship between the information on the layer 2 and the information on the layer 3 based on an image of the work; Equipped with the layer 1 includes image data for identifying the position of the crop in the field; Layer 2 includes image data for identifying the location of the fruit; The position information of the layer 1 is described in an absolute coordinate system, The position information of the layer 2 is described in a local coordinate system, The agricultural produce information processing device includes a positional relationship specifying unit that determines the relationship between the absolute coordinate system and the local coordinate system based on the corresponding relationship between the image data of the layer 1 and the image data of the layer 2.

7. The position of the crop in the field is stored as Layer 1 information, The position of the fruit of the crop is stored as information of layer 2, The content of the work performed on the object is stored as layer 3 information, Identifying the correspondence between the information on layer 2 and the information on layer 3 based on an image of the work; The layer 2 includes image data of the captured image of the fruit used to identify the position of the fruit, A computer-implemented agricultural product information processing method in which the correspondence is identified based on objects that appear in common in the photographed image included in layer 2 and the image of the work.

8. A program to be read and executed by a computer, Computer a storage unit that stores the position of a crop in a field as information on layer 1, the position of a fruit of the crop as information on layer 2, and the content of work performed on the fruit as information on layer 3; a correspondence relationship specifying unit that specifies a correspondence relationship between the information on the layer 2 and the information on the layer 3 based on an image of the work; and run it, The layer 2 includes image data of the captured image of the fruit used to identify the position of the fruit, A program for processing agricultural product information in which the correspondence is identified based on objects that appear in common in the photographed image included in layer 2 and the image of the work.

Images