Information processing device, information processing method, and program
The information processing device accurately calculates agricultural products by identifying and processing overlapping and non-overlapping image regions, addressing the challenges of omission and duplication in existing methods, ensuring a comprehensive and precise count of crops.
Patent Information
- Application Number
- PCT/JP2024/017326
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-05-10
- Publication Date
- 2024-11-28
AI Technical Summary
Existing methods for calculating the number of agricultural products from multiple images often result in inaccuracies due to omission or duplication, lacking effective techniques to accurately count crops without missing images or double counting.
An information processing device and method that acquires images from multiple imaging devices, identifies overlapping areas, calculates the number of agricultural products in overlapping and non-overlapping regions, and combines this data to accurately determine the total number of crops in a specific area, avoiding double counting and ensuring comprehensive counting.
The solution enables precise calculation of agricultural products by distinguishing between overlapping and non-overlapping areas, preventing omissions and duplications, thus providing a high-accuracy count from multiple images.
Smart Images

Figure JP2024017326_28112024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, and program
[0001] The present invention relates to an information processing device, an information processing method, and a program.
[0002] There are known techniques for calculating the number of agricultural crops grown in a field. For example, Patent Document 1 below discloses a system in which a mobile work vehicle travels between a plurality of cultivation beds while capturing images of the fruits of the plants with an imaging device, processes the captured images to detect the fruits, and calculates the total number of fruits.
[0003] JP 2019-17396 A
[0004] Here, when calculating the number of crops in a specific area of a field, it is necessary to capture multiple images using an imaging device. However, the technique disclosed in Patent Document 1 leaves room for technical improvement in accurately calculating the number of crops from the multiple images without omissions or overlaps.
[0005] Therefore, in order to solve the above problem, the present invention aims to provide an information processing device, an information processing method, and a program that can accurately calculate the number of crops from multiple images of the crops.
[0006] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires a plurality of images including one or more crops, each captured by one or more imaging devices in a specific area of a field; an identification unit that identifies an overlapping area between each area of the plurality of images; a first calculation unit that calculates a first number of crops in the overlapping area; a second calculation unit that calculates a second number of crops in a non-overlapping area other than the overlapping area in each area of the plurality of images; and a third calculation unit that calculates a third number of crops in the specific area based on the first number and the second number.
[0007] Another aspect of the present disclosure relates to a method in which a computer acquires a plurality of images including one or more crops, each captured by one or more imaging devices in a specific area of a field, identifies overlapping areas between each area of the plurality of images, calculates a first number of crops in the overlapping areas, calculates a second number of crops in non-overlapping areas other than the overlapping areas in each area of the plurality of images, and calculates a third number of crops in the specific area based on the first and second numbers.
[0008] A program according to another aspect of the present disclosure causes a computer to execute an acquisition function for acquiring a plurality of images including one or more crops, each imaged by one or more imaging devices in a specific area of a field; an identification function for identifying overlapping areas between each area of the plurality of images; a first calculation function for calculating a first number of crops in the overlapping areas; a second calculation function for calculating a second number of crops in non-overlapping areas other than the overlapping areas in each area of the plurality of images; and a third calculation function for calculating a third number of crops in the specific area based on the first and second numbers.
[0009] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that can accurately calculate the number of crops from a plurality of images of the crops.
[0010] FIG. 1 is a diagram showing an overview of a smart agriculture system according to the present embodiment. FIG. 2 is a conceptual diagram more specifically showing the field of view of a step surface S in the farm field shown in FIG. 1. FIG. 3 is a schematic diagram showing an example of an imaging method using an imaging device according to the present embodiment. FIG. 4 is a schematic diagram showing an example of a method for identifying overlapping areas in a plurality of images and a method for coordinate processing according to the present embodiment. FIG. 5 is a schematic diagram showing an example of a method for identifying overlapping areas in a plurality of images according to the present embodiment. FIG. 6 is a diagram showing an example of the functional configuration of an information processing device according to the present embodiment. FIG. 7 is a diagram showing an example of the operation of the information processing device according to the present embodiment. FIG. 8 is a diagram showing an example of the hardware configuration of the information processing device according to the present embodiment. FIG. 9 is a diagram showing an example of a method for combining images according to the present embodiment.
[0011] Hereinafter, an embodiment of the present invention (hereinafter referred to as "the present embodiment") will be described in detail with reference to the drawings as necessary. However, the present invention is not limited to this embodiment, and various modifications are possible without departing from the spirit of the present invention. In the drawings, the same elements are given the same reference numerals, and redundant explanations will be omitted. Furthermore, positional relationships such as up, down, left, and right are based on the positional relationships shown in the drawings unless otherwise specified. Furthermore, the dimensional ratios of the drawings are not limited to those shown in the drawings.
[0012] <1. System> Fig. 1 is a diagram showing an overview of a smart agriculture system 1 according to an embodiment of the present invention. As shown in Fig. 1, the smart agriculture system 1 includes an information processing device 200 and a measurement device 300. The measurement device 300 may be, for example, various sensors such as a mobile sensor 310 and / or a fixed sensor 320. The information processing device 200 may also be a terminal device 200a and / or a server device 200b. The information processing device 200 and the measurement device 300 may be connected via a network N.
[0013] In the smart agriculture system 1, various sensors such as the fixed sensor 320 and the mobile sensor 310 are used to acquire images (hereinafter simply referred to as "images") of agricultural products in a field. Next, the acquired images are stored by the information processing device 200 (for example, the server device 200b). Then, the information processing device 200 (for example, the terminal device 200a) calculates the number of agricultural products based on the images of the agricultural products. The agricultural products may be, for example, fruits grown on trellises, such as grapes, kiwis, or pears. Furthermore, the agricultural products may specifically be fruit flowers or berries. Furthermore, as another example, the agricultural products may be fruit vegetables, such as tomatoes, eggplants, or peppers.
[0014] [Information Processing Device] Information processing device 200 may be a terminal device such as a laptop or a smartphone, or a server device such as a storage server. By executing a predetermined program, information processing device 200 connects to measurement device 300 to send and receive various data, stores various received data, outputs various data to a user, and accepts instructions from the user for measurement device 300, etc., by voice input, operation input, etc.
[0015] The server device 200b may be a cloud server or an edge server. The edge server may be installed in or near the field and perform data processing and analysis. This reduces communication delays and distributes the processing load because data is processed on the edge server side without being sent to the cloud server. For example, the server device 200b, which is an edge server, may collect images of crops from the measuring device 300 and store the collected images in the server device 200b, which is a cloud server. The terminal device 200a may then acquire these stored images and calculate the number of crops. Note that instead of the edge server, a terminal device 200a having the same functions as the edge server may be provided.
[0016] [Measuring Device] The measuring device 300 measures the crops and the surrounding environment in the field, generates data indicating the results of this measurement (hereinafter also referred to as "measurement data"), and transmits it to the information processing device 200 or the like. The measuring device 300 is not particularly limited, and may be, for example, a device equipped with various sensors mounted at any position in the field, a drone equipped with various sensors that flies within the field or an unmanned aerial vehicle that can move within the field (hereinafter also referred to as a "mobile robot"), or a human-operated terminal such as a smartphone equipped with various sensors, a handheld computing device, or a wearable device. Here, the sensor is not particularly limited as long as it includes an image sensor, and may further include an environmental sensor.
[0017] The image sensor is one aspect of an imaging device. There are no particular limitations on the image sensor as long as it is a sensor capable of capturing still images or video. The image sensor may be, for example, a fixed camera installed in a field, a camera on a terminal such as a wearable device, or a camera installed on a mobile measuring device 300. The image sensor may be configured to continuously generate images in frame units, for example, and capture time-series images (e.g., video images).
[0018] The environmental sensor is a sensor for measuring environmental information in the field of the crops 520. The environmental sensor is not particularly limited, but examples thereof include a weather sensor, a soil sensor, and a gas sensor.
[0019] The mobile sensor 310 may be, for example, a drone traveling within a field equipped with various sensors. This drone is one type of mobile body. The sensors may be, for example, one or more image sensors for capturing images of agricultural crops and / or a GPS receiving antenna for measuring location information (e.g., latitude and longitude). While the present embodiment describes an example in which multiple image sensors are installed in the mobile sensor 310, the number of image sensors according to the present embodiment is not limited to this. The number of image sensors according to the present embodiment may be, for example, one, three, or more. The fixed sensor 320 may be, for example, a fixed camera.
[0020] [Network] The network N is configured by a wireless network or a wired network. Examples of the network include a mobile phone network, a PHS (Personal Handy-phone System) network, a wireless LAN (Local Area Network, including communication conforming to IEEE 802.11 (so-called Wi / Fi (registered trademark))), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), WiMax (registered trademark), infrared communication, visible light communication, Bluetooth (registered trademark), a wired LAN, a telephone line, a power line communication network, a network conforming to IEEE 1394, etc.
[0021] Figure 2 is a conceptual diagram more specifically illustrating the field of view of the step surface S in the field shown in Figure 1. As shown in Figure 2, agricultural crops with fruit at the ends of tree branches are shown, and shelves 530 for supporting the branches are installed below the branches. Here, the mobile sensor 310 may travel under the branches and shelves 530 while pointing the sensor substantially upward relative to the ground to capture images of the agricultural crops.
[0022] In the smart agriculture system 1, for example, multiple imaging devices installed in a mobile sensor 310 capture images of crops grown in a specific area, and the number of crops in the specific area is calculated from the captured images. The specific area is an area of the field for which the number of crops is to be calculated, and may be, for example, one or more plots among multiple plots in the field, or an area per unit area. When each of the multiple imaging devices captures images of the crops grown in the specific area in this way, multiple images are captured with portions of each of the multiple images overlapping to prevent any omissions. The number of crops can then be calculated based on the overlapping areas (hereinafter also referred to as "overlapping areas") and non-overlapping areas (hereinafter also referred to as "non-overlapping areas").
[0023] 2. Overview of Imaging and Identifying Overlapping Areas An example of the process of imaging and identifying overlapping areas using the mobile sensor 310 will be described with reference to FIGS. 3 to 5 . In this example, an example will be described in which a left-side image sensor (hereinafter also referred to as the "left image sensor") 311a and a right-side image sensor (hereinafter also referred to as the "right image sensor") 311b are arranged in parallel with each other. The left image sensor 311a and the right image sensor 311b are aspects of a first imaging device and a second imaging device. The left image sensor 311a and the right image sensor 311b, for example, capture images of crops grown on shelves 530 with the imaging direction facing upward relative to the ground. The left image sensor 311a and the right image sensor 311b will be collectively referred to as the "image sensor 311" when there is no particular need to distinguish between them.
[0024] 2-1. Imaging FIG. 3 is a schematic diagram showing the mobile sensor 310 traveling in a specific area of a field, as viewed from above. As shown in FIG. 3, the shelves 530 have support columns arranged at intervals of a width W (e.g., 4 to 5 m). The mobile sensor 310 travels along the area between the columns L, which serves as a travel path, and travels near the center line C of the travel path. The mobile sensor 310 is equipped with a left image sensor 311a and a right image sensor 311b arranged in the left and right directions so as to capture images of the respective left and right areas. The left and right directions refer to directions approximately perpendicular to the travel direction of the mobile sensor 310 (the longitudinal direction of the area between the columns). For example, after traveling from one end of a travel path to the other, the mobile sensor 310 turns around and travels in the opposite direction from the previous travel path, from one end of the longitudinal direction to the other of the adjacent travel path. The left image sensor 311a and the right image sensor 311b may be set to have their respective image capturing ranges overlap, for example, to avoid missing images of agricultural produce, etc. The mobile sensor 310 may be set to have their respective image capturing ranges overlap when capturing images of two adjacent travel lanes, for example, to avoid missing images of agricultural produce, etc.
[0025] 4A is a schematic diagram showing the overlap of the image capture ranges of the left image sensor 311a and the right image sensor 311b. An image P1 on the left side (hereinafter also referred to as the "left image") captured by the left image sensor 311a and an image P2 on the right side (hereinafter also referred to as the "right image") captured by the right image sensor 311b in the direction of travel contain overlapping areas of the same crops, etc. The information processing device 200 acquires the left image P1 and the right image P2 and identifies an overlapping area A3 in each image as an overlapping area, while identifying non-overlapping areas A1 and A2 as non-overlapping areas.
[0026] 2-3. Coordinate Transformation FIG. 4B is a schematic diagram illustrating an example of a method for image coordinate processing using the mobile sensor 310. The information processing device 200 may perform coordinate transformation based on the relative positions of the left image sensor 311a and the right image sensor 311b to align one coordinate system with the coordinate system S1 of the left image P1 or the coordinate system S2 of the right image P2. Specifically, the information processing device 200 may transform the center coordinates of the crops in the coordinate system S1 detected in the left image P1 into the coordinate system S2. The information processing device 200 then integrates the crops detected in the left image P1 and the right image P2 based on the center coordinates of the crops in the coordinate system S2. As an example of integration, the information processing device 200 may identify multiple crops whose center coordinates are within a predetermined range, in other words, whose center coordinates are within a threshold, as the same crop.
[0027] 2-4. Identifying Overlapping Areas Due to Turning Travel> FIG. 5A is a schematic diagram showing overlapping image capture ranges before and after a turnaround when the mobile sensor 310 turns back onto an adjacent travel lane. The image capture range of the right image sensor on travel lane A is from the center line C1 to the right edge + α in the short direction of travel lane A. The image capture range of the right image sensor on travel lane B is from the center line C2 to the right edge + α in the short direction of travel lane B. The right images captured on travel lane A and B may overlap by α (the area surrounded by the dashed ellipse). The value of α may be set, for example, so that the width W1 of the travel lane falls within the image capture range even if the mobile sensor 310 is displaced from the center or tilted left or right while traveling. Furthermore, if watering hoses are installed on shelves 530 along a row of support posts, information processing device 200 may detect these watering hoses as references (hereinafter also referred to as "marker objects") in each of the left and right images, and identify overlapping areas between the two right images of paths A and B. Specifically, information processing device 200 may identify, for each of the two right images of paths A and B, the areas outside the detected watering hoses as overlapping areas and the areas inside the detected watering hoses as non-overlapping areas. Furthermore, when combining the two right images of paths A and B or identifying overlapping areas, information processing device 200 may, as preprocessing, rotate the right image of path B by 180 degrees to align it with the orientation of the right image of path A, for example.
[0028] A marker object may be, for example, any object that can serve as a reference point among those installed for growing crops, such as a structure such as a cultivation shelf or part of a building such as an agricultural greenhouse (for example, the framework of the structure such as supports and joints), a sign installed in the field, or a string rope used for guiding crops.
[0029] 2-5. Identifying Overlapping Areas in the Traveling Direction FIG. 5B is a schematic diagram illustrating overlapping areas of captured images in a time series due to the travel of the mobile sensor 310. In time-series images, such as frame images of a video, the captured areas may overlap between adjacent images in the time series, for example, between a left image P1 and a right image P2 at a specific time point and a left image P1' and a right image P2' taken t seconds before the specific time point (e.g., the area surrounded by a dashed ellipse). The information processing device 200 may, for example, acquire position information associated with the measurement time of the mobile sensor 310 and calculate the amount of movement over t seconds based on this position information. Based on this amount of movement, the information processing device 200 may identify overlapping areas between the left image P1 and the right image P2 and the left image P1' and the right image P2'.
[0030] With the above configuration, the smart agriculture system 1 can calculate the number of crops in a specific area based on the number of crops calculated in each of the overlapping and non-overlapping areas. Therefore, the number of crops can be calculated taking into account overlaps between multiple images. Therefore, even if multiple images are captured with overlapping imaging ranges to ensure no crops are missed, double counting can be avoided. Therefore, the number of crops can be calculated with high accuracy from multiple images of the crops.
[0031] 3. Functional Configuration The functional configuration of the information processing device 200 according to this embodiment will be described with reference to Fig. 6. As shown in Fig. 6, the information processing device 200 includes a control unit 210, a communication unit 220, an input / output unit 230, and a storage unit 240.
[0032] [Control Unit] The control unit 210 includes an acquisition unit 211, an identification unit 212, and a calculation unit 216. The control unit 210 may also include a coordinate processing unit 213, a synthesis unit 214, and / or a detection unit 215, for example.
[0033] [Acquisition Unit] The acquisition unit 211 acquires various data from the measurement device 300, other information processing devices 200, the storage unit 240, etc. The acquisition unit 211 may store the acquired various data in the storage unit 240. The acquisition unit 211 acquires multiple images including one or more crops captured by one or more image sensors 311 in a specific area of the field. Specifically, the acquisition unit 211 receives data indicating the images including the crops (hereinafter also referred to as "image data") from the image sensors 311 or other information processing devices 200 via the communication unit 220. The image data is one type of measurement data and may include (or be associated with) information such as date and time information and / or location information when the image was captured. Furthermore, if the image data includes the left image P1 and the right image P2 described in the examples of FIGS. 3 to 5, the image data may also include association between these images.
[0034] The acquisition unit 211 may calculate the amount of movement of the mobile sensor 310 during a predetermined period such as an image capturing time based on the acquired position information. The acquisition unit 211 may store information indicating the calculated amount of movement in the storage unit 240.
[0035] The acquisition unit 211 may acquire, for example, an image including one or more agricultural products, in which one or more image sensors 311 capture images of the agricultural products along a predetermined direction (for example, the traveling direction of the mobile sensor 310).
[0036] For example, when the image sensor 311 captures images of agricultural products while moving, the acquisition unit 211 may acquire the amount of movement of the image sensor from the image sensor 311 or the like (or by referring to the storage unit 240). The amount of movement of the image sensor 311 may be, for example, the amount of movement of the mobile sensor 310 on which the image sensor 311 is installed.
[0037] [Identification Unit] The identification unit 212 identifies overlapping areas between respective areas of a plurality of images. Examples of overlapping areas that the identification unit 212 identifies include (1) overlapping caused by multiple adjacent image sensors as described in the example of 2-2 above, (2) overlapping caused by turning around while traveling as described in the example of 2-4 above, and (3) overlapping in the traveling direction as described in the example of 2-5 above. Regarding the overlapping in (1) above, for example, a combination of a left image P1 and a right image P2 may be identified as an overlapping area. Regarding the overlapping in (1) and (2) above, the identification unit 212 may identify a combination of a plurality of images as an overlapping area based on, for example, location information and / or date and time information included in the image data.
[0038] For example, the identification unit 212 may calculate the similarity for each unit region of the left image P1 and the right image P2, and identify each unit region for which the calculated similarity exceeds a threshold as an overlapping region. Furthermore, the identification unit 212 may calculate, as the similarity for each unit region of the left image P1 and the right image P2, the similarity for the feature amounts related to the shape of each of the crops detected by the detection unit 215 and the similarity for the feature amounts related to the arrangement of each of the crops. The identification unit 212 may calculate an overall similarity based on the respective calculated results. In this case, the identification unit 212 may perform the calculation by relatively weighting the feature amounts related to the arrangement of the crops more heavily than the feature amounts related to the shape, so as to prioritize the arrangement over the shape of the crops.
[0039] When the one or more image sensors 311 include a first image sensor (e.g., left image sensor 311a) and a second image sensor (e.g., right image sensor 311b), the identification unit 212 may identify, as the overlapping area, an area where an area of an image captured by the first image sensor (e.g., left image P1) overlaps an area of an image captured by the second image sensor (e.g., right image P2). With this configuration, even if two or more image sensors 311 capture images with overlapping imaging ranges to avoid missing images, it is possible to avoid redundantly calculating the number of crops.
[0040] For example, the identification unit 212 may identify an overlapping region as a region formed by overlapping edges of multiple images captured in adjacent rows of multiple rows (e.g., travel paths A and B) running in a predetermined direction. With this configuration, when capturing images of crops along a predetermined direction, even if the images captured in adjacent rows of multiple rows formed in the predetermined direction overlap, the number of crops can be calculated taking this overlap into account.
[0041] In the case of a plurality of images captured by the image sensor 311 in a time series (e.g., a video), the identification unit 212 may identify, for example, an overlapping area between adjacent images in the time series (in other words, between frames) as an overlapping area. With this configuration, the number of crops can be calculated based on the overlapping area in the time series images such as a video.
[0042] The identifying unit 212 may identify the overlapping area based on, for example, the amount of movement of the image sensor 311 acquired by the acquiring unit 211. With this configuration, even when the image sensor 311 captures images of agricultural products while moving, it is possible to identify the overlapping area in time-series images, etc., based on this movement.
[0043] [Coordinate Processing Unit] When the one or more image sensors 311 include a first image sensor and a second image sensor, the coordinate processing unit 213 aligns the coordinate system of the first image captured by the first image sensor with the coordinate system of the second image captured by the second image sensor based on the relative positions between the first image sensor and the second image sensor. The coordinate processing unit 213 may perform coordinate conversion, for example, to align the coordinate system of either the first image or the second image with the coordinate system of the other image.
[0044] The coordinate processing unit 213 includes an integration unit 213a, which integrates the crops in the overlapping area based on the central coordinates of the crops included in the first image and the second image, whose coordinate systems are aligned.
[0045] [Synthesis Unit] The synthesis unit 214 generates a composite image by synthesizing two or more images selected from a plurality of images that are the target of a specific region. For example, the synthesis unit 214 may generate a composite image by synthesizing a plurality of images based on an overlapping region identified by the identification unit 212. Specifically, the synthesis unit 214 may generate a composite image so that the overlapping portions become the overlapping portions when the identified overlapping region is synthesized. Note that when the synthesis process by the synthesis unit 214 and the coordinate conversion process by the coordinate processing unit 213 are performed in combination, the synthesis process may be performed after the coordinate conversion process, for example.
[0046] [Detection Unit] The detection unit 215 detects one or more agricultural products included in the image acquired by the acquisition unit 211 using image processing techniques such as machine learning, deep learning, or pattern recognition. During detection, the detection unit 215 may, for example, identify information such as the size, color, shape, or position within the image of each of the one or more agricultural products (hereinafter, these may be collectively referred to as "crop attribute information"). Furthermore, the detection unit 215 may, for example, assign identification information for identifying each of the one or more detected agricultural products, and store this identification information, the crop attribute information, date and time information and / or location information, and other measurement data included in the image data in association with each other in the storage unit 240.
[0047] For example, the detection unit 215 may detect the crops before or after the overlapping areas are identified by the identification unit 212. In the latter case, the detection unit 215 may detect the crops for each overlapping area and / or each non-overlapping area, for example.
[0048] For example, when detecting agricultural products, the detection unit 215 may determine whether the detected agricultural products are incomplete (or complete). An incomplete agricultural product may be, for example, a product that is not completely contained in an area of an image to be detected. Specifically, the detection unit 215 may determine that a crop that is partially missing due to a boundary between multiple areas or an edge of an area is not completely contained in an area. The detection unit 215 may associate information indicating the result of the determination of whether the crop is incomplete with each of the one or more detected agricultural products.
[0049] When the multiple images acquired by the acquisition unit 211 are time-series images captured by the image sensor 311, the detection unit 215 may, for example, detect one or more agricultural crops in each of the time-series images.
[0050] The detection unit 215 may include, for example, a tracking unit 215a. The tracking unit 215a tracks each of the one or more detected crops between adjacent images in the time series.
[0051] The tracking unit 215 a may track each of one or more crops, for example, further based on the result of interpolation by the interpolation unit 215 b. With this configuration, even if the tracked crop is hidden by leaves or the like and cannot be detected, it is possible to continue tracking by interpolation.
[0052] The detection unit 215 may include, for example, an interpolation unit 215b. The interpolation unit 215b interpolates crops that have not been detected by the detection unit 215 between adjacent images in the time series based on the amount of movement of the image sensor 311. Specifically, the interpolation unit 215b may predict the position in the image of each of one or more crops t seconds from now based on the amount of movement of the image sensor 311 over t seconds, and interpolate the undetected crops based on the result of this prediction.
[0053] [Calculation Unit] The calculation unit 216 calculates the number of crops in the field. Furthermore, for example, when there are multiple regions to be calculated, the calculation unit 216 may calculate a statistical value of the crops in the field as the calculation of the number of crops. The statistical value may be, for example, the total number of crops, the average value, the median value, the maximum value, the minimum value, or the mode. The calculation unit 216 may store count information indicating the calculation result in the storage unit 240 in association with the measurement data.
[0054] The calculation unit 216 includes a first calculation unit 216a. The first calculation unit 216a calculates the number of crops in the overlapping area identified by the identification unit 212 (hereinafter also referred to as the “first number”).
[0055] For example, the first calculation unit 216a may calculate the number of crops in each of the regions of the multiple images that correspond to the overlapping region, and use the number of crops in the region with the largest number as the first number as a result of this calculation. Specifically, the first calculation unit 216a may use the largest number of crops from the calculated numbers of crops in each of the regions of the multiple images that correspond to the overlapping region.
[0056] According to the above configuration, even if crops are not detected in some images corresponding to overlapping areas of multiple images, if they are detected in other images, the number calculated in these other images is used, so the number of crops can be calculated accurately without missing any counts.
[0057] For example, the first calculation unit 216a may calculate, as the first number, the average value of the number of crops in each of the multiple images corresponding to the overlapping region. With this configuration, even if there is a difference in the number of crops calculated for each of the multiple images corresponding to the overlapping region, it can be leveled out.
[0058] For example, the first calculation unit 216a may calculate the first number of crops further based on the result of the merging by the merging unit 213a. The first calculation unit 216a may calculate the merged crops, i.e., multiple crops identified as the same, as a single crop. With this configuration, even if the imaging directions of the first image sensor and the second image sensor are not parallel to each other, the coordinate systems can be aligned to merge the images of the crops captured by each sensor, and the number can be calculated using the merged crops, making it possible to accurately calculate the number of crops in the overlapping area.
[0059] The first calculation unit 216a may calculate the first number of overlapping areas based on, for example, the tracking result by the tracking unit 215a. With this configuration, the number of crops can be calculated with high accuracy even if areas overlap between adjacent images in the time series.
[0060] The calculation unit 216 includes a second calculation unit 216b. The second calculation unit 216b calculates the number of crops in non-overlapping areas other than overlapping areas in each of the multiple images (hereinafter also referred to as the "second number").
[0061] The second calculation unit 216b may calculate the number of crops in the composite image as the second number of crops in the non-overlapping areas. In other words, since the problem of double counting does not occur even in overlapping areas in a composite image made up of multiple images, the second calculation unit 216b calculates the number of crops by treating the areas of the composite image as non-overlapping areas. With this configuration, the composite image can be appropriately used to accurately calculate the number of crops in a specific area.
[0062] The second calculation unit 216b may calculate the second number of non-overlapping areas based on, for example, the tracking result. With this configuration, the number of crops can be calculated with high accuracy even if areas overlap between adjacent images in the time series.
[0063] For example, the first calculation unit 216a may calculate the first number based on the result of the detection unit 215's determination of whether each of the one or more crops is incomplete. Specifically, the first calculation unit 216a may exclude crops determined to be incomplete from the calculation of the first number. In other words, the first calculation unit 216a may not count incomplete crops in the overlapping area. On the other hand, the second calculation unit 216b may include incomplete crops in the calculation of the second number. Note that the first calculation unit 216a and the second calculation unit 216b may be reversed, so that the first calculation unit 216a counts incomplete crops. This configuration makes it possible to avoid overlapping calculations in both the overlapping and non-overlapping areas for crops located across the overlapping and non-overlapping areas.
[0064] The calculation unit 216 includes a third calculation unit 216c, which calculates the number of crops in the specific area (hereinafter also referred to as the "third number") based on the first number calculated by the first calculation unit 216a and the second number calculated by the second calculation unit 216b.
[0065] According to the above configuration, the third calculation unit 216c can calculate the number of crops in a specific area based on the number of crops calculated in each of the overlapping and non-overlapping areas. Therefore, the number of crops can be calculated taking into account overlaps between multiple images. Therefore, the third calculation unit 216c can avoid double counting even when multiple images capture crops that partially overlap to ensure no crops are missed. Therefore, the third calculation unit 216c can accurately calculate the number of crops in a specific area of a farm field.
[0066] For example, the third calculation unit 216c may divide the first number by the number of images constituting the overlapping area, sum the result of the sum and the second number, and calculate the third number based on the sum. With this configuration, the third calculation unit 216c can calculate the number of crops in the specific area based on the number of overlapping images.
[0067] [Communication Unit] The communication unit 220 transmits and receives various data such as measurement data to and from the measurement device 300 and / or other information processing devices 200 via the network N.
[0068] [Input / Output Unit] The input / output unit 230 transmits and receives measurement data and the like to and from external devices such as a display device and an external storage device.
[0069] [Storage Unit] The storage unit 240 stores crop identification information, crop attribute information, count information, and / or measurement data in association with each other. The storage unit 240 may store each piece of data using a database management system (DBMS), or may store each piece of information using a file system. When using a DBMS, the storage unit 240 may provide a table for each piece of data and manage the data by associating the tables with each other.
[0070] 4. Operation Example An operation example of the information processing device 200 will be described with reference to Fig. 7. Note that the order of the processes shown below is an example and may be changed as appropriate.
[0071] 7A is a flow diagram showing the flow of measurement data acquisition processing in the information processing device 200. As shown in FIG. 7A, the acquisition unit 211 of the information processing device 200 acquires measurement data from the measurement device 300 or another information processing device 200 (step S10). The acquisition unit 211 stores the acquired measurement data in the storage unit 240 based on time (step S11).
[0072] 7B is a flow diagram showing the flow of the crop calculation process in the information processing device 200. As shown in FIG. 7B, the acquisition unit 211 of the information processing device 200 acquires from the storage unit 240 a plurality of images including one or more crops, each of which has been captured by one or more image sensors 311 in a specific region of the field (step S20). The detection unit 215 detects one or more crops from each of the acquired images (step S21). The identification unit 212 identifies overlapping regions between the respective regions of the multiple images and non-overlapping regions other than the overlapping regions in the respective regions of the multiple images (step S22). Note that the order of steps S21 and S22 may be reversed.
[0073] The first calculation unit 216a of the information processing device 200 calculates a first number of crops in the overlapping region (step S23). The second calculation unit 216b calculates a second number of crops in the non-overlapping region in parallel with step S22 (step S24). Note that steps S23 and S24 may be processed in series, and if processed in series, the order does not matter. The third calculation unit 216c calculates a third number of crops in the specific region based on the first and second numbers (step S25).
[0074] 8, an example of a hardware configuration in which the above-described information processing device 200 is realized by a computer 800 will be described. Note that the functions of each device can also be realized by dividing them into multiple devices.
[0075] As shown in FIG. 8, a computer 800 includes a processor 801, a memory 803, a storage device 805, an input I / F unit 807, a data I / F unit 809, a communication I / F unit 811, and a display device 813.
[0076] The processor 801 controls various processes in the computer 800 by executing programs stored in the memory 803. For example, each functional unit included in the control unit 210 of the information processing device 200 can be realized by the processor 801 executing a program temporarily stored in the memory 803.
[0077] The memory 803 is a storage medium such as a RAM (Random Access Memory), and temporarily stores the program code of the program executed by the processor 801 or data required when the program is executed.
[0078] The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 805 stores an operating system or various programs for implementing the above-mentioned components. In addition, the storage device 805 can also store tables for registering measurement data, crop attribute information, etc., and a database for managing these tables. Such programs or data are loaded into the memory 803 as needed and referenced by the processor 801.
[0079] The input I / F unit 807 is a device for receiving input from a user. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, and a wearable device. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).
[0080] The data I / F unit 809 is a device for inputting data from outside the computer 800. A specific example of the data I / F unit 809 is a drive device for reading data stored in various storage media. The data I / F unit 809 may be provided outside the computer 800. In this case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.
[0081] The communication I / F unit 811 is a device for performing data communication via the Internet N, either wired or wirelessly, with devices external to the computer 800. The communication I / F unit 811 may be provided external to the computer 800. In this case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.
[0082] The display device 813 is a device for displaying various types of information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a display of a wearable device. The display device 813 may be provided outside the computer 800. In this case, the display device 813 is connected to the computer 800 via, for example, a display cable. Furthermore, when a touch panel is used as the input I / F unit 807, the display device 813 can be configured as an integral part of the input I / F unit 807.
[0083] The above-described embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to these embodiments. Furthermore, the present invention can be modified in various ways without departing from the spirit of the invention. Furthermore, those skilled in the art can adopt embodiments in which the elements described below are replaced with equivalents, and such embodiments are also within the scope of the present invention.
[0084] [Modifications] Although the present invention has been described based on the above embodiment, the following cases are also included in the present invention.
[0085] [Variation 1] In the above embodiment, an example has been described in which images of farm crops are combined based on the overlapping area identified by the identification unit 212, but the image combining method according to this embodiment is not limited to this. For example, the combining unit 214 may combine images of farm crops based on marker objects shown in the images, rather than the overlapping area. The marker object may be, for example, any object that serves as a reference among objects provided for growing farm crops, as described in the example of FIG. 5( a) above.
[0086] 9 is a schematic diagram showing an example of an imaging method and image synthesis process using the mobile sensor 310. As shown in Fig. 9, in the smart agriculture system 1, for example, a left image P1 and a right image P2 from the left image sensor 311a and the right image sensor 311b, respectively, may be synthesized to generate a synthesized image P3. For alignment during this synthesis, the images may be captured so that overlapping areas are provided between the multiple images.
[0087] The left image sensor 311a and the right image sensor 311b may set a marker object and capture images so that the marker object is included in the capture range (particularly the overlapping area). Then, the information processing device 200 may perform matching such as alignment based on the marker objects shown in the left image P1 and the right image P2, and generate a single composite image P3.
[0088] [Variation 2] In the above embodiment, an example was described in which two or more images selected from a plurality of target images in a specific region acquired by the acquisition unit 211 were combined. However, the image combination process according to this embodiment is not limited to this. For example, the combination unit 214 may combine all of the target images based on overlapping regions or marker objects. The third calculation unit 216c may calculate the third number of crops in the specific region based on this combined image. With this configuration, the use of marker objects makes it possible to calculate the number of crops in a specific region without identifying overlapping regions or separately calculating the number of crops in overlapping and non-overlapping regions.
[0089] [Variation 3] At least some of the components of the information processing device 200 according to the above embodiment may be provided in the measuring device 300. For example, the function of detecting crops from images provided in the information processing device 200, i.e., the function realized by the detection unit 215, may be provided in the measuring device 300. In this case, the information processing device 200 may acquire information indicating the results of this detection from the measuring device 300 and perform processing such as identifying overlapping areas and calculating the number of crops.
[0090] Aspects of this embodiment include the following disclosure.
[0091] [Supplementary Note 1] An information processing device (200) comprising: an acquisition unit (211) that acquires a plurality of images, each including one or more crops, captured by one or more imaging devices (311) in a specific region of a farm field; an identification unit (212) that identifies overlapping regions between each region of the plurality of images; a first calculation unit (216a) that calculates a first number of crops in the overlapping regions; a second calculation unit (216b) that calculates a second number of crops in non-overlapping regions other than the overlapping regions in each region of the plurality of images; and a third calculation unit (216c) that calculates a third number of crops in the specific region based on the first number and the second number.
[0092] [Supplementary Note 2] An information processing method in which a computer (200) acquires a plurality of images including one or more crops, each captured by one or more imaging devices (311) in a specific area of a field; identifies an overlapping area between each area of the plurality of images; calculates a first number of crops in the overlapping area; calculates a second number of crops in a non-overlapping area other than the overlapping area in each area of the plurality of images; and calculates a third number of crops in the specific area based on the first and second numbers.
[0093] [Supplementary Note 3] A program causing a computer (200) to execute: an acquisition function (211) for acquiring a plurality of images including one or more crops, the images being captured by one or more imaging devices (311) in a specific region of a field; an identification function (212) for identifying overlapping regions between the respective regions of the plurality of images; a first calculation function (216a) for calculating a first number of crops in the overlapping regions; a second calculation function (216b) for calculating a second number of crops in non-overlapping regions other than the overlapping regions in the respective regions of the plurality of images; and a third calculation function (216c) for calculating a third number of crops in the specific region based on the first number and the second number.
[0094] 1...Smart agriculture system, 200...information processing device, 210...control unit, 211...acquisition unit, 212...identification unit, 213...coordinate processing unit, 214...synthesis unit, 215...detection unit, 216...calculation unit, 220...communication unit, 230...input / output unit, 240...storage unit, 300...measuring device, 310...mobile sensor, 311...image sensor
Claims
1. An information processing device comprising: an acquisition unit that acquires a plurality of images including one or more agricultural crops, the images being captured by one or more respective imaging devices in a specific area of a field; an identification unit that identifies overlapping areas between each area of the plurality of images; a first calculation unit that calculates a first number of the agricultural crops in the overlapping areas; a second calculation unit that calculates a second number of the agricultural crops in non-overlapping areas other than the overlapping areas in each area of the plurality of images; and a third calculation unit that calculates a third number of the agricultural crops in the specific area based on the first number and the second number.
2. The information processing device according to claim 1, wherein the first calculation unit calculates the number of crops in each of the areas of the multiple images that correspond to the overlapping area, and adopts the number of crops in the area with the largest number as the result of the calculation as the first number.
3. The information processing device according to claim 1, wherein the first calculation unit calculates, as the first number, an average value of the number of crops in each of the areas of the plurality of images that correspond to the overlapping area.
4. An information processing device as described in any one of claims 1 to 3, wherein the third calculation unit sums the second number and a number obtained by dividing the first number by the number of the multiple images that constitute the overlapping area, and calculates the third number based on the result of the sum.
5. The information processing device according to claim 1, further comprising a synthesis unit that synthesizes the multiple images based on the overlapping area to generate a synthetic image, and the second calculation unit calculates the number of the crops in the synthetic image as the second number.
6. The information processing device according to claim 1, wherein the one or more imaging devices include a first imaging device and a second imaging device; a coordinate processing unit that aligns the coordinate systems of the image captured by the first imaging device and the image captured by the second imaging device based on the relative positions between the first imaging device and the second imaging device; and an integration unit that integrates the crops in the overlapping area based on the central coordinates of each of the crops included in the images whose coordinate systems have been aligned; and the first calculation unit calculates the first number further based on a result of the integration.
7. The information processing device according to claim 1 or 2, wherein the one or more imaging devices include a first imaging device and a second imaging device, and the identification unit identifies, as the overlapping area, an area where an area of an image captured by the first imaging device overlaps with an area of an image captured by the second imaging device.
8. The information processing device according to claim 1 or 2, wherein the acquisition unit acquires, as the image, an image of the agricultural crop captured by the one or more imaging devices along a predetermined direction, and the identification unit identifies, as the overlapping region, a region formed by overlapping ends of the multiple images captured in adjacent rows among multiple rows aligned in the predetermined direction.
9. The information processing device according to claim 1 or 2, wherein the plurality of images are images in a time series captured by the imaging device, and the identification unit identifies, as the overlapping area, an overlapping area between the images adjacent in the time series.
10. The information processing device according to claim 9, wherein the acquisition unit acquires the amount of movement of the one or more imaging devices when the imaging devices are moving while capturing images of the crops, and the identification unit identifies the overlapping area based on the amount of movement.
11. An information processing device as described in claim 1 or 2, wherein the multiple images are a time series of images captured by the imaging device, the information processing device further comprising: a detection unit that detects each of the one or more agricultural crops in each image in the time series; and a tracking unit that tracks each of the detected one or more agricultural crops between adjacent images in the time series; the first calculation unit calculates the first number of overlapping areas further based on the result of the tracking; and the second calculation unit calculates the second number of non-overlapping areas further based on the result of the tracking.
12. The information processing device of claim 11, wherein the acquisition unit acquires the amount of movement of the one or more imaging devices when the one or more imaging devices are moving to capture images of the crops, and an interpolation unit interpolates the crops not detected by the detection unit between adjacent images in a time series based on the amount of movement, and the tracking unit tracks each of the one or more crops based further on a result of the interpolation.
13. An information processing method in which a computer acquires a plurality of images including one or more crops, the images being captured by one or more respective imaging devices in a specific area of a field; identifies overlapping areas between each of the plurality of images; calculates a first number of the crops in the overlapping areas; calculates a second number of the crops in non-overlapping areas other than the overlapping areas in each of the plurality of images; and calculates a third number of the crops in the specific area based on the first and second numbers.
14. A program causing a computer to execute the following functions: an acquisition function for acquiring a plurality of images including one or more crops, the images being captured by each of one or more imaging devices in a specific area of a field; an identification function for identifying an overlapping area between each area of the plurality of images; a first calculation function for calculating a first number of the crops in the overlapping area; a second calculation function for calculating a second number of the crops in a non-overlapping area other than the overlapping area in each area of the plurality of images; and a third calculation function for calculating a third number of the crops in the specific area based on the first number and the second number.
Citation Information
Patent Citations
Harvesting system of fruit
JP2019017396A
Aerial photograph imaging method and aerial photograph imaging device
JP2013108927A
Nutritious liquid supply apparatus
JP2016010374A
Agricultural management prediction system, agricultural management prediction method, and server device
JP2019153109A
Harvest prediction device and harvest prediction method
JP2021108586A
