Information processing device, information processing method, and program

The information processing device improves crop positioning accuracy by integrating absolute position and tilt information, addressing the limitations of conventional methods in accurately determining crop growth status and location.

WO2024241899A1PCT designated stage expired Publication Date: 2024-11-28OMRON CORP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017296
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-05-09
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional methods for determining the growth status of crops over wide fields are insufficiently accurate, relying solely on location information and failing to provide precise positioning of agricultural products.

Method used

An information processing device and method that acquires first absolute position information, image data, and tilt information from an imaging device, allowing for the identification and estimation of the second absolute position of agricultural products within the image, thereby enhancing positional accuracy.

Benefits of technology

Enables precise specification of agricultural product positions in a field, improving accuracy by correcting for image sensor tilt and overlap, facilitating more effective crop monitoring and management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017296_28112024_PF_FP_ABST
    Figure JP2024017296_28112024_PF_FP_ABST
Patent Text Reader

Abstract

An information processing device 200 comprises: a position acquisition unit 211a that acquires first absolute position information indicating a first absolute position of an imaging device 311 that captures an image of one or more crops in a specific region of a field; an image acquisition unit 211b that acquires an image captured by the imaging device 311; an inclination acquisition unit 211c that acquires inclination information indicating an inclination of the imaging device measured by the measurement unit 300; an identification unit 212 that identifies an intra-image position in an image in each of the one or more crops shown in the image on the basis of the image and the inclination information; and an estimation unit 213 that estimates a second absolute position of each of the one or more crops shown in the image on the basis of the first absolute position information and the intra-image position.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] Conventionally, there are known techniques for identifying the positions of agricultural crops grown in a farm field. For example, Patent Document 1 below discloses an information processing device that acquires an image from an imaging device that captures an image in front of an agricultural vehicle and recognizes the positions of the agricultural crops in the farm field based on the image.

[0003] Japanese Patent Application Laid-Open No. 2021-100384

[0004] In the above-mentioned Patent Document 1, the position of each crop in the image can be identified. However, for example, when determining the growth status of crops over a wide area of ​​a farm field, it is not sufficient to use only such position information, and it is necessary to identify the position of the crops with higher accuracy.

[0005] Therefore, in order to solve the above-mentioned problems, an object of the present invention is to provide an information processing device, an information processing method, and a program that can identify the position of agricultural crops in a farm field with higher accuracy.

[0006] An information processing device according to one aspect of the present disclosure includes a position acquisition unit that acquires first absolute position information indicating a first absolute position of an imaging device that images one or more agricultural crops in a specific area of ​​a field, an image acquisition unit that acquires an image captured by the imaging device 311, a tilt acquisition unit that acquires tilt information indicating the tilt of the imaging device measured by a measurement unit, an identification unit that identifies an in-image position within the image for each of the one or more agricultural crops shown in the image based on the image and the tilt information, and an estimation unit that estimates a second absolute position of each of the one or more agricultural crops shown in the image based on the first absolute position information and the in-image position.

[0007] Another aspect of the present disclosure relates to a method in which a computer acquires first absolute position information indicating a first absolute position of an imaging device that images one or more agricultural crops in a specific area of ​​a field, acquires an image captured by the imaging device, acquires tilt information indicating the tilt of the imaging device measured by a measurement unit, identifies an in-image position within the image for each of the one or more agricultural crops shown in the image based on the image and the tilt information, and estimates a second absolute position of each of the one or more agricultural crops shown in the image based on the first absolute position information and the in-image position.

[0008] A program according to another aspect of the present disclosure causes a computer to execute a position acquisition function that acquires first absolute position information indicating the absolute position of an imaging device that images one or more agricultural crops in a specific area of ​​a field; an image acquisition function that acquires an image captured by the imaging device; a tilt acquisition function that acquires tilt information indicating the tilt of the imaging device measured by a measurement unit; an identification function that identifies an in-image position within the image for each of one or more agricultural crops shown in the image based on the image and the tilt information; and an estimation function that estimates a second absolute position of each of one or more agricultural crops shown in the image based on the first absolute position information and the in-image position.

[0009] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that can identify the position of agricultural crops in a field with higher accuracy.

[0010] FIG. 1 is a diagram illustrating an overview of a smart agriculture system according to an embodiment of the present invention. FIG. 2 is a conceptual diagram more specifically illustrating the field of view of a step surface S in the field shown in FIG. 1. FIG. 3 is a schematic diagram illustrating an example of an imaging method using an imaging device according to an embodiment of the present invention. FIG. 4 is a schematic diagram illustrating an example of an imaging method using an imaging device according to an embodiment of the present invention and a method for specifying the position of agricultural crops. FIG. 5 is a schematic diagram illustrating an example of an imaging method using an imaging device according to an embodiment of the present invention and a method for specifying the position of agricultural crops. FIG. 6 is a diagram illustrating an example of a functional configuration of an information processing device according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example of an operation example of the information processing device according to an embodiment of the present invention. FIG. 8 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment of the present invention.

[0011] Hereinafter, an embodiment of the present invention (hereinafter referred to as "the present embodiment") will be described in detail with reference to the drawings as necessary. However, the present invention is not limited to this embodiment, and various modifications are possible without departing from the spirit of the present invention. In the drawings, the same elements are given the same reference numerals, and redundant explanations will be omitted. Furthermore, positional relationships such as up, down, left, and right are based on the positional relationships shown in the drawings unless otherwise specified. Furthermore, the dimensional ratios of the drawings are not limited to those shown in the drawings.

[0012] <1. System> Fig. 1 is a diagram showing an overview of a smart agriculture system 1 according to an embodiment of the present invention. As shown in Fig. 1, the smart agriculture system 1 includes an information processing device 200 and a measurement device 300. The measurement device 300 may be, for example, various sensors such as a mobile sensor 310 and / or a fixed sensor 320. The information processing device 200 may also be a terminal device 200a and / or a server device 200b. The information processing device 200 and the measurement device 300 may be connected via a network N.

[0013] In the smart agriculture system 1, various sensors such as the fixed sensor 320 and the mobile sensor 310 are used to acquire images (hereinafter simply referred to as "images") of agricultural products in a field. Next, the acquired images are stored by the information processing device 200 (for example, the server device 200b). Then, the information processing device 200 (for example, the terminal device 200a) identifies the location of the agricultural products and calculates the number of agricultural products based on the images of the agricultural products. The agricultural products may be, for example, trellis-grown fruits such as grapes, kiwis, or pears. Furthermore, the agricultural products may specifically be fruit flowers or fruits. Furthermore, as another example, the agricultural products may be fruit vegetables such as tomatoes, eggplants, or peppers.

[0014] [Information Processing Device] Information processing device 200 may be a terminal device such as a laptop or a smartphone, or a server device such as a storage server. By executing a predetermined program, information processing device 200 connects to measurement device 300 to send and receive various data, stores various received data, outputs various data to a user, and accepts instructions from the user for measurement device 300, etc., by voice input, operation input, etc.

[0015] The server device 200b may be a cloud server or an edge server. The edge server may be installed in or near the field and perform data processing and analysis. This reduces communication delays and distributes the processing load because data is processed on the edge server side without being sent to the cloud server. For example, the server device 200b, which is an edge server, may collect images of crops from the measuring device 300 and store the collected images in the server device 200b, which is a cloud server. The terminal device 200a may then acquire these stored images and calculate the number of crops. Note that instead of the edge server, a terminal device 200a having the same functions as the edge server may be provided.

[0016] [Measuring Device] The measuring device 300 measures the crops and the surrounding environment in the field, generates data indicating the results of the measurement (hereinafter also referred to as "measurement data"), and transmits the data to the information processing device 200 or the like. The measuring device 300 is not particularly limited, and may be, for example, a device equipped with various sensors and installed at any position in the field, a drone equipped with various sensors and flying within the field, or an unmanned aerial vehicle capable of moving within the field (hereinafter also referred to as a "mobile robot"), or a human-operated terminal such as a smartphone equipped with various sensors, a handheld computing device, or a wearable device. Here, the sensor is not particularly limited, and may be, for example, an image sensor, a sensor that measures the position of the measuring device 300 (hereinafter also referred to as a "position sensor"), a sensor that measures the tilt of the measuring device 300 (hereinafter also referred to as a "tilt sensor"), a distance sensor and / or an environmental sensor that measures the distance between the measuring device 300 and the crops (hereinafter also referred to as a "first distance"), etc.

[0017] The image sensor is one aspect of the imaging device. The image sensor is not particularly limited as long as it is a sensor capable of capturing still images or videos. The image sensor may be, for example, a fixed camera installed in a field, a camera on a terminal such as a wearable device, or a camera installed on a mobile measuring device 300. The image sensor may be configured to continuously generate images in frame units, for example, and capture time-series images (for example, video images). The image sensor may also be, for example, a depth camera equipped with a distance measurement function.

[0018] The position sensor may be a GPS receiving antenna or the like for measuring position information of the measurement device 300 (for example, first absolute position information indicating latitude and longitude).

[0019] The tilt sensor is one aspect of a measurement unit that measures the tilt of the imaging device. The tilt sensor may be, for example, a gyro sensor, an acceleration sensor, or a gravity sensor.

[0020] The environmental sensor is a sensor for measuring environmental information in the field of the crops 520. The environmental sensor is not particularly limited, but examples thereof include a weather sensor, a soil sensor, and a gas sensor.

[0021] The mobile sensor 310 may be, for example, a drone that travels within a field and is equipped with various sensors. This drone is one form of a mobile object. The sensors may be, for example, one or more image sensors for capturing images of agricultural crops, one or more position sensors, and / or one or more tilt sensors.

[0022] The fixed sensor 320 may be equipped with, for example, a fixed camera. The fixed sensor 320 may also be a marker for the position of the mobile sensor 310 when it moves, such as a sensor-based sign using a Bluetooth beacon. The mobile sensor 310 may receive the beacon and determine its own position in a coordinate system of a specific area within the field. The specific area is an area of ​​the field for which the absolute position of the crops is to be determined, and may be, for example, the entire field, one or more plots among multiple plots within the field, or an area per unit area.

[0023] [Network] The network N is configured by a wireless network or a wired network. Examples of the network include a mobile phone network, a PHS (Personal Handy-phone System) network, a wireless LAN (Local Area Network, including communication conforming to IEEE 802.11 (so-called Wi / Fi (registered trademark))), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), WiMax (registered trademark), infrared communication, visible light communication, Bluetooth (registered trademark), a wired LAN, a telephone line, a power line communication network, a network conforming to IEEE 1394, etc.

[0024] Figure 2 is a conceptual diagram more specifically illustrating the field of view of the step surface S in the field shown in Figure 1. As shown in Figure 2, agricultural crops with fruit at the ends of tree branches are shown, and shelves 530 for supporting the branches are installed below the branches. Here, the mobile sensor 310 may travel under the branches and shelves 530 while pointing the sensor substantially upward relative to the ground to capture images of the agricultural crops.

[0025] In the smart agriculture system 1, for example, multiple imaging devices installed in a mobile sensor 310 or the like capture images of crops grown in a specific area, identify the positions of the crops in the captured images (hereinafter also referred to as "positions within the images"), and estimate the absolute positions of the crops (hereinafter also referred to as "second absolute positions") based on the absolute positions of the imaging devices (hereinafter also referred to as "first absolute positions") and the positions of the crops within the images. The absolute positions may be, for example, positions in a geodetic system such as latitude and longitude coordinates, positions in a plane rectangular coordinate system, or positions indicated by other specific coordinate systems (for example, coordinate systems within a specific area).

[0026] 2. Overview of Imaging and Identifying Overlapping Areas An example of the process of capturing images and identifying the position of crops using the mobile sensor 310 will be described with reference to FIGS. 3 to 7 . In this example, a left-side image sensor (hereinafter also referred to as the "left image sensor") 311a and a right-side image sensor (hereinafter also referred to as the "right image sensor") 311b are used, which are arranged in parallel with each other. The left image sensor 311a and the right image sensor 311b are aspects of a first imaging device and a second imaging device. The left image sensor 311a and the right image sensor 311b capture images of crops grown on shelves 530, for example, with the imaging direction facing upward relative to the ground. The left image sensor 311a and the right image sensor 311b will be collectively referred to as the "image sensor 311" when there is no need to distinguish between them.

[0027] 2-1. Imaging FIG. 3 is a schematic diagram showing the mobile sensor 310 traveling in a specific area of ​​a field, as viewed from above. As shown in FIG. 3, the shelves 530 have support columns arranged at intervals of a width W (e.g., 4 to 5 m). The mobile sensor 310 travels along the area between the columns L, which serves as a travel path, and travels near the center line C of the travel path. The mobile sensor 310 is equipped with a left image sensor 311a and a right image sensor 311b arranged in the left and right directions so as to capture images of the respective left and right areas. The left and right directions refer to directions approximately perpendicular to the travel direction of the mobile sensor 310 (the longitudinal direction of the area between the columns). For example, after traveling from one end of a travel path to the other, the mobile sensor 310 turns around and travels in the opposite direction from the previous travel path, from one end of the longitudinal direction to the other of the adjacent travel path. The left image sensor 311a and the right image sensor 311b may be set to have their respective image capturing ranges overlap, for example, to avoid missing images of agricultural produce, etc. The mobile sensor 310 may be set to have their respective image capturing ranges overlap when capturing images of two adjacent travel lanes, for example, to avoid missing images of agricultural produce, etc.

[0028] 2-2. Shift in Capture Range Due to Image Sensor Tilt> FIG. 4 is a schematic diagram showing the mobile sensor 310 capturing images while moving. FIG. 4( a) shows normal driving, and FIG. 4( b) shows the mobile sensor 310 tilted due to an obstacle D while driving. A left-side image P1 (hereinafter also referred to as the "left image") captured by the left image sensor 311a and a right-side image P2 (hereinafter also referred to as the "right image") captured by the right image sensor 311b with respect to the direction of travel may include overlapping areas of the same crops, etc. Furthermore, each of the left image P1 and the right image P2 may be time-series images captured while driving, and adjacent images in the time-series may include overlapping areas of the same crops, etc.

[0029] [Correction pursuant to Rule 91, 28.06.2024] In Figure 4(a), (1) shows the mobile sensor 310 as seen from behind, (2) shows the side view, and (3) shows the respective imaging ranges of the left image sensor 311a and the right image sensor 311b as seen from above. As shown in Figure 4(a), the left image sensor 311a and the right image sensor 311b each capture images of the tips of tree branches along the roadway, with their respective imaging ranges separated to the left and right. Normally, as shown in (3), the time-series left images P1a, P1b, and P1c and the time-series right images P2a, P2b, and P2c can be captured without any misalignment in their imaging ranges in the direction of travel or left-right.

[0030] [Correction pursuant to Rule 91, 28.06.2024] Figure 4(b) shows the situation when the mobile sensor 310, and therefore the image sensor 311, is tilted by an obstacle D, corresponding to (1) to (3) in Figure 4(a). As shown in Figure 4(b), as in Figure 4(a), the left image sensor 311a and the right image sensor 311b each have their respective imaging ranges split to the left and right to capture images of crops at the tips of tree branches along the roadway. When tilted by an obstacle D, as shown in (3), the imaging ranges of the time-series left images P1a', P1b', and P1c' and the time-series right images P2a', P2b', and P2c' may be misaligned in the direction of travel or left-right. In response to the shift in the imaging range caused by such obstacles or tilts due to unevenness in the road surface of the travel path, the smart farm system 1 can determine the position of the crops in the field with greater accuracy by estimating the second absolute position of each of one or more crops based on this tilt.

[0031] 5 to 7 are schematic diagrams showing an example of a method for identifying the position of agricultural crops in an image captured by the mobile sensor 310. In this example, for simplicity, an example will be described in which one image sensor 311 is used. In the coordinate system used in the example of FIGS. 5 to 7, the X-axis and Y-axis directions are the traveling direction and left-right direction, and the Z-axis direction is the height direction, and the values ​​of each coordinate in this coordinate system are expressed as (x-coordinate value, y-coordinate value, z-coordinate value). The unit used is meters (m).

[0032] FIG. 5 is a schematic diagram showing an overall image of the mobile sensor 310. As shown in FIG. 5, if the center coordinates of the lens of the image sensor 311 are (0,0,0) and the distance from the image sensor 311 to the crop to be imaged (i.e., the working distance) is z1, the center coordinates of the image capture range of the image sensor 311 can be expressed as (0,0,z1). The image capture range of the image sensor 311 can be calculated as 2Wz1 (m) in the X-axis direction and 2Wz1 (m) in the Y-axis direction. For example, the value of W is a value based on the horizontal angle of view of the image sensor 311 (for example, if the horizontal angle of view is θ 1 In this case, tan(θ 1 / 2), and the value of H is the vertical angle of view of the image sensor 311 (for example, the vertical angle of view is θ 2 In this case, tan(θ 2 / 2). This allows information processing device 200 to know the X and Y coordinate values ​​of the width and depth of the imaging range of image sensor 311, and therefore it can identify the position of each of one or more crops in this imaging range, i.e., their positions within the image, using X and Y coordinate values. Furthermore, if the first absolute position of information processing device 200 can be identified, it can estimate the second absolute position of the crops based on the identified positions within the image.

[0033] 6 and 7 are schematic diagrams showing the relationship between the movement and imaging range of the image sensor 311 in the mobile sensor 310 of Fig. 5. In this example, an example will be described in which the image sensor 311 moves x1 (m) in the X-axis direction.

[0034] As shown in Figure 6, before the image sensor 311 is moved, the coordinates of the vertex T1 of the shooting range are (Hz1, Wz1, z1), the coordinates of the vertex T2 are (-Hz1, -Wz1, z1), the coordinates of the center point T3 are (0, 0, z1), and the center coordinates of the lens of the image sensor 311 are (0, 0, 0).

[0035] After the image sensor 311 is moved, if there is no tilt in the image sensor 311, the coordinates of the vertex T1 are (x1+Hz1, Wz1, z1), the coordinates of the vertex T2 are (x1-Hz1, -Wz1, z1), the coordinates of the center point T3 are (x1, 0, z1), and the center coordinates of the lens of the image sensor 311 are (x1, 0, 0). On the other hand, if the image sensor 311 is moved and the angle θ 3 If the camera is tilted by a certain amount, the shooting range will shift, so the position of the crops must be determined taking this shift into account.

[0036] 7 is an enlarged view of the image sensor 311 shown in FIG. 6 when it is moved and tilted. As shown in FIG. 7, the coordinates of the center point T3 of the imaging range are (x1-z1 tan θ 3 , 0, z1). In addition, due to the tilt, the distance between the image sensor 311 and the crops to be imaged becomes z1′=z1 / cos θ 3The width 2Wz1' in the X-axis direction and the depth 2Hz1' in the Y-axis direction of the inclined imaging range can be calculated by, for example, using a keystone correction technique, by adjusting the inclination angle θ 3 In this case, the coordinates of the vertex T1 can be calculated using (x1-z tan θ 3 +Hz1', Wz1', z1), and the coordinates of vertex T2 are (x1-z tan θ 3 -Hz1', -Wz1', z1). This allows the information processing device 200 to know the shape of the image capture range of the image sensor 311 when tilted, and the X and Y coordinate values ​​of its width and depth. Therefore, the position of each of one or more crops in this image capture range, i.e., their position in the image, can be identified by the X and Y coordinate values ​​that take this tilt into account.

[0037] With the above configuration, in the smart agriculture system 1, when the image sensor 311 is tilted, the position of the crop in the image can be corrected based on the tilt. Therefore, the position of the crop in the field (the position in the image and the second absolute position) can be identified with higher accuracy.

[0038] 3. Functional Configuration The functional configuration of the information processing device 200 according to this embodiment will be described with reference to Fig. 8. As shown in Fig. 8, the information processing device 200 includes a control unit 210, a communication unit 220, an input / output unit 230, and a storage unit 240.

[0039] [Control Unit] The control unit 210 includes an acquisition unit 211, an identification unit 212, and an estimation unit 213. The control unit 210 may also include, for example, an integration unit 214, a calculation unit 215, a distance calculation unit 216, and / or a resolution calculation unit 217.

[0040] [Acquisition Unit] The acquisition unit 211 acquires various data from the measurement device 300, other information processing devices 200, the storage unit 240, etc. The acquisition unit 211 may store the acquired various data in the storage unit 240.

[0041] The acquisition unit 211 includes a position acquisition unit 211a. The position acquisition unit 211a acquires first absolute position information indicating the absolute position of an imaging device that images one or more agricultural products in a specific area of ​​the farm field. Specifically, the position acquisition unit 211a may receive the first absolute position information from a position sensor or another information processing device 200 via the communication unit 220. The first absolute position information is one of the measurement data, and may include (or be associated with) date and time information when the position is measured.

[0042] The acquisition unit 211 includes an image acquisition unit 211b. The image acquisition unit 211b acquires images captured by the image sensor 311. The image acquisition unit 211b may acquire, for example, a plurality of images including one or more crops, captured by each of the one or more image sensors 311 in a specific area of ​​a farm field. Specifically, the image acquisition unit 211b may receive data indicating images including crops (hereinafter also referred to as "image data") from the image sensors or another information processing device 200 via the communication unit 220. The image data is one type of measurement data and may include (or be associated with) date and time information and / or location information when the image was captured, for example.

[0043] The acquisition unit 211 includes a tilt acquisition unit 211c. The tilt acquisition unit 211c acquires tilt information indicating the tilt of the image sensor 311 measured by a tilt sensor that measures the tilt of the image sensor 311. The tilt information may include, for example, the angle of tilt of the image sensor 311. Specifically, the tilt acquisition unit 211c may receive the tilt information from the tilt sensor or another information processing device 200 via the communication unit 220. The tilt information is one type of measurement data and may include (or be associated with) date and time information and / or location information when the tilt is measured, for example.

[0044] The acquisition unit 211 may include, for example, a distance acquisition unit 211d. The distance acquisition unit 211d acquires distance information indicating the distance between the image sensor 311 and one or more crops. The distance indicated by this distance information may be, for example, (a) a distance measured by a distance measurement sensor, (b) a distance calculated by the distance calculation unit 216 described below, or (c) a fixed distance.

[0045] The acquisition unit 211 may include, for example, a size acquisition unit 211e. The size acquisition unit 211e acquires, for example, size information indicating the size of the agricultural products. The size of the agricultural products indicated by the size information may be, for example, a fixed value, a statistical value of the sizes measured for each of a plurality of agricultural products, or a size detected by the detection unit 212a for each of one or more agricultural products.

[0046] The acquisition unit 211 may include, for example, a device information acquisition unit 211f. The device information acquisition unit 211f may acquire, for example, information about the measuring device 300 (hereinafter also referred to as "device information"). When the measuring device 300 is an image sensor 311, the device information may include, for example, the angle of view (horizontal angle of view and vertical angle of view), focal length, and / or the number of pixels of the image sensor 311.

[0047] [Identification Unit] The identification unit 212 identifies the position within the image of each of one or more crops shown in the image based on the image acquired by the image acquisition unit 211 b. Specifically, the identification unit 212 identifies the position within the image of each of the crops detected from the image.

[0048] The identification unit 212 may identify the intra-image position of each of the one or more crops in the image, for example, further based on the tilt information acquired by the tilt acquisition unit 211c. Possible methods for the identification unit 212 to identify the intra-image positions of the crops using the tilt information include, for example, (a) identifying the position of each of the one or more crops in the image and then correcting each identified position, or (b) correcting the coordinates of a reference position in the image (hereinafter also referred to as an "in-image reference position") (e.g., coordinates of the vertices T1 and T2 or the center point T3 of the shooting range, etc.). In the above (b), the identification unit 212 may correct the intra-image reference position before the detection unit 212a, described later, detects the crops from the image.

[0049] The identification unit 212 may include, for example, a detection unit 212a. The detection unit 212a detects one or more agricultural products included in the image acquired by the acquisition unit 211 using image processing techniques such as machine learning, deep learning, or pattern recognition. During detection, the detection unit 212a may, for example, identify information such as the size, color, shape, or position within the image of each of the one or more agricultural products (hereinafter, collectively referred to as "crop attribute information"). Furthermore, the detection unit 212a may, for example, assign identification information for identifying each of the one or more detected agricultural products, and store this identification information, the crop attribute information, date and time information and / or location information, and other measurement data included in the image data in association with each other in the storage unit 240. Furthermore, the crop attribute information may include, for example, a second absolute position of the agricultural product estimated by the estimation unit 213 (described later).

[0050] For example, when detecting agricultural products, the detection unit 212a may determine whether the detected agricultural products are incomplete (or complete). An incomplete agricultural product may be, for example, a product that is not completely contained in an area of ​​an image to be detected. Specifically, the detection unit 212a may determine that a crop that is partially missing due to a boundary between multiple areas or an edge of an area is not completely contained in an area. The detection unit 212a may associate information indicating the result of the determination of whether the crop is incomplete with each of the one or more detected agricultural products.

[0051] When the multiple images acquired by the acquisition unit 211 are time-series images captured by an image sensor, the detection unit 212a may, for example, detect one or more agricultural products in each of the time-series images.

[0052] When the multiple images acquired by the acquisition unit 211 are time-series images captured by an image sensor, the detection unit 212a may, for example, detect one or more agricultural products in each of the time-series images.

[0053] The identification unit 212 may include a correction unit 212b. The correction unit 212b corrects the in-image position of the crop or the in-image reference position based on the tilt information acquired by the tilt acquisition unit 211c. Specifically, the correction unit 212b may correct the in-image position or the in-image reference position so that the first absolute position of the image sensor 311 and the angle of the image sensor 311 with respect to the crop are the same as those in an image when calibrated before starting image capture. When the image sensor 311 is mounted on the mobile sensor 310 and images are captured while moving, the correction unit 212b may correct the in-image position or the in-image reference position so that the same conditions as those in an image when calibrated before starting movement are the same.

[0054] 6 and 7, the correction unit 212b may correct the position of the crop in the image based on the tilt angle of the image sensor 311 and the distance indicated in the distance information. With this configuration, even if the image sensor 311 is tilted from the angle at which it was calibrated, the position of the crop can be identified with high accuracy based on the tilt angle and the distance between the crop.

[0055] The identification unit 212 may include, for example, a tracking unit 212c. The tracking unit 212c tracks each of the one or more detected crops between adjacent images in time series.

[0056] The tracking unit 212c may track each of one or more crops further based on the result of interpolation by the interpolation unit 212d, for example. With this configuration, even if the tracked crop is hidden by leaves or the like and cannot be detected, it is possible to continue tracking by interpolation.

[0057] The identification unit 212 may include, for example, an interpolation unit 212d. The interpolation unit 212d interpolates crops that have not been detected by the detection unit 212a between adjacent images in the time series based on the amount of movement of the image sensor 311. Specifically, the interpolation unit 212d may predict the position of each of one or more crops in the image t seconds from now based on the amount of movement of the image sensor 311 over t seconds, and interpolate the undetected crops based on the result of this prediction.

[0058] [Estimation unit] The estimation unit 213 estimates a second absolute position of each of one or more crops shown in the image based on the first absolute position information, the in-image positions of the crops identified by the identification unit 212 and / or the in-image positions of the crops corrected by the correction unit 212b. Specifically, when the absolute positions are positions in geodetic coordinates (e.g., latitude and longitude), the estimation unit 213 may convert the coordinates of the in-image positions of the crops into geodetic coordinates to estimate the second absolute positions of the crops.

[0059] With the above configuration, the estimation unit 213 can estimate the absolute position of the crop. Furthermore, when the image sensor 311 is tilted, the estimation unit 213 can estimate the absolute position of the crop based on the position of the crop in the image that has been corrected taking into account the tilt. Therefore, the position of the crop in the field can be identified with higher accuracy.

[0060] [Integration Unit] When there are multiple images, the integrating unit 214 integrates the crops shown in different images based on the second absolute positions of the crops shown in the multiple images. Specifically, the integrating unit 214 may identify, as the same crop, the second absolute positions of the crops detected by the detecting unit 212a that are within a predetermined range, in other words, the distance between the second absolute positions of the crops that are within a threshold value.

[0061] For example, the integrating unit 214 may assign identification information for identifying each of the integrated crops, and store this identification information in the storage unit 240 in association with the second absolute position of the crop, crop attribute information (including size information indicating the size of the crop), date and time information and / or location information and other measurement data included in the image data. Examples of information associated with this identification information include (a) individual information about each of the crops to be integrated, (b) statistical values ​​(e.g., maximum values, average values, etc.) of the information about the crops to be integrated, and (c) both of (a) and (b). Specifically, the integrating unit 214 may associate the identification information of crop A3, which is obtained by integrating crops A1 and A2, with at least one of (a) size information about crop A1, (b) size information about crop A2, and (c) information indicating the average size of crop A1 and crop A2. This configuration allows management of the second absolute position, size, etc. for each crop grown in the field.

[0062] [Calculation Unit] The calculation unit 215 calculates the number of crops in the field. For example, when there are multiple regions to be calculated, the calculation unit 215 may calculate a statistical value of the crops in the field as the calculation of the number of crops. The statistical value may be, for example, the total number of crops, the average value, the median value, the maximum value, the minimum value, or the mode. The calculation unit 215 may store count information indicating the calculation result in the storage unit 240 in association with the measurement data.

[0063] For example, the calculation unit 215 calculates the number of one or more crops in a specific region of the farm field based on the second absolute positions of the crops estimated by the estimation unit 213. Specifically, the calculation unit 215 may identify the crops for each second absolute position and calculate the number of identified crops. With this configuration, even if there are crops that are captured twice in multiple images, double counting can be avoided and the number of crops can be calculated with high accuracy.

[0064] The calculation unit 215 may calculate the number of one or more crops in a specific area of ​​the field based on the integration result by the integration unit 214, for example.

[0065] [Distance Calculation Unit] Based on the size information and the image, distance calculation unit 216 calculates a first distance between each of the one or more crops and image sensor 311. Distance calculation unit 216 may, for example, calculate the proportion of the image area corresponding to the crops to the entire area of ​​the image, and calculate the distance between the crops and image sensor 311 from this proportion and the focal length of image sensor 311 based on device information of image sensor 311. With this configuration, the distance between image sensor 311 and the crops can be determined without using information measured by a distance measurement sensor or the like, and therefore the second absolute position of the crops can be estimated or corrected.

[0066] [Resolution Calculation Unit] The resolution calculation unit 217 calculates the resolution of the image based on the distance calculated by the distance calculation unit 216 (or the distance indicated by the distance information) and the device information. Specifically, the resolution calculation unit 217 may calculate the resolution of the image based on the shooting range of the image, the number of pixels, and the first distance. The resolution calculation unit 217 may also calculate the resolution of the image based on the focal length of the image sensor 311. With this configuration, the size of the crops in the image can be measured with high accuracy based on the resolution of each image.

[0067] [Communication Unit] The communication unit 220 transmits and receives various data such as measurement data to and from the measurement device 300 and / or other information processing devices 200 via the network N.

[0068] [Input / Output Unit] The input / output unit 230 transmits and receives measurement data and the like to and from external devices such as a display device and an external storage device.

[0069] [Storage Unit] The storage unit 240 stores crop identification information, crop attribute information, count information, and / or measurement data in association with each other. The storage unit 240 may store each piece of data using a database management system (DBMS), or may store each piece of information using a file system. When using a DBMS, the storage unit 240 may provide a table for each piece of data and manage the data by associating the tables with each other.

[0070] 4. Operation Example An operation example of the information processing device 200 will be described with reference to Fig. 9. Note that the order of the processes shown below is an example and may be changed as appropriate.

[0071] 9A is a flow diagram showing the flow of measurement data acquisition processing in the information processing device 200. As shown in FIG. 9A, the acquisition unit 211 of the information processing device 200 acquires measurement data from the measurement device 300 or another information processing device 200 (step S10). The acquisition unit 211 stores this acquired measurement data in the storage unit 240 based on time (step S11).

[0072] 9(b) is a flow diagram showing the flow of the estimation process and calculation process of the second absolute position of the crop in the information processing device 200. As shown in FIG. 9(b), the position acquisition unit 211a of the information processing device 200 acquires, from the storage unit 240, first absolute position information indicating the first absolute position of the image sensor 311 that images one or more crops (step S20). The image acquisition unit 211b acquires, from the storage unit 240, images captured by the image sensor 311 (step S21). The tilt acquisition unit 211c acquires tilt information indicating the tilt of the image sensor 311 measured by the tilt sensor (step S22).

[0073] The detection unit 212a detects one or more crops from each of the acquired images (step S23). The identification unit 212 identifies the position within the image of each of the one or more crops shown in the image based on the detection result (step S24).

[0074] If the tilt of the image sensor 311 indicated by the tilt information is equal to or greater than the threshold (YES in step S25), the corrector 212b corrects the position within the image based on the tilt information (step S26). The estimator 213 estimates a second absolute position of each of the one or more crops shown in the image based on the first absolute position information and the position within the image of the crops (step S27). The calculator 215 calculates the number of the one or more crops in the specific region of the field based on the second absolute positions (step S28).

[0075] The processes of steps S25 and S26 may be executed before step S23. In this case, in the process of step S26, the correction unit 212b may correct the intra-image reference position instead of the intra-image position.

[0076] 10, an example of a hardware configuration in which the above-described information processing device 200 is realized by a computer 800 will be described. Note that the functions of each device can also be realized by dividing them into multiple devices.

[0077] As shown in FIG. 10, a computer 800 includes a processor 801, a memory 803, a storage device 805, an input I / F unit 807, a data I / F unit 809, a communication I / F unit 811, and a display device 813.

[0078] The processor 801 controls various processes in the computer 800 by executing programs stored in the memory 803. For example, each functional unit included in the control unit 210 of the information processing device 200 can be realized by the processor 801 executing a program temporarily stored in the memory 803.

[0079] The memory 803 is a storage medium such as a RAM (Random Access Memory), and temporarily stores the program code of the program executed by the processor 801 or data required when the program is executed.

[0080] The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 805 stores an operating system or various programs for implementing the above-mentioned components. In addition, the storage device 805 can also store tables for registering measurement data, crop attribute information, etc., and a database for managing these tables. Such programs or data are loaded into the memory 803 as needed and referenced by the processor 801.

[0081] The input I / F unit 807 is a device for receiving input from a user. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, and a wearable device. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).

[0082] The data I / F unit 809 is a device for inputting data from outside the computer 800. A specific example of the data I / F unit 809 is a drive device for reading data stored in various storage media. The data I / F unit 809 may be provided outside the computer 800. In this case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.

[0083] The communication I / F unit 811 is a device for performing data communication via the Internet N, either wired or wirelessly, with devices external to the computer 800. The communication I / F unit 811 may be provided external to the computer 800. In this case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.

[0084] The display device 813 is a device for displaying various types of information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a display of a wearable device. The display device 813 may be provided outside the computer 800. In this case, the display device 813 is connected to the computer 800 via, for example, a display cable. Furthermore, when a touch panel is used as the input I / F unit 807, the display device 813 can be configured as an integral part of the input I / F unit 807.

[0085] The above-described embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to these embodiments. Furthermore, the present invention can be modified in various ways without departing from the spirit of the invention. Furthermore, those skilled in the art can adopt embodiments in which the elements described below are replaced with equivalents, and such embodiments are also within the scope of the present invention.

[0086] [Modifications] Although the present invention has been described based on the above embodiment, the following cases are also included in the present invention.

[0087] [Variation 1] At least some of the components of information processing device 200 according to the above embodiment may be provided in measuring device 300. For example, the function of detecting crops from images provided in information processing device 200, i.e., the function realized by detection unit 212a, may be provided in measuring device 300. In this case, information processing device 200 may acquire information indicating the results of this detection from measuring device 300 and perform processing such as estimating the second absolute position of the crops.

[0088] Aspects of this embodiment include the following disclosure.

[0089] [Supplementary Note 1] An information processing device (200) comprising: a position acquisition unit (211a) that acquires first absolute position information indicating a first absolute position of an imaging device (311) that images one or more agricultural products in a specific area of ​​a farm field; an image acquisition unit (211b) that acquires an image captured by the imaging device (311); an identification unit (212) that identifies, based on the image, a position within the image of each of the one or more agricultural products shown in the image; a tilt acquisition unit (211c) that acquires tilt information indicating a tilt of the imaging device measured by a measurement unit (300); a correction unit (212b) that corrects the position within the image based on the tilt information; and an estimation unit (213) that estimates, based on the first absolute position information and the position within the image, a second absolute position of each of the one or more agricultural products shown in the image.

[0090] [Supplementary Note 2] An information processing method in which a computer (200) acquires first absolute position information indicating a first absolute position of an imaging device (311) that images one or more agricultural products in a specific area of ​​a farm field; acquires an image captured by the imaging device (311); identifies an in-image position within the image for each of the one or more agricultural products shown in the image based on the image; acquires tilt information indicating an inclination of the imaging device measured by a measurement unit (300); corrects the in-image position based on the tilt information; and estimates a second absolute position of each of the one or more agricultural products shown in the image based on the first absolute position information and the in-image position.

[0091] [Supplementary Note 3] A program causing a computer (200) to execute: a position acquisition function (211a) that acquires first absolute position information indicating the absolute position of an imaging device (311) that images one or more agricultural products in a specific area of ​​a farm field; an image acquisition function (211b) that acquires an image captured by the imaging device; an identification function (212) that identifies, based on the image, an intra-image position of each of the one or more agricultural products shown in the image; a tilt acquisition function (211c) that acquires tilt information indicating the tilt of the imaging device measured by a measurement unit; a correction function (212b) that corrects the intra-image position based on the tilt information; and an estimation function (213) that estimates, based on the first absolute position information and the intra-image position, a second absolute position of each of the one or more agricultural products shown in the image.

[0092] 1...Smart agriculture system, 200...Information processing device, 210...Control unit, 211...Acquisition unit, 212...Identification unit, 213...Estimation unit, 214...Integration unit, 215...Calculation unit, 216...Distance calculation unit, 217...Resolution calculation unit, 220...Communication unit, 230...Input / output unit, 240...Memory unit, 300...Measuring device, 310...Mobile sensor, 311...Image sensor.

Claims

1. An information processing device comprising: a position acquisition unit that acquires first absolute position information indicating a first absolute position of an imaging device that images one or more agricultural crops in a specific area of ​​a field; an image acquisition unit that acquires an image captured by the imaging device; a tilt acquisition unit that acquires tilt information indicating a tilt of the imaging device measured by a measurement unit; an identification unit that identifies an in-image position within the image for each of the one or more agricultural crops shown in the image based on the image and the tilt information; and an estimation unit that estimates a second absolute position of each of the one or more agricultural crops shown in the image based on the first absolute position information and the in-image position.

2. The information processing device according to claim 1, further comprising a calculation unit that calculates the number of said one or more crops in said specific area based on said second absolute position.

3. The information processing device according to claim 2, further comprising an integration unit which integrates the crops shown in different images based on the second absolute positions of each of the crops shown in the multiple images, and wherein the calculation unit calculates the number of the one or more crops in the specific area based on a result of the integration.

4. An information processing device as described in claim 1 or 2, wherein there are a plurality of the images, and further comprising an integration unit that integrates the crops shown in different images based on the second absolute positions of each of the crops shown in the plurality of images, and the integration unit stores in a memory unit identification information that identifies each of the integrated crops and the second absolute positions of each of the crops in association with each other.

5. An information processing device as described in claim 1 or 2, wherein there are a plurality of the images, and further comprising an integration unit that integrates the crops shown in different images based on the second absolute positions of each of the crops shown in the plurality of images, and the integration unit stores in a memory unit identification information that identifies each of the integrated crops and size information that indicates the size of each of the crops in association with each other.

6. An information processing device as described in claim 1 or 2, wherein the tilt information includes the tilt angle, and the information processing device further comprises a distance acquisition unit that acquires distance information indicating a distance between the imaging device and the one or more agricultural crops, the identification unit comprises a correction unit, the correction unit corrects the in-image position based on the tilt angle and the distance indicated in the distance information, and the estimation unit estimates the second absolute position based on the first absolute position information and the corrected in-image position.

7. An information processing device as described in claim 1 or 2, further comprising: a size acquisition unit that acquires size information indicating the size of the agricultural crop; and a distance calculation unit that calculates a distance between each of the one or more agricultural crops and the imaging device based on the size information and the image, wherein the identification unit comprises a correction unit, which corrects the in-image position based on the tilt angle and the calculated distance, and the estimation unit estimates the second absolute position based on the first absolute position information and the corrected in-image position.

8. The information processing device according to claim 7, further comprising: a device information acquisition unit that acquires device information related to the imaging device; and a resolution calculation unit that calculates a resolution of the image based on the distance and the device information.

9. An information processing method in which a computer acquires first absolute position information indicating a first absolute position of an imaging device that images one or more agricultural crops in a specific area of ​​a field, acquires an image captured by the imaging device, acquires tilt information indicating the tilt of the imaging device measured by a measurement unit, identifies an in-image position within the image for each of the one or more agricultural crops shown in the image based on the image and the tilt information, and estimates a second absolute position of each of the one or more agricultural crops shown in the image based on the first absolute position information and the in-image position.

10. A program causing a computer to execute the following functions: a position acquisition function for acquiring first absolute position information indicating an absolute position of an imaging device that images one or more agricultural crops in a specific area of ​​a field; an image acquisition function for acquiring an image captured by the imaging device; a tilt acquisition function for acquiring tilt information indicating the tilt of the imaging device measured by a measurement unit; a determination function for identifying an in-image position within the image for each of one or more agricultural crops shown in the image based on the image and the tilt information; and an estimation function for estimating a second absolute position of each of one or more agricultural crops shown in the image based on the first absolute position information and the in-image position.

Citation Information

Patent Citations

  • Information processor, agricultural vehicle and information processing method

    JP2021100384A

  • Information processing system, information processing apparatus, and program

    JP2021170171A

  • Information processing apparatus, program, system, and method for detecting grape grain from image

    JP2021189718A

  • Artificial intelligence (AI) learning device, fruit thinning object estimation device, estimation system, and program

    JP2022114418A

  • Information processing device, information processing method and program

    JP2022156431A