Information processing device, agricultural work support system, and information processing method

Abstract

The present invention determines compound information of a cultivated plant easily and accurately.　An information processing device (41) is provided with: an image acquisition unit (43) that acquires a captured image (51) of an object (101) that is at least a part of a cultivated plant (100); a first determination unit (44) that, on the basis of the captured image (51) acquired by the image acquisition unit (43), determines optical characteristics at predetermined position coordinates (Cn) of the object (101); and a second determination unit (45) that determines compound information pertaining to a compound at the position coordinates (Cn) on the basis of the optical characteristics at the position coordinates (Cn) determined by the first determination unit (44) and predetermined determination information indicating a relationship between the optical characteristics and a compound.

Description

Information processing device, agricultural work support system, and information processing method

[0001] The present invention relates to an information processing device, an agricultural work support system, and an information processing method.

[0002] The portable measuring device disclosed in Patent Document 1 includes a halogen light source that irradiates a measurement object with measurement light including near-infrared light, and a measuring unit that receives transmitted light and / or reflected light from the measurement object to measure the quality of the measurement object.

[0003] Japanese Patent Publication No. 2020-101409

[0004] The internal quality of the measurement object (e.g., fruits and vegetables) in Patent Document 1 can be measured non-destructively, and furthermore, the measurement can be performed without bringing the crop to be measured back from the field, etc. However, in the portable measurement device in Patent Document 1, the measurement unit receives transmitted light and / or reflected light from one point on the measurement object and uses the received measurement light as spectral data for analysis, so there is room for improvement in the accuracy of quality measurement.

[0005] The present invention has been made to solve the problems of the conventional technology, and aims to provide an information processing device, an agricultural work support system, and an information processing method that can easily and accurately determine compound information of cultivated plants.

[0006] An information processing device according to one aspect of the present invention includes an image acquisition unit that acquires an image of an object that is at least a part of a cultivated plant; a first determination unit that determines light characteristics of the object at a predetermined position coordinate based on the image acquired by the image acquisition unit; and a second determination unit that determines compound information related to the compound at the position coordinate based on the light characteristics at the position coordinate determined by the first determination unit and predetermined judgment information indicating the relationship between the light characteristics and the compound.

[0007] The second determination unit may determine, as the compound information, an amount of the compound at the position coordinates.

[0008] The first determination unit may determine the light characteristics of the object at a plurality of the position coordinates based on the captured image acquired by the image acquisition unit, and the second determination unit may determine the compound information at each of the position coordinates based on the light characteristics at each of the plurality of the position coordinates determined by the first determination unit and the determination information.

[0009] The information processing device may include an estimation unit that estimates a structure of the object based on the compound information for each of the plurality of position coordinates determined by the second determination unit.

[0010] When the target object includes a fruit of the cultivated plant, the estimation unit may estimate at least one of the skin, flesh, and seeds of the fruit as the structure of the fruit.

[0011] The first determination unit may determine the light characteristics at three-dimensional position coordinates of the object as the position coordinates based on the captured image acquired by the image acquisition unit.

[0012] The first determination unit may determine the light characteristics of the object at least at position coordinates inside the object, based on the captured image acquired by the image acquisition unit.

[0013] The image acquisition unit may acquire a plurality of captured images captured with reflected waves of different wavelengths, the first determination unit may determine spectral data at the position coordinates based on the plurality of captured images acquired by the image acquisition unit, and the second determination unit may determine the compound information at the position coordinates based on the spectral data at the position coordinates determined by the first determination unit and the determination information.

[0014] The first determination unit may determine the light characteristics from the captured image acquired by the image acquisition unit using a ray tracing method.

[0015] The first determination unit may use ray tracing as the ray tracing method.

[0016] An agricultural work support system according to one aspect of the present invention includes an imaging device that captures the image of the object, the information processing device, and a display device that displays information based on the compound information determined by the second determination unit.

[0017] The display device may display at least one of information on the moisture content, sugar content, and acidity of the object as information based on the compound information.

[0018] An information processing method according to one aspect of the present invention includes a first step in which an imaging device captures an image of an object that is at least a part of a cultivated plant; a second step in which a first determination unit determines light characteristics at a predetermined position coordinate of the object based on the image captured in the first step; and a third step in which the second determination unit determines compound information related to the compound at the position coordinate based on the light characteristics at the position coordinate determined in the second step and judgment information indicating the relationship between the light characteristics and a compound.

[0019] In the information processing method, in the first step, the imaging device may capture a plurality of images of the object using reflected waves of different wavelengths; in the second step, the first determination unit may determine spectral data at the position coordinates based on the plurality of captured images; and in the third step, the second determination unit may determine the compound information at the position coordinates based on the spectral data and the determination information.

[0020] The information processing method may include a fourth step in which, in the second step, the first determination unit determines the light characteristics at a plurality of the position coordinates of the object based on the captured image; in the third step, the second determination unit determines the amount of the compound at each of the plurality of position coordinates as the compound information at the position coordinates based on the light characteristics at each of the position coordinates and the determination information; and an estimation unit estimates a structure of the object based on the compound information at each of the plurality of position coordinates determined in the third step.

[0021] According to the information processing device, agricultural work support system, and information processing method, compound information of cultivated plants can be determined easily and accurately.

[0022] 1 is a schematic diagram of an agricultural work support system. FIG. 2 is a diagram showing how an object is imaged by an imaging device. FIG. 3 is a diagram explaining the flow of data in the agricultural work support system. FIG. 4 is a diagram for explaining ray tracing. FIG. 5 is a diagram schematically showing the relationship between each position coordinate of an object and light characteristics. FIG. 6 is a diagram showing the relationship between light characteristics and the amount of a predetermined compound. FIG. 7 is a first diagram schematically showing the relationship between predetermined position coordinates of an object and the amount of a predetermined compound. FIG. 8 is a second diagram schematically showing the relationship between predetermined position coordinates of an object and the amount of a predetermined compound. FIG. 9 is a diagram showing an example of the structure of a compound at each position coordinate determined by a second determination unit. FIG. 10 is a diagram showing an example of the structure of a compound at predetermined position coordinates of an object, and an example of the structure of a compound with a predetermined structure. FIG. 11 is a diagram showing an example of the structure of each position coordinate estimated by an estimation unit. FIG. 12 is an example of a support screen displayed by a display device. FIG. 13 is a diagram explaining the flow of a series of processes performed by the agricultural work support system.

[0023] An embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a schematic diagram of an agricultural work support system S. As shown in Fig. 1, the agricultural work support system S includes an imaging device 21 that captures an image of an object 101, which is at least a part of a cultivated plant 100; an information processing device 41 that acquires an image 51 captured by the imaging device 21 and determines light characteristics (e.g., light intensity, luminance, absorbance, etc.) at a predetermined position coordinate Cn (n = 1, 2, 3, ...) of the object 101 based on the image 51, thereby determining compound information related to the compound at the position coordinate Cn; and a display device 31 that displays information based on the compound information determined by the information processing device 41.

[0024] Here, the cultivated plant 100 is a plant cultivated by a farmer, such as a crop grown in a field such as a rice paddy, a field, or an orchard. Therefore, the imaging device 21 captures an image of an object 101 such as a fruit, a leaf, a stem, a seed, or a bud of the cultivated plant 100, and the information processing device 41 can determine compound information based on the captured image 51 captured by the imaging device 21.

[0025] The display device 31 can display information (quality information) on at least one of the moisture content, sugar content, and acidity of the object 101, allowing the farm worker to non-destructively grasp the quality information of the object 101. Note that the object 101 imaged by the imaging device 21 is not limited to at least a part of the cultivated plant 100, but may also be at least a part of a wild plant that grows naturally without being cultivated by a farm worker.

[0026] In the agricultural work support system S shown in Figure 1, an imaging device 21 and a display device 31 are provided in a portable terminal device 10 (support terminal) that can be carried by an agricultural worker or the like, and an information processing device 41 is provided in a fixed terminal device 40 (e.g., a server device) such as a fixed computer external to the support device.

[0027] 1 , the imaging device 21, the display device 31, and the information processing device 41 may be provided in the same terminal device or in separate terminal devices. For example, the imaging device 21 may be provided in a portable terminal device (imaging terminal), and the information processing device 41 and the display device 31 may be provided in a fixed terminal device (support terminal). In such a case, the terminal device is, for example, a personal computer used by a farm worker or a manager.

[0028] Alternatively, the imaging device 21 may be provided in a portable terminal device (imaging terminal) or a work machine (such as a harvesting machine for harvesting crops or a management machine for spraying pesticides), the display device 31 may be provided in a portable terminal device (support terminal), and the information processing device 41 may be provided in a fixed terminal device (e.g., a server device). Furthermore, the imaging device 21, the information processing device 41, and the display device 31 may be provided in a single terminal device (support terminal). In these cases, the support terminal is a portable terminal device such as a personal computer, tablet terminal, or smartphone with relatively high processing power. In the following explanation, the example of the agricultural work support system S shown in FIG. 1 will be mainly described, and detailed explanations of other variations will be omitted.

[0029] First, the imaging device 21 and the display device 31 will be described using a portable support terminal 10 as an example, as shown in Fig. 1. In the example shown in Fig. 1, the support terminal 10 has the imaging device 21 and a main body (a portion including the display device 31 and the like other than the imaging device 21) connected by a cable, and the main body is a tablet-shaped terminal device.

[0030] The support terminal 10 has a first control device 11 (processing circuit) including one or more processors. The first control device 11 is a controller of the support terminal 10 and performs various controls related to the support terminal 10. The first control device 11 is communicably connected to multiple devices (such as the image capture device 21 and the display device 31) installed in the support terminal 10 and can control these devices. For example, the first control device 11 controls (operates) the image capture of the object 101 by the image capture device 21 based on a signal (operation signal) input from an input interface. That is, in this embodiment, the first control device 11 also controls the image capture device 21, the display device 31, etc.

[0031] The first control device 11 includes one or more memories, various analog circuits, various digital circuits, etc. The one or more memories store (memorize) software programs and various data to be executed by one or more processors. The first control device 11 can read software programs from one or more memories using one or more processors and execute various processes based on the software programs. Note that the first control device 11 may also be able to execute various processes based on predetermined logic circuits using one or more processors.

[0032] The processor is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC).

[0033] The first control device 11 may execute various processes by having multiple physically separated processors cooperate with each other, and the configuration is not limited to the above-described configuration. In such a case, the multiple processors may be mounted on one or multiple computers that are physically separated from the support terminal 10, and these processors may be connected to each other via a network such as a LAN, a WAN, or the Internet so as to be able to communicate with each other.

[0034] In addition, the software program may be stored on a recording medium communicatively connected to the first control device 11 or on an external server device 40 connected via the network, and installed from there into the memory.

[0035] 1 , the support terminal 10 has a first storage unit 12. The first storage unit 12 is a storage medium such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and stores various information related to the support terminal 10. For example, the first storage unit 12 can store data (image data) of a captured image 51 of an object 101 captured by an imaging device 21.

[0036] As shown in FIG. 1 , the support terminal 10 includes a first communication device 13. The first communication device 13 is a device that transmits various data to an external device (e.g., a server device 40) of the support terminal 10 and receives data transmitted from the external device. The first communication device 13 wirelessly communicates with the external device via, for example, Wi-Fi (Wireless Fidelity, registered trademark) of the IEEE 802.11 series, a communication standard, a mobile phone communication network, or a data communication network. For example, the first communication device 13 transmits imaging data to the server device 40. The first communication device 13 also receives information from the server device 40 based on the compound information of the target object 101 determined by the information processing device 41.

[0037] 1 , the support terminal 10 has an imaging device 21. The imaging device 21 can capture an image 51 of an object 101, which is at least a part of a cultivated plant 100. The imaging device 21 captures, for example, a still image of the object 101 as the captured image 51. Note that it is sufficient for the imaging device 21 to capture at least the captured image 51 of the object 101, and the imaging device 21 may also capture a moving image of the object 101 and generate moving image data.

[0038] The imaging device 21 can image the object 101 using reflected waves (reflected light) of infrared rays (near infrared, mid-infrared), visible light, ultraviolet rays (near ultraviolet, far ultraviolet), other electromagnetic waves, etc. The imaging device 21 can image the object 101 using reflected waves of one or more wavelengths L. The imaging device 21 can image the object 101 using reflected light of wavelengths L in a relatively narrow, predetermined band (wavelength band). Each band of reflected light imaged by the imaging device 21 may be set to an appropriate band for determining compound information from the optical characteristics of the reflected light. Note that, although the present embodiment will be described mainly in terms of a case where the imaging device 21 images the object 101 using reflected waves of multiple wavelengths L, it is sufficient for the imaging device 21 to be able to image the object 101 using reflected waves of at least one wavelength L.

[0039] The imaging device 21, for example, irradiates the object 101 with measurement light of a relatively narrow predetermined band wavelength L using the irradiation unit 23, and captures an image of the object 101 using light reflected from the measurement light. Below, the imaging device 21 will be described in detail using an example in which the imaging device 21 has the irradiation unit 23.

[0040] 1, the imaging device 21 has an imaging unit 22 and an irradiation unit 23. The imaging unit 22 converts an optical image formed by incident light into an image signal and generates a captured image 51 from the image signal. The imaging unit 22 includes a lens 22a and an imaging element 22b. The lens 22a of the imaging unit 22 collects reflected light from the object 101, and the collected light beam forms an image on the light receiving surface of the imaging element 22b.

[0041] The image sensor 22b converts the optical image formed on the light receiving surface into an electrical signal (image signal). Examples of the image sensor 22b include a CMOS (Complementary Metal-Oxide Semiconductor) type and a CCD (Charge Coupled Device) type. The image sensor 22b has wavelength bands that can convert the optical image formed on the light receiving surface, i.e., the incident light, and wavelength bands that cannot. Therefore, the image sensor 21 has one or more image sensors 22b corresponding to appropriate wavelength bands for determining compound information from the optical characteristics of reflected light. For example, when the image sensor 21 captures an image of the object 101 using reflected light ranging from near ultraviolet to mid-infrared, the image sensor 21 has an image sensor 22b for near ultraviolet light and an image sensor 22b for visible light to mid-infrared light.

[0042] The irradiating unit 23 irradiates the object 101 with measurement light of a wavelength L in a relatively narrow predetermined band. Specifically, the irradiating unit 23 can irradiate the object 101 with measurement light of a plurality of different bands of wavelength L. The irradiating unit 23 irradiates the object 101 with measurement light of a wavelength L in a desired band based on a control signal from the first control device 11. The irradiating unit 23 irradiates the measurement light in a direction opposite to the incident direction of the reflected light on the lens 22a.

[0043] The irradiation unit 23 has one or more light sources 23a. The one or more light sources 23a are arranged around the lens 22a of the imaging unit 22. Examples of the light source 23a include an LED light source, a xenon lamp, a D2 lamp (deuterium lamp), a mercury lamp, etc. When the irradiation unit 23 has an LED light source as the light source 23a, the irradiation unit 23 has multiple LED light sources that are turned on with light of different wavelength bands L, and switches the multiple LED light sources on and off to irradiate measurement light of a desired wavelength band L. The irradiation unit 23 may have multiple LED light sources that are turned on with the wavelength L of each band. In such a case, it is preferable that the LED light sources are arranged at equal intervals around the outer periphery of the lens 22a.

[0044] The configuration of the irradiating unit 23 is not limited to the above example, as long as it can irradiate measurement light of at least the desired band of wavelengths L. For example, the irradiating unit 23 may irradiate measurement light of the desired band of wavelengths L by switching between various filters (e.g., band-pass filters) that transmit only light of a predetermined band among the light irradiated from the light source 23 a.

[0045] With the above configuration, the imaging device 21 can capture images of the object 101 with reflected light of multiple wavelengths L by capturing images of the object 101 multiple times with reflected light of different wavelengths L.

[0046] In the above example, the imaging device 21 captures an image of the object 101 using reflected light from measurement light irradiated by the irradiation unit 23. However, it is sufficient if the imaging device 21 can capture an image of the object 101 using reflected light of a relatively narrow, predetermined band of wavelengths L. Therefore, light of a relatively wide band of wavelengths L (e.g., sunlight) may be irradiated onto the object 101, and the imaging device 21 may capture an image of the object 101 using reflected light that has been dispersed into a predetermined band of wavelengths L using a spectroscope. In such a case, the imaging device 21 may or may not include an irradiation unit 23 that irradiates measurement light of a relatively wide band of wavelengths L. The irradiation unit 23 may, for example, have a plurality of LED light sources that are lit at different bands of wavelengths L, and these plurality of LED light sources are lit simultaneously to irradiate measurement light of a relatively wide band of wavelengths L.

[0047] Furthermore, in the above example, the image capturing device 21 captures an image of the object 101 multiple times with reflected light of different wavelengths L, thereby capturing an image of the object 101 with reflected light of multiple wavelengths L. However, the image capturing device 21 may also capture an image of the object 101 simultaneously with reflected light of multiple wavelengths L by dispersing light into different wavelengths L, as in a multispectral camera or hyperspectral camera. In such a case, the data (data cube) obtained by the image capturing device 21 capturing an image of the object 101 includes wavelength information at each position in the image, and therefore it is possible to extract from the data cube the captured image 51 captured with reflected light of each wavelength L.

[0048] The imaging device 21 captures an image of the object 101 from at least one direction using reflected waves of different wavelengths L (more specifically, reflected light of the wavelengths L in each band). It is preferable that the imaging device 21 captures an image of the object 101 from all directions using reflected light of the wavelengths L in each band. FIG. 2 is a diagram showing how the imaging device 21 captures an image of the object 101. In FIG. 2, the object 101 is shown as a single bunch of grapes (in other words, one or more fruits, hereinafter sometimes simply referred to as "fruit") on a grape vine grown in an orchard. In FIG. 2, the single bunch of grapes that is the object 101 is indicated by a solid line, and other parts (branches and leaves) are indicated by two-dot chain lines.

[0049] 2 also shows a case where the object 101 is surrounded by a light shielding body 25 that blocks light (external disturbances such as sunlight and lighting) directed toward the object 101 from outside the object 101. The light shielding body 25 preferably blocks infrared light, visible light, and ultraviolet light. In FIG. 2, the light shielding body 25 blocks external disturbances such as sunlight and lighting, but in an environment where such external disturbances do not exist (for example, at night when the sun has set or indoors with the lights turned off), the object 101 does not need to be surrounded by the light shielding body 25.

[0050] In FIG. 2 , the imaging device 21 captures 360° images of the object 101 (a bunch of fruit), including arrows A1 to A6, using reflected light of each wavelength L. Each of the arrows A1 to A6 in FIG. 2 indicates a direction from the periphery of the object 101 (a bunch of fruit, each fruit) toward the bunch. Specifically, arrow A1 in FIG. 2 indicates one vertical direction (first direction, upward), and arrow A2 indicates the other vertical direction (second direction, downward). Arrow A3 in FIG. 2 indicates one horizontal direction (third direction) perpendicular to the vertical direction, and arrow A4 indicates the opposite direction to the third direction (fourth direction). Arrow A5 in FIG. 2 indicates a horizontal direction (fifth direction) perpendicular to the vertical direction and perpendicular to the third and fourth directions, and arrow A6 indicates the opposite direction to the fifth direction (sixth direction).

[0051] Furthermore, in order to capture images of all possible blind spots, such as unevenness, of the object 101, the imaging device 21 may capture images of the object 101 from different distances in each direction in addition to capturing images of the 360° circumference of the object 101 including arrows A1 to A6. Therefore, in the example shown in Fig. 2, the imaging device 21 moves around the object 101 in a spiral around a central axis extending in a predetermined direction (the vertical direction in Fig. 2) (R1 in Fig. 2), captures images of the entire 360° circumference of the object 101, then changes the distance from the object 101 to continue imaging, and moves further in a spiral to capture images of the entire 360° circumference of the object 101 (R2 in Fig. 2).

[0052] In this case, the imaging device 21 may change the band of the reflected light to be captured each time the imaging device 21 moves around the object 101 and captures an image of the periphery of the object 101, or may repeatedly change the band of the reflected light to be captured while moving around the object 101. When the imaging device 21 changes the band of the reflected light to be captured each time the imaging device 21 moves around the object 101 and captures an image of the periphery of the object 101, the irradiation unit 23 changes the band of the wavelength L of the measurement light each time imaging of the periphery of the object 101 is completed. Furthermore, when the band of the reflected light to be captured while the imaging device 21 moves around the object 101 repeatedly, the irradiation unit 23 changes the band of the wavelength L of the measurement light each time the imaging device 21 captures a predetermined number of images (e.g., one image) of the object 101, thereby repeatedly changing the band of the reflected light to be captured while moving around the object 101. In such a case, the irradiation unit 23 may repeatedly change the band of the reflected light at predetermined elapsed times.

[0053] 2, for convenience of explanation, arrows A1 to A6 are used to indicate the directions in which the imaging device 21 captures the object 101, but it goes without saying that it is preferable for the imaging device 21 to capture images of the object 101 from other directions (for example, the direction between arrows A1 and A3, the direction between arrows A1 and A6, etc.). Furthermore, it is preferable for the imaging device 21 to capture images of the object 101 (fruit) from 360° around it, but it is not necessary to capture images of the entire 360° around it. Furthermore, in FIG. 2, a single imaging device 21 is moved to capture images of the object 101's surroundings, but multiple imaging devices 21 may surround the object 101 and capture images of the object 101's surroundings.

[0054] As described above, the imaging device 21 captures images of the 360° surrounding the object 101 (fruit), including the arrows A1 to A6, using reflected light of each band of wavelength L as the captured images 51 required for the information processing device 41 to perform a single process of determining compound information about the object 101 (hereinafter, sometimes referred to as the determination process). Therefore, the imaging device 21 captures multiple images 51 (for example, at least about 50 images) of the object 101 (fruit) for each reflected light of each band of wavelength L.

[0055] The imaging of the object 101 by the imaging device 21 may be performed entirely manually by the farmworker, or the first control device 11 may appropriately assist the farmworker in imaging or perform the imaging automatically.

[0056] One or more captured images 51 captured by the imaging device 21 are transmitted to the server device 40 by the first communication device 13. Specifically, data (imaging data) of the captured image 51 captured by the imaging device 21 is associated with information (band information) relating to the band of wavelength L of reflected light when the captured image 51 is captured. In the present embodiment, when the imaging device 21 captures the captured image 51, the first control device 11 associates the band information with the data (imaging data) of the captured image 51 and stores the data as processing data in the first storage unit 12. The first control device 11 also controls the first communication device 13 to transmit the processing data stored in the first storage unit 12 to the server device 40.

[0057] In addition to the band information, the data of the captured image 51 captured by the imaging device 21 (imaging data) may be associated with information that can be used to determine compound information of the object 101 in the information processing device 41. For example, the input interface may accept input of information about the object 101 (object information), and the first control device 11 may associate the object information with the imaging data. Examples of the object information include the type of cultivated plant 100 and a part of the object 101 (e.g., fruit, leaf, stem, etc.).

[0058] The object information may also be set without the farmer's own input operation. For example, a tag printed with an image code storing information for determining the type of cultivated plant 100 may be attached to the cultivated plant 100, and the first control device 11 may acquire the object information (the type of cultivated plant 100) by reading the image code with the imaging device 21. The object information may also include the date and time when the captured image 51 of the object 101 was captured. In addition, if the imaging device 21 (or the support terminal 10 on which the imaging device 21 is installed) is equipped with a position detection device 14 that detects its own position using satellite signals or the like, the object information may also include position information (latitude and longitude) of the location where the captured image 51 was captured. In the example shown in FIG. 1 , the support terminal 10 is equipped with the position detection device 14.

[0059] Furthermore, if the imaging device 21 (or the support terminal 10 on which the imaging device 21 is provided) has an inertial measurement unit (IMU) 24, camera parameters (external parameters) may be associated with the imaging data. The inertial measurement unit 24 has an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity, and the like. In the example shown in FIG. 1, the imaging device 21 is provided with the inertial measurement unit 24.

[0060] As shown in FIG. 1 , the support terminal 10 has a display device 31. The display device 31 can display information (quality information) based on the compound information determined by the information processing device 41. In addition to the quality information, the display device 31 may also display an image 51 captured by the imaging device 21 or display setting information for the support terminal 10. The information display device 31 is, for example, a liquid crystal display panel, an organic EL panel, or the like. The display device 31 is connected to the first control device 11 and can perform screen display based on a control signal from the first control device 11.

[0061] In this embodiment, the display device 31 has a display screen such as a touch display that can be operated for input, and outputs input information (such as an instruction to start capturing an image of the object 101 by the imaging device 21 and an instruction to transmit the captured image data) to the first control device 11. In this way, the display device 31 may serve as both an output interface that outputs information and an input interface that accepts input of information. Therefore, the farm worker can operate the display device 31 to capture images by the imaging device 21 and to operate the settings of the support terminal 10. Note that if the display device 31 does not also serve as an input interface, the support terminal 10 may have a physical switch or the like that accepts separate operations.

[0062] Next, the information processing device 41 will be described using a fixed terminal device 40 (server device) as an example, as shown in Fig. 1. The server device 40 is installed, for example, at an agricultural machinery manufacturer, an agricultural cooperative, a management company, or the like.

[0063] As shown in FIG. 1 , the server device 40 includes an information processing device 41 (second control device). The information processing device 41 includes one or more memories 42 (storage devices), various analog circuits, various digital circuits, etc. The one or more storage devices 42 store (store) software programs and various data to be executed by one or more processors. Like the first control device 11, the information processing device 41 can also read software programs from one or more storage devices 42 using one or more processors and execute various processes based on the software programs. Also, like the first control device 11, the information processing device 41 can also execute various processes based on predetermined logic circuits using one or more processors, and detailed description thereof will be omitted.

[0064] 1 , the server device 40 has a second storage unit 48. The second storage unit 48 is a database configured with a solid-state drive (SSD), a hard disk drive (HDD), etc., and stores various information related to determining compound information based on optical characteristics. For example, the database 48 can store data (imaging data) of an image 51 captured by the imaging device 21 of the target object 101, and determined compound information.

[0065] 1 , the server device 40 has a second communication device 49. The second communication device 49 is a device that transmits various data to an external device (e.g., the support terminal 10) of the server device 40 and receives data transmitted from the external device. The second communication device 49 is capable of wireless or wired communication with the first communication device 13. Specifically, the second communication device 49 performs wireless communication with the external device via, for example, Wi-Fi (Wireless Fidelity, registered trademark) of the IEEE 802.11 series, which is a communication standard, a mobile phone communication network, a data communication network, or the like.

[0066] For example, the second communication device 49 receives imaging data (processing data) from the support terminal 10 via the first communication device 13. Furthermore, the second communication device 49 transmits information based on the compound information of the object 101 to the support terminal 10 via the first communication device 13. The processing data received by the second communication device 49 from the support terminal 10 via the first communication device 13 is stored in the database 48.

[0067] Hereinafter, the determination of compound information based on the captured image 51 by the information processing device 41 will be described in detail with reference to Figures 1, 3, etc. Figure 3 is a diagram illustrating the flow of data in the agricultural work support system S. As shown in Figures 1 and 3, the information processing device 41 includes an image acquisition unit 43, a first determination unit 44, and a second determination unit 45. In addition to these, the information processing device 41 may also include an estimation unit 46. The image acquisition unit 43, the first determination unit 44, the second determination unit 45, and the estimation unit 46 are, for example, software programs stored in the memory or storage device 42, and are executed by the information processing device 41.

[0068] The image acquisition unit 43 acquires a captured image 51 obtained by capturing an image of the object 101. Specifically, the image acquisition unit 43 acquires a plurality of captured images 51 captured using reflected waves of different wavelengths L. More specifically, the image acquisition unit 43 acquires a plurality of captured images 51 obtained by capturing an image of the object 101 from different directions for each band of reflected light of the wavelength L. As shown in FIG. 3 , the information processing device 41 refers to processing data stored in the database 48 and temporarily stores the processing data in the storage device 42 (memory). The image acquisition unit 43 acquires the plurality of captured images 51 by acquiring the processing data stored in the storage device 42. At this time, the image acquisition unit 43 may shape (convert) the processing data into data that is easy for the first determination unit 44 to process.

[0069] The first determination unit 44 determines light characteristics at a predetermined position coordinate Cn of the object 101 based on the captured image 51 acquired by the image acquisition unit 43. Specifically, the first determination unit 44 determines light characteristics at a plurality of position coordinates Cn based on the captured image 51. At this time, the plurality of position coordinates Cn are three-dimensional position coordinates Cn of the object 101. Furthermore, these position coordinates Cn are at least position coordinates Cn inside the object 101. Since the processing data includes a plurality of captured images 51 captured with reflected waves of different wavelengths L, the first determination unit 44 can determine spectral data at the position coordinate Cn based on the plurality of captured images 51. The first determination unit 44 will be described in detail below.

[0070] The first determination unit 44 determines the optical characteristics from the captured image 51 acquired by the image acquisition unit 43 using a ray tracing method. An example of the ray tracing method is ray tracing. The first determination unit 44 performs ray tracing on each pixel of each captured image 51 acquired by the image acquisition unit 43. FIG. 4 is a diagram for explaining ray tracing. The first determination unit 44 defines the trajectory of a ray traveling from a predetermined reference P (observation point) to each pixel (x, y). After defining the trajectory of the ray, the first determination unit 44 integrates (adds) the optical characteristics of each position (position coordinate Cn) through which the ray passes to calculate the optical characteristics of each pixel. Specifically, the first determination unit 44 performs ray tracing on each pixel using the following equation (1):

[0071] where x: x coordinate of a pixel in the captured image 51, y: y coordinate of a pixel in the captured image 51, θ: angle of a light ray with respect to the captured image 51, In: optical characteristics at each position (position coordinate Cn), and σn: transmittance at each position (position coordinate Cn).

[0072] The first determination unit 44 calculates the light characteristics and transmittance at each position coordinate Cn from all the light characteristics calculated using simultaneous equations, thereby determining the relationship between each position coordinate Cn on the object 101 and the light characteristics (see FIG. 5 ).

[0073] More specifically, when calculating the light characteristics and transmittance at each position coordinate Cn by ray tracing, the first determination unit 44 extracts multiple captured images 51 for each wavelength L of the reflected light at the time of capture from the processing data (multiple captured images 51) acquired by the image acquisition unit 43, and reconstructs (restores) a three-dimensional space in which the captured images 51 are captured for each wavelength L. That is, when the processing data acquired by the image acquisition unit 43 includes multiple captured images 51 captured with near-infrared light and multiple captured images 51 captured with mid-infrared light, the first determination unit 44 reconstructs a three-dimensional space of the multiple captured images 51 captured with near-infrared light and a three-dimensional space of the multiple captured images 51 captured with mid-infrared light, respectively. The first determination unit 44 reconstructs the three-dimensional space of the multiple captured images 51 using a three-dimensional reconstruction method such as Neural Radiance Fields (NeRF) or Gaussian Splatting from the multiple captured images 51 extracted for each wavelength L.

[0074] Note that the first determination unit 44 may reconstruct the three-dimensional space by any method other than NeRF or Gaussian Splatting, as long as it can reconstruct the three-dimensional space of the multiple captured images 51 extracted for each wavelength L. In this embodiment, the first determination unit 44 reconstructs the three-dimensional space of the multiple captured images 51 extracted for each wavelength L using NeRF. At this time, the first determination unit 44 reconstructs the three-dimensional space using NeRF based on camera parameters included in the processing data and associated with each captured image 51. Note that if the processing data does not include camera parameters, the camera parameters may be estimated using SFM (Structure from Motion) such as COLMAP.

[0075] This allows the first determination unit 44 to determine the optical characteristics of the position coordinate Cn that cannot be observed in the captured image 51 acquired by the image acquisition unit 43, i.e., the position coordinate Cn inside the object 101. As described above, the first determination unit 44 performs three-dimensional reconstruction for each reflected light of each band of wavelength L, and can determine the optical characteristics for each reflected light of each band of wavelength L at each position coordinate Cn inside the object 101 from the reconstructed three-dimensional space. This allows the first determination unit 44 to determine the distribution of the optical characteristics for each reflected light of each band of wavelength L in the object 101. In other words, the first determination unit 44 can determine the spectral data for each position coordinate Cn inside the object 101.

[0076] The second determination unit 45 determines compound information about the compound at the position coordinate Cn based on the optical characteristics at the position coordinate Cn determined by the first determination unit 44 and predetermined determination information. The second determination unit 45 determines compound information at each position coordinate Cn based on the optical characteristics at each of the multiple position coordinates Cn determined by the first determination unit 44 and the determination information. In this embodiment, the first determination unit 44 determines spectral data for each position coordinate Cn, and therefore the second determination unit 45 determines compound information at the position coordinate Cn based on the spectral data and the determination information. The second determination unit 45 determines the amount of the compound at the position coordinate Cn as the compound information. Note that the compound information may be information about at least a compound, and the amount of the compound may be the amount of a single compound or the amount of a predetermined mixture containing multiple compounds.

[0077] The determination information indicates the relationship between the optical characteristics and the compound. In this embodiment, the determination information is stored in the storage device 42. The determination information is a calibration formula (calibration curve) that indicates the relationship between the optical characteristics of the wavelength L of each band and the amount of the compound (see FIG. 6). FIG. 6 is a diagram showing the relationship between the optical characteristics and the amount of a predetermined compound. FIG. 6 shows a calibration formula in which the amount of the compound increases (approximately proportional relationship) as the value of the optical characteristics increases. Note that the calibration formula is defined for each single compound or for each predetermined mixture consisting of multiple compounds, and for each optical characteristic of the wavelength L of one or multiple bands.

[0078] The second determination unit 45 determines the amount of a compound based on the optical characteristics (spectral data) at each position coordinate Cn determined by the first determination unit 44 and the calibration formula stored in the storage device 42. The second determination unit 45 determines the amount of each compound for each position coordinate Cn using the calibration formula corresponding to the wavelength L of each band. This allows the second determination unit 45 to determine the amount of a compound contained in each position coordinate Cn of the object 101, i.e., the composition of the compound (see FIGS. 7A and 7B). FIGS. 7A and 7B are schematic diagrams illustrating the relationship between predetermined position coordinates C1 and C2 of the object 101 and the amount of a predetermined compound. FIG. 7A shows the amount of a compound at the first position coordinate C1, and FIG. 7B shows the amount of a compound at the second position coordinate C2.

[0079] In this embodiment, each calibration formula is stored in the second storage unit 48 (database) in association with each compound. The second determination unit 45 may refer to (acquire) an appropriate calibration formula to determine compound information for the target object 101 based on the target object information included in the processing data, and temporarily store the calibration formula in the storage unit 42 (memory). Furthermore, each calibration formula (determination information) may be stored externally (e.g., an external server device or storage medium capable of wireless or wired communication with the information processing device 41) other than the database 48. In such a case, a device that receives information used by the information processing device 41 for information processing, such as the second communication device 49 of the server device 40, acquires (receives) each calibration formula (determination information) from the outside. For this reason, in this embodiment, the second communication device 49 also serves as an acquisition device for acquiring determination information from the outside, but this is not limited thereto. That is, the information processing device 41 (second determination unit 45) may acquire determination information from the outside via an acquisition device other than the second communication device 49.

[0080] For example, if the object 101 is a grape, the second determination unit 45 acquires corresponding calibration formulas from the second storage unit 48 to determine, for example, moisture, sugar content, acidity, pH, polyphenols, etc., and stores them in the storage unit 42. Furthermore, if the object 101 is a tomato, the second determination unit 45 acquires corresponding calibration formulas from the second storage unit 48 to determine, for example, moisture, sugar content, acidity, pH, lycopene, etc., and stores them in the storage unit 42.

[0081] As described above, the second determination unit 45 can determine the composition of compounds at each position coordinate Cn by determining the amount of the compound using the calibration formula for each position coordinate Cn and for each optical characteristic of the wavelength L in each band (see FIG. 8 ). This allows the second determination unit 45 to determine the distribution of each compound in the target object 101.

[0082] The estimation unit 46 estimates the structure of the object 101 based on the compound information for each of the multiple position coordinates Cn determined by the second determination unit 45. When the object 101 includes a fruit of the cultivated plant 100, the estimation unit 46 estimates at least one of the peel, flesh, and seeds of the fruit as the fruit structure. Figure 9 is a diagram showing an example of the structure of a compound at a predetermined position coordinate Cn of the object 101 and an example of the structure of a compound with a predetermined structure. The left diagram of Figure 9 shows the compound information at the predetermined position coordinate Cn determined by the second determination unit 45, and the right diagram of Figure 9 shows an example of the structure of a compound in the predetermined structure of the object 101.

[0083] The second storage unit 48 stores a first table relating each structure and the amount of a compound contained in the structure. The first table is defined for each object 101, for example. If the object 101 is a fruit, the first table defines the peel, pulp, seeds, etc., in association with the amounts of compounds that make them up. The estimation unit 46 may refer to (acquire) the first table corresponding to the object 101 from the database 48 based on the object information included in the processing data acquired by image acquisition, and temporarily store the first table in the storage unit 42 (memory).

[0084] The estimation unit 46 refers to the first table and estimates which structure each position coordinate Cn corresponds to based on the compounds contained in each structure in the first table and the amounts of those compounds. For example, the estimation unit 46 compares (matches) the compounds contained in each structure in the first table and the amounts of those compounds with the amounts of those compounds at each position coordinate Cn, and estimates the structure at the position coordinate Cn to be the one with the amounts of compounds that are relatively consistent.

[0085] Specifically, the estimation unit 46 determines whether the amount of each compound at each position coordinate Cn is within a predetermined range, using the amount of each compound included in each structure defined in the first table as a reference, and estimates the structure of each position coordinate Cn. For example, the estimation unit 46 determines whether the difference (Δdm, m = 1, 2, 3, ...) of the amount of each compound at each position coordinate Cn from the reference amount of the compound is less than a predetermined threshold. In this way, the estimation unit 46 can estimate the structure of the object 101 by estimating the structure of each position coordinate Cn (see FIG. 10 ).

[0086] 10 , the estimation unit 46 can estimate each structure with its position coordinate Cn, and can therefore estimate the position coordinate Cn of the feature point of each structure. Therefore, the estimation unit 46 can also estimate the size of each structure (volume or proportion to the total volume) based on the position coordinate Cn of the feature point of each structure and a known arithmetic expression, etc.

[0087] In addition, the information processing device 41 may have AI (Artificial Intelligence) as an estimation unit 46, and the estimation unit 46 may estimate the structure, size, etc. of the target object 101 based on the compound information of each of the multiple position coordinates Cn through machine learning of the AI.

[0088] In the above example, when the object 101 is a fruit, the estimation unit 46 estimates the peel, flesh, and seeds as the respective structures. However, the estimation unit 46 may estimate other structures in more detail. In this case, the estimation unit 46 may estimate, for example, insect damage or internal disease damage of each structure.

[0089] By estimating the structure of the object 101, the estimation unit 46 can estimate the amount of compounds for each structure. Therefore, if the object 101 is grapes, the estimation unit 46 can determine the moisture, sugar content, acidity, pH, polyphenols, etc. in each of the skin and flesh of the fruit based on the estimated structure and compound information. Furthermore, the estimation unit 46 can estimate the distribution of moisture, sugar content, acidity, pH, polyphenols, etc. in each structure (skin and flesh) of the object 101.

[0090] The estimation unit 46 may also estimate the harvest time of the object 101 based on the estimated structure, the size of the structure, compound information, etc. The estimation unit 46 estimates the harvest time by periodically referencing the estimated structure, the size of the structure, compound information, etc., or by chronologically referencing past estimation results for the same object 101. For example, the second storage unit 48 stores a second table in which the size of each structure at the harvest time is associated with the amount of a compound contained in each structure. The second table is defined for each object 101, for example. If the object 101 is a fruit, the second table may define the size of the peel and pulp at the harvest time and the amount of a compound contained in the peel and pulp. The estimation unit 46 may reference (acquire) the second table corresponding to the object 101 from the database 48 based on the object information included in the processing data acquired by image acquisition, and temporarily store the second table in the storage device 42 (memory).

[0091] The estimation unit 46 refers to the second table and estimates whether the object 101 is ready to be harvested based on the size of each structure and the amount of compound at the harvest time in the second table. For example, the estimation unit 46 determines whether the size of each structure and the amount of compound in the object 101 are within a predetermined range, based on the size of each structure and the amount of compound defined in the second table, and estimates whether the object 101 is ready to be harvested. In this way, the estimation unit 46 can estimate the harvest time of the object 101.

[0092] In addition, if the information processing device 41 has AI as the estimation unit 46, the estimation unit 46 may estimate the harvest time of the target object 101 through machine learning of the AI.

[0093] The estimation unit 46 stores the estimation results (e.g., the structure of the object 101, compound information for each structure, harvest time, etc.) in the storage device 42. The information processing device 41 stores the estimation results in the database 48 as estimation data in association with the object information.

[0094] In the above-described embodiment, the information processing device 41 has the estimation unit 46, and the results of estimation by the estimation unit 46 (estimated data) are stored in the second storage unit 48. However, in cases where the information processing device 41 does not have the estimation unit 46, for example, instead of or in addition to the estimated data, the compound information determined by the second determination unit 45 may be associated with the object information and stored as determined data in the database 48. In the following description, the estimated data and the determined data will be collectively referred to as result data.

[0095] The information processing device 41 controls the second communication device 49 to transmit information (e.g., result data) based on the compound information of the object 101 to the support terminal 10 via the first communication device 13. When the first communication device 13 receives the result data, the first control device 11 causes the display device 31 to display information (quality information) based on the result data. In other words, the display device 31 displays the quality information based on the compound information determined by the second determination unit 45. The display device 31 displays the quality information on a predetermined support screen M1. The display device 31 displays at least one of information on the moisture content, sugar content, and acidity of the object 101 as the quality information.

[0096] 11 is an example of a support screen M1 displayed by the display device 31. In the example shown in FIG. 11, the support screen M1 has a first information display section 111, a second information display section 112, and a third information display section 113. The first information display section 111 is an area that displays information about the object 101 captured by the imaging device 21. The first information display section 111 displays information based on the object information included in the result data, such as the type of cultivated plant 100, the location of the object 101 (e.g., fruit, leaves, stem, etc.), the date and time when the captured image 51 of the object 101 was captured, location information (latitude and longitude) of the location where the captured image 51 was captured, the field where the cultivated plant 100 is being cultivated, etc.

[0097] The second information display unit 112 is an area that displays information based on the compound information. The second information display unit 112 displays, for example, at least one of the moisture content, sugar content, and acidity content of the object 101 as the compound information. In the example shown in FIG. 11 , the second information display unit 112 displays compound information at each position (position coordinate Cn) along with one captured image 51 showing the object 101. The second information display unit 112 shown in FIG. 11 also displays the moisture content, sugar content, acidity, pH, and polyphenols of a specific grape bunch as the compound information. The second information display unit 112 can change the position (position coordinate Cn) at which the compound information is displayed by receiving an operation from the display device 31.

[0098] The third information display unit 113 is an area that displays the estimation results estimated by the estimation unit 46. The third information display unit 113 displays, for example, information such as compound information for each structure as an estimation result. The third information display unit 113 displays, for example, compound information for each peel, pulp, and seed. The third information display unit 113 in FIG. 11 displays the size of these structures and the estimated result of the harvest time of the object 101 instead of the compound information for each structure. In the example shown in FIG. 11, the third information display unit 113 displays the volume of each structure and its proportion to the whole as the size of each structure. The third information display unit 113 displays the estimated result of the harvest time in multiple stages (e.g., before the harvest time, at the harvest time, past the harvest time, etc.) and displays the difference between the harvest time and the current time (e.g., the number of days until the harvest time).

[0099] A series of processing steps performed by the agricultural work support system S will be described. FIG. 12 is a diagram illustrating a series of processing steps performed by the agricultural work support system S. That is, at least a portion of the series of processing steps shown in FIG. 12 can be considered to be an information processing method according to this embodiment. As shown in FIG. 12 , first, the imaging device 21 captures an image 51 of an object 101, which is at least a part of a cultivated plant 100 (S1, first step). Also, in the first step, the imaging device 21 captures multiple images 51 of the object 101 using reflected waves of different wavelengths L. Note that, when performing the first step, the farmworker may capture the object 101 in an environment that blocks light (external disturbances such as sunlight and lighting) that may interfere with the imaging of the object 101 by the imaging device 21 by surrounding the object 101 with a light shield 25, or by capturing the object 101 at night when the sun has set.

[0100] To explain the first step in more detail, the farm worker operates the input interface (display device 31) to input information (object information) about the object 101 (S1a). After inputting the object information, the farm worker operates the input interface to start capturing an image of the object 101 with the imaging device 21 (S1b).

[0101] The farm worker moves the imaging device 21 (support terminal 10) around the object 101 to capture images of the area around the object 101 from various distances and angles (S1c). At this time, the first control device 11 appropriately changes the band of the wavelength L of the measurement light irradiated by the irradiation unit 23, so that the imaging device 21 captures multiple captured images 51 of the object 101 using reflected waves of different wavelengths L.

[0102] Therefore, in the processing of step S1c, the imaging device 21 captures images of the surroundings of the target object 101 multiple times for each wavelength L in a different band. If the imaging device 21 (or the support terminal 10 on which the imaging device 21 is installed) has an inertial measurement unit (IMU) 24, the first control device 11 may control the display device 31 to display information such as the direction in which the imaging device 21 should be moved and whether the required number of captured images 51 for the determination process is sufficient. The first control device 11 determines that the capturing of the captured images 51 is complete when the farmworker operates the input interface or when the first control device 11 determines that the required number of captured images 51 for the determination process is sufficient (S1d: Yes). On the other hand, if the capturing of the captured images 51 is not complete (S1d: No), the processing returns to step S1c.

[0103] When the imaging device 21 completes capturing the captured image 51 of the object 101 (S1d: Yes), the first communication device 13 transmits one or more captured images 51 captured by the imaging device 21 to the server device 40 (second communication device 49) (S2). Specifically, the first control device 11 associates other information (bandwidth information, object information, etc.) with the data of the captured image 51 captured by the imaging device 21 (imaging data), and stores the data as processing data in the first storage unit 12 (S2a). The first control device 11 also controls the first communication device 13 to transmit the processing data stored in the first storage unit 12 to the server device 40 (S2b).

[0104] When the server device 40 (second communication device 49) receives processing data (one or more captured images 51) from the first communication device 13 (S3: Yes), the image acquisition unit 43 acquires the captured images 51 (S4). Specifically, the image acquisition unit 43 acquires multiple captured images 51 captured with reflected waves of different wavelengths L based on the processing data received by the second communication device 49. More specifically, the image acquisition unit 43 acquires the processing data stored in the storage device 42 and acquires multiple captured images 51 captured from different directions of the object 101 for each band of reflected light of wavelength L.

[0105] The first determination unit 44 determines light characteristics at a predetermined position coordinate Cn of the object 101 based on the captured image 51 captured in the first step (S5, second step). Specifically, in the second step, the first determination unit 44 determines light characteristics at a plurality of position coordinates Cn of the object 101 based on the captured image 51. Furthermore, in the second step, the first determination unit 44 determines spectral data at the position coordinate Cn based on the plurality of captured images 51.

[0106] In detail, the first determination unit 44 extracts a plurality of captured images 51 for each band of reflected light of wavelength L from the processing data acquired by the image acquisition unit 43 (S5a), and reconstructs (restores) (S5b) a three-dimensional space of the plurality of captured images 51 captured with reflected light of the same band of wavelength L. In S5a, the first determination unit 44 reconstructs the three-dimensional space using NeRF, for example, based on camera parameters included in the processing data and associated with each captured image 51.

[0107] When the three-dimensional space is reconstructed (S5b), the first determination unit 44 interpolates the plurality of captured images 51 in the three-dimensional space and determines light characteristics from the plurality of captured images 51 using a ray tracing method (S5c). The first determination unit 44 performs ray tracing on the image reconstructed from the three-dimensional space, i.e., on each pixel of each captured image 51 acquired by the image acquisition unit 43, and calculates the light characteristics and transmittance at each position coordinate Cn from all the light characteristics calculated using simultaneous equations.

[0108] The first determination unit 44 repeats the processes of S5a to S5c for each reflected light of wavelength L in each band (S5d: Yes), and determines the optical characteristics of each position coordinate Cn inside the object 101 for each reflected light of wavelength L in each band, in other words, determines the spectral data of each position coordinate Cn inside the object 101 (S5e).

[0109] The second determination unit 45 determines compound information at the position coordinate Cn based on the optical characteristics at the position coordinate Cn determined in the second step (S5) and determination information indicating the relationship between the optical characteristics and the compound (S6, third step). In the third step, the second determination unit 45 determines the compound information at the position coordinate Cn based on the spectral data and the determination information. In the third step, the second determination unit 45 also determines the amount of the compound at each of the multiple position coordinates Cn as compound information at that position coordinate Cn based on the optical characteristics and the determination information at each position coordinate Cn.

[0110] Specifically, the second determination unit 45 refers to (obtains) an appropriate calibration formula for determining compound information of the target object 101 based on the target object information included in the processing data (S6a). The second determination unit 45 determines compound information of each position coordinate Cn (amount of compound at each position coordinate Cn) based on the calibration formula and the optical characteristics (spectral data) at each position coordinate Cn determined by the first determination unit 44 in the second step (S5) (S6b).

[0111] The second determination unit 45 repeats the processes of S6a to S6b for each position coordinate Cn (S6c: Yes) and determines the compound information (the amount of compound contained in each position coordinate Cn, in other words, the composition of the compound) for each position coordinate Cn of the object 101 (S6d).

[0112] When the second estimation unit 46 has determined the compound information for all the position coordinates Cn (S6, third step), the estimation unit 46 estimates the structure of the object 101 based on the compound information for each of the multiple position coordinates Cn determined in the third step (S6) (S7, fourth step). For example, the estimation unit 46 refers to (obtains) the first table and the second table corresponding to the object 101 from the database 48 based on the object information included in the processing data acquired by image acquisition (S7a).

[0113] The estimation unit 46 refers to the first table and estimates which structure each position coordinate Cn corresponds to and its size (volume) based on the compounds contained in each structure in the first table and the amounts of those compounds (S7b).The estimation unit 46 also refers to the second table and estimates whether the target object 101 is at harvest time based on the sizes and amounts of compounds of each structure at harvest time in the second table (S7c).

[0114] In addition, if the information processing device 41 has AI as the estimation unit 46, the estimation unit 46 may estimate the structure, size of the structure, harvest time, etc. of the target object 101 through machine learning of the AI, without relying on the processing of S7a to S7c.

[0115] After the estimation unit 46 performs estimation through the processes of S7a to S7c, it stores the estimated results (e.g., the structure of the object 101, compound information for each structure, and harvest time) in the storage device 42, and the information processing device 41 associates the estimated results with the object information and stores them in the database 48 as estimated data (result data) (S7d).

[0116] When the estimation unit 46 stores the estimation data in the database 48 (S7d), the information processing device 41 controls the second communication device 49 to transmit information (e.g., result data) based on the compound information of the object 101 to the support terminal 10 via the first communication device 13 (S8). When the first communication device 13 receives the result data, the first control device 11 controls the display device 31 to display information based on the result data (quality information) (S9). As a result, as shown in Fig. 12, the display device 31 displays information based on the compound information determined by the second determination unit 45 (information on at least one of the moisture content, sugar content, and acidity of the object 101).

[0117] 12 is merely an example, and is not intended to be limiting. For example, when the imaging device 21 captures images of the surroundings of the object 101 for each reflected light of different wavelengths L, the first control device 11 may transmit processing data for each reflected light of each wavelength L to the server device 40. In such a case, after the processing of step S1c, the agricultural work support system S performs the processing of step S2 and then proceeds to the processing of step S1d.

[0118] Furthermore, when the imaging device 21 captures an image of the surroundings of the object 101 for each reflected light of a different band of wavelength L, and processing data is transmitted to the server device 40 for each reflected light of each band of wavelength L, the image acquisition unit 43 may perform the process of S5 for each reflected light of each band of wavelength L for which the processing data is received in step S4. In such a case, the agricultural work support system S repeats the processes of steps S1 to S5 for each reflected light of each band of wavelength L, and then proceeds to the process of step S6.

[0119] Furthermore, in the above-described embodiment, a case has been described in which a farm worker images the object 101 by surrounding it with a light shield 25 in order to capture images in an environment that blocks light that may hinder the imaging device 21 from capturing the image of the object 101, or in a time period such as at night when the sun has set. However, if the database 48 stores determination information (calibration formula) that takes the disturbance into account, the imaging device 21 may capture images of the object 101 in an environment where the disturbance is present.

[0120] A preferred embodiment of the present invention provides an information processing device 41, an agricultural work support system S, and an information processing method described in the following items.

[0121] (Item 1) An information processing device 41 including: an image acquisition unit 43 that acquires a captured image 51 of an object 101 that is at least a part of a cultivated plant 100; a first determination unit 44 that determines light characteristics at a predetermined position coordinate Cn of the object 101 based on the captured image 51 acquired by the image acquisition unit 43; and a second determination unit 45 that determines compound information related to the compound at the position coordinate Cn based on the light characteristics at the position coordinate Cn determined by the first determination unit 44 and predetermined determination information indicating a relationship between the light characteristics and a compound.

[0122] According to the information processing device 41 of this item 1, compound information at a predetermined position coordinate Cn of the object 101 can be easily and non-destructively determined simply by using a captured image 51 of the object 101. Therefore, the information processing device 41 can accurately determine the compound information of the object 101. In particular, when determining the compound information, it is sufficient for the farmer to prepare the captured image 51 of the object 101, so it is not necessary for the farmer to harvest the cultivated plant 100 that will become the object 101, which is highly convenient.

[0123] (Item 2) The information processing device 41 according to item 1, wherein the second determination unit 45 determines, as the compound information, an amount of the compound at the position coordinate Cn.

[0124] According to the information processing device 41 relating to this item 2, the amount of a compound at a predetermined position coordinate Cn of the object 101 can be easily and non-destructively determined simply by using an image 51 captured of the object 101.

[0125] (Item 3) The information processing device 41 according to Item 2, wherein the first determination unit 44 determines the light characteristics at a plurality of the position coordinates Cn of the object 101 based on the captured image 51 acquired by the image acquisition unit 43, and the second determination unit 45 determines the compound information at each of the position coordinates Cn based on the light characteristics at each of the plurality of the position coordinates Cn determined by the first determination unit 44 and the determination information.

[0126] According to the information processing device 41 of this item 3, compound information of a plurality of position coordinates Cn (in other words, a plurality of different parts) of the object 101 can be determined simply by using the captured image 51 obtained by capturing an image of the object 101. This makes it possible to easily determine the distribution of compounds contained in the object 101.

[0127] (Item 4) The information processing device 41 according to Item 3, further comprising an estimation unit 46 that estimates a structure of the object 101 based on the compound information of each of the plurality of position coordinates Cn determined by the second determination unit 45.

[0128] According to the information processing device 41 relating to this item 4, the structure of the object 101 can be estimated from the distribution of compounds contained in the object 101, and the usefulness of the determined compound information can be improved.

[0129] (Item 5) The information processing device 41 according to Item 4, wherein, when the target object 101 includes a fruit of the cultivated plant 100, the estimation unit 46 estimates at least one of a peel, a flesh, and a seed of the fruit as a structure of the fruit.

[0130] According to the information processing device 41 of this item 5, at least one of the peel, pulp, and seeds of a fruit can be easily and non-destructively determined as the structure of the object 101 simply by using the captured image 51 of the object 101. This further improves the usefulness of the determined compound information.

[0131] (Item 6) The information processing device 41 according to any one of items 2 to 5, wherein the first determination unit 44 determines the light characteristics at three-dimensional position coordinates Cn of the object 101 as the position coordinates Cn based on the captured image 51 acquired by the image acquisition unit 43.

[0132] According to the information processing device 41 relating to this item 6, compound information at a predetermined three-dimensional position coordinate Cn of the object 101 can be easily determined non-destructively by simply using the captured image 51 of the object 101.

[0133] (Item 7) The information processing device 41 according to any one of Items 2 to 6, wherein the first determination unit 44 determines the light characteristics of at least an internal position coordinate Cn of the object 101 based on the captured image 51 acquired by the image acquisition unit 43.

[0134] According to the information processing device 41 relating to this item 7, compound information at the position coordinate Cn inside the object 101 can be easily determined non-destructively by simply using the captured image 51 of the object 101.

[0135] (Item 8) The information processing device 41 according to any one of Items 2 to 7, wherein the image acquisition unit 43 acquires a plurality of captured images 51 captured with reflected waves of different wavelengths L, the first determination unit 44 determines spectral data at the position coordinate Cn based on the plurality of captured images 51 acquired by the image acquisition unit 43, and the second determination unit 45 determines the compound information at the position coordinate Cn based on the spectral data at the position coordinate Cn determined by the first determination unit 44 and the determination information.

[0136] According to the information processing device 41 relating to this item 8, spectral data at the position coordinate Cn inside the object 101 can be determined simply by using the captured image 51 of the object 101, and compound information of the object 101 can be determined with higher accuracy.

[0137] (Item 9) The information processing device 41 according to any one of items 1 to 8, wherein the first determination unit 44 determines the light characteristics from the captured image 51 acquired by the image acquisition unit 43 using a ray tracing method.

[0138] According to the information processing device 41 of this item 9, the first determination unit 44 can determine relatively realistic (photorealistic) light characteristics and transmittance by the ray tracing method, and therefore the second determination unit 45 can determine the compound information of the object 101 with higher accuracy, regardless of the directionality of the reflected light.

[0139] (Item 10) The information processing device 41 according to Item 9, wherein the first determination unit 44 uses ray tracing as the ray tracing method.

[0140] According to the information processing device 41 of this item 10, the first determination unit 44 can determine more realistic (photorealistic) light characteristics and transmittance by ray tracing, which allows the second determination unit 45 to determine compound information of the object 101 with even higher accuracy.

[0141] (Item 11) A farm work support system S including: an imaging device 21 that captures the captured image 51 of the target object 101; an information processing device 41 according to any one of items 1 to 10; and a display device 31 that displays information based on the compound information determined by the second determination unit 45.

[0142] According to the agricultural work support system S relating to this item 11, agricultural workers can easily recognize the compound information determined by the information processing device 41 that has the unique effects described above.

[0143] (Item 12) The agricultural work support system S according to Item 11, wherein the display device 31 displays at least one of information on moisture, sugar content, and acidity of the target object 101 as information based on the compound information.

[0144] According to the agricultural work support system S relating to item 12, a farmer can clearly recognize at a glance at least one of the moisture content, sugar content, and acidity of the object 101 simply by preparing an image 51 of the object 101 captured by the imaging device 21.

[0145] (Item 13) An information processing method comprising: a first step in which an imaging device 21 captures an image 51 of an object 101 that is at least a part of a cultivated plant 100; a second step in which a first determination unit 44 determines light characteristics at a predetermined position coordinate Cn of the object 101 based on the captured image 51 captured in the first step; and a third step in which a second determination unit 45 determines compound information related to the compound at the position coordinate Cn based on the light characteristics at the position coordinate Cn determined in the second step and judgment information indicating the relationship between the light characteristics and a compound.

[0146] According to the information processing method of this item 13, compound information at a predetermined position coordinate Cn of the object 101 can be easily and non-destructively determined simply by using the captured image 51 of the object 101. Therefore, the information processing device 41 can determine the compound information of the object 101 with high accuracy. In particular, when determining the compound information, the farmer only needs to prepare the captured image 51 of the object 101, so it is not necessary to harvest the cultivated plant 100 that will become the object 101, which is highly convenient.

[0147] (Item 14) The information processing method according to Item 13, wherein in the first step, the imaging device 21 captures a plurality of the captured images 51 of the object 101 using reflected waves of different wavelengths L, in the second step, the first determination unit 44 determines spectral data at the position coordinate Cn based on the plurality of the captured images 51, and in the third step, the second determination unit 45 determines the compound information at the position coordinate Cn based on the spectral data and the determination information.

[0148] According to the information processing method of this item 14, spectral data at the position coordinate Cn inside the object 101 can be determined simply by using the captured image 51 of the object 101, and compound information of the object 101 can be determined with higher accuracy.

[0149] (Item 15) The information processing method according to Item 13 further includes a fourth step in which, in the second step, the first determination unit 44 determines the light characteristics at the plurality of position coordinates Cn of the object 101 based on the captured image 51, in the third step, the second determination unit 45 determines the amount of the compound at each of the plurality of position coordinates Cn as the compound information at the position coordinates Cn based on the light characteristics of each position coordinate Cn and the determination information, and an estimation unit 46 estimates a structure of the object 101 based on the compound information at each of the plurality of position coordinates Cn determined in the third step.

[0150] According to the information processing method of this item 15, the structure of the object 101 can be estimated from the distribution of compounds contained in the object 101, and the usefulness of the determined compound information can be improved.

[0151] Although the present invention has been described above, the embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims.

[0152] 21: Imaging device 31: Display device 41: Information processing device (second control device) 42: Storage device (memory) 43: Image acquisition unit 44: First determination unit 45: Second determination unit 46: Estimation unit 51: Captured image 100: Cultivated plant 101: Object Cn: Position coordinates S: Support system

Claims

1. An information processing device comprising: an image acquisition unit that acquires an image of an object that is at least a part of a cultivated plant; a first determination unit that determines light characteristics of the object at a predetermined position coordinate based on the image acquired by the image acquisition unit; and a second determination unit that determines compound information related to the compound at the position coordinate based on the light characteristics at the position coordinate determined by the first determination unit and predetermined judgment information that indicates the relationship between the light characteristics and the compound.

2. The information processing device according to claim 1, wherein the second determination unit determines the amount of the compound at the position coordinates as the compound information.

3. The information processing device described in claim 2, wherein the first determination unit determines the light characteristics of the object at multiple position coordinates based on the captured image acquired by the image acquisition unit, and the second determination unit determines the compound information at each position coordinate based on the light characteristics at each of the multiple position coordinates determined by the first determination unit and the judgment information.

4. The information processing device according to claim 3, further comprising an estimation unit that estimates the structure of the object based on the compound information for each of the plurality of position coordinates determined by the second determination unit.

5. The information processing device according to claim 4, wherein when the object includes a fruit of the cultivated plant, the estimation unit estimates at least one of the peel, flesh, and seeds of the fruit as the structure of the fruit.

6. An information processing device according to claim 2, wherein the first determination unit determines the light characteristics at the three-dimensional position coordinates of the object as the position coordinates based on the captured image acquired by the image acquisition unit.

7. An information processing device as described in claim 2, wherein the first determination unit determines the light characteristics of at least the object at position coordinates inside the object based on the captured image acquired by the image acquisition unit.

8. The information processing device according to claim 2, wherein the image acquisition unit acquires a plurality of captured images captured using reflected waves of different wavelengths, the first determination unit determines spectral data at the position coordinates based on the plurality of captured images acquired by the image acquisition unit, and the second determination unit determines the compound information at the position coordinates based on the spectral data at the position coordinates determined by the first determination unit and the judgment information.

9. The information processing device according to claim 1, wherein the first determination unit determines the light characteristics from the captured image acquired by the image acquisition unit using a ray tracing method.

10. The information processing device according to claim 9, wherein the first determination unit uses ray tracing as the ray tracing method.

11. An agricultural work support system comprising: an imaging device that captures the image of the object; an information processing device according to any one of claims 1 to 10; and a display device that displays information based on the compound information determined by the second determination unit.

12. The agricultural work support system according to claim 11, wherein the display device displays at least one of the moisture content, sugar content, and acidity of the object as information based on the compound information.

13. An information processing method comprising: a first step in which an imaging device captures an image of an object that is at least a part of a cultivated plant; a second step in which a first determination unit determines light characteristics at a predetermined position coordinate of the object based on the image captured in the first step; and a third step in which a second determination unit determines compound information related to the compound at the position coordinate based on the light characteristics at the position coordinate determined in the second step and judgment information indicating the relationship between the light characteristics and a compound.

14. An information processing method as described in claim 13, wherein in the first step, the imaging device captures a plurality of images of the object using reflected waves of different wavelengths; in the second step, the first determination unit determines spectral data at the position coordinates based on the plurality of captured images; and in the third step, the second determination unit determines the compound information at the position coordinates based on the spectral data and the judgment information.

15. An information processing method as described in claim 13, further comprising: in the second step, the first determination unit determines the optical characteristics of the object at a plurality of position coordinates based on the captured image; in the third step, the second determination unit determines the amount of the compound at each of the plurality of position coordinates as the compound information at that position coordinate based on the optical characteristics of each position coordinate and the judgment information; and a fourth step in which an estimation unit estimates the structure of the object based on the compound information at each of the plurality of position coordinates determined in the third step.

Images