Leaf temperature acquisition device, crop cultivation system, leaf temperature acquisition method, program for acquiring leaf temperature, and crop cultivation method

Abstract

To grow crops based on measurement values.SOLUTION: The temperature of a crop leaf is obtained from a thermal image. In this process, separation of the leaf and the background in the thermal image is performed using deep learning. A growing environment is controlled based on the obtained leaf temperature, the area percentage of the leaf at that temperature, and the duration of the temperature. The environment is controlled so that VPD is in a specific range under the conditions of ΔT=T-Tm and VPD=((100-RH) / 100)×SVP, where RH is the humidity of the environment surrounding the leaf, SVP is the saturated vapor pressure of the leaf of the crop, T is the temperature of the leaf, Tm is the temperature of the environment surrounding the leaf.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for growing crops such as vegetables and fruits.

Background Art

[0002] Hitherto, the production of agricultural products has largely relied on the experience and intuition of producers. On the other hand, with the progress of the decrease and aging of the producer population, there are problems such as ensuring a stable production volume, reducing production costs, and suppressing physiological disorders and diseases for growing plants in a healthy manner. As a technology to address this issue, the systematic production management of agricultural products based on various sensing data has been studied.

[0003] For example, Patent Document 1 describes an agricultural production system using a thermal image. In addition, Non-Patent Documents 1 and 2 describe that the amount of saturated water vapor is related to the growth of crops.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Generally, in the production of agricultural products using a greenhouse, the temperature inside the greenhouse is measured, and based on this, the environment inside the greenhouse is adjusted. However, even within the same greenhouse, the growth states are different at the canopy unit (seedling unit or plant unit) and the community unit. In order to achieve higher productivity and suppress physiological disorders and diseases that promote the healthy growth of plants, fine measurement and control are required at the canopy unit and the community unit.

[0007] In addition, although there is a need for a technology to suppress the occurrence of physiological disorders based on objective measurement values, such a technology has not been established so far.

[0008] Against this background, the present invention aims to provide a technology capable of grasping the growth state of agricultural products based on measurement values and controlling the growth state. Further, as a secondary object, the present invention aims to provide a technology capable of growing crops healthily based on measurement values and suppressing physiological disorders and diseases.

Means for Solving the Problems

[0009] The present invention is a leaf temperature acquisition device for acquiring the temperature of a crop leaf, comprising a thermal image data acquisition unit for acquiring data of a thermal image obtained by imaging the leaf with an infrared camera, a background removal unit for removing the thermal image of the background of the leaf in the thermal image using a deep learning function, and a leaf temperature acquisition unit for acquiring the temperature of the leaf based on the thermal image from which the background has been removed. The deep learning function has a network including a first step of obtaining a thermal image obtained by infrared photographing of the leaf, a second step of obtaining a base feature map by performing a convolution process on the thermal image, a third step of obtaining at least a first feature map, a second feature map, a third feature map, and a fourth feature map by separately performing a plurality of different convolution processes on the base feature map, and a fourth step of performing further convolution processes on the first feature map, the second feature map, the third feature map, and the fourth feature map. The first image obtained by imaging the first feature map, the second image obtained by imaging the second feature map, the third image obtained by imaging the third feature map, and the fourth image obtained by imaging the fourth feature map each have different features. This is a leaf temperature acquisition device.

[0010] In the present invention, a preferable aspect is that the thermal image is a thermal image based on radiant heat having a wavelength in the range of 8 μm to 14 μm.In the present invention, a preferable aspect is that the deep learning function obtains, as a teacher image, an image in which the background is removed from a thermal image in which the leaves of a target crop are captured, based on an RGB image of the leaves of the target crop.

[0012] In the above network, when any two images among the first image, the second image, the third image, and the fourth image are considered, (1) It is clear throughout. (2) The features are emphasized and the visible parts are different. (3) The colors and light and dark areas are different. (4) The resolution is different. An embodiment satisfies the above requirement (1) and at least two of requirements (2), (3), and (4).

[0013] In the above network, n is a natural number equal to or greater than 4, and in the third step, a first feature map to an n-th feature map are obtained, and in the fourth step, further convolution processing is performed on each of the first feature map to the n-th feature map.

[0014] The present invention relates to A leaf temperature acquisition device including a thermal image data acquisition unit that acquires data of a thermal image obtained by imaging a leaf of a crop with an infrared camera, a background removal unit that removes a thermal image of the background of the leaf in the thermal image using a deep learning function, and a leaf temperature acquisition unit that acquires the temperature of the leaf based on the thermal image from which the background has been removed. A crop cultivation system that controls the cultivation environment of the crop based on the temperature of the leaf of the crop acquired by the leaf temperature acquisition device. The control of the growing environment includes controlling the temperature of the leaves of the crop to be within a specific temperature range. of the area of the leaf The ratio and in the ratio of the area of the leaf This is a crop growing system in which the relationship between the duration of the temperature range and the content of the control is based on a predetermined table. This invention can also be grasped as an invention of a method.

[0015] The present invention provides a leaf temperature acquisition device including a thermal image data acquisition unit that acquires data of a thermal image obtained by imaging a leaf of a crop with an infrared camera, a background removal unit that removes a thermal image of the background of the leaf in the thermal image using a deep learning function, and a leaf temperature acquisition unit that acquires the temperature of the leaf based on the thermal image from which the background has been removed. A crop cultivation system that controls the cultivation environment of the crop based on the temperature of the leaf of the crop acquired by the leaf temperature acquisition device. The humidity of the surrounding environment of the leaves of the crop is RH, the The leaf temperature of the crop acquired by the leaf temperature acquisition device is T(T leaf) The saturation vapor pressure of the leaves of the crop obtained from the T is defined as SVP, and VPD = ((100 - RH) / 100)×SVP. Regarding the VPD during the growth period of the flower buds of the crop, the range of VPD in which the physiological disorder of the crop does not manifest has been obtained in advance, and the control of the growth environment during the growth period of the flower buds of the crop is performed so as to be within the range of VPD in which the physiological disorder does not manifest. This is a crop cultivation system. This invention can also be grasped as an invention of a method.

[0016] The present invention provides a leaf temperature acquisition device including a thermal image data acquisition unit that acquires data of a thermal image obtained by imaging a leaf of a crop with an infrared camera, a background removal unit that removes a thermal image of the background of the leaf in the thermal image using a deep learning function, and a leaf temperature acquisition unit that acquires the temperature of the leaf based on the thermal image from which the background has been removed, and a crop cultivation system that controls the cultivation environment of the crop based on the temperature of the leaf of the crop acquired by the leaf temperature acquisition device, wherein The humidity of the ambient environment around the leaves of the crop is RH, the The temperature of the leaves of the crop acquired by the leaf temperature acquisition device is T (T leaf ) The saturation vapor pressure of the leaves of the crop obtained from the T is defined as SVP, the temperature of the ambient environment around the leaves is Tm (T temperature of the surrounding environment ) ΔT = T leaf -T temperature of the surrounding environment That is, ΔT = T - Tm, and VPD = ((100 - RH) / 100)×SVP. There is a linear relationship between the VPD and the ΔT during the growth period of the flower buds of the crop. The linear relationship has a region where the relationship is relatively strong and a region where the relationship is relatively weak. The control of the growth environment during the growth period of the flower buds of the crop is performed so as to be the value of VPD in the region where the relationship is relatively strong. This is a crop cultivation system.

[0017] In the present invention, it is preferable that the crop is a tomato, and the growth environment is controlled so that VPD < 2.2 during the period from at least the time when flower buds are formed to the time before the fruits turn red. Also, it is preferable that the objects related to the RH, the SVP, the T, and the Tm are the leaves of the part where the flower buds of the crop are formed.

[0018] The present invention relates to a leaf temperature acquisition device that acquires the temperature of the leaves of a crop of A method for obtaining the temperature of a leaf, comprising: a thermal image data acquisition step of acquiring data of a thermal image obtained by imaging the leaf with an infrared camera; a background removal step of removing the thermal image of the background of the leaf in the thermal image using a deep learning function; and a leaf temperature determination step of determining the temperature of the leaf based on the thermal image from which the background has been removed. the deep learning function has a network having a first step of obtaining a thermal image obtained by infrared imaging of the leaf, a second step of obtaining a base feature map by performing a convolution process on the thermal image, a third step of obtaining at least a first feature map, a second feature map, a third feature map, and a fourth feature map by individually performing a plurality of different convolution processes on the base feature map, and a fourth step of performing further convolution processes on the first feature map, the second feature map, the third feature map, and the fourth feature map, and the first image obtained by imaging the first feature map, the second image obtained by imaging the second feature map, the third image obtained by imaging the third feature map, and the fourth image obtained by imaging the fourth feature map each have different features It is a method for obtaining the temperature of a leaf.

[0019] The present invention is a program for obtaining the temperature of a crop leaf to be read and executed by a computer, which causes the computer to execute: a thermal image data acquisition step of acquiring data of a thermal image obtained by imaging the leaf with an infrared camera; a background removal step of removing the thermal image of the background of the leaf in the thermal image using a deep learning function; and a leaf temperature determination step of determining the temperature of the leaf based on the thermal image from which the background has been removed. The deep learning function has a network having a first step of obtaining a thermal image obtained by infrared photographing of the leaf, a second step of obtaining a base feature map by performing a convolution process on the thermal image, and a third step of obtaining at least a first feature map, a second feature map, a third feature map, and a fourth feature map by individually performing a plurality of different convolution processes on the base feature map, and a fourth step of performing further convolution processes on the first feature map, the second feature map, the third feature map, and the fourth feature map. The first image obtained by imaging the first feature map, the second image obtained by imaging the second feature map, the third image obtained by imaging the third feature map, and the fourth image obtained by imaging the fourth feature map each have different features. It is a program for obtaining the temperature of a leaf.

Advantages of the Invention

[0020] According to the present invention, the growth state of agricultural products can be grasped by measurement, and the growth state can be controlled.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Embodiments for Carrying Out the Invention

[0022] (1) Outline of the System Fig. 1 shows the outline of the system. Here, the case of tomatoes as the plant to be grown (cultivated) is shown. In this example, the case of tomatoes will be described, but the same applies to other plants, such as eggplants, strawberries, cabbages, bell peppers, cucumbers, and shishito peppers, etc. The system shown in Fig. 1 controls the environment (temperature, humidity, etc.) inside the greenhouse 300 based on various measurement data related to the tomato canopy 100. Here, the example of tomatoes is shown, but it can also be applied to plants other than tomatoes. Here, the canopy refers to the part composed of roots, stems and trunks extending from the roots, branches branching from the stems and trunks, leaves, flowers, and fruits attached to the branches. Generally, the canopy is denoted by the same concept as seedlings or stocks.

[0023] In the example of FIG. 1, the tomato is attracted to grow vertically upward by an attracting line (not shown). The growth direction can be arbitrarily set. Along the growth direction (the direction toward vertically upward), sensor groups are arranged in three stages. Each sensor group is composed of an infrared camera, a temperature sensor, a humidity sensor, and a photon sensor. The camera is an infrared camera that captures a thermal image. Here, an infrared camera having sensitivity in the wavelength band of 8 μm to 14 μm is used. Further, a camera that captures an RGB color image can also be arranged. Hereinafter, the thermal image in this specification is captured by an infrared camera having sensitivity in the above wavelength band.

[0024] According to the findings of the present inventors, the emissivity of plant leaves is high in the wavelength band of 8 μm to 14 μm. Therefore, in the thermal image captured by an infrared camera having sensitivity in the range of 8 μm to 14 μm wavelength, information related to the leaves can be obtained with relatively high sensitivity. That is, in the thermal image, there is a difference in the amount of information between the leaves and the rest. Specifically, in the thermal image showing the leaves, the information of the leaves is relatively large, and the information other than the leaves is relatively small. This difference is determined by an estimation model obtained by deep learning described later, and the separation between the leaves and their background is performed.

[0025] For example, in the case of photographing inside a greenhouse, the background of the leaves is glass or metal that constitutes the greenhouse. These have a significantly different thermal radiation spectrum from that of the leaves. Therefore, a difference occurs between the image information of the thermal image of the leaves and the image information of the thermal image of the background, and effective separation between the leaves and the background becomes possible.

[0026] Each sensor group is mounted on a clip, and by attaching the clip to an attracting line, a trunk, a branch, a support rod, etc., it is arranged at three locations in the vertical direction in proximity to the canopy 100 of the tomato. A technique of mounting a camera and various sensors on the clip, attaching this clip to an attracting line or the like, and performing various measurements on plants is described in, for example, Japanese Patent Application No. 2020-8993. Although three groups of sensor groups are shown in FIG. 1, more sensor groups may be arranged.

[0027] The system in FIG. 1 controls the environment inside the greenhouse 300. In FIG. 1, only one tomato canopy is shown. Similar to a normal greenhouse, it is possible to expand to a mode where multiple tomato canopies (seedlings) are planted.

[0028] The system in FIG. 1 controls the cultivation environment of tomatoes based on various sensing data acquired by the sensor group. According to the research of the present inventors, it has been found that there is a strong correlation between the temperature of tomato leaves and the growth state of fruits. What is important here is to accurately measure the leaf temperature.

[0029] In the invention disclosed in this specification, the leaf temperature is measured (predicted) from the thermal image. At this time, the separation of the target leaf and the background is performed using the characteristics of the thermal image. In addition, the cultivation environment of tomatoes is controlled using the measured leaf temperature. Also, in tomatoes, it is utilized that parameters related to the saturated vapor pressure of leaves at the initial stage of fruit development have a high correlation with the final physiological disorders of fruits (such as fruit cracking and blossom-end rot), and such physiological disorders are prevented. Here, the initial stage of fruit development includes the periods of flower bud formation, flower bud growth, flowering, fruit setting, and fruit growth. This period corresponds to the cell enlargement stage of the fruit. Also, a fruit is the fruit of a plant.

[0030] (2) Configuration of the sensor group The sensor group performs acquisition (imaging) of a thermal image by an infrared camera, measurement of the ambient temperature near the leaf by a temperature sensor, measurement of the ambient humidity near the leaf by a humidity sensor, and measurement of the amount of light (amount of visible light) required for photosynthesis near the leaf by a quantum sensor, with respect to the leaves of a specific part of the target canopy. Commercially available devices are used for the devices constituting each sensor. In the case of FIG. 1, an example of measurement at three locations: the lower, middle, and upper parts of the tomato canopy 100 is shown. Also, the system shown in FIG. 1 is equipped with a CO 2 sensor. The CO 2 sensor is arranged one for each seedling canopy. It is also possible to arrange the CO 2 sensors in a more detailed manner.

[0031] (3) Configuration of the analysis device Hereinafter, the analysis device 200 in FIG. 1 will be described. The analysis device 200 analyzes the measurement data measured by the sensor group, makes various determinations based on the results of this analysis, and determines the content of environmental control based on the results of this determination. The analysis device 200 is configured using a commercially available PC (personal computer). The analysis device 300 has various arithmetic functions, communication functions, user interface functions, data storage functions, etc. that a general PC has.

[0032] FIG. 2 shows a block diagram of the analysis device 200. The analysis device 200 includes a thermal image data acquisition unit 201, an ambient temperature acquisition unit 202, an ambient humidity acquisition unit 203, a light quantity acquisition unit 204, a leaf temperature acquisition unit 205, a determination unit 206, a data storage unit 207, an SVP acquisition unit 208, a VPD calculation unit 209, and a control content determination unit 210. These functional units are realized software-wise by a program installed in the PC to be used and by the CPU and other integrated circuits of the PC.

[0033] Part or all of the analysis device 200 may be configured by dedicated hardware. Also, a server or the like connected to a network line may be operated as the analysis device 200. Also, a form in which the functions of the analysis device 200 are performed by a plurality of distributed PCs or servers is also possible.

[0034] The thermal image data acquisition unit 201 receives data of a thermal image (thermal image data) captured by an infrared camera. The ambient temperature acquisition unit 202 receives the measurement data of a temperature sensor. The ambient humidity acquisition unit 203 receives the measurement data of a humidity sensor. The light quantity acquisition unit 204 receives the measurement data of a photon sensor.

[0035] The leaf temperature acquisition unit 205 acquires the temperature of the leaf based on the thermal image of the leaf acquired by the thermal image data acquisition unit 201. The leaf temperature acquisition unit 205 includes, as functional blocks, a background removal unit 211, a leaf thermal image extraction unit 212, a leaf temperature map creation unit 213, and a leaf temperature determination unit 214.

[0036] The background removal unit 211 uses an estimation model obtained by deep learning to separate the leaf part and the non-leaf background part from the thermal image, and removes the background part. Details of the estimation model obtained by deep learning will be described later.

[0037] The leaf thermal image extraction unit 212 extracts the thermal image of the leaf part from the thermal image with the background removed. The leaf temperature map creation unit 213 creates a map showing the temperature distribution of the leaf based on the extracted thermal image of the leaf part. As the leaf temperature map, if the thermal image is an enlarged image of the leaf, it is obtained as a map of the temperature distribution inside the leaf, and if the thermal image shows multiple leaves, it is a map of the temperature distribution showing the difference in temperature for each leaf. Of course, depending on the magnification state and resolution of the image, intermediate cases are also possible.

[0038] The leaf temperature determination unit 214 determines the temperature of the leaf based on the map of the temperature distribution. As a method for determining the temperature of the leaf, for example, in the case of a single leaf, there are methods such as calculating the average value and determining it as the temperature of the leaf, determining the median value of the temperature distribution as the temperature of the leaf, determining the temperature of the peak of the temperature distribution as the temperature of the leaf, and determining the average or peak value of the temperature distribution in a region with a specific area ratio or more as the temperature of the leaf. Also, when targeting multiple leaves, there are methods to obtain the overall average value or peak value, but data such as the ratio of the area of leaves in the first temperature range and the ratio of the area of leaves in the second temperature range can also be obtained.

[0039] The determination unit 206 performs various determinations based on the leaf temperature information acquired by the leaf temperature acquisition unit 205. Based on the results of this determination, the environment (temperature, humidity, etc.) inside the agricultural house 300 is adjusted. Details of the determination conditions and control will be described later.

[0040] The data storage unit 207 stores the data that is the basis for the determination in the determination unit 206, other data necessary for the operation of the analysis device 200, and the data obtained as a result of the operation of the analysis device 200.

[0041] The SVP acquisition unit 208 acquires the saturated vapor pressure (SVP) of the leaves near the tip 101 of the tomato canopy 100 based on the leaf temperature T acquired by the leaf temperature acquisition unit 205. For the target tomato, table data regarding the relationship between the leaf temperature T and the SVP has been acquired in advance, and based on this table data, the SVP is obtained from T. This process is performed by the SVP acquisition unit 208. Note that as the leaf temperature T, the average value of the group of target leaves is used. As the leaves to be observed, multiple or more leaves at the tip of the tomato canopy 100 are selected.

[0042] The VPD calculation unit 209 calculates the vapor pressure deficit (VPD) from VPD = ((100 - RH) / 100) × SVP. Here, RH is the humidity information near the tip of the canopy 100 acquired by the atmospheric humidity information acquisition unit 202.

[0043] Based on the result of the determination by the determination unit 206, the control content determination unit 210 determines the content of the control for controlling the environment inside the agricultural house 300. According to this determination, the environment is controlled by the environment control device 400 in FIG. 1.

[0044] (4) Environment control device The environment control device 400 in FIG. 1 adjusts the environment inside the agricultural house 300. The environments to be adjusted are temperature, humidity, brightness, fine mist cooling spray (spraying of water), irrigation state, ventilation, air blowing, forced air circulation, CO 2 concentration. As the devices to be controlled, a heating device (heating device) and a cooling device (a general air conditioner is used), a fine mist cooling spray device, a humidifying device, a dehumidifying device, a blower fan, an irrigation device, an opening / closing device for a ventilation window, an opening / closing device for a light-shielding curtain, a control device for a light (lighting), and a CO 2 fine mist cooling device are included.

[0045] (5) Acquisition of leaf temperature The processing performed by the leaf temperature acquisition unit 205 will be described below. In the technology disclosed in this specification, the temperature of a leaf is obtained from a thermal image. At this time, it is necessary to separate the thermal image of the leaf of interest from the thermal image, in other words, to perform a process of removing the background of the leaf of interest.

[0046] Here, taking advantage of the characteristics of the thermal image, deep learning is used to separate the thermal image of the leaf part from the thermal image. Then, the temperature distribution of the leaf is obtained from the thermal image of the leaf with the background removed, and finally the temperature of the leaf is determined.

[0047] In the present invention, deep learning is used to learn the difference between the thermal image of the leaf and the thermal image of the background caused by the difference in emissivity, and an arithmetic model (estimation model described later) for separating the leaf and the background on the thermal image is created. Deep learning will be described later. First, the discrimination between the thermal image of the leaf and the thermal image of the background caused by the difference in emissivity will be described.

[0048] FIG. 3 shows a color image (RGB image) (A) in the visible band that is the original image of the leaf, and a background removal image (B) in which the background is removed by color extraction using the difference in color information. In the case of a color image, the accuracy of separation based on color information is greatly affected by the lighting conditions. When the background is removed, there is a tendency for parts that must be identified as leaves to be missing (or conversely, for the background to remain). This is extremely disadvantageous in a farming work site where it is difficult to obtain a uniform lighting environment.

[0049] FIG. 4 is a thermal image (A) taken by an infrared camera from the same perspective as FIG. 3, a thermal image (B) in which the background is removed from the thermal image (A) using the estimation model described later, and a binary image (C) obtained by converting the thermal image (B) into a binary image with the background set to black. As shown in FIG. 4, in the case of a thermal image, there is little loss of leaf information. Also, a thermal image can be obtained regardless of the brightness of the environment (lighting conditions and illumination state). In particular, a thermal image can be obtained even at night.

[0050] FIG. 5 shows a color original image (RGB image) (A) in the visible band that captures the crops being cultivated inside the agricultural greenhouse, and images (B) and (C) obtained by removing the background from the image (A) through color extraction. The difference between (B) and (C) lies in the difference in the image processing conditions.

[0051] In the background-removed image of FIG. 5(B), a part of the leaf image is also removed together with the background. On the other hand, in the background-removed image of FIG. 5(C), the background remains. Thus, in background removal by color extraction, the setting of the extraction conditions is delicate, and it is also strongly affected by the lighting conditions, so it is difficult to stably remove the background.

[0052] FIG. 6(A) is a thermal image that captures the same part as FIG. 5. FIG. 6(B) is a thermal image obtained by removing the background from the thermal image (A) using an estimation model obtained by deep learning described later. FIG. 6(C) is an image obtained by converting the background of the thermal image (B) to black to form a binary image. As is clear from FIG. 6, by using the thermal image, the leaves and the background can be effectively separated.

[0053] To summarize the above, in the RGB image, separating the leaves and the background is delicate in terms of condition setting, and particularly affected by the lighting conditions. Also, it cannot be used in dark situations. In dark situations, a method of ensuring brightness by lighting can be considered, but in the case of lighting, problems such as lighting conditions and reflection occur, making it difficult to perform stable separation.

[0054] In contrast, in the case of the thermal image, regardless of the environmental brightness, stable separation of the leaves and the background is possible. This is presumably because the thermal image is not affected by visible light, and the difference in thermal radiation between the leaves and other materials due to the difference in the emissivity of the materials appears in the thermal image, and separation using this is performed.

[0055] Also, the point that it can be used at night is particularly useful from the perspective of continuously controlling crop growth.

[0056] (6) Regarding Deep Learning (Definition of Terms) The following describes the terms to be used. Here, the initial thermal image, the estimation model, the learning network, the estimated thermal image, and the teacher thermal image will be described.

[0057] The initial thermal image is a thermal image for which the process of separating the leaf of interest from the background has not been performed. Note that the initial thermal image may be normalized or noise removed.

[0058] The estimation model is obtained by training a learning network with an AI estimation function to learn a method for estimating an estimated thermal image. With the estimation model, an estimated thermal image in which the leaf of interest is separated from the background is estimated from the initial thermal image. In this example, the functions of the background removal unit 211 and the leaf thermal image extraction unit 212 in FIG. 2 are realized by the above-described estimation model.

[0059] The learning network is an algorithm that deepens the layers of a neural network and performs deep learning. The learning network is not particularly limited. Examples of the learning network include U-Net, Deeplab v3+, FPN (Feature Pyramid Networks), PSPNet (Pyramid Scene Parsing Network), LinkNet, etc.

[0060] The estimated thermal image is a thermal image estimated by the estimation model based on the initial thermal image. In the estimated thermal image, the background of the leaf is removed from the initial thermal image. That is, the estimated thermal image is obtained by separating the leaf and the background from the initial thermal image and making it a thermal image of only the leaf.

[0061] The teacher thermal image is a thermal image in which, in the initial thermal image, a boundary line based on an RGB image is manually drawn and the background is removed. The teacher thermal image is also the correct answer image in the case of being ideally estimated correctly.

[0062] (7) Creation of the Estimation Model Train the learning network to learn the function (ability) to estimate the estimated thermal image from the initial thermal image. In this training, the function that takes the initial thermal image as input and outputs the teacher thermal image is learned.

[0063] An example of the specific procedure of learning is shown below. For example, assume there are 1000 samples to be learned. These are divided into 300 samples (Group A: teacher data group) and 700 samples (Group B: test data group). First, using the samples in Group A, the above learning is performed to obtain an estimation model. That is, using the samples in Group A, a function that takes an initial thermal image as input and outputs a teacher thermal image is learned. Through this learning, an estimation model is obtained.

[0064] Then, for the above estimation model obtained by the samples in Group A, error backpropagation learning is performed using the samples in Group B. In this error backpropagation learning, using the samples in Group B, an estimated thermal image is estimated from the initial thermal image by the above estimation model, the difference between the obtained estimated image and the teacher image is calculated, and the weights of the neural network are adjusted so that this difference is minimized, that is, the learning network is adjusted.

[0065] The above process is repeated multiple times by randomly changing the combination of Group A and Group B. Through the above process, an estimation model obtained by deep learning is obtained. Using this estimation model, for example, the processes of FIG. 4(A)⇒(B) and FIG. 6(A)⇒(B) are performed.

[0066] (8) Example of environmental control Hereinafter, an example of environmental control based on the acquired leaf temperature information in the control of the growth of the tomato canopy 100 in the system of FIG. 1 will be described. Here, thermal images are taken of the leaves of the entire tomato canopy 100, and the temperature distribution of the leaves is acquired.

[0067] FIG. 7 is an example of a management table when managing the growth of tomatoes. The table in FIG. 7 is prepared in advance. Regarding the content of the table, it is preferable to prepare in advance the most suitable ones for the crop type and variety.

[0068] The management table in FIG. 7 is used as follows. Here, the temperature of the leaves throughout the entire tomato canopy 100 is monitored. For example, when considering the area of the monitored leaves, if 25% or more is in the range of 15°C to 20°C and this state continues for 30 minutes or more, it is determined as level 2. In this case, it is determined that the leaf temperature is low, and the control in which heating, which is a process of increasing the environmental temperature to increase the leaf temperature, is selected is performed.

[0069] For example, when considering the area of the monitored leaves, if 5% or more is in the range of 35°C to 40°C and this state continues for 10 minutes or more, it is determined as level 6. In this case, it is determined that the leaf temperature is high and the temperature stress is moderately large. Then, in order to alleviate this situation, fine mist cooling spraying (spraying of water), air circulation, and cooling, which is a process of reducing the environmental temperature, are performed to lower the leaf temperature.

[0070] Also, there may be a case where two different levels are determined simultaneously. There are several methods for dealing with this case. The first method is to prioritize the determination of the level with a larger area. For example, assume that level 4 and level 5 are determined simultaneously. At this time, if the leaf area ratio of the target of level 4 is 34% and the leaf area ratio of the target of level 5 is 21%, then the larger area, level 4, is prioritized, and the control corresponding to level 4 is performed.

[0071] The second method is to prioritize the more serious level. For example, assume that level 4 and level 5 are determined simultaneously. At this time, since level 5 is more serious, level 5 is prioritized, and the control corresponding to level 5 is performed.

[0072] It is also possible to use a method that combines the first method and the second method. In this case, when the difference in the area ratio is smaller than the threshold value, the more serious level is prioritized, and when the difference in the area ratio is equal to or greater than the threshold value, the level with the larger area is prioritized. These methods can be similarly applied when three or more different levels are determined simultaneously.

[0073] Here, changes in the time axis direction can also be incorporated into the determination. The temperature of the leaves fluctuates on the time axis. Therefore, the average on the time axis is taken into account to calculate the above area ratio. For example, observe the change in the area within a specific temperature range over 15 minutes, and specify the average as the area within that temperature range.

[0074] In addition, it is also possible to control multiple tree canopies. In this case, each tree canopy is the object of measurement and control, and the environment is controlled in units of tree canopies. As a direction, control is performed so as to approach level 3 as a whole. Since irrigation and the spraying of fine mist cooling can be controlled independently for each tree canopy, when multiple tree canopies are the object, a method can be adopted in which the state of irrigation and the spraying of fine mist cooling is controlled for each tree canopy so that each tree canopy stays within level 3. Also, it is possible to individually provide partitions and control the environment.

[0075] FIG. 8 shows an example of the processing procedure using the management table of FIG. 7. The program for executing the processing of FIG. 8 is stored in the storage area (such as a hard disk or semiconductor memory) of the PC constituting the analysis device 200 and is executed by the CPU provided in the analysis device 200. It is also possible to store the program in an appropriate storage medium, storage server, etc., and download and use it from there.

[0076] When the processing is started, first, thermal image data is acquired (step S101), and then steps S102 to S105, which are processing for obtaining the temperature T of the leaves, are performed. These processes are performed by the temperature acquisition unit 205 in FIG. 2. First, in the thermal image obtained in step S101, the background of the leaves is removed (step S102). This process is performed by the background removal unit 211 in FIG. 2.

[0077] After removing the background of the leaves, the thermal image of the leaves is extracted (step S103). This process is performed by the leaf thermal image extraction unit 212 in FIG. 2. Next, based on the extracted thermal image of the leaves, a map of the temperature distribution of the leaves is created (step S104). This process is performed by the leaf temperature map creation unit 213 in FIG. 2. In this case, a map of the temperature distribution of the leaves is created.

[0078] Once the leaf temperature map is obtained, based on this temperature map, the correspondence relationship between the leaf temperature and the area ratio is specified (step S105). This process is performed by the leaf temperature determination unit 214. In this process, the ratio (%) of the leaves at 10 to 15 °C in FIG. 7, the ratio (%) of the leaves at 15 to 20 °C, the ratio (%) of the leaves at 20 to 25 °C, the ratio (%) of the leaves at 25 to 30 °C, the ratio (%) of the leaves at 30 to 35 °C, the ratio (%) of the leaves at 35 to 40 °C, and the ratio (%) of the leaves at a temperature exceeding 40 °C are obtained.

[0079] Next, it is determined whether environmental control is necessary (step S106). If environmental control is necessary, environmental control corresponding to the level in FIG. 7 is performed (step S107). In the case of level 3 in FIG. 7, environmental control is not necessary, and the control below step S101 is repeated. The determination of the content of environmental control based on the management table in FIG. 7 is performed by the control content determination unit 210 in FIG. 2.

[0080] (9) Example of suppressing physiological disorders based on VPD Here, in the cultivation of tomatoes, a technique for suppressing physiological disorders based on the leaf temperature obtained based on a thermal image will be described. The causes of physiological disorders in tomatoes (such as cracked fruits, fruits with damaged bottoms, etc.) occur during the stage from flower buds to the stage where the fruits are set (the stage before the fruits turn from green to red). That is, the factors causing physiological disorders are already included in the initial stage of fruit development.

[0081] This is explained by the following model. In the case of plants that bear fruits such as tomatoes, cell division is repeated from the initial stage of fruit development, and finally fruits are formed. Here, factors that will cause subsequent physiological disorders occur at the initial stage of cell division.

[0082] The factors causing physiological disorders are mainly caused by insufficient or excessive water supply to cells. Therefore, it is important to control the appropriate growth environment during the process of flower bud formation and subsequent fruit setting so that the above-mentioned factors causing physiological disorders do not occur.

[0083] The inventors have found that the relationship shown in FIG. 9 can be used as a method for evaluating the factors causing the above physiological disorders.

[0084] In FIG. 9, for the leaves at the part where flower buds are formed at the tip (upper end) of the canopy in greenhouse-cultivated tomatoes, the humidity (relative humidity) of the surrounding environment of the leaves is RH, the saturated vapor pressure of the leaves is SVP (KPa), the temperature of the leaves is T (T leaf )(°C), the temperature of the surrounding environment of the leaves is Tm (T temperature of the surrounding environment )(°C), ΔT = T leaf -T temperature of the surrounding environment , ΔT = T - Tm, VPD = ((100 - RH) / 100) × SVP. A graph with ΔT on the vertical axis and VPD (KPa) on the horizontal axis is shown. Here, VPD (Vapor pressure deficit) is the difference between the saturated water vapor pressure of the air around the leaf and the leaf, and is a parameter for evaluating the ease of water evaporation from the leaf.

[0085] RH is the humidity (relative humidity) measured near the target leaf (at a position generally within a distance of 30 cm or less). The saturated vapor pressure SVP (Saturated Vapor pressure) of the leaf is obtained from the temperature T of the leaf. Generally, it is known that the saturated vapor pressure depends on temperature. Here, for the target tomatoes, the relationship between the temperature T of the leaf and SVP has been obtained in advance, and table data of the correspondence relationship has been prepared in advance. Based on this table data, SVP is obtained from the measured temperature T of the leaf. Generally, the higher the temperature, the higher the saturated vapor pressure. Therefore, the higher the temperature T of the leaf, the larger the value of SVP.

[0086] The temperature T of the leaf is obtained from the thermal image (infrared photographic image) obtained by imaging the leaf with an infrared camera. This thermal image is obtained by an infrared camera that is sensitive to infrared light (far-infrared light) with a wavelength of 8 μm to 14 μm.

[0087] The temperature Tm of the ambient environment of the leaf is measured by a temperature sensor installed near the target leaf (at a position generally within a distance of 30 cm or less). The humidity (relative humidity) RH of the ambient environment of the leaf is measured by a humidity sensor installed at the same position as the temperature sensor.

[0088] The plotted points shown in FIG. 9 are obtained by adjusting the temperature Tm (°C) and humidity (relative humidity) of the environment so that the values of VPD and ΔT become specific values. The period for maintaining this environment starts from the stage when flower buds are formed and ends at the stage before the fruit sets and turns red (the stage when the color of the fruit is green).

[0089] The value of VPD is affected by various parameters. For example, when the temperature Tm of the ambient environment of the leaf changes, the respiratory state of the leaf and the state of water release from the leaf change, the temperature T of the leaf changes, the saturation vapor pressure SVP of the leaf changes, and the value of VPD changes. Also, for example, if the humidity RH of the ambient environment of the leaf is changed, the value of VPD changes as is clear from the formula indicating VPD. Also, for example, by performing ventilation such as introducing outside air into the greenhouse, Tm and RH change, and the value of VPD changes. Also, when light shines on the leaf, due to photosynthetic activities etc. in the leaf, the temperature T of the leaf changes, and also the humidity RH of the ambient environment of the leaf is affected by the water released from the leaf. Also, if water is supplied to the roots in a state of water shortage, water is supplied to the leaf, and the temperature T of the leaf is affected. Also, when fine mist cooling is sprayed on the leaf, the temperature T of the leaf decreases, SVP becomes smaller, and the value of VPD becomes smaller.

[0090] Therefore, by adjusting the combination of parameters such as temperature, humidity, irradiated light, water supply to the roots, and spraying of fine mist cooling, the value of VPD can be controlled. In the case of FIG. 1, the illustrated data is obtained by adjusting the temperature Tm and humidity RH of the atmosphere near the tip of the target tree crown.

[0091] As shown in Fig. 9, a correlation approximated by a straight line is observed between ΔT on the vertical axis and VPD on the horizontal axis. However, the variation (deviation) from the straight line for fitting in the range of 2.2 < VPD becomes significantly large. That is, with VPD = 2.2 as the threshold, the region where the state of dispersion with respect to the linear relationship between ΔT and VPD is relatively small (the region of VPD < 2.2) and the region where the state of dispersion with respect to the straight line is relatively large (2.2 < VPD) are separated.

[0092] Fig. 10 is a graph with VPD on the horizontal axis and (y - ΔT) on the vertical axis. Here, y is the fitting straight line in Fig. 1 (y = -2.1138x - 3.2768). ΔT = T (leaf temperature) - Tm (ambient temperature).

[0093] In Fig. 10, (y - ΔT) on the vertical axis indicates the variation of the data. As shown in Fig. 10, in the region where VPD is 2.2 or less, the variation of the data is relatively small (STD (standard deviation) = 0.56), and in the region where VPD exceeds 2.2, the variation of the data is relatively large (STD = 3.36).

[0094] As is clear from Fig. 10, the variation of (y - ΔT) is small when VPD < 2.2. This indicates that the linear relationship between ΔT and VPD is relatively strong when VPD < 2.2. In contrast, when 2.2 < VPD, the variation of (y - ΔT) is large. This indicates that the linear relationship between ΔT and VPD is relatively weak. In this case, the deviation from the linear relationship has a five-fold difference in standard deviation. By performing data processing like that in Fig. 10, the threshold value of VPD can be determined.

[0095] Here, there is a clear correlation between the state of dispersion of the plot points and the actual cracking of the obtained tomatoes. First, in the group of samples with VPD < 2.2, the maximum value of the solids with actual cracking was about 5%. Also, some trusses had no actual cracking. On the other hand, in the sample group with 2.2 < VPD, actual cracking was observed in 50% or more.

[0096] Therefore, when considering VPD, it is concluded that the threshold value (VPDth) for the occurrence of physiological disorders is VPDth = 2.2.

[0097] From the above analysis, it can be concluded that by controlling the atmosphere (environment) in the greenhouse so that VPD < 2.2, it is possible to cultivate tomatoes with fewer physiological disorders.

[0098] Hereinafter, the method for obtaining VPDth will be described. First, the first method will be described. In the first method, data on the relationship between ΔT and VPD measured in advance is obtained to obtain data corresponding to Figure 9. Next, the situation of physiological disorders is investigated, and the value of VPD (VPDth) at which physiological disorders become apparent is obtained. For example, a graph is created with VPD on the horizontal axis and the physiological disorder rate in the truss on the vertical axis. Then, the VPD at the part where the slope in this graph changes rapidly is obtained as VPDth.

[0099] Next, the second method will be described. In the second method, data on the relationship between ΔT and VPD measured in advance is obtained to obtain data corresponding to Figure 9. Then, a straight line that fits the plotted points is obtained, the difference between this straight line and each plotted point is obtained, and a graph corresponding to Figure 10 is obtained. Then, the value of VPD at which the value of (y - ΔT) varies greatly as shown in Figure 10 is obtained as VPDth.

[0100] Specifically, data like that in Figure 10 is created, and two regions with a difference in STD (standard deviation) are obtained. In the case of Figure 10, there is a five-fold difference in STD. As a guideline, two regions with a difference of 2 times or more, preferably 3 times or more, in STD are found, and the VPD at the boundary part is obtained as VPDth.

[0101] The lower limit VPDmin of VPD is about VPDmin = 0.4. When VPD < 0.4, the possibility of occurrence of other physiological disorders and diseases increases. This is the same for cases other than tomatoes.

[0102] (10) Production system related to VPD control An example of a crop production system based on the findings obtained from the data in Fig. 9 will be described below. Here, the case of tomatoes as the crop will be described. Applicable crops include cucumbers, eggplants, bell peppers, melons, squash (pumpkins), grapes, oranges, strawberries, spinach, etc.

[0103] Here, the agricultural greenhouse 300 in Fig. 1 is utilized to grow the tomato canopy 100. In this case, as the threshold value of VPD (VPDth) related to physiological disorders, VPDth = 1.4 is adopted. Although the VPDth obtained from Fig. 1 is 2.2, here, with a margin, VPDth = 1.4 is adopted.

[0104] Note that if the tomato variety is different, this value may be slightly different. Therefore, precisely, it is preferable to obtain VPDth for each variety to be cultivated. However, for tomatoes, regardless of the variety, by setting VPDth = 2.2 (VPD < 2.2), and with a margin, VPDth = 1.4 (VPD < 1.4), a certain degree of effect of preventing physiological disorders can be obtained.

[0105] Here, an example of the case where the humidity and temperature inside the agricultural greenhouse are controlled to achieve VPD < 1.4 will be described. Here, the adjustment of humidity is performed by spraying and dehumidifying with fine mist cooling, and the adjustment of temperature is performed by a heating and cooling device. Of course, it is also possible to control VPD by controlling parameters such as ventilation, lighting, illumination, and irrigation.

[0106] In this example, the temperature Tm and humidity RH of the atmosphere near the tip (upper end) 101 of the tomato canopy 100 are measured. Also, the leaves near the tip (upper end) 101 of the tomato canopy 100 are imaged by an infrared camera to obtain the temperature T of the leaves near the tip (upper end) 101 of the tomato canopy 100. The acquisition of the leaf temperature T is performed by the leaf temperature acquisition unit 205 in Fig. 2.

[0107] In this case, the determination unit 206 in FIG. 2 makes a determination for control according to the management table shown in FIG. 11. Based on the result of this determination, the control content determination unit 210 in FIG. 2 determines the content of the control, and an instruction regarding the determined control is given to the cultivation environment control device 400. The determined control is executed by the cultivation environment control device 400.

[0108] For example, assume that the state of 0.2 ≦ VPD < 0.4 (KPa) continues for 30 minutes or more. In this case, the VPD stage 2 in FIG. 11 is determined by the determination unit 206, and dehumidification of the atmosphere is performed according to the table in FIG. 11.

[0109] The VPD stage 2 is determined to be in a high humidity state. Since it is in a high humidity state, the moisture in the leaves becomes excessive. Therefore, control is performed to dehumidify and reduce the moisture content in the atmosphere, and to create an environment in which moisture easily evaporates from the leaves.

[0110] In this case, by dehumidifying, the humidity (RH) is reduced, and the value of VPD represented by VPD = ((100 - RH) / 100) × SVP is increased.

[0111] Also, for example, assume that the state of VPD > 2.2 (KPa) continues for 15 minutes or more. In this case, the VPD stage 5 in FIG. 11 is determined by the determination unit 206, and irrigation (watering), humidification by the humidifying device, and cooling by the air conditioner (reduction of the atmosphere temperature) are performed according to the table in FIG. 11.

[0112] The VPD stage 5 is determined to be in an extremely low humidity state. Since it is in a low humidity state, moisture is easily lost from the leaves. Therefore, as environmental control, control is performed to supply moisture to the leaves by irrigation, suppress the evaporation of moisture from the leaves by humidification, and increase the atmosphere humidity by reducing the atmosphere temperature.

[0113] That is, by performing humidification, the humidity is increased, and the value of VPD represented by VPD = ((100 - RH) / 100) × SVP is decreased. Also, generally, the higher the temperature, the higher the saturation vapor pressure. Therefore, by lowering the ambient temperature, SVP (the saturation vapor pressure of the leaf) is decreased, and VPD is decreased.

[0114] Environmental control is performed continuously or intermittently while monitoring the value of VPD. And the control is stopped when 0.4 (KPa) ≤ VPD < 1.4 (KPa). As shown in the table of FIG. 11, in the range of 0.4 (KPa) ≤ VPD < 1.4 (KPa), it is determined as the optimal VPD range, and no special environmental control is performed.

[0115] The management table of FIG. 11 is created and prepared in advance. Also, the criteria for determination and the content of control vary depending on the target crop, and it is necessary to prepare the management table of FIG. 11 corresponding to the target crop.

[0116] (11) An example of the process related to VPD control Hereinafter, an example of the process performed by the system of FIG. 1 regarding the control of VPD will be described. An example of the procedure of the process is shown in FIG. 12. The program for executing the process of FIG. 12 is stored in the storage area (such as a hard disk or a semiconductor memory) of the PC constituting the analysis device 200 and is executed by the CPU provided in the analysis device 200. It is also possible to store the program in an appropriate storage medium or storage server and download and use it therefrom.

[0117] Prior to the process, the data of FIG. 9 related to the target crop is acquired in advance, and the threshold value of VPD related to physiological disorders is obtained. And the management table of FIG. 11 is created.

[0118] The process of FIG. 12 starts from the germination stage. When planting something that is already in the seedling state, the process starts from the time when the seedling is planted.

[0119] As for the frequency and timing of executing the process shown in FIG. 12, examples include a form in which it is continuously and constantly executed, or a form in which it is executed at specific time intervals.

[0120] When the process is started, first, data of the thermal image of the target leaf is acquired (step S201). Here, the tree crown (seedling) grows in a form that grows vertically upward, and the leaves at the tip (upper end) of the growth are the observation targets (targets for acquiring the temperature T of the leaves).

[0121] Next, processing steps S202 to S205 for obtaining the temperature T of the leaf are performed. These processes are performed by the temperature acquisition unit 205 in FIG. 2. First, in the thermal image obtained in step S201, the background of the leaf is removed (step S202). This process is performed by the background removal unit 211 in FIG. 2.

[0122] After removing the background of the leaf, the thermal image of the leaf is extracted (step S203). This process is performed by the leaf thermal image extraction unit 212 in FIG. 2. Next, based on the extracted thermal image of the leaf, a map of the temperature distribution of the leaf is created (step S204). This process is performed by the leaf temperature map creation unit 213 in FIG. 2.

[0123] Examples of the leaf temperature map include the following. For example, in the case of the thermal image of one or several leaves, the temperature distribution in one or each leaf is mapped. For example, in the case of a thermal image showing a large number of leaves, a map of the temperature distribution for each leaf can be obtained. Of course, an intermediate map between the two may also be obtained.

[0124] After obtaining the leaf temperature map, the temperature T of the leaf is determined (calculated) (step S205). In this case, based on the temperature map, the average value of the temperature distribution is determined as the temperature of the leaf. This process is performed by the leaf temperature determination unit 214.

[0125] After obtaining the temperature of the leaf, the temperature Tm around the leaf and the humidity RH around the leaf are acquired (step S206). The acquisition of Tm is performed by the ambient temperature acquisition unit 202 in FIG. 2, and the acquisition of RH is performed by the ambient humidity acquisition unit 203 in FIG. 2.

[0126] The ambient temperature Tm and the ambient humidity RH are obtained in the vicinity of the leaves near the tip (upper end) 101 of the tomato canopy 100, which is the object of observation. This is because flower buds are formed at the tip (upper end) of the canopy, and by controlling VPD for the leaves in that part, physiological disorders (such as cracked tomatoes and bottom-end damaged fruits, etc.) can be effectively suppressed.

[0127] Next, the saturated vapor pressure SVP (Saturated Vapor pressure) of the target leaf is obtained (step S208). SVP is obtained by applying the temperature T of the leaf obtained in step S205 to the table data in which the relationship between T and SVP has been previously determined. This process is performed by the SVP acquisition unit 208 in FIG. 2.

[0128] Next, VPD is calculated from VPD = ((100 - RH) / 100) × SVP (step S208). This process is performed by the VPD calculation unit 209 in FIG. 2.

[0129] Once VPD is obtained, a determination is made based on the management table in FIG. 11 to determine whether environmental control is necessary (step S209). This process is performed by the determination unit 206 in FIG. 2. If environmental control is necessary, the environmental control defined in the management table in FIG. 11 is performed (step S210). If environmental control is not necessary, the processes below S201 are repeated. Also, after step S210, the processes below S201 are repeated.

[0130] The process in FIG. 12 is performed until the tomato fruits set and turn red. After the stage when the fruits turn red, cell division, which causes physiological disorders, is almost complete, and the preventive effect of physiological disorders by controlling VPD becomes low. Therefore, the process in FIG. 12 may be terminated when redness is observed.

[0131] The part to be the target of VPD calculation is preferably the part where active cell division occurs during the process of fruit formation at the tip of the tree crown. This is because active cell division occurs during the process from flower bud formation to fruit set, and the environment during this process is greatly related to the generation of factors causing physiological disorders. Conversely, after the fruit has grown to a certain size, the effect of preventing physiological disorders by controlling VPD becomes less significant. In the case of tomatoes, since flower buds are formed at the tip (uppermost part) of the tree crown, it is preferable to control VPD for the leaves at the tip (upper part) of the tree crown. Also, it is desirable to control VPD during the period when cell division related to the fruit occurs.

[0132] (Application to other crops) For example, in the case of eggplants, by setting the range of 0.4 (KPa) ≤ VPD < 1.9 (KPa), the effect of suppressing physiological disorders can be obtained. Also, in the case of cucumbers, it is preferable to control in the range of 0.4 (KPa) ≤ VPD < 1.5 (KPa). In both cases, when VPD is below 0.4 (KPa), fungi or other diseases tend to occur, so it is appropriate to set the lower limit of VPD to 0.4 (KPa).

[0133] (12) Countermeasures for the case of growing multiple tree crowns When there are multiple tree crowns, each tree crown or a group of multiple tree crowns is set as the target for measurement and control. At this time, pay attention to being able to control the environment for each target as much as possible. Also, as a general tendency, it is also possible to perform control so that the parameters to be judged (leaf temperature, VPD, etc.) do not deviate extremely from the standard values. Also, with partitions or simple greenhouses, etc., it is possible to configure the space to be divided for each measurement and control target to make it easier to control each target individually.

[0134] (13) Conclusion In the above-described technique, the temperature of the leaves of the crop is obtained from a thermal image. At this time, the separation of the leaves and the background in the thermal image is performed using deep learning. Further, based on the obtained leaf temperature, the area ratio of the leaves at that temperature, and the duration of that temperature, the control of the growing environment is performed. Further, the humidity of the ambient environment of the leaves is RH, the saturation vapor pressure of the leaves of the crop is SVP, the temperature of the leaves is T, the temperature of the ambient environment of the leaves is Tm, ΔT = T - Tm, and VPD = ((100 - RH) / 100)×SVP. The environment is controlled so that VPD falls within a specific range. By controlling the environment so that VPD falls within a specific range, physiological disorders and the like can be prevented.

[0135] Since the above control is performed based on objective measurement values, it has high reproducibility. For this reason, stable harvesting of agricultural products can be achieved.

[0136] (14) Explanation of the network (learning network) for separating the leaves and the background in the thermal image Hereinafter, an example of the network used in deep learning will be described.

[0137] (First) When obtaining the temperature of the leaves from a thermal image, it is important to separate the leaves and the background in the thermal image. Heat is also radiated from the background, and it is necessary to eliminate its influence. The present inventors have been researching to achieve the above object by deep learning. However, it has been difficult to achieve the above object with known deep learning models.

[0138] (Outline) Therefore, the present inventors have developed the network models shown in FIGS. 13 and 30. Here, the input image is a thermal image (infrared image) taken by a thermographic camera having sensitivity in the wavelength range of 8 μm to 14 μm.

[0139] The networks in FIGS. 13 and 30 can be divided, as a global structure, into downsampling for feature extraction and upsampling for imaging the extracted features and finally obtaining a thermal image of a leaf with the background removed. The downsampling and upsampling are performed in multiple stages. A detailed description of each process will be given later.

[0140] Here, the main points will be briefly described. In this network model, in downsampling, the operation path branches into four channels. The part from block B to block C is that part. The upper part of FIG. 15 shows the details of this part.

[0141] The upper part of FIG. 15 is the case where an appropriate filter group (four filters) is selected, and the lower part is the case where an inappropriate filter group (four filters) is selected. The branching of the above four-channel path is performed by individually selecting the filters for performing the convolution process on the image data. Here, the selection of the filter is important.

[0142] As shown in images 30, 39, 12, and 21 in the upper part of FIG. 15, the images obtained by the filters used are different. This is because different parts of the image information are selectively filtered by the filters. In other words, by changing the way of convolution for a feature map at a certain stage, a difference occurs in the resulting feature map. This difference becomes the difference in the above images.

[0143] Filters are available in a variety of types related to wavelength, intensity, handling of edge parts, and handling of pixels. Here, four different filters are selected and the output images are visually inspected and judged. The four filters were selected and used from among those prepared by the software used for constructing the network.

[0144] In the software used, out of the 1024 types of prepared filters, 64 types of filters were randomly selected for one group convolution layer and displayed, and one good one was visually selected from among them. This was done for four group convolution layers, and as a result, four filters were selected. Among the four images obtained from them, (1) It is clear overall. (2) The parts where features appear to be emphasized are different. (3) The color tones and light and dark parts are different. (4) The resolutions are different. Those that meet the following requirements were selected. This selection was made visually by the inventor himself. Here, a set of four that best meets the above requirements was selected, and the set of four filters used at that time was adopted.

[0145] (Considerations regarding filter selection) A thermal image is an image of the intensity of the thermal radiation of an object. In a thermal image, not only temperature but also differences in the spectrum of the radiation wavelength affect the difference in the image. Also, not only temperature but also the effects of moisture and the emissivity of the object appear in the thermal image, and the situation is complex. This is different from the case of a visible light image that is imaged as a combination of wavelengths corresponding to RGB.

[0146] Here, consider the leaves of a plant as the object. The radiant temperature characteristics of the leaves are different from those of a simple solid substance. That is, the leaves are living organisms, and matters such as the presence of chlorophyll that absorbs light of a specific wavelength, photosynthesis related to chlorophyll, the presence of moisture in the action of photosynthesis, respiration at the stomata, and the release of moisture from the stomata accompanying respiration affect the thermal radiation.

[0147] Therefore, if the influence on the thermal image of the leaves can be extracted by a filter, the characteristics of the thermal image unique to the leaves can be extracted, and the separation of the leaves and the background in the thermal image can be effectively performed. The influence on the thermal image caused by these leaves is considered to extend to various parts.

[0148] Among them, the main factors are considered to lie in the differences in the form and intensity of the wavelength spectrum of thermal radiation. Also, the specificity of the leaf shape appears in the thermal image. For example, it is the shape of the edge part of the leaf, the shape of the stem and the leaf, their combinations, etc. Also, the presence of leaf veins (vascular bundles of the leaf) is considered to have a great influence on the thermal image. Leaf veins are the passageways for water and nutrients, and the thermal radiation from them affects the thermal image. The combination of these elements becomes the information unique to the leaf in the thermal image.

[0149] The combination of the four filters that meet the above requirements (1) to (4) is considered to be effective for extracting information that highlights the leaf-specific thermal image information described above. First, by meeting the requirement of (1), image information in which specific features are effectively extracted by the filter can be obtained. By meeting the requirements of (2), (3), and (4), various combinations of image information with differences in information related to wavelength and shape can be obtained. Here, those that meet at least two of the requirements of (2) to (4) are selected.

[0150] For example, the images of 39 and 21 in the upper part of Fig. 16 seem to be images in a mutually inverted relationship. This is considered to be due to the difference in the extracted wavelengths. This tendency also applies to the images of 12 and 21. Also, the images of 39 and 30 have a large difference in the prominent feature parts. This is considered to be due to the combination of the difference in wavelength information and the feature patterns and their distributions of the shape. Also, these two images have different brightnesses.

[0151] There are various types of wavelength information of thermal radiation. For example, there are differences in the peak wavelength, differences in the peak values of the wavelength, differences in the wavelength distribution (differences in the shape of the spectrum), etc. Also, there are various types of differences in the wavelength distribution. Also, these are affected by the state of the leaf as a living body. Furthermore, information related to the leaf shape is combined with these elements.

[0152] These differences in various information appear in the thermal image as differences in the appearance of the characteristic part, the presence or absence of the characteristic part in the image, differences in shadows, differences in light and darkness, and differences in their combinations. Therefore, by selecting a large number of variations of this difference, it is possible to effectively incorporate information unique to the leaf.

[0153] Also, it is important to select a filter that can obtain images with different resolutions in (4). When the resolution is high, information with a higher wavelength can be obtained, but on the other hand, the influence of noise also increases. Therefore, it can be said that an image with a low resolution also contains useful information for removing noise. In this sense, an image with a relatively low resolution is also required.

[0154] By the way, the combinations of the above requirements are enormous. In this regard, it is better to have as many logical branches as possible in order to incorporate more diverse information. However, increasing the number of filters and the number of logical branches increases the amount of calculation, increases the burden on the arithmetic unit (computer), and reduces practicality.

[0155] The inventors of the present invention have determined that in the process of transitioning from block B to block C in FIG. 13, if the number of branches is 4, it is possible to separate the leaf and the background significantly in practical use, and the increase in the amount of calculation can also be suppressed. If the number of branches is 4, it becomes a combination of four pieces of image information, so the above-described differences in various wavelength information can be incorporated into the next calculation process. Note that if the number of branches is 3 or less, the wavelength information is insufficient and the separation efficiency between the leaf and the background is significantly reduced. If the number of branches is 5 or more, the effect is enhanced, but the amount of calculation increases.

[0156] According to an experiment on tomato leaves, a numerical value of 95% or more has been obtained as the separation effect between the leaf and the background. In view of this superiority and practicality, the number of branches is set to 4 in the above example. If the progress of hardware enables larger-capacity calculations to be performed at lower cost and in a shorter time, it is also effective to set the number of branches to 5 or more.

[0157] (Details of the network) (Preliminary explanation) The details of the network in FIG. 13 will be described below. First, terms will be explained.

[0158] Convolution is a process that compresses feature amounts through convolution. Through convolution, the features of an image are efficiently converted into data.

[0159] Grouped Convolution divides the input data into groups in the layer direction, performs convolution on each divided data, and finally combines and outputs them.

[0160] MaxPooling performs compression by taking the maximum value within each region. For example, in a region where there are four (2×2) small regions (a total of 4×4 region), the maximum value within the 2×2 small region is adopted, and the features of the 4×4 region are compressed to 2×2. This maintains the features of the image while compressing the image size and reducing the computational burden in subsequent processes. In this case, 16 regions of 4×4 are compressed to 4 regions of 2×2.

[0161] ReLU is a method of a process that manifests features using an activation function.

[0162] Batch Normalization is a process that normalizes the output.

[0163] Depth concatenation is a process that performs concatenation in the depth direction of information. For example, assume there is image data A of 32×32×3 and image data B. Here, each element of the information is width × height × depth. For example, 32×32 is the information in the two-dimensional direction of the image (shape information), and the last 3 indicates that there are three types of information (e.g., color and density) for each pixel. Note that as the compression of information progresses, information related to the shape is also convolved and included in this depth-direction information.

[0164] In the above case, the process of concatenating image data A and image data B in the depth direction is Depth concatenation.

[0165] Transposed Convolution is a process that restores the original image from the image information that has been convolved and had its feature compressed by Convolution.

[0166] The Addition layer is a layer where the process of adding multiple inputs is performed.

[0167] Crop2D adjusts the height and width of the data.

[0168] (Schematic Explanation) The following explains the hierarchical diagram in FIG. 13. FIG. 30 is a conceptual diagram of FIG. 13 seen from another perspective. In this network, from BlockA to BlockD, the extraction of the features of the leaf image information in the thermal image data, that is, the manifestation of the information related to the thermal image of the leaf, is performed in multiple stages.

[0169] The image information to be extracted (manifested) here is the information unique to the leaf captured in the thermal image. In other words, by effectively extracting the thermal image information indicating that it is a leaf in multiple stages, the thermal image information unique to the leaf is compressed and manifested so that denser information can be obtained. On the other hand, by deepening the thermal image information unique to the leaf in multiple stages, the thermal image information that is not of the leaf is excluded accordingly. In other words, the features of the thermal image of the leaf are strengthened, and the features of the thermal image other than the leaf are weakened. This process is performed by the downsampling from BlockA to BlockD.

[0170] From BlockE to BlockG, based on the compressed thermal image information of the leaf, the process of reproducing the original (input-time) thermal image of the leaf is performed. The data at the end stage of BlockD is the data obtained by extracting the features of the thermal image of the leaf. However, as a result of only extracting the information of the feature part, the feature is abstracted as digital data, and the information as a visual image is lost, and it cannot be recognized as an image.

[0171] Therefore, after BlockE, image restoration processing based on leaf characteristics is performed. Since the final thermal image of the leaf is based on the leaf-specific data obtained up to BlockD, the background (parts other than the leaf) is removed, and only the leaf remains. That is, a thermal image of the leaf with the background separated can be obtained.

[0172] (Details of the network) The image to be processed is a thermal image (infrared image) taken by a thermography camera with sensitivity in the wavelength range of 8 μm to 14 μm. This thermal image has 1040×780 pixels and 24-bit grayscale information. This thermal image was resized to 240×240 pixels, noise was removed, and a binary image was used as the input image.

[0173] The original number of thermal images used was 13,766, which were increased to 55,064 through image conversion (e.g., enlargement, reduction, rotation, translation, etc.). For these thermal images, the pixels of each image were manually classified into two groups: the leaf group and the background group. The pixels classified as leaves were 77%, and the pixels classified as the background were 23%. Of these 55,064 thermal images, 60% were used for learning, 20% were used for verification, and 20% were used for testing purposes.

[0174] Figure 14 shows the details of the processing of BlockA in Figure 13 and the resulting image. Here, the input image is image data with pixel dimensions of 240 pixels × 240 pixels × 3 (RGB) in binary image form. Convolution is performed on this input image using 32 filters to create a 240×240×32 feature map (Map 1). Here, a 240×240×1 feature map is created for each filter.

[0175] Next, ReLU is used to manifest the features of the feature map of Map 1 and obtain Map 2. Furthermore, convolution is performed using 64 filters to create a feature map of 120×120×64 (Map 3). At this time, each feature map is compressed to 120×120 according to the convolution method.

[0176] After that, batch normalization is performed (Map 4), and ReLU is used to manifest the features of Map 4 (Map 5).

[0177] Next, proceed to the process of Figure 16. In the process of Map 6 in Figure 16 (max pooling), while maintaining the features of the image, the image size is compressed to 60×60×64. Figure 15 shows the presence or absence of the effect of performing max pooling at this stage. At the right end of the upper row of Figure 15, the image obtained as a result of performing max pooling at the stage of Map 6 is shown, and at the right end of the lower row of Figure 15, the image obtained when max pooling is not performed at the stage of Map 6 is shown.

[0178] It can be understood from Figure 15 that by performing max pooling at the stage of Map 6, the manifestation of features, in other words, the effective extraction of features, is carried out.

[0179] After Map 6, proceed with the processes of convolution (Map 7), batch normalization n (Map 8), ReLU (Map 9), and convolution (Map 10) to further advance the extraction of features. After obtaining Map 10, Map 11 is obtained by ReLU (the process of manifesting features using an activation function).

[0180] Here, grouped convolution is performed on Map 11 and convolution is performed by dividing it into 4 channels to obtain Maps 12, 21, 30, and 39. At this time, Maps 12, 21, 30, and 39 are obtained using different filters respectively.

[0181] The following will explain in detail the processing of the part that migrates from Map 11 to Maps 12, 21, 31, and 39.

[0182] First, assume that Convolution is performed on Map 11 (60×60×64) to obtain a Feature map 12 (Map 12) of 60×60×64. And assume that the image of this Feature map 12 is Image 12 in Fig. 16. Here, Map 11 is data of 60×60×64, and there are 64 images of 60×60 pixels. Here, one of these 64 images is selected and shown in the figure.

[0183] Here, when performing Convolution on Map 11, consider changing the method of Convolution. Changing the method of Convolution will change the combination of filters used, resulting in different convolutions and different results (of course, there may also be cases where this is not the case). This is because differences in the compression of feature quantities, the degree of compression of feature quantities, the balance of the degree of compression of multiple feature quantities, etc. occur due to differences in the convolution procedure.

[0184] The following steps will be described in detail. Here, there are 64 types of filters implemented in the software used. Therefore, in this case, there are a total of 64 convolutions, namely the 1st Convolution, the 2nd Convolution, the 3rd Convolution, ··· the 64th Convolution.

[0185] First, select 4 from the above 64 convolutions. The selection method is random. Here, these 4 convolutions are called the 1st Convolution, the 2nd Convolution, the 3rd Convolution, and the 4th Convolution. Naturally, the 1st Convolution, the 2nd Convolution, the 3rd Convolution, and the 4th Convolution are different convolutions.

[0186] Here, perform the first Convolution on map 11 to obtain the first feature map, perform the second Convolution on map 11 to obtain the second feature map, perform the third Convolution on map 11 to obtain the third feature map, and perform the fourth Convolution on map 11 to obtain the fourth feature map. This process becomes the first Grouped Convolution.

[0187] Then, visualize the first feature map to obtain the first image, visualize the second feature map to obtain the second image, visualize the third feature map to obtain the third image, and visualize the fourth feature map to obtain the fourth image.

[0188] After obtaining these first to fourth images, select any two of them and visually determine whether they satisfy the requirement in (1) below and at least two of the requirements in (2), (3), and (4). Of course, it is also possible to automate this determination using image analysis technology. Here, the inventor made a visual determination.

[0189] (1) It is clear overall. (2) The parts where features appear to be emphasized are different. (3) The color tone and light - dark parts are different. (4) The resolutions are different.

[0190] Perform this determination for all pairs of the two images selected from the above - mentioned first to fourth images, that is, for all combinations of pairs of two images among the four first to fourth images. If the determination for all pairs is YES, that is, if all pairs satisfy (1) and at least two of the requirements in (2) - (4), then adopt the first Grouped Convolution that is the basis of the selected first to fourth images as a candidate for adoption.

[0191] Here, there are 64 types of Convolution. By changing four combinations among them, the second Grouped Convolution, the third Grouped Convolution, etc. are prepared, and the same process is repeated. Although the ideal is like this, in reality, since the number of combinations becomes extremely large, the Grouped Convolution is configured by narrowing down to Convolution that can obtain a clear image and a resolution above a certain level, and the above determination is performed for each Grouped Convolution.

[0192] If there is no Grouped Convolution that passes (meets the criteria) the above determination, the criteria of the requirement (1) above are relaxed to search for a Grouped Convolution that meets the criteria. If there are multiple Grouped Convolutions that meet the criteria, the Grouped Convolution that obtains a set of images that more significantly meets the above requirements is adopted.

[0193] In this way, when any two images are selected, in all combinations, a set of Feature maps that serve as the basis for four images that meet the requirement (1) above and at least two of the requirements (2), (3), and (4) are selected as maps 12, 21, 30, and 39. The above process is the same when the number of branches is n, where n is a natural number of 5 or more.

[0194] Note that when Convolution is performed on map 11 instead of Grouped Convolution, the resolution of the obtained image becomes low as shown in the lower part of FIG. 16.

[0195] The processes after map 12, after map 21, after map 30, and after map 39 are the same. However, since the starting maps 12, 21, 30, and 39 are different, the results of the processes are different.

[0196] At this stage, it is important to perform convolution divided into four channels to obtain maps 12, 21, 30, and 39. This process is executed to effectively separate the leaves and the background in the thermal image.

[0197] Here, if the above process is performed at an earlier stage, the extraction (manifestation) of the characteristics unique to the leaves in the thermal image is insufficient, so the influence of the noise component is large, and the adverse effects of this noise component in each channel become manifest.

[0198] On the other hand, if the above process is performed after this stage, the extraction (manifestation) of the feature amounts progresses too much on the data, resulting in data that is difficult to recognize as an image. As the compression of the feature amounts progresses, the quantification of the features progresses, but on the other hand, the amount of data corresponding to the pixels decreases, and when viewed as an image, it gradually becomes difficult to recognize what kind of image it was originally visually. That is, as convolution is repeated, the quantification of the features of the image progresses and the abstraction of the features of the image progresses, but inversely, information is lost as the visible image.

[0199] Regarding the branches of map 11 ⇒ map 12, map 11 ⇒ map 21, map 11 ⇒ map 30, and map 11 ⇒ map 39, the selection of convolution is being performed by visually recognizing the images of maps 12, 21, 30, and 39. Therefore, if it is difficult to recognize the image at this stage, it becomes difficult to select effective convolution for branching into four channels.

[0200] For the above reasons, map 11 is selected as the feature map in which a certain degree of compression of the feature amounts is performed and the visibility of the imaged object is not impaired.

[0201] The details of the processing of each channel after map 12, after map 21, after map 30, and after map 39 are as shown in FIG. 16.

[0202] After Block B shown in FIG. 16, proceed to Block C in FIG. 17. At this time, Map 20 and Map 29 are combined (synthesized) by Depth concatenation to obtain Feature map 48 (Map 48). Also, Map 38 and Map 47 are combined (synthesized) by Depth concatenation to obtain Feature map 59 (Map 59).

[0203] Here, as a SKIP process, the data of Map 15 and 24 is combined with the data of Map 20 and 29 in the Depth concatenation for obtaining Map 48. Also, the data of Map 33 and 42 is combined with the data of Map 38 and 47 in the Depth concatenation for obtaining Map 59.

[0204] This SKIP process is performed to suppress information loss and stabilize the process. When performing Convolution in multiple stages, although the compression of features progresses, there is also a possibility that necessary information may be lost. In order to incorporate this lost information, the above SKIP process is performed.

[0205] At the stage of Map 48 and 59, the maps are not combined into one, but there are the following reasons for combining them into Map 48 and 59. At the stage of Map 15, 24, 33, and 42, the compression of features has progressed to 30×30×32.

[0206] Here, for the 4-channel processing of Map 12~20, 21~29, 31~38, and 39~47, the image information at the stage of the base maps Map 12, 23, 30, and 39 is selected to be a combination that is as different as possible significantly. Therefore, the features to be compressed in each channel may tend to be very different (it can also be said that it is intentionally done in this way).

[0207] When significantly different feature quantities are combined in Depth concatenation, in the subsequent Convolution, it becomes difficult to compress the feature quantities, and there is a tendency for it to interfere with the compression of effective feature quantities. To mitigate this problem, the data of four maps, map 20 and map 29, and map 15 and map 24 as SKIP data, are synthesized to obtain map 48. Also, the data of four maps, map 38 and map 47, and map 33 and map 42 as SKIP data, are synthesized to obtain map 59. In this way, the four channels are reorganized into two channels.

[0208] At the stage of map 48 and map 59, four 30×30×32 Feature maps are Depth concatenated, so the data volume of the Feature map obtained at this stage is 30×30×128.

[0209] MaxPooling is performed on map 48 to obtain map 49. Also, MaxPooling is performed on map 59 to obtain map 60. At this stage, the data volume is compressed to 10×10×128. Then, at the stage of map 50 and 61, through Convolution, the data volume becomes 10×10×64, and further at the stage of map 56 and 67, the data volume is convolved to 5×5×64.

[0210] Map 58 branches into map 70 and 76, and Convolution is performed on each. Here, the 5×5×64 data is branched into the first 5×5×32 data and the second 5×5×32 data, and the processing of the system of map 70⇒71⇒72⇒73⇒74⇒75 and the processing of the system of map 76⇒77⇒78 are performed. Here, by performing the processing of two systems with different layers, the compression of feature quantities in the wrong direction is suppressed.

[0211] The 5×5×64 SKIP data of map 56, the 5×5×32 data of map 75, and the 5×5×32 data of map 78 are synthesized in map 88 of Figure 18 to obtain 5×5×128 data.

[0212] Also, the 5×5×64 SKIP data of map 67, the 5×5×32 data of map 81, and the 5×5×32 data of map 87 are combined in map 95 of FIG. 18 to obtain 5×5×128 data.

[0213] The 5×5×128 data of map 88 is convolved with 1×1×64 data (map 89). Also, the 5×5×128 data of map 95 is convolved with 1×1×64 data (map 96).

[0214] It branches from map 94 to maps 102 and 105, and convolution of data is performed respectively to obtain two sets of 1×1×32 data. Also, it branches from map 101 to maps 108 and 111, and convolution of data is performed respectively to obtain two sets of 1×1×32 data.

[0215] For the data of map 102, Batch Normalization, which is a process of normalizing the output, is performed to obtain map 103, and ReLU, a process of manifesting features using an activation function, is applied to obtain map 104. Similarly, maps 107, 110, and 113 are obtained.

[0216] In this way, four sets of data, Feature map104, 107, 110, and 113, in which the feature amount of the leaf is compressed to 1×1×32, are obtained. Up to this point is the downsampling process, that is, the process of compressing the feature amount of the leaf. At this stage, digital data in which the feature amount of the thermal image of the leaf is highly abstracted is obtained.

[0217] This data extracts the features of the thermal image of the leaf at a high level, but the pixel data that was initially 240×240 is compressed into 1-pixel data and cannot be imaged as it is.

[0218] Therefore, restoration of the image based on maps 104, 107, 110, and 113 is performed. This restoration of the image is upsampling.

[0219] Details of upsampling are described from FIG. 19 onwards. The branching is repeated from maps 114 and 115 onwards to improve the processing efficiency. Note that in the upsampling stage, the computational burden is less than in the case of downsampling. Therefore, the processing branches shown in FIG. 19 are possible.

[0220] In map 176, the 5×5×64 data of map 56 and the 5×5×16 data of maps 133, 139, 145, and 151 are added to obtain 5×5×128 data.

[0221] In map 177, the 5×5×64 data of map 67 and the 5×5×16 data of maps 157, 163, 169, and 175 are added to obtain 5×5×128 data.

[0222] Proceed from map 183 in FIG. 19 to maps 190 and 210 in FIG. 20, and from map 189 in FIG. 19 to maps 227 and 253 in FIG. 20. In FIG. 20, the path branches unevenly. This is to enhance the reproducibility of the image.

[0223] Maps 209 and 226 in FIG. 20 are combined in map 263 in FIG. 21. Maps 242 and 252 in FIG. 20 are combined in map 273 in FIG. 21. At the stage of maps 267 and 277, it becomes 240×240×32 data and is combined at the stage of map 283.

[0224] Thereafter, at the stage of map 284, the data of map 283 and the input RGB thermal image are combined. Map 283 is data that compresses and digitizes the feature amount of the leaf. By combining map 283 and the input image, the leaf part in the input image is more emphasized and the image other than the leaf is weakened. That is, the data of map 283 functions as a filter that emphasizes the leaf part image (or masks the image other than the leaf) from the input image.

[0225] At the stage of map 290, it becomes 240×240×2 binary pixel data, and further softmax processing is performed to obtain the output image (Output).

[0226] Figs. 22 and 23 show a summary of the network performance. Figs. 24 and 25 show the results of image processing. Figs. 26 to 29 show the functional block diagrams of the network.

[0227] (Appendix) The network shown in this embodiment is developed for the purpose of separating leaves from the background in the thermal image (infrared image) of leaves, and is particularly effective when the thermal image is targeted, but there is no reason to inhibit its application to other than thermal images. Therefore, it is also possible to apply it to images other than thermal images (for example, visible light images). Also, the target is not limited to leaves, and it can also be used for separating a specific target from the image.

[0228] It is also possible to understand the above network as an invention of a method or an invention of a program.

Explanation of Signs

[0229] 100... Tomato canopy, 200... Analysis device, 300... Agricultural greenhouse, 400... Growth environment control device, 101... Near the upper end of the tomato canopy.

Claims

1. A leaf temperature acquisition device for acquiring the temperature of a crop leaf, comprising: a thermal image data acquisition unit that acquires data of a thermal image obtained by imaging the leaf with an infrared camera; a background removal unit that removes the thermal image of the background of the leaf in the thermal image using a deep learning function; a leaf temperature acquisition unit that acquires the temperature of the leaf based on the thermal image from which the background has been removed; and the deep learning function includes: a first step of obtaining a thermal image obtained by infrared imaging of the leaf; a second step of obtaining a base feature map by performing a convolution process on the thermal image; a third step of obtaining at least a first feature map, a second feature map, a third feature map, and a fourth feature map by individually performing a plurality of different convolution processes on the base feature map; a fourth step of performing further convolution processes on the first feature map, further convolution processes on the second feature map, further convolution processes on the third feature map, and further convolution processes on the fourth feature map; and has a network having the above steps, wherein the first image obtained by imaging the first feature map, the second image obtained by imaging the second feature map, the third image obtained by imaging the third feature map, and the fourth image obtained by imaging the fourth feature map each have different features, and it is a leaf temperature acquisition device.

2. The leaf temperature acquisition device according to claim 1, wherein the thermal image is a thermal image based on radiant heat in a wavelength range of 8 μm to 14 μm.

3. The leaf temperature acquisition device according to claim 1 or 2, wherein the deep learning function obtains, based on an RGB image of a leaf of a target crop, a teacher image obtained by removing the background from a thermal image in which the leaf is shown.

4. When paying attention to any two of the first image, the second image, the third image, and the fourth image, (1) It is clear as a whole. (2) The parts where features are emphasized and appear different. (3) The color tone and light and dark parts are different. (4) The resolutions are different. The temperature acquisition device according to claim 1, which satisfies the requirement of (1) above and at least two of the requirements of (2), (3), and (4).

5. Let n be a natural number of 4 or more. In the third step, a first feature map to an nth feature map are obtained. The temperature acquisition device according to claim 1, wherein in the fourth step, further convolution processing is performed on each of the first feature map to the nth feature map.

6. A leaf temperature acquisition device comprising: a thermal image data acquisition unit that acquires data of a thermal image obtained by imaging a leaf of a crop with an infrared camera; a background removal unit that removes a thermal image of the background of the leaf in the thermal image using a deep learning function; a leaf temperature acquisition unit that acquires the temperature of the leaf based on the thermal image from which the background has been removed. The crop growth system is provided with a leaf temperature acquisition device, and controls the growth environment of the crop based on the temperature of the leaf of the crop acquired by the leaf temperature acquisition device. The control of the growth environment is based on a table in which the relationship between the ratio of the area of the leaf of the crop whose temperature is within a specific temperature range, the time during which the temperature range continues in the ratio of the area of the leaf, and the content of the control is determined in advance.

7. A leaf temperature acquisition device comprising: a thermal image data acquisition unit that acquires data of a thermal image obtained by imaging a leaf of a crop with an infrared camera; a background removal unit that removes a thermal image of the background of the leaf in the thermal image using a deep learning function; a leaf temperature acquisition unit that acquires the temperature of the leaf based on the thermal image from which the background has been removed. The crop growth system is provided with a leaf temperature acquisition device, and controls the growth environment of the crop based on the temperature of the leaf of the crop acquired by the leaf temperature acquisition device. The humidity of the ambient environment around the leaf of the crop is RH, the saturated vapor pressure of the leaf of the crop obtained from T is SVP, VPD = ((100 - RH) / 100) × SVP As such, Let T be the temperature of the leaves of the crop acquired by the leaf temperature acquisition device (T leaf ). the range of VPD in which physiological disorders of the crop do not manifest during the growth period of the flower buds of the crop has been acquired in advance, and the control of the growth environment during the growth period of the flower buds of the crop is performed so as to be within the range of VPD in which physiological disorders do not manifest.

8. A leaf temperature acquisition device comprising: a thermal image data acquisition unit that acquires data of a thermal image obtained by imaging a leaf of a crop with an infrared camera; a background removal unit that removes a thermal image of the background of the leaf in the thermal image using a deep learning function; a leaf temperature acquisition unit that acquires the temperature of the leaf based on the thermal image from which the background has been removed. The crop growth system is provided with a leaf temperature acquisition device. ​ ​ ​ A crop cultivation system that controls the cultivation environment of the crop based on the temperature of the leaves of the crop acquired by the leaf temperature acquisition device, wherein the humidity of the ambient environment around the leaves of the crop is RH, Let T be the temperature of the leaf of the crop acquired by the leaf temperature acquisition device. T leaf ) the saturation vapor pressure of the leaves of the crop obtained from the T is SVP, Let the temperature of the surrounding environment of the leaf be Tm (T temperature of the surrounding environment ) ΔT = T leaf −T temperature of the surrounding environment 、 ΔT = T - Tm, VPD = ((100 - RH) / 100) × SVP is defined as, there is a linear relationship between the VPD and the ΔT during the growth period of the flower buds of the crop, in the linear relationship, there are a region where the relationship is relatively strong and a region where the relationship is relatively weak, the control of the cultivation environment during the growth period of the flower buds of the crop is performed so as to obtain the value of VPD in the region where the relationship is relatively strong. A crop cultivation system.

9. The crop is a tomato, The crop cultivation system according to claim 8, wherein the cultivation environment is controlled so that VPD < 2.2 during the period from at least the time when flower buds are formed to the time before the fruits turn red.

10. The object related to the RH, the SVP, the T, and the Tm is the leaf of the part where the flower buds of the crop are formed. The crop cultivation system according to claim 8.

11. A method for acquiring the temperature of a leaf that acquires the temperature of a leaf of a crop, a thermal image data acquisition step of acquiring data of a thermal image obtained by imaging the leaf with an infrared camera, a background removal step of removing the thermal image of the background of the leaf in the thermal image using a deep learning function, a leaf temperature determination step of determining the temperature of the leaf based on the thermal image from which the background has been removed, and comprising, the deep learning function, a first step of obtaining a thermal image obtained by infrared imaging of the leaf, a second step of obtaining a base feature map by performing a convolution process on the thermal image, a third step of obtaining at least a first feature map, a second feature map, a third feature map, and a fourth feature map by individually performing a plurality of different convolution processes on the base feature map, a fourth step of performing further convolution processes on the first feature map, the second feature map, the third feature map, and the fourth feature map, and having a network having. A method for obtaining the temperature of a leaf, wherein the first image obtained by imaging the first feature map, the second image obtained by imaging the second feature map, the third image obtained by imaging the third feature map, and the fourth image obtained by imaging the fourth feature map each have different features.

12. A program for causing a computer to read and execute to obtain the temperature of a leaf of a crop, causing the computer to a thermal image data acquisition step of acquiring data of a thermal image obtained by imaging the leaf with an infrared camera; a background removal step of removing the thermal image of the background of the leaf in the thermal image using a deep learning function; a leaf temperature determination step of determining the temperature of the leaf based on the thermal image from which the background has been removed; to execute, the deep learning function a first step of obtaining a thermal image obtained by infrared imaging of the leaf; a second step of obtaining a base feature map by performing a convolution process on the thermal image; a third step of obtaining at least a first feature map, a second feature map, a third feature map, and a fourth feature map by individually performing a plurality of different convolution processes on the base feature map; a fourth step of performing further convolution processes on the first feature map, further convolution processes on the second feature map, further convolution processes on the third feature map, and further convolution processes on the fourth feature map; having a network having A program for obtaining the temperature of a leaf, wherein the first image obtained by imaging the first feature map, the second image obtained by imaging the second feature map, the third image obtained by imaging the third feature map, and the fourth image obtained by imaging the fourth feature map each have different features.

13. A thermal image data acquisition step of acquiring data of a thermal image obtained by imaging a leaf of a crop with an infrared camera; a background removal step of removing the thermal image of the background of the leaf in the thermal image using a deep learning function; a leaf temperature acquisition step of acquiring the temperature of the leaf based on the thermal image from which the background has been removed; a control step of controlling the growth environment of the crop based on the temperature of the leaf of the crop acquired in the leaf temperature acquisition step; including the control of the growth environment A method for growing crops, which is performed based on a table in which the relationship between the ratio of the area of the leaf of the crop within a specific temperature range, the duration of the temperature range in the ratio of the area of the leaf, and the content of the control is determined in advance.

14. A thermal image data acquisition step of acquiring data of a thermal image obtained by imaging a leaf of a crop with an infrared camera, A background removal step of removing the thermal image of the background of the leaf in the thermal image using a deep learning function, A leaf temperature acquisition step of acquiring the temperature of the leaf based on the thermal image from which the background has been removed, A control step of controlling the growth environment of the crop based on the temperature of the leaf of the crop acquired in the leaf temperature acquisition step and including the humidity of the ambient environment around the leaf of the crop is RH, the temperature of the leaf of the crop acquired by the leaf temperature acquisition device is T (T leaf), the saturation vapor pressure of the leaf of the crop obtained from the T is SVP, VPD = ((100 - RH) / 100) × SVP is defined as Regarding the VPD during the growth period of the flower buds of the crop, the range of the VPD in which the physiological disorder of the crop does not manifest has been acquired in advance, A method for growing crops, wherein the control of the growth environment during the growth period of the flower buds of the crop is performed so as to be within the range of the VPD in which the physiological disorder does not manifest.

Images