Method for estimating tree health, system for estimating tree health, computer program, and optical simulation method

Abstract

This method for estimating tree health is executed by one or multiple computers and involves acquiring one or multiple pieces of image data showing a scene including a tree in a field in a physical space, and estimating an index value indicating the health of the tree on the basis of the image data. When estimating the index value, the relationship between the image of the scene and the total leaf area of the tree is used.

Description

Tree vigor estimation method, tree vigor estimation system, computer program, and optical simulation method

[0001] The present disclosure relates to a method, system, and computer program executed by one or more computers for tree vigor estimation. The present disclosure also relates to an optical simulation method for determining a relationship between an image of a tree and the total leaf area of ​​the tree.

[0002] Research and development is underway into smart agriculture, which utilizes ICT (Information and Communication Technology) and IoT (Internet of Things) as the next generation of agriculture. Smart agriculture aims to improve productivity, alleviate labor shortages, and reduce environmental impact.

[0003] Non-Patent Document 1 describes software that detects leaf and branch areas in an image captured by a monocular camera and estimates the leaf area index of a tree. The leaf area index is the total leaf area per unit land area (total leaf area / unit land area). The leaf area index is one of the index values ​​that indicates the vigor of plants such as trees, and is an important parameter for estimating the photosynthetic capacity, transpiration rate, etc. of plants.

[0004] De Bei, R., et al. “VitiCanopy: A free computer App to estimate canopy vigor and porosity for grapevine.” Sensors, 2016, Vol. 16, No. 4, p. 585

[0005] The above-mentioned conventional technology requires the use of an image captured by placing a monocular camera on the ground or a predetermined distance below the tree canopy, looking up at the tree canopy with the sky as the background.

[0006] The present disclosure provides a method, system, and computer program for estimating tree vigor, as well as an optical simulation method, that relaxes constraints on images used for tree vigor estimation.

[0007] The present invention has the configurations described in the following items.

[0008] [Item 1] A tree vigor estimation method executed by one or more computers, comprising: acquiring one or more image data representing a scene including trees in a field in real space; and estimating an index value representing the tree vigor based on the image data; wherein the index value is estimated using a relationship between the image of the scene and the total leaf area of ​​the trees.

[0009] [Item 2] The tree vigor estimation method according to Item 1, further comprising storing the index value as digital data associated with the scene in a memory within the computer or transmitting the index value to a device connected to the computer.

[0010] [Item 3] The tree vigor estimation method according to item 1 or 2, wherein the relationship is defined by an equation that associates a feature amount of the tree in the image of the scene with the total leaf area.

[0011] [Item 4] The tree vigor estimation method according to Item 3, wherein the formula is determined based on a three-dimensional model of the tree in the simulation space.

[0012] [Item 5] The tree vigor estimation method according to Item 4, wherein the relationship is determined by a process including: determining a total leaf area of ​​the tree based on the three-dimensional model; generating a plurality of images of the tree based on the three-dimensional model; extracting features from the plurality of images; and determining a relationship between the features and the total leaf area.

[0013] [Item 6] The tree vigor estimation method according to Item 5, wherein the feature amount includes pixel information of a leaf area in each of the plurality of images.

[0014] [Item 7] The tree vigor estimation method according to Item 6, wherein the feature amount includes the number of leaf pixels in each of the plurality of images, or the ratio of the number of leaf pixels to the total number of pixels in each of the plurality of images.

[0015] [Item 8] The tree vigor estimation method according to Item 7, wherein the feature amount includes a brightness histogram of leaves in each of the plurality of images.

[0016] [Item 9] The tree vigor estimation method according to Item 6, wherein the feature amount includes at least one of leaf size, tree height, and number of branches in each of the plurality of images.

[0017] [Item 10] The tree vigor estimation method according to any one of Items 1 to 9, wherein the index value includes at least one of the total leaf area, leaf area index, canopy volume, canopy height, average leaf size, variance of leaf size, leaf count density, average branch thickness or length, variance of branch thickness or length, and number of branches for the tree.

[0018] [Item 11] A tree vigor estimation method according to any one of Items 1 to 10, comprising: processing the image data to estimate the depth of each pixel of the scene; selecting leaf pixels within a predetermined depth range based on the depth and color of each pixel; and extracting features related to total leaf area from the leaf pixels.

[0019] [Item 12] The tree vigor estimation method according to any one of Items 1 to 11, wherein acquiring the image data includes: photographing the tree using one or more imaging devices; or receiving digital data acquired by photographing the tree using one or more imaging devices.

[0020] [Item 13] The tree vigor estimation method according to any one of Items 1 to 12, wherein the image data includes a side image of the tree.

[0021] [Item 14] The tree vigor estimation method according to any one of Items 1 to 8, wherein the image data includes a near-infrared image of the scene, and the method further includes extracting a brightness histogram of leaves in the near-infrared image as the feature amount.

[0022] [Item 15] The tree vigor estimation method according to Item 14, wherein the image data includes a visible light image taken from a camera viewpoint different from that of the near-infrared image.

[0023] [Item 16] An optical simulation method for determining a relationship between an image of a tree and a total leaf area of ​​the tree, the optical simulation method including: setting a three-dimensional model of the tree in a simulation space; generating a plurality of images of the tree based on the three-dimensional model; determining the total leaf area of ​​the tree based on the three-dimensional model; extracting feature values ​​from the plurality of images; and determining the relationship between the feature values ​​and the total leaf area.

[0024] [Item 17] The optical simulation method according to Item 16, wherein the feature amount is based on pixel information of a leaf region in each of the plurality of images.

[0025] [Item 18] The optical simulation method according to Item 16 or 17, wherein the three-dimensional model is defined by optical model parameters of the leaf, including at least one of the leaf's refractive index, surface roughness, density, size, absorbance, spectral transmittance, and spectral reflectance.

[0026] [Item 19] A tree vigor estimation system comprising: a storage device that stores one or more image data showing a scene including trees in a farm field in real space; and a processing device that estimates an index value indicating the vigor of the trees based on the image data, wherein the processing device uses a relationship between the image of the scene and the total leaf area of ​​the trees when estimating the index value.

[0027] [Item 20] The tree vigor estimation system described in Item 19, wherein the relationship is defined by an equation that associates feature amounts of the trees in the image of the scene with the total leaf area, and the processing device has a processor and memory, and the memory stores a computer program that causes the processor to execute the following: read image data stored in the storage device, extract feature amounts of the trees from the image of the scene, and apply the feature amounts to the equation to determine the total leaf area.

[0028] [Item 21] A computer program that causes one or more computers to execute a processing step of acquiring one or more image data showing a scene including trees in a farm field in real space, and a processing step of estimating an index value showing the tree vigor based on the image data, wherein the index value is estimated using a relationship between the image of the scene and the total leaf area of ​​the trees.

[0029] [Item 22] A recording medium on which the computer program according to Item 21 is stored.

[0030] [Item 23] A tree vigor estimation system comprising: means for acquiring one or more image data showing a scene including trees in a field in real space; and means for estimating an index value showing the tree vigor based on the image data, wherein when estimating the index value, a relational equation between the image of the scene and the total leaf area of ​​the trees is used.

[0031] According to an embodiment of the present disclosure, a method, system, computer program, and optical simulation method for estimating tree vigor are provided, in which constraints on images used for estimating tree vigor are relaxed.

[0032] FIG. 1 is a flowchart showing processing steps of a tree vigor estimation method according to an exemplary embodiment of the present disclosure. FIG. 2 is a flowchart showing processing steps of an optical simulation according to this embodiment. FIG. 3 is a diagram schematically showing two tree models (Model 1 and Model 2) at different growth stages. FIG. 4 is a scatter plot schematically showing the relationship between the feature values ​​of images obtained for Model 1 and Model 2 and the total leaf area. FIG. 5 is a scatter plot schematically showing the relationship between the feature values ​​of images obtained for multiple models and the total leaf area. FIG. 6 is a flowchart showing a process for extracting feature values ​​from an image showing a scene including trees in a field in real space. FIG. 7 is a diagram schematically showing a portion of a row of fruit-bearing trees in an orchard in real space. FIG. 8 is a diagram showing an example of a scene image acquired by photographing trees in real space with an imaging device. FIG. 9 is a diagram schematically showing a leaf area selected from the image of FIG. 8. FIG. 10 is a diagram of a depth image (depth map) showing depth information for each pixel in grayscale. FIG. 11 is a diagram schematically illustrating an example of a depth histogram in a scene image of a tree. FIG. 12 is a diagram illustrating an example of a binary image acquired by selecting pixels showing a predetermined range of depths from among the pixels constituting the leaf region shown in FIG. 9. FIG. 13 is a diagram illustrating the position and orientation of an imaging device that captures an image of a tree bathed in sunlight. FIG. 14 is a diagram schematically illustrating a leaf region composed of three overlapping leaf pixels in a captured image. FIG. 15 is a diagram schematically illustrating a leaf region composed of three overlapping leaf pixels. FIG. 16 is a diagram schematically illustrating an example of a brightness histogram of a near-infrared image acquired by capturing an image of a tree with a near-infrared camera. FIG. 17 is a diagram illustrating an example of a brightness histogram after normalization. FIG. 18 is a diagram schematically illustrating the position and orientation (orientation) of the near-infrared camera. FIG. 19 is a diagram schematically illustrating the positions and orientations of the near-infrared camera and visible light camera that capture images of the tree. FIG. 20 is a block diagram illustrating an example of the configuration of a tree vigor estimation system according to this embodiment.

[0033] <Terminology> "Tree" refers to a woody plant (standing tree), including tall trees and shrubs.

[0034] "Tree vigor" is an index that indicates the growth and health of a tree. Tree vigor indicates the growth potential, vitality, and resistance of a target tree or group of trees. Tree vigor can be evaluated based on the tree's appearance, physiological function, and ability to adapt to the environment.

[0035] A "tree vigor index" is a numerical value or measurable characteristic value used to quantitatively express tree vigor. These index values ​​are quantified versions of various morphological characteristics and / or physiological parameters of a tree and can be used to objectively evaluate the growth and health of the tree. In the present disclosure, the "tree vigor index" may include at least one of the following for a tree or group of trees of interest: total leaf area, leaf area index, canopy volume, canopy height, average leaf size, leaf size variance, leaf count density, average branch thickness or length, variance of branch thickness or length, and number of branches. These index values ​​may be used individually or in combination. By observing changes in the tree vigor index over time, it is possible to track the progression of the tree's growth and health.

[0036] The term "one or more computers" includes desktop PCs, laptops, tablets, mobile devices such as smartphones, wearable devices, embedded systems, and server computers interconnected via a network. These computers may operate independently or may be configured as multiple computer systems performing distributed processing in cloud computing or edge computing environments. Furthermore, computer systems equipped with specialized computing devices such as dedicated hardware accelerators, graphics processing units (GPUs), and field programmable gate arrays (FPGAs) are also included. In this disclosure, a "computer" typically includes a processor such as a GPU or central processing unit (CPU) and a memory device for storing computer programs and data. Any device or system equipped with these components and capable of executing the tree vigor estimation method of the present disclosure is included in the term "computer" in this disclosure, regardless of its name. Electronic control units (ECUs) mounted on and used in work vehicles such as tractors are also included in the term "computer."

[0037] Hereinafter, a tree vigor estimation method, system, computer program, and optical simulation method according to an embodiment of the present disclosure will be described with reference to the drawings. Parts that appear in multiple drawings with the same reference numerals indicate the same or equivalent parts.

[0038] The embodiments described below are examples for embodying the technical concept of the present invention, and do not limit the present invention. Furthermore, descriptions of the size, material, shape, relative arrangement, etc. of components are intended for illustration purposes only, and are not intended to limit the scope of the present invention. The size and positional relationship of components shown in each drawing may be exaggerated to facilitate understanding.

[0039] (Tree vigor estimation method) The tree vigor estimation method in this embodiment is a tree vigor estimation method executed by one or more computers.

[0040] First, the tree vigor estimation method according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a flowchart showing an example of basic processing steps of the tree vigor estimation method according to this embodiment.

[0041] As shown in Figure 1, the tree vigor estimation method of this embodiment includes step S10 of acquiring one or more image data showing a scene including trees in a field in real space, and step S20 of estimating an index value indicating the tree vigor based on the image data.

[0042] [Acquisition of Image Data] The acquisition of image data in step S10 can be performed by photographing the tree using one or more imaging devices. The imaging devices can acquire images using one or a combination of the following methods: - Portable imaging devices such as cameras mounted on smartphones operated by a person - Imaging devices mounted on agricultural machinery such as tractors - Imaging devices mounted on autonomous mobile robots - Imaging devices mounted on unmanned aerial vehicles such as drones - Fixed-mounted surveillance cameras or time-lapse cameras

[0043] Acquisition of image data is not limited to the above-mentioned method, and can also be performed by receiving digital data acquired by photographing trees using one or more image capture devices. Such digital data can be received via a communication line such as the Internet, a dedicated line, or a wireless communication network.

[0044] Image data is not limited to visible light images, but may also include special images such as near-infrared images, thermal images, multispectral images, and hyperspectral images. These special images are obtained by detecting electromagnetic waves in different wavelength bands. For example, near-infrared images detect near-infrared light (wavelength: 800 nm to 2500 nm), which has a wavelength longer than visible light, while thermal images detect near-mid-infrared light (wavelength: 800 nm to 14 μm) emitted from objects. Multispectral images and hyperspectral images capture a wider range of wavelength bands, dividing them into multiple bands.

[0045] Furthermore, the image data may include not only still images but also moving images (multiple time-series images). In the case of moving images, each frame is treated as an independent image, making it possible to capture changes over time.

[0046] In this embodiment, to acquire color image data, an imaging device is used that includes an image sensor in which each pixel has R (red), G (green), and B (blue) subpixels. Such imaging devices typically use a color filter array, such as a Bayer filter, placed in front of the image sensor. This allows the amount of R (red), G (green), and B (blue) radiation contained in the light beam incident on each pixel of the image sensor to be detected. Therefore, the image data acquired in step S10 includes color information (R value, G value, B value) for each pixel. These values ​​are typically recorded with 8-bit or 10-bit precision and expressed as integer values ​​ranging from 0 to 255 (in the case of 8-bit) or 0 to 1023 (in the case of 10-bit).

[0047] To acquire multispectral image data, an image sensor is used in which each pixel has four or more sub-pixels provided with color filters having different transmission wavelength ranges, which are not limited to the visible light range but may also be in the near-infrared range.

[0048] As described above, the conventional technology described in Non-Patent Document 1 requires a monocular camera to be placed on the ground or a predetermined distance below the tree canopy, and images taken by looking up at the tree canopy with the sky as the background must be used. In contrast, this embodiment alleviates such constraints. For example, it becomes possible to estimate index values ​​indicating tree vigor using color images (side views) taken from the side of a tree. The reason why index values ​​indicating tree vigor can be estimated using such images will be described later.

[0049] [Estimation of Tree Vitality Index Value] When estimating a tree vigor index value in step S20, a relationship between the scene image and the total leaf area of ​​the tree is used (step S25). This "relationship" can be defined, for example, by an equation (relational expression) that associates the "feature value" of the tree in the scene image with the "total leaf area." The feature value can include pixel information of the leaf region in each of the multiple images. For example, the feature value is the number of pixels of the leaves in each of the multiple images. When the number of pixels is used as the feature value, the total number of pixels contained in the images used must be standardized to a predetermined value (e.g., 200,000 pixels). Another example of a feature value is the ratio of the number of leaf pixels to the total number of pixels in each of the multiple images. Such a feature value is not limited to the number of pixels of the leaf region, but may also be a feature value extracted from the image by deep learning. In this embodiment, the number of pixels of the leaves in each of the multiple images is used as the feature. Note that the total number of pixels contained in each image is standardized to 200,000 pixels. In this way, when using images with a predetermined number of pixels, the number of pixels in the leaves can be used as a feature, but if the number of pixels in the images used can vary from image to image, it is preferable to use the ratio of the number of pixels in the leaves to the total number of pixels in each image.

[0050] [Formula relating the number of leaf pixels to the total leaf area] The formula relating the number of leaf pixels to the total leaf area can be determined, for example, by the following process: - Photographing several trees in a real-world field from various angles to obtain a large number of images of scenes containing trees. - Counting the feature value (here, "number of leaf pixels") in each of the above images. - Measure the area (one-sided area) of each leaf on the photographed tree to collect measurements of the total leaf area. - Deriving the relationship between the feature value and the total leaf area using a statistical method. An example of a statistical method is regression analysis.

[0051] However, with the above method, it takes a lot of time to acquire a large number of images for each of a large number of trees and measure the total leaf area.

[0052] In this embodiment, the above formula is determined based on a three-dimensional model of a tree in a simulation space. For example, an optical simulation including the process illustrated in Fig. 2 is executed. An example of the optical simulation method in this embodiment will be described below.

[0053] First, in step S40 shown in FIG. 2 , a three-dimensional model of a target tree is created. The three-dimensional tree model used in this embodiment may specifically include elements such as the trunk, branches, leaves, flowers, and fruit (in the case of fruit trees). These elements may be modeled, including their respective shapes, sizes, textures, and relative positions, depending on the type of tree, growth stage, etc. The three-dimensional model is also defined by leaf optical model parameters, including at least one of the leaf refractive index, surface roughness, density, size, absorbance, spectral transmittance, and spectral reflectance.

[0054] In this embodiment, after a three-dimensional model is set, in step S42, the total leaf area of ​​the tree is calculated based on the three-dimensional model. Once the three-dimensional model is set, the shapes and sizes of all leaves of the tree in the three-dimensional model are known, so the total leaf area of ​​the tree can be calculated from the specific three-dimensional model. For example, if each leaf is represented by a polygon mesh such as a triangular polygon, the leaf area of ​​each leaf can be calculated by calculating the area of ​​the polygon mesh that defines one side of each leaf. The total leaf area can be obtained by calculating the leaf area of ​​each leaf of the tree of interest and then calculating the sum of these leaf areas.

[0055] By performing such calculations, it is also possible to calculate the leaf area index, canopy volume, canopy height, mean leaf size, leaf size variance, leaf count density, mean branch thickness or length, variance of branch thickness or length, and number of branches for any tree in the three-dimensional model.When calculating the total area index, a step of obtaining information about the arrangement of tree rows in the field (such as tree spacing or density) may be performed.

[0056] Next, in step S44, multiple images of the tree are generated based on the set 3D model. Specifically, physically based rendering technology in computer graphics is applied to the 3D tree model to generate images from a virtual camera viewpoint. For example, image generation is performed using the following procedure: - Place one or more 3D tree models in the simulation space. - Set the position, orientation, and optical characteristics (focal length, angle of view, etc.) of the virtual camera. - Set the position of a light source such as the sun. - Use a physically based rendering algorithm to generate a 2D image of the tree as seen from the set camera viewpoint.

[0057] By freely changing the position and orientation (posture) of the camera (image capture device), images from various viewpoints can be generated from the same 3D model. This allows for the generation of multiple images (free-viewpoint images) of trees of any type and at any stage of growth, captured from any viewpoint. A feature indicating tree vigor can be determined from each of the multiple images. In this embodiment, when the feature is the number of leaf pixels in an image, pixels classified as leaves are first selected from all pixels constituting each image. In images obtained by applying physically based rendering technology to a 3D tree model, the correspondence of each pixel constituting the image to an object surface in the 3D model can be determined, for example, by geometrical optical ray tracing. The number of leaf pixels can be determined for each generated image by counting the number of pixels classified as leaves.

[0058] The order of steps S42 and S44 is not limited to the above example, and may be any order.

[0059] In step S46, it is determined whether or not the data necessary to derive the relationship between the aforementioned feature values ​​and the total leaf area by a statistical method has been collected. If it is determined that the data has not been collected (No), the process returns to step S40, and a new three-dimensional model is generated. The new three-dimensional model can be created, for example, by changing model parameters such as the shape, size, and number of at least one of the trunk, branches, and leaves of the tree. After the new three-dimensional model is set in step S40, the above-described processing steps are repeated, and new data is acquired.

[0060] If it is determined in step S46 that the necessary data has been collected (Yes), the process proceeds to step S48, where feature amounts are extracted from the multiple images, and the relationship between the feature amounts and the total leaf area is determined.

[0061] FIG. 3 is a diagram schematically showing two tree models (Model 1 and Model 2) at different growth stages. The total leaf area of ​​the tree in Model 1 is "A1," and the total leaf area of ​​the tree in Model 2 is "A2." As described above, after setting Model 1, a large number of images are generated by photographing the tree in Model 1 with a virtual camera. Similarly, after setting Model 2, a large number of images are generated by photographing the tree in Model 2 with a virtual camera. FIG. 4 schematically shows, with black dots, an example of a dataset determined by the feature values ​​extracted from each image and the total leaf area.

[0062] As shown in FIG. 4, the value of the total leaf area is constant for each tree model. Because the feature values ​​extracted from the image may differ for each image, the data obtained for each model is scattered along the horizontal axis. By performing similar processing for a large number of models, a scatter plot such as that shown in FIG. 5 can be obtained. In the example of FIG. 5, the total leaf area is linearly dependent on the image feature values. FIG. 5 also shows an example of a straight line K that approximately indicates the relationship between the image feature values ​​and the total leaf area. The relationship between the image feature values ​​and the total leaf area cannot always be approximated linearly.

[0063] Based on the dataset showing the relationship between features and total leaf area shown in the scatter plot above, various analyses can be performed to estimate the total leaf area of ​​trees from image features, using linear regression, polynomial regression, or machine learning algorithms (e.g., random forests, support vector machines, etc.).

[0064] Based on the results of the regression analysis, a mathematical model (function) can be constructed that estimates and outputs the total leaf area from the input feature quantities. This mathematical model can be expressed by the following formula: Total Leaf Area = f (Feature Quantity). Here, f(·) represents a function determined, for example, by regression analysis. The function f(·) may be a multivariate function that includes two or more feature quantities as variables, or may be a deep learning model. This function f(·) may vary depending on the type of tree being studied. To estimate the total leaf area for many types of trees from the image feature quantities, it is preferable to prepare a function f(·) for each type of tree.

[0065] To verify the estimation accuracy of the constructed mathematical model, it is preferable to test it using actual images of trees with known total leaf area. If necessary, adjust the parameters of the mathematical model and optimize the selection of features.

[0066] Note that limiting the camera height, orientation, and camera-to-tree distance to a certain range reduces the variance of features across multiple images acquired from the same tree model, improving the accuracy of estimations using regression equations, etc.

[0067] The feature extracted from an image may be, for example, the number of leaf pixels in the image. The feature may be the ratio of the number of leaf pixels to the total number of pixels in each of the multiple images, the size of the leaves in each of the multiple images, the height of the trees, or the number of branches. Furthermore, the type of feature (the number of independent variables) used as an independent variable in the regression equation is not limited to one. Different regression equations may be used depending on information (metadata) related to the date and time of photography, weather conditions, the distance from the trees to the camera, the height of the camera, the orientation of the camera, the angle of view, etc.

[0068] The mathematical model (e.g., function) obtained in this embodiment makes it possible to estimate the total leaf area of ​​a tree using features extracted from an actual tree image as input. This method provides an efficient and non-invasive method for leaf area estimation, replacing traditional destructive sampling techniques and labor-intensive manual measurements.

[0069] This embodiment, compared to the technology described in Patent Document 1, makes it possible to use a variety of images as real-space tree images required for estimation. For example, it is possible to use images of trees photographed from the side. Such images contain information about the vertical leaf arrangement (three-dimensional structure), enabling more accurate estimation of total leaf area. For low-growing shrubs (e.g., grapevines), it may be difficult to photograph the canopy of a tree from directly below the canopy using a handheld camera. In particular, in vineyards, grapevines (shrubs) are arranged in rows along wires, making it easier to photograph the side of the grapevine from an area between the rows than to photograph the canopy from directly below. Furthermore, scene images of shrubs such as grapevines photographed from the side contain information about the three-dimensional structure of the canopy, including its numerous leaves.

[0070] Referring again to FIG. 1 , in step S30, the estimated index value is stored in the computer's memory as digital data associated with the scene. This index value may also be transmitted to an external device connected to the computer. For example, if a computer server that provides users with the tree vigor estimation method of this embodiment is located on the cloud, the computer server can execute the processing steps of FIG. 1 based on image data uploaded by the user to estimate an index value indicating tree vigor, and then notify the user of the index value, for example, in response to a user request.

[0071] The index values ​​estimated by the above method may be stored in a storage device in association with an identification number that identifies the tree or the geographic coordinates of the tree. This makes it possible to display index values ​​indicating tree vigor for each tree, or for each position or segmented area within the field. The display format of the index values ​​is not limited to numerical values, and may include various formats such as the class to which the numerical value belongs, or colors or symbols that indicate whether the numerical value is high or low. The index values ​​may also be displayed in association with the positions of the trees on a map of the field.

[0072] [Feature Extraction] Next, an example of a process for extracting a feature from an image showing a scene including trees in a field in real space will be described with reference to Fig. 6. In this example, the feature is the number of pixels in the leaves. Hereinafter, a "scene including trees in a field in real space" may be simply referred to as a "tree scene," and an image of the tree scene may be simply referred to as a "scene image."

[0073] First, in step S52, the image data acquired in step S10 of Fig. 1 is processed to estimate the depth of each pixel in the tree scene. Depth information can be acquired by the following method.

[0074] Using a monocular camera and a deep learning model: From 2D images taken with a regular monocular camera, it is possible to estimate the "depth" of each pixel or segment classified as a leaf using a deep learning model. This method has the advantage of not requiring additional hardware and can be applied to existing image data.

[0075] Other methods include the following techniques:

[0076] Use of depth cameras: Depth cameras, such as ToF cameras and stereo cameras, can directly capture information about the depth of each pixel. These cameras use the time of flight and parallax of light to measure distance.

[0077] Stereo vision: A method of calculating depth using parallax information from images taken with multiple cameras.

[0078] - Structured light: A technology that projects a known pattern and estimates depth from the distortion of that pattern.

[0079] These methods each have different characteristics, and it is possible to select one depending on the usage environment and the required accuracy.

[0080] By using the depth information estimated by the above method, leaf pixels within a predetermined depth range are selected, which makes it easy to select only the leaf pixels of the target tree even if other trees are captured in the scene image.

[0081] FIG. 7 is a diagram schematically illustrating a portion of a row of fruit-bearing trees (fruit trees) 10 in an orchard in real space. FIG. 7 also illustrates a schematic representation of an imaging device 110, such as a camera. The imaging device 110 is located at a height H relative to the ground G. The line of sight of the imaging device 110 forms an angle θ with respect to the horizontal plane. The angle θ may be defined as having a "positive" value when the line of sight of the imaging device 110 is tilted in an elevation angle direction relative to the horizontal plane, and as having a "negative" value when the line of sight is tilted in a depression angle direction. This definition may also be reversed.

[0082] The position and orientation (attitude) of the image capture device 110 can be expressed by six parameters. Specifically, three parameters represent the position of the image capture device 110, and the other three parameters (e.g., Euler angles) represent the attitude of the image capture device 110. The attitude of the image capture device 110 may be expressed by a quaternion.

[0083] To acquire a scene image from which effective features for estimating the total leaf area can be extracted, it is undesirable for the distance L from the image capture device 110 to the subject tree 10 to be too close or too far. Therefore, the distance L may be set, for example, so that the crown of the target tree 10 occupies 10% or more of the image and so that the image includes 50% or more of the area from the bottom to the top of the crown. Specifically, the distance L may be set, for example, between 50 cm and 500 m. The height H of the image capture device 110 is, for example, between 30 cm and 250 cm. When the image capture device 110 acquires a side view of the tree 10, the angle θ between the line of sight of the image capture device 110 and the horizontal plane is, for example, between −70° and +70°. As described above, a scene image (side view) of the tree 10 captured from the side contains information about the arrangement of leaves in the vertical direction (three-dimensional structure), enabling more accurate estimation of the total leaf area. In particular, when the tree is a grapevine, since grapevines are long vertically, observing the tree from the side, i.e., from the side, has the advantage that the structure of the tree can be more easily recognized.

[0084] When capturing a scene image from which useful features can be extracted to estimate the total leaf area in this manner, it is preferable that the position and orientation of the virtual imaging device that defines the viewpoint of the image generated by simulation to obtain the "formula relating the number of leaf pixels to the total leaf area" is also set within a similar range.

[0085] If the distance from the imaging device to the tree varies significantly, the ratio of the number of leaf pixels to the total number of pixels in each image will vary significantly, even among multiple images of the same tree. Therefore, different "formulas relating the number of leaf pixels to the total leaf area" may be prepared for different distances or distance ranges from the imaging device to the tree. In this case, an appropriate formula can be selected based on the distance from the imaging device to the tree when the tree is photographed in real space. The same applies to the angle of view of the imaging device. The metadata about the imaging conditions described above is preferably used to select an appropriate "formula relating the number of leaf pixels to the total leaf area."

[0086] Specifically, the system allows a user to input several parameters that define the shooting conditions into a computer. For example, if a user is required to photograph a tree from directly to the side, the range of shooting conditions is limited, so it is sufficient to input only a few parameters, such as the distance from the image capture device to the tree. On the other hand, if the user is also allowed to photograph the tree canopy from diagonally below or above, it is preferable that the system be configured to have the user input information regarding the camera position and select an appropriate "formula relating the number of leaf pixels to the total leaf area" based on that information.

[0087] Fruit trees, such as grapes, are often planted in multiple rows. When trees in one row are photographed from the side, there is a high possibility that trees in other rows will be captured in the scene image. In this embodiment, depth information for each pixel in the scene image is used to distinguish between trees in the row of interest and trees in other rows that are captured in the scene image. When photographing trees from the side, there is a possibility that trees in other rows will be captured. However, by using the depth information, it is easy to identify the trees in the row of interest in the image.

[0088] Fig. 8 is a diagram showing an example of a scene image acquired by capturing an image of trees in real space using the image capture device 110. The scene image captured by the image capture device 110 is itself a color image, with each pixel having an R (red), G (green), and B (blue) value. However, Fig. 8 shows a monochrome (grayscale) image generated from this color image. In Fig. 8, some trees belonging to another row are reflected behind the row of trees of interest.

[0089] Referring again to Fig. 6, in this embodiment, in step S54, leaf pixels within a predetermined depth range are selected based on the depth and color of each pixel in the scene image acquired by the image capture device 110. Specifically, after selecting the leaf pixels, leaf pixels within the predetermined depth range are selected.

[0090] As described above, in this embodiment, color image data is acquired, and by using color information for each pixel, it is possible to classify whether each pixel belongs to a tree branch, fruit, flower, or leaf, or to the "background" such as the ground or sky. To detect leaf regions, a segmentation technique based on deep learning can be used. When using segmentation techniques, not only color information but also texture information of pixels can be used.

[0091] Fig. 9 is a diagram schematically showing a leaf region 12 selected from the image of Fig. 8. The leaf region 12 is surrounded by a solid line 12L. The leaf region 12 in the image of Fig. 9 includes not only the leaf pixels of the trees in the row to which the tree of interest belongs, but also the leaf pixels of the trees in the row behind it.

[0092] After selecting the leaf pixels in this way, by masking the pixels outside a predetermined depth range, it is possible to select and detect the leaves of the tree of interest from the scene image.

[0093] FIG. 10 is a diagram of a depth image showing depth information for each pixel in grayscale. Depth images are sometimes called depth maps. In FIG. 10, the shorter the distance between the object surface position corresponding to each pixel and the imaging device (the shallower the depth), the higher the brightness of that pixel. As is clear from the depth image in FIG. 10, the brightness of the leaf pixels in the tree in the back row is low, being black or a gray close to black. A depth histogram can be created from the depth image in FIG. 10. A depth histogram is a graph in which the horizontal axis represents bins (depth bins) divided by a predetermined width, and the vertical axis represents the frequency of pixels included in each bin. Hereinafter, the depth bins will be simply referred to as "depth," and the frequency of pixels included in each bin will be referred to as "pixel count."

[0094] 11 is a diagram illustrating an example of a depth histogram for a tree scene image, in which a first peak of the number of pixels is present in an area shallower than a certain depth TH, and another peak is present in an area deeper than the depth TH.

[0095] When trees belonging to multiple tree rows are captured in the same image, the trees belonging to different rows are located in different depth ranges, resulting in multiple peaks in the depth histogram. The pixels of the trees in the front row have low depths, forming the first peak in the depth histogram. The pixels of the trees in the rear row have successively higher depths, forming the second peak. From the depth histogram, a depth midway between adjacent peaks can be selected as a threshold. Pixels with depths equal to or greater than this threshold are pixels of the background or trees in the rear row. In the example of FIG. 11 , for example, a depth TH can be selected as the threshold. A pixel group R1 having a depth lower than the threshold depth TH can be determined to belong to a tree in the front row. Furthermore, a pixel group R2 having a depth higher than depth TH can be determined to belong to a tree in the rear row or to another object present between the tree rows.

[0096] Fig. 12 shows an example of a binary image obtained by selecting pixels showing depths in a predetermined range (for example, depths lower than depth TH) from among the pixels constituting leaf region 12 shown in Fig. 9. In the binary image of Fig. 12, the white pixels are composed of the selected pixels and are pixels of the leaves of trees belonging to the column of interest.

[0097] 9 may include areas other than leaves, such as branches, fruits, flowers, etc. In such cases, it is preferable to exclude pixels belonging to areas other than leaves based on color information of the image, etc.

[0098] Next, in step S56 shown in FIG. 6 , a feature related to the total leaf area is extracted from the selected leaf pixels. In the present embodiment, when the feature is the "number of leaf pixels in the image," the total number of selected pixels is counted. In the example of FIG. 12 , the total number of pixels included in the white region of the binary image is used as the feature. As mentioned above, the feature is not limited to this example. For example, for pixels included in the white region of the binary image of FIG. 12 , the feature may be extracted from various information, such as the ratio of the total number of pixels in the white region to the total number of pixels included in the image, or the shape of pixel group R1 in the depth histogram.

[0099] (Variation 1 of Tree Vigor Estimation Method) Next, a variation of the tree vigor estimation method will be described. In this variation, a near-infrared image of a tree scene is used as image data. By extracting the brightness histogram of leaves in the near-infrared image as a feature, it becomes possible to detect and evaluate overlapping leaves. The brightness histogram is a graph in which the horizontal axis represents classes into which the brightness (near-infrared irradiation amount) in each near-infrared image is divided by a predetermined width, and the vertical axis represents the frequency of pixels included in each class. Hereinafter, the brightness classes in the brightness histogram will be simply referred to as "brightness," and the frequency of pixels included in each class will be referred to as "pixel count."

[0100] 13 is a diagram showing the position and orientation of an imaging device that captures an image of a tree 10 bathed in sunlight from the sun 20. The diagram shows an X-Y-Z coordinate system (global coordinate system) fixed to the ground, an X1-Y1-Z1 coordinate system (camera coordinate system) fixed to the near-infrared camera, and an X1-Y1 plane 21 that is parallel to the image plane of the near-infrared camera. The Z1 axis indicates the line of sight of the near-infrared camera.

[0101] 13 has leaves 14 a, 14 b, 14 c, 14 d, 14 e, and 14 f. When the tree 10 is photographed with a near-infrared camera in the position and orientation shown in the figure, the other leaves 14 c, 14 d, 14 e, and 14 f may be only partially photographed due to the leaves 14 a and 14 b in the foreground.

[0102] 14 is a diagram showing a leaf region 12A in a captured image, which is composed of pixels of three overlapping leaves 14a, 14c, and 14e. This leaf region 12A does not contain information about the overlapping areas of the leaves 14a, 14c, and 14e. The number of pixels belonging to the leaf region 12A in FIG. 14 lacks information about the overlapping areas that is useful for estimating the sum of the areas of the leaves 14a, 14c, and 14e in three-dimensional space.

[0103] 15 is a diagram schematically illustrating a leaf region 12B composed of pixels of three overlapping leaves 14a, 14c, and 14e. In FIG. 14, the brightness of pixels belonging to the overlapping portion of two of the leaves 14a, 14c, and 14e is lower than the brightness of pixels belonging to leaves with no overlap. Also, the brightness of pixels belonging to the overlapping portion of three of the leaves 14a, 14c, and 14e is lower than the brightness of pixels belonging to the overlapping portion of two of the leaves.

[0104] If the pixels can be classified according to the degree of overlap of the leaves, it becomes possible to obtain information that correlates with the total area of ​​the leaves 14a, 14c, and 14e.

[0105] Grape leaves are known to transmit near-infrared light with wavelengths of approximately 800 nm or more, which is also contained in sunlight. In the case of grapes, the near-infrared transmittance of a single leaf is, for example, approximately 40% at a wavelength of 800 nm. Therefore, the transmittance in the area where two leaves overlap is approximately 16%, and the transmittance in the area where three leaves overlap is approximately 6%. Similar properties can be observed in leaves other than grape leaves.

[0106] Therefore, if you photograph a tree with a camera that can generate near-infrared images and create a brightness histogram of the acquired near-infrared image, the brightness (brightness) of pixels belonging to the leaves will show different values ​​depending on the degree of leaf overlap.

[0107] FIG. 16 is a diagram schematically illustrating an example of a brightness histogram of a near-infrared image acquired by photographing a tree with a near-infrared camera. As described above, the brightness histogram has peaks reflecting the degree of leaf overlap. Because branches transmit almost no near-infrared light, the brightness of pixels belonging to the branches is extremely low. Furthermore, if a background such as the sky is captured in part of the near-infrared image, some pixels may exhibit very high brightness. Furthermore, because the brightness of a near-infrared image depends on the intensity of sunlight reaching the tree from the sun, which is the light source, the average brightness and brightness distribution range in the near-infrared image may vary depending on weather conditions. Therefore, when creating a brightness histogram, it is preferable to normalize the brightness of pixels belonging to the background in each of the acquired multiple near-infrared images by setting the maximum brightness and the minimum brightness to the brightness of pixels belonging to the tree branches. This normalization allows for a brightness histogram that is easy to evaluate the degree of leaf overlap without being affected by the brightness value itself, which varies depending on the shooting conditions. An example brightness histogram after such normalization is shown in FIG. 17.

[0108] 15, leaf region 12B is divided into a group of pixels with the highest brightness, a group of pixels with intermediate brightness, and a group of pixels with the lowest brightness. If the number of pixels in the group of pixels with the highest brightness is N1, the number of pixels in the group of pixels with intermediate brightness is N2, and the number of pixels in the group of pixels with the lowest brightness is N3, the number of pixels belonging to leaf region 12B can be calculated by, for example, N1 + 2 N2 + 3 N3, rather than N1 + N2 + N3.

[0109] Note that since the proportion of near-infrared light that passes through the overlapping areas of the three leaves is low, there is a possibility that the pixels will not be accurately detected as leaf pixels. For this reason, the pixel group division based on the brightness histogram of the near-infrared image may be performed by dividing the pixel group into overlapping and non-overlapping areas.

[0110] The higher the intensity of near-infrared light incident on tree leaves, the higher the average brightness of the brightness histogram, making it easier to obtain accurate information about leaf overlap. For this reason, the position and orientation of the near-infrared camera are preferably such that the camera optical axis (Z1 axis) is directed toward the sun 20 or its vicinity, as shown in the X1-Y1-Z1 coordinate system (camera coordinate system) in Figure 18. Furthermore, it is preferable to acquire near-infrared images when the sun 20 is high in the sky and the weather is clear.

[0111] The amount of near-infrared radiation varies depending on the altitude of the sun 20 and weather conditions, but the effect of such variations can be reduced by using the normalized brightness histogram described above.

[0112] In this modified example, when deriving the function (relational equation) of total leaf area = f (feature quantity) through optical simulation, the refractive index, absorbance, spectral transmittance, and spectral reflectance of the leaf at near-infrared wavelengths are used as important model parameters.

[0113] As mentioned above, because grapevines are vertically long, it is easier to recognize the structure of the tree by observing the tree from the side, i.e., from the side. For this reason, it is preferable to use an image of the tree photographed from the side in addition to a near-infrared image acquired by a near-infrared camera positioned and oriented as shown in Fig. 18. The image of the tree 10 photographed from the side is preferably a visible light image containing color information, but may also be a near-infrared image.

[0114] Figure 19 is a schematic diagram illustrating how a tree 10 is photographed with a near-infrared camera and also photographed with a visible light camera at a camera viewpoint different from the camera viewpoint of the near-infrared image. Figure 19 schematically illustrates an X1-Y1-Z1 coordinate system and an X1-Y1 plane 21 fixed to the near-infrared camera, as well as an X2-Y2-Z2 coordinate system and an X2-Y2 plane 22 fixed to the visible light camera. The Z2 axis indicates the line of sight of the visible light camera.

[0115] By using a visible light image from a camera viewpoint different from that of the near-infrared image, there is an advantage that three-dimensional information about the leaves that reflects the three-dimensional structure of the crown of the tree 10 can be obtained.

[0116] (Variant example 2 of tree vigor estimation method) As shown in Figure 19, when image data including a visible light image and a near-infrared image is used, an equation that associates tree feature 1 in the visible light image and tree feature 2 in the near-infrared image with the total leaf area can be created using the optical simulation method described above, and the index value can be estimated using this equation.

[0117] This formula is generally expressed as follows, for example: Total leaf area=f(feature 1, feature 2) where f(·) represents a function determined by regression analysis.

[0118] Feature 1 is, for example, the total number of pixels belonging to leaves. Feature 2 is a feature extracted from a brightness histogram, and is, for example, a set (multivariate or vector) of counts of pixels belonging to multiple brightness groups with different brightness peaks. For example, if X1 represents the number of pixels in the group of pixels whose brightness is equal to or greater than a threshold, and X2 represents a prime number of pixels in the group of pixels whose brightness is less than the threshold, then the vector (X1, X2) can be used as feature 2. Such a feature may contain information that contributes to estimating the total leaf area, in that it contains information about leaf overlap rather than the number of leaf pixels in the visible light image.

[0119] In Variation 2, optical simulation is performed in a manner similar to that used for visible light images to generate multiple near-infrared images, and a brightness histogram is generated for each generated near-infrared image. To generate such near-infrared images using optical simulation, the three-dimensional leaf model used must have model parameters related to the transmittance and reflectance of near-infrared light. For example, a three-dimensional tree structure is created within the simulation space by setting an optical leaf model with transmittance and reflectance for near-infrared light at a wavelength of 800 nm. Using the simulation method described with reference to FIG. 2, near-infrared images captured from various camera viewpoints are generated for each of the multiple three-dimensional models using physically based rendering. Because the near-infrared images obtained using this simulation method are based on the optical properties of leaves, they exhibit a brightness distribution that reflects the degree of leaf overlap, which varies depending on the tree's leaf arrangement (canopy structure). As a result, it becomes possible to derive with greater accuracy an equation relating the above-mentioned Feature 2 to the total leaf area.

[0120] When a near-infrared camera used to photograph trees in a farm field in real space is equipped with a filter that transmits near-infrared light in a specific wavelength band, it is preferable to use a function f (feature 1, feature 2) created by performing an optical simulation tailored to that wavelength band.

[0121] (Tree Vigor Estimation Method Variation 3) Visible light images from multiple viewpoints may be used as image data. In this case, the camera position (e.g., height) and camera attitude (e.g., the tilt angle of the camera's line of sight relative to the horizontal plane) may be used as metadata for each image data. If different functions f(·) are prepared according to such metadata, an appropriate function f(·) can be selected by inputting information about the camera's position and attitude when the tree was photographed in real space into a computer, thereby enabling more accurate estimation of the tree vigor index value.

[0122] When performing optical simulation, if images are generated from individual 3D models by physically based rendering, a scatter plot with low variance may be obtained by restricting the camera position (e.g., height) and camera attitude (e.g., tilt angle of the camera's line of sight relative to the horizontal plane) to a specific range. In this case, when acquiring images of trees in a field in real space, it is preferable to align the camera position (e.g., height) and camera attitude (e.g., tilt angle of the camera's line of sight relative to the horizontal plane) to the above-mentioned specific range.

[0123] (Tree Vigor Estimation Method Variation 4) Near-infrared images from multiple viewpoints may be used as image data. In this case, the position (e.g., height) of the near-infrared camera and the attitude of the camera (e.g., the inclination angle of the camera's line of sight relative to the horizontal plane) may be used as metadata for each image data. If different functions f(·) are prepared according to such metadata, by inputting information about the position and attitude of the camera when photographing a tree in real space into a computer, an appropriate function can be selected from the multiple functions f(·) prepared in advance, enabling more accurate estimation of tree vigor index values.

[0124] As mentioned above, near-infrared images contain information about leaf overlap, so information about leaf overlap can be extracted as a feature based on the brightness histogram. By using near-infrared images from different camera viewpoints, it is possible to obtain information that reflects the three-dimensional structure of the tree crown.

[0125] (Tree vigor estimation system) Fig. 20 is a block diagram showing an example configuration of a tree vigor estimation system 1000 according to this embodiment. The tree vigor estimation system 1000 includes a computer 100, and an imaging device 110, a communication device 140, and a display device 150 connected to the computer 100. The computer 100 has a storage device 120 and a processing device 130. These components are connected to each other, for example, via a bus. The computer 100 can also be realized by a computing device installed in a mobile terminal device such as a smartphone owned by a user who receives the services of the tree vigor estimation method.

[0126] The tree vigor estimation system 1000 of this embodiment further includes another computer 200. The computer 200 is connected to a communication device 240, which is connected to the communication device 140 via wired or wireless network communication. The computer 200 includes a storage device 220 and a processing device 230. The storage device 220 and the processing device 230 are connected to each other, for example, via a bus. The computer 200 may be a personal computer that communicates with the computer 100 via the Internet. The computer 200 may also be realized by a cloud server. Such a cloud server may be configured to download a computer program (application) for executing the tree vigor estimation method to the storage device 120 in the mobile terminal device owned by the user.

[0127] In this embodiment, the imaging device 110 is used to capture images of trees in a real-world field. For example, an image (actually a color image) such as that shown in FIG. 8 is captured by the imaging device 110. The imaging device 110 may be a camera mounted on the mobile terminal device. A user may capture images of trees selected as subjects in the field using the imaging device 110, and process the image data using a computing device mounted on the user's mobile terminal device, thereby implementing the tree vigor estimation method.

[0128] The storage device 120 in this embodiment includes volatile and nonvolatile memory, and stores computer programs and data that cause the processing device 130 and the like to execute the processing steps shown in FIG. 1 . The storage device 120 also stores one or more image data representing a scene including trees in a field in real space. The volatile memory is, for example, semiconductor memory such as DRAM (Dynamic Random Access Memory) and SRAM (Static RAM). The nonvolatile memory is, for example, semiconductor memory such as ROM (Read Only Memory) and flash memory, an optical disk drive, or a hard disk drive.

[0129] The processing device 130 may be composed of a processor such as a GPU or a CPU. The processor is a hardware device realized by a semiconductor integrated circuit. Some or all of the processor may be a field programmable gate array (FPGA) equipped with a CPU, an application specific integrated circuit (ASIC), or an application specific standard product (ASSP). The processing device 130 operates according to a computer program recorded in the storage device 120. The processing device 130, either independently or in cooperation with the processing device 230 of the computer 200, estimates an index value indicating tree vigor based on image data acquired by the imaging device 110.

[0130] The communication devices 140 and 240 are interfaces for performing data communication between the computer 100 and the computer 200, or between the communication devices 140 and 240 and an external computing device. The communication devices 140 and 240 can perform wired communication using a controller area network (CAN) or the like, or wireless communication conforming to the Bluetooth (registered trademark) standard and / or the Wi-Fi (registered trademark) standard.

[0131] In this embodiment, the processing device 130 uses a relationship between the image of the scene and the total leaf area of ​​the trees when estimating the index value. In this embodiment, this relationship is determined by the computer 200 executing the above-mentioned simulation. The storage device (recording medium) 220 of the computer 200 stores a computer program that causes the processing device 230 to execute the following processing steps: setting a three-dimensional model of the tree in a simulation space; generating multiple images of the tree based on the three-dimensional model; determining the total leaf area of ​​the tree based on the three-dimensional model; and extracting feature values ​​from the multiple images and determining the relationship between the feature values ​​and the total leaf area.

[0132] As described above, the three-dimensional model is defined by optical model parameters of the leaf, including at least one of the leaf's refractive index, surface roughness, density, size, absorbance, spectral transmittance, and spectral reflectance.

[0133] The processing unit 130 can present the estimated index value to the user via the display unit 150 .

[0134] Note that all or part of the components and functions of computer 100 may be implemented by a cloud server. In this case, image data captured by a user is uploaded to the cloud server, and computer 100 in the cloud server may execute each process of the tree vigor estimation method.

[0135] The tree vigor estimation method, tree vigor estimation system, computer program, and optical simulation method of the present disclosure can be widely used in the agricultural field where trees are cultivated.

[0136] 10... Tree, 12... Leaf area, 14... Leaf, 20... Sun, 100... Computer, 110... Imaging device, 120... Storage device, 130... Processing device, 140... Communication device, 150... Display device, 200... Computer, 220... Storage device, 230... Processing device, 240... Communication device, 1000... Tree vigor estimation system

Claims

1. A tree vigor estimation method executed by one or more computers, comprising: acquiring one or more image data showing a scene including trees in a field in real space; and estimating an index value indicating the tree vigor based on the image data; and when estimating the index value, using a relationship between the image of the scene and the total leaf area of ​​the trees.

2. The tree vigor estimation method according to claim 1, further comprising storing the index value as digital data associated with the scene in a memory within the computer or transmitting the index value to a device connected to the computer.

3. The tree vigor estimation method according to claim 1, wherein the relationship is defined by an equation that associates the feature amount of the tree in the image of the scene with the total leaf area.

4. The tree vigor estimation method according to claim 3, wherein the formula is determined based on a three-dimensional model of the tree in the simulation space.

5. A tree vigor estimation method as described in claim 4, wherein the relationship is determined by a process including: determining the total leaf area of ​​the tree based on the three-dimensional model; generating multiple images of the tree based on the three-dimensional model; extracting features from the multiple images; and determining the relationship between the features and the total leaf area.

6. A tree vigor estimation method according to claim 5, wherein the feature amount includes pixel information of the leaf area in each of the plurality of images.

7. A tree vigor estimation method according to claim 6, wherein the feature value includes the number of leaf pixels in each of the plurality of images, or the ratio of the number of leaf pixels to the total number of pixels in each of the plurality of images.

8. The tree vigor estimation method according to claim 7, wherein the feature amount includes a brightness histogram of leaves in each of the plurality of images.

9. A tree vigor estimation method according to claim 6, wherein the feature amounts include at least one of leaf size, tree height, and number of branches in each of the plurality of images.

10. A tree vigor estimation method as described in claim 1, wherein the index values ​​include at least one of the total leaf area, leaf area index, canopy volume, canopy height, average leaf size, variance of leaf size, leaf count density, average branch thickness or length, variance of branch thickness or length, and number of branches for the tree.

11. The tree vigor estimation method of claim 1, comprising: processing the image data to estimate the depth of each pixel of the scene; selecting leaf pixels within a predetermined depth range based on the depth and color of each pixel; and extracting features related to total leaf area from the leaf pixels.

12. A tree vigor estimation method as described in claim 1, wherein acquiring the image data includes: photographing the tree using one or more imaging devices; or receiving digital data acquired by photographing the tree using one or more imaging devices.

13. A method for estimating tree vigor according to any one of claims 1 to 12, wherein the image data includes a side image of the tree.

14. The tree vigor estimation method according to claim 3, wherein the image data includes a near-infrared image of the scene, and the method includes extracting a brightness histogram of leaves in the near-infrared image as the feature.

15. A tree vigor estimation method according to claim 14, wherein the image data includes a visible light image taken from a camera viewpoint different from that of the near-infrared image.

16. An optical simulation method for determining a relationship between an image of a tree and the total leaf area of ​​the tree, the optical simulation method comprising: setting a three-dimensional model of the tree in a simulation space; generating a plurality of images of the tree based on the three-dimensional model; determining the total leaf area of ​​the tree based on the three-dimensional model; extracting features from the plurality of images; and determining the relationship between the features and the total leaf area.

17. The optical simulation method according to claim 16, wherein the feature amount is based on pixel information of a leaf region in each of the plurality of images.

18. The optical simulation method of claim 17, wherein the three-dimensional model is defined by leaf optical model parameters including at least one of leaf refractive index, surface roughness, density, size, absorbance, spectral transmittance, and spectral reflectance.

19. A tree vigor estimation system comprising: a storage device that stores one or more image data showing a scene including trees in a real-space farm field; and a processing device that estimates an index value indicating the vigor of the trees based on the image data, wherein the processing device uses a relationship between the image of the scene and the total leaf area of ​​the trees when estimating the index value.

20. A computer program that causes one or more computers to execute the following processing steps: acquiring one or more image data showing a scene including trees in a field in real space; and estimating an index value indicating the vigor of the trees based on the image data; and when estimating the index value, using a relationship between the image of the scene and the total leaf area of ​​the trees.

Images