Optical simulation method, optical simulation system, and computer program

Abstract

The light irradiance received by fruit in various field environments can be calculated. [Solution] The optical simulation method is executed by one or more computers and includes generating a three-dimensional model including multiple trees in a field, each tree bearing fruit, setting a light source within a simulation space in which the three-dimensional model is placed, moving the light source, calculating the irradiance of light received by the fruit in the three-dimensional model from the light source, and outputting an estimated value based on the irradiance that changes as the light source is moved.

Description

[Technical Field]

[0001] The present disclosure relates to an optical simulation method, an optical simulation system, and a computer program for calculating the amount of light irradiated onto a fruit. [Background technology]

[0002] Research and development is underway into smart agriculture, which utilizes ICT (Information and Communication Technology) and IoT (Internet of Things) as the next generation of agriculture. Smart agriculture aims to improve productivity, alleviate labor shortages, and reduce environmental impact.

[0003] Non-Patent Document 1 describes a method for estimating the amount of light received by the leaves of plants in a greenhouse by optical simulation. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Yuta Ohashi, "Estimation of light reception and growth of fruit vegetables using a 3D model in a greenhouse," Doctoral dissertation, Graduate School of Horticulture, Chiba University, January 2021 Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides an optical simulation method, an optical simulation system, and a computer program that are capable of calculating the amount of light irradiated onto fruits in various field environments. [Means for solving the problem]

[0006] The present invention has the configurations described in the following items.

[0007] [Item 1] 1. An optical simulation method executed by one or more computers, comprising: generating a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; setting a light source within a simulation space in which the three-dimensional model is placed; moving the light source; and calculating an irradiance of light received by the fruit in the three-dimensional model from the light source, and outputting an estimated value based on the irradiance that changes with the movement of the light source; An optical simulation method comprising:

[0008] [Item 2] Item 2. The optical simulation method according to item 1, wherein the three-dimensional model is generated based on sensing data of the field in real space.

[0009] [Item 3] 3. The optical simulation method according to item 1 or 2, wherein the three-dimensional model includes at least one of the slope, undulation, and elevation difference of the land in the field.

[0010] [Item 4] 4. The optical simulation method according to any one of items 1 to 3, wherein the estimated value includes at least one of a change in the irradiance, an integrated value of the irradiance over a predetermined period of time, and an index value indicating the state of the fruit.

[0011] [Item 5] 5. The optical simulation method according to item 4, wherein the transition of the irradiance and the integrated value of the irradiance over a predetermined period are calculated based on the wavelength of the selected light.

[0012] [Item 6] 6. The optical simulation method according to any one of items 1 to 5, wherein the index value includes a value indicating the state of sunburn or coloring of the fruit.

[0013] [Item 7] 7. The optical simulation method according to any one of items 1 to 6, wherein meteorological information including the temperature of the field is referenced when estimating the index value. [Item 8] 8. The optical simulation method according to any one of items 1 to 7, wherein the light source includes the sun, and the position of the sun is determined based on the latitude, longitude, and altitude of the field, as well as the date and time.

[0014] [Item 9] 9. The optical simulation method according to any one of items 1 to 8, wherein the plurality of trees are arranged in a row.

[0015] [Item 10] 10. The optical simulation method according to any one of items 1 to 9, wherein the three-dimensional model includes an optical model of a leaf having at least one of the following parameters: refractive index of the leaves of the tree, surface roughness of the leaves, density of the material inside the leaves, size of the material inside the leaves, and absorbance of the material inside the leaves.

[0016] [Item 11] a storage device that stores a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; a processing device that calculates the irradiance of light received by the fruit in the three-dimensional model from a light source and outputs an estimated value based on the irradiance that changes as the light source moves; Equipped with The processing device includes: setting a light source within a simulation space in which the three-dimensional model is placed; moving the light source; an optical simulation system configured to perform

[0017] [Item 12] the processing device includes a processor and a memory; The memory may include: setting a light source within a simulation space in which the three-dimensional model is placed; moving the light source; Item 12. The optical simulation system according to item 11, storing a computer program to be executed.

[0018] [Item 13] On one or more computers, generating a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; setting a light source within a simulation space in which the three-dimensional model is placed; moving the light source; calculating an irradiance of light received by the fruit in the three-dimensional model from the light source, and outputting an estimated value based on the irradiance that changes with the movement of the light source; A computer program that executes

[0019] [Item 14] A recording medium storing the computer program according to item 13.

[0020] [Item 15] means for storing a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; a means for calculating an irradiance of light received by the fruit in the three-dimensional model from a light source, and outputting an estimated value based on the irradiance that changes with the movement of the light source; and further comprising: means for setting a light source within a simulation space in which the three-dimensional model is placed; A means for moving the light source, An optical simulation system comprising: [Effects of the Invention]

[0021] According to an embodiment of the present disclosure, an optical simulation method, an optical simulation system, and a computer program are provided for calculating the irradiance of light received by fruit in various field environments. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a flowchart illustrating the processing steps of an optical simulation method according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 2 is a top view that schematically shows the row structure of trees in a farm field. [Figure 3] FIG. 3 is a diagram showing an example of an image generated when a tree in the generated simulation space is photographed from a certain camera viewpoint. [Figure 4] FIG. 4 is a schematic diagram showing the area of ​​the subject shown in FIG. [Figure 5] FIG. 5 is a diagram based on a micrograph of a cross section of a leaf. [Figure 6] FIG. 6 is a schematic cross-sectional view of a leaf for explaining an optical model of the leaf. [Figure 7] FIG. 7 is a diagram showing a schematic diagram of tree leaves and fruits, as well as a light source (the sun) and its trajectory. [Figure 8] FIG. 8 is a block diagram showing an example of the configuration of a simulation system according to this embodiment. [Figure 9] FIG. 9 is a flow chart illustrating the process steps in an embodiment of the optical simulation method. [Figure 10] FIG. 10 is a flow chart illustrating the process steps for generating a three-dimensional model of a leaf in an embodiment of the optical simulation method. DETAILED DESCRIPTION OF THE INVENTION

[0023] <Terminology> "Tree" refers to a woody plant (standing tree), including tall trees and shrubs.

[0024] "Fruit" refers to the organ in angiosperms that develops from the ovary and encases seeds, protecting and dispersing them. Fruits include simple, compound, and aggregate fruits. The edible part may be a true fruit with an enlarged ovary wall (such as apples and peaches), a false fruit with an enlarged receptacle (such as strawberries), or the seed itself (nuts). In this disclosure, "fruit" refers to a fruit that grows on a tree. This fruit, together with leaves and branches, is arranged in three-dimensional space and refers to an optical entity that is subject to sunlight irradiation.

[0025] Strictly speaking, the "irradiance of light received by a fruit from a light source" is expressed as the energy of light (electromagnetic waves with a wavelength of a specified value or range) passing through a unit area in a unit time. Irradiance is sometimes called "radiant flux density." The physical unit of "irradiance" is, for example, "Joules per second per square meter" (J / (s·m 2 )), or "watts per square metre" (W / m 2 ) The light is not limited to visible light, but may include ultraviolet or infrared light. In this disclosure, the integrated value (time-integrated value or cumulative value) of irradiance over a predetermined period may be simply referred to as "irradiance" or "integrated light irradiation amount," but these refer to values ​​per unit area.

[0026] The term "one or more computers" includes desktop PCs, laptops, tablets, mobile devices such as smartphones, wearable devices, embedded systems, and server computers interconnected via a network. These computers may operate independently or may be configured as multiple computer systems performing distributed processing in cloud computing or edge computing environments. Furthermore, computer systems equipped with specialized computing devices such as dedicated hardware accelerators, graphics processing units (GPUs), and field programmable gate arrays (FPGAs) are also included. In this disclosure, a "computer" typically includes a processor such as a GPU or central processing unit (CPU), and a memory device for storing programs and data. Any device or system equipped with these components and capable of executing the optical simulation method of this disclosure is included in the "computer" of this disclosure, regardless of its name. Electronic control units (ECUs) installed and used in work vehicles such as tractors are also included in the "computer."

[0027] <Embodiment> Hereinafter, an optical simulation method, an optical simulation system, and a computer program according to an embodiment of the present disclosure will be described with reference to the drawings. Parts that appear in multiple drawings with the same reference numerals indicate the same or equivalent parts.

[0028] The embodiments described below are examples for embodying the technical concept of the present invention, and do not limit the present invention. Furthermore, descriptions of the size, material, shape, relative arrangement, etc. of components are intended for illustration purposes only, and are not intended to limit the scope of the present invention. The size and positional relationship of components shown in each drawing may be exaggerated to facilitate understanding.

[0029] (Optical simulation method) The optical simulation method in this embodiment is executed by one or more computers.

[0030] First, the optical simulation method according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a flowchart showing an example of basic processing steps of the optical simulation method according to this embodiment.

[0031] As shown in FIG. 1, the optical simulation method of this embodiment includes step S10 of generating a three-dimensional model including a plurality of trees in a field, each of which has fruit; step S20 of setting a light source in a simulation space in which the three-dimensional model is placed; step S30 of moving the light source; and step S40 of calculating the irradiance of light (amount of light received per unit time) received by the fruit in the three-dimensional model from the light source and outputting an estimated value based on the irradiance that changes as the light source is moved.

[0032] Each step will be described in detail below.

[0033] [3D model generation] In step S10, a three-dimensional model is generated that includes a plurality of trees in a field, each bearing fruit.

[0034] Trees in the real world are located in a real field at a specific geographical location (latitude, longitude, altitude). Each individual tree also has its location coordinates within the field, and the location of the tree can be identified on a field map. A field may have multiple trees, for example, lined up in a row. Furthermore, the surface of the field is not necessarily flat, but may have slopes, undulations, and differences in elevation. The three-dimensional structure of the ground of the field can be represented by a terrain model.

[0035] The term "terrain model" refers to a model that represents the three-dimensional shape of the actual ground surface in real space as numerical data. The terrain model in this disclosure is a digital data set that accurately reproduces the slope, undulations, and elevation differences of the ground surface in and around a field where trees are planted. The terrain model in this embodiment will be described below.

[0036] · Terrain model data acquisition method: It is acquired using remote sensing technology mounted on unmanned aerial vehicles (UAVs), particularly LiDAR (Light Detection and Ranging) and SAR (Synthetic Aperture Radar). These sensors emit laser pulses or radio waves toward the ground surface and measure the reflection time and intensity to generate highly accurate 3D coordinate data.

[0037] Data format: Terrain model data consists of coordinates (X, Y) and altitude (Z) values ​​for each point arranged in a regular grid. Such data is expressed in a format known as a DEM (Digital Elevation Model) or DTM (Digital Terrain Model).

[0038] ·resolution: Depending on the performance of the sensor and the measurement conditions, the spatial resolution can be from a few centimeters to a few meters. The higher the resolution of the model, the more accurately it can represent subtle changes in the terrain.

[0039] Coordinate System: The coordinate system of geographic data can be expressed based on a geographic coordinate system (latitude and longitude) or a projected coordinate system (such as UTM coordinates), which allows for easier integration and analysis with other geospatial data.

[0040] Interpolation: Data between measurement points are estimated using an appropriate interpolation algorithm (e.g., kriging, inverse distance weighting, etc.), allowing for a continuous representation of the terrain.

[0041] In the optical simulations described herein, the terrain model is integrated with 3D models of individual trees and used as the basis for calculating fruit illumination. The slope, undulation, and elevation of the ground surface affect the placement and orientation of trees, as well as the shading caused by the surrounding terrain, and can therefore be used to accurately predict fruit illumination patterns.

[0042] The terrain model can also be updated as time-series data, which makes it possible to track changes in the terrain due to soil erosion and sedimentation, and can be used for long-term cultivation management and monitoring in fields.

[0043] The three-dimensional structure of multiple trees in a field is determined by the three-dimensional arrangement of multiple trees planted on the ground surface, which may have slopes, undulations, and differences in elevation. Therefore, when multiple trees are arranged in rows in a field, it is preferable to reproduce the tree row structure in the real world as accurately as possible in the simulation space.

[0044] Figure 2 is a top view that schematically illustrates the row structure of trees in a field. For reference, Figure 2 illustrates mutually orthogonal X-, Y-, and Z-axes. In the example of Figure 2, the Z-axis is parallel to the vertical direction, and the XY plane that includes the X- and Y-axes is parallel to the horizontal direction. Figure 2 omits illustration of the slope, undulations, and elevation differences of the ground surface.

[0045] 2, a plurality of trees 10 are arranged in rows, forming four tree rows 10-1, 10-2, 10-3, and 10-4. The row structure of trees 10 in a field can be defined by parameters such as the row direction (row direction), row length L, row spacing W, tree spacing P in each row, and the number of trees in each row.

[0046] In the present disclosure, the three-dimensional structure created by multiple trees in a field has two forms: a structure that an actual field has in real space (real space structure), and a structure defined by a three-dimensional model in a simulation space (simulation space structure). The latter simulation space structure is generated based on the former real space structure. In other words, the three-dimensional model of the field including trees is generated based on sensing data of the field in real space.

[0047] In order to construct a simulation space structure of a farm field in which many trees are planted, for example, the following processing is executed.

[0048] Sensing data collection: A user uses a video capture device such as a smartphone to capture video of tree rows 10-1, 10-2, 10-3, and 10-4 while moving through a field in real space. At the same time, the video capture device continuously acquires position coordinates during capture using a position estimation device such as a built-in GPS of the smartphone. Such video images can also be captured using agricultural machinery such as a tractor equipped with a video capture device, a mobile robot, or a drone.

[0049] Image analysis: The following elements can be recognized from the captured video: a) Three-dimensional row structure of trees b) Leaf shape and size on individual trees c) Branch shape d) Leaf attachment e) Fruit position

[0050] 3D model generation: By combining video data and location information and using 3D reconstruction techniques such as Structure from Motion (SfM), a 3D model of the tree 10 in the field is generated. This model includes the 3D structure of the leaves, branches, and fruit.

[0051] Figure 3 shows an example of an image obtained when a tree in the simulation space generated in this way is photographed from a certain camera viewpoint. Figure 4 is a schematic diagram showing the area of ​​the subject shown in Figure 3. In Figure 4, the area of ​​leaves 12 of tree 10 (the outline of the tree crown) is shown by a dashed line, and the boundary between the ground surface 20 of the field and the distant background 22 is shown by a solid line 20A and a dotted line 20B. Dotted line 20B indicates the part of tree 10 hidden by leaves 12, and this part is also shown for reference. Figure 4 also shows fruit 14 and trunk 16 of tree 10.

[0052] An image like the one shown in Figure 3 can be generated as an image from any camera viewpoint by applying the physically based rendering (PBR) technology in computer graphics to the above 3D model. By freely changing the "position" and "orientation (posture)" of the virtual camera, images from various viewpoints can be generated from the same 3D model.

[0053] Computer graphics technology simulates the behavior of light in real space by imitating phenomena such as reflection, refraction, and scattering of light based on the laws of physics. Specifically, optical simulation is performed based on the following: Light reflection characteristics (BRDF: Bidirectional Reflectance Distribution Function) Microstructure of objects Law of conservation of energy Light refraction and transmission Subsurface scattering (light scattering inside translucent materials)

[0054] In order to estimate the irradiance at the fruit using a 3D tree model, it is important to represent the optical properties of the tree leaves as accurately as possible. To achieve this, the following parameters can be set: Leaf surface reflectance: A value that indicates how much light the leaf surface reflects. Leaf refractive index: a value that determines the angle at which light refracts at the leaf surface Leaf surface roughness: A parameter that expresses the minute irregularities (microfacets) on the leaf surface · Leaf internal material: scattering particle density, size, and absorbance Leaf transmittance: A value that indicates the amount of light that passes through the leaf Leaf scattering coefficient: A value that indicates the degree to which light is scattered within the leaf

[0055] By using some of these parameters, it is possible to simulate phenomena such as reflection, transmission, and scattering when sunlight hits the leaves, improving the accuracy of estimating the irradiance of light received by the fruit.

[0056] Fig. 5 is a diagram based on a micrograph of the cross section of the leaf 12. Fig. 6 is a schematic cross-sectional view of a leaf to explain an optical model of the leaf 12. The optical model of the leaf 12 can be represented by a thin transparent plate (for example, "frosted glass") with fine irregularities on its surface. When the leaf 12 is given an internal structure, the internal structure can be represented by scattering particles.

[0057] As shown in Figure 6, when a light beam IN of a predetermined wavelength is incident on a leaf 12 at a predetermined angle of incidence θ, a portion of the light beam IN (light beam SR) is reflected by the surface 12A of the leaf 12, while the remaining light beams are scattered or absorbed within the leaf 12. The scattering or absorption can be represented by numerous particles 12P that mimic tissue structures such as chloroplasts and cell walls contained within the leaf 12. Note that chloroplasts have absorption peaks in the red and blue wavelength ranges due to photosynthesis. Some of the scattered light beams (DR1 and DR2) are emitted from the surface 12A of the leaf 12 to form part of the "reflected light." Of the light beams included in the light beam IN that entered the surface 12A of the leaf 12, light beams TR1 and TR2 that reach the back surface 12B of the leaf 12 without being reflected or absorbed are emitted to the outside as "transmitted light."

[0058] Reflection on the surface 12A of the leaf 12 can be classified into two types: "specular reflection" and "diffuse reflection." The direction of specular reflection depends on the angle of incidence θ and the surface roughness (a parameter that expresses fine irregularities) of the surface 12A of the leaf 12. The direction of diffuse reflection is caused by scattering within the leaf 12, and therefore exhibits an angular distribution that is more dependent on the internal structure of the leaf 12 and the wavelength of light than on the angle of incidence θ. To accurately simulate scattering and absorption within the leaf 12, it is preferable to assign appropriate values ​​to the density, size, and absorbance of the particles 12P that define the internal structure of the leaf 12. The density and size of the particles 12P affect the scattering of light, and scattering behaves differently depending on the wavelength. In addition, absorbance also has wavelength dependence.

[0059] The various optical parameters related to the leaves 12 may differ depending on the type of tree 10. Furthermore, even for the same tree 10, some or all of the optical parameters may be updated depending on the growth stage and season of the tree 10.

[0060] The intensity of light reflected from the surface of the leaf 12, as determined by the above optical parameters, exhibits an angular distribution that differs depending on the type of leaf 12, and this angular distribution may vary depending on the wavelength of the incident light. The values ​​of the above optical parameters can be estimated, for example, by using the angular distribution of light reflected from the surface for a given wavelength. Specifically, the optical parameters can be determined (estimated) by comparing the angular distribution value of the intensity of light reflected from the leaf surface obtained by simulation with the angular distribution of the intensity of light reflected from the leaf surface that is actually measured. This will be described later.

[0061] In the optical simulation method of this embodiment, the leaf 12 is represented as a three-dimensional model defined not only by structural parameters such as its shape, size, position, and posture, but also by optical parameters that indicate the optical properties of the leaf 12.

[0062] Light Source Settings Next, as shown in the flowchart of FIG. 1, in step S20, a light source is set within the simulation space in which the three-dimensional model is placed.

[0063] 7 is a diagram schematically showing leaves 12 and fruits 14 of a tree 10, as well as a light source 30 (the sun) and its trajectory 32. The position of the light source 30, which is the sun, is determined based on the latitude, longitude, and altitude of the field, as well as the date and time.

[0064] Within the three-dimensional structure of tree 10, leaves 12 and fruits 14 are exposed to light irradiation conditions that change depending on the position of light source 30. Fruit 14 receives sunlight that has passed through leaves 12 located above it and sunlight that has been reflected by leaves 12 located around it. Previously, there have been attempts to estimate the amount of light irradiance on tree leaves or canopies through simulation, but no attempts have been made to reproduce the three-dimensional structure of a real-world field in a simulation space and estimate the light irradiance on fruits by taking into account an optical model of the leaves.

[0065] The fruit 14 has specific position coordinates on the tree 10 and is under illumination conditions corresponding to the position of the sun at a specified date and time. The light emitted from the light source 30 has a spectrum whose intensity depends on the wavelength. In this embodiment, the light source 30 is the sun, so the spectrum is set to match the spectrum of sunlight. In this embodiment, the fruit 14 is represented as a three-dimensional model specified by parameters such as shape, size, position, and orientation. This model can also include optical characteristic parameters such as reflectance and absorptance for each wavelength.

[0066] The fruit 14 does not necessarily receive the light emitted from the light source 30 directly, but may also receive light that has passed through one or more leaves 12. Therefore, the irradiance at the fruit 14 is affected not only by the position of the light source 30, but also by the three-dimensional arrangement of the leaves 12 located around the fruit 14 and the optical properties of the leaves 12. Because the reflection and transmission of light by the leaves 12 depend on the wavelength of the light, the irradiance at the fruit 14 also changes depending on the wavelength of the light.

[0067] The optical simulation method of this embodiment can calculate the irradiance of light received by the fruit 14 from the light source 30 for each wavelength. Here, the irradiance is expressed as a function I(λ) of wavelength λ. The function I(λ) may be called spectral irradiance or spectral intensity. In order to estimate the irradiance of light received by the fruit 14 from the light source 30, it is preferable to accurately reconstruct within the simulation space the three-dimensional structure of the tree 10, including the numerous leaves 12, located around the fruit 14 of interest.

[0068] When calculating irradiance, a virtual plane that defines the irradiance can be assumed. Assume that the position of a fruit 14 of interest is expressed by coordinates (x1, y1, z1) in an XYZ coordinate system. Here, the Z axis of the XYZ coordinate system is parallel to the vertical direction and represents the distance (height) from the ground. The X-axis and Y-axis directions are parallel to the horizontal plane. In this embodiment, the irradiance at coordinates (x1, y1, z1) can be calculated according to the position of an arbitrarily set light source 30. This irradiance is calculated, for example, for light from the light source 30 incident on a unit area virtually placed at a height z1. To simplify the calculation, this virtual unit area is placed, for example, parallel to the XY plane.

[0069] The "horizontal plane" at height z1 can be represented by (x, y, z1), where x and y are variables that take any value within a range determined by the shape and size of the field. Note that this "horizontal plane" is a plane that is "horizontal (parallel to the XY plane)" in the XYZ coordinate system, and height z1 is not an absolute altitude but a displacement from the ground surface. According to this embodiment, a map (x, y, I(λ)) of irradiance I(λ) at (x, y, z1) can be obtained.

[0070] In a vineyard, although it depends on how the vines are trained, the position (height) of the fruit on each tree in a row often falls within a relatively narrow, predetermined range below the canopy. For example, in the example shown in Figure 4, many fruits 14 are located within a predetermined height range directly below the canopy. In such a case, by selecting the height z1 from the ground at which many fruits are formed, a map (x, y, I(λ)) of the irradiance I(λ) at the height of the fruit can be obtained.

[0071] The height z1 can be set to any value by, for example, a user. In this embodiment, such a map of irradiance I(λ) can be output (presented) to the user by a display device. The optical simulation method of this embodiment allows the user to obtain a map of irradiance I(λ) at any height above the ground, making it easy to visually recognize the influence of the tree canopy in the field on the sunlight conditions of fruits at various heights.

[0072] [Move Light Source] Next, as shown in the flowchart of FIG. 1, in step S30, the light source 30 is moved along a locus 32.

[0073] 7, the irradiance at the fruit 14 may vary depending on the location of the light source 30. For this reason, the location of the light source 30, which may be the sun, is determined based on the latitude, longitude, and altitude of the field and a specified date and time. Once the location of the light source 30 is determined, the irradiance of light reaching a specific tree 10 from the light source 30 can be calculated.

[0074] [Output of estimated value] Next, as shown in the flowchart of FIG. 1, in step S40, the irradiance of light received by the fruit in the three-dimensional model from the light source is calculated, and an estimated value based on the irradiance that changes as the light source moves is output.

[0075] Specifically, the irradiance that changes with the movement of the light source 30 is calculated, for example, from dawn (sunrise) to sunset (sunset). By integrating this irradiance over a predetermined period, the integrated light irradiation amount (e.g., "integrated daily received light amount") for that period can be calculated. The integrated light irradiation amount can be used as a basis for estimating a numerical value (index value) that indicates the quality or condition of the fruit that may be affected by light irradiation.

[0076] The estimated value based on the irradiance that changes with the movement of the light source may include at least one of the following: a change in irradiance, an accumulated amount of light irradiation over a predetermined period (e.g., "daily accumulated amount of received light"), and an index value indicating the condition of the fruit. Furthermore, the change in irradiance and the accumulated value of irradiance over a predetermined period may be calculated based on the wavelength of the selected light.

[0077] Similar to the map of irradiance I(λ) described above, a map of the integrated value of irradiance I(λ) over a predetermined period can also be generated.

[0078] The index value indicating the condition of the fruit may include a value indicating the state of sunburn or discoloration of the fruit. For example, by using a regression equation that inputs the number of days when the "daily accumulated amount of received light" obtained by the optical simulation method of this embodiment exceeds a predetermined threshold and outputs a numerical value indicating the degree of sunburn of the fruit, it is possible to output a numerical value indicating the degree of sunburn of the fruit as an index value. Furthermore, when estimating the index value, meteorological information including the temperature of the field may be referenced. For example, by using a trained model that inputs the accumulated temperature and accumulated amount of light irradiated over a predetermined period and outputs a class indicating the discoloration state of the fruit, it is possible to output a class indicating the discoloration state as an index value.

[0079] (Optical Simulation System) 8 is a block diagram showing an example configuration of an optical simulation system 1000 according to this embodiment. The optical simulation system 1000 includes a computer 100, and an imaging device 110, a communication device 140, and a display device 150 connected to the computer 100. The computer 100 includes a storage device 120 and a processing device 130. These components are connected to each other via a bus, for example. The computer 100 can also be realized by a computing device installed in a mobile terminal device such as a smartphone owned by a user who receives the services of the optical simulation method.

[0080] The optical simulation system 1000 of this embodiment further includes another computer 200. The computer 200 is connected to a communication device 240, which is connected to the communication device 140 via wired or wireless network communication. The computer 200 includes a storage device 220 and a processing device 230. The storage device 220 and the processing device 230 are connected to each other, for example, via a bus. The computer 200 may be a personal computer that communicates with the computer 100 via the Internet. The computer 200 may also be realized by a cloud server. Such a cloud server may be configured to download a computer program (application) for executing the optical simulation method to the storage device 120 in the mobile terminal device owned by the user.

[0081] In this embodiment, the imaging device 110 is used to capture video of trees in a field in real space. The video of the trees is captured by the imaging device 110 while moving through the field, for example, as shown in FIG. 2 . The imaging device 110 may be a camera mounted on the mobile terminal device described above. A user captures video of the trees 10 in the field using the imaging device 110, and processes the video data using a computing device mounted on the user's mobile terminal device, thereby recognizing the real-space structure of each tree 10 and generating a 3D model of each tree 10 in the simulated space structure.

[0082] The storage device 120 in this embodiment includes volatile and nonvolatile memory, and stores computer programs and data that cause the processing device 130 and the like to execute the processing steps shown in FIG. 1 . The storage device 120 also stores one or more image data representing a scene including trees in a field in real space. Volatile memory is, for example, semiconductor memory such as DRAM (Dynamic Random Access Memory) and SRAM (Static RAM). Nonvolatile memory is, for example, semiconductor memory such as ROM (Read Only Memory) and flash memory, optical disk drives, and hard disk drives.

[0083] The processing device 130 may be composed of a processor such as a GPU or a CPU. The processor is a hardware device realized by a semiconductor integrated circuit. Some or all of the processor may be an FPGA equipped with a CPU, an ASIC (Application Specific Integrated Circuit), or an ASSP (Application Specific Standard Product). The processing device 130 operates according to a computer program recorded in the storage device 120. The processing device 130, either independently or in cooperation with the processing device 230 of the computer 200, obtains the structure of a tree row based on video image data acquired by the imaging device 110 and generates a 3D model.

[0084] The communication devices 140 and 240 are interfaces for performing data communication between the computer 100 and the computer 200, or between the computer 100 and the computer 200 and an external computing device. The communication devices 140 and 240 can perform wired communication using a controller area network (CAN) or the like, or wireless communication conforming to the Bluetooth (registered trademark) standard and / or the Wi-Fi (registered trademark) standard.

[0085] The processing device 130 in this embodiment may be configured to read moving image data acquired by the imaging device 110 from the storage device 120 and transmit the data to the computer 200 via the communication devices 140, 240. In this case, the computer 200 executes the optical simulation method described above. That is, the storage device 220 of the computer 200 stores a computer program that causes the processing device 230 to execute the following processing steps. generating a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; Setting a light source in a simulation space in which the three-dimensional model is placed; moving the light source; and A step of calculating the irradiance of light received by the fruit in the three-dimensional model from the light source, and outputting an estimated value based on the irradiance that changes as the light source moves.

[0086] The estimated values ​​obtained by performing this optical simulation method are stored in the storage device 220. The processing device 230 may be configured to read the estimated values ​​from the storage device 220 and transmit them to the computer 100 via the communication devices 240, 140. The processing device 130 of the computer 100 can present the received estimated values ​​to a user on the display device 150. The user can provide information specifying a location or a tree in a field to the optical simulation system, and the optical simulation system 1000 can be configured to output an estimated value associated with the location or tree specified by the user.

[0087] (Example) An embodiment of an optical simulation method for grapes will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the processing steps of the optical simulation method in this embodiment. The optical simulation method in this embodiment is executed by one or more computers.

[0088] In this embodiment, first, in step S51, video images of the field (vineyard) are acquired while the camera viewpoint is moved, and a 3D reconstruction of the field, including the row structure of trees (vines), is performed from the video images. As mentioned above, by using 3D reconstruction techniques such as SfM, it is possible to acquire the information necessary to reproduce the field structure in real space in a simulation space from the video images.

[0089] In step S52, parameters defining the row structure, such as row direction, length, number of trees, etc., are determined based on the video image of the field. Some or all of these parameters may be estimated based on the video image, or may be provided to the computer by the user.

[0090] In step S53, parameters such as the three-dimensional structure of the branches of each tree, the number of leaves, the total leaf area, and the position and number of fruits are determined based on the video image of the farm field. The three-dimensional structure of the branches includes parameters such as the number of branches, branch length, and branch thickness of each tree.

[0091] In step S54, a 3D model of the tree is generated based on the above parameters. In order to calculate the amount of light irradiated on the fruit, a 3D model of the leaves is important in the 3D tree model. An example of a method for generating a 3D leaf model will be described later. Note that generating a detailed 3D model for each tree may require excessive computational costs. In such cases, one or several detailed 3D models (representative models) may be generated, and then an appropriate 3D model (representative model) may be assigned to each of all trees.

[0092] In step S55, a terrain model of the field is acquired. The terrain model may be generated by surveying the target field using the method described above, or by downloading a DEM or DTM of the area including the target field from a terrain database.

[0093] In step S56, multiple trees are placed on the terrain model based on the row structure of the field. In this way, the three-dimensional structure of the field can be reproduced in the simulation space. The three-dimensional structure of the field includes the slope, undulations, and elevation differences of the ground surface, as well as the three-dimensional structure of the tree rows (including branches, leaves, and fruit) arranged on the ground surface. Based on this three-dimensional structure, physically based rendering technology can generate an image (rendered image) from any camera viewpoint and display it on a display device. A user receiving services using the optical simulation method of this embodiment can view such a rendered image and identify the fruit for which the light irradiance or integrated light exposure is to be calculated.

[0094] The optical simulation method of this embodiment can generate a field map based on the above-described three-dimensional structure, e.g., a bird's-eye view of a row of trees from above. A "position" in such a field map is associated with geographical location information defined by longitude and latitude. By specifying a position or area on the field map using coordinates on the field map, a user can select several fruits ("grapes" in this embodiment) included in that position or area as targets for calculating optical irradiance.

[0095] In step S57, the latitude, longitude, altitude, and season (date and time) of the field are given, and in step S58, the position of the light source (sun) is set. Even if the latitude, longitude, and altitude of the field are the same, if the season (date and time) is different, the position of the light source (sun) as seen from the fruit of interest, i.e., the direction (solar altitude), may differ. In addition, the hours of sunlight also change depending on the season. When the latitude and longitude of the field are given, the altitude of the field is calculated based on the topographical data described above.

[0096] In step S59, the irradiance of light received by the fruit in the 3D model from the light source is calculated. In this embodiment, the position of the fruit of interest can be identified from the tree row having a 3D structure in the simulation space. The position of the fruit can be identified by various methods, such as using the farm field map described above.

[0097] The light received by each fruit from a light source includes not only direct sunlight but also light transmitted through one or more leaves surrounding the fruit of interest and light reflected by such leaves. Furthermore, the light received by the fruit from the light source has a spectral distribution. Therefore, the irradiance of light received by the fruit from the light source can be calculated for light of a specific wavelength or a specific wavelength range. The wavelengths of light that have a particularly strong effect on the tanning or coloring of fruits such as grapes may belong to a wavelength range that is relatively short compared to the central wavelength in the wavelength range of visible light, such as ultraviolet light or blue light. In such cases, it is preferable to calculate the irradiance of light received by the fruit in the ultraviolet and / or blue light wavelength range.

[0098] In step S60, the light source is moved, for example, from a position at dawn to a position at sunset. Then, in step S61, the integrated value of the irradiance of light (integrated light exposure) received by the fruit during the movement is calculated. Based on this integrated light exposure, it is possible to estimate the degree of sunburn and coloring of the fruit. As described above, the integrated value of the irradiance of light (integrated light exposure) received by the fruit while the light source is moving can be determined for a specific wavelength or a specific wavelength range. As a result, it is possible to estimate the integrated light exposure for light in a wavelength range that changes the state of the fruit.

[0099] Next, an example of a method for generating a 3D model of a leaf (particularly an optical model) in this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the processing steps for generating a 3D model of a leaf in this embodiment. Here, an example of a method for setting the refractive index and surface roughness of a leaf, which are among the optical parameters of a leaf required for generating a 3D model of a leaf, will be described.

[0100] In step S70, tentative parameter values ​​(initial values) for the unknown quantities of the grape leaf's refractive index and surface roughness are set. The surface roughness parameter value can be determined, for example, based on the standard deviation of the normal vectors of the multiple microfacets that define the leaf's surface. The surface roughness parameter value ranges, for example, from 0.0 to 1.0. The smaller this value, the smoother the leaf surface. The larger the value, the greater the surface roughness and the higher the light diffusion. The initial tentative parameter values ​​may be randomly selected, or parameter values ​​determined for other types of leaves may be used. Note that the optical model included in the 3D leaf model may include parameters such as the leaf's refractive index and surface roughness, as well as the density of the leaf's internal material, its size, and its absorbance. In this example, the density, size, and absorbance of the grape leaf's internal material are given fixed values ​​as known values. These parameters can be set to appropriate fixed values ​​based on existing information about the absorbance of chloroplasts in grape leaves and micrographs of grape leaves.

[0101] In step S71, a leaf model having formal parameter values ​​is generated in the simulation.

[0102] In step S72, light of a predetermined wavelength is irradiated onto the grape leaf model in a simulation. In step S73, the angular distribution of the intensity of light reflected by the leaf model (reflection intensity) is calculated. By varying the wavelength of the light irradiating the leaf in intervals of, for example, 5 nm, and determining the angular distribution of reflection intensity, it is possible to obtain the angular distribution of reflection intensity for each wavelength (spectral reflection intensity angular distribution based on tentative parameter values) for the entire or part of the spectral width of sunlight.

[0103] In step S74, measurements of the angular distribution of the reflection intensity of a real leaf in real space are obtained when the leaf is irradiated with light of a predetermined wavelength. Such measurements are obtained by varying the wavelength of light irradiating the real leaf in intervals of, for example, 5 nm and measuring the angular distribution of the reflection intensity. Measurements (reflection intensity characteristics) already obtained by a third party may also be used.

[0104] In step S75, a semitransparent member having the above reflection intensity characteristics is set within the simulation space, and light of a specific wavelength is irradiated onto the semitransparent member to calculate the angular distribution of the intensity of light reflected by the leaf (reflection intensity). By varying the wavelength of the light irradiating the leaf in 5-nm intervals, for example, and calculating the angular distribution of reflection intensity, the angular distribution of reflection intensity for each wavelength can be obtained for the entire or part of the spectral width of sunlight ("correct data" for the spectral reflection intensity angular distribution). This "correct data" is used in calculations to approximate the tentative parameters of the leaf's refractive index and surface roughness to their true values.

[0105] In step S76, the difference between the "spectral reflection intensity angular distribution based on the tentative parameter values" calculated in step S73 and the "spectral reflection intensity angular distribution in the correct answer data" is calculated. The difference here can be expressed, for example, by the following cost function:

[0106] The mean square error of sample values ​​at multiple different reflection angles is calculated between the reflection intensity angular distribution in the ground truth data at a certain wavelength λi and the reflection intensity angular distribution obtained by simulation at that wavelength λi. Here, the subscript "i" in λi is a positive integer that functions as an identification code for identifying the wavelength. The mean square error is a function (cost function) that takes formal parameter values ​​as arguments. The cost function is not limited to the mean square error.

[0107] The gradient method is applied to such a cost function. Specifically, the partial derivative of the cost function with respect to each parameter is calculated, and the gradient obtained thereby is used to determine the direction and magnitude of updating the parameters so as to reduce the value of the cost function (i.e., the difference).

[0108] When optimization is performed over a plurality of wavelengths λ1, λ2, . . . , λn, the sum of the mean square error at each wavelength may be used as the cost function, for example.

[0109] In step S77, the "difference" is compared with a predetermined threshold, and if the "difference" is equal to or greater than the threshold (YES), the process proceeds to step S78. In step S78, the unknown formal parameter values ​​of the leaf's refractive index and surface roughness are updated so as to reduce the difference. The above processing steps are then repeated based on the updated formal parameter values.

[0110] In step S77, if the difference is not equal to or greater than the threshold value (NO), the process proceeds to step S79.

[0111] In step S79, the refractive index and surface roughness of the leaf, which are the final values ​​of the temporary parameters, are determined as the refractive index and surface roughness of the leaf to be adopted in the optical model of the leaf.

[0112] In step S80, a three-dimensional model of the grape leaf is generated using the leaf's refractive index and surface roughness determined by the above method, as well as other known parameters.

[0113] In this embodiment, the 3D model of the grape leaf thus generated is used to generate a 3D model of the tree in step S54 of Fig. 9. Note that the 3D model of the grape leaf may be generated by a method other than the method described above.

[0114] In recent years, climate change, such as rising temperatures and increased solar radiation, has caused problems such as sunburn or abnormal coloring of fruits such as grapes. This embodiment makes it possible to improve such problems. In particular, since the amount of sunlight received by fruits such as grapes depends on the topography of the field and the three-dimensional structure of the trees, by restoring or reproducing these three-dimensional structures in a simulation space, it becomes possible to estimate with high accuracy the amount of light received by the fruits, and to modify the topography of the field and the three-dimensional structure of the trees in real space. [Industrial Applicability]

[0115] The optical simulation method and system, and computer program of the present disclosure can be widely used to calculate the amount of light irradiance received by fruits in various field environments. [Explanation of symbols]

[0116] 10...tree, 10-1, 10-2, 10-3, 10-4...tree row, 12...leaf, 12A...leaf surface, 12B...leaf underside, 12P...particles inside the leaf, 14...fruit, 16...stem, 20...field surface, 22...background, 30...sun (light source), 32...light source trajectory, 100...computer, 110...imaging device, 120...storage device, 130...processing device, 140...communication device, 150...display device, 200...computer, 220...storage device, 230...processing device, 240...communication device, 1000...optical simulation system

Claims

1. 1. An optical simulation method executed by one or more computers, comprising: generating a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; setting a light source within a simulation space in which the three-dimensional model is placed; moving the light source; and calculating an irradiance of light received by the fruit in the three-dimensional model from the light source, and outputting an estimated value based on the irradiance that changes with the movement of the light source; An optical simulation method comprising:

2. The optical simulation method according to claim 1 , wherein the three-dimensional model is generated based on sensing data of the field in real space.

3. The optical simulation method according to claim 1 , wherein the three-dimensional model includes at least one of a slope, an undulation, and an elevation difference of the land in the field.

4. The optical simulation method according to claim 3 , wherein the estimated value includes at least one of a transition of the irradiance, an integrated value of the irradiance over a predetermined period of time, and an index value indicating the state of the fruit.

5. The optical simulation method according to claim 4 , wherein the transition of the irradiance and the integrated value of the irradiance over a predetermined period are calculated based on a wavelength of the selected light.

6. The optical simulation method according to claim 4 , wherein the index value includes a value indicating a state of sunburn or coloring of the fruit.

7. The optical simulation method according to claim 6 , wherein meteorological information including the temperature of the field is referenced when estimating the index value.

8. The optical simulation method according to claim 1 , wherein the light source includes the sun, and the position of the sun is determined based on the latitude, longitude, and altitude of the field, and the date and time.

9. The optical simulation method according to claim 1 , wherein the plurality of trees are arranged in a row.

10. 2. The optical simulation method according to claim 1, wherein the three-dimensional model includes an optical model of a leaf having as parameters at least one of the refractive index of the leaf of the tree, the surface roughness of the leaf, the density of the material inside the leaf, the size of the material inside the leaf, and the absorbance of the material inside the leaf.

11. a storage device that stores a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; a processing device that calculates an irradiance of light received by the fruit in the three-dimensional model from a light source and outputs an estimated value based on the irradiance that changes as the light source moves; Equipped with The processing device includes: setting a light source within a simulation space in which the three-dimensional model is placed; moving the light source; an optical simulation system configured to perform

12. On one or more computers, generating a three-dimensional model including a plurality of trees in a field, each tree bearing fruit; setting a light source within a simulation space in which the three-dimensional model is placed; moving the light source; calculating an irradiance of light received by the fruit in the three-dimensional model from the light source, and outputting an estimated value based on the irradiance that changes with the movement of the light source; A computer program that executes

Images