Segmented image estimation device, segmented image estimation method, and program

Abstract

To provide a technology for evaluating characters of leaves in a leaf community using a method different from a 3D scanner.SOLUTION: A sectioned image estimation apparatus (1) includes one or more processors configured to execute a first estimation step which estimates boundaries between a leaf region and other regions in a leaf community, using a trained model (LM) constructed by machine learning, to generate a sectioned image formed by sectioning the leaf region and the other regions. An input of the trained model is leaf community image data which represents a leaf community image including a leaf community as a subject. An output of the trained model is sectioned image data which represents the sectioned image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for estimating boundaries between leaf regions and other regions in a leaf canopy using a trained model, and also to an apparatus and method for constructing such a trained model. [Background technology]

[0002] When managing the growth status of crops such as vegetables and fruits, genetic factor data indicating genetic potential, environmental factor data indicating environmental factors, and growth amount data indicating the growth amount of the crop are used. In managing the growth status, the genetic factor data and environmental factor data are explanatory variables, and the growth amount data is the objective variable.

[0003] With the advancement of sequencers and sensors, it has become possible to acquire large amounts of data on genetic factors and environmental factors without manual labor. On the other hand, for trait evaluation to obtain growth data (for example, the leaf area index in a leaf canopy), users still use a ruler or vernier calipers to evaluate leaf shape, etc. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ohashi et. al., Monitoring the Growth and Yield of Fruit Vegetables in Greenhouse Using a Three-Dimensional Scanner, Sensor, 20, 5270, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] A known technique for mechanically evaluating the characteristics of leaves in a leaf community is to use a 3D scanner equipped with a laser as a light source (see Non-Patent Document 1).

[0006] However, 3D scanners are expensive, making it unrealistic to spread their use to every farm. Also, because 3D scanners are large and heavy, they are difficult to move around in fields or greenhouses where aisles are not wide enough. In other words, 3D scanners are not suitable for use in crop cultivation sites.

[0007] In addition, 3D scanners cannot be used in environments with ambient light, so leaf canopy measurements cannot be performed during the day, and 3D scanner operators are forced to perform trait evaluations at night.

[0008] One aspect of the present invention has been developed in consideration of the above-mentioned problems, and its purpose is to provide a technology for evaluating leaf traits in a leaf community using a method other than a 3D scanner. [Means for solving the problem]

[0009] A segmented image estimation device according to one aspect of the present invention includes one or more processors that execute an estimation step of generating a segmented image in which a leaf region in a leaf canopy is segmented from another region by estimating a boundary between the leaf region and another region in the leaf canopy using a trained model constructed by machine learning. Also, a segmented image estimation method according to one aspect of the present invention includes an estimation step of generating a segmented image in which the leaf region is segmented from another region by estimating a boundary between the leaf region and another region in the leaf canopy using a trained model constructed by machine learning using one or more processors.

[0010] Furthermore, a machine learning device according to one aspect of the present invention includes one or more processors that execute a construction step of constructing a trained model that generates a segmented image in which a leaf region and another region in a leaf community are segmented by estimating a boundary between the leaf region and another region in a leaf community through supervised learning using a training dataset. Furthermore, a machine learning method according to one aspect of the present invention includes a construction step of constructing a trained model that generates a segmented image in which a leaf region and another region are segmented by estimating a boundary between the leaf region and another region in a leaf community through supervised learning using a training dataset.

[0011] In the segmented image estimation device, the segmented image estimation method, the machine learning device, and the machine learning method, (a) the input of the trained model is leaf canopy image data representing a leaf canopy image including a leaf canopy as a subject, and (b) the output of the trained model is segmented image data generated by estimating the boundary between a leaf area and other areas in a leaf canopy, and is segmented image data representing a segmented image in which the leaf area and the other areas are segmented. [Effects of the Invention]

[0012] According to one aspect of the present invention, a technology can be provided for evaluating leaf traits in a leaf community using a method different from a 3D scanner. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram showing the configuration of a segmented image estimation system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing the configuration of a segmented image estimation device included in the segmented image estimation system of FIG. 1. FIG. [Figure 3] 3(a) is a flowchart showing the flow of a segmented image estimation method implemented by the segmented image estimation device of Fig. 2. FIG. 3(b) is a flowchart showing the flow of post-processing steps included in the segmented image estimation method shown in (a). [Figure 4] FIG. 1 is a schematic plan view of a greenhouse in which tomatoes are grown. [Figure 5] (a) and (b) are examples of leaf canopy images that serve as input to the trained model in the segmented image estimation device included in the segmented image estimation system of Figure 1. [Figure 6] (a) and (b) are examples of segmented images that are the output of the trained model in the segmented image estimation device included in the segmented image estimation system of Figure 1. [Figure 7] 4 is an example of a segmented image after a cropping step included in the post-processing step shown in FIG. 3(b) is performed. [Figure 8] 3(b) shows the segmented image, weighting filter, and weighted image in the weighted image generation step included in the post-processing step. [Figure 9] 4 is a graph showing weight values of a weighting filter used in a weighted image generating step included in the post-processing step shown in FIG. 3(b). [Figure 10] 4 is a graph showing the correlation used in the second estimation step shown in FIG. 3(b). [Figure 11] FIG. 2 is a block diagram showing the configuration of a machine learning device included in the segmented image estimation system of FIG. 1. [Figure 12] 12 is a flowchart showing the flow of a machine learning method performed by the machine learning device of FIG. 11. [Figure 13] (a) is an example of a leaf canopy image input to the machine learning device included in the segmentation image estimation system of Figure 1. (b) is an example of a segmentation image generated by a user based on the leaf canopy image shown in (a). [Figure 14] (a) to (f) are examples of leaf canopy images used in the practical phase of an embodiment of the present invention. (a) to (c) are leaf canopy images of the first group, and (d) to (f) are leaf canopy images of the second group. [Figure 15]1A is a graph plotting the pixel ratio obtained by inputting the first group of leaf canopy image data to a trained model in an embodiment of the present invention, and the actual measured values of the LAI of the leaf canopy that is the subject of each leaf canopy image in the first group of leaf canopy image data. Also, in (a), a first correlation is shown by a straight line. Also, in (b), a graph plotting the pixel ratio obtained by inputting the second group of leaf canopy image data to a trained model in an embodiment of the present invention, and the actual measured values of the LAI of the leaf canopy that is the subject of each leaf canopy image in the second group of leaf canopy image data. Also, in (b), a second correlation is shown by a straight line. DETAILED DESCRIPTION OF THE INVENTION

[0014] [Segmentation Image Estimation System] A segmented image estimating system S according to one embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing the configuration of the segmented image estimating system S.

[0015] The classification image estimation system S is a system for generating a classification image in which a leaf region in a leaf canopy is classified into the other regions by estimating the boundary between the leaf region and the other regions. As shown in FIG. 1 , the classification image estimation system S includes a classification image estimation device 1 and a machine learning device 2. The classification image estimation system S also uses a camera C1 to generate leaf canopy image data to be input to the classification image estimation device 1. The classification image estimation system S also uses a camera C21 to generate leaf canopy image data to be input to the machine learning device 2. The classification image estimation system S also uses a computer C22 to generate classification image data to be input to the machine learning device 2.

[0016] Cameras C1 and C2 are digital cameras that generate image data representing captured images. In this embodiment, digital cameras included in smartphones are used as cameras C1 and C21. However, cameras C1 and C2 are not limited to digital cameras included in smartphones. Cameras C1 and C21 may be any digital cameras that can capture images including a leaf canopy as a subject. Furthermore, cameras C1 and C21 may be the same digital camera or different digital cameras.

[0017] <Segmented image estimation device> The segmented image estimation device 1 is a device for implementing a segmented image estimation method M1. The segmented image estimation method M1 is a method for generating a segmented image in which leaf regions and other regions in a leaf canopy are segmented by estimating boundaries between the leaf regions and other regions based on data provided by a camera C1 using a trained model LM constructed by machine learning. An example of the trained model LM is the U-Net Deep Neural Networks (U-Net DNN) architecture. U-Net DNN is described in Ghosh, S., Das, N., Das, I., & Maulik, U. (2019). Understanding deep learning techniques for image segmentation. ACM Computing Surveys, 52, 135. URL: https: / / doi.org / 10.1145 / 3329784. doi:10.1145 / 3329784. The configuration of the segmented image estimation device 1 and the flow of the segmented image estimation method M1 will be described in detail later with reference to other drawings.

[0018] The input of the trained model LM is leaf canopy image data representing a leaf canopy image that includes a leaf canopy as a subject.

[0019] In this embodiment, the leaf canopy photographed as a subject is a canopy of leaves from multiple crops planted along a row. Therefore, when viewed from the user holding the camera C1, this canopy has a pair of sides: a side closer to the user and a side farther from the user. When the user is on the right side of the row of crops being photographed as a subject, the right side of the canopy is the side closer to the user, and the left side of the canopy is the side farther from the user. Furthermore, when the user is on the right side of the row of crops being photographed as a subject, the above-described relationship is reversed.

[0020] In the classification image estimation system S, the leaf canopy image is obtained by photographing one of a pair of sides of the leaf canopy, which is the subject, from an oblique angle. Specific examples of the leaf canopy image will be described later with reference to different drawings.

[0021] In this embodiment, the classification image estimation system S will be described using tomatoes grown in a greenhouse as the crop. However, the crop is not limited to tomatoes, and any crop that forms a leaf canopy may be used. Examples of crops other than tomatoes include cucumbers and peppers. The classification image estimation system S is suitable for evaluating leaf traits of crops such as tomatoes, which have a high degree of leaf overlap in their leaf canopy.

[0022] The output of the trained model LM is segmented image data generated by estimating the boundary between the leaf area and other areas in a leaf canopy, and is segmented image data representing a segmented image in which the leaf area and the other areas are segmented.

[0023] The division image is an image generated based on the leaf canopy image, and is a binary image in which each pixel constituting the division image has a value of 0 or 1. In this embodiment, the value of pixels in the leaf canopy image that correspond to leaf areas in the leaf canopy is set to 1, and the value of pixels that correspond to other areas is set to 0. In this way, in the division image, each pixel takes on one of the two values, thereby separating the leaf areas from other areas in the leaf canopy.

[0024] Although it is difficult to explicitly specify a relational equation between the leaf canopy image that is the input of the trained model LM and the segmented image that is the output of the trained model LM, it is known that there is a certain relationship. Therefore, by using a trained model LM that takes a leaf canopy image as input, it is possible to accurately estimate the segmented image.

[0025] <Machine learning device> The machine learning device 2 is a device for implementing the machine learning method M2. The machine learning method M2 is a method for creating a training dataset DS using leaf canopy image data provided by a camera C21 and segmented image data that is generated by a computer C22 based on a boundary between a leaf region and other regions in a leaf canopy and that is input to the machine learning device 2, and for constructing a trained model LM by machine learning using the training data dataset DS. Details of the configuration of the machine learning device 2 and the flow of the machine learning method M2 will be described later with reference to the accompanying drawings.

[0026] <Each phase in the segmentation image estimation system> The segmentation image estimation system S goes through a preparation phase and a trial phase before reaching a practical phase. The preparation phase, trial phase, and practical phase will be briefly described below.

[0027] (1) Preparation Phase In the preparatory phase, the user uses the computer C22 to identify the boundary between the leaf region and other regions in the leaf canopy image. The computer C22 generates segmented image data representing a segmented image in which the leaf region and other regions are segmented according to the boundary identified by the user. Hereinafter, this operation will be referred to as generating a segmented image. Each time the user generates a segmented image, the machine learning device 2 creates training data (teacher data) from the leaf canopy image data provided by the camera C21 and the segmented image data generated by the user using the computer C22 and input to the machine learning device 2, and adds the created training data to the training dataset DS. The preparatory phase may end when a predetermined period (e.g., one week, one month, or one year) has elapsed since the start of the preparatory phase, or when the number of segmented images generated in the preparatory phase reaches a predetermined number (e.g., 100, 1,000, or 10,000). When the preparatory phase ends, the machine learning device 2 constructs a trained model LM through machine learning using the training dataset DS. The constructed trained model LM is transferred from the machine learning device 2 to the segmented image estimation device 1.

[0028] (2) Trial phase In the trial phase, the user generates a segmented image, and the segmented image estimation device 1 estimates the segmented image. Each time the user generates a segmented image, the segmented image estimation device 1 estimates the segmented image using the trained model LM based on the data provided by the camera C1 and the data input by the user. The trial phase may end when a predetermined period of time (e.g., one week, one month, or one year) has elapsed since the start of the trial phase, or when the number of segmented images generated in the trial phase reaches a predetermined number (e.g., 100, 1,000, or 10,000). Upon completion of the trial phase, the user compares the segmented image calculated according to the definition with the segmented image estimated by the segmented image estimation device 1 to evaluate the estimation accuracy of the segmented image estimation device 1. If the estimation accuracy is insufficient, the process returns to the preparatory phase. If the estimation accuracy is sufficient, the process proceeds to the practical phase. Note that the estimation accuracy may be confirmed using a portion of the training dataset DS after the preparatory phase is completed. In this case, the trial phase can be omitted.

[0029] (3) Practical phase In the practical phase, the segmented image estimation device 1 estimates a segmented image. The learned model LM used by the segmented image estimation device 1 in the practical phase has been confirmed to have sufficient estimation accuracy in the trial phase. In the practical phase, the generation of a segmented image by the user can be omitted. This relieves the user from the trouble of generating a segmented image and enables the segmented image to be estimated with high accuracy.

[0030] [Configuration of segmented image estimation device] The configuration of the segmented image estimation device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the segmented image estimation device 1.

[0031] The segmented image estimation device 1 is realized using a general-purpose computer, and includes a processor 11, a primary memory 12, a secondary memory 13, an input / output interface 14, a communication interface 15, and a bus 16. The processor 11, the primary memory 12, the secondary memory 13, the input / output interface 14, and the communication interface 15 are connected to each other via the bus 16.

[0032] The secondary memory 13 stores a segmented image estimation program P1, a trained model LM, and a correlation function representing the correlation between the pixel ratio in the segmented image and the Leaf Area Index (LAI). The processor 11 loads the segmented image estimation program P1 and the trained model LM stored in the secondary memory 13 onto the primary memory 12. The processor 11 then executes each step included in the segmented image estimation method M1 in accordance with instructions included in the segmented image estimation program P1 loaded onto the primary memory 12. The trained model LM loaded onto the primary memory 12 is used when the processor 11 executes a first estimation step M12 (described below) of the segmented image estimation method M1. Note that "storing the segmented image estimation program P1 in the secondary memory 13" refers to the storage of source code or an executable file obtained by compiling the source code in the secondary memory 13. Furthermore, "storing the trained model LM in the secondary memory 13" refers to the storage of parameters defining the trained model LM in the secondary memory 13.

[0033] The correlation function representing the correlation between the pixel ratio and LAI will be described later with reference to a different drawing.

[0034] Examples of devices that can be used as the processor 11 include a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The processor 11 is also sometimes called an "arithmetic unit."

[0035] Furthermore, an example of a device that can be used as the primary memory 12 is a random access memory (RAM). The primary memory 12 is sometimes called a "main storage device." Furthermore, an example of a device that can be used as the secondary memory 13 is a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an optical disk drive (ODD), or a combination thereof. The secondary memory 13 is sometimes called an "auxiliary storage device." The secondary memory 13 may be built into the segmented image estimation device 1, or may be built into another computer (e.g., a computer constituting a cloud server) connected to the segmented image estimation device 1 via the input / output interface 14 or the communication interface 15. While the present embodiment implements storage in the segmented image estimation device 1 using two memories (the primary memory 12 and the secondary memory 13), this is not limiting. That is, the storage in the segmented image estimation device 1 may be implemented using a single memory. In this case, for example, one storage area of the memory may be used as the primary memory 12, and the other storage area of the memory may be used as the secondary memory 13.

[0036] An input device and / or an output device are connected to the input / output interface 14. Examples of the input / output interface 14 include interfaces such as USB (Universal Serial Bus), ATA (Advanced Technology Attachment), SCSI (Small Computer System Interface), PCI (Peripheral Component Interconnect), and HDMI (High Definition Multimedia Interface, registered trademark). An example of an input device connected to the input / output interface 14 is a camera C1. In this case, the camera C1 is connected to the segmentation image estimation apparatus 1 by wire using an interface cable conforming to any of the above-mentioned standards (for example, USB or HDMI).

[0037] Data acquired from camera C1 in segmented image estimation method M1 is stored in primary memory 12. Examples of input devices connected to input / output interface 14 include a keyboard, a mouse, a touchpad, a microphone, or a combination thereof. Data acquired from a user in segmented image estimation method M1 is input to segmented image estimation apparatus 1 via these input devices and stored in primary memory 12. Examples of output devices connected to input / output interface 14 include a display, a projector, a printer, a speaker, headphones, or a combination thereof. Information provided to a user in segmented image estimation method M1 is output from segmented image estimation apparatus 1 via these output devices. Like a laptop computer, segmented image estimation apparatus 1 may incorporate a keyboard that functions as an input device and a display that functions as an output device. Alternatively, segmented image estimation apparatus 1 may incorporate a touch panel that functions as both an input device and an output device, like a tablet computer.

[0038] Other computers are connected to the communication interface 15 via a network, either wired or wirelessly. Examples of the communication interface 15 include interfaces such as Ethernet (registered trademark) and Wi-Fi (registered trademark). Examples of available networks include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a global area network (GAN), or an internetwork including these networks. The internetwork may be an intranet, an extranet, or the internet. An example of an input device connected to the communication interface 15 is a camera C1. In this case, the camera C1 is wirelessly connected to the segmentation image estimation device 1 using any of the networks described above.

[0039] In the segmented image estimation method M1, data (for example, the learned model LM) that the segmented image estimation device 1 acquires from another computer (for example, the machine learning device 2) is transmitted and received via these networks.

[0040] Although the present embodiment employs a configuration in which the segmented image estimation method M1 is executed using a single processor (processor 11), the present invention is not limited to this. That is, a configuration in which the segmented image estimation method M1 is executed using multiple processors may be employed. For example, the first estimation step M12 (described later) may be executed by a first processor, and the post-processing step M13 (described later) may be executed by a second processor. In this case, the multiple processors that cooperate to execute the segmented image estimation method M1 may be provided in a single computer and configured to be able to communicate with each other via a bus, or may be provided in multiple computers in a distributed manner and configured to be able to communicate with each other via a network. As an example, a processor built in a computer that constitutes a cloud server and a processor built in a computer owned by a user of the cloud server may cooperate to execute the segmented image estimation method M1.

[0041] Furthermore, in this embodiment, a configuration is adopted in which the trained model LM is stored in memory (secondary memory 13) built into the same computer as the processor (processor 11) that executes the segmented image estimation method M1, but the present invention is not limited to this. That is, a configuration may be adopted in which the trained model LM is stored in memory built into a computer different from the processor that executes the segmented image estimation method M1. In this case, the computer built into the memory that stores the trained model LM is configured to be able to communicate with a computer built into the processor that executes the segmented image estimation method M1 via a network. As an example, a configuration is conceivable in which the trained model LM is stored in memory built into a computer that constitutes a cloud server, and a processor built into a computer owned by a user of the cloud server executes the segmented image estimation method M1.

[0042] Furthermore, although the present embodiment employs a configuration in which the trained model LM is stored in a single memory (secondary memory 13), the present invention is not limited to this. That is, a configuration in which the trained model LM is distributed and stored in multiple memories may be employed. In this case, the multiple memories that store the trained model LM may be provided in a single computer (which may or may not be a computer incorporating a processor that executes the segmentation image estimation method M1), or may be distributed and provided in multiple computers (which may or may not include a computer incorporating a processor that executes the segmentation image estimation method M1). As an example, a configuration in which the trained model LM is distributed and stored in memories incorporated in each of multiple computers that constitute a cloud server may be considered.

[0043] [Flow of segmentation image estimation method] The flow of the segmented image estimation method M1 will be described with reference to FIGS. 3 to 10. FIG. 3(a) is a flowchart showing the flow of the segmented image estimation method M1. FIG. 3(b) is a flowchart showing the flow of the post-processing step M13 included in the segmented image estimation method M1. FIG. 4 is a schematic plan view of a greenhouse GH in which tomatoes are grown. FIGS. 5(a) and 5(b) are examples of leaf canopy images that are input to the trained model LM in the segmented image estimation device 1. FIGS. 6(a) and 6(b) are examples of segmented images that are output from the trained model LM. FIG. 7 is an example of a segmented image after performing a trimming step M131 included in the post-processing step M13. FIG. 8 shows a segmented image, a weighting filter, and a weighted image in a weighted image generation step M132 included in the post-processing step M13. FIG. 9 is a graph showing the weight values of the weighting filter used in the weighted image generation step M132 included in the post-processing step M13. FIG. 10 is a graph showing the correlation used in the second estimation step M135.

[0044] The segmented image estimation method M1 includes an acquisition step M11, a first estimation step M12, and a post-processing step M13 (see FIG. 3(a)).

[0045] <Images of the greenhouse and leaf canopy> Before explaining the segmented image estimation method M1, we will explain the leaf canopy image that is the input to the trained model LM and the greenhouse in which the leaf canopies LC1 to LC3 that are the subject of the image are grown.

[0046] Two rows of tomato seedlings are planted in the greenhouse GH. In Figure 4, 16 tomato seedlings are planted in 8 rows and 2 columns. In Figure 4, the main stem MS of each seedling is shown as a circle, and the leaf canopies LC1 to LC3 made up of the 16 seedlings are shown as rectangles.

[0047] Thus, each of the leaf communities LC1 to LC3 is formed along a row. Therefore, the leaf community LC1 has a pair of side surfaces along the row direction, a left side surface L1 and a right side surface R1. Similarly, the leaf community LC2 has a left side surface L2 and a right side surface R2, and the leaf community LC3 has a left side surface L3 and a right side surface R3.

[0048] In Fig. 4, this column direction is defined as the y-axis direction, and the direction perpendicular to the y-axis direction is defined as the x-axis direction. The direction from left to right of the user U is defined as the positive x-axis direction, and the direction from the front to the back when viewing the leaf canopies LC1 to LC3 from the user is defined as the positive y-axis direction. Each of Figs. 5 to 8 also schematically shows the x-axis direction shown in Fig. 4 when projected onto the leaf canopy image or classification image.

[0049] A user U uses a camera C1 to capture an image of the right side R1, which is one of a pair of sides of a leaf canopy LC1 serving as a subject, from an oblique angle. When the user captures an image of the leaf canopy LC1 in this manner, the camera C1 generates leaf canopy image data representing a leaf canopy image including the leaf canopy LC1 as a subject. Examples of leaf canopy images are shown in (a) and (b) of FIG. 5. (a) of FIG. 5 is a leaf canopy image in which only the leaf canopy LC1, which is a single row of leaf canopy, is the subject, and (b) of FIG. 5 is a leaf canopy image in which the leaf canopy LC1 and LC2, which are two adjacent rows of leaf canopies, are the subjects. In this way, the leaf canopy image may include two adjacent rows of leaf canopies as subjects.

[0050] Next, the acquisition step M11, the first estimation step M12, and the post-processing step M13 will be described.

[0051] <Acquisition steps> In the acquisition step M11, the processor 11 acquires leaf canopy image data input from the camera C1. The camera C1 may be connected to the classification image estimation device 1 using an interface cable as described above, or may be connected to the classification image estimation device 1 via a network. By connecting the camera C1 to the classification image estimation device 1 via a network, the classification image estimation method M1 can be performed on crops grown in a farm in a remote location, and the LAI of the crop can be estimated. The leaf canopy image data may be supplied from the camera C1 to the classification image estimation device 1 via a computer and a network. In the acquisition step M11, the processor 11 writes the acquired leaf canopy image data to the primary memory 12.

[0052] <First estimation step> The first estimation step M12 is a step in which the processor 11 uses the trained model LM to estimate the boundary between the leaf area and other areas in the leaf canopy, thereby generating a segmented image that segments the leaf area from the other areas. In the first estimation step M12, the processor 11 reads the leaf canopy image data from the primary memory 12 and inputs the read leaf canopy image data data to the trained model LM. Then, the processor 11 writes the segmented image data output from the trained model LM to the primary memory 12.

[0053] (a) and (b) in Figure 6 are segmented images estimated by the trained model LM using (a) and (b) in Figure 5 as input, respectively. In (a) and (b) in Figure 6, pixels in the leaf canopy image that correspond to leaf regions in the leaf canopy (pixels with a value of 1) are shown in white, and pixels that correspond to other regions (pixels with a value of 0) are shown in black.

[0054] The segmentation image estimation method M1 may further include an output step of outputting the segmentation image estimated in the first estimation step M12. In this output step, the processor 11 reads the segmentation image data from the primary memory 12 and presents the segmentation image represented by the read segmentation image data to a display, thereby allowing the user to visually confirm the segmentation image used to estimate the LAI.

[0055] <Post-processing steps> The post-processing step M13 includes a trimming step M131, a weighted image generation step M132, an output step M133, a ratio calculation step M134, and a second estimation step M135 (see FIG. 3(b)).

[0056] (Trimming step) The trimming step M131 is a step that the processor 11 executes between the first estimation step M12 and the weighted image generation step M132. When two adjacent rows of leaf canopies LC1 and LC2 are included in the segmentation image (see FIG. 6(b)), the trimming step M131 compares the widths of the leaf canopies (the length along the x-axis direction shown in FIG. 6(b)) in the segmentation image. Then, the trimming step M131 trims an area that includes the leaf canopy LC1 with the larger width and does not include the leaf canopy LC2 with the smaller width as a region of interest, and the image of the region of interest is used as a new segmentation image. For example, in the segmented image shown in FIG. 6(b), if the processor 11 trims the area to the left of the line A-A' as the region of interest, the new segmented image will be as shown in FIG.

[0057] (Weighted image generation step) The weighted image generating step M132 is a step that the processor 11 executes between the first estimation step M12 and the second estimation step M135. The weighted image generating step M132 is a step of applying a weighting filter to the segmented image (see, for example, FIG. 6(a) and FIG. 7) to generate a weighted image (see FIG. 8).

[0058] In the weighted image generating step M132, the weighting filter used by the processor 11 is configured so that the weighting value decreases or increases as the distance from the region corresponding to the front side of one side of the leaf canopy (in this embodiment, leaf canopy LC1) of the subject leaf canopy moves toward the region corresponding to the back side. In this embodiment, the former is adopted.

[0059] In the weighting filter shown in FIG. 8, pixels with a weight value of 1 are represented in white, pixels with a weight value of 0 are represented in black, and pixels with a weight value greater than 0 and less than 1 are represented in gray. The gray gradations correspond to the weight values. In the weighting filter shown in FIG. 8, the same weight value is set for pixels arranged in the column direction. On the other hand, this weighting filter is configured so that for pixels arranged in the row direction, the weight value decreases from 1 to 0 as one moves from the left end to the right end of the weighting filter. Here, the left end of the weighting filter corresponds to the front side of the leaf canopy LC1, and the right end of the weighting filter corresponds to the back side of the leaf canopy LC1.

[0060] FIG. 9 shows the position dependency of the weight values in the weight filter in the row direction. In FIG. 9, the left end of the weight filter is set as the origin, and the right end of the weight filter is normalized to 1. The position dependency of the weight values in the row direction may be a monotonous decrease or increase from point (0,1) to point (1,0), and the shape may be a straight line or a Bezier curve as shown in FIG. 9. By adjusting the shape of the position dependency of the weight values in the weight filter, the accuracy of the estimated LAI estimated in the second estimation step M135 described later can be improved.

[0061] For example, the shape of a Bezier curve is defined by a total of four control points: the control points (0,1) and (1,0) shown as black circles in Figure 9, and the two control points shown as white circles. Therefore, the accuracy of the estimated LAI can be improved by adjusting the positions of the two control points shown as white circles.

[0062] In this way, the weighting filter is configured so that the position-dependent shape of the weight value becomes smaller or larger as it moves from one of the left and right ends of the leaf canopy image, that is, the first end (in this embodiment, the left end of the leaf canopy image), to the other end, that is, the second end (in this embodiment, the right end of the leaf canopy image). Then, when the leaf canopy image is an image of the second side of the pair of side surfaces of the leaf canopy LC1 (in this embodiment, the right side R1), the processor 11 applies the weighting filter to the leaf canopy image as is. On the other hand, when the leaf canopy image is an image of the first side of the pair of side surfaces of the leaf canopy LC1 (in this embodiment, the left side L1), the processor 11 inverts the leaf canopy image left and right before applying the weighting filter.

[0063] In the weighting filter, it can be appropriately selected whether the first end having a large weight value corresponds to the left end or the right end of the leaf canopy image. When a Bezier curve obtained by left-right inverting the Bezier curve shown in Figure 9 is used as the shape of the position dependency of the weight value, it is only necessary to invert the handling of the two types of leaf canopy images described above (one photographed from the right side R1 and one photographed from the left side L1).

[0064] In this embodiment, the left and right of the leaf canopy image are reversed to match the first end portion having a large weight value with the front side of the leaf canopy that is the subject. However, instead of reversing the left and right of the leaf canopy image, the left and right of the shape of the Bezier curve can also be reversed.

[0065] (output step) In the output step M133, the processor 11 outputs the number of pixels included in the leaf region (ie, pixels with a value of 1) in the segmented image weighted in the weighted image generating step M132.

[0066] (Ratio calculation step) The ratio calculation step M134 is a step for calculating the ratio of the number of pixels to the total number of pixels of the segmented image weighted in the weighted image generation step M132 (hereinafter referred to as pixel ratio).

[0067] (Second estimation step) In the second estimation step M135, the processor 11 references the correlation between the pixel ratio and the LAI in the leaf canopy and estimates the LAI corresponding to the ratio. Hereinafter, the LAI estimated in the second estimation step M135 will be referred to as the estimated LAI.

[0068] This correlation can be determined in advance as follows. First, a user uses a computer to refer to the leaf canopy image represented by the leaf canopy image data. The user estimates the boundary between the leaf area and other areas in the leaf canopy image and inputs it into the computer. The computer generates segmented image data representing a segmented image in which the leaf area and other areas in the leaf canopy are segmented based on the boundary estimated by the user. Next, the segmented image data is subjected to the same post-processing as in post-processing step M13 to obtain the pixel ratio. The user also measures the LAI of the leaf canopy included as a subject in the leaf canopy image. Hereinafter, this measured value of LAI will be referred to as the measured LAI. The measured LAI can be calculated using the total leaf area and floor area. The total leaf area is obtained by measuring the length and width of all leaves included in the target leaf canopy. In this way, the correlation between the pixel ratio and the LAI in the leaf canopy is estimated using the pixel ratio calculated using the segmented image data based on the boundary estimated by the user and the measured LAI. In this embodiment, this correlation is estimated using a linear regression model. However, the model used to estimate this correlation is not limited to a linear regression model.

[0069] This correlation is stored in the secondary memory 13. The processor 11 expands this correlation in the primary memory 12 and refers to it in the second estimation step M135. In this embodiment, this correlation is expressed as a function (see FIG. 10). However, this correlation may also be expressed as a table.

[0070] For example, if the pixel ratio calculated in the ratio calculation step M134 is 0.4, the processor 11 estimates the estimated LAI to be 1.85 based on the correlation in FIG.

[0071] [Configuration of machine learning device] The configuration of the machine learning device 2 will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the configuration of the machine learning device 2.

[0072] The machine learning device 2 is realized using a general-purpose computer, and includes a processor 21, a primary memory 22, a secondary memory 23, an input / output interface 24, a communication interface 25, and a bus 26. The processor 21, the primary memory 22, the secondary memory 23, the input / output interface 24, and the communication interface 25 are connected to each other via the bus 26.

[0073] The secondary memory 23 stores a machine learning program P2 and a training dataset DS. The training dataset DS is a collection of training data DS1, DS2, .... The processor 21 expands the machine learning program P2 stored in the secondary memory 23 onto the primary memory 22. The processor 21 then executes each step included in the machine learning method M2 in accordance with the instructions included in the machine learning program P2 expanded onto the primary memory 22. The training dataset DS stored in the secondary memory 23 is created in a training dataset creation step M21 (described below) of the machine learning method M2 and is used in a trained model construction step M22 (described below) of the machine learning method M2. The trained model LM constructed in the trained model construction step M22 of the machine learning method M2 is also stored in the secondary memory 23. Note that the machine learning program P2 being stored in the secondary memory 23 refers to the source code or an executable file obtained by compiling the source code being stored in the secondary memory 23. Furthermore, the fact that the learned model LM is stored in the secondary memory 23 means that the parameters that define the learned model LM are stored in the secondary memory 23.

[0074] Examples of devices that can be used as the processor 21 include a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The processor 21 is also sometimes called an "arithmetic unit."

[0075] Furthermore, an example of a device that can be used as the primary memory 22 is a random access memory (RAM). The primary memory 22 is sometimes called a "main storage device." Furthermore, an example of a device that can be used as the secondary memory 23 is a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an optical disk drive (ODD), or a combination thereof. The secondary memory 23 is sometimes called an "auxiliary storage device." The secondary memory 23 may be built into the machine learning device 2, or may be built into another computer (e.g., a computer constituting a cloud server) connected to the machine learning device 2 via the input / output interface 24 or the communication interface 25. Note that, although the storage in the machine learning device 2 is realized by two memories (the primary memory 22 and the secondary memory 23) in this embodiment, this is not limiting. That is, the storage in the machine learning device 2 may be realized by a single memory. In this case, for example, one storage area of the memory may be used as the primary memory 22, and another storage area of the memory may be used as the secondary memory 23.

[0076] Input devices and / or output devices are connected to the input / output interface 24. Examples of the input / output interface 24 include interfaces such as USB (Universal Serial Bus), ATA (Advanced Technology Attachment), SCSI (Small Computer System Interface), and PCI (Peripheral Component Interconnect). An example of an input device connected to the input / output interface 24 is a data logger 5. Data acquired from the sensor group 4 in the machine learning method M2 is input to the machine learning device 2 via the data logger 5 and stored in the primary memory 22. Examples of input devices connected to the input / output interface 24 include a keyboard, a mouse, a touchpad, a microphone, or a combination thereof. Data acquired from a user in the machine learning method M2 is input to the machine learning device 2 via these input devices and stored in the primary memory 22. Examples of output devices connected to the input / output interface 24 include a display, a projector, a printer, a speaker, headphones, or a combination thereof. Information provided to the user in the machine learning method M2 is output from the machine learning device 2 via these output devices. The machine learning device 2 may have a built-in keyboard that functions as an input device and a built-in display that functions as an output device, like a laptop computer, or may have a built-in touch panel that functions as both an input device and an output device, like a tablet computer.

[0077] Other computers are connected to the communication interface 25 via a network, either wired or wirelessly. Examples of the communication interface 25 include interfaces such as Ethernet (registered trademark) and Wi-Fi (registered trademark). Available networks include, for example, a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a global area network (GAN), or an internetwork including these networks. The internetwork may be an intranet, an extranet, or the Internet. Data (e.g., the trained model LM) provided by the machine learning device 2 to other computers (e.g., the segmentation image estimation device 1) is transmitted and received via these networks.

[0078] Although the present embodiment employs a configuration in which the machine learning method M2 is executed using a single processor (processor 21), the present invention is not limited to this. That is, a configuration in which the machine learning method M2 is executed using multiple processors may be employed. For example, a configuration in which the training dataset creation step M21 (described below) is executed in a first processor and the trained model construction step M22 is executed in a second processor may be employed. In this case, the multiple processors that cooperate to execute the machine learning method M2 may be provided in a single computer and configured to be able to communicate with each other via a bus, or may be provided in multiple computers in a distributed manner and configured to be able to communicate with each other via a network. As an example, a processor built in a computer that constitutes a cloud server and a processor built in a computer owned by a user of the cloud server may cooperate to execute the machine learning method M2.

[0079] Furthermore, although this embodiment employs a configuration in which the training dataset DS is stored in memory (secondary memory 23) built into the same computer as the processor (processor 21) that executes the machine learning method M2, the present invention is not limited to this. That is, a configuration in which the training dataset DS is stored in memory built into a computer different from the processor that executes the machine learning method M2 may also be employed. In this case, the computer incorporating the memory that stores the training dataset DS is configured to be able to communicate with a computer incorporating the processor that executes the machine learning method M2 via a network. As an example, the training dataset DS may be stored in memory built into a computer that constitutes a cloud server, and a processor built into a computer owned by a user of the cloud server executes the machine learning method M2.

[0080] Furthermore, although this embodiment employs a configuration in which the training dataset DS is stored in a single memory (secondary memory 23), the present invention is not limited to this. That is, a configuration in which the training dataset DS is distributed and stored in multiple memories may be employed. In this case, the multiple memories that store the training dataset DS may be provided in a single computer (which may or may not be a computer incorporating a processor that executes the machine learning method M2), or may be distributed and stored in multiple computers (which may or may not include a computer incorporating a processor that executes the machine learning method M2). As an example, a configuration in which the training dataset DS is distributed and stored in memories incorporated in each of multiple computers that constitute a cloud server may be considered.

[0081] Furthermore, in this embodiment, a configuration is adopted in which the segmented image estimation method M1 and the machine learning method M2 are executed using different processors (processor 11 and processor 21), but the present invention is not limited to this. That is, the segmented image estimation method M1 and the machine learning method M2 may be executed using the same processor. In this case, by executing the machine learning method M2, a trained model LM is stored in memory built into the same computer as this processor. Then, when this processor executes the segmented image estimation method M1, it uses the trained model LM stored in this memory.

[0082] [Machine learning method flow] The flow of the machine learning method M2 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of the machine learning method M2.

[0083] The machine learning method M2 includes a learning dataset creation step M21 and a trained model construction step M22.

[0084] The learning data set creation step M21 is a step in which the processor 21 creates a learning data set DS which is a collection of learning data (teacher data) DS1, DS2, . . .

[0085] Each training data DSi (i = 1, 2, ...) contains leaf canopy image data. The leaf canopy image data contained in each training data DSi represents a leaf canopy image containing a leaf canopy as a subject, similar to the leaf canopy image data input to the trained model LM. In the training dataset creation step M21, the processor 21 acquires leaf canopy image data input from the camera C21 (see Figure 1). The method by which the processor 21 acquires the leaf canopy image data from the camera C21 is the same as the method by which the processor 11 acquires the leaf canopy image data from the camera C1.

[0086] Each training data DSi also includes classification image data as a label indicating the correct answer. The classification image data included in each training data DSi, like the classification image data output from the trained model LM, represents a classification image in which the leaf region and other regions in the leaf canopy are classified by estimating the boundary between the leaf region and the other regions. However, the classification image data included in each training data DSi differs from the classification image data output from the trained model LM in that the user intervenes when generating the classification image data, not the trained model LM. The user uses the computer C22 (see FIG. 1) to estimate the boundary between the leaf region and other regions included in the leaf canopy image (e.g., see FIG. 13(a)). Based on the boundary, the computer C22 sets the value of pixels corresponding to the leaf region to 1 and the value of pixels corresponding to other regions to 0, thereby generating classification image data representing a classification image (e.g., see FIG. 6). (a) of FIG. 13 is an example of a leaf canopy image represented by the leaf canopy image data included in each training data DSi. 13(b) is an example of a segmented image represented by segmented image data included in each learning data DSi. In the learning data set creation step M21, the processor 21 acquires segmented image data input from the computer C22.

[0087] Then, the processor 21 associates the leaf canopy image data with the classification image data and stores them in the secondary memory 23. The processor 21 repeats the above process to create a learning dataset DS.

[0088] The trained model construction step M22 is a step in which the processor 21 constructs the trained model LM. In the trained model construction step M22, the processor 21 constructs the trained model LM by supervised learning using the training dataset DS created in the training dataset creation step M21. The trained model LM constructed in the trained model construction step M22 is stored in the secondary memory 23.

[0089] [Example] A first example of a segmented image estimation system S according to an embodiment of the present invention will be described below.

[0090] <About leaf canopy image data> In this example, the preparation phase and practical phase were carried out using leaf canopy image data acquired in a greenhouse where tomatoes were grown.

[0091] Each leaf canopy image data set was created so that each leaf canopy image contained tomato leaf canopies of various sizes as the subject. To create this data set, 54 tomato seedlings of four varieties were cultivated. In addition, to verify the accuracy of the estimated LAI obtained by implementing segmentation image estimation method M1, the leaf length and width of all leaves contained in the leaf canopy included as the subject in each leaf canopy image were measured to obtain the measured LAI.

[0092] Furthermore, in each of the preparatory phase and practical phase, the multiple leaf canopy image data were divided into two groups. Some of the leaf canopy image data in the first group contained two or more rows of leaf canopies as subjects (see, for example, Figure 5(b)). On the other hand, the leaf canopy image data in the second group contained only one row of leaf canopies as subjects (see, for example, Figure 5(a)). The number of leaf canopy image data included in each of the first and second groups was 12 in the preparatory phase and 80 in the practical phase.

[0093] <Preparation Phase> In the preparation phase, a first trained model LM1 is constructed by machine learning using a first training dataset DS1 based on the leaf canopy image data of the first group (e.g., see (a) of FIG. 13) and segmented image data generated by the user (e.g., see (b) of FIG. 13). Similarly, a second trained model LM2 is constructed by machine learning using a second training dataset DS2 based on the leaf canopy image data of the second group and segmented image data generated by the user.

[0094] In the preparation phase, a user generated segmented image data using the first group of leaf canopy image data, and estimated a first correlation between the pixel ratio calculated using the segmented image data and the measured LAI. Also, a user generated segmented image data using the second group of leaf canopy image data, and estimated a second correlation between the pixel ratio calculated using the segmented image data and the measured LAI.

[0095] <Practical phase> In the practical phase, leaf canopy image data different from that used in the preparation phase was used. In the practical phase, segmentation image data was not generated by the user, but instead, the above-mentioned measured LAI was used to verify the accuracy of the estimated LAI obtained from the segmentation image estimation device 1. It is believed that the accuracy of this estimated LAI mainly reflects the accuracy of the pixel ratio obtained from the segmentation image estimation device 1.

[0096] Figures 14(a) to 14(f) show examples of leaf canopy images used in the practical phase. Figures 14(a) to 14(f) each show the measured LAI (measured LAI) and estimated LAI (estimated LAI) values for the leaf canopy that is the subject of each leaf canopy image. Figures 14(a) to 14(c) show leaf canopy images of the first group, and Figures 14(d) to 14(f) show leaf canopy images of the second group. The stepped leaf canopy that appears as the subject of the leaf canopy image in Figure 14(f) had grown large. Therefore, adjusting the elevation angle of camera C1 to match the height of the leaf canopy resulted in undesirable photographic conditions. As a result, the estimated LAI obtained from the leaf canopy image in Figure 14(f) was very small.

[0097] In addition, the pixel ratio obtained by implementing the classification image estimation method M1 using the leaf canopy image data of the first group as input and the actual measured values of the LAI of the leaf canopy that was the subject of each leaf canopy image of the leaf canopy image data of the first group are plotted in (a) of Figure 15. Note that the first correlation is also shown in (a) of Figure 15. In addition, the pixel ratio obtained by implementing the classification image estimation method M1 using the leaf canopy image data of the second group as input and the actual measured values of the LAI of the leaf canopy that was the subject of each leaf canopy image of the leaf canopy image data of the second group are plotted in (b) of Figure 15. Note that the second correlation is also shown in (b) of Figure 15. Figures 15(a) and (b) are graphs showing the error distribution in the estimated LAI.

[0098] Furthermore, from the results of the above examples, the mean absolute error (MAE) and The root mean square error (RMSE) of the total difference between the estimated LAI and the measured LAI, and R 2 The values were calculated.

[0099] The MAE was less than 0.4 when the leaf canopy image data of the first group was used as input, and was less than 0.5 when the leaf canopy image data of the second group was used as input.

[0100] In both cases where the leaf canopy image data of the second group was used as the input to the first trained model LM1 instead of the leaf canopy image data of the first group, and where the leaf canopy image data of the first group was used as the input to the second trained model LM2 instead of the leaf canopy image data of the second group, the increase in MAE was less than 0.1. This indicates that the segmented image estimation device 1 and segmented image estimation method M1 are highly robust to changes in conditions when capturing the leaf canopy images used to estimate the regression model.

[0101] The difference between the MAE when a Bezier curve is used as the shape of the position dependence of the weight values used in the trimming step M131 (see, for example, FIG. 9 ) and the MAE when a straight line is used as the shape is smaller than the difference in MAE caused by changes in conditions when capturing the leaf canopy images used to estimate the regression model described above.

[0102] <Modification of post-processing step> This embodiment employs the post-processing step M13 shown in Fig. 3(b), which includes a weighted image generation step M132 in which the leaf canopy image is flipped horizontally as necessary to match the ends of the weighting filter with the larger weight values to the front side of the leaf canopy, which is the subject.

[0103] In the first modification of the post-processing step M13, the inversion of the leaf canopy image, which is performed as needed in the weighted image generation step M132, is omitted.

[0104] In addition, in the second modification of the post-processing step M13, the weighted image generation step M132 is omitted.

[0105] MAE, RMSE, and R in the embodiment, the first modified example, and the second modified example 2 is shown below. [Table 1]

[0106] Comparing the embodiment, the first modified example, and the second modified example, in the first modified example and the second modified example, R 2It was found that the MAE significantly decreased in the second variant. It was also found that the MAE rapidly increased in the second variant. Therefore, it was found that it is preferable to perform a weighted image generation step M132 in the post-processing step M13, which includes inversion of the leaf canopy image as needed. However, the first and second variants of the post-processing step M13 are also included in the scope of the present invention.

[0107] <Comparison with reference example> The technique using a 3D scanner described in Non-Patent Document 1 and this embodiment are compared in terms of RMSE and R 2 The results of the comparison are shown below. Non-Patent Document 1 evaluates the estimated LAI in leaf canopies of cucumber, pepper, and tomato. [Table 2]

[0108] While Non-Patent Document 1 reports good results for cucumber and pepper, the effect obtained for tomato is limited because the degree of leaf overlap in the tomato leaf canopy is higher than that in the cucumber and pepper leaf canopies.

[0109] In this example, no results were obtained for the leaf canopies of cucumber and pepper, but in the case of the leaf canopy of tomato, a higher R 2 was obtained.

[0110] Note that the RMSE obtained by the second pattern of this embodiment shows a very large value, which is due to the leaf canopy image being captured in unfavorable conditions, such as the leaf canopy image shown in Figure 14 (f).

[0111] 〔summary〕 A segmented image estimation device according to aspect 1 includes one or more processors that execute a first estimation step of estimating boundaries between leaf regions and other regions in a leaf canopy using a trained model constructed by machine learning, thereby generating a segmented image in which the leaf regions are segmented from the other regions. In this segmented image estimation device, an input to the trained model is leaf canopy image data representing a leaf canopy image including a leaf canopy as a subject, and an output of the trained model is segmented image data representing the segmented image.

[0112] According to the segmented image estimation device of aspect 1, segmented image data can be output using a trained model, and therefore a technology can be provided for evaluating the characteristics of leaves in a leaf community using a method different from a 3D scanner.

[0113] In the segmented image estimation device according to aspect 2, in addition to the configuration of the segmented image estimation device according to aspect 1 described above, a configuration is adopted in which the one or more processors further execute an output step of outputting the number of pixels included in the leaf region of the segmented image.

[0114] According to the segmented image estimation device of aspect 2, the one or more processors output the number of pixels included in the leaf region, and therefore the number of pixels can be used to evaluate the characteristics of the leaf.

[0115] In the segmented image estimation device of aspect 3, in addition to the configuration of the segmented image estimation device of aspect 2 described above, the one or more processors further execute a ratio calculation step of calculating a ratio of the number of pixels to the total number of pixels of the segmented image, and a second estimation step of estimating the leaf area index corresponding to the ratio by referring to a predetermined correlation between the ratio and the leaf area index in the leaf canopy.

[0116] According to the classification image estimation device of aspect 3, the leaf area index can be estimated using the ratio of the number of pixels to the total number of pixels of the classification image. Therefore, the classification image estimation device can evaluate the leaf area index as one aspect of trait evaluation.

[0117] In a segmentation image estimation device according to aspect 4, in addition to the configuration of the segmentation image estimation device according to aspect 3 described above, the leaf canopy is formed along a row and has a pair of side surfaces, the leaf canopy image is one of the pair of side surfaces photographed at an oblique angle, and the one or more processors are configured to further execute a weighted image generation step executed between the first estimation step and the second estimation step, in which the one or more processors generate a weighted image by applying a weighting filter to the segmentation image. Here, the weighting filter is configured so that the weight value decreases or increases as the distance from a region corresponding to the front side of the one side surface increases toward a region corresponding to the back side.

[0118] According to the classification image estimation device of the fourth aspect, weighting can be applied to the classification image according to the position of the leaf in the leaf canopy, so that the number of pixels included in the leaf area can be counted with high accuracy, and thus the leaf area index can be estimated with high accuracy.

[0119] In the segmentation image estimation device according to aspect 5, in addition to the configuration of the segmentation image estimation device according to aspect 4 described above, the weighting filter is configured so that the weighting value becomes smaller or larger as the weighting value increases from one of the left and right ends of the leaf canopy image, i.e., a first end, to the other end, i.e., a second end. In addition, the one or more processors are configured so that, in the weighted image generation step, if the leaf canopy image is an image of a first side of the pair of sides, the one or more processors apply the weighting filter after flipping the leaf canopy image left and right, and if the leaf canopy image is an image of a second side of the pair of sides, the one or more processors apply the weighting filter to the leaf canopy image as is.

[0120] According to the segmented image estimation device of aspect 5, in a leaf canopy formed along a row, a weighting filter can be applied in an appropriate direction regardless of which of a pair of sides is the subject of the leaf canopy image.

[0121] In the segmentation image estimation device of aspect 6, in addition to the configuration of the segmentation image estimation device of aspect 5 described above, the one or more processors further execute a trimming step between the first estimation step and the weighted image generation step, and in the trimming step, when two adjacent rows of leaf canopies are included in the segmentation image, the widths of each leaf canopy in the segmentation image are compared, and an area that includes the leaf canopy with the larger width and does not include the leaf canopy with the smaller width is trimmed as a region of interest, and the image of the region of interest is used as a new segmentation image.

[0122] According to the classification image estimation device of aspect 6, even if two adjacent rows of leaf communities are included in the classification image because the two adjacent rows of leaf communities are included in the leaf community image, a classification image can be obtained that includes a leaf community that is suitable as a subject from among the two rows of leaf communities.

[0123] A segmented image estimation method according to aspect 7 includes a first estimation step in which one or more processors use a trained model constructed by machine learning to estimate boundaries between leaf regions and other regions in a leaf canopy, thereby generating a segmented image in which the leaf regions are segmented from the other regions. In this segmented image estimation method, an input to the trained model is leaf canopy image data representing a leaf canopy image including a leaf canopy as a subject, and an output of the trained model is segmented image data representing the segmented image.

[0124] The segmented image estimation method according to the seventh aspect has the same effects as the segmented image estimation device according to the first aspect.

[0125] The segmented image estimation method of aspect 8 employs a configuration that, in addition to the configuration of the segmented image estimation method of aspect 7 described above, further includes an output step of outputting the number of pixels included in the leaf region of the segmented image.

[0126] The segmented image estimation method according to the eighth aspect has the same effects as the segmented image estimation device according to the second aspect.

[0127] In the segmented image estimation method of aspect 9, in addition to the configuration of the segmented image estimation method of aspect 8 described above, a configuration is adopted which further includes a ratio calculation step of calculating the ratio of the number of pixels to the total number of pixels of the segmented image, and a second estimation step of estimating the leaf area index corresponding to the ratio by referring to a predetermined correlation between the ratio and the leaf area index in the leaf canopy.

[0128] The segmented image estimation method according to the ninth aspect has the same effects as the segmented image estimation device according to the third aspect.

[0129] A segmentation image estimation method according to aspect 10, in addition to the configuration of the segmentation image estimation method according to aspect 9 described above, further includes a weighting image generation step executed between the first estimation step and the second estimation step, in which the leaf canopy is formed along a row and has a pair of side surfaces, the leaf canopy image being obtained by photographing one of the pair of side surfaces at an oblique angle, and the weighting image generation step applies a weighting filter to the segmentation image to generate a weighted image. Here, the weighting filter is configured so that the weight value decreases as the distance from the region corresponding to the front side of the one side surface increases toward the region corresponding to the back side.

[0130] The segmented image estimation method according to the tenth aspect has the same effects as the segmented image estimation device according to the fourth aspect.

[0131] In the segmentation image estimation method of aspect 11, in addition to the configuration of the segmentation image estimation method of aspect 10 described above, the weighting filter is configured so that the weighting value becomes smaller as one moves from the first end, which is one of the left and right ends of the leaf canopy image, to the second end, which is the other end, and in the weighted image generation step, if the leaf canopy image is an image of the first side of the pair of sides, the leaf canopy image is flipped left and right and the weighting filter is applied, and if the leaf canopy image is an image of the second side of the pair of sides, the weighting filter is applied to the leaf canopy image as is.

[0132] The segmented image estimation method according to the eleventh aspect has the same effects as the segmented image estimation device according to the fifth aspect.

[0133] In the segmentation image estimation method of aspect 12, in addition to the configuration of the segmentation image estimation method of aspect 11 described above, a trimming step is further included which is executed between the first estimation step and the weighted image generation step, and in the trimming step, when two adjacent rows of leaf canopies are included in the segmentation image, the widths of each leaf canopy in the segmentation image are compared, and an area which includes the leaf canopy with the larger width and does not include the leaf canopy with the smaller width is trimmed as a region of interest, and the image of the region of interest is used as a new segmentation image.

[0134] The segmented image estimation method according to the twelfth aspect has the same effects as the segmented image estimation device according to the fifth aspect.

[0135] A machine learning device according to aspect 13 includes one or more processors that execute a construction step of constructing a trained model that generates a segmented image in which a leaf region in a leaf canopy is segmented from another region by estimating a boundary between the leaf region and another region in the leaf canopy through machine learning using a training dataset. In this machine learning device, an input of the trained model is leaf canopy image data representing a leaf canopy image including a leaf canopy as a subject, and an output of the trained model is segmented image data representing the segmented image.

[0136] According to the machine learning device of the thirteenth aspect, a trained model that can be used in the segmented image estimation device of any one of the first to sixth aspects can be constructed.

[0137] A machine learning method according to aspect 14 includes a construction step of constructing a trained model that generates a segmented image in which a leaf region in a leaf canopy is segmented from another region by estimating a boundary between the leaf region and another region in the leaf canopy through machine learning using a training dataset. In this machine learning method, an input of the trained model is leaf canopy image data representing a leaf canopy image including a leaf canopy as a subject, and an output of the trained model is segmented image data representing the segmented image.

[0138] According to the machine learning device of the fourteenth aspect, it is possible to construct a trained model that can be used in the segmented image estimation method of any one of the seventh to twelfth aspects.

[0139] The scope of the present invention also includes a program for causing a computer to function as a segmentation image estimation device according to any one of aspects 1 to 6, which causes the computer to execute the first estimation step, and a program for causing a computer to operate as a machine learning device according to aspect 13, which causes the computer to execute the construction step.

[0140] [Additional Notes] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Other embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0141] 1. Segmentation image estimation device 11 processors 12 Primary Memory 13 Secondary Memory 14 Input / Output Interface 15 Communication Interface 16 Bus M1 Segmentation Image Estimation Method M11 Acquisition Step M12 First estimation step M13 Post-processing step 2. Machine learning equipment 21 processors 22 Primary Memory 23 Secondary Memory 24 Input / Output Interface 25 Communication Interface 26 Bus M2 Machine Learning Method M21 Training Dataset Creation Steps M22 Trained model construction step S Segmentation Image Estimation System

Claims

1. The method includes one or more processors that execute a first estimation step of estimating a boundary between a leaf region and other regions in a leaf canopy using a trained model constructed by machine learning, thereby generating a segmented image in which the leaf region and the other regions are segmented, The input of the trained model is leaf canopy image data representing a leaf canopy image including a leaf canopy as a subject, The output of the trained model is segmented image data representing the segmented image, the one or more processors: an output step of outputting the number of pixels included in the leaf region of the segmented image; a ratio calculation step of calculating a ratio of the number of pixels to the total number of pixels of the segmented image; and (b) a second estimation step of estimating the leaf area index corresponding to the ratio by referring to a predetermined correlation between the ratio and the leaf area index in the leaf canopy, The leaf canopy has a pair of sides, the leaf canopy image is an image of one of the pair of side surfaces taken at an oblique angle, The one or more processors further perform a weighted image generation step, which is performed between the first estimation step and the second estimation step, in which a weighted image is generated by applying a weight filter having a weight according to the region on the one side surface to the segmented image. A segmented image estimation device characterized by:

2. The leaf canopy is formed along a row, The weighting filter is configured so that the weight value becomes smaller or larger as it moves from a region corresponding to the front side of the one side surface to a region corresponding to the back side.

2. The segmented image estimation device according to claim 1.

3. the weighting filter is configured so that the weight value becomes smaller or larger as it moves from a first end, which is one of the left and right ends of the leaf canopy image, to a second end, which is the other end, The one or more processors, in the weighted image generating step, If the leaf canopy image is an image of a first side surface of the pair of side surfaces, the leaf canopy image is inverted left and right and a weighting filter is applied thereto; If the leaf canopy image is an image of a second side surface of the pair of side surfaces, a weighting filter is applied to the leaf canopy image as is.

3. The segmented image estimation device according to claim 2.

4. the one or more processors further perform a cropping step between the first estimation step and the weighted image generation step; In the trimming step, when two adjacent rows of leaf canopies are included in the segmented image, the widths of the leaf canopies in the segmented image are compared, and an area that includes the leaf canopy with the larger width and does not include the leaf canopy with the smaller width is trimmed as a region of interest, and the image of the region of interest is used as a new segmented image.

4. The segmented image estimation device according to claim 3.

5. a first estimation step in which one or more processors estimate a boundary between a leaf region and another region in a leaf canopy using a trained model constructed by machine learning, thereby generating a segmented image in which the leaf region and the other region are segmented; The input of the trained model is leaf canopy image data representing a leaf canopy image including a leaf canopy as a subject, The output of the trained model is segmented image data representing the segmented image, an output step in which the one or more processors output the number of pixels included in the leaf region in the segmented image; a ratio calculation step in which the one or more processors calculate a ratio of the number of pixels to the total number of pixels of the segmented image; a second estimation step in which the one or more processors refer to a predetermined correlation between the ratio and a leaf area index in a leaf canopy, and estimate the leaf area index corresponding to the ratio; The leaf canopy has a pair of sides, the leaf canopy image is an image of one of the pair of side surfaces taken at an oblique angle, The method further includes a weighted image generation step executed by the one or more processors between the first estimation step and the second estimation step, in which a weighted image is generated by applying a weighting filter having a weight according to the region on the one side to the segmented image. A segmented image estimation method characterized by:

6. The leaf canopy is formed along a row, The weighting filter is configured so that the weight value decreases as it moves from a region corresponding to the front side of the one side surface to a region corresponding to the back side.

6. The segmented image estimation method according to claim 5.

7. the weighting filter is configured so that the weight value becomes smaller or larger as it moves from a first end, which is one of the left and right ends of the leaf canopy image, to a second end, which is the other end, In the weighted image generating step, When the leaf canopy image is an image of a first side surface of the pair of side surfaces, the leaf canopy image is inverted left and right and then a weighting filter is applied; When the leaf canopy image is an image of a second side surface of the pair of side surfaces, a weighting filter is applied to the leaf canopy image as is.

7. The segmented image estimation method according to claim 6.

8. further comprising a cropping step performed between the first estimation step and the weighted image generation step; In the trimming step, when two adjacent rows of leaf canopies are included in the segmented image, the widths of the leaf canopies in the segmented image are compared, and an area that includes the leaf canopy with the larger width and does not include the leaf canopy with the smaller width is trimmed as a region of interest, and the image of the region of interest is used as a new segmented image. The segmented image estimation method according to claim 7 .

9. 5. A program for causing a computer to function as the segmentation image estimation device according to claim 1, the program causing the computer to execute the first estimation step.

Images