Analysis system, analysis method, and analysis program

Abstract

To accurately calculate a wilting feature quantity for a plant.SOLUTION: An analysis system includes at least one processor. The at least one processor acquires a first image capturing a target plant including one or more leaves, performs segmentation on the first image to detect the one or more leaves in the first image, specifies, among the detected one or more leaves, a leaf located in the foreground as a target leaf, and sets a plurality of tracking points on the target leaf. The processor further acquires a plurality of second images capturing the target plant, detects positions of the tracking points in each of the second images, calculates transitions of the tracking points, and calculates a wilting feature quantity indicating features related to the wilting of the target plant based on the transitions of the tracking points.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an analysis system, an analysis method, and an analysis program. [Background technology]

[0002] Conventionally, there have been known methods for calculating feature amounts related to the wilting of plants. For example, Patent Document 1 describes a method for calculating the degree of wilting of plants using optical flow. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7198496 Summary of the Invention [Problem to be solved by the invention]

[0004] A method that can accurately calculate wilting features for plants is desired. [Means for solving the problem]

[0005] An analysis system according to one aspect of the present disclosure includes at least one processor that acquires a first image of a target plant including one or more leaves, performs segmentation on the first image to detect one or more leaves in the first image, identifies a leaf located in the foreground of the detected one or more leaves as a target leaf, sets a plurality of tracking points for the target leaf, acquires a plurality of second images of the target plant, detects positions of the plurality of tracking points in each of the plurality of second images, calculates transitions of the plurality of tracking points, and calculates wilting feature amounts that indicate feature amounts related to wilting of the target plant based on the transitions of the plurality of tracking points.

[0006] An analysis method according to one aspect of the present disclosure is executed by an analysis system including at least one processor. The analysis method includes the steps of acquiring a first image of a target plant including one or more leaves, performing segmentation on the first image to detect one or more leaves in the first image, identifying a leaf located in the foreground of the detected one or more leaves as a target leaf, setting a plurality of tracking points for the target leaf, acquiring a plurality of second images of the target plant, detecting positions of the plurality of tracking points in each of the plurality of second images and calculating transitions of the plurality of tracking points, and calculating wilting feature amounts indicating feature amounts related to wilting of the target plant based on the transitions of the plurality of tracking points.

[0007] An analysis method program according to one aspect of the present disclosure causes a computer to execute the following steps: acquiring a first image of a target plant including one or more leaves; performing segmentation on the first image to detect one or more leaves in the first image; identifying a leaf located in the foreground of the one or more detected leaves as the target leaf; setting multiple tracking points for the target leaf; acquiring multiple second images of the target plant; detecting the positions of the multiple tracking points in each of the multiple second images and calculating the progression of the multiple tracking points; and calculating wilting features that indicate features related to wilting of the target plant based on the progression of the multiple tracking points.

[0008] In one aspect of the present disclosure, a leaf located in the foreground in the first image is identified as a target leaf, and the transitions of multiple tracking points set on the target leaf are calculated. Then, a wilting feature amount for the target plant is calculated based on the transitions of the multiple tracking points. In this way, by identifying a foreground leaf with a larger transition amount than leaves in the background as the target leaf, wilting can be clearly detected. Furthermore, by calculating the transitions for multiple tracking points set on the target leaf rather than the entire target plant, the positions of the tracking points can be accurately detected even if the brightness values ​​of corresponding tracking points change between multiple second images due to factors such as sunlight. As a result, it is possible to accurately calculate wilting feature amounts for the plant. [Effects of the Invention]

[0009] According to one aspect of the present disclosure, it is possible to accurately calculate wilting feature amounts for plants. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 illustrates an example of a functional configuration of an analysis system. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer that constitutes the analysis system. [Figure 3] 10 is a flowchart illustrating an example of a process for calculating a withering feature amount. [Figure 4] FIG. 2 is a diagram showing an example of a first image. [Figure 5] 10 is a flowchart illustrating an example of a process for detecting one or more leaves in a first image. [Figure 6] FIG. 10 is a diagram showing an example of one or more leaves detected in the first image. [Figure 7] FIG. 10 is a diagram illustrating an example of the depth for each of one or more detected leaves. [Figure 8] FIG. 10 is a diagram illustrating an example of a specified target leaf. [Figure 9] FIG. 10 is a diagram illustrating another example of identified target leaves. [Figure 10]FIG. 10 is a diagram illustrating an example of a plurality of tracking points set on a target leaf. [Figure 11] 1 is a graph showing an example of the transition of a water stress index. [Figure 12] 10 is a flowchart showing an example of a process for estimating the amount of recovery from withering. [Figure 13] 10 is a flowchart illustrating an example of a process for determining whether or not irrigation is required for a target plant. DETAILED DESCRIPTION OF THE INVENTION

[0011] Various examples of the present disclosure will be described in detail below with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and redundant description will be omitted.

[0012] [System Overview] The analysis system according to the present disclosure is a computer system that performs analysis of wilting in plants that include one or more leaves. In one example, the analysis system performs analysis on plants grown in a farm or the like. In this example, a user of the analysis system can refer to the analysis results to understand the growth status of the plant and appropriately control the cultivation environment, such as irrigation control or air conditioning control, for the plant. Alternatively, the analysis system may perform analysis on wild plants. The plant in the present disclosure may be, for example, a fruit-bearing plant, such as a tomato, or a non-fruit-bearing plant. In one example, the analysis system performs the analysis at predetermined time intervals. The predetermined time interval may be, for example, one minute, five minutes, or ten minutes.

[0013] In one example, the analysis system performs the analysis by calculating a wilting feature for the plant. In the present disclosure, the wilting feature refers to the degree of water deficiency in the plant, i.e., a feature related to water stress. Therefore, the wilting feature in the present disclosure can also be said to be an index indicating the degree of water content in the plant.

[0014] In one example, the analysis system may perform the analysis by predicting the amount of recovery from wilting of the plant and determining whether irrigation is necessary for the plant. In the present disclosure, the amount of recovery from wilting refers to an index that indicates the extent to which the plant will recover from water stress when irrigated.

[0015] In one example, the analysis system may perform the above-mentioned calculation of wilting characteristics, prediction of wilt recovery amount, and determination of whether irrigation is necessary consecutively as a series of processes, or may perform these processes separately as individual processes.

[0016] [System Configuration] An example of application of an analysis system 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the functional configuration of the analysis system 10. In this example, the analysis system 10 acquires information necessary for analyzing plant wilting from an imaging device 20, an environmental sensor 30, and an irrigation control device 40, and performs the analysis. In the present disclosure, a plant that is the subject of analysis by the analysis system 10 is also referred to as a target plant.

[0017] In one example, the analysis system 10 includes functional modules including an acquisition unit 11, a detection unit 12, an identification unit 13, a calculation unit 14, an estimation unit 15, a determination unit 16, and an output unit 17. The acquisition unit 11 is a functional module that acquires an image of a target plant captured by an imaging device 20 and information obtained by an environmental sensor 30 and an irrigation control device 40. The captured image of a plant is also called a plant shape image. The detection unit 12 is a functional module that detects one or more leaves in the acquired image. The identification unit 13 is a functional module that identifies a target leaf from the detected one or more leaves. In the present disclosure, the target leaf refers to a leaf that is the target of analysis by the analysis system 10 among one or more leaves included in the target plant. The calculation unit 14 is a functional module that calculates wilting feature values ​​for the target plant. The estimation unit 15 is a functional module that estimates the amount of wilting recovery for the target plant. The determination unit 16 is a functional module that determines whether irrigation is necessary for the target plant. The output unit 17 is a functional module that outputs the processing results.

[0018] The imaging device 20 is a device that acquires images of the target plant at a predetermined interval. The predetermined interval may be, for example, one minute, five minutes, or ten minutes. The position, orientation, and angle of the imaging device 20 are set so that changes in the target plant can be detected. The imaging device 20 may capture an image of the entire target plant or only a portion of the target plant.

[0019] The environmental sensor 30 is a device that measures the environment surrounding the target plant at a predetermined interval. The predetermined interval may be, for example, one minute, five minutes, or ten minutes. The environmental sensor 30 is disposed, for example, in the cultivation environment of the target plant. In one example, the environmental sensor 30 is installed around the target plant in the cultivation environment. The environmental sensor 30 measures, for example, at least one of temperature, humidity, and illuminance. One environmental sensor 30 may acquire multiple types of values, or multiple types of environmental sensors 30 may acquire their own values.

[0020] The irrigation control device 40 is a device that controls irrigation of plants. In one example, the irrigation control device 40 controls the timing and amount of water irrigation. In this example, water is supplied to the plants through a hose under the control of the irrigation control device 40. In one example, the irrigation control device 40 records the time when irrigation is performed and outputs data indicating that time.

[0021] FIG. 2 is a diagram showing an example of the hardware configuration of a computer 100 constituting the analysis system 10. For example, the computer 100 includes a processor 101, a main memory 102, an auxiliary memory 103, a communication control unit 104, an input device 105, and an output device 106. The processor 101 executes an operating system and application programs. The main memory 102 is composed of, for example, ROM and RAM. The auxiliary memory 103 is composed of, for example, a hard disk or flash memory, and generally stores larger amounts of data than the main memory 102. The communication control unit 104 is composed of, for example, a network card or a wireless communication module. The input device 105 is composed of, for example, a keyboard, a mouse, a touch panel, etc. The output device 106 is composed of, for example, a monitor and speakers.

[0022] Each functional module of the analysis system 10 is realized by an analysis program 110 pre-stored in the auxiliary storage unit 103. Each functional module is realized by loading the analysis program 110 onto the processor 101 or the main storage unit 102 and having the processor 101 execute the analysis program 110. In accordance with the analysis program 110, the processor 101 operates the communication control unit 104, the input device 105, or the output device 106 to read and write data from and to the main storage unit 102 or the auxiliary storage unit 103.

[0023] The analysis program 110 may be provided in the form of being recorded on a non-transitory recording medium such as a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Alternatively, the analysis program 110 may be provided via a communication network as a data signal superimposed on a carrier wave.

[0024] The analysis system 10 may be configured with one computer 100 or multiple computers 100. When multiple computers 100 are used, these computers 100 are connected via a communication network such as the Internet or an intranet to logically construct a single analysis system 10. The analysis system 10 may also be constructed by combining multiple types of computers.

[0025] [System Operation] The operation of the analysis system 10 will be described, along with the analysis method according to the present disclosure.

[0026] (Calculation of wilting features) First, a process for calculating the wilting feature amount of a target plant will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of a process for calculating the wilting feature amount of a target plant as a process flow S1.

[0027] In step S11, the acquisition unit 11 acquires a first image. The first image is an image of a target plant. The first image is, for example, a color image. The first image may be a still image or a frame image constituting a video. In one example, the acquisition unit 11 acquires the first image directly from the imaging device 20. Alternatively, the acquisition unit 11 may acquire the first image by accessing a given database or file system, or may acquire the first image input by a user of the analysis system 10.

[0028] Here, an example of the first image will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the first image. In the example shown in Fig. 4, the target plant TP is located approximately in the center of the first image. As shown in this example, only a portion of the target plant TP may be captured in the first image. Alternatively, the first image may capture the entire target plant TP.

[0029] In step S12, the detection unit 12 detects one or more leaves in the first image. Details of this detection will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of processing for detecting one or more leaves in the first image.

[0030] In step S121, the detection unit 12 performs segmentation on the first image to detect one or more objects in the first image. Here, the one or more detected objects correspond to each object present in the first image including the target plant TP. In one example, the detection unit 12 detects each object without identifying the type of each object.

[0031] In one example, the detection unit 12 performs semantic segmentation on the first image to detect one or more objects in the first image. In this example, the detection unit 12 detects one or more objects using a segmentation model generated in advance. The segmentation model is, for example, a model generated by machine learning. Machine learning refers to a technique for autonomously finding laws or rules by iteratively learning based on given information. The segmentation model receives an image and outputs information indicating whether an object exists for each pixel in the received image. The detection unit 12 inputs the first image into such an analysis model to detect one or more objects in the first image.

[0032] In step S122, the detection unit 12 eliminates objects whose area in the first image satisfies a predetermined condition. In the present disclosure, "eliminating an object" refers to excluding the object from the processing targets in subsequent processing. In one example, the detection unit 12 eliminates objects whose area in the first image is equal to or greater than a first threshold and objects whose area in the first image is equal to or less than a second threshold.

[0033] In one example, the first threshold is set based on the relationship between the total area of ​​the first image and the typical area occupied by man-made objects in the first image. Since man-made objects are generally larger than plants, the area occupied by man-made objects in the first image also tends to be larger than the area occupied by plants. Therefore, by setting the first threshold based on this relationship, it is possible to appropriately remove objects corresponding to man-made objects in the first image. In this example, the first threshold may be set to 50% or 80%.

[0034] In one example, the second threshold is set based on the distance from the imaging device 20 that captured the image. In an image, the farther an object is located from the imaging device 20, the smaller the area it occupies in the image. Therefore, by setting the second threshold based on the distance from the imaging device 20, it is possible to appropriately remove objects located in the background of the image. In this example, the second threshold may be set to 1% or 0.1%.

[0035] In step S123, the detection unit 12 identifies one or more plant objects from the one or more remaining objects. In the present disclosure, a plant object refers to an object corresponding to an element constituting the target plant, and the element includes a stem and a leaf. In other words, it can be said that the detection unit 12 identifies one or more objects corresponding to a stem or a leaf from the remaining objects.

[0036] In one example, the detection unit 12 identifies one or more plant objects based on pixel values ​​of pixels included in the one or more remaining objects. The pixel value refers to a value corresponding to the color of the pixel. As described above, the first image is, for example, a color image in which the pixel value of each pixel is expressed as an RGB value.

[0037] In this example, the detection unit 12 performs the following process for each of the remaining one or more objects. The detection unit 12 identifies as green pixels pixels that represent the remaining objects and whose RGB values ​​are within a range interpreted as green. For example, the detection unit 12 determines as green pixels pixels those whose pixel values ​​corresponding to red and blue are equal to or less than a threshold value Ta and whose pixel value corresponding to green is equal to or greater than a threshold value Tb. Therefore, green pixels are not limited to pixels whose RGB values ​​are (0, 255, 0) but may also be pixels having other RGB values, such as (100, 230, 30) or (0, 240, 70). The detection unit 12 identifies an object as a plant object if the ratio of the number of green pixels to the total number of pixels representing the object is equal to or greater than a third threshold. In this example, the third threshold may be set to 50% or 80%. Identifying one or more plant objects based on the colors of the pixels included in each object prevents objects that do not correspond to elements constituting the target plant from being identified as plant objects.

[0038] In step S124, the detection unit 12 detects one or more leaves based on the shapes of the one or more identified plant objects. The detection unit 12 may identify one or more leaves, for example, as follows: First, the detection unit 12 calculates a circumscribing rectangle for each of the one or more plant objects. Next, for each of the calculated one or more circumscribing rectangles, the detection unit 12 calculates the ratio of the short side of the circumscribing rectangle to the long side of the circumscribing rectangle. Next, the detection unit 12 determines that a plant object whose ratio is equal to or less than a fourth threshold is a plant object corresponding to a stem, and detects the remaining plant objects as one or more leaves. In this example, the fourth threshold may be set to 10% or 30%. By taking the aspect ratio of the circumscribing rectangle into consideration in this way, it is possible to prevent an object corresponding to a stem, which is generally longer and thinner than a leaf, from being detected as a leaf.

[0039] FIG. 6 shows the results of performing the above-described detection process on the first image shown in FIG. 4. FIG. 6 is a diagram illustrating an example of one or more leaves Ls detected in the first image. In FIG. 6, one or more identified leaves Ls are hatched. In FIG. 6, stems and the like are not hatched, and it can be seen that the leaves Ls are properly detected through the above-described detection process. Furthermore, it can be seen that each identified leaf Ls is located in the foreground compared to leaves that are not hatched. In this way, by preliminarily eliminating artificial objects, stems, leaves located near the background, and the like, it is possible to reduce the number of objects to be targeted in subsequent processes, thereby reducing the processing load on the analysis system 10.

[0040] Returning to FIG. 3, in step S13, the identification unit 13 identifies a leaf located in the foreground of the one or more detected leaves as a target leaf. In one example, the identification unit 13 calculates a depth for each of the one or more detected leaves and identifies the target leaf based on the depth. Depth is an index indicating how far an object represented by a pixel in an image is located from the viewpoint of the image. Depth is expressed in 256 levels, for example, from 0 to 255, with a smaller value indicating a greater distance from the viewpoint of the image. In the case of a first image captured by the imaging device 20, the depth can also be said to be the distance between the object represented by each pixel in the first image and the imaging device 20.

[0041] In this example, the identification unit 13 identifies the target leaf as follows: First, the identification unit 13 calculates the depth for each pixel corresponding to each of the one or more leaves. Next, the identification unit 13 calculates a depth statistic for each of the one or more leaves based on the calculated depth for each pixel. This statistic is, for example, the average, median, or maximum value.

[0042] FIG. 7 shows the results of such depth calculation for one or more leaves Ls shown in FIG. 6. FIG. 7 is a diagram showing an example of the depth for each of the one or more detected leaves Ls, and is an enlarged diagram of a portion of the first image. In FIG. 7, the calculated depth statistical values ​​DP are superimposed on the corresponding leaves Ls. Note that in FIG. 7, for convenience of explanation, the statistical values ​​DP are superimposed on only some of the leaves Ls.

[0043] Next, the identification unit 13 identifies the leaf with the largest statistical value as the target leaf. Therefore, in this example, of the one or more detected leaves, the leaf closest to the image capture device 20, i.e., the leaf located in the foreground, is identified as the target leaf.

[0044] FIG. 8 shows the results of performing such identification processing on one or more leaves Ls in FIG. 7. FIG. 8 is a diagram showing an example of an identified target leaf TL. In FIG. 7, the leaf Ls with a statistical value DP of "240" is located in the foreground, and in FIG. 8, this leaf Ls is identified as the target leaf TL. In FIG. 8, the outer edge TLa of the target leaf TL is highlighted, thereby distinguishing the target leaf TL from other leaves Ls in the image. The following explanation will exemplify a case where the target leaf TL has been identified.

[0045] In one example, the identification unit 13 generates a mask image as shown in FIG. 9 as an image in which the target leaf TL is identified. FIG. 9 is a diagram showing another example of an identified target leaf TL. In FIG. 9, the target leaf TL is represented in white, and the rest of the background is represented in black. In other words, the mask image shown in FIG. 9 can also be said to be a binarized image. Note that, unlike FIGS. 7 and 8 which are enlarged views of a portion of the first image, FIG. 9 is an image in which the entire first image has been binarized.

[0046] Returning to FIG. 3, in step S14, the calculation unit 14 sets a plurality of tracking points TM for the target leaf TL. In one example, the calculation unit 14 sets a plurality of tracking points TM as follows. First, the calculation unit 14 reduces the target leaf TL to generate a reduced target leaf whose entirety is located inside the outer edge TLa. In one example, the calculation unit 14 generates a reduced target leaf by performing a morphological transformation or the like on the mask image shown in FIG. 9 to shrink the target leaf TL.

[0047] Next, the calculation unit 14 sets a plurality of tracking points TM for the target leaf TL. In one example, the calculation unit 14 sets a plurality of tracking points TM in a grid pattern for the target leaf TL. In this example, the calculation unit 14 sets a plurality of tracking points TM for the target leaf TL such that the plurality of tracking points TM are located inside the generated reduced target leaf. In the present disclosure, "setting a plurality of tracking points TM in a grid pattern" means setting a plurality of tracking points TM such that the distance between adjacent tracking points TM in the horizontal or vertical direction is equal.

[0048] By setting the tracking points TM on the target leaf TL so that the tracking points TM are located within the reduced target leaf that is entirely located inside the outer edge TLa of the target leaf TL, each tracking point TM is located inside the outer edge TLa without being located on the outer edge TLa, as shown in Fig. 10. Fig. 10 is a diagram showing an example of the tracking points TM set on the target leaf TL. That is, the calculation unit 14 sets the tracking points TM inside the outer edge TLa of the target leaf TL without setting the tracking points TM on the outer edge TLa.

[0049] Returning to FIG. 3 , in step S15, the acquisition unit 11 acquires a plurality of second images. Each of the second images is an image of a target plant. Each of the second images is, for example, a color image. The plurality of second images is configured as a series of images arranged in chronological order. In one example, the plurality of second images may include the first image acquired in step S11. In this case, the first image may be the oldest image in the chronological order of the plurality of second images. Each of the second images may be a still image or a single frame image constituting a video. In one example, the acquisition unit 11 acquires the plurality of second images directly from the imaging device 20. Alternatively, the acquisition unit 11 may acquire the plurality of second images by accessing a given database or file system, or may acquire the plurality of second images input by a user of the analysis system 10.

[0050] In step S16, the calculation unit 14 detects the positions of the tracking points TM in each of the second images and calculates the transitions of the tracking points TM. In the present disclosure, the transitions of the tracking points TM refer to changes in the positions of the tracking points TM in the second images. In other words, the transitions of the tracking points TM are trajectories that indicate how the positions of the tracking points TM move across the second images, and in step S16, the calculation unit 14 can also be said to calculate the trajectories of the tracking points TM.

[0051] In one example, the calculation unit 14 calculates the transition of the tracking points TM as follows. First, the calculation unit 14 calculates the position of each tracking point TM in each of the multiple second images. In this example, the position of each tracking point TM is defined by an X coordinate indicating the horizontal position of the tracking point TM and a Y coordinate indicating the vertical position of the tracking point TM. In one example, the process of calculating the positions of the multiple tracking points TM is realized by a technology called CoTracker. Then, the calculation unit 14 calculates a statistical value of the Y coordinate of each tracking point TM in each of the multiple second images. This statistical value is, for example, an average or median. Then, the calculation unit 14 calculates the transition of the height position of the target leaf TL by connecting the statistical values ​​of the Y coordinates across the multiple second images. This transition of the height position of the target leaf TL can be said to be an example of the transition of the multiple tracking points TM, and in this example, the process of calculating the transition of the height position can be said to be a process of calculating the transition of the multiple tracking points TM.

[0052] The height position of the target leaf TL is an index showing the degree of wilting of the target plant TP, and can also be said to be an index that quantifies the water stress applied to the target plant TP. In the present disclosure, such an index that quantifies water stress is called a water stress index. An example of a water stress index is Leaf Wilt Condition (LWC).

[0053] In step S17, the calculation unit 14 calculates the wilting feature amount based on the transition of the plurality of tracking points TM. In one example, the calculation unit 14 calculates the wilting feature amount based on the transition of the water stress index (the height position of the target leaf TL) calculated as the transition of the plurality of tracking points TM. In this example, the wilting feature amount includes the first to ninth wilting feature amounts. That is, the calculation unit 14 calculates the first to ninth wilting feature amounts as the wilting feature amounts.

[0054] The first to ninth wilting feature amounts will be described in detail with reference to Fig. 11. Fig. 11 is a graph showing an example of the transition of a water stress index. In the graph shown in Fig. 11, the horizontal axis represents time and the vertical axis represents the water stress index. As described above, the water stress index is the height position of the target leaf TL, and therefore Fig. 11 can also be said to show the transition of the height position of the target leaf TL.

[0055] In the example shown in Figure 11, time t1 indicates the time when the target plant TP most recovered from wilting after the irrigation two times before last. Time t2 indicates the time when the previous irrigation was performed. Time t3 indicates the time when the target plant TP most recovered from wilting after the previous irrigation. Time t4 indicates the time when the analysis was performed.

[0056] The water stress index WS1 indicates the water stress index at time t1 and is the maximum value of the water stress index from the time when the second-to-last irrigation was performed to time t2. The water stress index WS2 indicates the water stress index at time t2. The water stress index WS3 indicates the water stress index at time t3 and is the maximum value of the water stress index from time t2 to time t4. The water stress index WS4 indicates the water stress index at time t4.

[0057] The first wilting feature is the absolute value of the water stress index WS4. That is, the first wilting feature is an index indicating the degree to which the target plant TP has wilted at time t4, when the analysis is performed. The second wilting feature is the absolute value of the water stress index WS2. That is, the second wilting feature is an index indicating the degree to which the target plant TP has wilted at time t2, when the previous irrigation was performed.

[0058] The third wilting feature amount is the integrated value of the water stress index from time t2 to time t4. This integrated value is expressed as the sum of the water stress indexes at each analysis time point from time t2 to time t4. The fourth wilting feature amount is the difference value of the water stress index between time t2 and time t4. This difference value is expressed as the difference d1 between the water stress index WS2 and the water stress index WS4.

[0059] The fifth wilting feature is the applied stress for the target plant TP at time t4. In the present disclosure, applied stress refers to the amount of water stress applied to the target plant TP. The amount of water stress is an amount indicating the degree to which a water stress index that has recovered to its maximum due to irrigation has decreased at a certain point in time since the irrigation. Therefore, the applied stress can be said to be an index indicating the amount of wilting of the target leaf TL at a certain point in time. Such applied stress is represented by the difference between the water stress index at the point in time when the water stress index has recovered to its maximum due to irrigation and the water stress index at a certain point in time thereafter. In other words, the fifth wilting feature is represented by the applied stress GS1, which is the difference between the water stress index WS3 and the water stress index WS4. The sixth wilting feature is the applied stress for the target plant TP at time t2. The sixth wilting feature is represented by the applied stress GS2, which is the difference between the water stress index WS1 and the water stress index WS2.

[0060] The seventh wilting feature is an integrated value of the applied stress for the target plant TP at each analysis time point from time t3 to time t4. This integrated value is the sum of the differences between the water stress index WS3 and the water stress index at each analysis time point from time t3 to time t4. The larger this integrated value, the greater the rate at which the target plant TP wilts from time t3 to time t4. In other words, the seventh wilting feature can also be said to be an index that indicates the rate at which the target plant TP wilts. The eighth wilting feature is a difference value of the applied stress between time t2 and time t4. This difference value is represented by the difference between the applied stress GS1 and the applied stress GS2.

[0061] The ninth wilting feature value is the recovery stress at time t3. In the present disclosure, recovery stress refers to the maximum recovery amount of the water stress index due to irrigation. Such recovery stress is represented by the difference between the water stress index at the time of irrigation and the maximum value of the water stress index recovered by the irrigation. In other words, the ninth wilting feature value is represented by the recovery stress RS, which is the difference between the water stress index WS2 and the water stress index WS3.

[0062] Returning to FIG. 3, in step S18, output unit 17 outputs the processing result. In one example, output unit 17 outputs the withering feature as the processing result. In this example, output unit 17 outputs at least one of the first to ninth withering feature. In addition to the withering feature, output unit 17 may output at least one of the following first information, second information, third information, and fourth information.

[0063] The first information is information indicating one or more leaves detected in the target plant. The first information may be, for example, an image in which one or more leaves Ls detected in the image are highlighted, as shown in FIG.

[0064] The second information is information indicating the depth of one or more detected leaves. The second information may be, for example, an image in which depth statistics DP corresponding to each leaf Ls are superimposed on an image as shown in FIG.

[0065] The third information is information indicating which leaf among one or more leaves has been identified as the target object. The third information may be, for example, an image in which the outer edge TLa of the target leaf TL is highlighted in an image as shown in Fig. 8, or a binarized image in which the target leaf TL is represented in white and the rest of the background is represented in black as shown in Fig. 9.

[0066] The fourth information is information indicating the position of each tracking point TM in each of the plurality of second images. The fourth information may be, for example, an image in which each tracking point TM is superimposed on each of the plurality of second images. Alternatively, the fourth information may be a video in which such superimposed images are used as individual frames. In this example, the video may display the trajectory of each tracking point TM in addition to the position of each tracking point TM.

[0067] The output unit 17 may display the processing results on a display device, may store the processing results in a given storage device such as a memory, or may transmit the processing results to another computer system.

[0068] (Estimation of wilt recovery amount) Next, a process for estimating the amount of recovery from withering of a target plant will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of a process for calculating the amount of recovery from withering of a target plant as a process flow S2.

[0069] In step S21, the acquisition unit 11 acquires environmental features and temporal features. In the present disclosure, the environmental features indicate features related to the surrounding environment of the target plant, and the temporal features indicate features related to the elapsed time from a reference time point. In one example, the acquisition unit 11 receives information from the environmental sensor 30 and the irrigation control device 40 to acquire the environmental features and temporal features. In this example, the acquisition unit 11 receives, from the environmental sensor 30, values ​​of temperature, humidity, and illuminance at each analysis time point from time point t2 to time point t4. The acquisition unit 11 receives, from the irrigation control device 40, data indicating the time when the most recent irrigation was performed. The acquired environmental features and temporal features are described in detail below.

[0070] First, the environmental feature quantities will be described. In one example, the environmental feature quantities include a feature quantity related to at least one of temperature, humidity, illuminance, and vapor pressure deficit. In this example, the environmental feature quantities include a first environmental feature quantity to a sixteenth environmental feature quantity.

[0071] The first environmental feature quantity is the absolute value of the temperature at time t4. The second environmental feature quantity is the absolute value of the humidity at time t4. The third environmental feature quantity is the absolute value of the illuminance at time t4. The fourth environmental feature quantity is the absolute value of the vapor deficit at time t4.

[0072] The fifth environmental feature is the absolute value of the temperature at time t2. The sixth environmental feature is the absolute value of the humidity at time t2. The seventh environmental feature is the absolute value of the illuminance at time t2. The eighth environmental feature is the absolute value of the vapor deficit at time t2.

[0073] The ninth environmental feature quantity is a difference value of the air temperature between time t2 and time t4. This difference value is represented by the difference between the air temperature at time t2 and the air temperature at time t4. The tenth environmental feature quantity is a difference value of the humidity between time t2 and time t4. This difference value is represented by the difference between the humidity at time t2 and the humidity at time t4. The eleventh environmental feature quantity is a difference value of the illuminance between time t2 and time t4. This difference value is represented by the difference between the illuminance at time t2 and the illuminance at time t4. The twelfth environmental feature quantity is a difference value of the vapor pressure deficit between time t2 and time t4. This difference value is represented by the difference between the vapor pressure deficit at time t2 and the vapor pressure deficit at time t4.

[0074] The thirteenth environmental feature is the integrated value of the temperature from time t2 to time t4. This integrated value is represented by the sum of the temperatures at each analysis time from time t2 to time t4. The fourteenth environmental feature is the integrated value of the humidity from time t2 to time t4. This integrated value is represented by the sum of the humidity at each analysis time from time t2 to time t4. The fifteenth environmental feature is the integrated value of the illuminance from time t2 to time t4. This integrated value is represented by the sum of the illuminance at each analysis time from time t2 to time t4. The sixteenth environmental feature is the integrated value of the vapor deficit from time t2 to time t4. This integrated value is represented by the sum of the vapor deficit at each analysis time from time t2 to time t4.

[0075] In this example, the acquisition unit 11 acquires the first to third, fifth to seventh, ninth to eleventh, and thirteenth to fifteenth environmental feature quantities based on the values ​​of temperature, humidity, and illuminance at each analysis time point from time point t2 to time point t4 received from the environmental sensor 30. The acquisition unit 11 calculates the saturation deficit at each analysis time point based on the relationship between the corresponding temperature and humidity, and acquires the fourth, eighth, twelfth, and sixteenth environmental feature quantities.

[0076] Next, the temporal feature amount will be described. In one example, the temporal feature amount includes a feature amount relating to at least one of the time point when irrigation was performed and the time point of sunrise. In this example, the temporal feature amount includes a first temporal feature amount and a second temporal feature amount. The first temporal feature amount is the elapsed time from time point t2 to time point t4. In other words, the reference time point for the first temporal feature amount is time point t2 when the previous irrigation was performed. The second temporal feature amount is the elapsed time from time point t4 on the analysis execution date. In other words, the reference time point for the second temporal feature amount is time point t4 on the analysis execution date.

[0077] In this example, the acquisition unit 11 acquires a first temporal feature based on the time of time t2 and the time of time t4 received from the irrigation control device 40. The acquisition unit 11 acquires a second temporal feature based on the time of sunrise and the time of time t4. The acquisition unit 11 may acquire the time of sunrise by accessing a given database or file system, or may acquire the time of sunrise input by a user of the analysis system 10. The acquisition unit 11 may acquire the time of time t4 from a timing device (not shown) in the computer 100 constituting the analysis system 10, or may acquire the time of time t4 input by a user of the analysis system 10.

[0078] In step S22, the estimation unit 15 estimates the amount of recovery from wilting. In one example, the estimation unit 15 estimates the amount of recovery from wilting using an analytical model. The analytical model receives the wilting feature, the environmental feature, and the time feature, and outputs the amount of recovery from wilting when irrigation is performed. The analytical model is, for example, a learning device based on LightGBM. The analytical model may be a learning device based on a one-dimensional convolutional neural network, a learning device based on Transformer, or a learning device based on LSTM (Long Short-Term Memory).

[0079] In this example, the estimation unit 15 estimates the amount of recovery from wilting of the target plant TP by inputting the withering feature calculated in step S17 and the environmental feature and time feature acquired in step S21 into the analysis model. The estimated amount of recovery from wilting indicates the extent to which the target plant TP will recover from wilting if irrigation is performed at time t4.

[0080] In step S23, the output unit 17 outputs the processing result. In one example, the output unit 17 outputs the withering recovery amount as the processing result. The output unit 17 may display the processing result on a display device, may store the processing result in a given storage device such as a memory, or may transmit the processing result to another computer system.

[0081] (Determining whether or not irrigation is necessary for the target plant) Next, a process for determining whether or not irrigation is required for a target plant will be described with reference to Fig. 13. Fig. 13 is a flowchart showing, as a process flow S3, an example of a process for determining whether or not irrigation is required for a target plant.

[0082] In step S31, the acquisition unit 11 acquires the applied stress. The acquisition unit 11 acquires the applied stress for the target plant TP. In one example, the acquisition unit 11 acquires the applied stress GS1 (fifth wilting feature amount) for the target plant TP at time t4. The acquisition unit 11 may acquire the applied stress GS1 directly from the calculation unit 14, may acquire the applied stress GS1 by accessing a given database or file system, or may acquire the applied stress GS1 input by a user of the analysis system 10.

[0083] In step S32, the determination unit 16 determines whether or not irrigation is necessary for the target plant TP. In one example, the determination unit 16 makes this determination by comparing the amount of recovery from wilting with the applied stress GS1. The determination unit 16 determines that irrigation is necessary for the target plant TP if the amount of recovery from wilting is equal to or less than the applied stress GS1, and determines that irrigation is not necessary for the target plant TP if the amount of recovery from wilting is greater than the applied stress GS1.

[0084] If the determination unit 16 determines that irrigation of the target plant TP is necessary (YES in step S32), the process proceeds to step S33. If the determination unit 16 determines that irrigation of the target plant TP is not necessary (NO in step S32), the process ends without performing step S33.

[0085] In step S33, the output unit 17 outputs an irrigation command to the irrigation control device 40. Upon receiving the irrigation command, the irrigation control device 40 irrigates the target plant TP. The output unit 17 may output data indicating the amount of water required for the irrigation to the irrigation control device 40 together with the irrigation command.

[0086] As described above, the analysis system 10 may execute the process flows S1 to S3 consecutively as a series of processes, or may execute the process flows S1 to S3 separately as individual processes.

[0087] [Variations] Various examples of the present disclosure have been described above in detail. However, the present disclosure is not limited to the above examples. Various modifications can be made to the present disclosure without departing from the spirit and scope of the present disclosure.

[0088] In this disclosure, the expression "at least one processor executes a first process, executes a second process, ... executes an nth process" or a corresponding expression is a concept that includes cases where the entity executing the n processes from the first process to the nth process (i.e., the processor) changes midway through. In other words, this expression is a concept that includes both cases where all n processes are executed by the same processor and cases where the processor changes among the n processes according to an arbitrary policy.

[0089] The information processing method executed by at least one processor is not limited to the above examples. For example, some of the steps or processes described above may be omitted, or the steps may be executed in a different order. Furthermore, any two or more of the steps described above may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be executed in addition to the steps described above.

[0090] [Note] As can be seen from the various examples above, the present disclosure includes the following aspects. <Item 1> at least one processor; the at least one processor: acquiring a first image of a target plant including one or more leaves; performing segmentation on the first image to detect the one or more leaves in the first image; identifying a leaf located in the foreground among the one or more detected leaves as a target leaf; setting a plurality of tracking points on the target leaf; acquiring a plurality of second images of the target plant; Detecting positions of the plurality of tracking points in each of the plurality of second images and calculating a transition of the plurality of tracking points; calculating a wilting feature amount indicating a feature amount related to wilting of the target plant based on the transition of the plurality of tracking points; Analysis system. <Item 2> the at least one processor sets the plurality of tracking points inside the outer edge of the target leaf without setting the tracking points on the outer edge of the target leaf; Item 1. The analysis system according to item 1. <Item 3> the at least one processor: shrinking the identified target leaf to generate a reduced target leaf that is entirely located inside the outer edge of the target leaf; setting the plurality of tracking points with respect to the target leaf so that the plurality of tracking points are located within the reduced target leaf; Item 2. The analysis system according to item 2. <Item 4> the at least one processor sets the plurality of tracking points in a grid pattern with respect to the target leaf; The analysis system according to any one of items 1 to 3. <Item 5> the at least one processor: acquiring an environmental feature indicating a feature related to the surrounding environment of the target plant and a time feature indicating a feature related to the elapsed time from a reference time point; an analysis model that receives the wilting feature amount, the environmental feature amount, and the temporal feature amount, and outputs a wilting recovery amount that indicates the extent to which the plant will recover from wilting when irrigated, and inputs the calculated wilting feature amount, the acquired environmental feature amount, and the temporal feature amount into the analysis model, thereby estimating the wilting recovery amount for the target plant; The analysis system according to any one of items 1 to 4. <Item 6> the at least one processor determines whether irrigation of the target plant is necessary based on the amount of wilt recovery of the target plant; Item 5. The analysis system according to item 5. <Item 7> the at least one processor: acquiring an applied stress indicating the amount of water stress applied to the target plant; comparing the amount of recovery from wilting with the applied stress, and determining that irrigation of the target plant is necessary when the amount of recovery from wilting is equal to or less than the applied stress; Item 7. The analysis system according to item 6. <Item 8> 1. An analysis method performed by an analysis system comprising at least one processor, comprising: acquiring a first image of a target plant including one or more leaves; performing segmentation on the first image to detect the one or more leaves in the first image; identifying a leaf located in the foreground among the one or more detected leaves as a target leaf; setting a plurality of tracking points on the target leaf; acquiring a plurality of second images of the target plant; detecting positions of the plurality of tracking points in each of the plurality of second images and calculating a transition of the plurality of tracking points; calculating a wilting feature amount indicating a feature amount related to wilting of the target plant based on the transition of the plurality of tracking points; Including, Analysis method. <Item 9> acquiring a first image of a target plant including one or more leaves; performing segmentation on the first image to detect the one or more leaves in the first image; identifying a leaf located in the foreground among the one or more detected leaves as a target leaf; setting a plurality of tracking points on the target leaf; acquiring a plurality of second images of the target plant; detecting positions of the plurality of tracking points in each of the plurality of second images and calculating a transition of the plurality of tracking points; calculating a wilting feature amount indicating a feature amount related to wilting of the target plant based on the transition of the plurality of tracking points; An analysis program that causes a computer to execute the above.

[0091] According to items 1, 8, and 9, a leaf located in the foreground in the first image is identified as a target leaf, and the transitions of multiple tracking points set on the target leaf are calculated. Then, a wilting feature amount for the target plant is calculated based on the transitions of the multiple tracking points. In this way, by identifying a foreground leaf with a larger transition amount than leaves in the background as the target leaf, wilting can be clearly detected. Furthermore, by calculating the transitions for multiple tracking points set on the target leaf rather than the entire target plant, the positions of the tracking points can be accurately detected even if the brightness values ​​of corresponding tracking points change between multiple second images due to factors such as sunlight. As a result, it is possible to accurately calculate wilting feature amounts for a plant.

[0092] If a tracking point is set on the outer edge of a target leaf, the tracking point may correspond to an object other than the target leaf, depending on the detection accuracy of the segmentation. If the tracking point no longer corresponds to the target leaf, there is a risk that the wilting of the target plant cannot be accurately detected. According to item 2, a tracking point is not set on the outer edge of the target leaf, but multiple tracking points are set inside the outer edge, so that it is possible to prevent each of the set tracking points from corresponding to an object other than the target leaf. As a result, it is possible to more accurately calculate wilting features for a plant.

[0093] According to item 3, it is possible to easily and reliably set a plurality of tracking points inside the outer edge of the target leaf.

[0094] According to item 4, the positions of the multiple tracking points can be accurately detected in each of the multiple second images.

[0095] For example, when calculating the positions of multiple tracking points using CoTracker, CoTracker uses information on a certain tracking point as viewed from two or more tracking points adjacent to that tracking point. In this case, for example, if two or more tracking points are concentrated in a certain area, the direction in which the two or more tracking points view the certain tracking point may be biased in a particular direction. If such a bias occurs, the accuracy of calculating the positions of the multiple tracking points may decrease in processing using CoTracker. By setting multiple tracking points in a grid pattern on the target leaf, it is possible to prevent the tracking points from being concentrated in a certain area, and it is possible to prevent a decrease in the accuracy of calculating the positions of the multiple tracking points.

[0096] According to item 5, the accurately calculated wilting feature amount is input to the analysis model, and the resulting wilting recovery amount is also accurately estimated. In other words, the wilting recovery amount of a plant can be accurately estimated.

[0097] When a person determines whether or not a target plant needs irrigation, the criteria for this determination depend heavily on the person's level of skill. For example, a person with low skill may not be able to properly determine whether or not irrigation is necessary. According to item 6, the amount of recovery from wilting, which is an objective indicator, is used as the criterion for the determination, so that it is possible to properly determine whether or not the target plant needs irrigation without relying on the person's level of skill.

[0098] According to item 7, when the amount of recovery from wilting is equal to or less than the applied stress, it is determined that irrigation is necessary, so that excessive irrigation of the target plant can be prevented. [Explanation of symbols]

[0099] 10...analysis system, 11...acquisition unit, 12...detection unit, 13...identification unit, 14...calculation unit, 15...estimation unit, 16...determination unit, 17...output unit, 20...imaging device, 30...environmental sensor, 40...irrigation control device.

Claims

1. at least one processor; the at least one processor: acquiring a first image of a target plant including one or more leaves; performing segmentation on the first image to detect the one or more leaves in the first image; identifying a leaf located in the foreground among the one or more detected leaves as a target leaf; setting a plurality of tracking points on the target leaf; acquiring a plurality of second images of the target plant; Detecting positions of the plurality of tracking points in each of the plurality of second images and calculating a transition of the plurality of tracking points; calculating a wilting feature amount indicating a feature amount related to wilting of the target plant based on the transition of the plurality of tracking points; Analysis system.

2. the at least one processor sets the plurality of tracking points inside the outer edge of the target leaf without setting the tracking points on the outer edge of the target leaf; The analysis system according to claim 1 .

3. the at least one processor: shrinking the identified target leaf to generate a reduced target leaf that is entirely located inside the outer edge of the target leaf; setting the plurality of tracking points with respect to the target leaf so that the plurality of tracking points are located within the reduced target leaf; The analysis system according to claim 2 .

4. the at least one processor sets the plurality of tracking points in a grid pattern with respect to the target leaf; The analysis system according to any one of claims 1 to 3.

5. the at least one processor: acquiring an environmental feature indicating a feature related to the surrounding environment of the target plant and a time feature indicating a feature related to the elapsed time from a reference time point; an analysis model that receives the wilting feature amount, the environmental feature amount, and the temporal feature amount, and outputs a wilting recovery amount that indicates the extent to which the plant will recover from wilting when irrigated, and inputs the calculated wilting feature amount, the acquired environmental feature amount, and the temporal feature amount into the analysis model, thereby estimating the wilting recovery amount for the target plant; The analysis system according to any one of claims 1 to 3.

6. the at least one processor determines whether irrigation of the target plant is necessary based on the amount of wilt recovery of the target plant; The analysis system according to claim 5 .

7. the at least one processor: acquiring an applied stress indicating the amount of water stress applied to the target plant; comparing the amount of recovery from wilting with the applied stress, and determining that irrigation of the target plant is necessary when the amount of recovery from wilting is equal to or less than the applied stress; The analysis system according to claim 6 .

8. 1. An analysis method performed by an analysis system comprising at least one processor, comprising: acquiring a first image of a target plant including one or more leaves; performing segmentation on the first image to detect the one or more leaves in the first image; identifying a leaf located in the foreground among the one or more detected leaves as a target leaf; setting a plurality of tracking points on the target leaf; acquiring a plurality of second images of the target plant; detecting positions of the plurality of tracking points in each of the plurality of second images and calculating a transition of the plurality of tracking points; calculating a wilting feature amount indicating a feature amount related to wilting of the target plant based on the transition of the plurality of tracking points; Including, Analysis method.

9. acquiring a first image of a target plant including one or more leaves; performing segmentation on the first image to detect the one or more leaves in the first image; identifying a leaf located in the foreground among the one or more detected leaves as a target leaf; setting a plurality of tracking points on the target leaf; acquiring a plurality of second images of the target plant; detecting positions of the plurality of tracking points in each of the plurality of second images and calculating a transition of the plurality of tracking points; calculating a wilting feature amount indicating a feature amount related to wilting of the target plant based on the transition of the plurality of tracking points; An analysis program that causes a computer to execute the above.

Images