Learning method, leaf condition identification device, and program

Abstract

This training method comprises: a weight determination step for determining weights for leaves included in a captured image; and a first training step for training a leaf detection model for detecting leaves from the captured image, on the basis of the weights determined in the weight determination step such that leaves with high weights are more likely to be detected than leaves with low weights.

Description

[Technical Field]

[0001] The present invention relates to a technique for detecting leaves and identifying their state. [Background technology]

[0002] Because diseases and insect damage can cause significant damage to agricultural production, it is extremely important to detect them early and take measures to prevent them. However, visual detection of diseases and insect damage is difficult and time-consuming unless done by an agricultural expert (someone with specialized knowledge of agriculture).

[0003] Therefore, systems that automatically detect disease and insect damage have been proposed. Non-Patent Document 1 discloses a system that detects (extracts) leaves from a captured image and identifies the state of the detected leaves. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Bulletin of the Graduate School of Science and Engineering, Hosei University, Vol. 58, pp. 1-4, Published March 31, 2017 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology disclosed in Non-Patent Document 1, when a leaf that is not suitable for identifying its state (for example, a leaf that looks long and thin, a leaf that looks small, a leaf that is partially hidden by other leaves, an out-of-focus leaf, a dark leaf, etc.) is detected, an incorrect identification result is obtained for that leaf, resulting in a decrease in overall identification accuracy. Furthermore, when the overall identification accuracy is low, work (effort) such as having an agricultural expert check the identification results is required.

[0006] The present invention has been made in consideration of the above-mentioned situation, and aims to provide a method that enables leaf detection to be performed favorably, and thereby enables subsequent processes such as identifying the state of leaves to be performed with high accuracy. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention employs the following method.

[0008] A first aspect of the present invention provides a learning method comprising: a weight determination step for determining weights for leaves included in a captured image; and a first learning step for training a leaf detection model for detecting leaves from a captured image based on the weights determined in the weight determination step, so that leaves with large weights are more easily detected than leaves with small weights.

[0009] According to the above-described method, weights are determined for leaves, and a leaf detection model is trained so that leaves with large weights are more easily detected than leaves with small weights. This allows for optimal leaf detection, and ultimately enables subsequent processes such as identifying leaf states to be performed with high accuracy. For example, by determining large weights for leaves suitable for subsequent processes and small weights (or no weights) for leaves unsuitable for subsequent processes, leaves suitable for subsequent processes are more easily detected than leaves unsuitable for subsequent processes.

[0010] In the weight determination step, weights may be determined based on knowledge about agriculture. For example, in the weight determination step, weights may be determined based on knowledge obtained from at least one of the perspective of an agricultural expert and agricultural experience. In this way, a large weight can be determined for leaves that are suitable for subsequent processes, and a small weight (or no weight) can be determined for leaves that are not suitable for subsequent processes.

[0011] The weight determination step may determine the weight of a leaf based on at least one of the leaf's shape, size, and position. For example, a leaf that appears elongated when viewed from an angle or partially obscured by other leaves may be unlikely to be suitable for subsequent processes, as its condition cannot be accurately identified. Therefore, the weight determination step may determine a leaf whose bounding box is closer to a square, as a larger weight. A leaf that is underdeveloped or appears small because it is partially obscured by other leaves may be unlikely to be suitable for subsequent processes, as its condition cannot be accurately identified. Therefore, the weight determination step may determine a leaf whose size is larger, as a larger weight. Furthermore, because humidity is higher closer to the ground, leaves closer to the ground are more susceptible to fungal disease than leaves farther from the ground. Therefore, the weight determination step may determine a leaf whose size is greater, as its condition is younger (upper leaves) and therefore more susceptible to pests. The bounding box of a leaf is a rectangular frame that surrounds the leaf, and may be, for example, a rectangular frame that circumscribes the leaf.

[0012] The leaf detection model may be an inference model using Mask R-CNN or Faster R-CNN. In the first learning step, the value of the loss function may be reduced by a larger reduction amount as the weight increases. In this way, the allowable range of the leaves is adjusted so that the allowable range based on leaves with large weights is wide and the allowable range based on leaves with small weights is narrow. As a result, leaves with large weights (leaves included in the allowable range based on leaves with large weights) are more likely to be detected than leaves with small weights (leaves included in the allowable range based on leaves with small weights).

[0013] The method may further include a second learning step in which a leaf condition identification model for identifying the state of leaves is trained using the detection results of the leaf detection model trained in the first learning step. In this way, a leaf detection model that can suitably detect leaves can be obtained, and a leaf condition identification model that can identify leaves with high accuracy can be obtained. The leaf condition identification model may identify whether or not a leaf is infected with a pest.

[0014] A second aspect of the present invention provides a leaf condition identification device comprising: an acquisition means for acquiring a captured image; a detection means for detecting leaves from the captured image acquired by the acquisition means using the leaf detection model trained by the above-described learning method; and an identification means for identifying the state of the leaves detected by the detection means using a leaf condition identification model that identifies the state of the leaves. With this configuration, leaves are detected using the leaf detection model trained by the above-described learning method, so that the state of the leaves can be identified with high accuracy.

[0015] The present invention can be understood as a learning device, leaf condition identification device, learning system, or leaf condition identification system having at least a part of the above configuration or function. The present invention can also be understood as a learning method, leaf condition identification method, learning system control method, or leaf condition identification system control method, all of which include at least a part of the above processing, or a program for causing a computer to execute these methods, or a computer-readable recording medium on which such a program is non-temporarily recorded. The above configurations and processing can be combined with each other to constitute the present invention as long as no technical contradictions arise. [Effects of the Invention]

[0016] According to the present invention, leaf detection can be performed favorably, and consequently subsequent processes such as identification of leaf state can be performed with high accuracy. [Brief explanation of the drawings]

[0017] [Figure 1]FIG. 1A is a flowchart showing an example of a learning method to which the present invention is applied, and FIG. 1B is a block diagram showing an example of the configuration of a leaf condition identification device to which the present invention is applied. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a leaf condition identification system according to this embodiment. [Figure 3] FIG. 3A is a flowchart showing an example of the processing flow of a PC (leaf state identification device) in the learning phase, and FIG. 3B is a flowchart showing an example of the processing flow of a PC in the inference phase after the learning phase. [Figure 4] FIG. 4A is a schematic diagram showing an example of a captured image for learning, and FIGS. 4B and 4C are schematic diagrams showing examples of bounding boxes and the like. [Figure 5] FIG. 5 is a schematic diagram illustrating an example of a leaf detection model using Mask R-CNN. [Figure 6] Fig. 6A shows the detection result (leaf detection result) before narrowing down in the comparative example, Fig. 6B shows the detection result after narrowing down in the comparative example, and Fig. 6C shows the detection result of this embodiment. [Figure 7] Fig. 7A shows the detection result (leaf detection result) before narrowing down in the comparative example, Fig. 7B shows the detection result after narrowing down in the comparative example, and Fig. 7C shows the detection result of this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] <Application example> An application example of the present invention will be described.

[0019] A device (system) has been proposed that detects (extracts) leaves from a captured image and identifies the state of the detected leaves. When such a device detects a leaf that is not suitable for leaf state identification (e.g., a leaf that appears elongated, a leaf that appears small, a leaf that is partially hidden by other leaves, an out-of-focus leaf, a dark leaf, etc.), it will incorrectly identify the leaf, resulting in a decrease in overall identification accuracy. If the overall identification accuracy is low, it will require the work (effort) of an agricultural expert (a person with specialized agricultural knowledge) to confirm the identification results.

[0020] 1A is a flowchart showing an example of a learning method to which the present invention is applied. In step S101, weights are determined for leaves included in a captured image. In step S102, based on the weights determined in step S101, a leaf detection model is trained to detect leaves from the captured image so that leaves with higher weights are more easily detected than leaves with lower weights. Step S101 is an example of a weight determination step, and step S102 is an example of a first learning step. The captured image may or may not be a wide-area image with a wide angle of view.

[0021] According to the above-described method, weights are determined for leaves, and a leaf detection model is trained so that leaves with large weights are more easily detected than leaves with small weights. This allows for optimal leaf detection, and ultimately enables subsequent processes such as identifying leaf states to be performed with high accuracy. For example, by determining large weights for leaves suitable for subsequent processes and small weights (or no weights) for leaves unsuitable for subsequent processes, leaves suitable for subsequent processes are more easily detected than leaves unsuitable for subsequent processes.

[0022] In step S101, weights may be determined based on knowledge about agriculture. For example, in step S101, weights may be determined based on knowledge obtained from at least one of the gaze of an agricultural expert and agricultural experience. In this way, a large weight can be determined for leaves that are suitable for subsequent processes, and a small weight (or no weight) can be determined for leaves that are not suitable for subsequent processes. Information about the gaze may be acquired using existing gaze detection technology.

[0023] FIG. 1B is a block diagram showing an example configuration of a leaf condition identification device 110 to which the present invention is applied. The leaf condition identification device 110 includes an acquisition unit 111, a detection unit 112, and a classification unit 113. The acquisition unit 111 acquires a captured image. The detection unit 112 detects leaves from the captured image acquired by the acquisition unit 111 using a leaf detection model trained by the above-described learning method. The classification unit 113 classifies the state of the leaves detected by the detection unit 112 using a leaf condition identification model that classifies the state of the leaves. The acquisition unit 111 is an example of an acquisition means, the detection unit 112 is an example of a detection means, and the classification unit 113 is an example of a classification means. With this configuration, leaves are detected using the leaf detection model trained by the above-described learning method, so that the state of the leaves can be classified with high accuracy.

[0024] <Embodiment> An embodiment of the present invention will be described.

[0025] (composition) FIG. 2 is a block diagram showing an example of the configuration of a leaf condition identification system according to this embodiment. The leaf condition identification system has a camera 11 (imaging device), a PC 200 (personal computer; leaf condition identification device), and a display 12 (display device). The camera 11 and the PC 200 are connected to each other by wire or wirelessly, and the PC 200 and the display 12 are connected to each other by wire or wirelessly. The camera 11 captures images of a field or the like and outputs the captured images to the PC 200. The PC 200 detects leaves from the captured images of the camera 11 and identifies the state of the detected leaves. The PC 200 then displays the identification results and the like on the display 12. The display 12 displays various images and information.

[0026] The camera 11 may or may not be fixed. There is no particular limitation on the positional relationship between the camera 11, the PC 200, and the display 12. For example, the camera 11, the PC 200, and the display 12 may or may not be installed in the same room (e.g., a greenhouse).

[0027] In addition, in this embodiment, the camera 11 and the display 12 are separate devices from the PC 200, but at least one of the camera 11 and the display 12 may be part of the PC 200. The PC 200 (leaf condition identification device) may be a cloud-based computer. At least some of the functions of the camera 11, the PC 200, and the display 12 may be implemented by various terminals such as a smartphone or a tablet terminal.

[0028] The PC 200 includes an input unit 210, a control unit 220, a storage unit 230, and an output unit 240.

[0029] The input unit 210 acquires the captured image from the camera 11. For example, the input unit 210 is an input terminal. The input unit 210 is an example of an acquisition unit.

[0030] The control unit 220 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and controls each component, performs various information processing, etc. In this embodiment, the control unit 220 detects leaves from the captured image of the camera 11 (captured image acquired by the input unit 210), and identifies the state of the detected leaves.

[0031] The storage unit 230 stores programs executed by the control unit 220 and various data used by the control unit 220. For example, the storage unit 230 is an auxiliary storage device such as a hard disk drive or a solid state drive.

[0032] The output unit 240 outputs the identification result of the control unit 220 and the like to the display 12. As a result, the identification result and the like are displayed on the display 12. For example, the output unit 240 is an output terminal.

[0033] The control unit 220 will be described in more detail below. The control unit 220 includes an annotation unit 221, a weight determination unit 222, a detection unit 223, and a recognition unit 224.

[0034] The annotation unit 221 performs annotation on the image captured by the camera 11. The weight determination unit 222 determines weights for the leaves included in the image captured by the camera 11. The detection unit 223 uses a leaf detection model to detect leaves from the image captured by the camera 11. The classification unit 113 uses a leaf state classification model to classify the state of the leaves detected by the detection unit 112. Details of these processes will be described later. The detection unit 112 is an example of a detection means, and the classification unit 113 is an example of a classification means.

[0035] (Learning phase process flow) 3A is a flowchart showing an example of the processing flow of the PC 200 in the learning phase. In the learning phase, a leaf detection model is learned. In this embodiment, a leaf state identification model is also learned.

[0036] First, the input unit 210 acquires a captured image for learning (step S301). The captured image for learning may or may not be an image captured by the camera 11. Fig. 4A shows an example of a captured image for learning. The captured image in Fig. 4A shows one plant, but a captured image may also show multiple plants.

[0037] Next, the annotation unit 221 performs annotation on the captured image acquired in step S301 (step S302). The annotation is a process of setting a true value (correct answer) in learning, and the true value is specified based on information specified (input) by the worker.

[0038] For example, the worker specifies the outline of a leaf in a captured image. In response to the specification of the outline, the annotation unit 221 sets a leaf mask in the area surrounded by the outline. Then, as shown in FIG. 4B, the annotation unit 221 automatically sets a bounding box, which is a rectangular frame surrounding the leaf mask (leaf). For example, the annotation unit 221 sets a rectangular frame circumscribing the leaf mask (leaf) as the bounding box.

[0039] It is preferable for an operator to select only leaves suitable for the subsequent process (in this embodiment, for identifying the leaf condition) and specify their outlines, but it is difficult for anyone other than an agricultural expert to determine whether a leaf is suitable for the subsequent process, and the operator specifying the outlines is not necessarily an agricultural expert. For this reason, in annotation, leaf masks and bounding boxes may be set for leaves that are not suitable for the subsequent process.

[0040] In this embodiment, the leaf condition is identified by identifying whether the leaves are infected with pests (diseases or harmful insects) (whether the leaves are healthy or not). For this purpose, an operator inputs information on whether the leaves are infected with pests or diseases, and the annotation unit 221 sets the information. The information on whether the leaves are infected with pests or diseases is input by an agricultural expert. Note that the type of disease or pest may also be identified when identifying the leaf condition.

[0041] Returning to the description of FIG. 3A, after step S302, the weight determination unit 222 determines a weight for each leaf included in the captured image acquired in step S301 based on the information set in step S302 (step S303). In this embodiment, the weight determination unit 222 determines the weight for each leaf based on at least one of the shape, size, and position of the leaf. Step S302 is an example of a weight determination step.

[0042] Leaves that appear elongated when viewed from an angle or partially hidden by other leaves are unlikely to be suitable for subsequent processing, as their state cannot be identified with high accuracy. Therefore, the weight determination unit 222 may determine a larger weight for a leaf whose bounding box is closer to a square. For example, the weight determination unit 222 determines a weight ω1 from the width w and height h of the bounding box shown in FIG. 4C using the following equations 1-1 and 1-2. When w / h≦1: ω1=w / h (Eq. 1-1) When w / h>1: ω1=h / w (Eq. 1-2)

[0043] Leaves that appear small because they are underdeveloped or partially hidden by other leaves are unlikely to be suitable for subsequent processing, as their state cannot be identified with high accuracy. Therefore, the weight determination unit 222 may assign a larger weight to a leaf as the leaf size increases. For example, the weight determination unit 222 determines the weight ω2 using the following equation 2 based on the width W (number of pixels in the horizontal direction) and height H (number of pixels in the vertical direction) of the captured image shown in FIG. 4B and the number of pixels s of the leaf mask shown in FIG. 4C. W × H is the total number of pixels in the captured image. ω2=s / (W×H) (Formula 2)

[0044] The weight determination unit 222 may determine the weight ω2 using the following equations (2-1) to (2-3). The threshold values Th1 and Th2 are not particularly limited. For example, when W = 1200 and H = 1000, Th1 = 5000 and Th2 = 10000 may be used. Note that the number of levels of the weight ω2 may be more or less than three levels. When s ≤ Th1: ω2 = 0.1 ··· (Equation 2-1) When Th1 < s ≤ Th2: ω2 = 0.5 ··· (Equation 2-2) When Th2 < s: ω2 = 0.9 ··· (Equation 2-3)

[0045] Since the humidity is higher closer to the ground, leaf diseases are more likely to occur in the leaves closer to the ground than in the leaves farther from the ground. Therefore, the weight determination unit 222 may determine a larger weight for the leaf the closer it is to the ground. For example, when the captured image is an image of a plant captured from the side, the weight determination unit 222 determines the weight ω3 using equations (3-1) to (3-3) from the vertical position c_y (vertical direction position) of the center of the bounding box. The threshold values Th3 and Th4 are not particularly limited. For example, the threshold value Th3 corresponds to the vertical position where the vertical distance from the lower end of the captured image (vertical direction distance) is H / 3, and the threshold value Th4 corresponds to the vertical position where the vertical distance from the lower end of the captured image is (2 / 3)×H. Here, it is assumed that the value (coordinate value) of the vertical position increases from the lower end to the upper end of the captured image. Note that the number of levels of the weight ω3 may be more or less than three levels. When c_y ≤ Th3: ω3 = 0.9 ··· (Equation 3-1) When Th3 < c_y ≤ Th4: ω3 = 0.5 ··· (Equation 3-2) When Th4 < c_y: ω3 = 0.1 ··· (Equation 3-3)

[0046] When the captured image is an image captured so as to overlook a field, leaves close to the ground may be located at the upper part of the captured image. In such a case, as shown in FIG. 4B, a bounding box of the entire plant may be set, and instead of the vertical distance from the lower end of the captured image, the vertical distance from the lower end of the bounding box of the entire plant may be regarded as the distance from the ground.

[0047] The weight determination unit 222 may determine any one of the above-described weights ω1 to ω3, or may determine the final weight ω by combining two or three of the weights ω1 to ω3. For example, the weight determination unit 222 may determine ω1×ω2, ω1×ω3, ω2×ω3, or ω1×ω2×ω3 as the final weight ω. Further, the weight determination unit 222 may determine the weight ω only for the leaves that satisfy a predetermined condition (for the leaves that do not satisfy the predetermined condition, ω = 0 may be determined). The predetermined condition may include the condition of 0.75 < w / h < 1.3. When W = 1200 and H = 1000, the predetermined condition may include the condition of s > 10000.

[0048] Note that the method for determining the weight is not limited to the above method. For example, since young leaves (upper leaves) are more likely to be invaded by pests, the weight determination unit 222 may determine a larger weight for the leaves as the distance of the leaves from the ground is farther. The weight determination unit 222 may increase the weight of the leaves with proper exposure (proper brightness) or increase the weight of the clear leaves based on the luminance value or sharpness of the leaf image.

[0049] Returning to the description of FIG. 3A. Next to step S303, the control unit 220 performs learning of the leaf detection model included in the detection unit 223 so that the leaves with larger weights are more easily detected than the leaves with smaller weights based on the weights determined in step S303 (step S304). Step S304 is an example of the first learning step. By performing learning of the leaf detection model so that the leaves with larger weights are more easily detected than the leaves with smaller weights, leaf detection can be suitably performed, and as a result, subsequent processes such as discrimination of leaf states can be performed with high accuracy.

[0050] Various methods such as Mask R-CNN and Faster R-CNN can be used for the leaf detection model. In this embodiment, the leaf detection model is an inference model (learning model) using Mask R-CNN, as shown in Fig. 5. Since Mask R-CNN is a well-known method, an overview will be provided below.

[0051] In the leaf detection model (Mask R-CNN), first, a convolutional neural network (CNN) extracts features from a captured image and generates a feature map. Next, an RPN detects candidate regions that are candidates for leaf regions (bounding boxes) from the feature map. A fixed-size feature map is then obtained by RoI Align, and after processing in a fully connected layer (not shown), inference results for each candidate region (probability that the candidate region is a leaf region (correct answer probability), the position of the candidate region, the size of the candidate region, leaf mask candidates, etc.) are obtained. After training the leaf detection model, the detection unit 223 detects candidate regions with a correct answer probability equal to or greater than a predetermined threshold as leaf bounding boxes.

[0052] When training the leaf detection model, the control unit 220 compares the inference result with the true value (correct answer) for each candidate region to calculate the loss L. The loss L is calculated using, for example, the following equation 4 (loss function). The loss Lcls is the classification loss of the bounding box, and becomes smaller when the candidate region matches the correct bounding box. The loss Lloc is the regression loss of the bounding box, and becomes smaller the closer the candidate region is to the correct bounding box. The loss Lmask is the matching loss of the leaf mask, and becomes smaller the closer the leaf mask candidate is to the correct leaf mask. The coefficients f(ω) and g(ω) depend on the weight ω determined by the weight determination unit 222, and are, for example, f(ω) = g(ω) = e -ωIn this embodiment, the weight determination unit 222 determines the weight of a leaf based on at least one of the shape, size, and position of the leaf. Since the losses associated with the shape, size, and position of the leaf are the loss Lloc and the loss Lmask, the loss Lloc and the loss Lmask are multiplied by the coefficient f(ω) and the coefficient g(ω), respectively. L=Lcls+Lloc×f(ω)+Lmask×g(ω) (Formula 4)

[0053] The control unit 220 then updates the RPN based on the loss L for each candidate region. The coefficients f(ω) and g(ω) become smaller as the weight ω increases. Therefore, the value of the loss function (L = Lcls + Lloc + Lmask) that does not consider the weight ω is reduced by a larger amount as the weight ω increases. By updating the RPN based on the reduced loss L in this way, the allowable range of the leaves is adjusted so that the allowable range based on leaves with a large weight ω is wider and the allowable range based on leaves with a small weight ω is narrower. As a result, candidate regions of leaves with a large weight ω (leaves included in the allowable range based on leaves with a large weight ω) are more likely to be detected than candidate regions of leaves with a small weight ω (leaves included in the allowable range based on leaves with a small weight ω). Furthermore, the control unit 220 updates the entire leaf detection model based on the sum (average) of the losses L for each candidate region.

[0054] Although an example of reducing the candidate regions of leaves with a small weight ω has been described, it is also possible to use another method to make leaves with a large weight ω more likely to be detected than leaves with a small weight ω. For example, the leaf detection model may be trained so as to reduce the probability of correct detection for candidate regions of leaves with a small weight ω.

[0055] Returning to the description of FIG. 3A, after step S304, the control unit 220 uses the detection results of the detection unit 223 having the leaf detection model trained in step S304 to train the leaf state discrimination model of the discrimination unit 224 (step S305). Step S305 is an example of a second learning step. By using the detection results of the detection unit 223 having the trained leaf detection model, it is possible to obtain a leaf state discrimination model that can discriminate leaves with high accuracy. Various techniques can also be used for the leaf state discrimination model.

[0056] (Inference phase processing flow) 3B is a flowchart showing an example of a processing flow of the PC 200 in the inference phase after the learning phase. First, the input unit 210 acquires a captured image from the camera 11 (step S311). Next, the detection unit 223 detects leaves from the captured image acquired in step S311 using the learned leaf detection model (step S312). Next, the identification unit 113 identifies the state of the leaves detected in step S312 using the learned leaf state identification model (step S313). Next, the output unit 240 outputs and displays the identification result of step S313 on the display 12 (step S314).

[0057] (effect) The effects of this embodiment will be described. In this embodiment, weights are determined for leaves, and a leaf detection model is trained so that leaves with large weights are more likely to be detected than leaves with small weights. As another method (comparative example), a method of narrowing down leaf detection results using a predetermined threshold can be considered. However, such a method does not provide detection results (leaf detection results) as favorable as those of the method of this embodiment.

[0058] Figures 6A and 6B show the detection results of a comparative example. Figure 6A shows the detection result before narrowing down. Since weights were not taken into account in the learning process, all leaves were detected. Furthermore, fruits were falsely detected. Figure 6B shows the result of narrowing down using a size threshold to remove small leaves. In Figure 6B, small leaves have been removed from the detection result, but the fruits are large and therefore not removed.

[0059] 6C shows the detection results of this embodiment. By taking weights into consideration in the learning process (learning was performed by increasing the weight of leaves that better represent leaf characteristics), small leaves and fruits were not detected, and only large leaves suitable for subsequent processing were detected.

[0060] Figures 7A and 7B show the detection results of a comparative example. Figure 7A shows the detection result before narrowing down. Since weights were not taken into account in the learning process, all leaves were detected. Bright and clear leaves were also detected. Such leaves are likely to be suitable for subsequent processing even if they are small (for example, leaves whose leaf state can be identified with high accuracy). Figure 7B shows the results of narrowing down using a size threshold to remove small leaves. In Figure 7B, bright and clear leaves that should be left as leaves suitable for subsequent processing are removed because they are small.

[0061] 7C shows the detection results of this embodiment. Although small leaves are difficult to detect due to the weighting factor used in the learning process, bright and clear leaves can be detected because they clearly represent the characteristics of the leaf even if they are small.

[0062] (summary) As described above, according to this embodiment, weights are determined for leaves, and the leaf detection model is trained so that leaves with large weights are more easily detected than leaves with small weights. This allows for optimal leaf detection, and ultimately allows subsequent processes such as identifying leaf states to be performed with high accuracy.

[0063] <Other> The above-described embodiment merely exemplifies the configuration of the present invention, and the present invention is not limited to the specific embodiment described above, and various modifications are possible within the scope of the technical concept thereof.

[0064] <Appendix 1> a weight determination step (S101, S303) for determining weights for leaves included in the captured image; a first learning step (S102, S304) of learning a leaf detection model that detects leaves from the captured image based on the weights determined in the weight determination step so that leaves with large weights are more easily detected than leaves with small weights; have A learning method characterized by:

[0065] <Appendix 2> Acquisition means (111, 210) for acquiring a captured image; a detection means (112, 223) for detecting leaves from the captured image acquired by the acquisition means using the leaf detection model trained by the training method according to any one of claims 1 to 9; a classification means (113, 224) for classifying the state of the leaves detected by the detection means using a leaf state classification model for classifying the state of the leaves; have A leaf condition identification device (110, 200) characterized by: [Explanation of symbols]

[0066] 110: Leaf condition identification device 111: Acquisition unit 112: Detection unit 113: Identification unit 200: PC (information processing device) 210: Input unit 220: Control unit 230: Storage unit 240: Output unit 221: Annotation unit 222: Weight determination unit 223: Detection unit 224: Classification unit 11: Camera 12: Display

Claims

1. a weight determination step of determining a weight of a leaf based on a position of the leaf included in the captured image; a first learning step of learning a leaf detection model that detects leaves from the captured image based on the weights determined in the weight determining step so that leaves with large weights are more easily detected than leaves with small weights; and In the weight determination step, a larger weight is determined for a leaf as the leaf is closer to the ground. A learning method characterized by:

2. a weight determination step of determining a weight of a leaf based on a position of the leaf included in the captured image; a first learning step of learning a leaf detection model that detects leaves from the captured image based on the weights determined in the weight determining step so that leaves with large weights are more easily detected than leaves with small weights; and In the weight determination step, a larger weight is determined for a leaf as the leaf is farther from the ground. A learning method characterized by:

3. In the weight determination step, weights are determined based on agricultural knowledge.

3. The learning method according to claim 1 or 2.

4. In the weight determination step, weights are determined based on knowledge obtained from at least one of the agricultural expert's perspective and agricultural experience.

4. The learning method according to claim 3.

5. In the weight determination step, a weight of the leaf is determined based on the shape of the leaf, and the closer the shape of the bounding box of the leaf is to a square, the larger the weight is determined.

5. The learning method according to claim 4.

6. In the weight determination step, a weight of the leaf is determined based on the size of the leaf, and a larger weight is determined for the leaf as the size of the leaf increases.

6. The learning method according to claim 4 or 5.

7. The leaf detection model is an inference model using Mask R-CNN or Faster R-CNN.

7. The learning method according to claim 1, wherein:

8. In the first learning step, the value of the loss function is reduced by a larger reduction amount as the weight is larger.

8. The learning method according to claim 7.

9. a second learning step of learning a leaf state identification model that identifies leaf states using the detection results of the leaf detection model learned in the first learning step; Further having 9. The learning method according to claim 1, wherein the learning method is a learning method for learning a plurality of words.

10. The leaf condition discrimination model discriminates whether a leaf is infected with a pest or not.

10. The learning method according to claim 9.

11. an acquisition means for acquiring a captured image; a detection means for detecting leaves from the captured image acquired by the acquisition means using the leaf detection model trained by the training method according to any one of claims 1 to 10; a discrimination means for discriminating the state of the leaves detected by the detection means using a leaf state discrimination model for discriminating the state of the leaves; have A leaf condition identification device characterized by:

12. A program for causing a computer to execute each step of the learning method according to any one of claims 1 to 10.

Images