Learning method, leaf state recognition device, and computer program product

Abstract

The learning method includes a weight determination step of determining a weight for a leaf included in a captured image, and a first learning step of performing learning of a leaf detection model that detects a leaf from a captured image, based on the weight determined in the weight determination step, so that a leaf having a large weight is more likely to be detected than a leaf having a small weight.

Description

Technical Field

[0001] This invention relates to a technology for detecting leaves and identifying leaf conditions. Background Technology

[0002] Since diseases and pests can cause significant damage to agricultural production, it is crucial to detect them as early as possible and take appropriate countermeasures. However, when detecting diseases and pests through observation, it is difficult to do so early without the expertise of an agricultural specialist, and it is also labor-intensive.

[0003] Therefore, a system for automatically detecting diseases and pests has been proposed. Non-patent literature 1 discloses a system for detecting (extracting) leaves from captured images and identifying the state of the detected leaves.

[0004] Existing technical documents

[0005] Non-patent literature

[0006] Non-patent document 1: Journal of the Graduate School of Science and Engineering, Hosei University, Vol. 58, pp. 1-4, published March 31, 2017. Summary of the Invention

[0007] The technical problem that the invention aims to solve

[0008] However, in the technology disclosed in Non-Patent Document 1, when a leaf unsuitable for leaf condition recognition is detected (e.g., a leaf that appears slender, a leaf that appears small, a leaf partially hidden among other leaves, a blurry, out-of-focus leaf, a dark leaf, etc.), an incorrect recognition result is obtained for that leaf, resulting in a decrease in overall recognition accuracy. Moreover, when the overall recognition accuracy is low, manual labor, such as that required from agricultural experts, is needed to verify the recognition results.

[0009] The present invention was made in view of the above-mentioned actual situation, and its object is to provide a method that enables proper leaf detection and subsequent processes such as leaf state identification with high accuracy.

[0010] Solutions for solving technical problems

[0011] To achieve the above objectives, the present invention employs the following method.

[0012] A first aspect of the present invention provides a learning method, characterized by comprising: a weight determination step for determining weights for leaves included in a captured image; and a first learning step for learning a leaf detection model for detecting leaves from the captured image based on the weights determined in the weight determination step, such that leaves with larger weights are easier to detect than leaves with smaller weights.

[0013] Based on the above method, weights are assigned to leaves, and a leaf detection model is trained to ensure that leaves with higher weights are easier to detect than those with lower weights. This allows for appropriate leaf detection, leading to more accurate leaf state identification and other subsequent processes. For example, if leaves suitable for subsequent processes are assigned higher weights, while leaves unsuitable for subsequent processes are assigned lower weights (or have uncertain weights), then leaves suitable for subsequent processes are easier to detect than those unsuitable.

[0014] Alternatively, the weight determination step can be configured to determine weights based on agricultural-related knowledge. For example, the weight determination step can also be configured to determine weights based on knowledge obtained from at least one of the perspectives of agricultural experts and agricultural-related experience. In this way, larger weights can be assigned to leaves suitable for subsequent processes, and smaller weights (or uncertain weights) can be assigned to leaves unsuitable for subsequent processes.

[0015] Alternatively, in the weight determination step, the weight of a leaf can be determined based on at least one of its shape, size, and position. For example, leaves that appear slender when viewed at an angle or partially hidden among other leaves are more likely to be unsuitable for subsequent processes due to the inability to accurately identify their leaf condition. Therefore, in the weight determination step, the closer the shape of the leaf's bounding box is to a square, the greater the weight assigned to that leaf. Similarly, leaves that are undeveloped or partially hidden among other leaves are more likely to be unsuitable for subsequent processes due to the inability to accurately identify their leaf condition. Therefore, in the weight determination step, the larger the leaf size, the greater the weight assigned to that leaf. Furthermore, since humidity is higher closer to the ground, leaves closer to the ground are more susceptible to mold than those farther away. Therefore, in the weight determination step, the closer a leaf is to the ground, the greater its weight assigned to that leaf. Additionally, since younger leaves (upper leaves) are more susceptible to pests, in the weight determination step, the farther a leaf is from the ground, the greater its weight assigned to that leaf. The bounding box of a leaf is a rectangular box that surrounds the leaf, such as a rectangular box that is circumscribed around the leaf.

[0016] Alternatively, the leaf detection model can be configured to use an inference model based on Mask R-CNN or Faster R-CNN. Furthermore, in the first learning step, a larger weight corresponds to a larger reduction in the loss function value. This adjusts the allowable range of leaves, widening the allowable range based on leaves with larger weights and narrowing the allowable range based on leaves with smaller weights. Consequently, leaves with larger weights (those included within the allowable range based on leaves with larger weights) are easier to detect than leaves with smaller weights (those included within the allowable range based on leaves with smaller weights).

[0017] Alternatively, the learning method may further include a second learning step, in which the detection results of the leaf detection model learned in the first learning step are used to learn a leaf state recognition model for identifying leaf states. In this way, a leaf detection model capable of appropriate leaf detection and a leaf state recognition model capable of high-precision leaf identification can be obtained. Alternatively, the leaf state recognition model may identify whether a leaf has been affected by disease or pests.

[0018] A second aspect of the present invention provides a leaf state recognition device, characterized by comprising: an acquisition unit for acquiring a captured image; a detection unit for detecting leaves from the captured image acquired by the acquisition unit using a leaf detection model learned through the aforementioned learning method; and a recognition unit for recognizing the state of the leaf detected by the detection unit using a leaf state recognition model for recognizing the state of the leaf. According to this configuration, since the leaf detection model learned through the aforementioned learning method is used for leaf detection, the leaf state can be recognized with high accuracy.

[0019] It should be noted that the present invention can be understood as a learning device, leaf state recognition device, learning system, or leaf state recognition system having at least a part of the above-described configuration or function. Furthermore, the present invention can also be understood as a learning method, leaf state recognition method, control method of a learning system, or control method of a leaf state recognition system including at least a part of the above-described processing, a program for causing a computer to execute these methods, or a computer-readable storage medium that non-transitoryly stores such a program. The above-described configurations and processes can be combined with each other to constitute the present invention as long as they do not create technical contradictions.

[0020] Invention Effects

[0021] According to the present invention, leaves can be appropriately detected, and subsequent processes such as leaf state identification can be performed with high precision. Attached Figure Description

[0022] Figure 1 (A) is a flowchart illustrating an example of applying the learning method of the present invention. Figure 1 (B) is a block diagram showing an example of the configuration of the leaf state recognition device of the present invention.

[0023] Figure 2 A block diagram illustrating an example configuration of the leaf state recognition system according to this embodiment.

[0024] Figure 3 (A) is a flowchart illustrating an example of the processing flow of the PC (leaf state recognition device) during the learning phase. Figure 3(B) is a flowchart illustrating an example of the processing flow of the PC in the inference phase after the learning phase.

[0025] Figure 4 (A) is a schematic diagram illustrating an example of a photograph taken for learning purposes. Figure 4 (B) and (C) are schematic diagrams illustrating an example of a bounding box, etc.

[0026] Figure 5 This is a schematic diagram illustrating an example of a leaf detection model using Mask R-CNN.

[0027] Figure 6 (A) shows the detection results of the comparative example before restriction (the detection results of the leaf). Figure 6 (B) shows the detection results of the comparative example after the narrowing. Figure 6 (C) shows the detection results of this embodiment.

[0028] Figure 7 (A) shows the detection results of the comparative example before restriction (the detection results of the leaf). Figure 7 (B) shows the detection results of the comparative example after the narrowing. Figure 7 (C) shows the detection results of this embodiment. Detailed Implementation

[0029] <Application Example>

[0030] Application examples of the present invention will be described.

[0031] A device (system) for detecting (extracting) leaves from captured images and identifying the state of the detected leaves has been proposed. In such a device, if a leaf unsuitable for leaf state identification is detected (e.g., a leaf that appears slender, a leaf that appears small, a leaf partially hidden among other leaves, an out-of-focus, blurry leaf, a dark leaf, etc.), an incorrect identification result will be obtained for that leaf, reducing the overall identification accuracy. Furthermore, with low overall identification accuracy, tasks (labor) such as having an agricultural expert (someone with agricultural expertise) verify the identification results are required.

[0032] Figure 1 (A) is a flowchart illustrating an example of applying the learning method of the present invention. In step S101, weights are determined for the leaves included in the captured image. In step S102, based on the weights determined in step S101, a leaf detection model for detecting leaves from the captured image is learned, such that leaves with larger weights are easier to detect than leaves with smaller weights. Step S101 is an example of a weight determination step, and step S102 is an example of a first learning step. The captured image may be a wide-field image with a wide view, or it may not be a wide-field image with a wide view.

[0033] Based on the above method, weights are assigned to leaves, and a leaf detection model is trained to ensure that leaves with higher weights are easier to detect than those with lower weights. This allows for appropriate leaf detection, leading to more accurate leaf state identification and other subsequent processes. For example, if leaves suitable for subsequent processes are assigned higher weights, while leaves unsuitable for subsequent processes are assigned lower weights (or have uncertain weights), then leaves suitable for subsequent processes are easier to detect than those unsuitable.

[0034] Alternatively, in step S101, weights based on agricultural knowledge can be determined. For example, in step S101, weights based on knowledge obtained from at least one of the perspectives of agricultural experts and agricultural experience can be determined. In this way, larger weights can be assigned to leaves suitable for subsequent processes, and smaller weights (or uncertain weights) can be assigned to leaves unsuitable for subsequent processes. Perspective information can be obtained using existing perspective detection techniques.

[0035] Figure 1 (B) is a block diagram showing an example of the configuration of the leaf state recognition device 110 according to the present invention. The leaf state recognition device 110 includes an acquisition unit 111, a detection unit 112, and a recognition unit 113. The acquisition unit 111 acquires captured images. The detection unit 112 detects leaves from the captured images acquired by the acquisition unit 111 using a leaf detection model learned by the learning method described above. The recognition unit 113 recognizes the state of the leaves detected by the detection unit 112 using a leaf state recognition model that recognizes the state of the leaves. The acquisition unit 111 is an example of an acquisition unit, the detection unit 112 is an example of a detection unit, and the recognition unit 113 is an example of a recognition unit. According to this configuration, since the leaf detection model learned by the learning method described above is used for leaf detection, the leaf state can be recognized with high accuracy.

[0036] <Implementation Method>

[0037] The embodiments of the present invention will be described.

[0038] (constitute)

[0039] Figure 2This is a block diagram illustrating an example configuration of the leaf state recognition system according to this embodiment. The leaf state recognition system includes a camera 11 (imaging device), a PC 200 (personal computer; leaf state recognition device), and a display 12 (display device). The camera 11 and the PC 200 are interconnected via wired or wireless means, and the PC 200 and the display 12 are interconnected via wired or wireless means. The camera 11 takes pictures of fields, etc., and outputs the captured images to the PC 200. The PC 200 detects leaves from the captured images of the camera 11 and identifies the state of the detected leaves. Then, the PC 200 displays the identification results, etc., on the display 12. The display 12 displays various images and information.

[0040] It should be noted that the camera 11 may or may not be fixed. The positional relationship between the camera 11, PC 200, and monitor 12 is not particularly limited. For example, the camera 11, PC 200, and monitor 12 may or may not be located in the same room (e.g., a plastic greenhouse).

[0041] In this embodiment, the camera 11 and the display 12 are set as separate devices from the PC 200, but at least one of the camera 11 and the display 12 may also be part of the PC 200. The PC 200 (leaf state recognition device) may also be a computer in the cloud. The functions of at least some of the camera 11, the PC 200, and the display 12 may also be implemented through various terminals such as smartphones and tablets.

[0042] PC200 includes an input unit 210, a control unit 220, a storage unit 230, and an output unit 240.

[0043] The input unit 210 acquires captured images from the camera 11. For example, the input unit 210 is an input terminal. The input unit 210 is an example of an acquisition unit.

[0044] The control unit 220 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and performs control of each component and various information processing. In this embodiment, the control unit 220 detects leaves from the captured image of the camera 11 (the captured image acquired through the input unit 210) and identifies the state of the detected leaves.

[0045] The storage unit 230 stores programs executed in the control unit 220, various data used in the control unit 220, etc. For example, the storage unit 230 is an auxiliary storage device such as a hard disk drive or a solid-state drive.

[0046] The output unit 240 outputs the recognition results, etc., from the control unit 220 to the display 12. The recognition results, etc., are then displayed on the display 12. For example, the output unit 240 is an output terminal.

[0047] The control unit 220 will be described in more detail. The control unit 220 includes a labeling unit 221, a weight determination unit 222, a detection unit 223, and an identification unit 224.

[0048] The annotation unit 221 annotates the images captured by the camera 11. The weight determination unit 222 determines the weights of the leaves included in the images captured by the camera 11. The detection unit 223 detects leaves from the images captured by the camera 11 using a leaf detection model. The recognition unit 113 recognizes the state of the leaves detected by the detection unit 112 using a leaf state recognition model. Details of these processes will be described later. The detection unit 112 is an example of a detection unit, and the recognition unit 113 is an example of a recognition unit.

[0049] (Processing flow during the learning phase)

[0050] Figure 3 (A) is a flowchart illustrating an example of the processing flow of PC200 during the learning phase. During the learning phase, a leaf detection model is learned. In this embodiment, a leaf state recognition model is also learned.

[0051] First, the input unit 210 acquires images for learning purposes (step S301). The images for learning purposes can be images captured by the camera 11 or images not captured by the camera 11. Figure 4 (A) shows an example of a photograph taken for learning purposes. Figure 4 In image (A), only one plant was captured, but there are also cases where many plants were captured in an image.

[0052] Next, the annotation unit 221 annotates the captured image obtained in step S301 (step S302). Annotation is a process of setting the truth value (correct solution) in the learning process, and the truth value is specified based on the information specified (input) by the operator.

[0053] For example, the operator specifies the outline of a leaf captured in the image. The annotation unit 221 responds to the specified outline by setting a leaf mask within the area enclosed by that outline. Then, as... Figure 4 As shown in (B), the annotation unit 221 automatically sets the rectangular frame, i.e., the bounding box, that surrounds the leaf mask (leaf). For example, the annotation unit 221 sets the rectangular frame that is circumscribed to the leaf mask (leaf) as the bounding box.

[0054] It should be noted that the preferred operator only selects leaves suitable for subsequent processes (leaf state identification in this embodiment) and specifies their outlines. However, if the operator is not an agricultural expert, it is difficult to determine whether a leaf is suitable for subsequent processes, and the operator specifying the outline is not necessarily an agricultural expert. Therefore, in the annotation, leaf masks and bounding boxes for leaves that are not suitable for subsequent processes are sometimes set.

[0055] In this embodiment, leaf condition identification is performed to determine whether the leaves have been affected by diseases or pests (whether the leaves are healthy). Therefore, the operator inputs information about whether the leaves have been affected by diseases or pests, and the labeling unit 221 sets this information. The information about whether the leaves have been affected by diseases or pests is set to be input by an agricultural expert. It should be noted that leaf condition identification can also identify the type of disease, the type of pest, etc.

[0056] Return to Figure 3 Explanation of (A). After step S302, the weight determination unit 222 determines the weight of the leaf included in the captured image acquired in step S301 based on the information set in step S302 (step S303). In this embodiment, the weight determination unit 222 determines the weight of the leaf based on at least one of the leaf's shape, size, and position. Step S302 is an example of the weight determination step.

[0057] For leaves that appear slender when viewed at an angle or partially hidden among other leaves, it is difficult to accurately identify their state, making them unsuitable for subsequent processes. Therefore, the closer the shape of the leaf's bounding box is to a square, the greater the weight assigned to that leaf by the weight determination unit 222. For example, the weight determination unit 222 uses the following equations 1-1 and 1-2, based on... Figure 4 The width w and height h of the bounding box shown in (C) determine the weight ω1.

[0058] When w / h ≤ 1: ω1 = w / h…(Equation 1-1)

[0059] When w / h > 1: ω1 = h / w…(Equation 1-2)

[0060] Leaves that are underdeveloped or partially hidden among other leaves and appear small cannot be accurately identified, making them unsuitable for subsequent processes. Therefore, the larger the leaf size, the greater the weight assigned to that leaf by the weight determination unit 222. For example, the weight determination unit 222 uses the following formula 2, based on... Figure 4 The width W (number of pixels in the horizontal direction) and height H (number of pixels in the vertical direction) of the captured image shown in (B) are as follows: Figure 4 The number of pixels s of the leaf mask shown in (C) is used to determine the weight ω2. W×H is the total number of pixels in the captured image.

[0061] ω2=s / (W×H)…(Equation 2)

[0062] The weight determination unit 222 can also determine the weight ω2 using the following equations 2-1 to 2-3. The thresholds Th1 and Th2 are not particularly limited; for example, when W = 1200 and H = 1000, they can be set to Th1 = 5000 and Th2 = 10000. It should be noted that the number of levels for weight ω2 can be more than three or less than three.

[0063] When s≤Th1: ω2=0.1…(Equation 2-1)

[0064] When Th1 < s ≤ Th2: ω2 = 0.5…(Equation 2-2)

[0065] When Th2 < s: ω2 = 0.9…(Equation 2-3)

[0066] Since humidity is higher closer to the ground, leaves closer to the ground are more susceptible to mold than those farther away. Therefore, the weight determination unit 222 may assign a greater weight to a leaf closer to the ground. For example, when the image is obtained by photographing the plant from the side, the weight determination unit 222 uses equations 3-1 to 3-3 to determine the weight ω3 based on the vertical position c_y (vertical position) of the center of the bounding box. The thresholds Th3 and Th4 are not particularly limited. For example, threshold Th3 corresponds to a vertical position with a vertical distance (vertical distance) of H / 3 from the bottom of the image, and threshold Th4 corresponds to a vertical position with a vertical distance of (2 / 3) × H from the bottom of the image. Here, the value (coordinate value) of the vertical position is assumed to increase from the bottom of the image upwards. It should be noted that the number of levels for weight ω3 may be more than three or less than three.

[0067] When c_y≤Th3: ω3=0.9…(Equation 3-1)

[0068] When Th3 < c_y ≤ Th4: ω3 = 0.5…(Equation 3-2)

[0069] When Th4 < c_y: ω3 = 0.1…(Equation 3-3)

[0070] When the image is taken from an overhead view of the field, leaves near the ground may sometimes appear at the top of the image. In such cases, it is also possible that... Figure 4As shown in (B), the overall plant bounding box is set not by the vertical distance from the bottom of the captured image, but by treating the vertical distance from the bottom of the overall plant bounding box as the distance from the ground.

[0071] The weight determination unit 222 can determine any one of the weights ω1 to ω3, or it can combine two or three of the weights ω1 to ω3 to determine the final weight ω. For example, the weight determination unit 222 can determine ω1×ω2, ω1×ω3, ω2×ω3, or ω1×ω2×ω3 as the final weight ω. Furthermore, the weight determination unit 222 can also determine the weight ω only for leaves that satisfy a predetermined condition (or determine ω = 0 for leaves that do not satisfy the predetermined condition). The predetermined condition may include the condition 0.75 < w / h < 1.3. In the case of W = 1200 and H = 1000, the predetermined condition may include the condition s > 10000.

[0072] It should be noted that the method for determining the weights is not limited to the methods described above. For example, since younger leaves (upper leaves) are more susceptible to pest infestation, the weight determination unit 222 may assign a greater weight to a leaf that is further away from the ground. The weight determination unit 222 may also increase the weight of leaves that are appropriately exposed (appropriate brightness) or increase the weight of sharp leaves based on the brightness value and sharpness of the leaf image.

[0073] Return to Figure 3 Explanation of (A). After step S303, the control unit 220 learns the leaf detection model possessed by the detection unit 223 based on the weights determined in step S303, so that leaves with larger weights are easier to detect than leaves with smaller weights (step S304). Step S304 is an example of the first learning step. By learning the leaf detection model so that leaves with larger weights are easier to detect than leaves with smaller weights, leaf detection can be performed appropriately, thereby enabling subsequent processes such as leaf state recognition with high accuracy.

[0074] Various techniques, such as Mask R-CNN or Faster R-CNN, can be used in leaf detection models. In this embodiment, such as... Figure 5 As shown, the leaf detection model is assumed to be an inference model (learning model) using Mask R-CNN. Since Mask R-CNN is a well-known technique, it will only be briefly described below.

[0075] In the leaf detection model (MaskR-CNN), firstly, a CNN (Convolutional Neural Network) is used to extract features from the captured image to generate a feature map. Next, a RPN (Recursive Processing Network) is used to detect candidate regions (bounding boxes) of leaves from the feature map. Then, a fixed-size feature map is obtained through Rol Align, and after processing by fully connected layers (not shown), the inference result for each candidate region is obtained (the probability that the candidate region is a leaf region (positive probability), the position of the candidate region, the size of the candidate region, and the candidate for the leaf mask, etc.). After learning the leaf detection model, the detection unit 223 detects candidate regions with a positive probability above a predetermined threshold as the bounding boxes of the leaves.

[0076] During the learning of the leaf detection model, the control unit 220 compares the inference results with the ground truth (positive solution) according to the candidate regions to calculate the loss L. The loss L is calculated, for example, using Equation 4 (loss function). Loss Lcls is the classification loss of the bounding box, which decreases when the candidate region matches the bounding box of the positive solution. Loss Lloc is the regression loss of the bounding box, which decreases as the candidate region gets closer to the bounding box of the positive solution. Loss Lmask is the matching loss of the leaf mask, which decreases as the candidate leaf mask gets closer to the leaf mask of the positive solution. The coefficients f(ω) and g(ω) are coefficients that depend on the weight ω determined by the weight determination unit 222, for example, f(ω) = g(ω) = e -ω In this embodiment, the weight determination unit 222 determines the weight of the leaf based on at least one of the leaf's shape, size, and position. Since the losses associated with the leaf's shape, size, and position are loss Lloc and loss Lmask, loss Lloc and loss Lmask are multiplied by coefficients f(ω) and g(ω), respectively.

[0077] L=Lcls+Lloc×f(ω)+Lmask×g(ω)…(Formula 4)

[0078] Then, the control unit 220 updates the RPN based on the loss L for each candidate region. The larger the weight ω, the smaller the coefficients f(ω) and g(ω). Therefore, the larger the weight ω, the greater the reduction in the value of the loss function (L = Lcls + Lloc + Lmask) without considering the weight ω. By updating the RPN based on this reduced loss L, the allowable range of the leaves is adjusted so that the allowable range based on leaves with large weight ω becomes wider, and the allowable range based on leaves with small weight ω becomes narrower. As a result, the candidate regions of leaves with large weight ω (leaves included in the allowable range based on leaves with large weight ω) are easier to detect than the candidate regions of leaves with small weight ω (leaves included in the allowable range based on leaves with small weight ω). Moreover, the control unit 220 updates the overall leaf detection model based on the sum (average) of the losses L for each candidate region.

[0079] It should be noted that while the example of reducing candidate regions for leaves with small weights ω has been illustrated, other methods can also be used to make leaves with large weights ω easier to detect than those with small weights ω. For example, the leaf detection model can be trained by reducing the positive probability of candidate regions for leaves with small weights ω.

[0080] Return to Figure 3 Explanation of (A). After step S304, the control unit 220 uses the detection results of the detection unit 223, which has the leaf detection model learned in step S304, to learn the leaf state recognition model possessed by the recognition unit 224 (step S305). Step S305 is an example of the second learning step. By using the detection results of the detection unit 223, which has the learned leaf detection model, a leaf state recognition model capable of high-precision leaf recognition can be obtained. Various techniques can also be used in the leaf state recognition model.

[0081] (Processing flow during the reasoning phase)

[0082] Figure 3 (B) is a flowchart illustrating an example of the processing flow of PC200 in the inference stage after the learning stage. First, the input unit 210 acquires a captured image from the camera 11 (step S311). Next, the detection unit 223 uses a learned leaf detection model to detect leaves from the captured image acquired in step S311 (step S312). Next, the recognition unit 113 uses a learned leaf state recognition model to recognize the state of the leaves detected in step S312 (step S313). Next, the output unit 240 outputs the recognition result of step S313 to the display 12 for display (step S314).

[0083] (Effect)

[0084] The effects of this embodiment will be explained. In this embodiment, weights are assigned to leaves, and a leaf detection model is trained so that leaves with larger weights are easier to detect than leaves with smaller weights. As another method (comparative example), it is conceivable to limit the leaf detection results with a predetermined threshold. However, such methods cannot obtain the appropriate detection results (leaf detection results) as the method of this embodiment.

[0085] Figure 6 (A) and Figure 6 (B) shows the detection results of the comparative example. Figure 6 (A) shows the detection results before weighting. Because weights were not considered during learning, all leaves were detected. Furthermore, fruits were falsely detected. Figure 6 (B) shows the result after limiting the size by a threshold to remove smaller leaves. Figure 6 In (B), smaller leaves were removed from the test results, but fruits were not removed because they were too large.

[0086] Figure 6 (C) shows the detection results of this embodiment. By considering weights in the learning process (increasing the weight of leaves that well represent the characteristics of leaves and learning accordingly), it is possible to detect only large leaves suitable for subsequent processes without detecting smaller leaves and fruits.

[0087] Figure 7 (A) and (B) show the detection results of the comparative example. Figure 7 (A) shows the detection results before narrowing. Since weights were not considered during learning, all leaves were detected. Bright and clear leaves were also detected. Such leaves, even if small, are likely to be suitable for subsequent processes (e.g., leaves whose state can be identified with high precision). Figure 7 (B) shows the result after limiting the size by a threshold to remove smaller leaves. Figure 7 In (B), the bright and clear leaves that should be left as leaves suitable for subsequent processes are removed because they are too small.

[0088] Figure 7 (C) shows the detection results of this embodiment. Smaller leaves are difficult to detect because weights are taken into account during learning; however, bright and clear leaves can be detected because they well represent the characteristics of leaves even when they are small.

[0089] (Summarize)

[0090] As described above, according to this embodiment, weights are assigned to leaves, and a leaf detection model is trained so that leaves with higher weights are easier to detect than those with lower weights. This allows for appropriate leaf detection, and subsequently, high-precision leaf state identification and other follow-up processes.

[0091] <Other>

[0092] The above embodiments are merely illustrative examples illustrating the configuration of the present invention. The present invention is not limited to the specific embodiments described above, and various modifications can be made within the scope of its technical concept.

[0093] <Postscript 1>

[0094] A learning method, characterized by having:

[0095] The weight determination steps (S101, S303) determine the weights of the leaves included in the captured image; and

[0096] The first learning step (S102, S304) involves learning a leaf detection model for detecting leaves from captured images based on the weights determined in the weight determination step, so that leaves with larger weights are easier to detect than leaves with smaller weights.

[0097] <Appendix 2>

[0098] A leaf state recognition device (110, 200), characterized in that it has:

[0099] Acquisition units (111, 210) acquire captured images;

[0100] Detection units (112, 223) detect leaves from the captured images acquired by the acquisition unit using the leaf detection model learned by the learning method according to any one of claims 1 to 9; and

[0101] The identification unit (113, 224) uses a leaf state identification model to identify the state of the leaf detected by the detection unit.

[0102] Explanation of reference numerals in the attached figures

[0103] 110: Leaf state recognition device; 111: Acquisition unit; 112: Detection unit; 113: Recognition unit; 200: PC (information processing device); 210: Input unit; 220: Control unit; 230: Storage unit; 240: Output unit; 221: Labeling unit; 222: Weight determination unit; 223: Detection unit; 224: Recognition unit; 11: Camera; 12: Display.

Claims

1. A learning method, characterized in that, have: The weight determination step determines the weight of a leaf based on the position of the leaf included in the captured image. as well as The first learning step involves learning a leaf detection model from captured images based on the weights determined in the weight determination step, so that leaves with larger weights are easier to detect than leaves with smaller weights. In the weight determination step, the closer a leaf is to the ground, the greater the weight assigned to that leaf.

2. A learning method, characterized in that, have: The weight determination step determines the weight of a leaf based on the position of the leaf included in the captured image. as well as The first learning step involves learning a leaf detection model from captured images based on the weights determined in the weight determination step, so that leaves with larger weights are easier to detect than leaves with smaller weights. In the weight determination step, the farther the leaf is from the ground, the greater the weight assigned to that leaf.

3. The learning method according to claim 1, characterized in that, The learning method also has the following characteristics: The step of learning a disease identification model for leaves by using the detection results of the leaf detection model learned in the first learning step.

4. The learning method according to claim 2, characterized in that, The learning method also has the following characteristics: The step of learning a model for identifying leaf pests using the detection results of the leaf detection model learned in the first learning step.

5. The learning method according to claim 1, characterized in that, In the weight determination step, weights based on agricultural-related knowledge are determined.

6. The learning method according to claim 5, characterized in that, In the weight determination step, weights are determined based on knowledge obtained from at least one of the perspectives of agricultural experts and agricultural-related experience.

7. The learning method according to claim 1, characterized in that, In the weight determination step, the weight of the leaf is determined based on at least one of the leaf's shape and size.

8. The learning method according to claim 7, characterized in that, In the weight determination step, the closer the shape of the leaf's bounding box is to a square, the greater the weight is assigned to that leaf.

9. The learning method according to claim 7 or 8, characterized in that, In the weight determination step, the larger the size of the leaf, the greater the weight assigned to that leaf.

10. The learning method according to any one of claims 1 to 8, characterized in that, The leaf detection model uses an inference model that employs Mask R-CNN or Faster R-CNN.

11. The learning method according to claim 10, characterized in that, In the first learning step, the larger the weight, the greater the reduction in the value of the loss function.

12. The learning method according to any one of claims 1 to 8, characterized in that, The learning method also has the following characteristics: The second learning step involves using the detection results of the leaf detection model learned in the first learning step to learn a leaf state recognition model for identifying the state of the leaves.

13. The learning method according to claim 12, characterized in that, The leaf condition recognition model identifies whether the leaves have been affected by diseases or pests.

14. A leaf state recognition device, characterized in that, have: The acquisition unit acquires the captured image; The detection unit uses the leaf detection model learned by the learning method according to any one of claims 1 to 13 to detect leaves from the captured image obtained by the acquisition unit; as well as The identification unit uses a leaf state identification model to identify the state of the leaf detected by the detection unit.

15. A computer program product for causing a computer to perform the steps of the learning method according to any one of claims 1 to 13.

Images