Diagnostic device, diagnostic method, and program
A multi-task learning model for plant health diagnosis processes ground and wide-range images to accurately diagnose lawn diseases, improving prediction accuracy and enabling effective disease management.
Patent Information
- Application Number
- JP2023131080
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-03-14
- Filing Date
- 2023-08-10
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2043-08-10
AI Technical Summary
Diagnosing the health status of plants, particularly lawns, is challenging due to the need for specialized knowledge and the difficulty in observing plant conditions, especially when historical data on pest and disease occurrences is lacking.
A diagnostic device utilizing a multi-task learning model that processes ground surface and wide-range images of plants to predict health conditions, incorporating both multi-value and binary classification tasks, enabling accurate disease diagnosis without relying on historical data.
The model enhances prediction accuracy for plant health conditions, allowing for timely and effective management of diseases such as anthracnose, brown patch, dollar spot, dry spot, and snow mold by providing detailed diagnosis results and visualizations.
Smart Images

Figure 0007591628000001 
Figure 0007591628000002 
Figure 0007591628000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a diagnostic device and a diagnostic method. , and and programs. [Background technology]
[0002] Diagnosing the health of plants requires advanced expertise. In addition, it can be difficult for experts with such knowledge to directly observe plants. To solve this problem, Patent Document 1 discloses a learning model that inputs images of a target plant and estimates the pests and diseases of the target plant. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2022-94783 A Summary of the Invention [Problem to be solved by the invention]
[0004] This learning model discloses a method for diagnosing pests in a target plant by referring to the probability of pest occurrence calculated from the past pest occurrence history data in addition to the estimated pests. However, since obtaining pest occurrence history data is time-consuming and laborious, it is desirable to be able to appropriately judge the health of a plant without the history data.
[0005] The present disclosure aims to provide a diagnostic device, a diagnostic method, a learning model generation method, and a program that can appropriately diagnose the health condition of a plant based on an image. [Means for solving the problem]
[0006] A diagnostic device according to one embodiment of the present disclosure is a diagnostic device for diagnosing the health condition of a plant, the diagnostic device including a multi-task learner configured to output a plurality of prediction results respectively associated with a plurality of tasks when an input image is input, the multi-task learner being trained with training data including a ground image group including a plurality of ground images capturing the plant covering the ground surface, and a wide-range image group including a plurality of wide-range images capturing the plant over an area wider than the ground image, each of the ground image group and the wide-range image group including a plurality of asymptomatic images capturing the plant in a healthy state and a plurality of symptomatic images capturing the plant in an unhealthy state, the unhealthy state including a plurality of disease symptoms, the plurality of tasks including a multi-value classification task of predicting whether the plant captured in the input image corresponds to one of the plurality of disease symptoms or asymptomatic symptoms, and a binary classification task of predicting whether the input image corresponds to the ground image or the wide-range image.
[0007] A diagnostic method according to one embodiment of the present disclosure is a method for diagnosing the health of a plant, the method including: acquiring an input image depicting the plant covering a ground surface; inputting the input image to a multi-task learner; and outputting a plurality of prediction results respectively associated with a plurality of tasks, wherein the multi-task learner is trained with training data including a ground image group including a plurality of ground images depicting the plant covering the ground surface; and a wide-range image group including a plurality of wide-range images depicting the plant over an area wider than the ground image, wherein each of the ground image group and the wide-range image group includes a plurality of asymptomatic images depicting the plant in a healthy state and a plurality of symptomatic images depicting the plant in an unhealthy state, wherein the unhealthy state includes a plurality of disease symptoms, and the plurality of tasks include a multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the plurality of diseases or asymptomatic, and a binary classification task of predicting whether the input image corresponds to either the ground image or the wide-range image.
[0008] In a method for generating a learning model according to one embodiment of the present disclosure, a learning model for diagnosing the health condition of a plant is configured to output a plurality of prediction results respectively associated with a plurality of tasks when an input image is input, the generation method includes acquiring a group of images that serve as training data, and training the learning model using the training data, the training data including a group of ground images including a plurality of ground images depicting the plant covering the ground surface, and a group of wide-range images including a plurality of wide-range images depicting the plant over an area wider than the ground images, each of the group of ground images and the group of wide-range images including a plurality of asymptomatic images depicting the plant in a healthy state and a plurality of symptomatic images depicting the plant in an unhealthy state, the unhealthy state including a plurality of symptoms of a disease, and the plurality of tasks include a multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the plurality of diseases or asymptomatic, and a binary classification task of predicting whether the input image corresponds to either the ground image or the wide-range image.
[0009] A program according to one embodiment of the present disclosure is a plant diagnosis program that causes one or more computers to execute the following steps: acquire an input image depicting a plant covering the ground surface; input the acquired input image to a multi-task learner; and output a plurality of prediction results associated with a plurality of tasks, wherein the multi-task learner is trained with training data including a ground image group including a plurality of ground images depicting the plants covering the ground surface, and a wide-range image group including a plurality of wide-range images depicting the plants over an area wider than the ground image, each of the ground image group and the wide-range image group includes a plurality of asymptomatic images depicting the plant in a healthy state and a plurality of symptomatic images depicting the plant in an unhealthy state, the unhealthy state including a plurality of disease symptoms, and the plurality of tasks include a multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the plurality of diseases or asymptomatic, and a binary classification task of predicting whether the input image corresponds to either the ground image or the wide-range image. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a schematic diagram for explaining a diagnostic device according to an embodiment. [Diagram 2] FIG. 2 is an example of a wide-range image input to the diagnostic device of FIG. [Diagram 3] FIG. 3 is an example of a ground image input to the diagnostic device of FIG. [Figure 4] FIG. 4 is a table illustrating training data for a learning model according to an embodiment. [Diagram 5] FIG. 5 is a schematic diagram illustrating a learning model according to the embodiment. [Figure 6] FIG. 6 is a graph showing the relationship between the loss function of the learning model in FIG. 5 and the prediction accuracy. [Figure 7] FIG. 7 is a table showing the accuracy rate of a learning model including subtasks according to the embodiment and a learning model of a comparative example not including subtasks. [Figure 8] FIG. 8 is a schematic diagram for explaining additional functions of the diagnostic device of FIG. [Figure 9] Figure 9 shows an example of an input image and a heat map. [Figure 10] FIG. 10 is a flow diagram illustrating a diagnostic method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Examples of the diagnostic device, diagnostic method, learning model generation method, and program of the present disclosure will be described below with reference to the drawings. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.
[0012] [Diagnostic equipment] FIG. 1 illustrates an example of a diagnostic device 30 configured to diagnose the health of a plant. In this disclosure, grass is illustrated as an example of a plant. Hereinafter, grass growing to cover the ground surface is referred to as lawn 18. Plants that grow in clusters to cover the ground surface are also called ground cover plants, and are often evergreen perennial plants or evergreen woody plants with a height of about 50 cm or less, about 30 cm or less, or about 10 cm or less. Such plants, such as ground cover plants, are planted to densely cover the ground surface with a single species or multiple species for the purpose of adding color to the landscape, cushioning impacts, protecting the topsoil, suppressing temperature changes, or suppressing the growth of other weeds.
[0013] Grass is generally cut short to keep the leaves short. Mowing the lawn drastically reduces the surface area available for the grass to photosynthesize. To compensate for the reduced efficiency of photosynthesis, the grass grows denser and increases the number of leaves, forming a green carpet. Mowed grass is less resistant to disease than unmowed grass, and requires management such as spraying chemicals according to the symptoms of disease. The diagnostic device 30 can be used to determine whether or not management of plants such as grass is required.
[0014] Regarding the health of the lawn, if the lawn grows evenly all over, it is asymptomatic and can be determined to be in a healthy state. In contrast, if the planted area of the lawn is bald, withered, discolored, or patchy, it is symptomatic and can be determined to be in an unhealthy state.
[0015] 1 shows a golf course 11 as an example of the location where the grass 18 is planted, but it may also be a park, a kindergarten yard, a schoolyard, a horse racing track, or an athletic field such as a soccer field, an American football field, a baseball field, or a tennis court. Additionally, the diagnostic device 30 may be used to diagnose the health of plants that are natural vegetation.
[0016] A photographer 20 operates a photographing device 21 to capture one or more images 10 of grass laid on a golf course 11. The photographing device 21 may be, for example, but is not limited to, a portable digital camera, a video camera, a smartphone, or a tablet. The images 10 may be captured by a photographing device mounted on an unmanned aerial vehicle such as a drone or an autonomous vehicle, or may be captured by a fixed-point camera. The images 10 are captured periodically or irregularly. The images 10 may be captured at a fixed point, or may be captured at a different position each time.
[0017] Photographer 20 may photograph one or more predetermined photographing locations, for example, one or more areas such as green 12 that requires particular maintenance. Photographer 20 may photograph multiple images 10 for each location. The multiple images 10 may include, for example, one or more ground surface images 10A (see FIG. 3) depicting vegetation (lawn 18) covering the ground surface, and one or more wide-range images 10B (see FIG. 2) depicting vegetation (lawn 18) covering an area wider than ground surface image 10A.
[0018] 1 and 2, the wide-range image 10B may include natural or man-made objects in or around the lawn 18. An example of a natural object in the lawn 18 is the boundary line of the green 12, and an example of a natural object around the lawn 18 is a bunker 14, a pond 15, or a tree 16. An example of a man-made object in the lawn 18 is a cup hole 13 or a pin, and an example of a man-made object around the lawn 18 is a building such as a maintenance facility 17.
[0019] As shown in Fig. 3, the ground surface image 10A should be captured primarily to show the health state of the lawn 18. In particular, if the photographer 20 visually observes an unhealthy condition of the lawn 18 that may be due to disease, such as peeling, withering, discoloration, or mottling, the photographer 20 should capture the ground surface image 10A so as to capture that condition. An unhealthy condition includes multiple symptoms of disease. If an unhealthy condition is confirmed, the photographer 20 should capture a wide-range image 10B in addition to the ground surface image 10A so that the distribution of symptoms can be seen.
[0020] The ground surface image 10A1 shown in FIG. 3 is an example of a symptomless image showing a healthy lawn 18. The ground surface images 10A2 to 10A7 shown in FIG. 3 are examples of symptomatic images showing unhealthy lawn 18. The types of disease may include, for example, one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, and snow mold. The ground surface images 10A2, 10A3, 10A4, 10A5, 10A6, and 10A7 show lawns 18 exhibiting symptoms of anthracnose, brown patch, dollar spot, dry spot, pythium, or snow mold, respectively.
[0021] The captured image 10 may be stored in a user terminal 22. The user terminal 22 may be located, for example, in a management facility 17 of the golf course 11. The user terminal 22 is, for example, a personal computer. The user terminal 22 may be configured to communicate with the diagnostic device 30 through a network 23. In this case, the image 10 stored in the user terminal 22 may be transmitted to the diagnostic device 30 through the network 23.
[0022] The network 23 includes, for example, the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, etc. When the imaging device 21 is a terminal having a communication function, such as a smartphone or a tablet, the captured image may be transmitted directly to the diagnostic device 30 without going through the user terminal 22. Alternatively, the user terminal 22 or a terminal having the imaging device 21 may be used as the diagnostic device 30.
[0023] The diagnostic device 30 is, for example, a server or a computer including a processor 31, a memory 32, and a communication IF (interface) 33. The diagnostic device 30 may further include an input device 34, an output device 35, and a storage 36. The processor 31, the memory 32, the communication IF 33, the input device 34, the output device 35, and the storage 36 are connected to each other by a communication bus 37.
[0024] The processor 31 is, for example, a processing circuit configured to execute various software processes. The processing circuit may include a dedicated hardware circuit (e.g., ASIC, etc.) that processes at least a part of the software processes. In other words, the software processes may be executed by a processing circuitry that includes at least one of one or more software processing circuits and one or more dedicated hardware circuits.
[0025] The processor 31 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), a field-programmable gate array (FPGA), or other arithmetic device. The processor 31 executes a series of instructions included in a program stored in the memory 32 or the storage 36 in response to a given signal or in response to the establishment of a predetermined condition.
[0026] The memory 32 is, for example, a random access memory (RAM) or other volatile memory. The memory 32 is configured to temporarily store programs and data. The programs are read from, for example, the storage 36. The data stored in the memory 32 may include data received by the diagnostic device 30 and data generated by the processor 31.
[0027] The communication IF 33 is configured to connect to the network 23. The communication IF 33 is configured to communicate with other devices connected to the network 23. The communication IF 33 is realized, for example, as a LAN or other wired communication IF. The communication IF 33 can also be realized, for example, but not limited to, Wi-Fi (registered trademark), Bluetooth (registered trademark), or other wireless communication IF.
[0028] The input device 34 is, for example, a keyboard and a mouse, and may include a button, a key, a switch, a touch pad, or a microphone. The output device 35 may be, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor, or may be a touch screen having a touch panel that also serves as the input device 34. The output device 35 may be a speaker or a terminal for audio output.
[0029] The storage 36 is a type of memory and may be, for example, a read-only memory (ROM), a hard disk drive, a flash memory, or other non-volatile storage device. The storage 36 may be a removable storage device such as a memory card. The storage 36 is configured to permanently store programs and data. The storage 36 may store a learning device 40, image data 41, one or more applications 43 for realizing various functions, and output data 44. The image data 41 includes one or more images 10 received from the user terminal 22. The output data 44 includes output results from the learning device 40 and the application 43.
[0030] The learning device 40 may be a multi-task learning device configured to output a plurality of prediction results associated with a plurality of tasks, respectively, when the input image 10 is input. The learning device 40 may include a learning model 45 that uses a convolutional neural network. The learning model 45 included in the learning device 40 may be created as follows.
[0031] [How to generate a learning model] 1 is, for example, a computer including a processor 51, a memory 52, and a communication IF (interface) 53. The information processing device 50 may further include an input device 54, an output device 55, and a storage 56. The processor 51, the memory 52, the communication IF 53, the input device 54, the output device 55, and the storage 56 are connected to one another by a communication bus 57. The processor 51, the memory 52, the communication IF 53, the input device 54, the output device 55, and the storage 56 may have the same configurations as the processor 31, the memory 32, the communication IF 33, the input device 34, the output device 35, and the storage 36, respectively.
[0032] In order to generate a learning model, the processor 51 generates or acquires one or more learning programs 58 for machine learning and stores them in the storage 56. The processor 51 also generates or acquires training data 59 and stores them in the storage 56. Thereafter, the processor 51 executes the learning program 58 to train, verify, and test the learning model using the training data 59 shown in FIG. 4. The learning model 45 generalized (generated) using the training data 59 is stored in the storage 56. The learning model 45 included in the learner 40 is a trained learning model 45 acquired from the information processing device 50 via the network 23.
[0033] The training data 59 includes a ground image group and a wide-range image group. The ground image group includes a plurality of ground images capturing plants (lawn 18 in this example) covering the ground surface. The ground images can also be considered as zoomed-in images of plants captured by zooming in more than the wide-range image. The wide-range image group includes a plurality of wide-range images capturing plants (lawn 18 in this example) over an area wider than the ground image. The wide-range images can also be considered as zoomed-out images of plants captured by zooming out more than the ground image. The ground image and the wide-range image may be captured at different positions.
[0034] The ground surface image and the wide-range image are generally distinguished by the objects included in the shooting range. For example, the ground surface image may be an image in which the ground surface covered with vegetation occupies the entire shooting range. The wide-range image may be a landscape image that captures at least one of natural objects such as trees growing on the ground surface or the sky, or artificial objects including buildings, in addition to the ground surface covered with vegetation (grass 18).
[0035] The wide-range image may be an image captured by an aircraft such as a drone from a higher position than the ground image. In this way, even if the ground surface occupies the entire shooting range of an image, an image captured at a high position higher than a person's eye level may be classified as a wide-range image. When a person takes a picture by holding the camera 21 directly in his / her hand, the ground image may be an image captured at a high angle (indicated by a diagonally downward arrow in FIG. 1), and the wide-range image may be an image captured at a lower angle than the ground image, for example, a horizontal angle (indicated by a leftward arrow in FIG. 1) or a low angle (indicated by a diagonally upward arrow in FIG. 1).
[0036] Each of the ground image group and the wide-range image group includes a plurality of asymptomatic images of healthy plants and a plurality of symptomatic images of unhealthy plants. Each of the images used as training data 59 is labeled with a first correct answer label indicating whether the image is a disease type or asymptomatic, and a second correct answer label indicating whether the image is a ground image or a wide-range image.
[0037] The symptom image group includes images of symptoms of a plurality of diseases. The diseases include, for example, one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, and snow mold. The types of diseases prepared as training data 59 become the types of diseases that can be diagnosed by the diagnostic device 30. Each of the ground surface image group and the wide-area image group may include, as training data 59, a plurality of images for each symptom.
[0038] As shown in FIG. 4, the training data 59 includes learning data, verification data, and test data. FIG. 4 shows an example of the number of each disease image used in training. The training data 59 may be collected, for example, by crawling data on the web using the name of the disease as a keyword. In this case, the name of the disease used as the keyword may be attached to the collected images as a first correct label. In addition, objects or feature points captured in the collected images may be extracted to determine whether they are surface images or wide-area images, and the result may be attached as a second correct label.
[0039] The collected images may be subjected to one or more image processing operations, such as rotation, movement, enlargement, reduction, inversion, color change, noise addition, blurring, high frequency component emphasis, and distortion, to increase the amount of training data 59. By adding images that have been subjected to image processing to the training data 59, it becomes possible to learn about images with different positions, orientations, sizes, brightness, or saturation.
[0040] As shown in Fig. 5, the learning model 45 uses a backbone of a convolutional neural network (CNN). Examples of convolutional neural networks are Resnet34, ResNet50, ResNet101, or EfficientNetB7, but in this example, ResNet50 is adopted. The learning model 45 includes a common layer 46, which is a backbone, and multiple (two in this example) output layers 47 (47A, 47B).
[0041] The common layer 46 includes an input layer to which the input image 10 is input, and multiple intermediate layers trained with training data 59. The multiple (two in this example) output layers 47A, 47B are associated with multiple (two in this example) different tasks. The learning model 45 includes a program that causes a computer to input the input image 10, which depicts vegetation covering the ground surface, to the input layer (common layer 46), and output multiple prediction results from the multiple output layers 47, each associated with a multiple task.
[0042] The multiple tasks include a multi-value classification task of predicting whether the plant (lawn 18) depicted in the input image 10 corresponds to a disease type or to asymptomatic, and a binary classification task of predicting whether the input image 10 corresponds to the ground surface image 10A or the wide-range image 10B. The multi-value classification task is also called a first task or main task. The binary classification task is also called a second task or subtask. The multi-value classification task is for predicting plant disease. The binary classification task is for predicting the zoom level of the input image 10. The learning model 45 is also a classifier or a neural network system for simultaneously performing classification according to plant disease and classification according to the zoom level of the input image 10.
[0043] 5, the first output layer 47A is configured to output a prediction result 48A of the first task, and the second output layer 47B is configured to output a prediction result 48B of the second task. In this manner, the learning model 45 is configured to output multiple (two) prediction results 48 (48A, 48B) for one input image 10.
[0044] The multi-value classification task may be configured to output probabilities corresponding to multiple diseases or asymptomatic as a prediction result. The output probabilities are not limited to percentage display, and may be displayed so that the sum of the probabilities is 1. For example, the multi-value classification task may output the probabilities corresponding to each of the seven classifications (also called classes) of six types of diseases and asymptomatic, or may output the top one class with the highest probability and its probability, for example, "asymptomatic: 96.4%". Alternatively, the multi-value classification task may output the top two or three classes with the highest probability among the seven classes and numerical values corresponding to the respective probabilities, for example, "asymptomatic: 0.964, dry spot: 0.015, brown patch: 0.008". The binary classification task may output whether the input image 10 corresponds to the earth surface image 10A or the wide-range image 10B, and the probability of the corresponding image.
[0045] In the learning model 45, the cross entropy loss (hereafter, CELoss) is set as the loss for the main task, and the binary cross entropy loss (hereafter, BCELoss) is set as the loss for the subtask. The loss function of the learning model 45 for the two tasks is integrated as (1-λ)CELoss+λBCELoss.
[0046] Figure 6 shows the verification results of the classification accuracy of the main task and the subtasks according to λ. As shown in Figure 6, the accuracy of the main task is highest when λ = 0.05, but the prediction accuracy of the subtasks is low at this time. The main purpose of the learning model 45 is to diagnose the health condition of the plant. Therefore, λ = 0.05 was set so that the prediction accuracy of the main task would be high. In other words, the loss function of the learning model 45 is set so that the prediction performance of the main task is higher than the prediction performance of the subtasks.
[0047] Figure 7 shows the accuracy comparison results between the learning model 45 of this example with subtasks and a single-task learning model without subtasks. The top-1 accuracy is the probability (accuracy rate) that the most probable prediction result of the multi-value classification task matches the first correct label. In addition, the accuracy rate of whether the prediction result of the multi-value classification task is symptomatic or asymptomatic is shown as the "symptomatic vs. asymptomatic accuracy rate" (prediction of the presence or absence of disease).
[0048] As shown in Figure 7, the accuracy rate of symptom vs. asymptomatic does not differ with or without the subtask, but the first place accuracy rate is higher for the learning model 45 with the subtask. Therefore, it can be said that adding the subtask of predicting the zoom level to the learning model 45 for diagnosing the health condition of plants increases the prediction accuracy. In this way, the subtask improves the generalization performance of the main task.
[0049] [Additional functions of diagnostic equipment] The diagnostic device 30 has a basic function of predicting plant diseases using a learning model 45, but additionally has an application 43 that has a function of outputting diagnostic results based on the prediction results of the learning model 45.
[0050] 8, the application 43 may include one or more of a diagnosis output unit 71, a heat map generating unit 72, a shooting position acquiring unit 73, a position information creating unit 74, a shooting range estimating unit 75, a three-dimensional data generating unit 76, and a planar image generating unit 77, which are optional additional programs. When the processor 31 executes the programs 71 to 77, the diagnostic device 30 operates as each of the functional units described below. In other words, it can be said that the diagnostic device 30 includes the diagnosis output unit 71, the heat map generating unit 72, the shooting position acquiring unit 73, the position information creating unit 74, the shooting range estimating unit 75, the three-dimensional data generating unit 76, and the planar image generating unit 77.
[0051] The diagnosis output unit 71 is configured to be executed by the processor 31 to output a diagnosis result regarding whether or not the plant needs care, based on the prediction result of the learning device 40. If the prediction result of the multi-value classification task is asymptomatic, the diagnosis output unit 71 outputs a diagnosis result that no care is required.
[0052] The diagnosis output unit 71 may output a diagnosis result indicating that care is required when the prediction result of the multi-value classification task is any one of a plurality of diseases. In this case, the output diagnosis result may include the content of care, such as spraying of a drug according to the disease. In one example, the diagnosis output unit 71 is configured to output a diagnosis result indicating that care is required for the plant when the probability of any one of the diseases output by the multi-value classification task exceeds a symptom threshold (e.g., 70%). Alternatively, the diagnosis output unit 71 may output a diagnosis result indicating that care is required for the plant if the probability of the disease exceeds a first symptom threshold (e.g., 90%), and output a diagnosis result indicating that observation is required if the probability of the disease is less than the first symptom threshold (e.g., 90%) and a second symptom threshold (e.g., 70%).
[0053] The diagnosis output unit 71 may output a diagnosis result regarding the necessity of plant care after taking into account the prediction result of the binary classification task into the prediction result of the multi-value classification task. For example, the diagnosis output unit 71 may be configured to output a diagnosis result regarding the necessity of plant care when the prediction result of the binary classification task corresponds to the wide-range image 10B and the prediction result of the multi-value classification task is any one of a plurality of diseases. In this case, the diagnosis output unit 71 may output a diagnosis result that the plant needs care based on the wide-range image 10B. In this case, it is preferable to photograph the ground surface image 10A and the wide-range image 10B as a set at each shooting location. More specifically, when the distribution value related to the spread of the disease in the wide-range image 10B exceeds a threshold value, the diagnosis output unit 71 may output a diagnosis result that the plant needs care.
[0054] After obtaining the prediction result of the multi-value classification task, the diagnosis output unit 71 may perform additional diagnosis and then output a diagnosis result indicating that maintenance is necessary. The additional diagnosis may be performed based on one or more of the type, range, degree, or timing of disease, for example. The additional diagnosis may be realized as an additional function of the learning device 40, or may be realized by executing a program included in the application 43. The prediction result output by the diagnosis output unit 71 is associated with the additional diagnosis and related data as necessary and stored in the storage 36.
[0055] The heat map generating unit 72 is configured to generate a heat map 65 in which prediction grounds for the multi-value classification task are superimposed on the input image 10 as shown in FIG. 9 by being executed by the processor 31. The heat map generating unit 72 can use, for example, Grad-CAM. In this case, the heat map 65 is a feature map of the final output of the multi-value classification task, with the portions that are the basis for prediction being shown in darker colors. The generated heat map 65 is associated with the input image 10 and the prediction result and stored in the storage 36.
[0056] By referring to the heat map 65, it is possible to visualize which part of the image 10 is the basis for the judgment. This allows the user to confirm that the judgment has been made correctly if the judgment is based on dead parts of the diseased grass in the image 10, and to surmise that the prediction is inaccurate if, for example, an artificial object other than grass is the basis for the judgment.
[0057] The photographing position acquisition unit 73 is configured to be executed by the processor 31 to acquire position information related to the photographing position 60 of the input image 10. The position information may be attached to the image 10 by the photographing device 21, for example. The position information acquired by the photographing position acquisition unit 73 is stored in the storage 36. The photographing position 60 may be output or displayed together with the diagnosis result output by the diagnosis output unit 71.
[0058] When the application 43 includes the position information creation unit 74, the storage 36 stores map data 61 including the location where the input image 10 was taken, for example, the golf course 11. The map data 61 may include a map including latitude and longitude, or in the case of the golf course 11, may include a layout diagram unique to the facility, such as a course layout. The position information creation unit 74 is configured to create position information in which the shooting position 60 is plotted on the map data 61 by being executed by the processor 31.
[0059] When using the three-dimensional data generating unit 76, the photographer 20 acquires a plurality of landscape images taken from different viewpoints so as to include the photographing range. The plurality of landscape images may be replaced with a video. A landscape image group including the plurality of landscape images is transmitted from the photographing device 21 to the diagnosis device 30 via the network 23 and stored in the storage 36. The three-dimensional data generating unit 76 is configured to generate three-dimensional data 62 of the target area (e.g., golf course 11) from the landscape image group or the video.
[0060] The planar image generating unit 77 is configured to generate a planar image 63 from the three-dimensional data 62. Specifically, the planar image generating unit 77 generates a planar image 63 which is a plan view of the three-dimensional data 62 seen from above. The position information creating unit 74 may create position information in which the shooting position 60 is plotted on the planar image 63 created by the planar image generating unit 77. The position information creating unit 74 may overlay the planar image 63 on map data 61 including latitude and longitude, and then create position information in which the shooting position 60 is plotted. The planar image 63 including the shooting position 60 of the input image 10 thus generated is stored in the storage 36.
[0061] The shooting range estimation unit 75 is configured to be executed by the processor 31 to estimate the shooting range 64 of the input image 10. The shooting range estimation unit 75 may be configured to extract characteristic objects or feature points from the image 10 and compare them with the three-dimensional data 62. The shooting range estimation unit 75 may then be configured to estimate the shooting position 60 and the shooting range 64 by determining corresponding objects or feature points in the image 10 and the three-dimensional data 62.
[0062] When the image 10 is predicted to be a ground image 10A, the shooting range estimation unit 75 may estimate the shooting position 60 (point data) or a specified area including the shooting position 60 (for example, an area of 30 to 50 cm square) as the shooting range 64 without extracting objects or feature points. The shooting range estimation unit 75 may be configured to overlay the shooting position 60 and the shooting range 64 on the planar image 63.
[0063] The diagnosis output unit 71 may output one or more of the heat map 65, the three-dimensional data 62, the planar image 63, the photographing position 60, or the photographing range 64 as a diagnosis result together with the prediction result. For example, when the prediction result of the main task corresponds to any one of a plurality of diseases, the diagnosis output unit 71 may add information indicating that the disease is predicted to the photographing range 64 of the planar image 63 and output the information. The application 43 may include a program for displaying the diagnosis result output by the diagnosis output unit 71 by combining the results on one screen or by dividing the results on a plurality of switchable screens.
[0064] [Methods for diagnosing plant health] A method for diagnosing the health condition of a plant will be described with reference to Fig. 10. Steps S10 to S14 and S17 are steps relating to displaying the diagnosis results, and steps S8 and S15 to S16 are steps relating to predicting disease damage by the learning device 40. Steps S10 to S14 may be performed after step S16.
[0065] First, in step S8, processor 31 acquires input image 10 showing vegetation covering the ground surface. Subsequent steps S10 to S12 are processes related to the creation of a background map for displaying the prediction result. First, in step S10, processor 31 acquires a landscape image group. In step S11, three-dimensional data generation unit 76 generates three-dimensional data 62 based on the landscape image group. Then, in step S12, planar image generation unit 77 generates planar image 63 based on the three-dimensional data 62. As a result, planar image 63, which becomes the background image, is obtained.
[0066] Steps S13 to S14 are preparation steps for linking the shooting position 60 and the position of the shooting range 64 with the background image. In step S13, the shooting position acquisition unit 73 acquires the shooting position 60. In the following step S14, the shooting range estimation unit 75 estimates the shooting range 64. The estimated shooting position 60 and shooting range 64 may be superimposed on the planar image 63.
[0067] In step S15, the processor 31 inputs the image 10 to the learning device 40. The image 10 input at this time is the same as the image 10 for which the shooting position 60 has been acquired and the shooting range 64 has been estimated in steps S13 and S14. Subsequently, in step S16, the learning device 40 outputs prediction results of the main task and the subtask.
[0068] In step S17, the heat map generator 72 generates a heat map 65 to display the basis for the prediction result in step S16. The generated heat map 65 may be synthesized so as to be aligned with the original image 10.
[0069] In step S18, the diagnosis output unit 71 outputs the diagnosis result. The diagnosis result may include the prediction result of the main task output in step S16 and a diagnosis of whether or not the plant needs care, and may not include the prediction result of the subtask. The prediction result of the main task may be output in association with the shooting position 60 and shooting range 64 superimposed on the planar image 63, the original image 10, and the heat map 65.
[0070] Finally, in step S19, processor 31 transmits the diagnosis result to user terminal 22 through network 23. If diagnostic device 30 does not include diagnosis output unit 71, processor 31 may transmit the prediction result of step S16 to user terminal 22 instead of the diagnosis result of step S18.
[0071] [Effects of the present disclosure] In the case of agricultural or ornamental plants, disease and pests are often judged from the condition of the leaves. However, grasses such as grass have thin leaves, and the condition of the leaves is difficult to see when the grass is cut short. Therefore, when a lawn is infected with disease, the disease is often discovered when the disease has spread to a certain extent and the green area covering the ground becomes partially peeled, withered, discolored, or mottled.
[0072] To diagnose disease symptoms in lawns based on images, it is necessary to capture images of a fairly wide area. However, to determine the type of disease, it is necessary to capture images close enough to the lawn to see the condition of the grass blades.
[0073] In this regard, since the learning device 40 is trained with the training data 59 including both ground surface images and wide-range images, it becomes possible to diagnose the health condition of the plants depicted in the image 10 regardless of whether the input image 10 is a ground surface image 10A or a wide-range image 10B. For the photographer 20, too, management becomes easier since the image 10 can be acquired without being restricted by the shooting range 64.
[0074] Since the learning model 45 of the learning device 40 includes subtasks in addition to the multi-value classification task related to disease diagnosis, the prediction accuracy of the multi-value classification task is higher than that of a learning model that does not include subtasks. This makes it possible to properly diagnose the health of the lawn based on the image 10.
[0075] [Effects of this disclosure] According to the present disclosure, the following effects can be achieved. (1) By adding a binary classification task to a multi-value classification task, the prediction accuracy of the multi-value classification task can be improved.
[0076] (2) The diagnosis output unit 71 can output a diagnosis result regarding the necessity of caring for the plant based on the prediction result of the learning device 40. (3) By outputting the prediction results of a multi-class classification task as a probability, it is possible to show the user an indicator of the reliability of the prediction results.
[0077] (4) When the prediction result of the binary classification task corresponds to the wide-range image 10B and the prediction result of the multi-value classification task is one of a plurality of diseases, it is possible to determine the extent of the disease based on the wide-range image 10B that is the basis of the prediction. Therefore, it is possible to diagnose that the plant needs care based on the spread of the disease grasped from the wide-range image 10B.
[0078] (5) By comparing the heat map 65 with the input image 10, the user can confirm which part of the image 10 was the basis for the output of the prediction result. (6) By acquiring location information related to the photographing location 60 of the image 10, the photographing location can be associated with the prediction result.
[0079] (7) By creating location information by plotting the photographing location 60 on map data, the user can easily manage plants based on the diagnosis results. (8) By generating a planar image 63 based on the three-dimensional data 62, the environment in which the plant grows can be illustrated.
[0080] (9) The area related to diagnosis can be illustrated by superimposing the shooting range 64 of the wide-range image 10B on the planar image 63. This makes it easier for the user to manage the plant. [Example of change] This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction.
[0081] [Modification 1] One or more ground surface images 10A and one or more wide-range images 10B captured at the same capturing position 60 may be input as a set to the learning device 40. In this case, the type and presence or absence of disease may be determined based on the prediction result of the multi-value classification task when the ground surface image 10A is input, and the diagnosis output unit 71 may output a diagnosis result regarding the need for plant care based on the wide-range image 10B.
[0082] [Modification 2] The subtask may be changed from a binary classification task to a task that performs classification into three or more classes. For example, three or more zoom levels may be set, or a three-class classification may be performed by adding aerial images taken from the sky by an aircraft such as a drone to ground images and wide-area images taken from the ground at different angles.
[0083] [Modification 3] The learning device 40 may be a single-task learning device. The single-task learning device may be a multi-modal model that inputs an image and other information. For example, along with the input image 10, whether the image 10 is a ground image or a wide-range image may also be input to solve a multi-value classification task, which is a single task. Other examples of information to be added to the image include the conditions at the time of shooting, such as temperature, weather, location, and season.
[0084] [Modification 4] The diagnostic device 30 may include a plurality of single-task learners instead of the multi-task learner 40. For example, the diagnostic device 30 may include a first learner trained on a group of surface images for making predictions related to plant diseases, and a second learner trained on a group of wide-area images for making predictions related to plant diseases. Furthermore, the diagnostic device 30 may include a third learner for predicting the zoom level of an image. In this case, after the third learner predicts whether the input image corresponds to a surface image or a wide-area image, if the input image is a surface image, it may be input to the first learner, and if the input image is a wide-area image, it may be input to the second learner to obtain a prediction result related to plant diseases.
[0085] [Modification 5] The learning device 40 may be a multitask learning device including three or more tasks. The additional tasks may be, for example, prediction of the type of plant, prediction of the management state of the plant (for example, whether the lawn has been mowed), prediction of diseased areas, prediction of the ratio of diseased areas to harmless areas, and prediction of the season when the image was taken. When predicting the type of plant, a single diagnostic device 30 can diagnose a plurality of plants. Examples of plants other than the grass include ground cover plants such as Diachondra, Rockweed, Clavia, and White Clover. Alternatively, different types of grass, such as wild grass, Korean grass, and Western grass, may be diagnosed separately, or may be classified according to growth type, such as summer grass and winter grass.
[0086] [Modification 6] A portion of a landscape image may be extracted to generate one or more ground images and one or more wide-range images for input or training. [Variation 7] In addition to disease, the unhealthy condition may include one or more of the following: symptoms of insect damage, withering, and physical damage (such as hollows in the ground).
[0087] [Modification 8] The diagnostic device 30 may also function as the information processing device 50. In other words, the computer that is the diagnostic device 30 may generate (train) the learning model 45. [Modification 9] The learning device 40 may be provided as a diagnostic program for diagnosing the health condition of a plant together with at least a part of the application 43. The diagnostic program causes one or more computers to acquire an input image 10 depicting a plant covering the ground surface, input the acquired input image 10 to the learning device 40, and output a plurality of prediction results associated with a plurality of tasks, respectively. The diagnostic program may be downloaded to a personal computer, tablet, or smartphone such as the user terminal 22 and used as a diagnostic application. The diagnostic program may be used as a cloud application provided by the diagnostic device 30 via the network 23.
[0088] [Modification 10] In addition to the ground surface image group and the wide-range image group, the training data 59 may include attribute information related to the user or photographing device (for example, a camera or an unmanned vehicle such as a drone used for photographing) who photographed the ground surface image. This attribute information may include at least one of the following information 1) to 4).
[0089] 1) Photographing subject identification information indicating whether the subject who photographed the ground image is a human or not; 2) Photographer information including at least one of the gender and age group of the person who took the surface image; 3) biographical information including at least one of a medical history and a golfing history of the person who captured the ground image; 4) Equipment information including at least one of the year and model number of the imaging device (e.g., drone) that captured the ground image.
[0090] The learning device 40 may train the learning model 45 based on training data 59 including such attribute information. In this case, the diagnostic device 30 may obtain a diagnostic result by inputting the attribute information in addition to the input image 10 to the input layer of the learning model 45.
[0091] Aspects grasped based on the above embodiment and modified examples are listed below. [1] A diagnostic device for diagnosing the health condition of a plant, the diagnostic device comprising: a multi-task learner configured to receive an input image and output a plurality of prediction results associated with a plurality of tasks, The multi-task learner comprises: a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image, Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; Diagnostic equipment.
[0092] [2] A diagnosis output unit configured to output a diagnosis result regarding whether or not the plant needs care based on a prediction result of the multi-task learning device. The diagnostic device according to [1] above.
[0093] [3] The multi-class classification task is configured to output probabilities corresponding to the plurality of diseases or asymptomatic diseases as a prediction result; The diagnosis output unit is configured to output a diagnosis result indicating that care of the plant is required when the probability of any of the diseases output by the multi-value classification task exceeds a symptom threshold. The diagnostic device according to [2] above.
[0094] [4] The diagnosis output unit is configured to output, when a prediction result of the binary classification task corresponds to the wide-range image and a prediction result of the multi-value classification task is any one of the plurality of diseases, a diagnosis result that the plant needs care based on the wide-range image. The diagnostic device according to the above [2] or [3].
[0095] [5] A heat map generator configured to generate a heat map in which a prediction basis of the multi-class classification task is superimposed on the input image. The diagnostic device according to any one of the above [1] to [4].
[0096] [6] A photographing position acquisition unit configured to acquire position information relating to a photographing position of the input image; A location information generating unit; one or more memories; The memory stores map data including the photographing position of the input image, the location information creation unit is configured to create location information by plotting the shooting location on the map data. The diagnostic device according to any one of the above [1] to [5].
[0097] [7] A three-dimensional data generating unit configured to generate three-dimensional data from a landscape image group including a plurality of landscape images captured from different viewpoints; a planar image generator configured to generate a planar image from the three-dimensional data; The diagnostic device according to any one of the above [1] to [6].
[0098] [8] A shooting range estimation unit configured to estimate a shooting range of the input image. The diagnostic device according to any one of the above [1] to [7].
[0099] [9] The multi-task learner includes a learning model using a convolutional neural network; A loss function of the learning model is set so that the predictive performance of the multi-value classification task is higher than the predictive performance of the binary classification task. The diagnostic device according to any one of the above [1] to [8].
[0100]
[10] The plant is a ground cover plant that grows in clusters to cover the ground surface. The diagnostic device according to any one of the above [1] to [9].
[0101]
[11] The plant is grass. The plurality of diseases include one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, or snow mold; The diagnostic device according to any one of the above [1] to
[10] .
[0102]
[12] The ground surface image is an image in which the ground surface covered with the vegetation occupies an entire shooting range, The wide-range image is a landscape image that includes at least one of trees growing on the ground surface and artificial objects including buildings in addition to the ground surface covered with vegetation. The diagnostic device according to any one of the above [1] to
[11] .
[0103]
[13] A method for diagnosing the health of a plant, the method comprising: acquiring an input image showing the vegetation covering the ground surface; inputting the input image into a multitask learner; the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks; Including, The multi-task learner comprises: a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image, Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; Diagnostic methods.
[0104]
[14] A method for generating a learning model for diagnosing a health condition of a plant, the learning model being configured to output a plurality of prediction results associated with a plurality of tasks when an input image is input; The method for producing the same comprises: Obtaining a group of images that serve as training data; training the learning model using the training data; The training data is a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image; Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; How to generate a learning model.
[0105]
[15] On one or more computers, acquiring an input image showing vegetation covering a ground surface; inputting the acquired input image into a multitask learning device; the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks; A program for plant diagnosis, The multi-task learner comprises: a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image, Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; program.
[0106]
[16] A diagnostic device for diagnosing a health condition of a plant, the diagnostic device comprising: One or more processors and one or more memories; The memory stores a multi-task learner configured to receive an input image and output a plurality of prediction results associated with a plurality of tasks, The multi-task learner comprises: a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image, Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; Diagnostic equipment.
[0107]
[17] the processor is configured to perform one or more of a plurality of operations; The plurality of processes include: A process of inputting the input image to the multi-task learner and outputting the plurality of prediction results; outputting a diagnosis result regarding whether or not the plant needs care based on the prediction result of the multi-task learner; A process of generating a heat map in which a prediction basis of the multi-class classification task is superimposed on the input image; A process of acquiring position information relating to a photographing position of the input image; The diagnostic device according to
[16] above,
[0108]
[18] An input layer to which an input image is input; A hidden layer trained with the training data; a plurality of output layers respectively associated with a plurality of tasks; A learning model for causing a computer to function in such a way that an input image showing plants covering a ground surface is input to the input layer, and a plurality of prediction results respectively associated with the plurality of tasks are output from the plurality of output layers, The training data is a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image; Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; Learning model. [Explanation of symbols]
[0109] 10...image (input image), 10A (10A1 to 10A7)...surface image, 10B...wide-range image, 11...golf course, 12...green, 13...cup hole, 14...bunker, 15...pond, 16...trees, 17...management facility, 18...grass, 20...photographer, 21...photographing device, 22...user terminal, 23...network, 30...diagnostic device, 31...processor, 32...memory, 33...communication IF, 34...input device, 35...output device, 36...storage, 37...communication bus, 40...learning device, 41...image data, 43...application, 44...output data, 45...learning model, 46...common layer , 47 (47A, 47B)...output layer, 48 (48A, 48B)...prediction result, 50...information processing device, 51...processor, 52...memory, 53...communication IF, 54...input device, 55...output device, 56...storage, 57...communication bus, 58...learning program, 59...training data, 60...shooting position, 61...map data, 62...3D data, 63...planar image, 64...shooting range, 65...heat map, 71...diagnosis output unit, 72...heat map generation unit, 73...shooting position acquisition unit, 74...position information creation unit, 75...shooting range estimation unit, 76...3D data generation unit, 77...planar image generation unit.
Claims
1. A diagnostic device for diagnosing a health condition of a plant, the diagnostic device comprising: a multi-task learner configured to receive an input image and output a plurality of prediction results associated with a plurality of tasks; a capturing range estimating unit configured to estimate a capturing range of the input image; Equipped with The multi-task learner comprises: a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image, Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; The shooting range estimation unit is when a prediction result of the binary classification task is the wide-range image, a characteristic object or feature point is extracted from the input image, the extracted characteristic object or feature point is compared with three-dimensional data generated from a plurality of landscape images taken from different viewpoints in the region, and corresponding objects or feature points between the input image and the three-dimensional data are identified, thereby estimating the shooting range; When a prediction result of the binary classification task is the ground image, a specified area including a shooting position of the input image is estimated as the shooting range without extracting the characteristic object or feature point from the input image. Diagnostic equipment.
2. a diagnosis output unit configured to output a diagnosis result regarding whether or not the plant needs care based on a prediction result of the multi-task learning device; The diagnostic device of claim 1 .
3. The multi-value classification task is configured to output, as a prediction result, probabilities corresponding to the plurality of diseases or asymptomatic diseases; The diagnosis output unit is configured to output a diagnosis result indicating that care of the plant is required when the probability of any of the diseases output by the multi-value classification task exceeds a symptom threshold. The diagnostic device of claim 2.
4. the diagnosis output unit is configured to output, when a prediction result of the binary classification task corresponds to the wide-range image and a prediction result of the multi-value classification task is any one of the plurality of diseases, a diagnosis result that the plant needs care based on the wide-range image. The diagnostic device of claim 2.
5. A heat map generator configured to generate a heat map in which a prediction basis of the multi-class classification task is superimposed on the input image. A diagnostic device according to any one of claims 1 to 4.
6. a photographing position acquisition unit configured to acquire position information related to the photographing position of the input image; A location information generating unit; one or more memories; The memory stores map data including the photographing position of the input image, the location information creation unit is configured to create location information by plotting the shooting location on the map data. A diagnostic device according to any one of claims 1 to 4.
7. A three-dimensional data generation unit configured to generate the three-dimensional data from a landscape image group including the plurality of landscape images; a planar image generator configured to generate a planar image from the three-dimensional data; The diagnostic device according to any one of claims 1 to 4, comprising:
8. The input image is an image of grass laid on a golf course, the ground surface image is an image of the ground surface covered with grass, the wide-range image is a landscape image that captures at least one of a natural object or an artificial object in or around the lawn in addition to the ground surface covered with the lawn, the natural object is a boundary line of a green, a bunker, a pond, or trees growing around the green; The artifact is a cup hole or a pin. A diagnostic device according to any one of claims 1 to 4.
9. The multi-task learner includes a learning model using a convolutional neural network; a loss function of the learning model is set so that the predictive performance of the multi-value classification task is higher than the predictive performance of the binary classification task. A diagnostic device according to any one of claims 1 to 4.
10. The plant is a ground cover plant that grows in clusters to cover the ground surface. A diagnostic device according to any one of claims 1 to 4.
11. the plant is grass; The plurality of diseases include one or more of anthracnose, brown patch, dollar spot, dry spot, pythium, or snow mold; A diagnostic device according to any one of claims 1 to 4.
12. the ground surface image is an image in which the ground surface covered with the vegetation occupies an entire shooting range, The wide-range image is a landscape image that includes at least one of trees growing on the ground surface and artificial objects including buildings in addition to the ground surface covered with vegetation. A diagnostic device according to any one of claims 1 to 4.
13. A method for diagnosing a plant's health, the method comprising: acquiring an input image showing the vegetation covering the ground surface; inputting the input image into a multitask learner; the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks; Estimating a capture range of the input image; Including, The multi-task learner comprises: a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image, Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; The estimating of the imaging range includes: When the predicted result of the binary classification task is the wide-range image, extracting characteristic objects or feature points from the input image, comparing the extracted characteristic objects or feature points with three-dimensional data generated from a plurality of landscape images taken in the region from different viewpoints, and determining corresponding objects or feature points between the input image and the three-dimensional data, thereby estimating the shooting range; When a prediction result of the binary classification task is the ground image, estimating a specified area including a shooting position of the input image as the shooting range without extracting the characteristic object or feature point from the input image. Diagnostic methods.
14. On one or more computers, acquiring an input image showing vegetation covering a ground surface; inputting the acquired input image into a multitask learning device; the multi-task learner outputs a plurality of prediction results respectively associated with a plurality of tasks; Estimating a capture range of the input image; A program for plant diagnosis that executes the following: The multi-task learner comprises: a ground surface image group including a plurality of ground surface images each showing the vegetation covering the ground surface; and a wide-range image group including a plurality of wide-range images capturing the plants in an area wider than the ground surface image, Each of the ground surface image group and the wide range image group is A plurality of asymptomatic images of the plant in a healthy state; and a plurality of symptomatic images of the plant in an unhealthy state; The unhealthy condition includes symptoms of multiple diseases; The plurality of tasks include: A multi-value classification task of predicting whether the plant depicted in the input image corresponds to one of the multiple diseases or asymptomatic diseases; a binary classification task of predicting whether the input image corresponds to the ground image or the wide-area image; The estimating of the imaging range includes: When the predicted result of the binary classification task is the wide-range image, extracting characteristic objects or feature points from the input image, comparing the extracted characteristic objects or feature points with three-dimensional data generated from a plurality of landscape images taken in the region from different viewpoints, and determining corresponding objects or feature points between the input image and the three-dimensional data, thereby estimating the shooting range; When a prediction result of the binary classification task is the ground image, estimating a specified area including a shooting position of the input image as the shooting range without extracting the characteristic object or feature point from the input image. program.
Citation Information
Patent Citations
Lawn disease and insect damage predicting and diagnosing device
JP1994035897A
Plant disease diagnostic system, plant disease diagnostic method, and program
JP2016168046A
Disease and insect damage diagnostic device, disease and insect damage diagnostic method, disease and insect damage diagnostic program, model generation device, model generation method, and model generation program
JP2022094783A
