Diagnostic system, diagnostic method, and diagnostic program

Abstract

To realize a technology for suitably recognizing the damaged parts of plants and for accurately diagnosing the conditions of the plant.SOLUTION: A diagnostic system (100) for diagnosing plant conditions comprises: an estimation device (10) including an estimation unit (13) for estimating the state of a test plant using a three-dimensional image of a damaged part of the test plant as input information using an estimation model; and a diagnostic device (20) comprising a diagnostic unit (23) for diagnosing the conditions of the test plant with reference to the estimation result representing the conditions of the test plant.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a diagnostic system, a diagnostic method, a diagnostic program, and a learning model generation device. [Background technology]

[0002] Due to the progression of global warming and the globalization of human movement and logistics, there are concerns that transboundary pests and livestock infectious diseases that have caused great damage overseas will enter Japan. Once these pests spread, they can cause great damage and require control work, so early containment is important. However, while there are only a limited number of people with the specialized knowledge to diagnose, it is necessary to establish a system to ensure early diagnosis and prevent the spread of diseases. In addition, with remote control and digitalization rapidly progressing, there is a demand for the establishment of technology to identify and diagnose pests and diseases from remote locations.

[0003] Patent Document 1 describes a technology for diagnosing a plant disease by inputting an image of a plant to be diagnosed into a trained model generated by performing machine learning using image data of the plant disease and corresponding diagnosis result information as training data. Patent Document 2 describes a technology for diagnosing a plant disease by referring to cultivation history information of the plant in addition to an image of the plant to be diagnosed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-168046 A [Patent Document 2] JP 2020-149307 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technologies described in Patent Documents 1 and 2 diagnose plant diseases from plant images using a trained model, making it possible to diagnose without a diagnostician with specialized knowledge and also enabling diagnosis from a remote location. However, it is not easy to recognize three-dimensional damaged areas such as depressions or the depth of scratches on the plant surface from plant images, and there is a possibility that information required for diagnosis cannot be obtained.

[0006] One aspect of the present invention aims to realize a technology for appropriately recognizing damaged parts of a plant and accurately diagnosing the condition of the plant. [Means for solving the problem]

[0007] A diagnostic system according to one embodiment of the present invention is a diagnostic system for diagnosing the condition of a plant, and includes an estimation device having an estimation unit that estimates the condition of a test plant using an estimation model generated by performing machine learning using three-dimensional image data of a damaged portion of the plant and information representing a diagnosis result indicating the condition of the plant as learning data, and a diagnostic device having a diagnosis unit that diagnoses the condition of the test plant by referring to the estimation result indicating the condition of the test plant estimated by the estimation device.

[0008] A diagnostic method according to one embodiment of the present invention is a method for diagnosing the condition of a plant, and includes an estimation step of estimating the condition of a test plant by using a three-dimensional image of a damaged portion of the plant as input information using an estimation model generated by performing machine learning using three-dimensional image data of the damaged portion of the plant and information representing a diagnosis result indicating the condition of the plant as learning data, and a diagnosis step of diagnosing the condition of the test plant by referring to the estimation result indicating the condition of the test plant estimated in the estimation step.

[0009] A diagnostic program according to one embodiment of the present invention is a diagnostic program for causing a computer to function as the above-mentioned diagnostic system, and executes an estimation step of estimating the condition of a test plant using a three-dimensional image of the damaged portion of the test plant as input information, using an estimation model generated by performing machine learning using three-dimensional image data of the damaged portion of the plant and information representing a diagnosis result indicating the condition of the plant as learning data, and a diagnosis step of diagnosing the condition of the test plant by referring to the estimation result indicating the condition of the test plant estimated in the estimation step.

[0010] A diagnostic system according to another aspect of the present invention is a diagnostic system for diagnosing the condition of a plant, and includes a cultivation condition estimation device having a cultivation condition estimation unit that estimates the cultivation conditions of a test plant by using a three-dimensional image captured of the environment surrounding the plant as input information, using a cultivation condition estimation model generated by performing machine learning using three-dimensional image data captured of the environment surrounding the plant and information representing the cultivation conditions of the plant as learning data, and a diagnostic device that diagnoses the condition of the test plant by referring to the cultivation conditions estimated by the cultivation condition estimation device. A learning model generation device according to one embodiment of the present invention includes a generation unit that generates an estimation model by performing machine learning using three-dimensional image data of a damaged portion of a plant and information representing a diagnosis result showing the condition of the plant as learning data.

[0011] Each device of the diagnostic system according to each aspect of the present invention may be realized by a computer. In this case, the control program of each device that causes the computer to operate as each part (software element) of each device to realize each device on a computer, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present invention. Effect of the Invention

[0012] According to one aspect of the present invention, it is possible to appropriately recognize damaged parts of a plant and accurately diagnose the condition of the plant. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of a configuration of a main part of a diagnostic system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating an overview of a diagnostic system. [Diagram 3] FIG. 13 is a diagram showing an example of learning data for generating an estimation model used in the estimation device. [Figure 4] FIG. 11 is a diagram showing an example of learning data for generating a cultivation condition estimation model used in the cultivation condition estimation device. [Diagram 5] 1 is a flowchart illustrating a flow of a diagnostic process in the diagnostic system. [Figure 6] 11 is a flowchart illustrating a flow of an estimation process in the estimation device. [Figure 7] 10 is a flowchart illustrating a flow of a cultivation condition estimation process in the cultivation condition estimation device. [Figure 8] FIG. 2 is a diagram illustrating a concept of request processing in the diagnostic system. [Figure 9] 11 is a flowchart illustrating the flow of a request process in the diagnostic system. [Figure 10] 11 is a flowchart illustrating a flow of a switching process in the diagnostic system. [Figure 11] 11 is a flowchart illustrating the flow of a learning process in the learning model generating device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Diagnostic System] A diagnostic system 100 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of a configuration of a main part of the diagnostic system 100 according to an embodiment of the present invention.

[0015] A diagnostic system 100 according to one embodiment of the present invention is a diagnostic system that diagnoses the condition of a plant. The condition of a plant diagnosed by the diagnostic system includes, but is not limited to, the presence or absence of a plant disease, the degree of the plant disease, the presence or absence of damage by pests, the degree of damage by pests, the presence or absence of poor growth, the degree of poor growth, bird damage, plant vigor, etc.

[0016] The diagnostic system 100 may be provided for each plant crop (e.g., for apples, tomatoes, eggplants, strawberries, cucumbers, etc.) or may target multiple crops at once.

[0017] The diagnostic system 100 may be used by service providers such as companies or organizations that offer a service to diagnose the condition of plants, and may also be used by agricultural workers who receive the service and use the results of the diagnosis of the condition of plants.

[0018] An overview of the diagnostic system 100 will be described with reference to Fig. 2. Fig. 2 is a diagram for explaining an overview of the diagnostic system 100. In the diagnostic system 100, first, a three-dimensional image of a damaged part of a plant photographed by a producer (photographer) in a certain production field 1 is sent to a diagnostician or diagnostic device located away from the production field 1. The diagnostician, for example, observes the three-dimensional image displayed on a VR (virtual reality) display device from any viewpoint in VR and diagnoses the condition of the plant.

[0019] The diagnostic device uses a trained model that has been machine-learned using, for example, three-dimensional image data and diagnostic result information as learning data, and outputs a diagnostic result for the condition of the plant using the three-dimensional image as input information. The diagnostic result by the diagnostician or diagnostic device is sent to the photographer in the production field 1, and is displayed, for example, superimposed on an AR (augmented reality) display device. The diagnostic result is shared between the diagnostician and the diagnostic device. Similarly, three-dimensional images from other production fields are also sent to the diagnostician or diagnostic device for diagnosis.

[0020] In this way, the diagnostic system 100 uses three-dimensional images, so that it is possible to recognize three-dimensional damaged areas such as depressions and the depth of scratches on the surface of a plant. In addition, diagnosis is possible even if a diagnostician with specialized knowledge is not present at the production field, and diagnosis from a remote location is also possible. Furthermore, by sharing the diagnosis results between the diagnostician and the diagnostic device, it is possible to improve the accuracy of diagnosis. In addition, by accepting three-dimensional images from multiple fields, it is also possible to provide diagnostic support for multiple fields at the same time.

[0021] As shown in Fig. 1, the diagnostic system 100 includes an estimation device 10 and a diagnostic device 20. The diagnostic system 100 may include, as components, a photographer-side system 30, a diagnoser-side system 40, a cultivation condition estimation device 50, a learning model generation device 60, and a storage device 70. The diagnostic system 100 may include the estimation device 10 and the diagnostic device 20 as an integrated device, or may include each of them as separate devices. The diagnostic system 100 may include the estimation device 10, the diagnostic device 20, the cultivation condition estimation device 50, the learning model generation device 60, and the storage device 70 as an integrated device.

[0022] (Estimation device 10) The estimation device 10 uses an estimation model to estimate the state of a test plant from a three-dimensional image obtained by photographing a damaged portion of the test plant.

[0023] The estimation device 10 includes a control unit 11. The control unit 11 controls each part of the estimation device 10, and is realized by a processor and a memory, for example. In this example, the processor accesses a storage (not shown), loads a program (not shown) stored in the storage into the memory, and executes a series of instructions included in the program. This constitutes each part of the control unit 11. As each part, the control unit 11 includes a three-dimensional image acquisition unit 12, an estimation unit 13, and a display control unit 15.

[0024] The three-dimensional image acquisition unit 12 acquires a three-dimensional image of a damaged part of a test plant. The three-dimensional image acquisition unit 12 acquires a three-dimensional image captured by an imaging device 31 of a photographer-side system 30, which will be described later. The three-dimensional image acquisition unit 12 outputs the acquired three-dimensional image to the estimation unit 13.

[0025] The estimation unit 13 estimates the condition of the test plant using an estimation model generated by performing machine learning using three-dimensional image data of the damaged part of the plant and information representing the diagnosis result showing the condition of the plant as learning data. The estimation unit 13 acquires the estimation model generated by the learning model generation device 60 described later, inputs the three-dimensional image data acquired from the three-dimensional image acquisition unit 12, and acquires information representing the condition of the test plant to be output. The estimation unit 13 outputs the acquired condition of the test plant to the diagnosis device 20 as an estimation result.

[0026] Here, the learning data used to generate the estimation model will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the learning data 200 for generating the estimation model used in the estimation device. The learning data 200 includes three-dimensional image data of a damaged part of a plant and diagnostic result information of the plant. The diagnostic result is a diagnostic result of the state of the plant diagnosed by a diagnosing person or the diagnostic device 20 for the plant in the three-dimensional image. By using an estimation model generated by performing machine learning using such learning data 200, the state of the test plant is output using a three-dimensional image of the damaged part of the test plant as input information. Therefore, it is possible to estimate the state of the test plant based on the three-dimensional damaged part such as a depression or the depth of a scratch on the surface of the test plant.

[0027] The determination unit 14 determines whether the estimation accuracy when the three-dimensional image of the damaged part of the test plant is used as input information is less than a predetermined threshold. If the determination unit 14 determines that the estimation accuracy is less than the predetermined threshold, it outputs the determination result to the display control unit 15. If the determination unit 14 determines that the estimation accuracy is equal to or greater than the predetermined threshold, it outputs the determination result to the diagnosis device 20.

[0028] When the determination unit 14 determines that the damage is less than the predetermined threshold, the display control unit 15 causes the photographer-side display device to display information requesting a three-dimensional image of a part of the test plant other than the damaged part. In this way, when the accuracy of the estimation result using a certain three-dimensional image is low, a diagnosis with higher accuracy can be made by requesting the photographer to obtain a three-dimensional image for more accurate estimation. For example, a three-dimensional image showing the damaged part more clearly or a three-dimensional image of a normal part of the plant can be requested, and these images can be used for estimation in the estimation unit 13.

[0029] (Diagnostic device 20) The diagnosis device 20 diagnoses the condition of the test plant by referring to the estimation result indicating the condition of the test plant estimated by the estimation device 10.

[0030] The diagnostic device 20 includes a control unit 21. The control unit 21 controls each part of the diagnostic device 20, and is realized by a processor and a memory, for example. In this example, the processor accesses a storage (not shown), loads a program (not shown) stored in the storage into the memory, and executes a series of instructions included in the program. This constitutes each part of the control unit 21. As each part, the control unit 21 includes a data acquisition unit 22, a diagnosis unit 23, and a display control unit 24.

[0031] The data acquiring unit 22 acquires the estimation result sent from the estimation device 10 or the diagnoser-side system 40. The data acquiring unit 22 outputs the acquired estimation result to the diagnosing unit .

[0032] The diagnosis unit 23 diagnoses the condition of the test plant by referring to the estimation result indicating the condition of the test plant estimated by the estimation device 10. The diagnosis unit 23 also diagnoses the condition of the test plant by referring to the diagnosis result indicating the condition of the test plant sent from the diagnoser's system 40. That is, the diagnosis unit 23 diagnoses the condition of the test plant by referring to at least one of the estimation result indicating the condition of the test plant estimated by the estimation device and the diagnosis result received by the input device. The diagnosis unit 23 outputs the diagnosis result to the display control unit 24 and the storage device 70.

[0033] When the estimation accuracy by the estimation device is less than a predetermined threshold, the diagnosis unit 23 diagnoses the state of the test plant by referring to the diagnosis result received by the input device. That is, the diagnosis unit 23 diagnoses the state of the test plant by referring to either the estimation result by the estimation device 10 or the diagnosis result by the diagnoser, depending on the estimation accuracy. When the estimation accuracy by the estimation device 10 is high, diagnosis by the diagnoser is not necessary, and more systematic diagnosis is possible. On the other hand, when estimation by the estimation device 10 is difficult, such as when a disease or physiological disorder with insufficient case data occurs in a plant, diagnosis by the diagnoser allows continuous diagnosis without interruption. In this way, more suitable diagnosis is possible by appropriately switching between automatic diagnosis by the diagnosis device 20 and manual diagnosis by the diagnoser.

[0034] The diagnosis unit 23 further refers to the estimated result of the cultivation conditions estimated by the cultivation condition estimation device to diagnose the state of the subject plant. Here, the cultivation conditions include, but are not limited to, sunlight conditions, humidity conditions, wind conditions (ventilation, strong winds, etc.). The diagnosis unit 23 diagnoses the state of the subject plant by referring to the estimated result of the cultivation conditions together with a three-dimensional image of the damaged part of the subject plant. For example, brown spot disease is likely to occur when the cultivation conditions are poorly drained, so by making a diagnosis with reference to such information, a more accurate diagnosis is possible. In addition, by referring to the cultivation conditions of the subject plant, it is possible to diagnose the early stage of damage occurrence or the stage where the damage is minor.

[0035] The display control unit 24 controls the display on the photographer-side display device 32 of the photographer-side system 30. The display control unit 24 outputs information representing the diagnosis result by the diagnosis unit 23 to the photographer-side display device 32, and causes the photographer-side display device 32 to display it.

[0036] When the input device receives input information from a diagnosing person requesting a three-dimensional image of a part of the test plant other than the damaged part, the display control unit 24 causes the operator's display device to display information representing the request from the diagnosing person. In this way, when the input device receives that the diagnosing person has determined that an accurate diagnosis cannot be made using a certain three-dimensional image, the display control unit 24 requests the operator to obtain a three-dimensional image for a more accurate diagnosis. For example, by requesting a three-dimensional image that more clearly shows the damaged part or a three-dimensional image of a normal part of the plant, the diagnosing person can use these images in the diagnosis and make a more accurate diagnosis.

[0037] (Photographer's system 30) The operator-side system 30 includes an image capturing device 31 and an operator-side display device 32 .

[0038] The imaging device 31 captures a three-dimensional image showing a damaged part of the test plant. The imaging device 31 may be, for example, a three-dimensional scanner. The imaging device 31 may also be a device that synthesizes a three-dimensional image from a plurality of two-dimensional images captured by an imaging device built into a smartphone. The imaging device 31 outputs the captured three-dimensional image to the estimation device 10 and the diagnostician system 40.

[0039] The photographer-side display device 32 is a display device that is visually recognized by the photographer of the test plant. The photographer-side display device 32 displays the diagnosis result acquired from the diagnosis device 20 and provides the diagnosis result to the photographer. The photographer-side display device 32 may be a monitor of a smartphone, a monitor connected to a PC, a wearable AR display device such as glasses, etc. When the photographer-side display device 32 is an AR display device, it displays an image in which the diagnosis result is superimposed on the test plant.

[0040] The operator-side display device 32 also displays a request for a three-dimensional image of a part of the subject plant other than the damaged part from the estimation device 10 and the diagnoser. If the operator-side display device 32 is an AR display device, it displays an image of the request superimposed on a part of the subject plant other than the damaged part that is requested.

[0041] (Diagnostician System 40) Diagnostician's side system 4 0 The diagnostician side display device 41 and the input device 42 are provided.

[0042] The diagnostician-side display device 41 acquires a three-dimensional image of the test plant and displays it to the diagnostician. The diagnostician-side display device 41 may be a VR display device that displays the three-dimensional image of the test plant in a virtual space. By using the VR display device, the diagnostician can observe the three-dimensional image of the test plant projected in the virtual space from various viewpoints and directions, which allows the diagnostician to observe shallow scratches and dents on the test plant as well as the back and sides of the damaged area.

[0043] The input device 42 accepts input information of a diagnostic result indicating the state of the test plant diagnosed by a diagnoser with reference to the three-dimensional image of the test plant. The input device 42 may be, for example, a keyboard, a mouse, a touch panel, or the like. The diagnoser inputs, via the input device 42, the diagnostic result of the state of the test plant diagnosed with reference to the three-dimensional image of the test plant displayed on the diagnoser-side display device 41. The input device 42 outputs the input information indicating the input from the diagnoser to the diagnoser device 20.

[0044] The input device 42 receives an input from a diagnoser requesting a three-dimensional image of a part of the test plant other than the damaged part. If the diagnoser determines that an accurate diagnosis cannot be made by referring to a certain three-dimensional image, the diagnoser inputs a request for a three-dimensional image for a more accurate diagnosis via the input device 42. For example, by requesting a three-dimensional image that more clearly shows the damaged part or a three-dimensional image of a normal part of the plant, the diagnoser can use these images in the diagnosis, enabling a more accurate diagnosis.

[0045] (Storage device 70) The storage device 70 stores programs and data used in the diagnosis system 100. The storage device 70 stores the diagnosis results by the diagnosis device 20, requests for three-dimensional images of parts other than the damaged part from the estimation device 10 and the diagnosis device 20, the estimation results of the cultivation conditions by the cultivation condition estimation device 50, learning data used for machine learning by the learning model generation device, three-dimensional images captured by the imaging device 31, and the like.

[0046] (Cultivation condition estimation device 50) The cultivation condition estimation device 50 uses a cultivation condition estimation model to estimate the cultivation conditions of a test plant from an image captured of the environment surrounding the test plant.

[0047] The cultivation condition estimation device 50 includes a control unit 51. The control unit 51 controls each unit of the cultivation condition estimation device 50, and is realized by a processor and a memory, for example. In this example, the processor accesses a storage (not shown), loads a program (not shown) stored in the storage into the memory, and executes a series of instructions included in the program. This constitutes each unit of the control unit 51. As each unit, the control unit 51 includes a data acquisition unit 52 and an estimation unit (cultivation condition estimation unit) 53.

[0048] The data acquisition unit 52 acquires an image of the environment surrounding the test plant. Such an image is an image captured by the imaging device 31 of the photographer side system 30, and may be a two-dimensional image or a three-dimensional image. Such an image may be, for example, an image showing the soil of the test plant, or an image showing at least a part (walls, pillars, windows, etc.) of the facility in which the test plant is cultivated. The data acquisition unit 52 outputs the acquired image to the estimation unit 53.

[0049] The estimation unit 53 estimates the cultivation conditions of the test plant using a cultivation condition estimation model generated by performing machine learning using image data of the environment around the plant and information indicating the cultivation conditions of the plant as learning data, and using an image of the environment around the test plant as input information. The estimation unit 53 acquires a cultivation condition estimation model generated by a learning model generation device 60 described later, inputs the image acquired from the data acquisition unit 52, and acquires the cultivation conditions of the test plant to be output. The estimation unit 53 outputs the acquired cultivation conditions of the test plant to the diagnosis device 20 as an estimation result.

[0050] Here, learning data used to generate a cultivation condition estimation model will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of learning data for generating a cultivation condition estimation model used in a cultivation condition estimation device. The learning data 300 includes image data of an environment around a plant and cultivation condition information represented by the image. The cultivation conditions are cultivation conditions specified by a photographer for the environment of the image, or cultivation conditions estimated by a diagnostician or the cultivation condition estimation device 50. If a cultivation condition estimation model generated by performing machine learning using such learning data 300 is used, the cultivation conditions of the test plant are output using an image of the environment around the test plant as input information. This makes it possible to estimate the cultivation conditions of the test plant from an image of the environment around the test plant.

[0051] (Learning model generating device 60) The learning model generation device 60 generates an estimation model by performing machine learning using three-dimensional image data of a damaged part of a plant and information representing a diagnosis result showing the state of the plant as learning data.

[0052] The learning model generating device 60 includes a control unit 61. The control unit 61 controls each unit of the learning model generating device 60, and is realized by a processor and a memory, for example. In this example, the processor accesses a storage (not shown), loads a program (not shown) stored in the storage into the memory, and executes a series of instructions included in the program. This constitutes each unit of the control unit 61. As each unit, the control unit 61 includes a learning data acquisition unit 62, a learning model generating unit 63, and a re-learning unit 64.

[0053] The learning data acquisition unit 62 acquires learning data. The learning data acquisition unit 62 receives an input signal representing an instruction to start learning input by a user who uses the diagnostic system 100, and reads out the learning data stored in the storage device 70. The learning data acquisition unit 62 outputs the acquired learning data to the learning model generation unit 63. An example of the learning data acquired by the learning data acquisition unit 62 is the learning data 200 shown in FIG. 3.

[0054] The learning model generating unit 63 generates an estimation model by performing machine learning using three-dimensional image data of a damaged part of a plant and information representing a diagnosis result showing the state of the plant as learning data. As an example, the learning model generating unit 63 generates a learning model using a known machine learning method such as a neural network, a decision tree, a random forest, or a support vector machine. The learning model generated by the learning model generating unit 63 uses a three-dimensional image of a damaged part of a plant as input information and outputs a diagnosis result showing the state of the plant. The learning model generating unit 63 stores the generated learning model in the storage device 70.

[0055] The re-learning unit 64 re-learns the estimation model using as learning data three-dimensional image data of the damaged part of the test plant and information representing the diagnosis result of the condition of the test plant diagnosed by the diagnosis device. The re-learning unit 64 can improve the estimation accuracy of the estimation model by re-learning the estimation model using the diagnosis result by the diagnosis device 20. By improving the estimation accuracy of the estimation model by re-learning the estimation model using the diagnosis result by the diagnoser, it becomes possible to perform automatic diagnosis with higher accuracy even when the diagnoser is not present.

[0056] In addition, the learning model generation device 60 can also be used to respond to requests from the estimation device 10 for three-dimensional images of areas other than the damaged area by generating an estimation model that has learned information about three-dimensional images requested by the diagnostician from the photographer in order to make a more accurate diagnosis.

[0057] Furthermore, the learning model generation device 60 generates a cultivation condition estimation model. For example, the learning model generation device 60 performs machine learning using the learning data 300 shown in Fig. 4 to generate a cultivation condition estimation model that outputs the cultivation conditions of the test plant using an image of the environment around the test plant as input information.

[0058] (Diagnosis Processing) The flow of diagnostic processing (diagnostic method) in diagnostic device 20 will be described with reference to Fig. 5. Fig. 5 is a flowchart illustrating the flow of diagnostic processing in diagnostic device 20.

[0059] First, the data acquisition unit 22 acquires at least one of the estimation result by the estimation device 10 and the diagnosis result by the diagnoser (step S1). The diagnosis unit 23 diagnoses the state of the plant by referring to at least one of the estimation result by the estimation device 10 and the diagnosis result by the diagnoser acquired by the data acquisition unit 22 (step S2, diagnosis step). The diagnosis unit 23 outputs information representing the diagnosis result to the display control unit 24 and the storage device 70 (step S3).

[0060] The display control unit 24 causes the operator-side display device of the operator-side system 30 to display information indicating the diagnosis result by the diagnosing unit 23 (step S4), and ends the diagnosis process.

[0061] (Estimation process) The flow of the estimation process in the estimation device 10 will be described with reference to Fig. 6. Fig. 6 is a flowchart illustrating the flow of the estimation process in the estimation device 10.

[0062] First, the three-dimensional image acquisition unit 12 acquires a three-dimensional image of a damaged part of a test plant photographed by the imaging device 31 (step S11). The estimation unit 13 reads out an estimation model stored in the storage device 70 (step S12). The estimation unit 13 inputs the acquired three-dimensional image data into the read out estimation model, and acquires information representing the state of the test plant to be output (step S13). The estimation unit 13 outputs the acquired information representing the state of the test plant to the diagnosis device 20 as an estimation result (step S14), and ends the estimation process.

[0063] (Cultivation condition estimation process) The flow of the cultivation condition estimation process in the cultivation condition estimation device 50 will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating the flow of the cultivation condition estimation process in the cultivation condition estimation device 50.

[0064] First, the data acquisition unit 52 acquires an image of the environment surrounding the test plant captured by the imaging device 31 (step S21). The estimation unit 53 reads out the cultivation condition estimation model stored in the storage device 70 (step S22). The estimation unit 53 inputs the acquired image data into the read cultivation condition estimation model, and acquires information representing the cultivation conditions of the test plant to be output (step S23). The estimation unit 53 outputs the acquired information representing the cultivation conditions of the test plant as an estimation result to the diagnosis device 20 (step S24), and ends the cultivation condition estimation process.

[0065] (Request Processing) The flow of request processing in the diagnostic system 100 will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a diagram for explaining the concept of request processing in the diagnostic system. Fig. 9 is a flowchart for explaining the flow of request processing in the diagnostic system.

[0066] As shown in FIG. 8, in the diagnosis process in the diagnosis system 100, a three-dimensional image of a damaged part of a test plant photographed by a photographer is sent from the photographer to a diagnosing person or estimation device 10. Then, the diagnosing person or estimation device 10 diagnoses or estimates the condition of the test plant by referring to the acquired three-dimensional image. Here, there are cases where the damaged part of the test plant is not properly photographed in the three-dimensional image sent from the photographer, or where it is preferable to make a diagnosis by also referring to a three-dimensional image other than the damaged part. There is a possibility that the photographer does not have specialized knowledge for diagnosis and cannot photograph a three-dimensional image suitable for diagnosis.

[0067] In the diagnosis system 100, if the diagnosing person or the estimation device 10 judges that an accurate diagnosis cannot be made even by referring to the acquired three-dimensional image, the diagnosing person or the estimation device 10 can request the photographer to provide a three-dimensional image for a more accurate diagnosis. For example, the diagnosing person directly inputs the part for which a three-dimensional image is requested into the image of the test plant projected in the virtual space of the diagnosing person side display device 41, and an image in which the request is superimposed on the part of the test plant that is requested is displayed to the photographer.

[0068] In this way, in the diagnostic system 100, appropriate diagnosis can be made not only by one-way diagnosis request processing from the photographer to the diagnosator or estimation device 10, but also by two-way request processing, such as image request processing from the diagnosator or estimation device 10 to the photographer.

[0069] FIG. 9 shows a request process from the estimation device 10. As shown in FIG. 9, the judgment unit 14 of the estimation device 10 acquires an estimation result from the estimation unit 13 (step S31). The judgment unit 14 judges whether the estimation accuracy of the acquired estimation result is less than a predetermined threshold value (step S32). When the judgment unit 14 judges that the estimation accuracy is less than the predetermined threshold value (Yes), the display control unit 15 causes the photographer-side display device to display information requesting a three-dimensional image of a part of the test plant other than the damaged part (step S33). When the judgment unit 14 judges that the estimation accuracy is not less than the predetermined threshold value (No), the estimation unit 13 outputs the estimation result to the diagnosis device 20 (step S34) and ends the process.

[0070] (Switching process) The flow of the switching process by the diagnostic device 20 will be described with reference to Fig. 10. Fig. 10 is a flowchart illustrating the flow of the switching process in the diagnostic system.

[0071] First, the diagnosis unit 23 acquires the estimation result by the estimation device 10 (step S41). The diagnosis unit 23 judges whether the estimation accuracy by the estimation device is less than a predetermined threshold (step S42). When the diagnosis unit 23 judges that the estimation accuracy is less than the predetermined threshold (Yes), the diagnosis unit 23 acquires the diagnosis result by the diagnoser accepted by the input device 42 (step S43). The diagnosis unit 23 outputs the acquired diagnosis result by the diagnoser to the display control unit 24 (step S45). When the diagnosis unit 23 judges that the estimation accuracy is not less than the predetermined threshold (No), the diagnosis unit 23 outputs the estimation result by the estimation device 10 to the display control unit 24 as the diagnosis result (step S44), and ends the process.

[0072] (Learning process) The flow of the learning process in the learning model generating device 60 will be described with reference to Fig. 11. Fig. 11 is a flowchart illustrating the flow of the learning process in the learning model generating device 60.

[0073] The learning data acquisition unit 62 reads out the learning data stored in the storage device 70 (step S51). The learning model generation unit 63 performs machine learning using the read out learning data to generate an estimation model for estimating the state of the plant (step S52). The learning model generation unit 63 outputs the generated estimation model to the estimation device 10 or the storage device 70 (step S53) and ends the process.

[0074] (effect) The diagnostic system according to one aspect of the present invention is a VR (Virtual Reality) / AR (Augmented Reality) system that combines a technology for instantly identifying and diagnosing the condition of a plant from a remote location with a technology for acquiring 3D images in real time on the producer's side and a technology for reproducing 3D image data on the diagnostician or diagnostic device side. This technology provides a 3D image capture system that can capture 3D images of plants suffering from pests or physiological disorders in real time at the producer's site. This allows diagnosticians with specialized knowledge of plant pests and physiological disorders to view images of the affected areas from any viewpoint, as if they were observing the area on-site, greatly improving the identification and diagnosis of plant conditions. Furthermore, the use of VR / AR technology is expected to support not only pest and disease diagnosis, but also plant vigor diagnosis in everyday farm work.

[0075] The diagnostic system according to one embodiment of the present invention contributes to the maintenance and development of agriculture, thereby contributing to the achievement of the Sustainable Development Goals (SDGs).

[0076] [Software implementation example] The functions of each of the devices constituting the diagnostic system 100, namely the estimation device 10, the diagnosis device 20, the cultivation condition estimation device 50, and the learning model generation device 60, are programs for causing a computer to function as these devices, and can be realized by programs for causing a computer to function as each control block of these devices (particularly each part included in the control unit 11, the control unit 21, the control unit 51, and the control unit 61).

[0077] In this case, these devices are provided with a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the above program. The above program is executed by the control device and the storage device, thereby realizing each function described in the above embodiment.

[0078] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0079] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0080] Furthermore, each process described in the above embodiment may be executed by AI (Artificial Intelligence). In this case, the AI ​​may be operated by the control device or another device (for example, an edge computer or a cloud server).

[0081] That is, the following diagnostic programs are also included in the scope of the present invention: a diagnostic program for causing a computer to function as any of the diagnostic systems described above, which executes an estimation step of estimating the condition of a test plant using a three-dimensional image of the damaged portion of the test plant as input information, using an estimation model generated by performing machine learning using three-dimensional image data of the damaged portion of the plant and information representing a diagnostic result indicating the condition of the plant as learning data, and a diagnosis step of diagnosing the condition of the test plant by referring to the estimation result indicating the condition of the test plant estimated in the estimation step.

[0082] Other Aspects In the diagnostic system 100, the diagnostic device 20 may diagnose the state of the plant from the cultivation conditions estimated by the cultivation condition estimation device 50. That is, the following diagnostic system is also included in the scope of the present invention: a diagnostic system including a cultivation condition estimation device having a cultivation condition estimation unit that estimates the cultivation conditions of a test plant by using a three-dimensional image captured of the environment surrounding the test plant as input information, using a cultivation condition estimation model generated by performing machine learning using three-dimensional image data captured of the environment surrounding the plant and information indicating the cultivation conditions of the plant as learning data, and a diagnostic device that diagnoses the state of the test plant by referring to the cultivation conditions estimated by the cultivation condition estimation device.

[0083] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0084] 10 Estimation device 13 Estimation part 14 Judgment section 15 Display control section 20 Diagnostic Equipment 23 Diagnostic Department 24 Display control section 30 Photographer's system 31 Imaging device 32 Photographer's display device 40 Diagnostician System 41 Diagnostician's display device 42 Input Devices 50 Cultivation condition estimation device 53 Estimation Department (Cultivation Condition Estimation Department) 60 Learning model generation device 63 Learning model generation unit 64 Re-learning section 100 Diagnostic Systems 200 training data

Claims

1. A diagnostic system for diagnosing a condition of a plant, comprising: an estimation device including an estimation unit that estimates the state of a test plant by using, as input information, a three-dimensional image of a damaged portion of the test plant, and an estimation model generated by performing machine learning using, as learning data, three-dimensional image data of the damaged portion of the test plant and information showing a diagnosis result indicating the state of the test plant; a diagnostic device including a diagnostic unit that diagnoses a state of the test plant by referring to an estimation result indicating a state of the test plant estimated by the estimation device; a diagnostician-side display device that acquires the three-dimensional image of the test plant and displays it to a diagnostician; an input device that receives input information of a diagnosis result indicating a state of the test plant diagnosed by a diagnoser with reference to the three-dimensional image of the test plant; Equipped with the diagnosing unit diagnoses the state of the test plant by referring to at least one of an estimation result indicating the state of the test plant estimated by the estimation device and the diagnosis result received by the input device; Diagnostic system.

2. The diagnostic system according to claim 1 , wherein the diagnosing unit diagnoses the state of the test plant by referring to the diagnosis result received by the input device when the estimation accuracy by the estimation device is less than a predetermined threshold value.

3. A photographer-side display device that is a display device visually recognized by a photographer of the test plant, The estimation device includes: a determination unit that determines whether or not the estimation accuracy is less than a predetermined threshold value when the three-dimensional image of the damaged portion of the test plant is used as input information; The diagnostic system of claim 1 or 2, further comprising a display control unit that, when the judgment unit determines that the amount is less than a predetermined threshold, displays information requesting a three-dimensional image of a part of the test plant other than the damaged part on a photographer-side display device viewed by a person photographing the test plant.

4. A photographer-side display device that is a display device visually recognized by a photographer of the test plant, The diagnostic device comprises: The diagnostic system described in claim 1 or 2, further comprising a display control unit that, when the input device receives input information from a diagnoser requesting a three-dimensional image of a part of the test plant other than the damaged part, displays information representing the request from the diagnoser on a photographer-side display device viewed by the photographer of the test plant.

5. The present invention further includes a cultivation condition estimation device including a cultivation condition estimation unit that estimates the cultivation conditions of a test plant by using an image of the environment surrounding the test plant as input information, using a cultivation condition estimation model generated by performing machine learning using image data of the environment surrounding the plant and information representing the cultivation conditions of the plant as learning data, and The diagnostic system according to claim 1 , wherein the diagnosing unit further refers to an estimation result of the cultivation conditions estimated by the cultivation condition estimation device to diagnose a state of the test plant.

6. The diagnostic system according to any one of claims 1 to 5, further comprising a learning model generation device having a generation unit that generates an estimation model by performing machine learning using three-dimensional image data of a damaged portion of a plant and information representing a diagnosis result showing the condition of the plant as learning data.

7. The diagnostic system of claim 6, further comprising a re-learning unit configured to re-learn the estimation model using as learning data three-dimensional image data of the damaged portion of the test plant and information representing the diagnosis result of the condition of the test plant diagnosed by the diagnosis device.

8. A diagnostic system for diagnosing a condition of a plant, comprising: an estimation device including an estimation unit that estimates the state of a test plant by using, as input information, a three-dimensional image of a damaged portion of the test plant, and an estimation model generated by performing machine learning using, as learning data, three-dimensional image data of the damaged portion of the test plant and information showing a diagnosis result indicating the state of the test plant; a diagnostic device including a diagnostic unit that diagnoses a state of the test plant by referring to an estimation result indicating a state of the test plant estimated by the estimation device; a photographer-side display device which is a display device visually recognized by a photographer of the test plant; Equipped with The estimation device includes: a determination unit that determines whether or not the estimation accuracy is less than a predetermined threshold value when the three-dimensional image of the damaged portion of the test plant is used as input information; and a display control unit that, when the determination unit determines that the damage is less than a predetermined threshold, displays information requesting a three-dimensional image of a part of the test plant other than the damaged part on a photographer-side display device visually recognized by a photographer of the test plant. Diagnostic system.

9. A method for diagnosing a condition of a plant, comprising: an estimation step of estimating the state of a test plant by using, as input information, a three-dimensional image of the damaged portion of the test plant, by using an estimation model generated by performing machine learning using, as learning data, three-dimensional image data of the damaged portion of the test plant and information showing a diagnosis result indicating the state of the test plant; a diagnosis step of diagnosing a state of the test plant by referring to an estimation result indicating a state of the test plant estimated in the estimation step; a diagnostician-side display step of acquiring the three-dimensional image of the test plant and displaying it to a diagnostician; an input step of receiving input information of a diagnosis result indicating a state of the test plant diagnosed by a diagnoser with reference to the three-dimensional image of the test plant; Inclusive of In the diagnosis step, a state of the test plant is diagnosed by referring to at least one of an estimation result indicating a state of the test plant estimated in the estimation step and the diagnosis result received in the input step. Diagnostic methods.

10. A diagnostic program for causing a computer to function as an estimation device constituting the diagnostic system according to claim 3, comprising: an estimation step of estimating the state of a test plant by using an estimation model generated by performing machine learning using, as learning data, three-dimensional image data of the damaged portion of the plant and information showing a diagnosis result indicating the state of the plant, as input information, a three-dimensional image of the damaged portion of the test plant; a determination step of determining whether or not the estimation accuracy when the three-dimensional image obtained by photographing the damaged portion of the test plant is used as input information is less than a predetermined threshold value; a display control step of displaying information requesting a three-dimensional image of a part of the test plant other than the damaged part on a photographer-side display device viewed by a photographer of the test plant when it is determined in the determination step that the damage is less than a predetermined threshold value; Run the diagnostic program.

11. A diagnostic system for diagnosing a condition of a plant, comprising: a cultivation condition estimation device including a cultivation condition estimation unit that estimates the cultivation conditions of a test plant by using a three-dimensional image captured of the environment surrounding the test plant as input information, using a cultivation condition estimation model generated by performing machine learning using three-dimensional image data captured of the environment surrounding the plant and information representing the cultivation conditions of the plant as learning data; a diagnosis device that diagnoses a state of the test plant by referring to the cultivation conditions estimated by the cultivation condition estimation device; a diagnostician-side display device that acquires the three-dimensional image of the test plant and displays it to a diagnostician; an input device that receives input information of a diagnosis result indicating a state of the test plant diagnosed by a diagnoser with reference to the three-dimensional image of the test plant; Equipped with the diagnosis device diagnoses a state of the test plant by referring to at least one of the estimation result of the cultivation condition estimated by the cultivation condition estimation device and the diagnosis result received by the input device. Diagnostic system.

Images