Plant diagnosis method and maintenance system

Abstract

The invention relates to a plant diagnosis method and a maintenance system. The plant diagnosis method comprises the following steps: acquiring information about a plant to be diagnosed; determining a preliminary disease cause set based on the acquired information through the diagnosis model, wherein each preliminary disease cause in the preliminary disease cause set has confidence; in response to the preliminary cause set including the preliminary causes whose confidence is lower than a confidence threshold, determining information required to be supplemented for cause reasoning through a diagnosis model, and generating questions based on the information required to be supplemented through a question and answer model, obtaining an answer of the user to the question, and re-determining a preliminary disease cause set through the diagnosis model based on the obtained information and the answer; in response to the fact that the confidence coefficient of each preliminary cause in the re-determined preliminary cause set is not lower than a confidence coefficient threshold value, determining the re-determined preliminary cause set as a final cause set; generating, via a diagnostic model, a diagnostic result based on the final etiological set; and presenting the diagnosis result on a user interface.

Description

Technical Field

[0001] This disclosure relates to the field of information processing technology, and more specifically, to a plant diagnostic method and associated electronic equipment, computer-readable storage media, computer program products, and a maintenance system. Background Technology

[0002] As living standards improve, more and more people are starting to cultivate plants. Because plants may experience various growth events during their development, it is necessary to constantly monitor their health in order to better care for them. Summary of the Invention

[0003] A brief overview of this disclosure is given below to provide a basic understanding of some aspects of it. However, it should be understood that this overview is not an exhaustive summary of this disclosure. It is not intended to identify key or essential parts of this disclosure, nor is it intended to limit the scope of this disclosure. Its purpose is merely to present certain concepts of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0004] According to a first aspect of this disclosure, a plant diagnosis method is provided. The method includes: acquiring information about a plant to be diagnosed, including images of the plant; determining a preliminary set of causes based on the acquired information via a diagnostic model, each preliminary cause in the preliminary set having a confidence level; in response to the preliminary set of causes including preliminary causes whose confidence levels are below a confidence threshold, performing a question-and-answer interaction process: determining supplementary information required for cause inference via the diagnostic model and generating a question via the question-and-answer model based on the supplementary information, and acquiring user answers to the question and redetermining the preliminary set of causes based on the acquired information and the answers via the diagnostic model; in response to each preliminary cause in the redetermined preliminary set having a confidence level not lower than a confidence threshold, determining the redetermined preliminary set of causes as a final set of causes; generating a diagnostic result based on the final set of causes via the diagnostic model; and presenting the diagnostic result on a user interface.

[0005] In some embodiments, the method further includes determining at least one of the plant species and disease based on the acquired information via a diagnostic model, wherein outputting a diagnostic result via the diagnostic model based on a final set of causes includes generating a diagnostic result via the diagnostic model based on at least one of the plant species and disease determined and the final set of causes.

[0006] In some embodiments, in response to the redefined preliminary etiology set including preliminary etiologies whose confidence level is below a confidence threshold, the question-and-answer interaction process is re-executed to redefine the preliminary etiology set until the confidence level of each preliminary etiology in the redefined preliminary etiology set is not lower than the confidence threshold.

[0007] In some embodiments, the question is further generated based on contextual information via a question-answering model, the contextual information including at least one of question-answering contextual information and diagnostic contextual information.

[0008] In some embodiments, the question-answering model is configured to perform at least one of the following operations: increasing the depth of the question in subsequent question-answering interactions in response to user impatience during the question-answering interaction; or decreasing the depth of the question in subsequent question-answering interactions in response to inaccurate answers from the user during the question-answering interaction.

[0009] In some embodiments, the question-answering model is configured to determine that a user is showing impatience in response to detecting at least one of a user quickly skipping a question, a user forcibly terminating a question, or a user's answer including negative feedback; or the question-answering model is configured to determine that a user's answer is inaccurate in response to detecting that a user's answer does not match the question asked.

[0010] In some embodiments, generating questions based on supplementary information via a question-answering model includes: setting the priority of corresponding questions based on the priority of the supplementary information via a question-answering model, wherein higher-priority questions are presented to the user before lower-priority questions.

[0011] In some embodiments, the question-and-answer interaction process further includes: after obtaining the user's answer to the question, in response to determining via a diagnostic model that the answer contradicts the preliminary cause of the problem, verifying the reliability of the obtained information via a diagnostic model.

[0012] In some embodiments, the question-and-answer interaction process further includes: after obtaining the user's answer to the question, performing semantic parsing of the answer based on a knowledge graph for plant diagnosis via a question-and-answer model to determine whether the answer needs to be corrected; in response to the answer needing to be corrected, correcting the answer based on a knowledge graph for plant diagnosis via a question-and-answer model, or prompting the user to correct the answer; and re-determining the preliminary etiology set via a diagnostic model based on the obtained information and the corrected answer.

[0013] In some embodiments, the preliminary etiology set has an etiology probability distribution indicating the probability of each preliminary etiology therein, and wherein generating a diagnostic result based on the final etiology set via a diagnostic model includes generating the diagnostic result based on the etiology probability distribution of the final etiology set via the diagnostic model.

[0014] In some embodiments, determining a preliminary set of causes based on the acquired information via a diagnostic model includes: determining the plant species and disease based on the acquired information via a diagnostic model, wherein the disease has multiple possible causes associated with the disease; and screening multiple possible causes based on the determined species via a diagnostic model to determine a preliminary set of causes.

[0015] In some embodiments, the information obtained may also include text about the plant, the text including the plant's metadata, the metadata including at least one of time data, geographic data, environmental data and historical diagnostic data related to the plant.

[0016] In some embodiments, the diagnostic model includes a multimodal large model, which includes a visual encoder, a projector, and a large language model, wherein the large language model is configured to extract textual features from text about plants; the visual encoder is configured to extract image features from images of plants; the projector is configured to map the image features to a latent space unified with the textual features and fuse the mapped image features with the textual features to obtain fused features; and the large language model is further configured to determine the species, symptoms, and preliminary etiology set of the plant based on the fused features.

[0017] In some embodiments, the large language model includes multiple sub-models fine-tuned by corpora from different sub-segments of plant diagnosis.

[0018] In some embodiments, the large language model is configured to semantically associate the identified plant species and / or symptoms with the outputs of sub-models among a plurality of sub-models to adjust the causal probability distribution of the initial causal set.

[0019] In some embodiments, the multiple sub-models include a first sub-model for determining soil moisture, a second sub-model for determining leaf physiological state, and a third sub-model for determining light traces.

[0020] In some embodiments, the visual encoder includes an image feature extraction module and an image feature optimization module, wherein the image feature extraction module is configured to extract image features from an image of a plant, and the image feature optimization module is configured to optimize the extracted image features to enhance the feature weights of key lesion regions.

[0021] In some embodiments, the diagnostic model and the question-answering model are optimized via reinforcement learning, wherein an improvement in diagnostic accuracy is used as a reward signal in the reinforcement learning.

[0022] In some embodiments, the diagnostic results include at least one of etiological reasoning and care recommendations.

[0023] In some embodiments, the method further includes: generating a maintenance plan based on the determined maintenance recommendations, the maintenance plan including maintenance tasks and identifiers of maintenance devices for performing the maintenance tasks; and outputting the maintenance plan.

[0024] In some embodiments, the method further includes controlling the corresponding maintenance device according to the identifier of the maintenance device in the maintenance plan to complete the maintenance task.

[0025] According to a second aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-executable instructions, which, when executed by the processor, cause the processor to perform the method described according to any embodiment of the first aspect of this disclosure.

[0026] According to a third aspect of this disclosure, a computer-readable storage medium is provided having computer-executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described according to any embodiment of the first aspect of this disclosure.

[0027] According to a fourth aspect of this disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor, implement the method described according to any embodiment of the first aspect of this disclosure.

[0028] According to a fifth aspect of this disclosure, a maintenance system is provided, comprising: an acquisition module configured to acquire information about a plant to be diagnosed, the information including images of the plant; and a diagnosis module configured to: determine a preliminary set of causes based on the acquired information via a diagnosis model, each preliminary cause in the preliminary set having a confidence level; and, in response to the preliminary set of causes including preliminary causes whose confidence levels are below a confidence level threshold, perform a question-and-answer interaction process as follows: determine supplementary information required for cause inference via the diagnosis model and generate a question via the question-and-answer model based on the supplementary information; and acquire a user's answer to the question and redetermine the preliminary set of causes based on the acquired information and the answer via the diagnosis model; and, in response to each preliminary cause in the redetermined preliminary set of causes... The system comprises: a preliminary etiology set with a confidence level not lower than a confidence threshold; a final etiology set determined by re-determining the preliminary etiology set; a diagnostic model generating a diagnostic result based on the final etiology set, the diagnostic result including etiology inference and maintenance recommendations; a presentation module configured to present the diagnostic result on a user interface; a maintenance plan generation module configured to generate a maintenance plan based on the maintenance recommendations, the maintenance plan including maintenance tasks and identifiers of maintenance devices for performing the maintenance tasks; a control module configured to control the corresponding maintenance device based on the identifier of the maintenance device in the maintenance plan to complete the maintenance task; and an execution module including a maintenance device communicatively coupled to the control module, the maintenance device being configured to execute the maintenance task in response to receiving a command from the control module. Attached Figure Description

[0029] The foregoing and other features and advantages of this disclosure will become clear from the following description of embodiments illustrated in conjunction with the accompanying drawings. The drawings, incorporated herein and forming a part of the specification, are further used to explain the principles of this disclosure and to enable those skilled in the art to make and use it. Wherein:

[0030] Figure 1 This is a flowchart illustrating a plant diagnostic method according to some embodiments of the present disclosure;

[0031] Figures 2 to 7 This is a schematic diagram illustrating a non-limiting example of a user interface according to some embodiments of the present disclosure;

[0032] Figure 8 This is a schematic block diagram illustrating an electronic device according to some embodiments of the present disclosure;

[0033] Figure 9 This is a schematic block diagram illustrating a computer system on which embodiments of the present disclosure may be implemented;

[0034] Figure 10 This is a schematic block diagram illustrating a maintenance system according to some embodiments of the present disclosure.

[0035] Note that in the embodiments described below, the same reference numerals are sometimes used across different figures to denote the same parts or parts with the same function, and repeated descriptions are omitted. In some cases, similar reference numerals and letters are used to denote similar items, so once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0036] For ease of understanding, the positions, dimensions, and extents of the structures shown in the accompanying drawings and other materials may not represent actual positions, dimensions, and extents. Therefore, this disclosure is not limited to the positions, dimensions, and extents disclosed in the accompanying drawings and other materials. Detailed Implementation

[0037] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0038] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this disclosure or its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in this disclosure. However, those skilled in the art will understand that they merely illustrate exemplary ways that can be used to implement this disclosure, and not exhaustive ways. Furthermore, the drawings are not necessarily drawn to scale, and some features may be enlarged to show details of specific components.

[0039] In addition, techniques, methods and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods and equipment should be considered part of the specification.

[0040] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0041] In related technologies, plant diagnosis methods are generally divided into two categories. In the first category, a diagnosis is given directly based on the plant image uploaded by the user. Therefore, when the information provided by the uploaded plant image is insufficient to support a diagnosis, the accuracy of the diagnosis obtained by the first category is low, and it cannot interact with the user to obtain supplementary information. In the second category, after the user uploads a plant image, the method can interact with the user based on a fixed list of questions (such as questions about watering frequency, light duration, etc.) to obtain corresponding supplementary information, and then give a diagnosis based on the plant image and supplementary information. However, the second category of plant diagnosis methods only asks fixed questions and cannot generate personalized questions for different plant states shown in different plant images. Therefore, the second category of plant diagnosis methods lacks reasoning ability and personalized adjustment ability, resulting in low diagnostic accuracy in scenarios with uncertain factors and a poor user experience.

[0042] Therefore, this disclosure provides a plant diagnosis method that can determine a preliminary set of causes based on the information originally provided by the user, and dynamically generate questions to obtain the supplementary information needed for cause reasoning when the confidence of the preliminary causes is insufficient, thereby obtaining a more accurate diagnosis result based on the information originally provided by the user and the information supplemented by the user.

[0043] The plant diagnostic method according to this disclosure will now be described in detail with reference to the accompanying drawings. It should be understood that actual plant diagnostic methods may include other additional steps, but in order to avoid obscuring the key points of this disclosure, these additional steps will not be discussed herein and will not be shown in the accompanying drawings.

[0044] Figure 1 A flowchart of a plant diagnosis method 100 (hereinafter referred to as "method 100") according to some embodiments of the present disclosure is shown. Method 100 can be executed by an application (APP) installed on an electronic device such as a computer or mobile phone to provide plant diagnosis functionality to a user. Figure 1 As shown, method 100 includes steps S102 to S112.

[0045] In step S102, information about the plant to be diagnosed is obtained. This information may include an image of the plant.

[0046] Reference Figure 2 This illustrates a non-limiting example of a user interface 200 according to some embodiments of the present disclosure. For example... Figure 2 As shown, the user can aim the viewfinder 202 of the user interface 200 at the plant to be diagnosed and take a picture of the plant by interacting with the user interaction element 204. The captured image can be displayed in the display box 206. When the user finishes taking the picture, they can interact with the user interaction element 208 to provide one or more images in the display box 206 to the diagnostic model. Of course, the user can also select one or more saved images by using the user interaction element 210, and the selected image can also be displayed in the display box 206.

[0047] In step S104, a preliminary etiology set is determined based on the acquired information via a diagnostic model, and each preliminary etiology in the preliminary etiology set has a confidence level.

[0048] In some embodiments, determining a preliminary set of causes based on acquired information via a diagnostic model includes: determining, based on the acquired information, the plant species and disease, which has multiple possible causes associated with the disease; and filtering the multiple possible causes based on the determined species to determine a preliminary set of causes. In some examples, the plant species may be considered to exclude causes that are not applicable to that species.

[0049] In some embodiments, the acquired information may also include text about the plant. This text may include plant metadata, which may include at least one of the following: plant-related time data, geographic data, environmental data, and historical diagnostic data. For example, the acquired text may describe the current month / season, the plant's geographical location, environmental conditions such as temperature / humidity / light, the plant's medical history and treatment records, etc.

[0050] In some embodiments, the diagnostic model may include a multimodal large model, which may include a visual encoder, a projector, and a large language model. As a non-limiting example, the visual encoder may be a convolutional neural network (CNN), the projector may be a multi-layer perceptron (MLP), and the large language model may be a large language model employing a transformer architecture. It is understood that various current or future visual encoders, projectors, and large language models may be applied to this disclosure.

[0051] In this embodiment, the large language model can be configured to extract text features from text about plants; the visual encoder can be configured to extract image features from images of plants; the projector can be configured to map the image features to a latent space that is consistent with the text features and fuse the mapped image features with the text features to obtain fused features; and the large language model can also be configured to determine the species, symptoms, and preliminary etiology set of the plant based on the fused features.

[0052] Therefore, the diagnostic model can extract multimodal features related to plants based on multiple data sources, enabling comprehensive plant diagnosis and avoiding misdiagnosis caused by isolated data sources, thereby improving the accuracy of plant diagnosis. Furthermore, it can leverage the visual understanding capabilities and knowledge reserves of the multimodal large-scale model, combined with the user's questions and key points in the image, to provide an optimal preliminary set of causes and output content that meets the user's preferences based on the next token prediction technique.

[0053] In some embodiments, the large language model may comprise multiple sub-models fine-tuned from corpora of different sub-domains of plant diagnosis. The outputs of these sub-models can be used to assist the large language model in determining the etiology. In some examples, the multiple sub-models may include a first sub-model for determining soil moisture, a second sub-model for determining leaf physiological state, and a third sub-model for determining light traces.

[0054] As a non-limiting example, the first sub-model may include a classification model (such as MobileNetV3) for determining soil moisture based on soil color and texture. The training data for the first sub-model may include images of multiple soils with different moisture levels. For example, the training data for the first sub-model may include images of dark brown soil with few clumps (labeled "wet soil"), light brown soil with many cracks (labeled "dry soil"), and soil with strong reflectivity and uniform color (labeled "waterlogged soil"), etc.

[0055] As a non-limiting example, the second sub-model can be configured to output semantic indicators such as leaf water adequacy and disease infection probability by calculating the leaf texture entropy and edge sharpness. The training data for the second sub-model can include images of multiple leaves in different physiological states. For example, the training data for the second sub-model can include leaf images with coarse texture and blurred edges (labeled "water-deficient leaves"), leaf images with irregularly spotted textures (labeled "disease-infected leaves"), etc.

[0056] As a non-limiting example, the third sub-model can be configured to analyze the shadow gradients of a plant or its parts (e.g., leaves, stems, etc.) to output light-related features (light trails, i.e., the traces left by light under the plant) of the plant or its parts (e.g., leaves, stems, etc.), thereby assisting the large language model in identifying light-related causes in plants (e.g., scorching, etiolation, etc.). The training data for the third sub-model can include images of plant parts with different light trails. For example, the training data for the third sub-model can include leaf images with uniform shadows (labeled "leaves with prolonged lack of light"), leaf images with localized bright spots (labeled "leaves scorched by strong light"), etc.

[0057] Therefore, by using the first, second, and third sub-models, key characteristics of plant-environment interaction can be determined, such as soil moisture, leaf water content, and the traces left by ambient light on the leaves. Based on these key characteristics, the accuracy of subsequent disease diagnosis and preliminary etiology diagnosis can be improved.

[0058] In some embodiments, the visual encoder may include an image feature extraction module and an image feature optimization module, wherein the image feature extraction module is configured to extract image features from an image of a plant, and the image feature optimization module is configured to optimize the extracted image features to enhance the feature weights of key lesion regions.

[0059] As a non-limiting illustrative example, a visual encoder may include a CNN and a Convolutional Block Attention Module (CBAM). The CNN can extract image features from plant images, and then input these extracted features into the CBAM to strengthen the feature weights of "key symptom regions" (such as yellowing edges of leaves, wrinkled textures on stems, etc.) and suppress "background regions" (such as flowerpots, debris, etc.), thereby obtaining optimized image features. This makes the subsequent identification of symptoms and preliminary etiologies based on the optimized image features more accurate.

[0060] In some embodiments, the diagnostic model can be optimized via reinforcement learning from diagnosis feedback (RLDF), where improved diagnostic accuracy can be used as a reward signal in reinforcement learning, thereby optimizing the diagnostic model's accuracy in diagnosing species, symptoms, and preliminary causes.

[0061] In some examples, diagnostic models can be fine-tuned using corpora in the field of plant diagnosis, as well as datasets that indicate the relationships between plant species characteristics, symptom characteristics and causes, and care recommendations. For instance, training data for diagnostic models can include plant pathology data, diagnostic manuals, and case studies compiled by experts.

[0062] In step S106, in response to the preliminary etiology set including preliminary etiologies whose confidence level is lower than the confidence level threshold, the following question-and-answer interaction process is performed: the information that needs to be supplemented for etiology reasoning is determined by the diagnostic model and a question is generated by the question-and-answer model based on the information that needs to be supplemented, and the user's answer to the question is obtained and the preliminary etiology set is re-determined by the diagnostic model based on the obtained information and the answer.

[0063] In some examples, determining the additional information needed for etiological reasoning via a diagnostic model may include: determining information item labels related to at least one preliminary etiology in the preliminary etiology set via the diagnostic model, and determining the additional information needed based on the determined information item labels.

[0064] As a non-limiting example, the preliminary etiology set may include {fungal infection, root hypoxia, root damage}, where the confidence level for fungal infection is 50%, the confidence level for root hypoxia is 90%, and the confidence level for root damage is 60%. Assuming a confidence threshold of 60%, it can be determined that the confidence level of the preliminary etiology "fungal infection" in the preliminary etiology set is below the confidence threshold. Therefore, the diagnostic model can determine information item labels associated with at least one of "fungal infection," "root hypoxia," and "root damage," such as watering frequency, maintenance duration, repotting time, placement location, temperature, and humidity.

[0065] You can refer to the following: Figures 3 to 6 , Figures 3 to 6 Non-limiting examples of user interface 300 according to some embodiments of this disclosure are shown. Based on the determined information item label "watering frequency," it is determined that supplementary information, including the watering frequency of the plant, can be generated, such as... Figure 3 The user interaction element 302A shown can include the question "How often do you water it?" and answer options such as "Grows in water", "Once every 1-2 days", "Twice a week", "Once a week", "Once every 2 weeks", "More than 2 weeks", and "Relies on normal rainfall". Users can select "Once every 1-2 days" as their answer. Therefore, the answer based on the watering frequency can help determine the reliability and cause of root hypoxia.

[0066] Based on the identified information item label "repotting time", the required supplementary information includes the repotting time of the plant, and can be generated as follows: Figure 4The user interaction element 302B shown can include the question "When was the last time you repotted or transplanted?" and answer options "less than 2 weeks", "2 weeks to 3 months", "3 months to 1 year", and "more than 1 year". Users can select "less than 2 weeks" as their answer. This allows the answer based on the repotting time to help determine the reliability and cause of root damage.

[0067] Based on the identified information tags "temperature" and "humidity," the required supplementary information includes the temperature and humidity of the plant's environment, and can be generated as follows: Figure 5 The user interaction element 302C shown may include the question "Which city is your plant located in?", an explanation of the question "This can help us understand the temperature and humidity", and a user interaction element 308 for selecting the location city. The user can select "Hangzhou" as the answer. Thus, based on temperature and humidity, the diagnostic model can further help determine possible causes.

[0068] Question-answering models can further generate, for example... Figure 6 The user interaction element 302D shown can include an open-ended question, "Any other details or questions about your plant?" and a dialog box where the user can input their answer. The user can then supplement this information with details such as, "After repotting last week, the upper leaves started to wither..." This additional information can further assist the diagnostic model in identifying potential causes.

[0069] It is understandable that the specific form of questions and answers can be varied, such as multiple choice questions, fill-in-the-blank questions, and essay questions.

[0070] Therefore, the question-answering model can dynamically determine the information that needs to be supplemented based on the initial etiology, thereby generating highly relevant questions, breaking away from the fixed binding of "symptom-question", realizing personalized questioning driven by "etiology", reducing invalid interactions, improving diagnostic efficiency and enhancing user experience.

[0071] In some embodiments, generating questions based on supplementary information via a question-answering model includes: prioritizing corresponding questions based on the priority of the supplementary information via the question-answering model, wherein higher-priority questions are presented to the user before lower-priority questions. For example, continue to refer to Figures 3 to 6 , Figure 3 The question about "watering frequency" has the highest priority and is therefore shown on the first page of the user interface 300; Figure 4 The issue of "repotting time" has the second highest priority, and therefore it is shown on the second page of the user interface 300; Figure 5The questions about "temperature" and "humidity" have the third highest priority, and are therefore shown on the third page of the user interface 300; Figure 6 The open-ended questions in the interface have the lowest priority, so they are shown on the fourth (and last) page of the user interface 300.

[0072] In some embodiments, the question-and-answer interaction process may further include: after obtaining the user's answer to the question, in response to a diagnostic model determining that the answer contradicts the preliminary cause, verifying the reliability of the obtained information via the diagnostic model. For example, the set of preliminary causes determined by the diagnostic model includes drought, but the user's answer obtained through the question-and-answer interaction indicates that the soil for the plants is very moist. That is, if the user's answer contradicts the preliminary cause determined by the diagnostic model, it is possible to reverse-check whether there are anomalies in the obtained information (e.g., image upload errors, environmental sensor malfunctions, etc.) that caused the diagnostic model to misdiagnose the preliminary cause as drought.

[0073] In other embodiments, the question-and-answer interaction process may further include: after obtaining the user's answer to the question, in response to the diagnostic model determining that the answer contradicts the preliminary etiology, the diagnostic model regenerates the preliminary etiology set based on the obtained information. This avoids misdiagnosis caused by misclassification of images by the diagnostic model.

[0074] Therefore, the diagnostic process can be corrected through question-and-answer interaction, thereby reducing the probability of misdiagnosis caused by inaccurate information or inaccurate initial diagnosis by the diagnostic model, and improving the accuracy and efficiency of plant diagnosis.

[0075] In some embodiments, the question-and-answer interaction process may further include: after obtaining the user's answer to the question, performing semantic parsing of the answer based on a knowledge graph for plant diagnosis via a question-and-answer model to determine whether the answer needs to be corrected; in response to the answer needing to be corrected, correcting the answer based on a knowledge graph for plant diagnosis via a question-and-answer model, or prompting the user to correct the answer; and re-determining the preliminary etiology set via a diagnostic model based on the obtained information and the corrected answer.

[0076] In some examples, knowledge graphs for plant diagnosis can be used to describe various entities (e.g., plant species, symptoms, causes) and the relationships between entities, where entities are represented by nodes in the knowledge graph, and relationships between entities are represented by edges between nodes. Question answering models can validate and automatically correct user-inputted answers based on the knowledge graph. The question answering model can compare semantically parsed user-added information with standard nodes in the knowledge graph. When spelling errors, aliases, or ambiguous expressions are found in the supplementary information, the question answering model can automatically correct these issues or prompt the user to correct the answer based on the knowledge graph. Furthermore, the question answering model can also determine the reasonableness of the answer based on the relationships and constraints between entities in the knowledge graph. If the answer is unreasonable, it can correct the answer or prompt the user to correct it.

[0077] In some examples, question-answering models can also autocomplete dynamic key information slots. A slot is a field in a knowledge graph used to describe a specific attribute or relationship of an entity or concept. Specifically, question-answering models can normalize time representations. For example, when a user's answer includes "last Friday," the model can automatically convert the slot value for "time" from "last Friday" to a specific date. Question-answering models can also quantify fuzzy units. For example, when a user's answer includes "frequently," the model can automatically convert the slot value for "frequency" from "frequently" to the number of regular days.

[0078] In some embodiments, the question-answering model can be optimized via RLDF, where the improvement in diagnostic accuracy can be used as the reward signal in RLDF, thereby optimizing the question-answering model's "question selection strategy that maximizes information gain".

[0079] In some embodiments, the question-answering model can be built upon a pre-trained language model and fine-tuned using plant diagnosis cases, multi-turn dialogue cases, and corpora in the plant diagnosis domain. For example, the question-answering model may include a language model for Natural Language Processing (NLP) (such as LlaMa2 or BERT) and a multimodal retrieval model. Training data may include historical interaction data and synthetic samples, wherein historical interaction data may include consultation dialogues, image descriptions, diagnostic suggestions, and final confirmation results, and synthetic samples may include paraphrasing or slot replacement based on professional templates and real-world statements to cover different expressions and colloquial / regional vocabulary.

[0080] In some embodiments, in response to the redefined preliminary etiology set including preliminary etiologies whose confidence level is below a confidence threshold, the question-and-answer interaction process is re-executed to redefine the preliminary etiology set until the confidence level of each preliminary etiology in the redefined preliminary etiology set is not lower than the confidence threshold.

[0081] In some examples, the methods disclosed herein support the generation of single-round or multi-round questions. After each round of responses (e.g., a round of responses may include, for example, ...), Figures 3 to 6 The four questions shown can be used to update the diagnostic inference via the diagnostic model, and the decision to continue the question-and-answer interaction or output the inference result is made based on the confidence level of the updated preliminary cause.

[0082] In some embodiments, questions are further generated by a question-answering model based on contextual information, which includes at least one of question-answering contextual information and diagnostic contextual information. After multiple question-answering interactions, the question-answering model can jointly generate questions based on the question-answering contextual information of the previous interactions and the diagnostic contextual information of the previous diagnostic processes. This allows historical question-answering and diagnostic information to be taken into account when generating new questions, thereby optimizing question generation, improving diagnostic efficiency, and enhancing user experience.

[0083] In some embodiments, the question-answering model is configured to perform at least one of the following operations: in response to user impatience during a question-answering interaction, increase the depth of the question during subsequent question-answering interactions.

[0084] In some examples, the question-answering model is configured to determine that a user is showing impatience in response to detecting at least one of the following: a user quickly skips a question, a user forcibly terminates a question, or a user's answer includes negative feedback.

[0085] Continue to refer to Figure 3 Users can quickly interact with user interaction element 304 in user interface 300 to skip the current question. Users can also interact with user interaction element 306 in user interface 300 to forcibly terminate the question and exit the question-and-answer process. (Continue to refer to...) Figure 6 Users can also enter negative feedback words such as "I don't know," "No," "I don't want to answer," or meaningless symbols in the dialog box. If the question-answering model detects such behavior, it can determine that the user is showing impatience. Therefore, in subsequent question-answering interactions, the model can increase the depth of its queries. For example, it can output a more in-depth question such as, "Please upload a close-up photo of the plant leaves to ensure that the main lesions and color changes are clearly visible." This reduces redundant questions by increasing query depth, directly presenting high-value questions to the user, allowing the plant diagnosis process to converge faster, and reducing user waiting time and operational burden.

[0086] In some embodiments, the question-answering model is configured to reduce the question depth in subsequent question-answering interactions in response to inaccurate user answers during the question-answering interaction process.

[0087] In some examples, the question-answering model is configured to determine that the user's answer is inaccurate in response to detecting that the user's answer does not match the question asked.

[0088] For example, suppose the current question is "Any other details or questions about your plant?", and the user's answer is "I recently traveled to another city...". Based on this, the question-answering model can detect a mismatch between the user's answer and the question, thus determining that the user's answer is inaccurate. After determining that the user's answer is inaccurate, the question-answering model can reduce the question depth in subsequent question-answering interactions. For example, it can output a question with lower question depth, such as "Are the leaves turning yellow?" Therefore, by reducing the question depth, the number of questions can be increased, the questions can be refined, and the amount of information in each question can be reduced, in order to better understand the user's true situation.

[0089] At step S108, in response to the fact that the confidence level of each preliminary cause in the redefined preliminary cause set is not lower than the confidence level threshold, the redefined preliminary cause set is determined as the final cause set.

[0090] In step S110, a diagnostic result is generated based on the final etiology set via a diagnostic model.

[0091] In step S112, the diagnostic results are presented on the user interface.

[0092] In some embodiments, the preliminary etiology set has an etiology probability distribution indicating the probability of each preliminary etiology therein, and wherein generating a diagnostic result based on the final etiology set via a diagnostic model includes generating the diagnostic result based on the etiology probability distribution of the final etiology set via the diagnostic model.

[0093] For example, a preliminary set of causes could include {fungal infection, root hypoxia, root damage}, where the probability distribution of fungal infection as a cause is 5%, root hypoxia as a cause is 85%, and root damage as a cause is 10%. Therefore, the diagnosis could include "the primary cause is currently root hypoxia, and secondary causes include fungal infection and root damage".

[0094] In some embodiments, method 100 may further include: determining at least one of the plant species and disease based on the acquired information via a diagnostic model, wherein outputting a diagnostic result via the diagnostic model based on a final set of causes includes generating a diagnostic result via the diagnostic model based on at least one of the plant species and disease determined and the final set of causes.

[0095] In some embodiments, the large language model can also be configured to semantically associate the identified plant species and / or symptoms with the outputs of sub-models among multiple sub-models to adjust the causal probability distribution of the initial causal set.

[0096] As a non-limiting illustration, in some examples of the large language model that includes a first sub-model for determining soil moisture and a second sub-model for determining leaf physiological state, bi-branch features can be fused using a Transformer cross-attention layer to semantically associate "plant species-disease" with "soil moisture-leaf state," thereby adjusting the probability of the corresponding cause. For example, associating drought-tolerant cacti with moist soil further reduces the probability of the cause "water deficiency."

[0097] Therefore, the diagnostic method disclosed herein can break through the limitations of "single symptom identification" by combining image information with plant growth logic. It can achieve multimodal and fine-grained analysis from multiple perspectives, such as plant species, symptom status, and environmental characteristics (such as soil moisture, leaf physiology, and light traces), thereby improving the accuracy of plant diagnosis.

[0098] In some embodiments, the diagnostic results may further include at least one of etiological inference and care recommendations. In some embodiments, the diagnostic results may further include at least one of species inference and symptom inference.

[0099] refer to Figure 7 This illustrates a non-limiting schematic diagram of a user interface 400 according to some embodiments of the present disclosure. Figure 7 As shown, the user interface 400 can include images, species, symptoms, causes, and care suggestions for the plant. Specifically, the plant species is "Pothos," and the symptoms include "yellowing leaves with a few irregular spots." The causes include "the primary cause is root hypoxia, with secondary causes being fungal infection and root damage. Specifically: 1. Overwatering leads to excessive soil moisture, causing root hypoxia; 2. Repotting last week may have caused minor root damage; 3. Recent high temperatures and humidity easily breed bacteria, leading to minor fungal infection." Care suggestions include "1. Reduce watering frequency; 2. Loosen the soil to aerate the roots; 3. Use a fungicide if necessary; 4. You can water with a small amount of nutrient solution." Thus, by reading the user interface 400, the user can obtain information about the plant to be diagnosed, including species, symptoms, causes, and care suggestions, allowing the user to better understand the plant's growth status and better care for it.

[0100] As a non-limiting illustration, the following shows an example of a complete diagnostic process for pothos.

[0101] 1. The user uploaded an image of the entire potted pothos plant. The diagnostic model determined that the symptoms included yellowing of the leaves with a few irregular spots and textures, and also determined that the soil moisture in the pot was too high.

[0102] 2. The preliminary etiology set determined by the diagnostic model includes {fungal infection, root hypoxia, and root damage}, of which the etiology probability distribution of fungal infection is 30%, root hypoxia is 40%, and root damage is 30%.

[0103] 3. The following question-and-answer interaction process is executed via the question-and-answer model:

[0104] A. "How often do you water it?", with answer options including "Grows in water", "Once every 1-2 days", "Twice a week", "Once a week", "Once within 2 weeks", "More than 2 weeks" and "Relies on normal rainfall", the user selected "Once every 1-2 days";

[0105] B. "When was the last time you repotted or transplanted the plant?", with answer options including "less than 2 weeks", "2 weeks to 3 months", "3 months to 1 year" and "more than 1 year". The user selected "less than 2 weeks".

[0106] C. "Which city is your plant located in? This can help us understand the temperature and humidity." The user selected "Hangzhou".

[0107] D. "Any other details or questions about your plant?" The user entered, "After repotting last week, the upper leaves started to wither..."

[0108] 4. Diagnostic model update inference: Overwatering leads to excessive soil moisture, resulting in root hypoxia, with a probability distribution of 85%. Repotting last week and overwatering can cause minor root damage, with a probability distribution of 10%. Temperature and humidity in Hangzhou have been at a high level recently, making pothos susceptible to minor fungal infections, with a probability distribution of 5%.

[0109] 5. Output species: Pothos.

[0110] 6. Symptoms: Yellowing of leaves with a few irregular spots and textures.

[0111] 7. Causes of the disease: The primary cause is currently root hypoxia, with secondary causes being fungal infection and root damage. Specifically: 1. Overwatering leads to excessive soil moisture, resulting in root hypoxia; 2. Repotting last week may cause minor root damage; 3. Recent high temperatures and humidity promote the growth of fungi, leading to minor fungal infections.

[0112] 6. Maintenance suggestions: 1. Reduce watering frequency; 2. Loosen the soil to improve root aeration; 3. Use fungicide if necessary; 4. You can water with a small amount of nutrient solution.

[0113] In some embodiments, method 100 may further include: generating a maintenance plan based on the determined maintenance recommendations, the maintenance plan including maintenance tasks and identifiers of maintenance devices for performing the maintenance tasks; and outputting the maintenance plan. The maintenance plan may include routine maintenance plans, therapeutic maintenance plans, preventative maintenance plans, and combinations thereof.

[0114] Continuing with the previous example of diagnosing pothos, based on the output care suggestions, corresponding care tasks are generated: "reduce the watering frequency of pothos", "loosen the soil for pothos", "spray pothos with a low concentration of fungicide" and "spray pothos with a small amount of nutrient solution".

[0115] After generating a maintenance task, since the database also stores the identifiers of maintenance devices coupled with the user terminal, the database can be used to determine the identifiers of the maintenance devices associated with the maintenance task, thereby generating a maintenance plan. Once the maintenance plan is obtained, it can be displayed on the user interface. This allows the user to know what maintenance devices (e.g., irrigation devices, fertilization devices, moving devices, pruning devices, drug delivery devices, light control devices, temperature control devices, humidity control devices, etc., or combinations thereof) should be used to maintain the pothos. Maintenance tasks can include, for example, watering, spraying, fertilizing, pruning, weeding, rotating the pot, moving the plant, providing sunlight, providing shade, adjusting temperature, adjusting humidity, applying insecticides, and applying fungicides. Specifically, the maintenance task can also include various parameters, such as the time, interval, and amount of watering; the dosage, time, and interval of nutrient solutions; the location of pruning; the dosage and location of fungicide spraying; and the target location for moving the plant.

[0116] In some embodiments, method 100 may include: controlling a corresponding maintenance device according to the identifier of the maintenance device in the maintenance plan to complete the maintenance task.

[0117] Since maintenance devices typically have communication capabilities, commands can be transmitted to them (e.g., via Bluetooth, Zigbee, etc.). For example, in the aforementioned example, the maintenance plan may include four maintenance tasks: {watering task, watering device identifier}, {soil loosening task, soil loosening device identifier}, {fungicide spraying task, drug delivery device identifier}, and {nutrient solution spraying task, fertilizer application device identifier}. Based on the identifiers, commands are sent to the corresponding watering device, soil loosening device, drug delivery device, and fertilizer application device to instruct them to perform the watering, soil loosening, fungicide spraying, and nutrient solution spraying tasks, respectively. This controls the watering, soil loosening, drug delivery, and fertilizer application devices to automatically complete these tasks, thereby reducing the user's maintenance burden and improving maintenance efficiency.

[0118] In addition to automatically executing the maintenance plan, in some embodiments, after displaying the maintenance plan, the user may be further asked whether to confirm the execution of the maintenance plan. After the user confirms the execution of the maintenance plan, the corresponding maintenance device is controlled according to the maintenance plan to complete the corresponding maintenance task.

[0119] This disclosure also provides an electronic device in another aspect. (See reference...) Figure 8 This illustrates a schematic block diagram of an electronic device 500 according to some embodiments of the present disclosure. Figure 8 As shown, the electronic device 500 includes a processor 502 and a memory 504 storing computer-executable instructions that, when executed by the processor 502, cause the processor 502 to perform the plant diagnostic method according to any of the foregoing embodiments of this disclosure. The processor 502 may be, for example, a central processing unit (CPU) of the electronic device 500. The processor 502 may be any type of general-purpose processor, or it may be a processor specifically designed for plant diagnostics, such as an application-specific integrated circuit (“ASIC”). The memory 504 may be coupled to the processor 502 and may include various computer-readable media accessible by the processor 502. In various embodiments, the memory 504 described herein may include volatile and non-volatile media, removable and non-removable media. For example, the memory 504 may include any combination of: random access memory (“RAM”), dynamic RAM (“DRAM”), static RAM (“SRAM”), read-only memory (“ROM”), flash memory, cache memory, and / or any other type of non-transitory computer-readable media. The memory 504 may store instructions that, when executed by the processor 502, cause the processor 502 to perform the method 100 according to any of the foregoing embodiments of this disclosure. In some embodiments, the electronic device 500 may be implemented as a smartphone, a smart camera, a computer, etc.

[0120] The electronic device 500 is configured to perform the method 100 described in any of the foregoing embodiments, and therefore reference can be made to the description of the various embodiments of method 100 above, which will not be repeated here.

[0121] This disclosure also provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the plant diagnostic method according to any of the foregoing embodiments.

[0122] This disclosure also provides a computer program product including instructions that, when executed by a processor, implement the plant diagnostic method according to any of the foregoing embodiments. The instructions can be any set of instructions that will be executed directly by one or more processors, such as machine code, or any set of instructions that will be executed indirectly, such as a script. The instructions can be stored in an object code format for direct processing by one or more processors, or stored in any other computer language, including scripts or sets of independent source code modules that are interpreted on demand or compiled in advance.

[0123] Figure 9A schematic block diagram of a computer system 600 on which embodiments of the present disclosure may be implemented is shown. The computer system 600 includes a bus 602 or other communication mechanism for transmitting information, and a processing means 604 coupled to the bus 602 for processing information. The computer system 600 also includes a memory 606 coupled to the bus 602 for storing instructions to be executed by the processing means 604; the memory 606 may be random access memory (RAM) or other dynamic storage device. The memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processing means 604. The computer system 600 also includes a read-only memory (ROM) 608 or other static storage device coupled to the bus 602 for storing static information and instructions for the processing means 604. A storage device 610, such as a magnetic disk or optical disk, is provided and coupled to the bus 602 for storing information and instructions. Computer system 600 may be coupled via bus 602 to output device 612 for providing output to a user, such as, but not limited to, a display (such as a cathode ray tube (CRT) or liquid crystal display (LCD)), speakers, etc. Input device 614, such as a keyboard, mouse, microphone, etc., is coupled to bus 602 for transmitting information and command selections to processing device 604. Computer system 600 may execute embodiments of the present disclosure. Consistent with certain implementations of the present disclosure, results are provided by computer system 600 in response to processing device 604 executing one or more sequences of one or more instructions contained in memory 606. Such instructions may be read into memory 606 from another computer-readable medium, such as storage device 610. Execution of the sequence of instructions contained in memory 606 causes processing device 604 to perform the methods described herein. Alternatively, the teachings may be implemented using hardwired circuitry in place of or in combination with software instructions. Therefore, implementations of the present disclosure are not limited to any particular combination of hardware circuitry and software. In various embodiments, computer system 600 can be connected across a network to one or more other computer systems, such as computer system 600, to form a networked system via network interface 616. This network may include a private network or a public network such as the Internet. In a networked system, one or more computer systems can store data and supply data to other computer systems. As used herein, the term "computer-readable medium" refers to any medium that participates in providing instructions to processing device 604 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks such as storage device 610. Volatile media include dynamic memory such as memory 606. Transmission media include coaxial cables, copper wires, and optical fibers, including wiring that includes bus 602.Common forms of computer-readable media or computer program products include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, CD-ROMs, digital video discs (DVDs), Blu-ray discs, any other optical media, thumb drives, memory cards, RAM, PROMs and EPROMs, fast EPROMs, any other memory chips or cartridges, or any other tangible media from which a computer can read. Various forms of computer-readable media may be involved when carrying one or more sequences of one or more instructions to processing device 604 for execution. For example, instructions may initially be carried on a disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions over a telephone line using a modem. A modem local to computer system 600 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector coupled to bus 602 may receive the data carried in the infrared signal and place the data on bus 602. Bus 602 carries the data to memory 606, from which processing device 604 retrieves and executes the instructions. For example, instructions received by memory 606 may be stored on storage device 610 before or after execution by processing device 604.

[0124] According to various embodiments, instructions configured to be executed by a processing device to perform a method are stored on a computer-readable medium. The computer-readable medium may be a device for storing digital information. For example, a computer-readable medium includes a compact disc read-only memory (CD-ROM) as known in the art for storing software. The computer-readable medium is accessed by a processor adapted to execute the instructions configured to be executed.

[0125] Figure 10 A schematic block diagram of a maintenance system 700 according to some embodiments of the present disclosure is shown. The maintenance system 700 includes an acquisition module 702, a diagnostic module 704, a presentation module 706, a maintenance plan generation module 708, and a control module 710.

[0126] The acquisition module 702 is configured to acquire information about the plant to be diagnosed, including images of the plant. The diagnosis module 704 is configured to determine a preliminary set of causes based on the acquired information via a diagnostic model, each preliminary cause in the preliminary set having a confidence level. In response to the preliminary set including preliminary causes whose confidence levels are below a confidence threshold, the following question-and-answer interaction process is performed: determining the supplementary information needed for cause inference via the diagnostic model and generating questions based on the supplementary information via the question-and-answer model; acquiring the user's answers to the questions; and re-determining the preliminary set of causes based on the acquired information and answers via the diagnostic model. The diagnosis module 704 is also configured to determine the re-determined preliminary set of causes as a final set of causes in response to each preliminary cause in the re-determined set having a confidence level not lower than a confidence threshold; and to generate a diagnostic result based on the final set of causes via the diagnostic model, the diagnostic result including cause inference and care recommendations. The presentation module 706 is configured to present the diagnostic results on a user interface. The maintenance plan generation module 708 is configured to generate a maintenance plan based on the determined maintenance recommendations. The maintenance plan includes maintenance tasks and identifiers of maintenance devices used to perform the maintenance tasks. The control module 710 is configured to control the corresponding maintenance devices according to the identifiers of the maintenance devices in the maintenance plan to complete the maintenance tasks.

[0127] The acquisition module 702, diagnosis module 704, presentation module 706, maintenance plan generation module 708 and control module 710 can be implemented together as the aforementioned electronic device 500 or computer system 600, and can be implemented as, for example, but not limited to, smartphones, smart cameras, computers, etc.

[0128] The maintenance system 700 also includes an execution module 720, which may include one or more maintenance devices (e.g., 7201, 7202, ..., 720) that are communicatively coupled to the control module 710. n It is configured to perform maintenance tasks in response to receiving commands from control module 710.

[0129] In some embodiments, the maintenance device (e.g., 7201, 7202, ..., 720) n The device is configured to transmit execution data to the control module 710 in response to the execution of a maintenance task. Therefore, the maintenance system 700 can also be configured to update the plant's maintenance information based on the execution data received from the maintenance device. In this way, when the user's plant image is acquired again, the updated plant maintenance information can be combined to provide the user with more personalized information and update the corresponding maintenance plan.

[0130] In some embodiments, the maintenance system 700 may also include a user interface (not shown). For example, a results page including diagnostic results may be displayed on the user interface, and / or a maintenance plan may be displayed on the user interface.

[0131] In some embodiments, the acquisition module 702 may be implemented as a camera communicatively coupled to the diagnostic module 704. The camera may be configured to capture plant images and transmit the captured plant images to the diagnostic module 704. The camera may be any suitable imaging device for monitoring plants.

[0132] For example, the diagnostic module 704 can be configured to determine a preliminary set of causes based on plant images received from the camera. For instance, it might require plant images or videos showing the local or overall condition of a plant being cared for by the user. In this case, the camera can automatically acquire these plant images or videos without requiring the user to input them. Alternatively, if a plant image from the user is unrecognizable due to insufficient clarity or other reasons, the camera can automatically acquire the plant image without requiring the user to re-input it. In other words, plant images captured by the camera can be used to assist in determining the preliminary causes of plant diseases.

[0133] Various embodiments of the maintenance system 700 can be referenced similarly to any embodiment of any of the foregoing aspects of this disclosure, and will not be elaborated further here.

[0134] The foregoing has described one or more exemplary embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0135] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this disclosure does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0136] While one or more embodiments of this disclosure provide the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or terminal product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment).

[0137] The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first" or "second" to denote names does not indicate any particular order.

[0138] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0139] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0142] Those skilled in the art will understand that one or more embodiments of this disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] One or more embodiments of this disclosure can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0144] The same or similar parts between the various embodiments of this disclosure can be referred to mutually, and each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to the description of the method embodiments. In the description of this disclosure, the descriptions of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., mean that the specific feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of this disclosure. In this disclosure, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this disclosure and the features of the different embodiments or examples.

[0145] Additionally, when used in this disclosure, the terms “here,” “above,” “below,” “below,” “in the following,” “overall,” and similar terms should refer to the entirety of this disclosure and not any particular part thereof. Furthermore, unless expressly stated otherwise or otherwise understood in the context in which they are used, conditional language used herein, such as “may,” “possibly,” “for example,” “like,” etc., is generally intended to express that certain embodiments include, while other embodiments do not, certain features, elements, and / or states. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or states in any way, or whether such features, elements, and / or states are included or performed in any particular embodiment.

[0146] In addition, embodiments of this disclosure may also include the following examples:

[0147] Example 1. A plant diagnostic method, comprising:

[0148] Obtain information about the plant to be diagnosed, including images of the plant;

[0149] A preliminary set of causes is determined based on the acquired information via a diagnostic model, and each preliminary cause in the preliminary set of causes has a confidence level;

[0150] In response to the preliminary etiology set including preliminary etiologies whose confidence level is below a confidence threshold, the following question-and-answer interaction process is performed:

[0151] The diagnostic model determines the additional information needed for etiological reasoning, and the question-answering model generates questions based on this additional information.

[0152] Obtain the user's answer to the question and, based on the obtained information and the answer, redetermine the preliminary set of causes via the diagnostic model;

[0153] In response to the fact that the confidence level of each preliminary cause in the redefined preliminary cause set is not lower than the confidence level threshold, the redefined preliminary cause set is determined as the final cause set;

[0154] The diagnostic model generates a diagnostic result based on the final set of causes; and

[0155] The diagnostic results are presented on the user interface.

[0156] Example 2. The method according to Example 1, further comprising determining, via the diagnostic model, at least one of the species and disease of the plant based on the acquired information.

[0157] Specifically, the diagnostic result output by the diagnostic model based on the final etiology set includes generating the diagnostic result by the diagnostic model based on at least one of the identified species and diseases of the plant and the final etiology set.

[0158] Example 3. According to the method of Example 1, wherein, in response to the redetermined preliminary etiology set including preliminary etiologies whose confidence is lower than the confidence threshold, the question-and-answer interaction process is re-executed to redetermine the preliminary etiology set until the confidence of each preliminary etiology in the redetermined preliminary etiology set is not lower than the confidence threshold.

[0159] Example 4. According to the method of Example 3, wherein the question is further generated based on contextual information via the question-answering model, the contextual information including at least one of question-answering contextual information and diagnostic contextual information.

[0160] Example 5. The method according to Example 3, wherein the question-answering model is configured to perform at least one of the following operations:

[0161] In response to user impatience during the Q&A interaction, the depth of the questions will be increased in subsequent Q&A interactions; or

[0162] In response to inaccurate answers from users during the question-and-answer interaction, the depth of the inquiry is reduced in subsequent question-and-answer interactions.

[0163] Example 6. According to the method described in Example 5, wherein,

[0164] The question-answering model is configured to determine that a user is showing impatience in response to at least one of the following: a user quickly skips a question, a user forcibly terminates a question, or a user's answer includes negative feedback; or

[0165] The question-answering model is configured to determine that the user's answer is inaccurate in response to detecting a mismatch between the user's answer and the question asked.

[0166] Example 7. According to the method described in Example 1, generating a question based on the required supplementary information via a question-answering model includes:

[0167] The question-and-answer model prioritizes questions based on the information that needs to be supplemented, with higher-priority questions presented to the user before lower-priority questions.

[0168] Example 8. The method described in Example 1, wherein the question-and-answer interaction process further includes:

[0169] After obtaining the user's answer to the question, in response to the determination by the diagnostic model that the answer contradicts the preliminary cause of the disease, the reliability of the obtained information is verified by the diagnostic model.

[0170] Example 9. The method described in Example 1, wherein the question-and-answer interaction process further includes:

[0171] After obtaining the user's answer to the question, the question-answering model performs semantic parsing on the answer based on a knowledge graph used for plant diagnosis to determine whether the answer needs to be corrected.

[0172] In response to the need for correction of the answer, the answer is corrected via the question-answering model based on the knowledge graph used for plant diagnosis, or the user is prompted to correct the answer; and

[0173] The preliminary etiology set is re-determined based on the acquired information and the corrected response via the diagnostic model.

[0174] Example 10. The method according to Example 1, wherein the preliminary etiology set has an etiology probability distribution indicating the probability of each preliminary etiology therein, and

[0175] Specifically, generating a diagnostic result based on the final etiology set via the diagnostic model includes generating the diagnostic result based on the etiology probability distribution of the final etiology set via the diagnostic model.

[0176] Example 11. The method according to Example 1, wherein determining a preliminary set of etiologies based on the acquired information via a diagnostic model includes:

[0177] The species and disease of the plant are determined by the diagnostic model based on the acquired information, and the disease has multiple possible causes associated with the disease;

[0178] The diagnostic model is used to screen multiple possible causes based on the identified species to determine the preliminary set of causes.

[0179] Example 12. The method according to Example 11, wherein the acquired information further includes text about the plant, the text including metadata of the plant, the metadata including at least one of time data, geographic data, environmental data and historical diagnostic data related to the plant.

[0180] Example 13. The method according to Example 12, wherein the diagnostic model includes a multimodal large model, the multimodal large model including a visual encoder, a projector, and a large language model, wherein,

[0181] The large language model is configured to extract textual features from the text about the plant;

[0182] The visual encoder is configured to extract image features from the image of the plant;

[0183] The projector is configured to map the image features to a latent space unified with the text features and fuse the mapped image features with the text features to obtain fused features; and

[0184] The large language model is also configured to determine the species, symptoms, and preliminary etiology set of the plant based on the fusion features.

[0185] Example 14. The method according to Example 13, wherein the large language model comprises multiple sub-models fine-tuned by corpora from different sub-segments of plant diagnosis.

[0186] Example 15. The method according to Example 14, wherein the large language model is configured to semantically associate the determined species and / or symptoms of the plant with the outputs of the sub-models among the plurality of sub-models to adjust the causal probability distribution of the preliminary causal set.

[0187] Example 16. The method according to Example 14, wherein the plurality of sub-models includes a first sub-model for determining soil moisture, a second sub-model for determining leaf physiological state, and a third sub-model for determining light traces.

[0188] Example 17. The method according to Example 13, wherein the visual encoder includes an image feature extraction module and an image feature optimization module, wherein the image feature extraction module is configured to extract image features from the image of the plant, and the image feature optimization module is configured to optimize the extracted image features to enhance the feature weights of key lesion regions.

[0189] Example 18. The method according to Example 1, wherein the diagnostic model and the question-answering model are optimized via reinforcement learning, wherein an improvement in diagnostic accuracy is used as a reward signal in reinforcement learning.

[0190] Example 19. The method according to Example 1, wherein the diagnostic result includes at least one of etiological reasoning and care recommendations.

[0191] Example 20. The method according to Example 19 further includes:

[0192] A maintenance plan is generated based on the determined maintenance recommendations, the maintenance plan including maintenance tasks and identifiers of maintenance devices for performing the maintenance tasks;

[0193] Output the maintenance plan.

[0194] Example 21. The method according to Example 20 includes:

[0195] Control the corresponding maintenance device according to the identification of the maintenance device in the maintenance plan to complete the maintenance task.

[0196] Example 22. An electronic device comprising:

[0197] Processor; and

[0198] A memory storing computer-executable instructions, which, when executed by the processor, cause the processor to perform the method according to any one of Examples 1 to 21.

[0199] Example 23. A computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of Examples 1 to 21.

[0200] Example 24. A computer program product comprising instructions that, when executed by a processor, implement the method according to any one of Examples 1 to 21.

[0201] Example 25. A maintenance system comprising:

[0202] An acquisition module is configured to acquire information about the plant to be diagnosed, the information including an image of the plant;

[0203] The diagnostic module is configured as follows:

[0204] A preliminary set of causes is determined based on the acquired information using a diagnostic model, and each preliminary cause in the preliminary set has a confidence level.

[0205] In response to the preliminary etiology set including preliminary etiologies whose confidence levels are below a confidence threshold, the following question-and-answer interaction process is performed: the diagnostic model determines the supplementary information needed for etiology reasoning, and the question-and-answer model generates questions based on the supplementary information; the user's answers to the questions are obtained, and the diagnostic model re-determines the preliminary etiology set based on the obtained information and the answers.

[0206] In response to the fact that the confidence level of each preliminary cause in the redefined preliminary cause set is not lower than the confidence level threshold, the redefined preliminary cause set is determined as the final cause set.

[0207] The diagnostic model generates a diagnostic result based on the final set of causes, and the diagnostic result includes cause inference and maintenance recommendations.

[0208] A presentation module is configured to present the diagnostic results on a user interface;

[0209] The maintenance plan generation module is configured to generate a maintenance plan based on the maintenance recommendations, the maintenance plan including maintenance tasks and identifiers of maintenance devices for performing the maintenance tasks;

[0210] The control module is configured to control the corresponding maintenance device according to the identifier of the maintenance device in the maintenance plan to complete the maintenance task; and

[0211] The execution module includes a maintenance device communicatively coupled to the control module, the maintenance device being configured to perform a maintenance task in response to receiving a command from the control module.

[0212] The above description is merely an embodiment of one or more embodiments of this disclosure and is not intended to limit the scope of the one or more embodiments of this disclosure. Various modifications and variations can be made to the one or more embodiments of this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims.

Claims

1. A plant diagnosis method, comprising: obtaining information about a plant to be diagnosed, the information including an image of the plant; determining, via a diagnosis model, a preliminary cause set based on the obtained information, each preliminary cause in the preliminary cause set having a confidence; in response to the preliminary cause set including a preliminary cause whose confidence is lower than a confidence threshold, performing a question-and-answer interaction process as follows: determining, via the diagnosis model, information needed to supplement a cause reasoning and generating, via a question-and-answer model, a question based on the needed supplemental information, and obtaining a user’s answer to the question and redetermining, via the diagnosis model, a preliminary cause set based on the obtained information and the answer; in response to each preliminary cause in the redetermined preliminary cause set having a confidence not lower than the confidence threshold, determining the redetermined preliminary cause set as a final cause set; generating, via the diagnosis model, a diagnosis result based on the final cause set; and presenting the diagnosis result on a user interface.

2. The method of claim 1, further comprising determining, via the diagnosis model, at least one of a species and a condition of the plant based on the obtained information, outputting, via the diagnosis model, a diagnosis result based on the final cause set includes generating, via the diagnosis model, the diagnosis result based on the at least one of the determined species and condition of the plant and the final cause set. wherein 3. The method of claim 1, wherein: optionally, in response to the redetermined preliminary cause set including a preliminary cause whose confidence is lower than the confidence threshold, re-performing the question-and-answer interaction process to redetermine a preliminary cause set until each preliminary cause in the redetermined preliminary cause set has a confidence not lower than the confidence threshold; optionally, the question is generated, via the question-and-answer model, further based on context information, the context information including at least one of a question-and-answer context information and a diagnosis context information; optionally, the question-and-answer model is configured to perform at least one of: in response to a user showing impatience in the question-and-answer interaction process, increasing an inquiry depth in a subsequent question-and-answer interaction process, or in response to a user’s answer being inaccurate in the question-and-answer interaction process, decreasing an inquiry depth in a subsequent question-and-answer interaction process; optionally, the question-and-answer model is configured to determine that a user shows impatience in response to detecting at least one of the user quickly skipping a question, the user forcibly terminating a question, or the user’s answer including negative feedback, or the question-and-answer model is configured to determine that a user’s answer is inaccurate in response to detecting that the user’s answer does not match a question being asked.

4. The method of claim 1, wherein: optionally, generating, via a question-and-answer model, a question based on needed supplemental information includes: setting, via the question-and-answer model, a priority of a respective question based on a priority of the needed supplemental information, wherein a question with a higher priority is presented to a user before a question with a lower priority; optionally, the question-and-answer interaction process further comprises: ​ After obtaining the answer of the user to the question, in response to determining via the diagnostic model that the answer contradicts the preliminary causes, verifying reliability of the obtained information via the diagnostic model.

5. The method of claim 1, Optionally, the question and answer interaction process further comprises: After obtaining the answer of the user to the question, performing semantic analysis on the answer via the question and answer model based on a knowledge graph for plant diagnosis to determine whether the answer needs to be corrected, in response to the answer needing to be corrected, correcting the answer via the question and answer model based on the knowledge graph for plant diagnosis, or prompting the user to correct the answer, and re-determining the preliminary cause set via the diagnostic model based on the obtained information and the corrected answer; Optionally, the preliminary cause set has a cause probability distribution indicating a probability of each preliminary cause therein, and wherein generating a diagnosis result via the diagnostic model based on the final cause set comprises generating the diagnosis result via the diagnostic model based on the cause probability distribution of the final cause set; Optionally, determining a preliminary cause set via the diagnostic model based on the obtained information comprises: determining a species and a condition of the plant via the diagnostic model based on the obtained information, the condition having a plurality of possible causes associated with the condition, filtering the plurality of possible causes via the diagnostic model based on the determined species to determine the preliminary cause set; Optionally, the obtained information further comprises text about the plant, the text comprising metadata of the plant, the metadata comprising at least one of temporal data, geographical data, environmental data, and historical diagnosis data related to the plant; Optionally, the diagnostic model comprises a multi-modal large model, the multi-modal large model comprising a visual encoder, a projector, and a large language model, wherein, the large language model is configured to extract text features from the text about the plant, the visual encoder is configured to extract image features from the image of the plant, the projector is configured to map the image features to a latent space unified with the text features and fuse the mapped image features with the text features to obtain fused features, and the large language model is further configured to determine the species, the condition, and the preliminary cause set of the plant based on the fused features; Optionally, the large language model comprises a plurality of sub-models respectively fine-tuned with corpora of different sub-fields of plant diagnosis; Optionally, the large language model is configured to semantically associate the determined species and / or condition of the plant with outputs of a sub-model of the plurality of sub-models to adjust a cause probability distribution of the preliminary cause set; Optionally, the plurality of sub-models comprises a first sub-model for judging soil humidity, a second sub-model for judging leaf physiological state, and a third sub-model for judging light trace; Optionally, the visual encoder comprises an image feature extraction module and an image feature optimization module, wherein the image feature extraction module is configured to extract image features from the image of the plant, and the image feature optimization module is configured to optimize the extracted image features to emphasize feature weights of key disease areas. Optionally, the diagnosis model and the question-answering model are optimized via reinforcement learning, wherein diagnosis accuracy improvement is taken as a reward signal in the reinforcement learning.

6. The method of claim 1, Optionally, the diagnosis result comprises at least one of aetiological reasoning and maintenance suggestion. Optionally, the method further comprises: generating a maintenance plan according to the determined maintenance suggestion, the maintenance plan comprising a maintenance task and an identification of a maintenance device for performing the maintenance task, outputting the maintenance plan; Optionally, the method comprises: controlling a corresponding maintenance device according to the identification of the maintenance device in the maintenance plan to complete the maintenance task.

7. An electronic device comprising: a processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-6.

8. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1-6.

9. A computer program product comprising instructions that, when executed by a processor, implement the method of any one of claims 1-6.

10. A maintenance system comprising: an acquisition module configured to acquire information about a plant to be diagnosed, the information comprising an image of the plant; a diagnosis module configured to: determine, via a diagnosis model, a preliminary cause set based on the acquired information, each preliminary cause in the preliminary cause set having a confidence level, in response to the preliminary cause set including a preliminary cause whose confidence level is lower than a confidence threshold, perform a question-answering interaction process of determining, via the diagnosis model, information needed to supplement aetiological reasoning, generating, via a question-answering model, a question based on the needed supplemental information, acquiring a user's answer to the question, and redetermining, via the diagnosis model, a preliminary cause set based on the acquired information and the answer, in response to each preliminary cause in the redetermined preliminary cause set having a confidence level not lower than the confidence threshold, determining the redetermined preliminary cause set as a final cause set, and generate, via the diagnosis model, a diagnosis result based on the final cause set, the diagnosis result comprising aetiological reasoning and maintenance suggestion; a presentation module configured to present the diagnosis result on a user interface; a maintenance plan generation module configured to generate a maintenance plan according to the maintenance suggestion, the maintenance plan comprising a maintenance task and an identification of a maintenance device for performing the maintenance task; a control module configured to control a corresponding maintenance device according to the identification of the maintenance device in the maintenance plan to complete the maintenance task; and ​ An execution module including a maintenance device communicatively coupled to the control module, the maintenance device configured to perform a maintenance task in response to receiving a command from the control module.

Images