A multi-modal knowledge graph pest and disease question answering method fusing agricultural environment features

Abstract

The application discloses a kind of multi-modal knowledge graph pest and disease question and answer method fusing agricultural environment characteristics, belong to artificial intelligence and wisdom agricultural technology field, including: collection text and image, text and image are preprocessed, feature text, feature image and visual recognition result are obtained using distillation MiniCPM-V 4.5 model after text and image are preprocessed, then retrieve using agricultural knowledge graph, obtain entity, entity in community, agricultural knowledge guide abstract and control core measure, then obtain structured answer in combination with visual recognition result, then utilize end side interface and show to obtain accurate consulting demand answer display.The application solves the problems of insufficient data professional of existing system, poor field adaptation of retrieval enhancement based on graph, low knowledge extraction accuracy, after fine tuning, pest and disease recognition accuracy reaches 93.3%, and can be deployed on end side equipment, to provide efficient consulting service for grass-roots practitioners.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, multi-modal processing, knowledge graph and smart agriculture, in particular to an agricultural multi-modal pest and disease question and answer method and system based on multi-source professional data fusion and improved GraphRAG framework, which is suitable for pest and disease identification, feature query and prevention scheme consultation of main crops such as rice, wheat and corn, and can be deployed on mobile phones, agricultural tablets and other terminal devices for use by primary growers, agricultural practitioners and township agricultural technicians. BACKGROUND

[0002] The development of smart agriculture puts forward the demand of "high professionalism, high adaptability and terminal availability" for intelligent pest and disease consultation tools, but the existing technology has four core defects, which cannot meet the actual scene: multi-source data fusion is insufficient, and professionalism is lacking: the existing system mainly depends on general data sets (such as a small number of pest and disease images in ImageNet) or single platform text, and does not integrate the structured data of professional platforms, resulting in incomplete data coverage (such as supporting less than 50 kinds of pests and diseases) and low professionalism (prevention suggestions do not meet GB / T 8321 standards); and lacks the linkage design of "image - text - knowledge", which cannot support accurate question answering; the GraphRAG framework has poor field adaptation.

[0003] Traditional GraphRAG (such as Microsoft GraphRAG) designs community detection (Leiden algorithm) and summary generation strategy based on general text, without considering the "temporality" (such as high incidence of rice leaf roller in summer) and "spatiality" (such as diseases and pests unique to the Yangtze River Basin) of agricultural pests and diseases, resulting in chaotic community division (such as grouping "winter wheat rust" and "summer rice blast" into the same community) and redundant summary (containing a large number of general descriptions irrelevant to pests and diseases), and the retrieval accuracy is less than 70%; the knowledge extraction accuracy is low, and the long-tail entity coverage is poor: there are a large number of long-tail entities in agricultural pests and diseases (such as "rice moth trichogramma" and "25% thiamethoxam water dispersible granules"), and the traditional BERT-BiLSTM-CRF model lacks field knowledge guidance, so the F1 value of long-tail entity extraction is only 65%-70%, and it is easy to misjudge "avermectin" as "fungicide" (actually it is an insecticide), so the knowledge extraction credibility is low; there is a contradiction between terminal deployment and performance: although general multi-modal models (such as GPT-4V and MiniCPM-V 4.5 original version) can process multi-modal input, the parameter scale is large (≥8B), the terminal memory occupation is ≥8GB, the response time is ≥2 seconds, and they are not optimized for agricultural scenes, so they cannot meet the demand of "fast query and low device threshold" in the field. SUMMARY

[0004] In order to solve the above technical problems, the present application provides a multi-modal knowledge graph pest and disease question and answer method fusing agricultural environment characteristics.

[0005] To achieve the above object, the application designs a multi-modal knowledge graph pest and disease question and answer method fusing agricultural environment characteristics, which is characterized in that the method comprises the following steps:

[0006] S1: Collecting text data and image data and performing preprocessing, constructing structured data based on the text data and the image data, constructing a data set, and constructing an agricultural pest and disease knowledge graph, the agricultural pest and disease knowledge graph comprising entity design, knowledge extraction, and knowledge storage;

[0007] S2: Based on the MiniCPM-V 4.5 model, a distilled MiniCPM-V 4.5 model is constructed, the structured data is input into the distilled MiniCPM-V 4.5 model to obtain feature text, feature image, and visual recognition result, the visual recognition result is used to generate a structured answer, and the distilled MiniCPM-V 4.5 model comprises a visual coding layer and a text coding layer;

[0008] S3: Based on the GraphRAG retrieval model, an agricultural adapted GraphRAG retrieval model is constructed, the feature text and the feature image are input into the agricultural adapted GraphRAG retrieval model and the agricultural knowledge graph is used as a retrieval knowledge base to obtain entities, communities where the entities are located, agricultural knowledge oriented summaries, and core prevention measures, the agricultural adapted GraphRAG retrieval model comprises a time-space coordinated hierarchical community detection algorithm and an agricultural knowledge oriented summary pruning strategy, the agricultural knowledge oriented summary pruning strategy comprises an agricultural key information screening script, and the agricultural key information screening script comprises a term matching module, an information entropy calculation module, a pruning execution module, and a summary formatting module;

[0009] S4: The entities, the communities where the entities are located, the agricultural knowledge oriented summaries, the core prevention measures, and the visual recognition result are integrated by using the distilled MiniCPM-V 4.5 model to obtain a structured answer, and the structured answer comprises a recognition result, a morphological feature, an occurrence law, a prevention scheme, and a data source;

[0010] S5: The structured answer is displayed by using an end-side interface to obtain a precise consulting demand answer display, and the end-side interface is an interface in end-side application development;

[0011] Preferably, the preprocessing in step S1 comprises text data preprocessing and image data preprocessing, and specifically as follows:

[0012] The text data preprocessing is to perform duplicate elimination, term standardization, and sentence splitting processing on the collected text data to obtain standard text data;

[0013] The image data preprocessing is to label the collected image data in COCO format, and the label contains "pest category, disease site, severity", to obtain standard image data;

[0014] The specific process of constructing the structured data set is as follows:

[0015] Using standard text data and standard image data, construct structured data, each containing "user multi-round question + professional answer + data traceability", and construct a structured data set;

[0016] The agricultural pest knowledge graph is constructed, comprising:

[0017] 1) Entity design: On the basis of traditional entities, add "natural enemy" entities and "natural enemy inhibition" relationships to form a new ontology system, which is a specific head entity + relationship + tail entity triple structure;

[0018] 2) Knowledge extraction: Take BERT-BiLSTM-CRF as the skeleton, and integrate the pre-training word vectors of the "agricultural pest terminology dictionary" into the BERT word embedding layer to build a semi-supervised knowledge extraction model enhanced by the field dictionary. Use the model to extract entities and relationships from the ontology system;

[0019] 3) Knowledge storage: stored in Neo4j graph database, supports Cypher statement query.

[0020] Preferably, in step S2, the structured data is input into the distillation MiniCPM-V 4.5 model to obtain feature text data, feature image data and visual recognition result, and the specific process is as follows:

[0021] The standard text data in the structured data is input into the text encoding layer to obtain the feature text, and the standard image data in the structured data is input into the visual encoding layer to obtain the feature image and the visual recognition result;

[0022] Preferably, in step S3, the GraphRAG retrieval model is constructed, and the specific process is as follows:

[0023] Taking the GraphRAG retrieval model as the skeleton, adding a time-space collaborative hierarchical community detection algorithm and an agricultural knowledge-oriented summary pruning strategy in the GraphRAG retrieval model to obtain an agricultural adapted GraphRAG retrieval model;

[0024] The time-space collaborative hierarchical community detection algorithm: based on the Leiden algorithm open source code, adding a "time-space" calculation module, writing Calculation function, through agricultural data cross validation to determine the weight, as follows:

[0025] Environment configuration installation dependent library: pip install python-louvain numpy pandas networkx, among which python-louvain is used for basic community detection, networkx is used for graph structure processing, and numpy is used for numerical calculation;

[0026] Core module code implementation, including five steps of graph structure loading, basic module degree calculation, time - space factor calculation, community division comprehensive score calculation, and community division execution;

[0027] Algorithm optimization: according to the sparsity characteristics of agricultural pest knowledge graph, the resolution parameter of Leiden algorithm is optimized, resolution=0.8 is set to ensure that the entities in the community are closely related, and at the same time avoid too many communities to reduce the retrieval efficiency; Introducing community merging mechanism, merging small communities with node number <5 to adjacent large communities to improve the representativeness of the community;

[0028] The agricultural knowledge-oriented summary pruning strategy: write "agricultural key information screening script", identify and retain professional terms based on term dictionary, calculate information entropy through Transformer, prune redundant sentences, which includes the following:

[0029] Term matching module: load agricultural pest term dictionary, use regular expression to match professional terms in text, and mark the sentences containing terms as candidate key sentences;

[0030] Information entropy calculation module: based on the Transformer model, use bert-base-chinese to calculate the information entropy of the sentence, the specific process is as follows: first, input the sentence into the bert-base-chinese model to get the word vector; Then calculate the conditional probability p(w_i|s) of each word; Finally, according to the information entropy formula, it is expressed as follows:

[0031]

[0032] Among them, Information entropy, sum represents the summation symbol, and p(w_i|s) represents the conditional probability;

[0033] Calculate the information entropy of the sentence, and the sentence with information entropy ≥1.5 is regarded as a high information density sentence;

[0034] Pruning execution module: fuse the term matching result and the information entropy calculation result, screen the sentences that meet "contain at least 1 professional term" and "information entropy >= 1.5" at the same time; sort the screened sentences in descending order of information entropy, and retain the first 80% of the sentences as the abstract content; force to retain the sentences containing the core industry terms of "GB / T", "safety interval period" and "prevention and control window period", so as to ensure that the professional information is not lost;

[0035] Abstract formatting module: reorganize the pruned sentences according to the logical order of "morphological characteristics -> occurrence law -> prevention and control method -> matters needing attention", and generate a structured community summary.

[0036] Compared with the prior art, the present application has the following remarkable beneficial effects: data professionalism and traceability: multi-source data come from authoritative platforms such as the national agricultural scientific data center, containing 32,000 professional texts, 16,000 labeled images, the knowledge graph contains 15,620 entities, and the prevention and control suggestions are 100% consistent with the agricultural industry standards; GraphRAG field adaptability: the improved GraphRAG framework, the community matching accuracy is improved by 7.2%, the key information retention rate of the summary is high, and the problems of "community confusion and summary redundancy" of the traditional GraphRAG in the agricultural field are solved; knowledge extraction accuracy: the semi-supervised model enhanced by the field dictionary is significantly improved compared with the traditional model, and the quality of the knowledge graph construction is ensured; end-side deployment advantage: after multi-modal feature distillation, the model memory occupation is reduced, the response time is less than or equal to 1.5 seconds, offline reasoning is supported, and it is suitable for the scene of field without network and low configuration device; practical value: covering 102 kinds of pests and diseases, more than 10 kinds of tropical crops, grassroots practitioners can quickly obtain accurate prevention and control schemes without professional knowledge, which can replace 60% of the grassroots agricultural technical personnel consultation workload, and help the landing of intelligent agriculture. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The flowchart of the multi-modal knowledge graph pest and disease question answering method fusing the agricultural environment characteristics.

[0038] Figure 2 The agricultural adaptation GraphRAG framework diagram of the multi-modal knowledge graph pest and disease question answering method fusing the agricultural environment characteristics. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0040] The present application proposes a multi-modal knowledge graph pest and disease question answering method fusing the agricultural environment characteristics, the flowchart of the method is as shown in Figure 1 The steps of the method are described in detail as follows.

[0041] S1: Collecting text data and image data and preprocessing, constructing structured data based on text data and image data, constructing a data set, and constructing an agricultural pest knowledge graph, which includes entity design, knowledge extraction, and knowledge storage;

[0042] Among them, 18,000 texts are selected from the rice / wheat pest sub-database text crawled from the National Agricultural Science Data Center, 12,000 structured description texts of 102 pests are downloaded from the China Crop Pest Knowledge Graph, and 2,000 prevention and control method sentence texts are extracted from the agricultural pest database, a total of 32,000 professional texts are obtained; professional texts are manually cleaned (remove duplicate / error sentences such as "wheat rust control dosage error"), standardized (reference GB / T 8321.1-2023 to GB / T 8321.9-2023, and "spraying" is unified as "chemical control agent spraying"), and sentence splitting (classified according to "morphological characteristics / occurrence law / control method" to facilitate subsequent knowledge extraction) to obtain standard text data.

[0043] 1.2 million pest images are collected from IP102 dataset, and 40,000 pest images are collected from "Agricultural Pest Database", a total of 16,000 images; the images are labeled in COCO format, and the labels include "pest category (such as rice leaf roller), disease site (such as leaf), severity (mild / medium / severe), etc.", and standard image data is obtained.

[0044] 2.8 million structured data are constructed from standard text data and standard image data, each containing "user multi-round question (including image placeholder) + professional answer + data traceability", and the data answer comes from professional knowledge base without any large model participation, ensuring the reliability of subsequent model fine-tuning; the structured data is constructed into "image-multi-round question and answer" structured data set.

[0045] The agricultural pest knowledge graph includes:

[0046] Entity design: Based on the traditional 8 entities, the "natural enemy" entity (such as trichogramma, chalcid wasp) is added, and the "natural enemy inhibition" relationship (such as "trichogramma → natural enemy inhibition → rice leaf roller") is added, forming a 9-entity, 8-relationship ontology system;

[0047] Knowledge extraction: Based on BERT-BiLSTM-CRF, the pre-training word vector of "agricultural pest terminology dictionary" is integrated into the BERT word embedding layer to construct a semi-supervised knowledge extraction model enhanced by domain dictionary, and the model is used to extract entities and relationships from the ontology system;

[0048] Knowledge storage: stored in Neo4j graph database, supports Cypher statement query (such as "MATCH (p: pest) - [r: natural enemy inhibition] -> (q: natural enemy) WHERE p.name='rice leaf roller' RETURN q.name, q.release_dosage"), provides structured knowledge support for GraphRAG;

[0049] The semi-supervised knowledge extraction model enhanced by the field dictionary optimization step is as follows:

[0050] The architecture aspect takes BERT-BiLSTM-CRF as the backbone, and the pre-training word vector (trained by Word2Vec, dimension 300) of the "agricultural pest and disease terminology dictionary" is integrated into the BERT word embedding layer to enhance the model's semantic perception of terms such as "rice leaf roller" and "25% abamectin emulsion". After the model extracts entities / relationships, Prompt is added to guide the correction, such as: "Please check the following extraction results: 1. Entity: 'abamectin' (type: fungicide);

[0051] 2. Relationship: 'rice leaf roller -> harms crops -> wheat'. Please judge whether it is correct according to agricultural knowledge, if it is wrong, please correct and explain the basis (reference GB / T 8321.9-2018)." The model corrects the "abamectin type misjudgment" (changed to "insecticide"), "harm crop error" (changed to "rice"), and other issues through self-correction;

[0052] Semi-supervised training strategy: stage 1 uses 1000 artificial annotation data (including long-tail entities) to train model basic parameters, ensures core entity (such as rice leaf roller) extraction F1 value ≥85%; stage 2 uses 5000 unannotated data to generate "pseudo-labels", selects samples with pseudo-label confidence ≥0.85 (about 3800), and merges them with artificial annotation data as the training set; stage 3 introduces "field adaptation loss function", increases the punishment weight of samples with agricultural term extraction errors (such as term type misjudgment punishment weight ×1.5), and improves the model's sensitivity to professional knowledge;

[0053] S2: based on MiniCPM-V 4.5 model, construct distillation MiniCPM-V 4.5 model, input structured data into distillation MiniCPM-V 4.5 model to obtain feature text, feature image and visual recognition result, visual recognition result is used to generate structured answer, the distillation MiniCPM-V 4.5 model includes: visual coding layer and text coding layer;

[0054] The standard text data in the structured data is input into a text encoding layer to obtain feature text, and the standard image data in the structured data is input into a visual encoding layer to obtain feature images and visual recognition results.

[0055] The visual encoding layer (SigLIP 2-400M) contains a large number of general image convolution kernels (such as recognizing "sky" and "land"), which are useless for pest and disease recognition. The scheme is to retain the convolution kernels for recognizing "leaf spots", "insect morphology" and "disease site" (about 70% are retained) through heat map analysis, and to crop the general convolution kernels (the cropping ratio is 25%); the visual feature dimension is reduced from 512 to 384, and the parameter redundancy is reduced;

[0056] The text encoding layer contains a large number of general semantic vectors (such as "today" and "yesterday"), which have low usage rate in agricultural scenarios. Through knowledge distillation (the teacher model is the original MiniCPM-V 4.5, and the student model is the distilled model), the agricultural-related semantic vectors such as "disease and pest terminology" and "control measures" are retained, and the general vector dimension is compressed (from 768 to 512); for core terms such as "trichogramma" and "rice leaf roller", their complete semantic vectors are forcibly retained to avoid loss of professional information caused by distillation;

[0057] The visual convolution kernel screening weight formula (used to retain disease and pest related convolution kernels) is expressed as follows:

[0058]

[0059] wherein, is the screening weight of the kth convolution kernel, is the convolution kernel response region, is the i-th disease and pest target annotation region, and m is the number of annotation samples, is the intersection over union (range 0-1), and the convolution kernel with is retained; the text semantic distillation loss function is expressed as follows:

[0060]

[0061] wherein, is the total number of agricultural terms, is the term semantic vector of the teacher model (original MiniCPM-V 4.5), is the term semantic vector of the student model (distilled model), is the cosine similarity function, denotes the text semantic distillation loss function;

[0062] S3: based on the GraphRAG retrieval model, an agricultural adapted GraphRAG retrieval model is constructed, the feature text and the feature image are input into the agricultural adapted GraphRAG retrieval model, and the entity, the community where the entity is located, the agricultural knowledge oriented abstract and the prevention and control core measure are obtained by using the agricultural knowledge graph as a retrieval knowledge base, the agricultural adapted GraphRAG retrieval model comprises:

[0063] a time-space coordinated hierarchical community detection algorithm and an agricultural knowledge oriented abstract pruning strategy, the agricultural knowledge oriented abstract pruning strategy comprises an agricultural key information screening script, and the agricultural key information screening script comprises a term matching module, an information entropy calculation module, a pruning execution module and an abstract formatting module;

[0064] Wherein, based on the GraphRAG retrieval model, an agricultural adapted GraphRAG retrieval model is constructed, and the specific process is as follows:

[0065] Taking the GraphRAG retrieval model as the framework, the time-space coordinated hierarchical community detection algorithm and the agricultural knowledge oriented abstract pruning strategy are added in the GraphRAG retrieval model to obtain the agricultural adapted GraphRAG retrieval model;

[0066] The time-space coordinated hierarchical community detection algorithm is based on the Leiden algorithm open source code, and a "time-space" calculation module is added, and a Calculation function is written, the weight is determined through agricultural data cross validation, and the specific implementation steps are as follows:

[0067] Environment configuration installs dependent library: pip install python-louvain numpy pandas networkx, wherein python-louvain is used for basic community detection, networkx is used for graph structure processing, and numpy is used for numerical calculation;

[0068] Core module code implementation, including five steps of graph structure loading, basic module degree calculation, time-space factor calculation, community division comprehensive score calculation and community division execution;

[0069] The community division comprehensive score calculation introduces the incidence season weight And the correlation degree of the occurrence area , the traditional module The community division comprehensive score is calculated, and the formula is as follows:

[0070]

[0071] Wherein, Indicates the community division comprehensive score, This represents the modularity of the traditional Leiden algorithm. Indicates the seasonal weighting of pest and disease outbreaks. Indicates the degree of regional correlation. Indicates the weighting coefficient;

[0072] Algorithm optimization: To address the sparsity of agricultural pest and disease knowledge graphs, the resolution parameter of the Leiden algorithm was optimized by setting resolution=0.8 to ensure close relationships between entities within a community while avoiding a decrease in retrieval efficiency due to an excessive number of communities; a community merging mechanism was introduced to merge small communities with fewer than 5 nodes into adjacent large communities to improve the representativeness of the communities.

[0073] The agricultural knowledge-oriented summary pruning strategy involves: developing an "agricultural key information screening script," identifying and retaining specialized terms based on a terminology dictionary, calculating information entropy using Transformer, and pruning redundant statements. Specifically, this includes the following:

[0074] Terminology matching module: Loads an agricultural pest and disease terminology dictionary, uses regular expressions to match technical terms in the text, and marks statements containing the terms as candidate keywords;

[0075] Information entropy calculation module: Based on the Transformer model, the bert-base-chinese algorithm is used to calculate the information entropy of a sentence. The specific process is as follows: First, the sentence is input into the bert-base-chinese model to obtain word vectors; then, the conditional probability p(w_i|s) of each word is calculated; finally, the information entropy formula is used, as follows:

[0076]

[0077] in Let w_i represent information entropy, sum represent the summation symbol, and p(w_i|s) represent the conditional probability.

[0078] Calculate the information entropy of the statement; statements with an information entropy ≥ 1.5 are considered high information density statements.

[0079] The pruning execution module integrates terminology matching results and information entropy calculation results to filter statements that simultaneously meet the criteria of "containing at least one professional term" and "information entropy ≥ 1.5". The filtered statements are sorted in descending order of information entropy, and the top 80% of the statements are retained as the summary content. Statements containing core industry terms such as "GB / T", "safety interval", and "prevention window period" are forcibly retained to ensure that professional information is not lost.

[0080] Abstract formatting module: Reorganizes the pruned statements in the logical order of "morphological characteristics → occurrence pattern → prevention and control methods → precautions" to generate a structured community abstract;

[0081] The domain dictionary enhanced semi-supervised knowledge extraction model optimization steps are as follows:

[0082] The architecture aspect takes BERT-BiLSTM-CRF as the backbone, and the pre-training word vector (trained by Word2Vec, dimension 300) of the "agricultural pest terminology dictionary" is integrated into the BERT word embedding layer to enhance the model's semantic perception of terms such as "rice leaf roller" and "25% abamectin emulsion". After the model extracts entities / relationships, Prompt guidance correction is added, such as: "Please check the following extraction results: 1. Entity: 'abamectin' (type: fungicide); 2. Relationship: 'rice leaf roller → harm crops → wheat', please judge whether it is correct according to agricultural knowledge, if it is wrong, please correct and explain the basis (reference GB / T 8321.9-2018)". The model corrects the "abamectin type misjudgment" (changed to "insecticide") and "harm crops error" (changed to "rice") through self-correction;

[0083] Semi-supervised training strategy: Stage 1 uses 1000 artificial labeled data (including long-tail entities) to train the model's basic parameters, ensuring that the core entity (such as rice leaf roller) extraction F1 value ≥ 85%; Stage 2 uses 5000 unlabeled data to generate "pseudo-labels", and selects samples with pseudo-label confidence ≥ 0.85 (about 3800) to merge with artificial labeled data as the training set; Stage 3 introduces a "domain adaptation loss function" to increase the punishment weight of samples with agricultural terminology extraction errors (such as terminology type misjudgment punishment weight × 1.5), and improve the model's sensitivity to professional knowledge.

[0084] S4: The entity, entity community, agricultural knowledge oriented summary, prevention and control core measures, and visual recognition results are integrated using the distillation MiniCPM-V 4.5 model to obtain a structured answer, which includes the recognition result, morphological characteristics, occurrence regularity, prevention and control scheme, and data source;

[0085] Among them, the entity, entity community, agricultural knowledge oriented summary, prevention and control core measures, and visual recognition results are integrated using the distillation MiniCPM-V 4.5 model to obtain a structured answer, which specifically includes:

[0086] The visual recognition features and the knowledge graph constructed in the search results are integrated to give a structured answer;

[0087] To solve the problem of high memory occupation and slow response of the original MiniCPM-V 4.5 model, a "multi-modal feature distillation strategy" is designed to compress the model while preserving key agricultural information:

[0088] Visual feature distillation: the visual encoding layer (SigLIP 2-400M) of MiniCPM-V 4.5 contains a large number of general image convolution kernels (such as identifying "sky" "land"), which are useless for pest identification; the scheme is to retain the convolution kernels that identify "leaf spots" "insect morphology" "disease site" (about 70% are retained) through heat map analysis, and crop general convolution kernels (crop ratio 25%); reduce the visual feature dimension from 512 to 384, reduce parameter redundancy.

[0089] Text feature distillation: the model text encoding layer contains a large number of general semantic vectors (such as "today" "yesterday"), which have low usage rate in agricultural scenarios. Through knowledge distillation (the teacher model is the original MiniCPM-V 4.5, and the student model is the distilled model), the "disease and pest terminology" "prevention and control measures" and other agricultural related semantic vectors are retained, and the general vector dimension is compressed (from 768 to 512); for core terms such as "Trichogramma" and "rice leaf roller", the complete semantic vector is forced to be retained to avoid the loss of professional information caused by distillation;

[0090] S5: The structured answer is displayed on the end-side interface to obtain the accurate consulting demand answer display, and the end-side interface is an interface in the development of the end-side application;

[0091] Among them, the accurate consulting demand answer display, the end-side interface completes multi-dimensional result presentation, realizes the "image comparison" function in the image layer, visually and horizontally compares and displays the input image uploaded by the user and the standard image built-in in the system, and intuitively presents the image matching and difference information; realize the "knowledge graph visualization" function in the knowledge association layer, based on the recognition analysis result, dynamically render and display the hierarchical association logic of "chlorantraniliprole→rice leaf roller→GB / T8321.8" in the form of node connection graph; realize the "prevention and control scheme download" function in the scheme output layer, first search the corresponding question and answer result text, then structure and encapsulate the text content according to the preset format specification and dynamically generate a PDF file, and finally provide a PDF format prevention and control scheme download return entry for the user;

[0092] End-side application development: interface development is based on Python language (BeeWare framework is adopted for mobile terminal to adapt to Android / iOS native capabilities), input area supports camera shooting / photo album uploading, and result area uses TabView to realize multi-tab page switching; MiniCPM-V model after distillation+quantization is deployed through ONNX Runtime, and image preprocessing and inference interface is written; community summary and core knowledge of 102 kinds of pests and diseases are pre-downloaded (about 500MB after light index processing, structured knowledge is stored in SQLite+FAISS vector index, and graph structure is stored in SQLite in the form of adjacency list), when there is no network, local knowledge index matching information is called first, and then knowledge enhancement reasoning is completed through the end-side model. The specific implementation details are as follows:

[0093] Technology stack selection: BeeWare+FastAPI lightweight service architecture is adopted for mobile terminal, BeeWare is responsible for UI interaction, and FastAPI encapsulates local model inference interface; computer terminal (server) model deployment is VLLM framework, which supports high-throughput inference of MiniCPM-V4.5 model and improves response speed; ONNX Runtime inference engine (supports CPU / GPU / NPU acceleration, low-end machine type TFLite) is adopted for mobile terminal (end-side), which is suitable for mobile phone / tablet hardware resources; Neo4j graph database is connected, Cypher statement query is supported, SQLite-Graph extension is carried on mobile terminal to realize light graph query, light database SQLite is used for offline storage, and offline knowledge and user data are stored.

[0094] Development environment configuration: NVIDIA 3090 GPU (server inference and model distillation), PyCharm2023.2 (integrates BeeWare / Android / iOS compilation plug-in), Python3.10, Node.js16.14 (only web management background needs); mobile terminal development relies on PyCharm to complete Android / iOS compilation, and no additional development tools are needed; model conversion environment needs to be configured with ONNX Runtime Mobile, TensorRT and other optimization tool chains;

[0095] The overall process of the end-side multi-modal question and answer of the application is as follows:

[0096] Multi-modal input: the user uploads pest and disease images (JPG / PNG) through the agricultural tablet, and inputs text questions (such as "Hainan Sanya April rice, what is this disease, how to treat? ");

[0097] Data preprocessing: image Resize to 1344x1344 pixels, normalized and input into the distilled MiniCPM-V4.5 visual encoding layer; text term standardization (e.g. "treatment" -> "prevention and treatment"), input text encoding layer;

[0098] Improved GraphRAG retrieval: community retrieval, such as Figure 2 as shown: call "time-space coordinated hierarchical community detection algorithm", based on "April" (time) and "Sanya, Hainan" (space), retrieve the community of "rice" and the identified pests and diseases in the knowledge graph; take the "agricultural knowledge-oriented summary" of the community, which contains the key information of the control agents, dosage, and application time of the disease and pest;

[0099] Multi-modal reasoning and question answering generation: the model fuses the visual understanding results (identified as "rice leaf roller" and "rice blast") and the community summary retrieved by GraphRAG, and generates a structured answer. The answer contains "identification result (confidence [93.3]%) -> morphological characteristics -> occurrence regularity -> control scheme (including GB standard) -> data source";

[0100] The result is displayed on the terminal side interface, which supports "image comparison" (input image and standard image), "knowledge graph visualization" (chlorantraniliprole -> rice leaf roller -> GB / T8321.8), and "control scheme download" (PDF format).

[0101] The present application can be modified as follows, all within the scope of protection: agricultural pest type expansion: by adding "fruit tree pest" data (such as apple rot disease), the system can support 200+ types of pests and diseases, and only needs to retrain the knowledge extraction model and GraphRAG community detection parameters; if it is a disease identification of other types such as fish, it cannot be supported. Model upgrade: when MiniCPM-V releases a higher version (such as 5.0), the model weight can be directly replaced, the existing distillation strategy and fine-tuning data can be reused, and only 1-2 rounds of fine-tuning adaptation are required; cloud collaboration: the terminal side and the cloud form a "light reasoning + massive knowledge" collaboration, the terminal side relies on ONNX Runtime to handle common pests and diseases, and complex problems (such as new pests and diseases) are uploaded to the cloud, which calls the complete GraphRAG and expert library answer, and the terminal and cloud reasoning is based on the unified ONNX model format and API interface to realize seamless connection, supporting differential model update and knowledge base incremental synchronization.

[0102] The contents not described in detail in the specification belong to the prior art known to those skilled in the art.

[0103] Finally, it should be noted that the above detailed description is merely illustrative of the patent technical solutions and is not limiting, and although the patent has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the patent can be modified or equivalently replaced without departing from the spirit and scope of the patent, and all should be covered in the scope of the claims of the patent.

Claims

1. A multimodal knowledge graph-based question-and-answer method for pests and diseases that integrates agricultural environmental characteristics, comprising: S1: Collect and preprocess text and image data, construct a structured dataset based on the text and image data, and construct an agricultural pest and disease knowledge graph, which includes entity design, knowledge extraction, and knowledge storage. S2: Based on the MiniCPM-V 4.5 model, construct a distillation MiniCPM-V 4.5 model. Input structured data into the distillation MiniCPM-V 4.5 model to obtain feature text, feature images, and visual recognition results. The visual recognition results are used to generate structured answers. The distillation MiniCPM-V 4.5 model includes: a visual encoding layer and a text encoding layer. S3: Based on the GraphRAG retrieval model, an agriculture-adapted GraphRAG retrieval model is constructed. Feature text and feature images are input into the agriculture-adapted GraphRAG retrieval model, and an agricultural knowledge graph is used as the retrieval knowledge base to obtain entities, the communities in which the entities belong, agricultural knowledge-oriented summaries, and core prevention and control measures. The agriculture-adapted GraphRAG retrieval model includes two strategies: a time-series-spatial collaborative hierarchical community detection algorithm and an agricultural knowledge-oriented summary pruning strategy. The agricultural knowledge-oriented summary pruning strategy includes an agricultural key information screening script, which includes a term matching module, an information entropy calculation module, a pruning execution module, and a summary formatting module. S4: The entity, the community where the entity is located, the agricultural knowledge-guided summary, the core prevention and control measures, and the visual recognition results are integrated using the distillation MiniCPM-V 4.5 model to obtain a structured answer. The structured answer includes the recognition results, morphological features, occurrence patterns, prevention and control plans, and data sources. S5: The structured answers are displayed using the client-side interface to obtain accurate answers to consultation needs. The client-side interface is the interface in client-side application development.

2. The multimodal knowledge graph-based pest and disease question-and-answer method integrating agricultural environmental characteristics as described in claim 1, characterized in that, In step S1, the preprocessing includes text data preprocessing and image data preprocessing, as detailed below: Text data preprocessing involves removing duplicates, standardizing terms, and splitting sentences from the collected text data to obtain standard text data. Image data preprocessing involves labeling the collected image data in COCO format, with labels including "pest / disease type, affected area, and severity," to obtain standard image data. The specific process for constructing the structured dataset is as follows: Using standard text and image data, structured data is constructed, with each entry containing "multiple rounds of user questions + professional answers + data source tracing," thus building a structured dataset; The construction of the agricultural pest and disease knowledge graph includes: 1) Entity design: Based on traditional entities, add "natural enemy organism" entities and "natural enemy inhibition" relationships to form a new ontology system. The new ontology system is specifically a head entity + relationship + tail entity triplet structure. 2) Knowledge extraction: Using BERT-BiLSTM-CRF as the backbone, pre-trained word vectors from the "Agricultural Pests and Diseases Terminology Dictionary" are integrated into the BERT word embedding layer to construct a domain dictionary-enhanced semi-supervised knowledge extraction model. The model is used to extract entities and relationships from the ontology system. 3) Knowledge storage: Stored using Neo4j graph database, supporting Cypher query statements.

3. The multimodal knowledge graph-based pest and disease question-and-answer method integrating agricultural environmental characteristics as described in claim 1, characterized in that, In step S2, structured data is input into the distillation MiniCPM-V 4.5 model to obtain feature text data, feature image data, and visual recognition results. The specific process is as follows: Standard text data from structured data is input into a text encoding layer to obtain feature text, and standard image data from structured data is input into a visual encoding layer to obtain feature images and visual recognition results.

4. The multimodal knowledge graph-based pest and disease question-and-answer method integrating agricultural environmental characteristics according to claim 1, characterized in that, In S3, an agriculture-adapted GraphRAG retrieval model is constructed based on the GraphRAG retrieval model. The specific process is as follows: Using the GraphRAG retrieval model as the backbone, a time-space collaborative hierarchical community detection algorithm and an agricultural knowledge-oriented summary pruning strategy are added to the GraphRAG retrieval model to obtain an agricultural-adapted GraphRAG retrieval model; The time-space collaborative hierarchical community detection algorithm is based on the Leiden algorithm open-source code, with the addition of a "time-space" calculation module, and is written as follows: The calculation function, with weights determined through cross-validation using agricultural data, is as follows: Configure the environment and install dependencies: pip install python-louvain numpy pandas networkx, where python-louvain is used for basic community detection, networkx is used for graph structure processing, and numpy is used for numerical computation. The core module code implementation includes five steps: graph structure loading, basic module degree calculation, temporal-spatial factor calculation, community division comprehensive score calculation, and community division execution. Algorithm optimization: In view of the sparsity of agricultural pest and disease knowledge graph, the resolution parameter of Leiden algorithm is optimized and set to resolution=0.8 to ensure that entities within the community are closely related, while avoiding the decrease in retrieval efficiency due to too many communities. A community merging mechanism is introduced to merge small communities with fewer than 5 nodes into adjacent large communities, thereby improving the representativeness of the communities. The agricultural knowledge-oriented summary pruning strategy involves: developing an "agricultural key information screening script," identifying and retaining specialized terms based on a terminology dictionary, calculating information entropy using Transformer, and pruning redundant statements. Specifically, this includes the following: Terminology matching module: Loads an agricultural pest and disease terminology dictionary, uses regular expressions to match technical terms in the text, and marks statements containing the terms as candidate keywords; Information entropy calculation module: Based on the Transformer model, the bert-base-chinese algorithm is used to calculate the information entropy of a sentence. The specific process is as follows: First, the sentence is input into the bert-base-chinese model to obtain word vectors; then, the conditional probability p(w_i|s) of each word is calculated; finally, the information entropy formula is used, as follows: ; in Let w_i represent information entropy, sum represent the summation symbol, and p(w_i|s) represent the conditional probability. Calculate the information entropy of the statement; statements with an information entropy ≥ 1.5 are considered high information density statements. The pruning execution module integrates terminology matching results and information entropy calculation results to filter statements that simultaneously meet the criteria of "containing at least one professional term" and "information entropy ≥ 1.5". The filtered statements are sorted in descending order of information entropy, and the top 80% of the statements are retained as the summary content. Statements containing core industry terms such as "GB / T", "safety interval", and "prevention window period" are forcibly retained to ensure that professional information is not lost. Summary formatting module: Reorganizes the pruned statements in the logical order of "morphological characteristics → occurrence pattern → prevention and control methods → precautions" to generate a structured community summary.

5. A system for a question-and-answer method for pests and diseases using a multimodal knowledge graph that integrates agricultural environmental characteristics, as described in any one of claims 1-4, characterized in that... include: Multi-source data construction module: used to crawl, clean, and label text and image data from professional platforms to construct an "image-multi-turn question answering" dataset and a professional corpus; Knowledge graph construction module: integrates "semi-supervised knowledge extraction model with domain dictionary enhancement", extracts entities and relations, and constructs and stores agricultural pest and disease knowledge graph; Improved GraphRAG module: Includes "temporal-spatial collaborative hierarchical community detection algorithm" and "agricultural knowledge-oriented summary pruning strategy" to achieve efficient knowledge graph retrieval and summary generation; Multimodal fine-tuning module: Fine-tunes the MiniCPM-V 4.5 model based on the "image-multi-turn question answering" dataset, and optimizes edge deployment by combining the "multimodal feature distillation strategy"; Edge-side inference module: Deploys the fine-tuned model, receives multimodal queries, calls the improved GraphRAG module to retrieve knowledge, and provides structured answers; Data update module: Supports data synchronization from professional platforms to continuously optimize model and knowledge graph performance.

6. The system according to claim 5, characterized in that, The multi-source data construction module includes a "data traceability mechanism" that labels each piece of data with its source platform and release time, ensuring high data credibility.

7. The system according to claim 5, characterized in that, The improved GraphRAG module supports "dynamic community updates". When a pest or disease entity is added to the knowledge graph, the "temporal-spatial collaborative hierarchical community detection algorithm" is automatically triggered to update the community division and summary. There is no need to retrain the model, and the update time is ≤30 minutes.

Images