Multi-modal agricultural knowledge graph construction method based on large language model

Abstract

The invention provides a multi-modal agricultural knowledge graph construction method based on a large language model. The method comprises the following steps: firstly, carrying out standardized preprocessing on multi-modal data such as texts, images and sensors in the agricultural field; constructing an initial agricultural knowledge ontology by adopting a top-down and bottom-up combined mixed mode, and identifying a new concept based on a large language model to realize dynamic updating of the ontology; fusing the text entity, the image feature vector and the sensor time sequence feature into a multi-modal triple and storing the multi-modal triple into an image database; and finally, analyzing a user query intention by using a large language model, and performing multi-hop path reasoning in combination with a graph neural network to generate a structured result. The problems that in the prior art, multi-modal fusion is difficult, knowledge updating lags behind, and semantic reasoning capacity is weak are solved, the integrity, timeliness and intelligent decision support capacity of the knowledge graph are improved, lightweight deployment and a federated learning mechanism can be combined, an edge computing scene can be adapted, and the data privacy protection requirement is met.

Description

Technical Field

[0001] This invention relates to the fields of agricultural information technology and artificial intelligence technology, specifically to a method for constructing a multimodal agricultural knowledge graph based on a large language model. Background Technology

[0002] With the improvement of agricultural informatization, agricultural information systems have accumulated massive amounts of multimodal data, including text, images, and sensor data. Knowledge graphs, as structured semantic network knowledge bases, demonstrate powerful knowledge organization and semantic association capabilities in the agricultural field. However, existing methods for constructing agricultural knowledge graphs face the following technical bottlenecks: Insufficient multimodal data processing capabilities: Traditional methods mainly rely on single text data for construction, making it difficult to effectively integrate multimodal information such as images (e.g., images of diseased fruit spots on trees) and sensor data (e.g., soil moisture, temperature, and humidity). For example, when processing wheat stripe rust, existing methods only extract symptoms through text descriptions, failing to correlate disease image features with environmental sensor data (e.g., high incidence when temperature ≥20℃ and humidity >85%), resulting in fragmented and incomplete semantic representation of the data.

[0003] The lack of a dynamic knowledge update mechanism: Agricultural knowledge is highly time-sensitive (such as new crop varieties, pesticide resistance variations in pests and diseases, and the impact of climate change), but the updating of existing static knowledge maps relies heavily on human intervention and cannot respond to data changes in real time. For example, when new pesticides are launched or pesticide resistance monitoring data is updated, traditional methods struggle to update prevention and control plans in a timely manner, resulting in knowledge maps lagging behind actual production needs.

[0004] Weak cross-domain semantic association: The agricultural industry chain covers multiple links such as planting, processing, storage, and sales, but existing maps are mostly limited to a single sub-domain (such as crop planting or pest and disease control), lacking the ability to establish cross-domain semantic associations (such as the chain relationship of "crop variety - growing environment - processing technology - market demand"). For example, agricultural product traceability maps do not integrate planting environment data and processing quality standards, making it difficult to support risk management and decision optimization throughout the entire chain from field to table.

[0005] Limited ability for complex semantic reasoning: Traditional reasoning relies on simple rule engines (such as "temperature > 30℃ → increased irrigation demand"), but lacks the ability for multi-hop reasoning and implicit relationship mining in complex scenarios (such as the interaction of multiple factors affecting crop yield and the evolution mechanism of pest and disease resistance). For example, it is difficult to achieve the linkage analysis of precision fertilization and disease prevention through multi-layer semantic chains of "soil pH value → suitable fertilizer type → crop root development → disease resistance". Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for constructing a multimodal agricultural knowledge graph based on a large language model, so as to realize the effective fusion of multimodal data, dynamic knowledge updating and intelligent semantic reasoning.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for constructing a multimodal agricultural knowledge graph based on a large language model includes the following steps: S1. Acquire text data, image data, and sensor data in the agricultural field, and perform standardized preprocessing on the multimodal data; S2. An initial agricultural knowledge ontology is constructed using a hybrid approach combining top-down and bottom-up methods, and new concepts are identified based on a large language model to achieve dynamic updates of the ontology. S3. Fuse text entities, image feature vectors, and sensor temporal features into multimodal triples and store them in a graph database; S4. Use a large language model to parse the user's query intent, and combine a graph neural network to traverse the knowledge graph path to generate structured reasoning results.

[0008] Furthermore, in step S1: the text data includes unstructured text from agricultural journals, agricultural technology Q&A platforms, and e-commerce reviews; the image data includes crop lesion images and agricultural machinery operation videos, using a target detection model to extract lesion area features, and generating image feature vectors through a cross-modal model; the sensor data includes time-series data collected by weather stations and soil sensors, using a time-series model to extract features and map them to a knowledge graph relationship chain.

[0009] Furthermore, in step S2: the top-down mode defines the core ontology structure by domain experts; the bottom-up mode analyzes agricultural texts through a large language model, automatically extracts sub-concepts, and optimizes the ontology hierarchy; the dynamic update includes periodically crawling agricultural literature, identifying new concepts, and determining their relationship with existing entities through entity linking technology to trigger ontology updates.

[0010] Furthermore, in step S3: the multimodal triple includes entities, relations and corresponding multimodal attributes; a graph database is used to store entity relations, the sensor raw data is stored in a distributed file system, and associated with graph nodes through indexes.

[0011] Furthermore, in step S4: the large language model is used to parse the user's natural language query and identify key entities and query intent; the graph neural network is used to perform multi-hop path traversal in the knowledge graph and combine reinforcement learning to optimize the reasoning path; the output result is structured information containing variety name, resistance data and planting suggestions.

[0012] Furthermore, it also includes a lightweight deployment solution: using knowledge distillation and quantization techniques to compress large language models, reducing model size and computing resource requirements; and adapting to agricultural IoT devices and mobile application scenarios.

[0013] Furthermore, it also includes a federated learning mechanism: jointly training models across multiple farms to construct regional knowledge graphs; achieving knowledge fusion without sharing original data, thus meeting data privacy protection requirements.

[0014] As can be seen from the above technical solutions, the present invention has the following technical advantages compared with the prior art: 1. This invention uses multimodal data fusion to uniformly represent text, image, and sensor data, significantly improving the completeness and richness of the knowledge graph; 2. This invention adopts a dynamic ontology update mechanism to realize real-time updates of agricultural knowledge, ensuring the timeliness and accuracy of the atlas; 3. This invention combines the inference frameworks of LLM and GNN to support multi-hop inference and implicit relationship mining in complex scenarios; 4. This invention provides a lightweight deployment solution that adapts to the computing power limitations of agricultural edge computing scenarios; 5. This invention adopts a federated learning mechanism to achieve cross-farm knowledge sharing while protecting data privacy. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the steps of the multimodal agricultural knowledge graph construction method based on a large language model according to the present invention. Detailed Implementation

[0016] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0017] like Figure 1 The method for constructing a multimodal agricultural knowledge graph based on a large language model, as shown, is based on multimodal data fusion and realizes the automated construction and application of agricultural knowledge graphs through dynamic ontology construction and intelligent reasoning engine.

[0018] Example 1: Construction of a Knowledge Graph for Crop Disease and Pest Diagnosis and Control Based on Multimodal Data This embodiment uses wheat stripe rust as an example to illustrate the specific implementation process of this method: S1. Multimodal data acquisition and preprocessing: Acquire text data, image data, and sensor data in the agricultural field, and perform standardized preprocessing on the multimodal data.

[0019] The text data described in this embodiment was crawled from unstructured text sources such as the agricultural journal "Transactions of the Chinese Society of Agricultural Engineering", the agricultural technology Q&A platform "Nongguanjia", and e-commerce reviews, collecting a total of 23,000 text entries on "wheat stripe rust". The spaCy tool was used for word segmentation and part-of-speech tagging, and keywords were extracted using the TF-IDF method to construct an agricultural thesaurus.

[0020] The image data described in this embodiment consists of 1,500 wheat field images collected by drones. The YOLOv8 model is used to detect lesion areas and extract features such as lesion area ratio, color distribution (e.g., yellowish-brown accounts for 85%), and edge complexity (e.g., fractal dimension 1.8) to establish a "disease image-wheat" labeled dataset. The CLIP model is used to generate image feature vectors, and cross-modal mapping is established with the "symptom" entity in the text data to form a "wheat stripe rust-symptom-image" feature vector.

[0021] The sensor data described in this embodiment is obtained by accessing a weather station to acquire temperature, humidity, and light data, and by using a soil sensor to acquire pH value and nutrient content data. The LSTM-Attention model is used to extract time-series features (such as the frequency of humidity >90% for three consecutive days and temperature change trends).

[0022] The aforementioned feature vectors are mapped to the "environment-crop growth" relationship chain of the knowledge graph using knowledge graph embedding technology (ComplEx algorithm).

[0023] S2. Dynamic Ontology Construction and Update: An initial agricultural knowledge ontology is constructed using a hybrid approach combining top-down and bottom-up methods. New concepts are identified based on a large language model to achieve dynamic updates of the ontology and realize multimodal knowledge fusion and storage.

[0024] The initial agricultural knowledge ontology construction described in this embodiment includes: From top to bottom, experts through The tool defines the core ontology, such as "wheat (crop) → diseases → stripe rust → disease conditions → control measures".

[0025] Bottom-up approach: The LLM Large Language Model (GPT-4) is used to analyze the text and automatically extract the sub-concepts of the core concept. For example, the sub-concepts of "striped rust" such as "yellow rust" and "leaf rust" are clustered into the same entity by the hierarchical clustering algorithm DBSCAN, thereby reducing concept redundancy.

[0026] The dynamic update described in this embodiment includes: Regularly crawl agricultural journals (such as PlantDisease), use the BERT-Whitespace model to detect new words, identify the new concept "Zhengmai 1860, a new wheat variety resistant to stripe rust", determine it as a subclass of "wheat variety" through entity linking technology Dedupe, and find that there are no synonyms or subclasses of this entity, and automatically update the ontology.

[0027] S3. Multimodal knowledge fusion and storage: Text entities, image feature vectors and sensor time-series features are fused into multimodal triples and stored in a graph database.

[0028] The multimodal triplet constructed in this embodiment is: <wheat, susceptible diseases, stripe rust, image features: [area ratio 0.15, color yellowish-brown, edge complexity 1.8], environmental threshold: [temperature 15-22℃, humidity >85%], control measures: [spraying triadimefon, planting disease-resistant varieties]>.

[0029] The data is stored in the Neo4j graph database, with 850,000 associated nodes and 4.2 million edges.

[0030] S4. Intelligent Reasoning and Application: Utilize a large language model to parse user query intent, combine graph neural networks to traverse the knowledge graph path, and generate structured reasoning results; realize multi-hop reasoning and agricultural scenario applications based on knowledge graph.

[0031] For example, a user enters a natural language query: "How to prevent and control wheat stripe rust in Shandong Province?" The entity ["Shandong", "wheat stripe rust", "prevention and control"] was parsed using an LLM large language model (such as Flan-T5), and the intent was identified as "recommendation of disease prevention and control plan".

[0032] GNN traversal path: Shandong → Climate characteristics (temperate monsoon climate) → Common diseases (stripe rust); Stripe rust → Control measures (triadimefon) → Disease-resistant varieties (Zhengmai 1860); Based on sensor data, it is determined that the current period is a high-incidence period (humidity 87%, temperature 18℃). Output structured results: Recommended action: "Spray triadimefon immediately, at a dosage of 40 ml per acre." Disease-resistant variety: "Zhengmai 1860" Warning: "The current environment is at a high risk of infection. It is recommended to complete the application of pesticides within 3 days."

[0033] In practical applications, reinforcement learning models (such as PPO) can be combined to optimize the recommendation path and generate structured results that include variety names, resistance data, and planting suggestions.

[0034] Example 2: Lightweight Deployment and Federated Learning Regional Crop Recommendation System The lightweight deployment described in this embodiment is suitable for mobile pest and disease diagnosis scenarios: Using distillation technology to compress LLM (MobileBERT) instead of heavy LLM reduces model size by 70%; When used in conjunction with the lightweight CV model MobileNet-v2 for image processing, inference speed is improved by 2 times; Offline pest and disease diagnosis can be implemented in an Android app with accuracy loss controlled within 5%.

[0035] The federated learning application described in this embodiment, while protecting farmers' privacy, constructs a regional map by combining data from multiple farms through federated learning, thus solving the data silo problem and complying with GDPR (General Data Protection Regulation) requirements. A soil pH-maize variety adaptation model was trained in collaboration with three county-level farms in Shandong Province. Each farm only uploads model gradients and does not share specific plot data; After constructing a regional knowledge graph, the accuracy of corn variety recommendations increased to 88%, while also complying with data privacy regulations.

[0036] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for constructing a multimodal agricultural knowledge graph based on a large language model, characterized in that, Includes the following steps: S1. Acquire text data, image data, and sensor data in the agricultural field, and perform standardized preprocessing on the multimodal data; S2. An initial agricultural knowledge ontology is constructed using a hybrid approach combining top-down and bottom-up methods, and new concepts are identified based on a large language model to achieve dynamic updates of the ontology. S3. Fuse text entities, image feature vectors, and sensor temporal features into multimodal triples and store them in a graph database; S4. Use a large language model to parse the user's query intent, and combine a graph neural network to traverse the knowledge graph path to generate structured reasoning results.

2. The method for constructing a multimodal agricultural knowledge graph based on a large language model according to claim 1, characterized in that, In step S1: the text data includes unstructured text from agricultural journals, agricultural technology Q&A platforms, and e-commerce reviews; the image data includes crop lesion images and agricultural machinery operation videos, using a target detection model to extract lesion area features and generating image feature vectors through a cross-modal model; the sensor data includes time-series data collected by weather stations and soil sensors, using a time-series model to extract features and map them to a knowledge graph relationship chain.

3. The method for constructing a multimodal agricultural knowledge graph based on a large language model according to claim 1, characterized in that, In step S2: the top-down mode defines the core ontology structure by domain experts; the bottom-up mode analyzes agricultural texts through a large language model, automatically extracts sub-concepts and optimizes the ontology hierarchy; the dynamic update includes periodically crawling agricultural literature, identifying new concepts and determining their relationship with existing entities through entity linking technology, triggering ontology updates.

4. The method for constructing a multimodal agricultural knowledge graph based on a large language model according to claim 1, characterized in that, In step S3: the multimodal triple includes entities, relations and corresponding multimodal attributes; a graph database is used to store entity relations, the sensor raw data is stored in a distributed file system, and associated with graph nodes through indexes.

5. The method for constructing a multimodal agricultural knowledge graph based on a large language model according to claim 1, characterized in that, In step S4: the large language model is used to parse the user's natural language query and identify key entities and query intent; the graph neural network is used to perform multi-hop path traversal in the knowledge graph and combine reinforcement learning to optimize the reasoning path. The output is structured information that includes variety name, resistance data, and planting recommendations.

6. The method for constructing a multimodal agricultural knowledge graph based on a large language model according to claim 1, characterized in that, It also includes a lightweight deployment solution: using knowledge distillation and quantization techniques to compress large language models, reducing model size and computing resource requirements, and adapting to agricultural IoT devices and mobile application scenarios.

7. The method for constructing a multimodal agricultural knowledge graph based on a large language model according to claim 1, characterized in that, It also includes a federated learning mechanism: jointly training models across multiple farms to build regional knowledge graphs, achieving knowledge fusion without sharing the original data, and meeting data privacy protection requirements.

Images