Agricultural knowledge base construction method based on large language model

Abstract

The invention discloses an agricultural knowledge base construction method based on a large language model, relates to the field of agricultural knowledge base construction, and aims to solve the problems of dependence on a large amount of manpower, poor data quality, inaccurate search and the like in existing agricultural knowledge base construction. The method comprises the following steps: firstly, collecting agricultural knowledge texts from multiple channels such as professional books and academic papers, and segmenting the agricultural knowledge texts into smaller knowledge fragments to form an original knowledge fragment information base; then, a knowledge corpus set containing questions, abstracts and keywords is generated by designing interaction between a prompt project and the large language model; and then the corpora are converted into a vector form by using an NLP technology, a problem index vector library and an auxiliary index vector library are constructed, and the retrieval efficiency and accuracy are improved. And further automatically extracting keywords by using a large language model to form a structured keyword set. And finally, forming a comprehensive and structured agricultural knowledge base, and ensuring the quality and timeliness of the knowledge base through continuous dynamic optimization and user feedback.

Description

Technical Field

[0001] The present invention relates to the field of agricultural knowledge base construction, and in particular to a method for constructing an agricultural knowledge base based on a large language model. Background Art

[0002] Agriculture is one of the basic industries in modern society and has accumulated a large number of important research results. These results can promote crop yields, improve the stability of agricultural production, and provide key technical support for sustainable agricultural development, which will help improve the production quality of crops and promote rural economic growth. However, the knowledge system in the agricultural field is very complex, including seed science, soil management, crop cultivation, water resources management, biotechnology, and agricultural machinery. It is a multidisciplinary knowledge system involving biology, chemistry, ecology, engineering, etc. Therefore, it is not easy to make full use of the knowledge contained in agricultural research results.

[0003] An agricultural knowledge base refers to the classified storage and management of a large number of valuable and important research results in the agricultural field, the accumulation of knowledge assets, the avoidance of knowledge loss, the promotion of the learning, sharing, training, reuse and innovation of agricultural knowledge, the effective reduction of agricultural production costs and the improvement of agricultural production efficiency.

[0004] There are still many drawbacks in the construction of the existing knowledge base for agricultural scientific research results: (1) Relying on a large amount of manpower to input, upload and edit content, which not only consumes time and resources, but also has no unified standards and specifications, resulting in inconsistent formats, structures and styles of knowledge entries, thus affecting effective knowledge retrieval; (2) Existing agricultural knowledge bases are prone to data quality problems, such as data errors, duplications, incompleteness, or outdatedness, which directly affect the accuracy and credibility of the results; (3) Existing knowledge base search engine technology cannot accurately capture users’ query intent, context, and personal preferences, resulting in the returned search results not meeting users’ actual needs and poor retrieval results. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method for constructing an agricultural knowledge base based on a large language model. First, agricultural-related knowledge texts are collected from professional sources and divided into smaller knowledge fragments to form an agricultural original knowledge fragment information library. Then, a prompt project is designed to interact with the large language model to generate an agricultural knowledge corpus containing questions, summaries and keywords. These corpora are then converted into vector form to construct a question index vector library and an auxiliary index vector library to improve retrieval efficiency and accuracy. Further, a large language model is used to automatically extract keywords to form a rich keyword vocabulary. Finally, all libraries are integrated to form a structured and complete agricultural knowledge base, and the quality and timeliness of the knowledge base are guaranteed through continuous dynamic optimization and verification.

[0006] To achieve the above objectives, the present invention is implemented by the following technical solutions: A method for constructing an agricultural knowledge base based on a large language model, comprising the following steps: S1: Build an agricultural original knowledge fragment information database, collect agricultural-related knowledge text information from reliable sources such as professional books, academic papers, research reports and technical documents. Then segment and process these collected materials, breaking them down into smaller knowledge fragments, thus forming an agricultural original knowledge fragment information database, laying the foundation for the subsequent construction of the knowledge base; S2: Generate an agricultural knowledge corpus. First, configure the prompt project and design a series of questions or instructions to guide the large language model to understand and extract professional knowledge in the agricultural field. Then, use these carefully designed prompt projects to interact with the large language model to generate a large amount of agricultural-related corpus, thereby establishing a rich agricultural knowledge corpus. S3: Build a question index vector library and convert the generated agricultural knowledge corpus into vector form for efficient retrieval and query. Use natural language processing techniques such as word embedding or semantic encoding models to convert text content into numerical vector representations. In addition, in order to enhance the performance and accuracy of the retrieval system, an auxiliary index vector library will be created, which includes synonyms, industry terms and their related contextual information to ensure that users can obtain the most relevant results when querying. A mapping relationship is established to ensure that each piece of agricultural knowledge content can be accurately mapped to the corresponding question index vector and auxiliary index vector, and the knowledge fragment information library is associated with their vector representations to facilitate efficient and accurate information retrieval based on these vectors; S4: Build an agricultural keyword lexicon and use a large language model to automatically extract keywords from the constructed agricultural knowledge corpus. Identify important agricultural terms and concepts by analyzing the vocabulary frequency, context relevance, and other semantic features in the corpus, and then form a structured keyword set to provide key support for subsequent knowledge retrieval; S5: Build a complete agricultural knowledge base, integrating all the resources generated in the previous steps, including the agricultural knowledge fragment information library, agricultural knowledge corpus, question index vector library, and agricultural keyword thesaurus, to form a comprehensive and structured agricultural knowledge base to facilitate efficient retrieval, management, and application of professional knowledge in the agricultural field; S6: Dynamically optimize and verify the agricultural knowledge base, continuously update the knowledge base to reflect the latest agricultural research results, and ensure the quality and timeliness of the knowledge base content through user feedback and other means.

[0007] Preferably, in said S1, it also includes: According to the construction scale, goal and type of knowledge base, targeted collection of literature, technical solutions and other knowledge text information in the field of agriculture. Develop a self-created knowledge text segmentation strategy, segment text according to the priority order of chapters, subsections, paragraphs and symbols, and ensure that each part after segmentation still maintains semantic coherence and integrity through a semantic constraint model. Use the Scrapy crawler framework to obtain agricultural literature data, retrieve keywords closely related to agricultural disciplines from relevant databases, and obtain the corresponding literature content. After obtaining the content, perform data preprocessing, including removing tags, special characters, processing missing data and abnormal characters, and use the unique identifiers of the document such as title and author to deduplicate the data to ensure that the information in the knowledge base is accurate and non-repetitive.

[0008] Preferably, in said S2, further comprising: A series of prompt projects are designed based on the constructed agricultural original knowledge fragment information library, which will guide the large language model to generate specific questions, summaries and keywords, thereby forming a comprehensive corpus. Specifically, a customized prompt project is created for each agricultural book knowledge fragment, such as asking questions about the core concepts of the fragment, requiring the summary of the main content, and extracting key terms. Next, the service interface of the large language model will be called to use these carefully designed prompt projects to generate a corpus including a question set, a summary set, and a keyword set. In order to further enrich the keyword information, the prior knowledge capabilities of the large language model will be used to expand the associated information of the keywords through the prompt word project, and finally the word embedding model will be combined to obtain the alias and synonym information of the keywords to build a detailed and accurate agricultural knowledge corpus.

[0009] Preferably, in said S3, further comprising: The generated agricultural knowledge corpus is evaluated for effectiveness and sampled, and then these annotated corpora are used to fine-tune the large language model. Once the fine-tuned model reaches the predetermined corpus effectiveness annotation accuracy threshold, the model can be used to annotate the entire corpus for effectiveness, thereby screening out effective corpora. Then, by performing text vectorization on these effective corpora, a question index vector library is constructed. At the same time, the abstract set and keyword set also need to be text vectorized to create an auxiliary index vector library. Finally, by establishing the main mapping relationship between the agricultural original knowledge fragment information library and the question index vector library, as well as the auxiliary mapping relationship between the information library and the auxiliary index vector library, the construction of the agricultural question index knowledge base is completed.

[0010] Preferably, in said S4, further comprising: The ChatGLM4.0-9B large language model was used as a keyword expander to expand the associated information of keywords in the agricultural original knowledge fragment information library through the prompt word project. By setting specific task prompt words, the model was guided to generate agricultural-related keywords and their aliases or synonyms to enrich the keyword library content. After the expansion was completed, the BGE-Large-zh-V1.5 word embedding model was used to vectorize the keywords to better capture the semantic information of the keywords themselves and their contextual meaning. A higher similarity threshold of 0.95 was set to identify keywords with high similarity, classify them into one category, and build a more comprehensive and accurate keyword thesaurus. In order to further optimize the selection of keywords, the Gensim library integrated with the TF-IDF statistical model was also used to measure the importance of keywords, filter out those words that appear too low or are too general, screen out more closely related keywords, and better enrich the keyword thesaurus.

[0011] Compared with the prior art, the present invention discloses a method for constructing an agricultural knowledge base based on a large language model. The present invention has the following beneficial effects: 1. By automatically collecting and segmenting agricultural-related knowledge text materials, smaller knowledge fragments are formed, which reduces the dependence on manpower and helps to unify the format, structure and style of knowledge entries. 2. A well-designed prompting project is used to guide the large language model to understand and extract professional knowledge, automatically generate a large amount of agricultural-related corpus, and improve the quality of content and the accuracy of retrieval. 3. Natural language processing technology is used to convert text content into numerical vector representation, and an auxiliary index vector library is created to enhance the performance and accuracy of the retrieval system, improving the existing search engine technology so that it can more accurately capture the user's query intent and context. 4. The large language model is used to automatically extract keywords, and important agricultural terms and concepts are identified by analyzing features such as vocabulary frequency and context relevance to form a rich structured keyword set. 5. The knowledge base is continuously updated to reflect the latest research results, and user feedback is collected to ensure the quality and timeliness of the knowledge base content.

[0012] Example: Application of smart agricultural knowledge base in rice planting management Background: In the rice farming industry, farmers face complex challenges such as pest and disease identification, crop growth cycle management, and irrigation and fertilization strategy formulation. Traditional methods rely on limited local knowledge and experience, and often cannot cope with the uncertainties brought about by climate change and market fluctuations. To this end, an agricultural technology company decided to adopt an intelligent agricultural knowledge base solution to improve the intelligence level of rice planting; Method implementation: S1: Collecting knowledge points related to rice planting and management S11 collects multiple data sources related to rice cultivation, including professional books and textbooks, academic papers, technical reports, weather stations, soil sensors, etc., and collects real-time data on climate conditions, soil moisture, and crop growth status; S12 uses crawler technology to capture rice pest and disease information, the latest research papers, industry standards, and agricultural policies and regulations on the Internet; S13 uses Python's Pandas library to clean data, remove outliers and duplicate records, and ensure data quality; S2: Generate a corpus of rice planting knowledge S21 extracts the key words of each knowledge segment, such as "rice blast" and "nitrogen fertilizer", from the established agricultural original knowledge segment information database to create a customized prompt project; S22 calls the service interface of the large language model and uses the above-mentioned carefully designed prompt engineering to generate a question set, a summary set and a keyword set. It also uses the prior knowledge ability of the large language model to expand the associated information of keywords through the prompt word engineering; S23 integrates the above-generated question set, summary set and keyword set, and automatically classifies them into corresponding knowledge categories to form a comprehensive rice planting knowledge corpus; S3: Construction of question index vector library S31 combines word embedding and semantic encoding models to convert the rice planting knowledge corpus into vector representation and create an auxiliary index vector library containing synonyms and industry terms. At the same time, it establishes a primary-auxiliary mapping relationship to ensure that the content corresponds accurately to the index vector, thereby enhancing the performance and accuracy of the retrieval system; S32 fine-tunes the large language model by evaluating the effectiveness and sampling and annotating the rice planting knowledge corpus. After the model reaches the predetermined accuracy threshold, it uses it to annotate the entire corpus for effectiveness to screen out effective content. S33 then performs text vectorization processing on the effective corpus of rice planting knowledge and establishes a main and auxiliary vector index library and its mapping relationship to construct a rice planting problem index knowledge base; S4: Constructing a keyword database for rice cultivation S41 uses the ChatGLM4.0-9B large language model to automatically extract keywords from the constructed rice planting knowledge corpus, and expands the associated information of keywords through the prompt word project to generate keywords related to agriculture and their aliases or synonyms, such as associating "rice smut" with "jinggangmycin" and "rice blast" with "nitrogen fertilizer"; S42 uses the BGE-Large-zh-V1.5 word embedding model to vectorize keywords and sets a similarity threshold of 0.95 to identify and classify highly similar keywords and build a comprehensive and accurate vocabulary. S43 uses the Gensim library to integrate the TF-IDF statistical model: it measures the importance of keywords, filters out words that appear too infrequently or are too general, and selects more closely related keywords; S5: Building a complete rice planting knowledge base S51 integrates all the resources generated in the previous steps, including the rice planting knowledge fragment information database, the rice planting knowledge corpus, the question index vector database, and the rice planting keyword thesaurus, to form a comprehensive and structured rice planting knowledge database; S52 dynamically optimizes and verifies the rice planting knowledge base; S53 continuously updates the knowledge base to reflect the latest rice cultivation research results; S54 ensures the quality and timeliness of knowledge base content through user feedback and other means. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those skilled in the art, other relevant drawings can be obtained based on these drawings without creative work. Figure 1 It is a flow chart of the construction of the agricultural original knowledge fragment information database of the present invention; Figure 2 is a flow chart of the present invention for generating an agricultural knowledge corpus; Figure 3 It is a flow chart of constructing a problem index vector library of the present invention; Figure 4 is a flow chart of constructing an agricultural keyword thesaurus of the present invention; Figure 5 is a flow chart of constructing a complete agricultural knowledge base of the present invention; Figure 6 It is a flow chart of the present invention for dynamically optimizing and verifying the agricultural knowledge base. DETAILED DESCRIPTION

[0014] Step 1: Build an information base of original agricultural knowledge fragments S11 determines the goal, scope and type of data collection, such as collecting information on rice cultivation. According to the scale, goal and type of knowledge base construction, targeted collection of literature, technical solutions and other knowledge text information in the agricultural field from reliable sources such as professional books, academic papers, research reports and technical documents. Text segmentation is performed according to the priority order of chapters, subsections, paragraphs and symbols, and the semantic constraint model is used to ensure that each segmented part still maintains semantic coherence and integrity; S12 uses the Scrapy+selenium+requests crawler framework to crawl relevant agricultural literature website data, retrieve keywords closely related to agricultural disciplines from relevant websites, and obtain the corresponding literature content. After obtaining the content, data preprocessing is performed, including removing tags, special characters, processing missing data and abnormal characters, and deduplicating data through the unique identifiers of the literature such as title and author to ensure that the information in the knowledge base is accurate and non-repetitive; S13 integrates the segmented and pre-processed data to form an information base of agricultural original knowledge fragments.

[0015] Step 2: Generate agricultural knowledge corpus S21 analyzes the core concepts, key terms, structure and context of the constructed agricultural original knowledge fragments, and then combines question-based prompts, summary prompts and extraction prompts to create customized prompt projects; S22 calls the service interface of the ChatGLM4.0-9B large language model and sends a request to the large language model according to the prompt project designed in the previous step to generate a corpus including a question set, a summary set, and a keyword set; S23 uses the prior knowledge of the large language model to expand the associated information of keywords through prompt word engineering, and combines the word embedding model to obtain the alias and synonym information of keywords to enrich the content of the keyword set; S24 integrates the above-generated question set, summary set and keyword set to construct a detailed and accurate agricultural knowledge corpus.

[0016] Step 3: Build a question index vector library S31 randomly extracts samples from the generated agricultural knowledge corpus and evaluates its effectiveness. The evaluation indicators include but are not limited to accuracy, completeness, relevance, etc. The sampled corpus is intelligently annotated, including the effectiveness of questions, the accuracy of summaries, and the relevance of keywords. In addition, this process retains the channel for human intervention. Fine-tune the large language model using the annotated dataset to improve its performance in the agricultural field. During the fine-tuning process, the performance of the model needs to be monitored to ensure that the predetermined accuracy threshold is reached; S32 uses the fine-tuned large language model to annotate the entire corpus for validity and select valid corpus. The selected valid question set is vectorized to build a question index vector library. Similarly, the abstract set and keyword set are also vectorized to create an auxiliary index vector library. S33 organizes all valid original knowledge fragments and their related information, including source, author, publication date, etc. Establishes the main mapping relationship between the original knowledge fragment information library and the question index vector library to ensure that users can find relevant knowledge fragments by asking questions. Creates an auxiliary mapping relationship between the auxiliary index vector library and the original knowledge fragment information library so that users can retrieve information by keywords or abstracts, completing the construction of the agricultural question index knowledge library.

[0017] Step 4: Build an agricultural keyword database The ChatGLM4.0-9B large language model is used as a keyword expander, and the associated information of the keywords in the agricultural original knowledge fragment information library is expanded through the prompt word project. By setting specific task prompt words, the model is guided to generate agricultural-related keywords and their aliases or synonyms to enrich the keyword library content. After the expansion is completed, the BGE-Large-zh-V1.5 word embedding model is used to vectorize the keywords to better capture the semantic information of the keywords themselves and their contextual meaning. A higher similarity threshold of 0.95 is set to identify keywords with high similarity, classify them into one category, and build a more comprehensive and accurate keyword thesaurus. In order to further optimize the selection of keywords, the Gensim library integrated with the TF-IDF statistical model is also used to measure the importance of keywords, filter out those words that appear too low or are too general, screen out more closely related keywords, and better enrich the keyword thesaurus; S41 designs corresponding prompt words for each keyword in the agricultural original knowledge fragment information database. The prompt words contain the keyword itself and instructions for the model to generate relevant information, such as: "Please provide other names and related terms about 'rice'". Using the ChatGLM4.0-9B model, the aliases or synonyms related to the keywords are generated based on the designed prompt words. This step will greatly enrich the content of the keyword library; S42 uses the BGE-Large-zh-V1.5 word embedding model to vectorize the expanded keywords. This model can capture the semantic information of keywords and their contextual meaning. Set a higher similarity threshold (such as 0.95) and use the cosine similarity measurement method to identify highly similar keywords. Keywords with similarity exceeding the threshold are classified into the same category to reduce duplication and improve the refinement of the vocabulary; S43 uses the TF-IDF statistical model in the Gensim library to calculate the importance of each keyword. TF-IDF can help us measure the importance of keywords in the corpus. Filter out words that appear too infrequently (low TF) or are too general (low IDF) based on the TF-IDF score. This step ensures that the retained keywords are more closely related, which helps improve the accuracy of subsequent searches and matches. Combined with the above results, organize and optimize the keyword library to ensure that it is both comprehensive and accurate, so as to build a high-quality agricultural keyword index. This will help improve the efficiency and accuracy of agricultural information retrieval systems.

[0018] Step 5: Build a complete agricultural knowledge base S51 integrates the agricultural original knowledge fragment information base, agricultural knowledge corpus, question index vector library and agricultural keyword vocabulary generated in all previous steps to form a comprehensive and structured agricultural knowledge base to facilitate the efficient retrieval, management and application of professional knowledge in the agricultural field.

[0019] Step 6: Dynamically optimize and verify the agricultural knowledge base S61 regularly monitors the latest research results and technical trends in the agricultural field. Use crawler technology to regularly crawl the latest literature, reports and technical documents from authoritative sources. Update the content in the knowledge base to ensure that it reflects the latest agricultural research results; S62 Establish a user feedback mechanism to collect problems and suggestions encountered by users when using the knowledge base. Regularly analyze user feedback to revise and improve the content of the knowledge base. Adjust the structure and retrieval function of the knowledge base according to user usage; S63 tracks the latest technological advances in the field of natural language processing, such as word embedding models, deep learning, etc. Updates the technology stack of the knowledge base and adopts more advanced technologies to improve retrieval efficiency and accuracy. Upgrades models and tools in a timely manner to adapt to new data processing requirements; S64 regularly monitors the performance of the knowledge base, including query speed, retrieval accuracy, etc. Adjust the index structure and data storage method based on the monitoring results to optimize performance. Monitor system stability to ensure that the knowledge base can still run stably under high concurrent access.

Claims

1. A method for constructing an agricultural knowledge base based on a large language model, characterized in that: The method comprises the following steps: S1: Construct an information base of original agricultural knowledge fragments, including collecting agricultural-related knowledge text information from reliable sources such as professional books, academic papers, research reports and technical documents; segmenting the collected information into smaller knowledge fragments to form an information base of original agricultural knowledge fragments; S2: Generate an agricultural knowledge corpus, including configuring prompt engineering to design a series of questions or instructions to guide the large language model to understand and extract expertise in the agricultural field; using carefully designed prompt engineering to interact with the large language model to generate an agricultural knowledge corpus containing questions, summaries and keywords; S3: Construct a question index vector library, including converting the agricultural knowledge corpus into vector form and creating an auxiliary index vector library to enhance the performance and accuracy of the retrieval system; S4: Constructing an agricultural keyword lexicon, including using a large language model to automatically extract keywords from the constructed agricultural knowledge corpus; S5: Building a complete agricultural knowledge base, including integrating the resources generated in all previous steps to form a comprehensive and structured agricultural knowledge base; S6: Dynamically optimize and verify the agricultural knowledge base, including continuously updating the knowledge base to reflect the latest agricultural research results, and ensuring the quality and timeliness of the knowledge base content through user feedback and other means.

2. The method according to claim 1, characterized in that The S1 further comprises: According to the scale, goal and type of knowledge base construction, targeted collection of literature, technical solutions and other knowledge text information in the agricultural field; Develop self-created knowledge text segmentation strategies and segment texts according to the priority order of chapters, sections, paragraphs and symbols; The semantic constraint model is used to ensure that each segmented part still maintains semantic coherence and integrity; The Scrapy crawler framework is used to obtain agricultural literature data, retrieve keywords closely related to agricultural disciplines from relevant databases, and obtain corresponding literature content; Data preprocessing includes removing labels, special characters, handling missing data and abnormal characters, and deduplicating data using unique identifiers of documents such as title and author.

3. The method according to claim 1, characterized in that The S2 further includes: Create customized prompts for each knowledge segment of agricultural books, asking questions about the core concepts of the segment, summarizing the main content, and extracting key terms; Call the service interface of the large language model and use the carefully designed prompt engineering to generate a corpus including a question set, a summary set, and a keyword set; Leverage the prior knowledge of large language models to expand the associated information of keywords through prompt word engineering; Combined with word embedding models to obtain aliases and synonyms of keywords, a detailed and accurate agricultural knowledge corpus is constructed.

4. The method according to claim 1, characterized in that: The S3 further includes: Conduct validity evaluation and sampling on the generated agricultural knowledge corpus; Use these annotated corpora to fine-tune the large language model to reach a predetermined corpus validity annotation accuracy threshold; Use the fine-tuned model to annotate the validity of the entire corpus and select valid corpus; By performing text vectorization processing on these valid corpora, a question index vector library is constructed; Create an auxiliary index vector library and establish primary and auxiliary mapping relationships.

5. The method according to claim 1, characterized in that The S4 further comprises: Using ChatGLM4.0-9B large language model as a keyword expander, the associated information of keywords in the agricultural original knowledge fragment information database is expanded through prompt word engineering; The BGE-Large-zh-V1.5 word embedding model is used to vectorize keywords; A high similarity threshold of 0.95 was set to identify keywords with high similarity and group them into one category; The Gensim library that integrates the TF-IDF statistical model is used to measure the importance of keywords, filter out words that appear too infrequently or are too general, and select more closely related keywords.

Images