Large model optimization agricultural knowledge field question and answer application method

Abstract

The invention relates to the technical field of AI text processing, in particular to a large model optimization agricultural knowledge field question and answer application method. According to the agricultural knowledge construction method suitable for the RAG, more professional agricultural knowledge can be constructed only through simple prompt design, and therefore the value of the RAG is truly achieved. Through the agricultural knowledge construction method provided by the invention, the accuracy of agricultural knowledge questions and answers by a large model plug-in knowledge base can be greatly improved, specifically, PDF book knowledge coarse segmentation is performed, agricultural knowledge is finely segmented by prompt agricultural coarse knowledge point input large model semantics, the knowledge construction of agricultural books is completed, and the professionality of agricultural knowledge answers is improved; the agricultural Excel data and the prompt template form the Excel data, the Excel data is endowed with intentions, user problems and the semantic gap of the Excel are solved, and the answer accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of AI text processing technology, and in particular to a large-model optimization agricultural knowledge field question-and-answer application method. Background Art

[0002] In recent years, large language models have officially started to be widely applied in the industry. The excellent general question-answering ability of large language models has led to the research and development personnel of relevant companies in the industry to invest in research. RAG, a key technology in the application of large language models, can enable users to achieve domain knowledge question-answering capabilities without training models. It is low-cost and will not cause catastrophic forgetting of large models. It is loved by most small and medium-sized enterprises. RAG first divides the user's text content into knowledge points, then vectorizes the user's questions, retrieves relevant knowledge points through semantic retrieval models, and then uses prompt engineering to input large models. Because the content input to the large model contains user content, it is equivalent to a student taking an exam with reference content, so that the large language model can answer domain-related questions without training. However, due to various reasons such as the non-standardization of knowledge points, the inaccuracy of semantic retrieval models, the noise in knowledge content, the complexity of knowledge points, and the excessive length of knowledge point content, large language models usually have hallucination problems. In order to solve this problem, this patent proposes a large model to optimize the large model to solve the large model hallucination problem. A complete set of solutions, we have practiced in the development process of Shennong No. 1 agricultural large model and proposed a complete set of technical solutions to solve the large model hallucination. The value of RAG lies in domain knowledge question answering. If domain knowledge is not as valuable as open model knowledge, RAG will lose its meaning. This patent proposes a solution to construct domain knowledge and solve the illusion of large models at the same time. Summary of the invention

[0003] In the process of developing agricultural knowledge questions and answers based on the Shennong No. 1 large model, hallucinations often occur in the process of answering domain knowledge questions based on the open source large language model ChatGLM3-6B and the Langchain framework as the RAG engine technology. For example, when describing the characteristics of a certain agricultural pest, the body length and other values ​​are described as other pests. Through analysis, this patent proposes a solution to solve the hallucination of large models by optimizing large models. More professional agricultural knowledge is more valuable in RAG. This patent proposes a method for constructing agricultural knowledge suitable for RAG. It only requires a simple prompt design to construct more professional agricultural knowledge, thereby truly realizing the value of RAG.

[0004] The present invention solves the deficiencies of the prior art and provides a large-scale model optimization agricultural knowledge field question-and-answer application method, comprising the following steps: S1. Collect data sources Collect data through open source datasets; S2. Agricultural PDF collection and processing: First, extract text from PDF files using the PyPDF2 toolkit in Python, and then segment the text; Step 1: Determine whether a line of text ends with a period or an exclamation mark or not with a punctuation mark; Step 2: Determine whether the length of the line of text is consistent with the length of the previous lines of text; Step 3: If the above two requirements are met, it means that the text content is a preliminary complete semantic content block, which is segmented into corpus blocks; Repeat the above steps until the entire document is segmented; Secondly, we use the open source Qwen1.5-14B-Chat model to finely segment the corpus. Knowledge point in 30 Up to 50 words , through the open source big model to finely divide the knowledge points; S3. Agricultural Excel table data: Knowledge points are replaced with table data through prompts; knowledge points are composed of letter segments, AAA represents [crops], BBB represents [crop varieties], CCC represents [soil conditions], DDD represents [planting techniques], EEE represents [diseases and pests], and FFF represents [storage methods]. Agricultural knowledge points are constructed through prompt technology; prompts are constructed by combining different columns in Excel to form different knowledge points composed of letter segments, and then an agricultural dictionary is formed; S4. Open source dataset processing: The agricultural dictionary performs a rough filtering on the open source data set and selects data related to the agricultural dictionary.

[0005] In the S1, data is collected through open source data sets and processed, including data cleaning, feature scaling, feature encoding, feature selection, data conversion, data normalization, outlier processing, data balancing, dimensionality reduction, and time series processing steps.

[0006] The following steps are also included: S5. Agricultural question answering dataset processing: Accumulate agricultural question-answering data sets through agricultural question-answering robots, supplement data source collection, and expand the data source library; S6. Knowledge point total processing: The first step is to vectorize the agricultural question-answering dataset. The second step is to retrieve relevant knowledge points through semantic vector similarity. The third step is to filter knowledge points based on the similarity score threshold.

[0007] Beneficial effects of the invention: This patent proposes a technical solution for optimizing a large model to solve the problem of large models in agricultural knowledge questions and answers. The core of this solution is not the application of a prompt technical point, but a methodology. Through the agricultural knowledge construction method proposed in this patent, the accuracy of large model plug-in knowledge base for agricultural knowledge questions and answers can be greatly improved. Specifically, PDF book knowledge is roughly segmented, prompt agricultural rough knowledge points are input into large model semantics to finely segment agricultural knowledge, and the knowledge construction of agricultural books is completed, which is conducive to improving the professionalism of agricultural knowledge answers; agricultural Excel data and prompt templates form Excel data, which gives intent to Excel data, solves the semantic gap between user problems and Excel, and improves the accuracy of answers; open source data set dictionary filtering and processing, large model industry classification filtering collects a wider range of agricultural data sets, and improves the coverage of agricultural knowledge questions and answers; knowledge point total processing, all knowledge points are compressed through large model semantic understanding, and short knowledge points are expanded through large model semantic understanding. This avoids the accuracy problem of too many short texts searching for long texts and long texts searching for short texts in semantic retrieval models. Through the combination of the above-mentioned operations, the accuracy of the large model in agricultural knowledge question and answer can be greatly improved, and the methodology proposed in this patent has a wide range of reference significance for knowledge question and answer in any industry and any field. Because most of the technical solutions proposed in this patent solve the problem of large model accuracy through large model technology, and are put into practice in the process of agricultural large model development, this patent is named: A large model optimization large model to solve the application method of large model in the field of agricultural knowledge question and answer. Specific implementation plan

[0008] S1. Data source collection: Agriculture is the basic industry for national economic construction and development, the most critical factor for social stability, and is related to the national economy and people's livelihood. In the historical development of our country, agriculture has played a vital role. Therefore, generations have accumulated a wide range of agricultural books and agricultural knowledge data. These agricultural data play a very important guiding role in agricultural development. The data sources of this patent mainly come from agricultural PDF books, some Excel spreadsheet agricultural data in the process of agricultural development, and agricultural robot question-and-answer data sets, open source large model data sets, and open source agricultural dictionaries collected at the same time. The above content is the knowledge source of this patent. The standards for collecting data should include but are not limited to the following points: 1. Uniqueness of data. The collected data should be unique in the agricultural field to avoid overlapping data with general large models; 2. Accuracy and consistency. Agriculture is a production-oriented enterprise. The data used for large model training must ensure accuracy and consistency to avoid the model giving wrong content to mislead users. 3. The data is representative. Collect representative samples covering data from different regions, seasons, crop types, etc. This helps ensure that the model can generalize to different situations and conditions. 4. Data diversity. Collect diverse data covering different types of agricultural activities, crops, livestock and poultry. This helps improve the generalization ability of the model and allows the model to learn patterns and laws in various agricultural scenarios. The collected data is preprocessed as basic data, including the following steps: 1. Data cleaning: Identify and handle errors, missing values, outliers and duplicates in the data to ensure data quality. 2. Feature scaling: Scale the values ​​of different features to a similar range to prevent some features from having too much influence on the model. Common scaling methods include min-max scaling and standardization. 3. Feature encoding: Convert categorical variables to numerical variables so that the model can handle them. Common encoding methods include one-hot encoding, label encoding, etc. 4. Feature selection: Select the most representative and relevant features from the original features to reduce the dimension of the feature space and improve the efficiency and generalization ability of the model. 5. Data transformation: Transform the data to meet the assumptions or requirements of the model. For example, logarithmic transformation, exponential transformation, etc. 6. Data normalization: Scale the data to a fixed range, usually between 0 and 1 or -1 to 1, to eliminate the dimensional effects of different features. 7. Outlier processing: Identify and process outliers in the data, which can be processed by deletion, replacement or transformation. 8. Data balancing: Process unbalanced data sets to ensure that the model can fully consider the differences between categories during training and evaluation. 9. Dimensionality reduction: Reduce the number of features by retaining the most important features or projecting them into a low-dimensional space to improve the efficiency and generalization ability of the model. Common dimensionality reduction methods include principal component analysis (PCA) and linear discriminant analysis (LDA).10. Time series processing: Smoothing, differencing, sliding window processing, etc. are performed on time series data to identify trends and periodicity, or reduce the impact of noise. The data processed in this step will be used as input data for subsequent steps.

[0009] S2. Collection and processing of agricultural PDF books: With the research of generations of agricultural experts, a large number of agricultural-related books have been accumulated. These agricultural books play an important role in my country's planting. However, if these PDF texts are directly deployed based on the open source LangChain framework, the large model usually has poor results due to the non-standard knowledge point cutting and the semantic gap between knowledge point content and user questions. For these PDF format documents, first extract them from PDF files as text through the PyPDF2 toolkit in python, and then perform preliminary rough knowledge segmentation through the following steps. Step 1: Determine whether the end of a line of text is "." or "!" or the end is not punctuation. Step 2: Determine whether the length of the line of text is longer than the previous lines of text. Step 3: If the above two steps are met, it means that the text content is a preliminary complete semantic content block and can be segmented into corpus blocks. Repeat the above steps until the entire document is segmented. This segmentation scheme is a preliminary coarse-grained segmentation based on semantics based on the characteristics of Chinese. Compared with the method of segmenting text according to fixed length in the Langchain framework, this scheme can effectively avoid the truncation of complete sentences and ensure the validity of knowledge.

[0010] Secondly, we use open source-based large models such as Qwen1.5-14B-Chat to perform fine segmentation on each knowledge segment that has been initially segmented. The segmentation method is as follows to form knowledge points, that is, add prompt instructions as follows: { \nThe above content is divided into several knowledge points. Each knowledge point is 30 to 50 words.}, by repeatedly calling the open source large model, the knowledge points of the PDF book are formed. When the large model is reasoning, it is less likely to produce hallucinations when using short text as background knowledge than long text. This phenomenon can be understood from several perspectives: 1. Information volume and complexity: Short texts usually contain less information, which allows the model to focus more when processing, thereby reducing the errors that may occur when processing complex information. Due to the large amount of information in long texts, the model may be confused due to too many details when processing. 2. Context matching: Short texts tend to target specific issues or topics, so they may have a higher degree of context matching, which helps the model understand and apply this information more accurately. On the contrary, long texts may contain information that is irrelevant to the current reasoning task, increasing the risk of the model making misleading inferences. 3. Attention mechanism: Modern large language models often use attention mechanisms to focus on important parts of the text. Short texts, due to their conciseness, may allow the model to allocate its attention resources more effectively, thereby reducing attention to irrelevant information. 4. Memory and generalization: Long texts may contain more memory traces, which may cause the model to overfit to specific details in the training data. Short texts may prompt the model to focus more on generalization rather than memorizing specific pieces of information. 5. Error accumulation: When processing long texts, the model may make mistakes in the early parts of the text, and these errors may be amplified later in the text processing. Short texts have shorter information flows, which reduces the chance of error accumulation. Because the content of each knowledge point is not long, large models are less likely to hallucinate, and the reasoning effect is greatly improved. This is one of the technical solutions to be protected by this patent.

[0011] Specific application: Most vegetable products have a water content of more than 90%. It can be seen that water is one of the important conditions for the growth and development of vegetables. Due to the different origins and the result of system development, different types of vegetables have different requirements for water. They can be roughly divided into the following five categories: Category 1, which consumes a lot of water and requires a higher soil temperature. Most of these vegetables originate from rainy and humid areas, including cabbage and fungus originating from the Mediterranean coast, cucumbers and various green leafy vegetables originating from the Southeast Asia, and cabbage originating from southern China. These vegetables have relatively small root groups, densely distributed in the shallow soil layer, and the stem area of ​​the leaves is large. They need to be irrigated frequently during cultivation, and the amount of water should be large. If the water supply is insufficient, the yield will be low and the quality will deteriorate. Because the soil in the origin of these vegetables is sufficient in water and there is more rainfall during the growing season, they have formed the characteristics of high water demand, and they also pay attention to the results of irrigation during the long-term cultivation process. The second category does not consume much water, but requires high soil humidity and can tolerate low air temperature. These vegetables originated from the high mountains of Central Asia, such as onions, scallions, garlic, etc. Their origin is snowy in winter, and the soil is sufficient after the snow melts in spring, but the air is dry. Their morphological characteristics are tubular leaves with waxy surfaces, small transpiration, weak root system, and weak water absorption. Therefore, the soil should be kept moist in cultivation, and the amount of irrigation should not be too large, but it should be irrigated frequently. The third category consumes more water and requires moderate soil humidity. It is mainly pumpkins, beans, tomatoes, peppers, potatoes, etc. that originated from tropical South America. Their main feature is that their root system is strong and the leaf area is not small. Although they consume more water, they have strong absorption capacity. Appropriate irrigation in cultivation can meet their water needs. The fourth category is vegetables that adapt to lower soil humidity and release less water. These vegetables include watermelons, which originated in Central Africa; carrots, which originated in the Near East. Because of their strong root system and broken leaves, they have a strong ability to absorb water and consume very little water, so they can tolerate relatively dry soils and are called early-resistant vegetables. In particular, watermelons store a lot of water in their fruits, which can regulate the plant's need for water when encountering severe droughts. The fifth category, vegetables that grow in water, originated in the marshland along the South China Sea, are called aquatic vegetables. The root system of aquatic vegetables is not well developed, and the transpiration area of ​​their leaves is large. They must be cultivated in ponds or paddy fields, which are suitable for rainy and humid climates. The above are the requirements of vegetables for soil moisture, in addition to the requirements for relative air humidity. Vegetables from different origins have different requirements for relative air humidity. Vegetables originating from dry areas require lower air temperatures, while vegetables originating from warm areas require higher air temperatures. The appropriate relative air humidity for various vegetables can be divided into 4 categories. \nThe above content is divided into several knowledge points. Each knowledge point is between 30 and 50 words.

[0012] S4. Agricultural Excel table data: Agriculture is the foundation of the country, so a large amount of valuable and meaningful agricultural Excel data has been accumulated in the product project. However, if the Excel table data is directly plugged into the knowledge base, the accuracy is not high. Because there is a semantic gap between the user's questions and the Excel header, errors will occur. This patent proposes a technical solution that converts table data into knowledge points through prompts, giving intent to Excel table data, thereby improving the accuracy of knowledge questions and answers: The specific implementation measures are as follows: AAA represents [crops], BBB represents [crop varieties], CCC represents [soil conditions], DDD represents [planting technology], EEE represents [pests and diseases], and FFF represents [storage method]. Agricultural knowledge points are constructed through prompt technology. For example, the construction of knowledge points on plant pests and diseases is taken as an example: AAA related pest and disease knowledge includes: BBB Replace AAA and BBB with the corresponding columns in the Excel table to form knowledge points related to crop diseases and pests.

[0013] By combining different columns in Excel to construct prompts, various knowledge points are formed, and finally the abstract Excel data in agriculture is converted into easy-to-understand text content. In this solution, Prompt converts Excel into valuable and meaningful knowledge, reducing the semantic gap between user questions and external knowledge, thereby improving the accuracy of the external knowledge base, which is one of the technical solutions to be protected by this patent. For example, a user consultation question: What are the diseases and pests of soybeans? If there is no prompt template designed to form a knowledge point, RAG usually inputs the relevant knowledge of what diseases and pests potatoes have as background knowledge into the large language model during the retrieval stage, which usually causes hallucinations. By designing multiple prompts to generate a plant knowledge point, the semantic retrieval module can easily retrieve knowledge points related to soybeans, thereby improving the accuracy of the large model.

[0014] S5. Processing of open source data sets: Although the above scheme has formed more valuable agricultural data, it cannot achieve a high coverage. User questions are not in the knowledge point content, and external knowledge will interfere. However, the open source data set is a data set accumulated over the years in the field of artificial intelligence, and therefore has very important value. In order to make the large model have good generalization, this patent, based on the previous scheme, filters the open source data set. First, it performs a rough filtering on the open source data set based on the agricultural dictionary to form preliminary data with agricultural relevance. If these agricultural data are used as external plug-ins, the storage through the vectorized model will greatly increase the storage requirements. In order to solve this problem, this patent proposes a solution of filtering industry data with a large model, that is, through the prompt technology, the agricultural data is further finely screened: Specific application: the water content of most vegetable products is above 90%, which shows that water is one of the important conditions for the growth and development of vegetables. Due to the different origin regions and the result of system development, different types of vegetables have different requirements for water. They can be roughly divided into the following five categories: the first category, which consumes a lot of water and requires a higher soil temperature. Most of these vegetables originate from rainy and warm air areas, including cabbage and lettuce originating from the Mediterranean coast, cucumbers and various green leafy vegetables originating from the Southeast Asia, and cabbage originating from southern China. These vegetables have relatively small root groups, densely packed in the shallow soil layer, and a large transpiration area of ​​the leaves. They need to be irrigated frequently during cultivation, and irrigation is very important. If the water supply is insufficient, the yield will be low and the quality will change. Because the soil of the origin of these vegetables is sufficient in water, the more rainfall in the growing season has formed the characteristics of high water demand, and the result of paying attention to irrigation during the long-term cultivation process, the second category, which consumes the least water itself, but requires a higher soil temperature and can tolerate lower air temperatures. These vegetables originated from the high mountains of Central Asia, such as onions, green onions, garlic, etc. Its origin is snowy in winter, and the soil is moist after the snow melts in spring, but the air is dry. Its morphological characteristics are tubular leaves, waxy surface, small transpiration, weak root system, and weak water absorption. Therefore, the soil should be kept warm and the amount of irrigation should not be too large, but it should be irrigated frequently. The third category consumes more water and requires moderate soil moisture. It mainly originates from the tropical South American pumpkins, beans, tomatoes, peppers, potatoes, etc. Its characteristics are strong root system and large leaf area. Although the potential water consumption is large, the absorption capacity is strong. Appropriate irrigation in cultivation can meet its water needs. The fourth category is vegetables that adapt to lower soil temperatures and consume less water. These vegetables include watermelons that originated in Central Africa and carrots that originated in the Near East. Because of their strong root system and broken leaves, they have a strong ability to absorb water and consume very little water, so they can tolerate relatively dry soils and are called drought-tolerant vegetables. In particular, watermelons store a lot of water in their fruits, which can adjust the plant's need for water when encountering severe droughts. The fifth category, vegetables that grow in water, originated from the swamps of the South Sea, are called aquatic vegetables. The root system of aquatic vegetables is not well developed, and the base area of ​​the leaves is large. They must be cultivated in ponds or paddy fields, which are suitable for rainy and humid climates. The above are the requirements of vegetables for soil moisture, in addition to the requirements for relative air humidity. Vegetables from different origins have different requirements for relative air humidity. Vegetables originating from dry areas require lower air humidity, while vegetables originating from humid areas require higher air temperature. The relative air humidity suitable for various vegetables can be divided into 4 categories. \nWhich industry does the above content belong to?

[0015] If the generated content contains the keyword "agriculture", the text is considered to belong to the agricultural category knowledge point, and the text content is stored as a knowledge point, and all other data are deleted. This operation can not only expand the agricultural knowledge base with the help of open source datasets, but also avoid the problem of loading too much non-agricultural data and taking up more storage space. For example, if a text content is agricultural, the corpus will be used as a knowledge base for the semantic retrieval module of RAG. Because of the extensive collection of open source general agricultural corpus, RAG's general agricultural question-answering capabilities have been greatly improved.

[0016] S6. Agricultural question-answering dataset processing: Agricultural question-answering robots have accumulated a lot of agricultural question-answering datasets. Usually, these question-answering datasets are high-quality datasets accumulated by the company over a long period of time. In order to ensure that these high-quality datasets are not overwhelmed by the general knowledge of plug-ins, this patent uses the following method to enhance the large model instructions: concentrate the questions in the agricultural question-answering dataset, expand the knowledge through the method of expanding similar questions through the large model, for example: Question: What are the diseases and pests of corn? Answer: Regarding corn pests and diseases, the following are some common types: Leaf spot: This is a disease caused by a fungus that causes yellow or brown spots to appear on corn leaves.

[0017] Stripe disease: This is a disease caused by a virus, and the symptoms are green streaks and yellowing on corn leaves.

[0018] Downy mildew: This is a disease caused by a fungus, and the symptom is a white frost that appears on the corn leaves.

[0019] Wheat rust: This is a disease caused by a fungus, and the symptoms are brown or black spots on corn leaves.

[0020] Smut: This is a disease caused by a fungus, the symptoms of which are the formation of black lesions on the cobs and kernels of corn.

[0021] Then expand the question and prompt command to similar questions, that is, "What diseases and pests does corn have\nWhat words express the same meaning as the above sentence", and then input the above content into the big model, the big model will give the following results: Here are some expressions that are similar in meaning to "What are the pests and diseases of corn?" 1. “What diseases and pests are common in corn?” 2. “What diseases and pests is corn susceptible to?” 3. “What are the common diseases and pests of corn?” 4. “What are some common diseases and pests of corn?” 5. “What diseases and pests may plague corn?” 6. “What diseases and pests are common to corn?” Note: The results given by different large models may be different, but it does not affect the feasibility of this solution.

[0022] The answer obtained in the first step (about corn pests and diseases, the following are some common types: XX) is combined with the 6 ways of expressing the results of this large model generation, that is, 6 question-answer pairs about the knowledge point of "common corn pests and diseases" are constructed, realizing the expansion of this knowledge point. By doing the same processing on all the collected data, each question-answer pair in the question-answer pair data set is expanded into multiple knowledge points, so that when the user enters a question, it is easier to match the corresponding question and answer, and thus more valuable.

[0023] S7. Total processing of knowledge points: The knowledge points formed by the above operations are used as the plug-in knowledge base, and then the knowledge points are vectorized based on the open source vectorization model. When the user asks a question, the first step is to vectorize the user's question, the second step is to retrieve related knowledge points through semantic vector similarity, and the third step is to filter the matched knowledge points according to the score threshold, and select the first 4 as background knowledge to input into the large model. Usually, the large model can complete the user's answer task by combining the retrieved background knowledge, but the number of words in the user's question is usually inconsistent, such as "How to grow cold-region rice?", "My home is in XX City, Heilongjiang Province. I want to grow cold-region rice. What should I pay attention to in order to get a relatively high yield? Where is it more suitable to sell it after I finish growing it?...". Usually, we call user questions with less than 15 words short text, and those with more than 15 words long text. Semantic matching usually has technical bottlenecks of short text searching long text and long text searching short text, that is, long text is usually easy to recommend long text, and short text is usually easy to match short text. In order to deal with this situation, for the knowledge points in long texts, by adding prompts as: \nThe above content is re-expressed with fewer words and easier to understand. Then the new content is input into the big model to complete the knowledge compression; for the knowledge points in short texts, by adding prompts as: \nThe above content is re-expressed with more words and easier to understand. Then the content is input into the big model to complete the knowledge expansion. Ultimately, no matter whether the user input is long or short, the text content with corresponding semantics can be searched, thereby improving the accuracy. The reason why the length text matching accuracy is not high is mainly due to the small number of long and short text matching training corpora for building the semantic retrieval model. Because the length of the long and short knowledge in this patent is lengthened or compressed, the pressure on the semantic retrieval model is alleviated to a certain extent.

[0024] Glossary LLM: LLM usually refers to "Large Language Model". These models are an important breakthrough in the field of artificial intelligence, capable of understanding and generating natural language text. With the advancement of technology, LLM has grown to a huge scale and is capable of handling complex language tasks such as text translation, question answering, writing, and conversation.

[0025] Grand Model Illusion: Models may generate responses that appear plausible but are inaccurate, especially when dealing with questions that require deep knowledge or commonsense understanding.

[0026] Intelligent question and answer: refers to the function of automatically generating personalized answers to natural language questions raised by users through AI technology in a question-and-answer format.

[0027] RAG: RAG stands for Retrieval-augmented Generation, which is a text generation method that combines retrieval and generation. The core idea of ​​this method is to combine the pre-trained retrieval semantic model with the generation model so that the retrieved relevant information can be used when generating text, thereby realizing domain knowledge question answering.

Claims

1. A large-scale model optimization agricultural knowledge field question-and-answer application method, characterized in that The steps include: S1. Collect data sources Collect data through open source datasets; S2. Agricultural PDF collection and processing: First, extract text from PDF files using the PyPDF2 toolkit in Python, and then segment the text; Step 1: Determine whether a line of text ends with a "period" or "exclamation mark" or does not end with a punctuation mark; Step 2: Determine whether the length of this line of text is consistent with the length of the previous lines of text; Step 3: If the above two requirements are met, it means that the text content is a preliminary complete semantic content block, which is segmented into corpus blocks; Repeat the above steps until the entire document is segmented; Secondly, we use the open source Qwen1.5-14B-Chat model to finely segment the corpus. Knowledge points are between 30 and 50 Character , through the open source big model to finely divide the knowledge points; S3. Agricultural Excel table data: Knowledge points are replaced with table data through prompts; knowledge points are composed of letter segments, AAA represents [crops], BBB represents [crop varieties], CCC represents [soil conditions], DDD represents [planting techniques], EEE represents [diseases and pests], and FFF represents [storage methods]. Agricultural knowledge points are constructed through prompt technology; prompts are constructed by combining different columns in Excel to form different knowledge points composed of letter segments, and then an agricultural dictionary is formed; S4. Open source dataset processing: The agricultural dictionary performs a rough filtering on the open source data set and selects data related to the agricultural dictionary.

2. According to claim 1, a large model optimization agricultural knowledge field question and answer application method is characterized in that In the S1, data is collected through open source data sets and processed, including data cleaning, feature scaling, feature encoding, feature selection, data conversion, data normalization, outlier processing, data balancing, dimensionality reduction, and time series processing steps.

3. According to claim 1, a large model optimization agricultural knowledge field question and answer application method is characterized in that The following steps are also included: S5. Agricultural question answering dataset processing: Accumulate agricultural question-answering data sets through agricultural question-answering robots, supplement data source collection, and expand the data source library; S6. Knowledge point total processing: Step 1: vectorize the agricultural question-answering dataset; Step 2: retrieve relevant knowledge points through semantic vector similarity; Step 3: Filter knowledge points based on similarity score threshold.