Language model question and answer quality optimization method oriented to vertical field

Abstract

The invention provides a vertical domain-oriented language model question and answer quality optimization method, which comprises the following steps of: acquiring a complex question proposed by a user, and dividing a long document in a corresponding vertical domain knowledge base into standardized text units based on the complex question; related text units related to the complex problem in the standardized text units are inquired and extracted through a mixed retriever, the mixed retriever comprises sparse retrieval and dense retrieval, and the related text units comprise sparse retrieval results and dense retrieval results with preset weight coefficients; reordering and extracting the related text units through a preset selection rule to obtain related documents; according to the related document, first output is obtained through a first language model, the first language model comprises a first prompt, the first prompt comprises an inference and action framework, and the first prompt inferes the answer of the user according to the complex question. Through a knowledge retrieval and model reasoning collaborative enhancement method, the problem of insufficient knowledge coverage of a general large model is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a language model question and answer quality optimization method for a vertical field. BACKGROUND

[0002] With the explosive growth of Internet information and the diversification of user needs, the development of question and answer systems has experienced evolution from simple fact question and answer to complex question and answer involving multi-dimensional and multi-level knowledge relationships. Such complex question and answer has very high requirements for the multi-source knowledge integration, logical reasoning and cross-neighborhood association capabilities of the system.

[0003] In the field of complex question and answer, two classical methods of semantic parsing and information retrieval are usually used. The semantic parsing method converts a natural language question into an executable structured query statement and executes the query in a knowledge graph to obtain an answer. The information retrieval method first identifies the subject entity in the question, and then extracts the relevant subgraph in the knowledge graph according to the semantic relationship implied in the question, and then infers the target answer entity accordingly.

[0004] However, both methods generally have the problem of knowledge coverage when dealing with knowledge-intensive vertical neighborhood complex question and answer tasks. SUMMARY

[0005] (I) Technical problem to be solved In view of the deficiencies of the prior art, the present application provides a language model question and answer quality optimization method for a vertical field, which solves the problem of insufficient knowledge coverage of traditional complex question and answer models.

[0006] (II) Technical scheme To achieve the above object, the present application is realized by the following technical scheme: In a first aspect, the present application provides a language model question and answer quality optimization method for a vertical field, comprising: obtaining a complex question proposed by a user, and dividing long documents in a corresponding vertical field knowledge base into standardized text units based on the complex question; querying and extracting relevant text units related to the complex question in the standardized text units through a hybrid retriever; wherein the hybrid retriever includes sparse retrieval and dense retrieval, and the relevant text units include sparse retrieval results and dense retrieval results with a preset weight coefficient; reordering and extracting the relevant text units through a preset selection rule to obtain relevant documents; According to the related documents, a first output is obtained through a first language model; wherein the first language model comprises a first prompt, the first prompt comprises an inference and action framework, and the first prompt infers the answer of the user according to the complex question.

[0007] Preferably, the sparse retrieval comprises keyword matching of the standardized text units based on the complex question and scoring according to document relevance, and extracting high-relevance text units as sparse retrieval results.

[0008] Preferably, the dense retrieval comprises vectorizing the standardized text units as paragraph vectors using semantic vector similarity, encoding the complex question as a query vector, calculating the similarity between the paragraph vectors and the query vector, and returning high-similarity text units as dense retrieval results.

[0009] Preferably, the method further comprises: According to the related documents, a second output is obtained through a second language model; wherein the second language model is obtained by supervised fine-tuning of a pre-constructed training set.

[0010] Preferably, the construction method of the training set comprises: A certain size of data set is divided as initial training data based on conventional question and answer data, the initial training data comprising complex questions, initial thinking processes and background knowledge; Retrieval results of the initial training data are obtained through the hybrid retriever and re-ranked after weighted summation to obtain an initial relevant document list; The initial relevant document list is expert proofread to obtain an improved relevant document list; According to the improved relevant document list, a thinking process is generated through a large language model with a second prompt; the complex question, the background knowledge and the thinking process serve as the training set.

[0011] Preferably, the second prompt comprises background knowledge, user questions and thinking processes, and the second prompt generates a thinking process according to the background knowledge and user questions.

[0012] Preferably, the second language model is supervised fine-tuned through LoRA technology.

[0013] In a second aspect, the present application also provides a language model question and answer quality optimization system for vertical fields, comprising: An acquisition module acquires a complex question raised by a user, and divides long documents in a corresponding vertical field knowledge base into standardized text units based on the complex question; The retrieval module queries and extracts relevant text units related to the complex problem in the standardized text units through a hybrid retriever; wherein the hybrid retriever includes sparse retrieval and dense retrieval, and the relevant text units include sparse retrieval results and dense retrieval results of a preset weight coefficient; The extraction module reorders and extracts the relevant text units through a preset selection rule to obtain relevant documents; The processing module obtains a first output through a first language model according to the relevant documents; wherein the first language model includes a first prompt, the first prompt includes an inference and action framework, and the first prompt infers the answer of the user according to the complex problem.

[0014] In a third aspect, the present application further provides a computer readable storage medium storing a computer program for optimizing the quality of question and answer of a language model facing a vertical field, wherein the computer program causes a computer to execute the method for optimizing the quality of question and answer of a language model facing a vertical field described above.

[0015] In a fourth aspect, the present application further provides an electronic device comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include a program for executing the method for optimizing the quality of question and answer of a language model facing a vertical field described above.

[0016] (Three) beneficial effects The present application provides a method for optimizing the quality of question and answer of a language model facing a vertical field. Compared with the prior art, the following beneficial effects are achieved: The present application provides a complex question and answer method based on knowledge retrieval and model inference collaborative enhancement, which adopts a method of hybrid retrieval and ReAct prompt technology in the first prompt to work collaboratively, and can recall relevant knowledge of a vertical field for a general large model, effectively solving the problem of insufficient knowledge coverage of the general large model. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0018] Figure 1 The present application provides a method for optimizing the quality of question and answer of a language model facing a vertical field. Compared with the prior art, the following beneficial effects are achieved: Figure 2 A framework diagram of the language model question and answer quality optimization method for the vertical field provided for an embodiment of the present application is shown in the figure. Figure 3 A framework diagram of the language model question and answer quality optimization method for the vertical field provided for another embodiment of the present application is shown in the figure. Figure 4 A framework diagram of the language model question and answer quality optimization method for the vertical field provided for yet another embodiment of the present application is shown in the figure. Figure 5 A framework diagram of the language model question and answer quality optimization method for the vertical field provided for still another embodiment of the present application is shown in the figure. Figure 6 A first prompt diagram provided for an embodiment of the present application is shown in the figure. Figure 7 A second language model training diagram provided for an embodiment of the present application is shown in the figure. Figure 8 A training set training diagram provided for an embodiment of the present application is shown in the figure. Figure 9 A second prompt diagram provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0020] The embodiments of the present application provide a language model question and answer quality optimization method for a vertical field, solve the problem of weak complex semantic understanding ability of a traditional model, and achieve accurate reasoning of a large-scale language model and a small-scale language model when facing complex problems in a vertical field.

[0021] The technical solutions in the embodiments of the present application are as follows to solve the above technical problems: Breakthrough progress in natural language processing technology, large language models (LLMs) represented by the GPT series are trained on large-scale general corpus, have excellent natural language understanding ability, and show excellent generation ability by using their own knowledge, especially show the potential of deep semantic analysis and complex reasoning in open domain scenarios, provide a new direction for the research of complex question and answer, and complex question and answer technology also usher in further development.

[0022] A vertical field complex question answering technical solution based on a large language model (LLM) has the following core steps: (1) question direct input and processing: directly input the complex question in the vertical field into a general large language model (such as GPT, GLM, etc.); (2) internal knowledge reasoning of the model: relying on the knowledge obtained by pre-training the LLM in a large-scale general corpus (such as Wikipedia, web text), reasoning through the self-recurrent generation mechanism, without external knowledge retrieval or domain adaptation operation; (3) answer direct generation: the model generates answers based on the internally stored knowledge. The technical features are: (1) relying on the "pre-trained knowledge + generation ability" of the LLM to directly process the question, without an additional knowledge retrieval module; (2) the performance is strongly related to the number of model parameters; (3) the unified reasoning process does not distinguish between types.

[0023] Although LLMs exhibit significant performance advantages in general tasks, they generally face the challenge of insufficient knowledge coverage when dealing with knowledge-intensive vertical field complex question answering tasks. Small-scale models also have limited reasoning capabilities. Insufficient knowledge coverage: Since LLMs are mainly trained on open-domain general corpora, the coverage of fine-grained knowledge in vertical fields is low, leading to frequent "hallucinations" or factual errors when dealing with professional problems; Small Language Models (SLMs) have limited reasoning capabilities: existing SLMs often lack systematic logical reasoning paths, especially when dealing with complex structured problems that require multi-hop reasoning, set intersection, and temporal comparison, which can easily lead to broken reasoning chains or semantic misinterpretations.

[0024] Experiments show that, under the same retrieval method and prompting method, smaller parameter models perform worse than large-scale language models. Even if the retriever recalls relevant background knowledge, the smaller parameter model still cannot reason correctly. The main reason is that the model's own reasoning ability is weak, and the prompting method has limited effect on enhancing the reasoning effect of small parameter models. On this basis, the present application explores and experiments to improve the reasoning ability of small parameter models, and proposes a complex question answering method based on background knowledge and thinking process fine-tuning. By supervising and fine-tuning small parameter models (ChatGLM2-6B, Llama2-7B, and Baichuan2-7B), the complex question answering performance of small models is further improved.

[0025] In order to better understand the above technical solution, the above technical solution will be described in detail in conjunction with the drawings in the specification and the specific embodiments.

[0026] Embodiment 1: As shown in Figure 1 In one embodiment, a vertical field language model question answering quality optimization method includes the following steps: Step S1, obtaining a complex question raised by a user, and dividing a long document in a corresponding vertical field knowledge base into a standardized text unit based on the user question.

[0027] Specifically, the embodiment as a whole is implemented based on the LangChain framework, aiming to build an efficient and scalable complex question answering system. First, a complex question raised by a user is obtained, and a vertical field knowledge base corresponding to the user question is determined. The long document in the vertical field knowledge base is preprocessed and block stored by using a text splitting module (Text Splitter) provided by the framework to generate a standardized text unit suitable for retrieval, laying a foundation for subsequent information retrieval. It should be noted that the present application is a question answering system for vertical fields, and therefore the questions asked by the user have a very high degree of professionalism, which cannot be compared with general simple questions. For these complex questions with a very high degree of professionalism, the language model is optimized and adjusted.

[0028] The vertical field-oriented language model question answering quality optimization method of the embodiment of the present application provides a complex question answering system applicable to medical diagnosis, industrial fault diagnosis, financial risk control and other knowledge-intensive vertical fields based on the LangChain framework. By replacing the field corpus, adjusting the reasoning link and supplementing the external tool interface, it can be reused in various knowledge-intensive scenarios, and has very high scalability.

[0029] Step S2, querying and extracting relevant text units related to the complex question from the standardized text units by using a hybrid retriever; wherein the hybrid retriever includes sparse retrieval and dense retrieval, and the relevant text units include sparse retrieval results and dense retrieval results with a preset weight coefficient. The sparse retrieval includes keyword matching of the complex question based on the standardized text units and scoring according to document relevance, and extracting high-relevance text units as sparse retrieval results. The dense retrieval includes vectorizing the standardized text units as paragraph vectors using semantic vector similarity, and encoding the user's complex question as a query vector, calculating the similarity of the paragraph vector and the query vector and returning high-similarity text units as dense retrieval results.

[0030] Specifically, referring to Figure 5 To realize the collaborative fusion of the sparse retriever and the dense retriever, the embodiment of the present application constructs a hybrid retriever based on the EnsembleRetriever module provided by the LangChain framework. The module supports the combination of multiple retrievers, and by setting the weighting coefficients (such as sparse retrieval 0.3 and dense retrieval 0.7), the retrieval results output by different retrievers are weighted and then normalized and fused.

[0031] Referring to Figure 3 , Figure 3Sparse retrieval method based on text keywords is represented. To support large-scale and efficient retrieval, the embodiments of the present application use the Elasticsearch search engine, create an index, and write all the segmented and normalized text units as document items one by one into the index, thus completing the construction of the inverted index. Based on the ElasticsearchBM25Retriever module provided by the LangChain framework, the module is connected with the Elasticsearch system, encapsulates the retrieval interface call of Elasticsearch, and automatically adopts the BM25 algorithm to match the keywords and score the document relevance of the natural language question input by the user, and finally returns the sorted high-relevance paragraph result TOP-K, which is the sparse retrieval result for the downstream large model reasoning.

[0032] Referring to Figure 4 , Figure 4 The dense retrieval method based on semantics is represented, which has good recall ability in the context with more fuzzy or variants, by means of semantic vector similarity (chroma). The embodiments of the present application use the embedding model bge-m3 to vectorize the segmented and normalized text units, and encode each text unit into a fixed-dimensional semantic vector. Subsequently, these vectors are stored in the high-performance vector database chroma, and the dense semantic index is constructed to support subsequent semantic similarity retrieval. In the retrieval execution phase, the system is connected with the vector database based on the chromaRetriever module provided in the LangChain framework, which can automatically load and query the vectorized knowledge blocks, encode the user's natural language question into a query vector, and calculate the similarity (such as cosine similarity) with the paragraph vectors in chroma. Then, the high-similarity paragraph vectors are extracted and returned as high-similarity text units, which constitute the most relevant semantic paragraph set, i.e., the dense retrieval result.

[0033] The language model question and answer quality optimization method for vertical fields provided by the embodiments of the present application uses sparse retrieval through keyword matching strategy, which has strong accuracy in the scene with high word consistency; the dense retrieval has high scalability and real-time response capability, and provides more semantically relevant context input for large language models, significantly improves the understanding ability and generation quality in complex question and answer scenarios; by constructing a hybrid retriever to fuse and reorder the sparse retrieval result and the dense retrieval result, the relevance and ranking quality of the paragraphs in the retrieval result are significantly improved, the accuracy of context matching in complex question and answer tasks is effectively enhanced, and higher quality retrieval input is provided for the downstream large language model generation module.

[0034] Step S3, reordering and extracting the relevant text units by a preset selection rule to obtain relevant documents.

[0035] Specifically, after the Elasticsearch BM25Retriever module fuses the relevant text units returned by the hybrid retriever, the text units are globally sorted according to the weighted total score to generate a unified and sorted document list. Finally, the most relevant text units (TOP-K paragraphs) are selected according to the score of the document list.

[0036] The reordering mechanism of the embodiment of the present application significantly improves the relevance sorting quality of the paragraphs, effectively enhances the accuracy of context matching in complex question answering tasks, and provides higher quality retrieval input for the downstream large language model generation module.

[0037] Step S4, obtaining a first output by a first language model according to the relevant documents. The first language model includes a first prompt, the first prompt includes an inference and action framework, and the first prompt infers the user's answer according to the user's question.

[0038] For example, the first language model of the embodiment is a large-scale language model, such as Wenxin 3.5, GLM-4, etc. Specifically, for complex problems, after obtaining the retrieval results, the retrieval results need to be input into the language model for inference to obtain the answer. The embodiment proposes a system prompt inference method, and uses the ReAct prompt method. By combining inference and action, the interaction of large language models and external retrievers is improved, and the inference and decision-making ability of the model in complex tasks is improved. See Figure 6 , Figure 6 The ReAct prompt structure of the embodiment of the present application.

[0039] In the embodiment, the relevant documents are input into the large model, and the ReAct prompt structure of the large model is used for inference to obtain the output result. ReAct not only focuses on the inference process inside the model, but also combines the model's inference with the actual action steps, allowing the model to actively obtain external information and interact with other tools, thereby providing more accurate and effective answers in actual tasks. In this way, the model can not only generate answers through inference, but also further optimize decisions through action to ensure the accuracy and relevance of the answers.

[0040] Step S41, obtaining a second output by a second language model according to the relevant document list; wherein the second language model is obtained by supervising and fine-tuning the pre-constructed training set.

[0041] Exemplary second language models are smaller parameter scale models, such as ChatGLM2-6B, Llama2-7B, and Baichuan2-7B. See Figure 7 , Figure 7 The model fine-tuning framework that fuses background knowledge and thinking process is a second language model fine-tuned under supervision, FLM (Finetuned Language Model). LoRA (Low-Rank Adaptation) is a technique for efficiently fine-tuning large pre-trained models. The second language model is fine-tuned under supervision using the LoRA technique.

[0042] The implementation of step S41 also includes the construction of the training set, and the specific method of construction is as follows: S411, divide a certain scale of data set as initial training data based on conventional question and answer data, and the initial training data includes complex questions and background knowledge.

[0043] S412, obtain the retrieval results of the initial training data by the hybrid retriever and reorder after weighted summation to obtain an initial relevant document list.

[0044] S413, expert proofing of the initial relevant document list to obtain an improved relevant document list.

[0045] S414, generate a thinking process by a large language model with a second prompt according to the improved relevant document list; the complex question, the background knowledge, and the thinking process are the training set.

[0046] See Figure 7 , Figure 8 , Figure 7 The construction framework of the training set is shown in FIG. 4. Figure 8 The second prompt structure for generating a thinking process is shown in FIG. 5. The second prompt includes background knowledge, a user question, and a thinking process, and the second prompt generates a thinking process according to the background knowledge and the user question. Specifically, a certain scale of data set is divided as initial training data based on conventional question and answer data, and the initial training data includes complex questions and background knowledge. Then, the hybrid retriever is used to recall the TOP-K passages most relevant to the question, and after expert proofing, an improved relevant document list is obtained for downstream tasks. Next, an advanced large-scale model (such as DeepSeek-V3) is used to generate a thinking process according to the background knowledge based on the second prompt structure by the few-shot prompting method. The complex question, the background knowledge, and the thinking process will be used as the training set of the model.

[0047] In this embodiment, by performing supervised fine-tuning on models with small parameter sizes, the complex question-answering performance of small models is further improved, which solves the problem that small models have weak reasoning ability and that prompting methods have limited effect on enhancing the reasoning effect of small parameter models.

[0048] Example 2: This technical solution possesses good versatility and can be extended to multiple knowledge-intensive vertical fields such as medical diagnosis, industrial fault diagnosis, and financial risk control. By replacing the domain corpus, adjusting the inference chain, and supplementing external tool interfaces, it can be reused in various knowledge-intensive scenarios. Taking the medical diagnosis scenario as an example, its key migration and evolution process is as follows: At the knowledge resource level, the original knowledge bases, which were geared towards general or specific vertical fields, are replaced with medical knowledge resources such as medical literature, standard treatment guidelines, case databases, and electronic medical records. A hybrid retrieval strategy is used to achieve precise retrieval of medical information: the sparse retrieval part matches keywords and symptoms (such as "chest pain," "fever," and "shortness of breath"), while the dense retrieval part uses a pre-trained embedding model in the medical field to perform semantic vector encoding on the case text, improving the recall ability for ambiguous or polysemous expressions.

[0049] At the reasoning mechanism level, a ReAct-style prompting reasoning framework is introduced to construct a chain-like thinking structure of "symptom identification - examination suggestion - differential diagnosis". The model can simulate the reasoning path of clinicians, such as thinking "the patient has chest pain, and coronary heart disease or pneumonia should be considered as a preliminary possibility", and take action accordingly, calling on auxiliary examination resources such as electrocardiogram and CT images, and integrating symptoms, examination results and past medical history through multiple rounds of reasoning to provide more targeted diagnostic suggestions.

[0050] To enhance model capabilities, a domain-specific large language model (such as MedGPT-6B) can be used as the base model. Symptom descriptions, examination results, and expert diagnostic thought processes from medical cases can be used as training samples for supervised fine-tuning. Combined with efficient parameter fine-tuning techniques such as LoRA, the model's multi-hop reasoning and semantic understanding capabilities for complex and rare cases (such as "fever of unknown origin complicated by multi-organ damage") can be improved. Simultaneously, a medical knowledge graph can be introduced to supplement knowledge related to rare diseases or low-frequency entities, effectively alleviating the "hallucination" problem that occurs with general-purpose models in professional scenarios.

[0051] Through the aforementioned technology migration and module adaptation, the original question-answering enhancement solution can be smoothly transformed into an intelligent auxiliary diagnostic system for the medical field, demonstrating the good scalability and engineering adaptability of this technical solution in complex cross-domain task scenarios.

[0052] This application also provides a language model question-answering quality optimization system for vertical domains, including: The acquisition module acquires complex questions raised by users and divides long documents in the corresponding vertical domain knowledge base into standardized text units based on the complex questions.

[0053] The retrieval module uses a hybrid retrieval tool to query and extract relevant text units related to complex issues from standardized text units. The hybrid retrieval tool includes sparse retrieval and dense retrieval, and the relevant text units include sparse retrieval results and dense retrieval results with preset weight coefficients.

[0054] The extraction module reorders and extracts relevant text units using preset selection rules to obtain the relevant documents.

[0055] The processing module obtains a first output based on relevant documents and a first language model; wherein the first language model includes a first prompt, the first prompt includes a reasoning and action framework, and the first prompt infers the user's answer based on the complex question.

[0056] It is understood that the language model question answering quality optimization system for vertical domains provided in this embodiment of the invention corresponds to the language model question answering quality optimization method for vertical domains described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the language model question answering quality optimization method for vertical domains, and will not be repeated here.

[0057] This invention also provides a computer-readable storage medium storing a computer program for optimizing the quality of language model question answering for a vertical domain, wherein the computer program causes a computer to execute the above-described method for optimizing the quality of language model question answering for a vertical domain.

[0058] This invention also provides an electronic device, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include methods for performing the above-described vertical domain-oriented language model question-answering quality optimization method.

[0059] In summary, compared with existing technologies, it has the following beneficial effects: 1. This application provides a complex question answering method based on knowledge retrieval and prompting reasoning. It adopts a hybrid retrieval and ReAct prompting technology to recall relevant knowledge in vertical domains for general large models and coordinate reasoning and action (retrieval), effectively solving the challenge of insufficient knowledge coverage faced by general large models.

[0060] 2. This application proposes a complex question-answering method based on background knowledge and fine-tuning of thought processes. By introducing high-quality background knowledge to construct semantically enhanced context, and combining it with the model's thought chain reasoning process for fine-tuning training, the model with a relatively small parameter scale can exhibit stronger capabilities in understanding the intent of the question, inferring implicit information, and constructing reasoning paths. This enables the small model to effectively capture key information and carry out reasoning when faced with long contexts, complex and scattered information, effectively solving the problem that the reasoning ability of models with a relatively small parameter scale is still limited.

[0061] 3. The embodiments of this application provide a method for complex question answering in vertical domains based on LLMs. Relying on the powerful natural language understanding capabilities of LLMs, it effectively solves the challenges of insufficient complex semantic understanding capabilities and difficulty in dealing with multi-level semantic reasoning in traditional complex question answering methods, laying a solid foundation for complex question answering research.

[0062] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0063] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing the quality of language model question answering in a vertical domain, characterized in that, include: Get complex questions raised by users, and divide long documents in the corresponding vertical domain knowledge base into standardized text units based on the complex questions; A hybrid search engine is used to query and extract relevant text units related to the complex problem from the standardized text units; wherein, the hybrid search engine includes sparse search and dense search, and the relevant text units include sparse search results and dense search results with preset weight coefficients; The relevant text units are reordered and extracted using preset selection rules to obtain the relevant documents. According to the relevant documents, a first output is obtained through a first language model; wherein, the first language model includes a first prompt, the first prompt includes a reasoning and action framework, and the first prompt infers the user's answer based on the complex question.

2. The method according to claim 1, characterized in that, The sparse retrieval includes performing keyword matching on the standardized text units based on the complex problem and scoring them according to document relevance, and extracting highly relevant text units as sparse retrieval results.

3. The method according to claim 1, characterized in that, The intensive retrieval includes vectorizing the standardized text units into paragraph vectors using semantic vector similarity, encoding the complex question into a query vector, calculating the similarity between the paragraph vector and the query vector, and returning highly similar text units as intensive retrieval results.

4. The method according to claim 1, characterized in that, The method further includes: According to the relevant documents, a second output is obtained through a second language model; wherein the second language model is obtained through supervised fine-tuning using a pre-constructed training set.

5. The method according to claim 4, characterized in that, The methods for constructing the training set include: A dataset of a certain size is divided based on regular question-and-answer data as initial training data. The initial training data includes complex questions, initial thought processes, and background knowledge. The initial list of relevant documents is obtained by using the hybrid retrieval system to obtain the retrieval results of the initial training data, which are then weighted, summed, and reordered. The initial list of relevant documents is reviewed by experts to obtain an improved list of relevant documents; Based on the improved list of relevant documents, a thought process is generated using a large language model with a second prompt; the complex problem, the background knowledge, and the thought process serve as the training set.

6. The method according to claim 5, characterized in that, The second prompt includes background knowledge, user questions, and thought processes, and the second prompt generates the thought processes based on the background knowledge and user questions.

7. The method according to claim 4, characterized in that, The second language model is fine-tuned under supervision using LoRA technology.

8. A language model question-answering quality optimization system for vertical domains, characterized in that, include: The acquisition module acquires complex questions raised by users and divides long documents in the corresponding vertical domain knowledge base into standardized text units based on the complex questions. The retrieval module queries and extracts relevant text units related to the complex problem from the standardized text units through a hybrid retrieval tool; wherein, the hybrid retrieval tool includes sparse retrieval and dense retrieval, and the relevant text units include sparse retrieval results and dense retrieval results with preset weight coefficients; The extraction module reorders and extracts the relevant text units according to preset selection rules to obtain the relevant documents; The processing module obtains a first output based on the relevant documents using a first language model; wherein the first language model includes a first prompt, the first prompt includes a reasoning and action framework, and the first prompt infers the user's answer based on the complex question.

9. A computer-readable storage medium, characterized in that, It stores a computer program for optimizing the quality of language model question answering for a vertical domain, wherein the computer program causes a computer to execute the method for optimizing the quality of language model question answering for a vertical domain as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing vertical domain-oriented language model question-answering quality optimization as described in any one of claims 1 to 7.

Images