Method for finely adjusting language model based on domain knowledge and related equipment

Abstract

The invention provides a method for finely adjusting a language model based on domain knowledge and related equipment, and the method comprises the steps: obtaining a plurality of reply texts according to a first question text related to a target domain and first knowledge information corresponding to the first question text; determining a first reply text set of which the reply quality meets a set condition from the plurality of reply texts; generating a training prompt word according to a second question text associated with the first question text and second knowledge information associated with the first knowledge information; and taking the training prompt word as input data, taking at least one reply text in the first reply text set as an annotation result, and performing fine adjustment on the target language model. The method can improve the reasoning effect of the target language model in the target domain. For example, in the fields of IT management, operation and maintenance, IT intelligent questions and answers and the like, the model can be finely adjusted according to the scheme on the basis of IT field knowledge information so as to complete IT field tasks.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for fine-tuning a language model based on domain knowledge. Background Art

[0002] With the rapid development of computer technology, language models have emerged. Language models usually have natural language processing capabilities and can handle different types of natural language tasks. In order to improve the reasoning ability of language models in specific fields, the industry usually uses retrieval-augmented generation (RAG) technology to assist language models in task processing.

[0003] Specifically, RAG technology is divided into two stages: knowledge retrieval stage and response generation stage. The knowledge retrieval stage is used to retrieve knowledge information related to the question text from the knowledge base of a specific field, and the response generation stage is used to combine the knowledge information related to the question text to generate the response text.

[0004] However, in the above method, during the response generation stage, since the amount of knowledge information related to the question text is usually large, the quality of the response text generated by the language model is difficult to control. Summary of the invention

[0005] The present application provides a method for fine-tuning a language model based on domain knowledge. The method can improve the reasoning effect of the target language model in the target domain during the response generation phase of the RAG technology. The present application also provides a device, an electronic device, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0006] In a first aspect, the present application provides a method for fine-tuning a language model based on domain knowledge, the method comprising:

[0007] According to a first question text related to the target field and first knowledge information corresponding to the first question text, a plurality of reply texts are obtained; wherein the plurality of reply texts are texts replying to the first question text according to the first knowledge information;

[0008] Determine, from the plurality of reply texts, a first reply text set whose reply quality meets a set condition;

[0009] Generate training prompt words according to a second question text associated with the first question text and second knowledge information associated with the first knowledge information;

[0010] The training prompt words are used as input data, and at least one reply text in the first reply text set is used as a labeling result to fine-tune the target language model.

[0011] In a second aspect, the present application provides a device for fine-tuning a language model based on domain knowledge, the device comprising:

[0012] An acquisition module, configured to obtain a plurality of reply texts according to a first question text related to a target domain and first knowledge information corresponding to the first question text; wherein the plurality of reply texts are texts replying to the first question text according to the first knowledge information;

[0013] A determination module, configured to determine, from the plurality of reply texts, a first reply text set whose reply quality meets a set condition;

[0014] A generating module, configured to generate a training prompt word according to a second question text associated with the first question text and second knowledge information associated with the first knowledge information;

[0015] The fine-tuning module is used to use the training prompt word as input data and at least one reply text in the first reply text set as a labeling result to fine-tune the target language model.

[0016] In a third aspect, the present application provides an electronic device, the electronic device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory, so that the electronic device performs the method for fine-tuning a language model based on domain knowledge in the first aspect or any implementation of the first aspect.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct an electronic device to execute the method for fine-tuning a language model based on domain knowledge as described in the first aspect or any one of the implementations of the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method for fine-tuning a language model based on domain knowledge as described in the first aspect or any one of the implementations of the first aspect.

[0019] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.

[0020] It can be seen from the above technical solutions that this application has the following advantages:

[0021] The present application provides a method for fine-tuning a language model based on domain knowledge. The method first obtains multiple reply texts based on a first question text related to a target domain and first knowledge information corresponding to the first question text, wherein the multiple reply texts are texts that reply to the first question text based on the first knowledge information. Next, a first reply text set whose reply quality meets set conditions is determined from the multiple reply texts. Training prompt words are generated based on a second question text associated with the first question text and second knowledge information associated with the first knowledge information. The training prompt words are used as input data, and at least one reply text in the first reply text set is used as a labeling result to fine-tune the target language model.

[0022] In this method, for the first question text in the target domain, multiple reply texts are obtained in combination with the first knowledge information, and a set of first reply texts with better reply quality is screened from the multiple reply texts. The knowledge retrieval stage in the RAG technology is simulated by the second question text associated with the first question text and the second knowledge information associated with the first knowledge information, and the training prompt words are generated, and the set of first reply texts with better reply quality is used as the annotation result, so that the target language model can learn the reply texts with better reply quality when the input is the training prompt words, and improve the reasoning effect of the target language model in the target domain in the reply generation stage of the RAG technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0024] Figure 1 A flowchart of a method for fine-tuning a language model based on domain knowledge provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for fine-tuning a language model based on domain knowledge provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a device for fine-tuning a language model based on domain knowledge provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0029] First, some technical terms and application scenarios involved in the embodiments of this application are introduced.

[0030] With the rapid development of computer technology, language models have emerged. Language models have powerful expression and learning capabilities and can handle many different types of natural language tasks, such as dialogue tasks, translation tasks, speech recognition tasks, text generation tasks, semantic analysis tasks, etc.

[0031] Generally, language models can perform natural language processing on the question text entered by the user and generate the corresponding reply text. Language models can have good reasoning ability in general fields that are not related to professional technology. However, for specific fields related to professional technology, language models are difficult to reason with good results.

[0032] In related technologies, in order to use language models to integrate domain knowledge information in specific fields and improve the reasoning ability of language models in specific fields, the industry usually adopts a method of directly fine-tuning the language model based on domain knowledge, or a method of assisting the language model in task processing based on retrieval-augmented generation (RAG) technology.

[0033] However, fine-tuning the language model directly based on domain knowledge will lead to the problem of language model hallucination. Moreover, if the domain knowledge of a specific domain is added or modified, the fine-tuned language model cannot adapt to the change and needs to be retrained. The method of processing tasks based on RAG technology to assist the language model can cope with the addition or modification of domain knowledge in a specific domain.

[0034] Specifically, RAG technology is divided into two stages: knowledge retrieval stage and response generation stage. In the knowledge retrieval stage, for the question text input by the user, the recall model is called to retrieve the knowledge information related to the question text from the knowledge base of the specific field. In the response generation stage, prompt words are generated based on the knowledge information related to the question text retrieved in the knowledge retrieval stage and the question text, and the prompt words are sent to the language model, so that the language model can analyze the question text in combination with the knowledge information related to the question text and generate a response text.

[0035] However, in the above method, there are the following problems in the response generation stage: since the amount of knowledge information related to the question text is usually large, the quality of the response text generated by the language model (such as the output format, the format of calling the knowledge information in the knowledge base, etc.) is difficult to control. To solve this problem, it is usually necessary to add a large amount of information to the prompt words to indicate the output format and the format of calling the knowledge in the knowledge base. However, the increase in prompt words will increase the time consumption of requesting the language model. At the same time, too long prompt words will also cause the language model's ability to follow instructions to deteriorate.

[0036] In view of this, the present application provides a method for fine-tuning a language model based on domain knowledge. The method first obtains multiple reply texts based on a first question text related to a target domain and first knowledge information corresponding to the first question text, wherein the multiple reply texts are texts that reply to the first question text based on the first knowledge information. Next, a first reply text set whose reply quality meets set conditions is determined from the multiple reply texts. Training prompt words are generated based on a second question text associated with the first question text and second knowledge information associated with the first knowledge information. The training prompt words are used as input data, and at least one reply text in the first reply text set is used as a labeling result to fine-tune the target language model.

[0037] In this method, for the first question text in the target domain, multiple reply texts are obtained in combination with the first knowledge information, and a set of first reply texts with better reply quality is screened from the multiple reply texts. The knowledge retrieval stage in the RAG technology is simulated by the second question text associated with the first question text and the second knowledge information associated with the first knowledge information, and the training prompt words are generated, and the set of first reply texts with better reply quality is used as the annotation result, so that the target language model can learn the reply texts with better reply quality when the input is the training prompt words, and improve the reasoning effect of the target language model in the target domain in the reply generation stage of the RAG technology.

[0038] To facilitate understanding of the technical solution provided by the embodiments of the present application, the following will be described in conjunction with the accompanying drawings. Figure 1 A flow chart of a method for fine-tuning a language model based on domain knowledge is shown, and the method specifically includes:

[0039] S101: Obtain multiple reply texts according to a first question text related to a target field and first knowledge information corresponding to the first question text.

[0040] In the embodiments of the present application, the target domain can be understood as any knowledge domain related to professional technology. Among them, the knowledge domain can be understood as a concept corresponding to the general domain. The general domain is usually unrelated to professional technology. For example, the general domain may include the field of common sense, the field of language communication, etc. The knowledge domain is usually related to professional technology. For example, the knowledge domain may include the field of mathematics, the field of physics, etc. In other words, the knowledge information in the general domain is usually not professional, and the knowledge information in the knowledge domain is usually professional.

[0041] In actual business scenarios, knowledge domains can also be domains related to specific businesses. For example, in a product sales scenario, knowledge information related to product sales (such as knowledge related to product introduction, knowledge related to product sales cases, knowledge related to potential customers, etc.) can constitute the product sales domain. For another example, in a store management scenario, knowledge information related to store management (such as knowledge related to store information, knowledge related to business conditions, etc.) can constitute the store management domain.

[0042] The first question text related to the target field can be understood as the question text under the target field. For example, when the target field is the product sales field, the first question text can be a question related to product sales.

[0043] The first knowledge information corresponding to the first question text can be understood as information that can assist in answering the first question text. Since the first question text is related to the target field, the first knowledge information can also be information related to the target field. The first knowledge information can provide knowledge related to the target field, thereby helping to answer the first question text.

[0044] The multiple reply texts can be understood as reply texts generated after multiple answers to the first question text. In an embodiment of the present application, the multiple reply texts can be texts that reply to the first question text based on the first knowledge information. In other words, multiple answers are given to the first question text in combination with the knowledge related to the target field provided by the first knowledge information to generate multiple reply texts.

[0045] In the embodiment of the present application, different methods are supported for obtaining the first question text and the first knowledge information. Figure 2 As shown, in some embodiments, user questions in the target field are collected, and a first question text is determined based on the user questions. Then, multiple knowledge articles are retrieved from the knowledge base of the target field based on the first question text, and first knowledge information is determined based on the multiple knowledge articles.

[0046] That is to say, the first question text may be derived from the question text actually input by the user. For example, when the target field is related to a specific business, the first question text may be questions asked by users collected in the production environment of the business. For example, in the IT management and service process of an enterprise, IT-related questions asked by users may be collected, such as how to deal with computer freezes and when to apply for a replacement of an office computer.

[0047] Next, knowledge information related to the first question text is retrieved from the knowledge base of the target field to obtain a plurality of knowledge articles, wherein the knowledge base of the target field may store knowledge articles related to the target field.

[0048] In some possible implementations, the knowledge base of the target domain may be a vector knowledge base. The knowledge base of the target domain is constructed by collecting knowledge articles related to the target domain from different source channels, dividing the knowledge articles related to the target domain into blocks, vectorizing the knowledge articles after the block processing, generating vectorized representations corresponding to the knowledge articles related to the target domain, and storing the vectorized representations corresponding to the knowledge articles related to the target domain.

[0049] By vectorizing the first question text, determining the vectorized representation corresponding to the first question text, and calculating the similarity between the vectorized representation corresponding to the first question text and the vectorized representation corresponding to each knowledge article in the knowledge base of the target domain, the knowledge articles whose similarity meets the similarity condition (for example, the similarity is greater than the similarity threshold) are determined as multiple knowledge articles related to the first question text.

[0050] Based on the plurality of knowledge articles, the first knowledge information can be determined. In some possible implementations, the first knowledge information may include information related to each of the plurality of knowledge articles. For example, for each of the plurality of knowledge articles, knowledge information in the following format may be formed:

[0051] "Article:{{order}}

[0052] Link:{{link}}

[0053] Title:{{title}}

[0054] Content:{{content}}”

[0055] Among them, {{order}} can be used to indicate the order of the knowledge article among the multiple knowledge articles retrieved. Usually, the multiple knowledge articles retrieved can be arranged in descending order of similarity. {{link}} can be used to indicate the location of the knowledge article in the knowledge base of the target field. For example, {{link}} can be the storage directory or uniform resource locator (URL) of the knowledge article in the knowledge base of the target field. {{content}} is used to indicate the title of the knowledge article, and {{content}} is used to indicate the content of the knowledge article. The knowledge information of each knowledge article integrated in the above format is spliced ​​to form the first knowledge information.

[0056] In other embodiments, continue as Figure 2 As shown, a first knowledge article is obtained from the knowledge base of the target domain, the title of the first knowledge article is determined as the first question text, and the content of the first knowledge article is determined as the first knowledge information. The first knowledge article can be any knowledge article in the knowledge base of the target domain.

[0057] That is to say, considering that the user's question may only cover a small number of knowledge articles in the knowledge base of the target domain, and most of the knowledge articles are not used, it may make it difficult to handle long-tail problems after fine-tuning the model. Therefore, the first question text and the first knowledge information are directly generated based on the knowledge articles in the knowledge base of the target domain, that is, the title of the first knowledge article is determined as the first question text, and the content of the first knowledge article is determined as the first knowledge information, which effectively guarantees the diversity of subsequent training data and makes full use of the knowledge articles in the knowledge base of the target domain.

[0058] After determining the first question text and the first knowledge information by any of the above methods, multiple reply texts can be obtained based on the first question text and the first knowledge information. The embodiment of the present application does not limit the method for obtaining multiple reply texts. In some embodiments, when the first question text is related to the user's question and the first knowledge information is related to multiple knowledge articles, the multiple reply texts can be derived from the reply text of the actual reply. For example, when the target field is related to a specific business, the multiple reply texts can be the reply texts that actually reply to the first question text input by the user in the production environment of the business. In other embodiments, the multiple reply texts can also be the reply texts after multiple users (for example, multiple annotation personnel) reply to the first question text in combination with the first knowledge information.

[0059] S102: Determine, from a plurality of reply texts, a first reply text set whose reply quality meets a set condition.

[0060] In an embodiment of the present application, multiple reply texts to the first question text are screened, and the reply texts whose reply quality meets the set conditions are combined into a first reply text set.

[0061] Among them, the reply quality can be understood as a preference for the reply text. In the embodiment of the present application, based on the actual model training requirements, such as the requirements for the output format, the requirements for the format of the knowledge in the call knowledge base, etc., set the setting conditions, and then compare multiple reply texts with the setting conditions to determine whether the reply quality of the multiple reply texts meets the setting conditions and whether the reply texts meet the output preference.

[0062] In some possible implementations, the set conditions can be divided into a first set condition and a second set condition. The first set condition is first used to qualitatively determine whether the reply text does not belong to the first reply text set, and then the second set condition is used to further quantitatively determine whether the reply text belongs to the first reply text set.

[0063] In specific implementation, it is determined whether multiple reply texts satisfy a first set condition, and the reply texts that do not satisfy the first set condition are determined as reply texts in a second reply text set. Then, for the reply texts deployed in the second reply text set, a quality assessment result is determined according to the second set condition, and the reply texts whose quality assessment results satisfy the quality assessment threshold are determined as reply texts in the first reply text set, and the reply texts whose quality assessment results do not satisfy the quality assessment threshold are determined as reply texts in the second reply text set.

[0064] The second set condition is used to perform quality assessment on the reply text from multiple dimensions, and the first reply text set and the second reply text set include all reply texts in the multiple reply texts.

[0065] That is to say, the first set condition can be understood as the basic set condition. If the reply text does not meet the first set condition, it is considered to not meet the output preference and have poor reply quality. It belongs to the second reply text set and does not belong to the first reply text set.

[0066] In some embodiments, the first setting condition may be:

[0067] “1. A single sentence in the reply text is too long: the single sentence is longer than the single sentence threshold;

[0068] 2. The reply text is too long: the reply text is longer than the text threshold;

[0069] 3. The link in the reply text is not in the first knowledge information;

[0070] 4. The link in the reply text is not output in the markdown syntax format, that is, it is not output in the format of [anchor](link);

[0071] 5. The anchor text of the link in the reply text does not match the link;

[0072] 6. The language of the reply text is inconsistent with the language of the first question text. "

[0073] Among them, the single sentence threshold, text threshold, etc. can be set based on actual output preferences. At the same time, the first setting condition related to the target field can be added in combination with the actual needs of the target field.

[0074] For the reply text that meets the first set condition, the second set condition is further used to perform quality assessment from multiple dimensions. In some embodiments, the multiple dimensions may include multiple of the following: the difficulty of the user to understand the reply text, the number of grammatical types in the output format of the reply text, the degree of association between the reply text and the first question text, and the degree of association between the reply text and the first knowledge information. In other words, the second set condition can be used to perform quality assessment from multiple dimensions such as readability, output format diversity, semantic relevance, and the number of links that reference the first knowledge information. The quality assessment result can be presented in the form of a score. For example, the second set condition can be:

[0075] “1. Readability score: The score is given based on the average sentence length and punctuation of the reply text. The better the readability, the higher the score;

[0076] 2. Markdown syntax diversity scoring: The more syntax such as titles (#), bold (*), and lists (-) are used in the reply text and the better the diversity, the higher the score;

[0077] 3. Semantic relevance score: The response text is divided into specific blocks, such as sentences or paragraphs, and the average score of the relevance of each block to the first question text. The higher the relevance, the higher the score.

[0078] 4. The more links that cite the first knowledge information in the reply text, the higher the score. "

[0079] For each reply text that meets the first set condition, the scores of each item in the second set condition are obtained, and the sum of the scores is determined as the quality evaluation result. In some possible implementations, the sum of the scores can also be divided by (base_len+output_len), that is, the sum of the scores is divided by the sum of the base length (flexibly configurable) and the length of the reply text, so as to encourage the use of shorter reply texts to express richer information.

[0080] The quality evaluation result of each reply text that meets the first set condition is compared with the quality evaluation threshold, and the reply text that meets the quality evaluation threshold is determined to belong to the first reply text set. In this way, multiple reply texts are classified to form a first reply text set with better reply quality and a second reply text set with poor reply quality, so that they can be used as annotation results in the subsequent model fine-tuning process, so that the language model (referred to as the target language model in the embodiment of the present application) can improve the reply quality.

[0081] S103: Generate training prompt words according to the second question text associated with the first question text and the second knowledge information associated with the first knowledge information.

[0082] In the model fine-tuning process of the embodiment of the present application, the target language model is fine-tuned. Among them, the target language model has natural language processing capabilities, can understand the meaning of natural language, and process different types of natural language tasks. For example, the target language model can be a deep learning model trained using text data.

[0083] The target language model performs natural language processing based on prompt learning. Among them, prompt words can be used to guide the language model to perform specific outputs in generative tasks (such as text generation tasks, question-answering tasks, and dialogue tasks). By configuring prompt words, the language model can understand the background and requirements of the task, so that the language model can handle different types of natural language processing tasks without retraining the language model, increasing the scalability and flexibility of the language model.

[0084] Therefore, in the embodiment of the present application, the training prompt words can be understood as the prompt words input into the target language model during the model fine-tuning process. In other words, the training prompt words can be used as the input part during the model fine-tuning process.

[0085] In the present embodiment, continue as Figure 2 As shown, since a plurality of question-answer pairs consisting of a first question text and a reply text in a first reply text set have been obtained, and the first reply text is generated based on the first knowledge information, in the process of generating training prompt words, the knowledge retrieval process in the RAG technology is simulated, that is, the process of retrieving the second knowledge information based on the second question text is simulated, and then the second question text, the second knowledge information and the information indicating that the language model replies to the second question text based on the second knowledge information are combined to generate training prompt words.

[0086] In some possible implementations, the training prompt words may be as follows:

[0087] "As a RAG technical expert, please select the relevant section from the given article to answer the user query.

[0088] User query:{{query}}

[0089] Related articles:

[0090] {{information}}”

[0091] Among them, {{query}} is the second question text, {{information}} is the second knowledge information, and the remaining content is information instructing the language model to reply to the second question text based on the second knowledge information.

[0092] In some possible implementations, the second knowledge information may include multiple knowledge articles. In this case, each knowledge article in the second knowledge information may be presented in {{information}} in the following format:

[0093] "Article:{{order}}

[0094] Link:{{link}}

[0095] Title:{{title}}

[0096] Content:{{content}}”

[0097] Among them, {{order}} can be used to indicate the order of the knowledge article among the multiple knowledge articles of the second knowledge information. {{link}} can be used to indicate the position of the knowledge article in the knowledge base of the target domain, {{content}} is used to indicate the title of the knowledge article, and {{content}} is used to indicate the content of the knowledge article.

[0098] Different first question texts from different sources may correspond to different second question texts. In some embodiments, the first question text is related to a user's question, and the first knowledge information is related to multiple knowledge articles. In this case, the first question text is determined as the second question text, and the order of the multiple knowledge articles is randomly arranged to determine the second knowledge information, and then, the training prompt words are generated according to the second question text and the second knowledge information.

[0099] That is to say, when the first question text comes from a user's question and the first knowledge information is retrieved from the knowledge base of the target field based on the first question text, the second question text is the same as the first question text, and the second knowledge information is a plurality of knowledge articles formed by randomly arranging a plurality of knowledge articles in the first knowledge information.

[0100] In this way, by randomly arranging the multiple knowledge articles retrieved in the knowledge retrieval stage, the target language model can answer the question based on the relevance of each knowledge article to the second question text during the subsequent model fine-tuning process, without being affected by the order of the multiple knowledge articles.

[0101] In other embodiments, the first question text is the title of the first knowledge article, the first knowledge information is the content of the first knowledge article, and the first knowledge article is any knowledge article in the knowledge base of the target field. In this case, an enhanced title with semantic similarity to the title of the first knowledge article is determined, and the enhanced title is determined as the second question text, and multiple second knowledge articles are randomly selected from the knowledge base of the target field, and the order of the first knowledge article and the multiple second knowledge articles is randomly arranged to determine the second knowledge information, and the training prompt words are generated according to the second question text and the second knowledge information.

[0102] That is to say, in the case where the first question text comes from the knowledge base of the target domain, since the title of the first knowledge article is the first question text, considering that in the subsequent model fine-tuning process, the target language model can better learn to integrate the knowledge of the target domain, in this embodiment of the application, an enhanced title with similar semantics to the title of the first knowledge article is used as the second question text.

[0103] The embodiments of the present application do not limit the method for determining the enhanced title. In some embodiments, keywords with similar semantics to the title of the first knowledge article are determined, and the keywords are replaced to obtain the enhanced title. In other embodiments, the title of the first knowledge article can also be rewritten by a user (e.g., an annotator) to obtain a semantically similar enhanced title.

[0104] In addition to the first knowledge article, the second knowledge information also includes multiple second knowledge articles randomly sampled from the knowledge base of the target field, and the order of the first knowledge article and the multiple second knowledge articles is randomly arranged. In this way, adding multiple second knowledge articles that are unrelated to the first knowledge article as noise in the training prompt words helps the target language model learn useful information and avoid noise interference during the model fine-tuning process. In this way, in the reasoning process, for the actual request of the user, there is a greater tolerance for the knowledge articles retrieved in the knowledge retrieval stage, which improves the reasoning effect.

[0105] S104: taking the training prompt words as input data and at least one reply text in the first reply text set as a labeling result, and fine-tuning the target language model.

[0106] That is to say, the training prompt word and at least one reply text in the first reply text set can be understood as training data. For example, the first reply text set includes multiple reply texts. In some embodiments, the reply text with the best quality evaluation result in the first reply text set is selected, and the training prompt word and the reply text with the best quality evaluation result are used as a piece of training data. In other embodiments, K reply texts with the top-k quality evaluation results in the first reply text set are selected to generate K pieces of training data consisting of training prompt words and reply texts respectively.

[0107] In the embodiment of the present application, the target language model is fine-tuned based on the self-play fine-tuning (SFT) method. In this way, the target language model can learn the reply text in the first reply text set with better reply quality when the training prompt word is used as input. By adjusting the model parameters of the target language model, the output of the target language model in the target domain is more in line with the output preference, and the output reply text quality is better.

[0108] Furthermore, a second reply text set whose reply quality does not meet the set conditions can be determined from multiple reply texts, and the training prompt words can be used as input data. At least one reply text in the first reply text set can be used as a positive annotation result, and at least one reply text in the second reply text set can be used as a negative annotation result. Based on the self-supervised fine-tuning method, the target language model can be fine-tuned.

[0109] In this way, the target language model can learn the reply texts in the first reply text set with better reply quality when the training prompt words are used as input, and at the same time learn the reply texts in the second reply text set with poor reply quality when the training prompt words are used as input. By adjusting the model parameters of the target language model, the output of the target language model in the target domain is closer to the reply texts in the first reply text set, and the output of the reply texts in the second reply text set is avoided, and the output reply texts have better quality.

[0110] In this method, for the first question text in the target domain, multiple reply texts are obtained in combination with the first knowledge information, and a set of first reply texts with better reply quality is screened from the multiple reply texts. The knowledge retrieval stage in the RAG technology is simulated by the second question text associated with the first question text and the second knowledge information associated with the first knowledge information, and the training prompt words are generated, and the set of first reply texts with better reply quality is used as the annotation result, so that the target language model can learn the reply texts with better reply quality when the input is the training prompt words, and improve the reasoning effect of the target language model in the target domain in the reply generation stage of the RAG technology.

[0111] Combination of the above Figure 1 and Figure 2 The method for fine-tuning the language model based on domain knowledge provided in the embodiment of the present application is introduced in detail. The following will introduce the apparatus and device provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0112] See also Figure 3 The schematic diagram of the structure of the device for fine-tuning the language model based on domain knowledge is shown, and the device 30 includes:

[0113] The acquisition module 301 is used to obtain a plurality of reply texts according to a first question text related to a target field and first knowledge information corresponding to the first question text; wherein the plurality of reply texts are texts replying to the first question text according to the first knowledge information;

[0114] A determination module 302 is used to determine, from the multiple reply texts, a first reply text set whose reply quality meets a set condition;

[0115] A generating module 303, configured to generate a training prompt word according to a second question text associated with the first question text and second knowledge information associated with the first knowledge information;

[0116] The fine-tuning module 304 is configured to use the training prompt word as input data and at least one reply text in the first reply text set as a labeling result to fine-tune the target language model.

[0117] In some possible implementations, the acquisition module 301 is specifically used to:

[0118] Collecting user questions in the target field, and determining a first question text according to the user questions;

[0119] Retrieving a plurality of knowledge articles from a knowledge base of the target domain according to the first question text;

[0120] Determining first knowledge information corresponding to the first question text according to the plurality of knowledge articles;

[0121] A plurality of reply texts are obtained according to the first question text and the first knowledge information.

[0122] In some possible implementations, the acquisition module 301 is specifically used to:

[0123] Obtain the first knowledge article from the knowledge base of the target domain;

[0124] Determine the title of the first knowledge article as the first question text, and determine the content of the first knowledge article as the first knowledge information corresponding to the first question text;

[0125] A plurality of reply texts are obtained according to the first question text and the first knowledge information.

[0126] In some possible implementations, the first question text is related to a user's question, and the first knowledge information is related to a plurality of knowledge articles; the generating module 303 is specifically used to:

[0127] Determining the first question text as the second question text, and randomly arranging the order of the plurality of knowledge articles to determine second knowledge information;

[0128] Generate training prompt words according to the second question text and the second knowledge information.

[0129] In some possible implementations, the first question text is the title of the first knowledge article, the first knowledge information is the content of the first knowledge article, and the first knowledge article is any knowledge article in the knowledge base of the target field; the generating module 303 is specifically used to:

[0130] Determine an enhanced title that is semantically similar to the title of the first knowledge article, and determine the enhanced title as the second question text;

[0131] arbitrarily selecting a plurality of second knowledge articles from the knowledge base of the target domain;

[0132] Randomly arrange the order of the first knowledge article and the plurality of second knowledge articles to determine second knowledge information;

[0133] Generate training prompt words according to the second question text and the second knowledge information.

[0134] In some possible implementations, the determining module 302 is specifically configured to:

[0135] Determining whether the multiple reply texts satisfy a first set condition, and determining reply texts that do not satisfy the first set condition as reply texts in a second reply text set;

[0136] For the reply text that does not belong to the second reply text set, determine the quality assessment result according to the second set condition; wherein the second set condition is used to perform quality assessment on the reply text from multiple dimensions;

[0137] The reply text whose quality assessment result meets the quality assessment threshold is determined as the reply text in the first reply text set; and the reply text whose quality assessment result does not meet the quality assessment threshold is determined as the reply text in the second reply text set.

[0138] In some possible implementations, the multiple dimensions include multiple of the following: the difficulty for the user to understand the reply text, the number of grammatical types in the output format of the reply text, the degree of association between the reply text and the first question text, and the degree of association between the reply text and the first knowledge information.

[0139] In some possible implementations, the determining module 302 is further configured to:

[0140] Determining, from the plurality of reply texts, a second reply text set whose reply quality does not meet a set condition;

[0141] The fine-tuning module 304 is specifically used for:

[0142] The target language model is fine-tuned by taking the training prompt words as input data, taking at least one reply text in the first reply text set as a positive annotation result, and taking at least one reply text in the second reply text set as a negative annotation result.

[0143] The apparatus 30 for fine-tuning a language model based on domain knowledge according to an embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each module / unit of the apparatus 30 for fine-tuning a language model based on domain knowledge are respectively to implement Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0144] The present application also provides an electronic device. The electronic device is specifically used to implement Figure 3 The function of the device 30 for fine-tuning the language model based on domain knowledge in the illustrated embodiment.

[0145] Figure 4 A schematic diagram of the structure of an electronic device 400 is provided. Figure 4 As shown, the electronic device 400 includes a bus 401, a processor 402, a communication interface 403 and a memory 404. The processor 402, the memory 404 and the communication interface 403 communicate with each other via the bus 401.

[0146] The bus 401 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] The processor 402 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0148] The communication interface 403 is used for communicating with the outside. For example, the communication interface 403 can be used for communicating with a terminal.

[0149] The memory 404 may include a volatile memory, such as a random access memory (RAM). The memory 404 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0150] The memory 404 stores executable codes, and the processor 402 executes the executable codes to perform the aforementioned method for fine-tuning a language model based on domain knowledge.

[0151] Specifically, in implementing Figure 3 In the case of the embodiment shown, and Figure 3 When each module or unit of the device 30 for fine-tuning a language model based on domain knowledge described in the embodiment is implemented by software, the execution Figure 3 The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 404. The processor 402 executes the program code corresponding to each unit stored in the memory 404 to perform the aforementioned method for fine-tuning the language model based on domain knowledge.

[0152] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the method for fine-tuning a language model based on domain knowledge applied to the device 30 for fine-tuning a language model based on domain knowledge.

[0153] The embodiment of the present application further provides a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.

[0154] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0155] When the computer program product is executed by a computer, the computer executes any of the aforementioned methods for fine-tuning a language model based on domain knowledge. The computer program product may be a software installation package, and when any of the aforementioned methods for fine-tuning a language model based on domain knowledge is needed, the computer program product may be downloaded and executed on a computer.

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the unit / module does not, in some cases, constitute a limitation on the unit itself.

[0158] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0159] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0160] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0161] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0162] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0163] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0164] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for fine-tuning a language model based on domain knowledge, characterized in that: The method comprises: According to a first question text related to the target field and first knowledge information corresponding to the first question text, a plurality of reply texts are obtained; wherein the plurality of reply texts are texts replying to the first question text according to the first knowledge information; Determine, from the plurality of reply texts, a first reply text set whose reply quality meets a set condition; Generate training prompt words according to a second question text associated with the first question text and second knowledge information associated with the first knowledge information; The training prompt words are used as input data, and at least one reply text in the first reply text set is used as a labeling result to fine-tune the target language model.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of reply texts according to a first question text related to the target field and first knowledge information corresponding to the first question text includes: Collecting user questions in the target field, and determining a first question text according to the user questions; Retrieving a plurality of knowledge articles from a knowledge base of the target domain according to the first question text; Determining first knowledge information corresponding to the first question text according to the plurality of knowledge articles; A plurality of reply texts are obtained according to the first question text and the first knowledge information.

3. The method according to claim 1, characterized in that The step of obtaining a plurality of reply texts according to a first question text related to the target field and first knowledge information corresponding to the first question text includes: Obtain the first knowledge article from the knowledge base of the target domain; Determine the title of the first knowledge article as the first question text, and determine the content of the first knowledge article as the first knowledge information corresponding to the first question text; A plurality of reply texts are obtained according to the first question text and the first knowledge information.

4. The method according to claim 1, characterized in that The first question text is related to a user's question, and the first knowledge information is related to a plurality of knowledge articles; The step of generating a training prompt word according to the second question text associated with the first question text and the second knowledge information associated with the first knowledge information includes: Determining the first question text as the second question text, and randomly arranging the order of the plurality of knowledge articles to determine second knowledge information; Generate training prompt words according to the second question text and the second knowledge information.

5. The method according to claim 1, characterized in that The first question text is the title of the first knowledge article, the first knowledge information is the content of the first knowledge article, and the first knowledge article is any knowledge article in the knowledge base of the target field; The step of generating a training prompt word according to the second question text associated with the first question text and the second knowledge information associated with the first knowledge information includes: Determine an enhanced title that is semantically similar to the title of the first knowledge article, and determine the enhanced title as the second question text; arbitrarily selecting a plurality of second knowledge articles from the knowledge base of the target domain; Randomly arrange the order of the first knowledge article and the plurality of second knowledge articles to determine second knowledge information; Generate training prompt words according to the second question text and the second knowledge information.

6. The method according to claim 1, characterized in that The step of determining, from the plurality of reply texts, a first reply text set whose reply quality satisfies a set condition comprises: Determining whether the multiple reply texts satisfy a first set condition, and determining reply texts that do not satisfy the first set condition as reply texts in a second reply text set; For the reply text that does not belong to the second reply text set, determine the quality assessment result according to the second set condition; wherein the second set condition is used to perform quality assessment on the reply text from multiple dimensions; The reply text whose quality assessment result meets the quality assessment threshold is determined as the reply text in the first reply text set; and the reply text whose quality assessment result does not meet the quality assessment threshold is determined as the reply text in the second reply text set.

7. The method according to claim 6, characterized in that The multiple dimensions include multiple ones of the following: the difficulty of the user to understand the reply text, the number of grammatical types in the output format of the reply text, the degree of association between the reply text and the first question text, and the degree of association between the reply text and the first knowledge information.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Determining, from the plurality of reply texts, a second reply text set whose reply quality does not meet a set condition; The step of taking the training prompt word as input data and taking at least one reply text in the first reply text set as a labeling result to fine-tune the target language model includes: The target language model is fine-tuned by taking the training prompt words as input data, taking at least one reply text in the first reply text set as a positive annotation result, and taking at least one reply text in the second reply text set as a negative annotation result.

9. A device for fine-tuning a language model based on domain knowledge, characterized in that: The device comprises: An acquisition module, configured to obtain a plurality of reply texts according to a first question text related to a target domain and first knowledge information corresponding to the first question text; wherein the plurality of reply texts are texts replying to the first question text according to the first knowledge information; A determination module, configured to determine, from the plurality of reply texts, a first reply text set whose reply quality meets a set condition; A generating module, configured to generate a training prompt word according to a second question text associated with the first question text and second knowledge information associated with the first knowledge information; The fine-tuning module is used to use the training prompt word as input data and at least one reply text in the first reply text set as a labeling result to fine-tune the target language model.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct an electronic device to execute the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises computer readable instructions for implementing the method according to any one of claims 1 to 8.

Images