High-efficiency comprehensive dialogue engine based on large language model and construction method of high-efficiency comprehensive dialogue engine

Abstract

The invention provides an efficient comprehensive dialogue engine based on a large language model and a construction method of the efficient comprehensive dialogue engine. The method comprises the steps that LoRA modules are added into each linear layer in the large language model; performing fine tuning on the linear layer added with the LoRA module to obtain a fine tuning model; selecting an optimal fine tuning model through LoRA gating; and selecting an FAQ dialogue sub-engine, an active guide type dialogue sub-engine, a task type dialogue sub-engine or a dialogue sub-engine based on a knowledge base to reply based on the optimal fine tuning model and different user scenes. According to the method, the cost of constructing the large language model is reduced, the time delay of the model is reduced on the premise of keeping the reasoning speed of the large language model, and the model processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to an efficient comprehensive dialogue engine based on a large language model and a construction method thereof. Background Art

[0002] Large language models are of great significance in building dialogue systems. First, large language models can process and generate natural language, allowing dialogue systems to understand and respond to user input more naturally. These models are trained with large-scale data and have powerful language understanding and generation capabilities. They can capture semantic, contextual, and pragmatic information to provide accurate and fluent answers. Secondly, large language models are highly flexible and adaptable, and can cope with dialogue needs in different fields and scenarios, from simple questions and answers to complex multi-round dialogues. In addition, large language models can gradually improve the performance and user experience of dialogue systems through continuous learning and optimization.

[0003] However, large language models face great challenges in actual business implementation: a dialogue system usually contains multiple modules and engines to cope with different scenarios. In existing work or solutions, these engines need to be supported by different models, resulting in a very large server overhead required for the deployment of a complete dialogue system. Although the use of efficient parameter fine-tuning technology can reduce the size of video memory, it will significantly increase the delay of the model and reduce the effectiveness of the customer service system's responses. Therefore, it is very necessary to design an efficient and comprehensive dialogue engine based on a large language model and its construction method. Summary of the invention

[0004] The purpose of the present invention is to provide an efficient comprehensive dialogue engine based on a large language model and a construction method thereof, so as to reduce costs and reduce the delay of the model while maintaining the reasoning speed of the large language model.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An efficient comprehensive dialogue engine based on a large language model, comprising: a dialogue management sub-engine, an FAQ dialogue sub-engine, an active guidance dialogue sub-engine, a task-based dialogue sub-engine, a knowledge base-based dialogue sub-engine and a large language model; the dialogue management sub-engine, the FAQ dialogue sub-engine, the active guidance dialogue sub-engine, the task-based dialogue sub-engine and the knowledge base-based dialogue sub-engine are all connected to the large language model;

[0007] The large language model is used to select the corresponding dialogue sub-engine for reply according to different user scenarios;

[0008] The dialogue management sub-engine is used to update the dialogue state according to the response type of the dialogue sub-engine selected by the large language model;

[0009] The FAQ dialogue engine is used to compare user questions with questions in the FAQ database and respond based on the comparison results;

[0010] The active guided conversation sub-engine is used to guide users to obtain information step by step;

[0011] The task-based dialogue engine is used to complete specific tasks according to user intentions;

[0012] The knowledge base-based dialogue sub-engine is used to analyze the semantics and context of user questions, match the information in the knowledge base according to the analysis results, and respond based on the matching results.

[0013] Optionally, user scenarios include: consulting services, account management, fee inquiry and payment, repair services, complaints and suggestions, heating problem answers, heating stop notifications and energy saving suggestions.

[0014] A method for constructing an efficient comprehensive dialogue engine based on a large language model comprises the following steps:

[0015] Add the LoRA module to each linear layer in the large language model;

[0016] Fine-tune the linear layer with the LoRA module added to obtain a fine-tuned model;

[0017] Select the optimal fine-tuning model through LoRA gating;

[0018] Based on the optimal fine-tuning model and different user scenarios, the corresponding conversation sub-engine is selected for reply.

[0019] Optionally, the LoRA module is added to each linear layer in the large language model respectively. The specific steps are: add a set of parameter matrices to each linear layer in the large language model; the specific calculation formula is: x'=Wx+ABx; wherein x' is the output of the current linear layer, x is the output of the previous linear layer, W is the parameter matrix of the linear layer, and AB is the added parameter matrix.

[0020] Optionally, the intermediate dimension of the LoRA module is k, the hidden state dimension of the linear layer is d, the size of the parameter matrix of the linear layer is d*d, the size of the A matrix in the added parameter matrix is ​​d*k, and the size of the B matrix in the added parameter matrix is ​​k*d.

[0021] Optionally, the linear layer with the LoRA module added is fine-tuned to obtain a fine-tuned model, and the specific steps are: predict the tth word according to the first t-1 words in the sentence input by the large language model, and combine the predicted tth word with the first t-1 words to form a predicted sentence; the calculation formula of the predicted sentence is: Among them, LH is the possibility of predicting the sentence, T is the length of the predicted sentence, and x t is the tth word, x <t is the first t-1 words, P θ () is the operation for calculating log-likelihood.

[0022] Optionally, the optimal fine-tuning model is selected through LoRA gating. The specific steps include:

[0023] Input the prompt text into the large language model, and use the output of the last linear layer of the large language model as the vector representation of the prompt text;

[0024] The vector representation is calculated through LoRA gating to obtain the calculation result;

[0025] Select the optimal fine-tuning model based on the calculation results.

[0026] Optionally, the optimal fine-tuning model is selected through LoRA gating, and the specific calculation formula is: prob = Softmax (h Pt,m W g,m );l *,m = argmax(prob); where Pt is the prompt text, h Pt,m is a vector representation, l *,m is the optimal fine-tuning model, Softmax() is the normalization function calculation, W g,m is the parameter of the mth module in the gth layer of the large language model, prob is the model selection probability, and argmax() is the maximum value operation.

[0027] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention provides an efficient comprehensive dialogue engine construction method based on a large language model, the method comprising: adding LoRA modules to each linear layer in the large language model respectively; fine-tuning the linear layer with the LoRA module added to obtain a fine-tuning model; selecting the optimal fine-tuning model through LoRA gating; and selecting the corresponding dialogue sub-engine for reply based on the optimal fine-tuning model and different user scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1This is a structural diagram of an efficient comprehensive dialogue engine based on a large language model according to an embodiment of the present invention;

[0030] Figure 2 This is a flow chart of a method for building an efficient comprehensive dialogue engine based on a large language model according to an embodiment of the present invention;

[0031] Figure 3 This is a flow chart of LoRA gating selection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] like Figure 1 As shown, the embodiment of the present invention provides an efficient comprehensive dialogue engine based on a large language model, including: a dialogue management sub-engine, a FAQ dialogue sub-engine, an active guidance dialogue sub-engine, a task-based dialogue sub-engine, a knowledge base-based dialogue sub-engine and a large language model; the dialogue management sub-engine, the FAQ dialogue sub-engine, the active guidance dialogue sub-engine, the task-based dialogue sub-engine and the knowledge base-based dialogue sub-engine are all connected to the large language model;

[0035] The large language model is used to select the corresponding dialogue sub-engine for response according to different user scenarios.

[0036] Specifically, the dialogue management sub-engine is an upper-level encapsulation of the FAQ dialogue sub-engine, the active guidance dialogue sub-engine, the task-based dialogue sub-engine and the knowledge base-based dialogue sub-engine. It is used to manage the dialogue status, and is also used to select the appropriate dialogue sub-engine to generate a reply based on the historical dialogue status and the current user question, and update the dialogue status according to the type of reply.

[0037] The FAQ dialogue engine is a system that uses natural language processing technology to understand user questions and automatically provide relevant answers based on a pre-defined set of common questions and their answers. The FAQ dialogue engine uses a semantic matching model to compare user questions with questions in the FAQ library to select the most relevant answers for reply. The FAQ dialogue engine can quickly and efficiently answer users' common questions, greatly reducing the workload of manual customer service, improving response speed and user satisfaction, and ensuring the consistency and accuracy of information.

[0038] An active-guidance conversation engine is a conversation system that guides users to gradually obtain information or complete specific tasks through preset conversation processes and prompts. An active-guidance conversation engine can predict user needs and guide users through complex interactions by proactively asking questions, providing options, and clarifying questions. An active-guidance conversation engine significantly improves user experience and interaction efficiency. It can also help users discover important information that they may have overlooked and reduce the number of multiple rounds of conversations, thereby improving overall service efficiency and user satisfaction.

[0039] A task-based dialog engine is a dialog system that focuses on completing specific tasks. It uses natural language processing technology to understand user intent, automatically perform corresponding operations, and provide feedback. The task-based dialog engine can not only answer user questions, but also handle complex user requests, such as booking services, modifying account information, or submitting repair orders.

[0040] The knowledge-based conversation engine uses a pre-built information base, natural language understanding and generation technology to enable the system to understand the questions raised by users and give corresponding answers or solutions. The knowledge-based conversation engine can accurately match and retrieve information in the knowledge base by analyzing the semantics and context of user questions, and provide users with personalized and effective support, thereby improving customer interaction efficiency and problem-solving capabilities.

[0041] Specifically, user scenarios include: consulting services, account management, fee inquiry and payment, repair services, complaints and suggestions, answers to heating questions, heating stop notifications and energy saving suggestions.

[0042] like Figure 2 As shown, the embodiment of the present invention also provides a method for constructing an efficient comprehensive dialogue engine based on a large language model, comprising the following steps:

[0043] Step 100: Add the LoRA module to each linear layer in the large language model respectively; the specific steps are: add a set of parameter matrices to each linear layer in the large language model, and change the operation of the current linear layer to:

[0044] x'=Wx+ABx;

[0045] Among them, x' is the output of the current linear layer, x is the output of the previous linear layer, W is the parameter matrix of the linear layer, and AB is the added parameter matrix.

[0046] Specifically, the intermediate dimension of the LoRA module is k, the hidden state dimension of the linear layer is d, the size of the parameter matrix of the linear layer is d*d, the size of the A matrix in the added parameter matrix is ​​d*k, and the size of the B matrix in the added parameter matrix is ​​k*d. Since the value of k is much smaller than d, the number of parameters of the A and B matrices is very small compared to the number of parameters of the original large model.

[0047] Specifically, the scale of the large language model base is 20 billion parameters, and the parameter volume of the LoRA module is close to 10M, accounting for less than 1% of the large language model backbone.

[0048] Step 200: Fine-tune the linear layer with the LoRA module added to obtain a fine-tuned model; the specific steps are: according to the first t-1 words x in the sentence input by the large language model <t =[x1,x2,x3...,x t-1 ]Predict the tth word x t , and combine the predicted t-th word with the previous t-1 words to form a predicted sentence; the calculation formula for the predicted sentence is:

[0049]

[0050] Among them, LH is the possibility of predicting the sentence, T is the length of the predicted sentence, and x t is the tth word, x <t is the first t-1 words, P θ () indicates that the fine-tuned model calculates the probability of word x.

[0051] Specifically, the result of the fine-tuning model calculating the probability of word x is the log-likelihood.

[0052] Specifically, fine-tuning uses efficient parameter fine-tuning rather than full parameter fine-tuning to prevent overfitting.

[0053] Specifically, the model without the LoRA module can generate an average of 35.8 tokens per second, while the model with the LoRA module can generate an average of 28.4 tokens per second.

[0054] Step 300: Select the optimal fine-tuning model through LoRA gating; the specific steps are as follows Figure 3 As shown, including:

[0055] Step 301: input the prompt text into the large language model, and use the output of the last linear layer of the large language model as the vector representation of the prompt text Pt;

[0056] Specifically, the parameter of the mth module in the gth layer of the large language model is defined as W g,m .

[0057] More specifically, the g-th layer of the large language model has 6 modules, namely query, key, value, output, up-projection and down-projection.

[0058] Step 302: Calculate the vector representation through LoRA gating to obtain a calculation result; the calculation formula is:

[0059] prob = Softmax(h Pt,m W g,m );

[0060] Among them, h Pt,m is a vector representation, Softmax() is a normalization function calculation, prob is the model selection probability, W g,m is the parameter of the mth module in the gth layer of the large language model.

[0061] Step 303: Select the optimal fine-tuning model according to the calculation results; the calculation formula of the selection process is:

[0062] l *,m = argmax(prob);

[0063] Among them, l *,m is the optimal fine-tuning model, and argmax() is the maximum value operation.

[0064] Specifically, step 300 makes only l *,m The LoRA parameters are used, that is, the large language model no longer performs LoRA gating calculations in the subsequent multiple Token generation process for the prompt text.

[0065] Specifically, through LoRA gating selection, the inference speed of the large language model is increased from 28.4 / second to 34.7 / second.

[0066] Step 400: Select the corresponding dialogue sub-engine for reply based on the optimal fine-tuning model and different user scenarios.

[0067] The beneficial effects of the present invention are as follows:

[0068] 1) The potential of large language models is explored through the LoRA module;

[0069] 2) LoRA gating avoids the problem of adding unnecessary LoRA parameters;

[0070] 3) The optimal fine-tuning model selected for each layer of the large language model is different, which can adapt to different inputs to achieve better response effects;

[0071] 4) Reduced the delay caused by the LoRA module and improved the response speed of the model.

[0072] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0073] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An efficient comprehensive dialogue engine based on a large language model, characterized in that: include: A dialogue management sub-engine, a FAQ dialogue sub-engine, an active guidance dialogue sub-engine, a task-based dialogue sub-engine, a knowledge-based dialogue sub-engine and a large language model; the dialogue management sub-engine, the FAQ dialogue sub-engine, the active guidance dialogue sub-engine, the task-based dialogue sub-engine and the knowledge-based dialogue sub-engine are all connected to the large language model; The large language model is used to select the corresponding dialogue sub-engine for reply according to different user scenarios; The dialog management sub-engine is used to update the dialog state according to the reply type of the dialog sub-engine selected by the large language model; The FAQ dialogue sub-engine is used to compare the user's question with the questions in the FAQ library and respond according to the comparison result; The active guidance type dialogue sub-engine is used to guide the user to gradually obtain information; The task-based dialogue sub-engine is used to complete a specific task according to the user's intention; The knowledge base-based dialogue sub-engine is used to analyze the semantics and context of user questions, match the information in the knowledge base according to the analysis results, and respond according to the matching results.

2. The efficient comprehensive dialogue engine based on a large language model according to claim 1, characterized in that: The user scenarios include: consulting services, account management, fee inquiry and payment, repair services, complaints and suggestions, heating problem answers, heating stop notifications and energy saving suggestions.

3. A method for constructing an efficient comprehensive dialogue engine based on a large language model, applied to the efficient comprehensive dialogue engine based on a large language model as claimed in any one of claims 1 to 2, characterized in that: The steps include: Add the LoRA module to each linear layer in the large language model; Fine-tune the linear layer with the LoRA module added to obtain a fine-tuned model; Select the optimal fine-tuning model through LoRA gating; Based on the optimal fine-tuning model and different user scenarios, a corresponding dialogue sub-engine is selected for reply.

4. The method for constructing an efficient comprehensive dialogue engine based on a large language model according to claim 3, characterized in that: The LoRA module is added to each linear layer in the large language model respectively. The specific steps are: add a set of parameter matrices to each linear layer in the large language model; the specific calculation formula is: x'=Wx+ABx; where x' is the output of the current linear layer, x is the output of the previous linear layer, W is the parameter matrix of the linear layer, and AB is the added parameter matrix.

5. The method for constructing an efficient comprehensive dialogue engine based on a large language model according to claim 4, characterized in that: The intermediate dimension of the LoRA module is k, the hidden state dimension of the linear layer is d, the size of the parameter matrix of the linear layer is d*d, the size of the A matrix in the added parameter matrix is ​​d*k, and the size of the B matrix in the added parameter matrix is ​​k*d.

6. The method for constructing an efficient comprehensive dialogue engine based on a large language model according to claim 3, characterized in that: The linear layer with the LoRA module added is fine-tuned to obtain a fine-tuning model. The specific steps are: predicting the tth word according to the first t-1 words in the sentence input by the large language model, and combining the predicted tth word with the first t-1 words to form a predicted sentence; the calculation formula of the predicted sentence is: Among them, LH is the possibility of predicting the sentence, T is the length of the predicted sentence, and x t is the tth word, x <t is the first t-1 words, P θ () is the operation for calculating log-likelihood.

7. The method for constructing an efficient comprehensive dialogue engine based on a large language model according to claim 3, characterized in that: Select the optimal fine-tuning model through LoRA gating. The specific steps include: Inputting the prompt text into the large language model, and using the output of the last linear layer of the large language model as the vector representation of the prompt text; Calculate the vector representation through the LoRA gating to obtain a calculation result; The optimal fine-tuning model is selected according to the calculation result.

8. The method for constructing an efficient comprehensive dialogue engine based on a large language model according to claim 7, characterized in that: The optimal fine-tuning model is selected through LoRA gating. The specific calculation formula is: prob = Softmax (h Pt,m W g,m );l *,m = argmax(prob); where Pt is the prompt text, h Pt,m is a vector representation, l *,m is the optimal fine-tuning model, Softmax() is the normalization function calculation, W g,m is the parameter of the mth module in the gth layer of the large language model, prob is the model selection probability, and argmax() is the maximum value operation.

Images