Decoding training method capable of improving dialogue model effect
By positioning the learning focus, calculating and adjusting the weights and modifying the loss function in the large language model, the problem of insufficient attention to the large model illusion and learning focus is solved, and the dialogue model is achieved to improve the knowledge relevance and response consistency.
Patent Information
- Application Number
- CN202510165871.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-14
- Publication Date
- 2025-03-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing large language models are prone to "big model hallucinations" when generating replies, that is, the generated content is inconsistent with real-world facts or user input, and traditional training methods ignore the marking of Tokens that significantly affect the loss, resulting in the model tending to focus on simple words and ignore knowledge-related markings.
Through a decoding training method, we can locate learning focus, calculate and adjust weights, and modify cross-entropy loss, and improve the loss sensitivity of token positions highly correlated with a given knowledge, thereby achieving token-level knowledge constraints.
This improves the attention of the model to the tokens that are highly related to a given knowledge during the training process, enhances the consistency and credibility of the responses, and reduces the occurrence of "big model hallucinations".
Smart Images
Figure CN119623525A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of large language models, and in particular to a decoding training method that can improve the effect of a dialogue model. Background Art
[0002] Large Language Model (LLM): refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.
[0003] Transformer: is a revolutionary architecture in the field of natural language processing (NLP). Transformer is a sequence model based on the attention mechanism, originally proposed by Google's research team and applied to machine translation tasks. Unlike traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), Transformer only uses the self-attention mechanism to process input sequences and output sequences, so it can be calculated in parallel, greatly improving computational efficiency.
[0004] Large model hallucination: Large models experience hallucinations, which refers to the phenomenon that the content generated by the model is inconsistent with real-world facts or user input. In order to enrich the model's response content, a very effective method is to introduce external knowledge. However, such models that introduce external knowledge often generate responses that seem to have knowledge, but are not derived from the given knowledge. This means that the correctness of the knowledge used in the response cannot be guaranteed, which is the "hallucination" phenomenon. Since the knowledge cannot be verified, hallucinations may mislead users and reduce the credibility of the model.
[0005] There are many methods to deal with the problem of large model hallucination, such as knowledge graphs, contrastive learning, or control codes. These models enhance the model's attention to knowledge by adding parameters and carefully designing data engineering. One of the important assumptions is that the model has the ability to pay more attention to knowledge, but this assumption is not necessarily true. The usual training method ignores the tracking of the tags of tokens (i.e., learning focus) that significantly affect the loss. Unlike the traditional attention mechanism that focuses on important information in the input, the learning focus mentioned here focuses on important information in target generation. In general learning scenarios, the learning focus is often uncontrolled, and the model tends to focus on simple words, thereby ignoring those tags that are highly relevant to knowledge. These tags are crucial to improving consistency. Focusing the model's attention on these tags can make the optimization target more suitable for the task. Therefore, it is necessary to modify the original learning focus. To achieve the above goals, there are two main challenges: ① How to locate the desired learning focus. Since the learning focus is on different words in each sentence and the manual annotation of the Toekn-level replies is very time-consuming, there is no fine-grained annotation of key semantic words in the replies in the existing datasets. ② Given the desired learning focus, how to modify the original learning focus. Existing training methods using cross-entropy loss functions lack direct guidance on learning focus. Summary of the invention
[0006] In view of this, the present application provides a decoding training method that can improve the effect of the dialogue model, so as to increase the loss sensitivity of the token position that is highly relevant to the given knowledge, thereby realizing knowledge constraints at the token level.
[0007] To achieve the above purpose, the technical solution of this application is: A decoding training method for improving the effect of a dialogue model, comprising: Step 1: Data input: a conversation containing n rounds of questions , and n Knowledge related to wheel problems And the answers to the questions Input into the large language model, where For reply Y Length ，m Represents the amount of knowledge, Indicates i knowledge, Reply Y Middle t Tokens; Step 2: Data preprocessing: The embedding vector layer of the large language model is used to process the input dialogue. C ,Knowledge K and reply YPerform preprocessing and output question vector, knowledge vector and response vector respectively; Step 3: Large language model training: Use the open source large model as the base model of the large language model for integrated learning and training. C As the input of the large language model, the large language model outputs the probability distribution of the predicted value ,in For reply Y Middle t Tokens The corresponding predicted probability distribution; Step 4. Calculate relevance: Calculate responses Y With given knowledge K The correlation between , For reply Y Middle t Tokens The dense vector representation of k It is given knowledge The dense vector mean of ; Step 5: Calculate the adjustment weight: Based on the correlation Calculating Adjusted Weights for Large Language Models ; Step 6: Probability distribution adjustment: adjust the weights according to the and the original probability distribution Calculate the probability of each Token; Step 7, calculate the loss function: calculate the final loss function according to the probability of each Token obtained in step 6; Step 8, model parameter update: follow the final loss function to update the parameters of the large language model, the parameters of the large language model include the number of neurons, embedding vector dimension and Transformer structure parameters.
[0008] Furthermore, the knowledge vector and the response vector in step 2 are processed by the embedding layer Embedding technology. The dense vector form is expressed as follows: ; in, k It is given knowledge The dense vector mean of .
[0009] Furthermore, the response is calculated in step 4 Y With given knowledge K The correlation between Specifically: ; in, k It is given knowledge The dense vector mean of .
[0010] Further, in step 5, according to the correlation Calculating Adjusted Weights for Large Language Models Specifically: ; in, , is a constant smoothing factor.
[0011] Furthermore, in step 6, the weight is adjusted according to and the original probability distribution The specific calculation probability of each Token is: ; in, is the input of the base model, C For a conversation consisting of n rounds of questions , = C ; is the normalized exponential function.
[0012] Furthermore, in step 7, the final loss function is calculated based on the probability of each Token obtained in step 6 as follows: ; in, For reply Y Length, For reply Y Middle t Tokens, is the conditional probability function, that is, the input is And known under the conditions probability.
[0013] The beneficial effects of this application are: The method of this application improves the loss sensitivity of token positions that are highly relevant to given knowledge by locating learning focus, calculating adjustment weights, and modifying cross-entropy loss, so that the model pays more attention to these tokens during training and realizes knowledge constraints at the token level. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0015] Figure 1 A flowchart of a decoding training method for improving the effect of a dialogue model for this application; Figure 2 This is a schematic diagram of a decoding training method for improving the effect of a dialogue model in this application. DETAILED DESCRIPTION
[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.
[0017] This application provides corresponding solutions to the challenges of the prior art. For the first challenge mentioned above, the expected learning focus in the knowledge-driven dialogue task is first defined as a knowledge-aware token, and then the learning focus is located by calculating the correlation matrix. For the second challenge, a weight-related conversion method is used to provide dynamic token-level weights for the cross-entropy loss, and the corrected learning focus is used to guide model training. This method can widen the gap between knowledge-aware tokens and other tokens, thereby increasing the impact of changes in knowledge-aware tag loss on the final loss, thereby affecting the optimization direction and guiding the model to pay more attention to tokens that are more relevant to knowledge. The token refers to the basic unit that converts raw data such as text, images, and audio into a readable form for machine learning models.
[0018] In simple terms, this application solution uses multiple open source large models as base models for training. The base model outputs the probability distribution of the predicted value, and then locates the learning focus by calculating the correlation matrix, calculates the corresponding adjustment weights, adjusts the original predicted probability distribution to the final probability distribution, and finally trains the model to generate the next round of dialogue by minimizing the cross entropy loss.
[0019] The specific implementation methods of this application are as follows: like Figure 1 and Figure 2 As shown, the present application provides a decoding training method that can improve the effect of a dialogue model, including: Step 1: Data input: a conversation containing n rounds of questions , and n Knowledge related to wheel problems And the answers to the questions Input into the large language model, where For reply Y Length ，m Represents the amount of knowledge, Indicates knowledge, Reply Y Middle t Tokens; Step 2, data preprocessing: The embedding vector layer of the large language model preprocesses the input dialogue C, knowledge K and response Y, and outputs a question vector, a knowledge vector and a response vector respectively; In order to get (reply Y Middle t Tokens) and knowledge K In the representation of semantic space, this paper uses the embedding layer Embedding processing technology of the base model To get its dense vector representation: ; in, k It is given knowledge The dense vector mean of .
[0020] EMB (Embedding) is the process of mapping high-dimensional data (such as text, images, and audio) to a low-dimensional space in machine learning and natural language processing. An embedding vector is usually a vector of real numbers that represents the input data as a point in a continuous numerical space.
[0021] Step 3: Large language model training: Use the open source large model as the base model of the large language model for integrated learning and training. Take the dialogue C as the input of the large language model, and the large language model outputs the probability distribution of the predicted value. ,in For reply Y Middle t Tokens The corresponding predicted probability distribution; The large language models include but are not limited to Qwen, LIama, Gemma, Transformer and other open source large models.
[0022] Step 4. Calculate relevance: Calculate responses Y With given knowledge K The correlation between , To reply to Y t Tokens Dense vector representation of ; The tokens output by the model each time have a high semantic similarity with the given knowledge, and will eventually be used as new learning focus tokens. In order to adjust the learning focus tokens of the model, first reply Y Defined as a new learning focus, which is in line with the knowledge-based dialogue task. Then by replyingY The distance between the given knowledge and the semantic space is used to measure their relevance. The following is the calculation method of the relevance matrix: ; Compared with directly outputting knowledge-aware Tokens, this localization method uses the correlation matrix to provide more information for the subsequent adjustment of weights.
[0023] Step 5: Calculate the adjustment weight: Based on the correlation Calculating Adjusted Weights for Large Language Models ; In order to adaptively generate adjustment weights for each Token, while maximizing the loss of knowledge Tokens and making the overall loss basically the same as the original loss, the following formula is used to calculate the adjustment weights.
[0024] ; in, is a constant smoothing factor.
[0025] Step 6: Probability distribution adjustment: adjust the weights according to the and the original probability distribution Calculate the probability of each Token; After obtaining the adjusted weights, the original probability distribution output by the large language model in step 3 is scaled and the probability of each token is calculated. The adjusted probability distribution is calculated using the following formula, where is the input of the base model, C For a conversation consisting of n rounds of questions , = C ; is the normalized exponential function; ; Step 7, calculate the loss function: calculate the final loss function according to the probability of each Token obtained in step 6; After the probability distribution is adjusted, the final loss function is calculated as: ; in, For reply Y Length, For reply Y Middle t Tokens, is the conditional probability function, that is, the input is And known under the conditions probability.
[0026] The adjusted loss function changes the original loss distribution and shifts the learning focus to the knowledge token.
[0027] Step 8, model parameter update: follow the final loss function to update the parameters of the large language model, the parameters of the large language model include the number of neurons, embedding vector dimension and Transformer structure parameters.
[0028] Backward transfer loss function to the large language model and update model parameters.
[0029] The method of this application improves the loss sensitivity of token positions that are highly relevant to given knowledge by locating learning focus, calculating adjustment weights, and modifying cross-entropy loss, so that the model pays more attention to these tokens during training and realizes knowledge constraints at the token level.
[0030] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A decoding training method for improving the effect of a dialogue model, characterized in that: include: Step 1: Data input: a conversation containing n rounds of questions , and n Knowledge related to wheel problems And the answers to the questions Input into the large language model, where For reply Y Length ，m Represents the amount of knowledge, Indicates knowledge, Reply Y Middle t Tokens; Step 2: Data preprocessing: The embedding vector layer of the large language model is used to process the input dialogue. C ,Knowledge K and reply Y Perform preprocessing and output question vector, knowledge vector and response vector respectively; Step 3: Large language model training: Use the open source large model as the base model of the large language model for integrated learning and training. C As the input of the large language model, the large language model outputs the probability distribution of the predicted value ,in For reply Y Middle t Tokens The corresponding predicted probability distribution; Step 4. Calculate relevance: Calculate responses Y With given knowledge K The correlation between , For reply Y Middle t Tokens The dense vector representation of k It is given knowledge The dense vector mean of ; Step 5: Calculate the adjustment weight: Based on the correlation Calculating Adjusted Weights for Large Language Models ; Step 6: Probability distribution adjustment: adjust the weights according to the and the original probability distribution Calculate the probability of each Token; Step 7, calculate the loss function: calculate the final loss function according to the probability of each Token obtained in step 6; Step 8, model parameter update: follow the final loss function to update the parameters of the large language model, the parameters of the large language model include the number of neurons, embedding vector dimension and Transformer structure parameters.
2. A decoding training method for improving the effect of a dialogue model as claimed in claim 1, characterized in that: The knowledge vector and the response vector in step 2 are processed by the embedding layer Embedding technology The dense vector form is expressed as follows: ; in, k It is given knowledge The dense vector mean of .
3. A decoding training method for improving the effect of a dialogue model as claimed in claim 2, characterized in that: Calculate the response in step 4 Y With given knowledge K The correlation between Specifically: ; in, k It is given knowledge The dense vector mean of .
4. A decoding training method for improving the effect of a dialogue model as claimed in claim 3, characterized in that: In step 5, according to the correlation Calculating Adjusted Weights for Large Language Models Specifically: ; in, , is a constant smoothing factor.
5. A decoding training method for improving the effect of a dialogue model as claimed in claim 4, characterized in that: In step 6, the weight is adjusted according to and the original probability distribution The specific calculation probability of each Token is: ; in, is the input of the base model, C For a conversation consisting of n rounds of questions , = C ; is the normalized exponential function.
6. A decoding training method for improving the effect of a dialogue model as claimed in claim 5, characterized in that: In step 7, the final loss function is calculated based on the probability of each Token obtained in step 6: ; in, For reply Y Length, For reply Y Middle t Tokens, is the conditional probability function, that is, the input is And known under the conditions probability.