Large language model training method, reply generation method and related device

Abstract

The invention discloses a large language model training method, a reply generation method and a related device, and the method comprises the steps: processing a current training sample in a training set through a large language model, and obtaining a text prediction result; based on the text prediction result, at least one loss item is obtained through calculation, the at least one loss item comprises at least one of a mutual information enhancement loss item and an entropy calibration loss item, and the mutual information enhancement loss item is used for encouraging a big language model to improve the relevancy between the text prediction result and a current training sample; the entropy calibration loss item is used for calibrating uncertainty estimation of the large language model; and adjusting parameters of the large language model based on the at least one loss item. In this way, knowledge illusion of the large language model can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a large language model training method, a reply generation method and related devices. BACKGROUND

[0002] The rapid evolution of large language models greatly enhances their ability in language generation, understanding and complex reasoning, and thus triggers a profound change in the paradigm of human-computer interaction. However, accompanied by this powerful ability is a serious challenge - model hallucination, which refers to the phenomenon that the text generated by the model, although showing high rationality in grammar and fluency, contains factual errors, internal logical contradictions or lack of realistic basis. And how to reduce the large language model hallucination is a technical problem to be solved. SUMMARY

[0003] The technical problem solved by the present application is to provide a large language model training method, a reply generation method and related devices, which can reduce the large language model hallucination.

[0004] To solve the above technical problem, one technical solution adopted by the present application is to provide a large language model training method, which comprises: processing a current training sample in a training set by using a large language model to obtain a text prediction result; based on the text prediction result, at least one loss term is calculated, wherein the at least one loss term includes at least one of a mutual information enhancement loss term and an entropy calibration loss term, the mutual information enhancement loss term is used to encourage the large language model to improve the correlation between the text prediction result and the current training sample, and the entropy calibration loss term is used to calibrate the estimation of uncertainty by the large language model; and based on the at least one loss term, the parameters of the large language model are adjusted.

[0005] To solve the above technical problem, another technical solution adopted by the present application is to provide a reply generation method, which comprises: obtaining user input data; and generating a reply text based on the user input data by using a large language model; wherein the large language model is trained by using the above method.

[0006] To solve the above technical problems, another technical solution adopted by the present application is to provide a large language model training device, which comprises a text prediction module, a loss determination module and a parameter adjustment module. The text prediction module is configured to process a current training sample in a training set by using a large language model to obtain a text prediction result. The loss determination module is configured to calculate at least one loss term based on the text prediction result, wherein the at least one loss term comprises at least one of a mutual information enhancement loss term and an entropy calibration loss term. The mutual information enhancement loss term is used to encourage the large language model to improve the correlation between the text prediction result and the current training sample. The entropy calibration loss term is used to calibrate the estimation of uncertainty of the large language model. The parameter adjustment module is configured to adjust the parameters of the large language model based on the at least one loss term.

[0007] To solve the above technical problems, another technical solution adopted by the present application is to provide a reply generation device, which comprises a data acquisition module and a reply module. The data acquisition module is configured to acquire user input data. The reply module is configured to generate a reply text based on the user input data by using a large language model. The large language model is trained by using the above method.

[0008] To solve the above technical problems, another technical solution adopted by the present application is to provide an electronic device comprising a memory and a processor. The memory is configured to store program instructions. The processor is configured to execute the program instructions to implement the above method.

[0009] To solve the above technical problems, another technical solution adopted by the present application is to provide a computer readable storage medium, which is configured to store program instructions. The program instructions can be executed to implement the above method.

[0010] The above scheme is based on at least one loss term for model parameter adjustment during the training of the large language model. The at least one loss term comprises at least one of a mutual information enhancement loss term and an entropy calibration loss term. The mutual information enhancement loss term is used to encourage the large language model to improve the correlation between the text prediction result and the current training sample. The entropy calibration loss term is used to calibrate the estimation of uncertainty of the large language model. It can be seen that the present application can improve the correlation between the text prediction result and the input content, i.e. the model can select effective knowledge for text prediction, or the knowledge learned by the model does not exceed the knowledge distribution of the original data, so that the content output by the model conforms to the real basis, thereby reducing the knowledge illusion of the model. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of an embodiment of the large language model training method provided by the present application; Figure 2is a flowchart of an embodiment of the reply generation method provided in the present application. Figure 3 is a framework diagram of an embodiment of the large language model training apparatus provided in the present application. Figure 4 is a framework diagram of an embodiment of the reply generation apparatus provided in the present application. Figure 5 is a framework diagram of an embodiment of the electronic device provided in the present application. Figure 6 is a framework diagram of the computer-readable storage medium provided in the present application. DETAILED DESCRIPTION

[0012] To make the objectives, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the accompanying drawings and embodiments.

[0013] In addition, if the present application embodiments involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection claimed by the present application.

[0014] It should be noted that the large language model training method provided in the present application is a training method that can alleviate model hallucination.

[0015] To facilitate understanding of the present application scheme, the background content and knowledge theory related to the present application scheme are briefly described first.

[0016] First, the background content related to the present application scheme is described as follows: Model knowledge hallucination refers to the text generated by the model, although it shows high rationality in grammar and fluency, contains factual errors, internal logical contradictions or lacks realistic basis.

[0017] During their research, the inventors discovered that the root of this problem can be attributed to the fact that large language models, at their core, are probabilistic text generators trained on massive corpora. Their primary goal is to predict the most likely word sequence, rather than achieving true cognition and understanding. Existing training and evaluation paradigms, to some extent, incentivize models to produce superficially coherent but not necessarily truthful outputs. This leads the models, when faced with their knowledge boundaries, to tend to fabricate seemingly reasonable but factually unsupported answers, rather than proactively acknowledging their cognitive limitations.

[0018] Several solutions exist to address the model illusion problem. At the data level, data-centric approaches such as data cleaning and filtering, and the introduction of external structured knowledge bases, are used to improve the quality and factual accuracy of training data from the source. At the model level, alignment techniques such as instruction fine-tuning and reinforcement learning based on human feedback (RLHF) are commonly employed to optimize the model's generative behavior. Simultaneously, uncertainty estimation mechanisms are introduced, enabling the model to quantify and identify its own cognitive boundaries, thus choosing to express questionable content rather than forcibly generating potentially fictitious content when uncertainty is high.

[0019] For example, in the inference phase, Retrieval-Augmented Generation (RAG) significantly reduces the risk of model illusion in open-domain question-answering scenarios by introducing credible external knowledge sources as the basis for generation. Meanwhile, prompting techniques such as Chain-of-Thought enhance the transparency and interpretability of the generation logic by explicitly illustrating the model's reasoning process, thereby helping researchers identify potential fallacies in the reasoning chain. Furthermore, methods such as fact-checking, post-editing, and model self-criticism and iterative correction further validate and calibrate the model after content generation. In summary, mitigating the large model illusion requires integrating strategies across multiple stages, including data, model, inference, and post-processing, to construct a defense-in-depth architecture.

[0020] Nevertheless, existing methods still face several key challenges: From the perspective of model training, current common model training methods are relatively coarse-grained data alignment methods, lacking fine-grained alignment schemes at the knowledge level. This leads to models mimicking data distribution patterns, easily resulting in uncontrollable knowledge illusion risks. As for uncertainty estimation techniques for large models, they are far from mature. Letting the model judge its own "uncertainty" is inherently uncertain. Turning our attention to interventions in the more widely applied reasoning stage, the situation becomes even more complex. The effectiveness of Retrieval Augmentation (RAG) depends entirely on the accuracy of the retrieval system and the quality of the knowledge base. Once irrelevant or erroneous sources are retrieved, RAG not only fails to correct the illusion but also "anchors" it to a seemingly authoritative but incorrect basis, fostering a more subtle and deceptive illusion. Seemingly sophisticated prompting strategies such as thought chains and self-criticism come at a heavy cost in terms of computational overhead and response latency, and the reasoning process they guide may itself be flawed. Post-processing fact-checking, due to limitations in coverage, timeliness, and automation, is difficult to scale. In summary, most existing solutions merely patch the surface of the model without truly impacting its core as a probability generator, thus failing to endow the model with genuine understanding and reasoning capabilities. This "band-aid" nature, coupled with the system complexity, high costs, and performance degradation resulting from stacking multiple technologies, reveals the deep-seated dilemma currently facing illusion mitigation techniques.

[0021] Therefore, in order to fundamentally alleviate the problem of knowledge illusion in large models, this application proposes a training method for large language models, which alleviates the knowledge illusion in large models through model training itself.

[0022] Secondly, the relevant theoretical knowledge for this application is explained as follows: During model training, it's crucial to learn domain-specific effective knowledge from the full range of knowledge contained in the data to mitigate the risk of illusion. Therefore, if a portion of information (X) significantly reduces our uncertainty about another portion (Y), the correlation between X and Y constitutes effective knowledge. For example, in finance, if knowing that "a company's core product's quarterly revenue growth rate far exceeds market expectations" (information X) significantly reduces our uncertainty about "whether the company's stock price will rise sharply in the coming week" (information Y), then this strongly correlated information constitutes high-value information. The core of effective knowledge measurement is precisely quantifying the reduction in uncertainty resulting from the injection of key information. A key idea behind model illusion reduction is enabling the model to learn the ability to select effective knowledge from the total amount of knowledge. Therefore, from an information theory perspective, the quantification of effective knowledge is closely related to the total information content, the amount of related knowledge, and the amount of predictive knowledge.

[0023] For the total information capacity, the training data includes a knowledge metric, representing data richness. Specifically: Shannon entropy measures the uncertainty of a random variable, and the principle of entropy increase can be used to quantify the richness of knowledge. The higher the entropy, the more unpredictable the variable's value, and the greater the average amount of information contained in the data. Taking a financial scenario as an example, for a discrete variable X (e.g., a stock code) in a financial dataset, its entropy H(X) is calculated as follows:

[0024] Where X represents any token in the data (such as security code, industry category, credit rating, etc.); x is all the tokens that can be obtained within the specific value space of X; p(x) represents the frequency of this value in the entire dataset; ∑ represents the summation of all possible values ​​x of X.

[0025] In financial scenarios, the entropy of key tokens can be used to quantify their information content and uncertainty. For example, H (security code) can be used to measure the diversification of assets in a portfolio; H (industry category) reflects the level of diversification of industries covered by investments; and H (credit rating) characterizes the complexity and uncertainty of credit risk distribution. The higher the entropy value, the richer the information contained in the token.

[0026] Therefore, it is understandable that entropy is used to measure the average uncertainty of data. The higher the entropy value, the greater the uncertainty and the more information it can carry; the lower the entropy value, the stronger the certainty and the less information it can carry.

[0027] For related knowledge quantity, the aim is to directly measure the strength of the intrinsic association between different variables (tokens), thus providing a method to measure the effective knowledge learned by the model. Its core metric is mutual information. Mutual information I(X;Y) assesses the degree to which the uncertainty of the training variable X is reduced given that the model is aligned with variable Y (and vice versa). Its mathematical definition is expressed as I(X;Y) = H(X) - H(X|Y), or I(X;Y) = H(Y) - H(Y|X), which can also be equivalently expressed as:

[0028] To define the total amount of effective knowledge, a threshold τ can be set, and only variable pairs with mutual information greater than τ are considered to contain strong knowledge. Based on this, the total amount of associated knowledge in the dataset can be defined as:

[0029] in, The total amount of related knowledge can effectively measure the overall strength and density of the entire financial knowledge network.

[0030] In the process of measuring model knowledge, the most valuable knowledge is often reflected in its predictive power for key future variables. Predictive knowledge aims to quantify the joint predictive power of a continuous token sequence X={X1,X2,…,Xn} (such as macroeconomic indicators, market technical factors, and company fundamental data) for a target variable Y (such as quarterly stock price returns or credit default status). Its mathematical definition is as follows:

[0031] This refers to the difference between the initial uncertainty H(Y) of Y and the residual uncertainty H(Y|X) after the known token sequence. This value directly quantifies the total amount of knowledge that all relevant token features in the dataset can contribute to a specific financial prediction task (such as asset return prediction or extreme risk warning), and is an important indicator for evaluating the value of data information. Distance metrics can be used during the optimization process to measure the amount of predictive knowledge.

[0032] The training method for the large language model provided in this application is described below: See Figure 1 , Figure 1 This is a flowchart illustrating one implementation method of the large language model training method provided in this application. The method includes: S11: Use a large language model to process the current training samples in the training set to obtain text prediction results.

[0033] Among them, data from the corresponding scenario can be selected as training samples in the training set according to the application scenario of the large language model, and then the model can be trained to learn knowledge in the corresponding domain using the corresponding training set.

[0034] In a specific implementation scenario, the training samples in the training set are financial data, and the trained large language model can learn financial knowledge based on the training set.

[0035] The text prediction result is the content generated by processing the current training samples using a large language model.

[0036] The text prediction results include the answers generated by the large language model based on the training samples (including word sequences composed of several word units), and of course, the probability distribution of several word units may also be included.

[0037] S12: Based on the text prediction results, at least one loss term is calculated, wherein the at least one loss term includes at least one of a mutual information enhancement loss term and an entropy calibration loss term. The mutual information enhancement loss term is used to encourage the large language model to improve the correlation between the text prediction results and the current training samples, and the entropy calibration loss term is used to calibrate the large language model's estimation of uncertainty.

[0038] First, the mutual information enhanced loss term in step S12 is introduced as follows: It should be noted that in order to reduce the knowledge hallucination of the large language model, not only does the large language model need to generate the correct answer, but the answer generated by the large language model also needs to be highly relevant and informative to the question, rather than allowing the model to give a general and templated answer that is correct but lacks effective knowledge.

[0039] To achieve this goal, during the training of the large language model, the model parameters need to be adjusted in the direction of reducing the difference between the text prediction result and the annotated text, so that the model can generate the correct answer, and the model parameters need to be adjusted in the direction of increasing the information content of the text prediction result, so that the model can generate an answer that is highly relevant and informative to the question.

[0040] Therefore, in one embodiment, the first sub-loss term for quantifying the difference between the text prediction result and the annotated text, and the second sub-loss term for quantifying the information content of the text prediction result can be used as the mutual information enhanced loss term.

[0041] Among them, for the quantification of the difference between the text prediction result and the annotated text, it can be quantified by the mean square error, mean absolute error, cross-entropy loss, etc. That is, the first sub-loss term can be the mean square error, mean absolute error, cross-entropy loss, etc.

[0042] In a specific embodiment, the first sub-loss term is the cross-entropy loss term. Among them, the text prediction result includes the probability distribution of several tokens; the calculation steps of the cross-entropy loss term include: obtaining the sum of the logarithms of the probability distributions of each token as the first sum, and taking the negative of the first sum as the cross-entropy loss term.

[0043] Among them, the calculation of the cross-entropy loss term can refer to the following formula:

[0044] In the formula, y<t represents all the tokens before the t-th token among several tokens, X represents the current training sample, Pθ(… ) represents the probability distribution of the next token predicted by the large language model (parameters are θ), and ∑ represents the cumulative sum of the logarithms of the probability distributions of all tokens among several tokens (that is, the first sum).

[0045] Among them, by training the large language model with this cross-entropy loss term, it can be ensured that the generated content is consistent with the annotated text (the annotated text corresponding to the training sample) at the token level, which is the basis for ensuring knowledge accuracy. During the model training process, minimizing the cross-entropy loss term is equivalent to minimizing the KL divergence between the model prediction distribution and the real data distribution. From the perspective of "predictive knowledge", this is also equivalent to minimizing the conditional entropy H of the model for the target Y (text prediction result)θ (Y|X). In other words, it aims to minimize the uncertainty of the model regarding Y after observing X.

[0046] Regarding the quantification of the information content of text prediction results, any relevant method can be used to quantify the information content of the model-generated response (text prediction result).

[0047] As described above regarding the background and knowledge theory, entropy can be used to quantify the richness of information (i.e., the amount of information it carries). Therefore, in one embodiment, a second sub-loss term can be determined based on the entropy of the output of the large language model to quantify the information content of the text prediction result. Here, the output of the large model refers to the answer generated by the large model; furthermore, the higher the entropy of the output of the large model, the richer the information content of the text prediction result (e.g., containing more specific, diverse, and detailed information).

[0048] Therefore, in this embodiment, to encourage the model to output more informative answers, it is necessary to train the model to output high-entropy answers during the training process. Encouraging the model to generate high-entropy answers actually penalizes the model for making empty promises without providing specific information in order to avoid confusion, thereby indirectly suppressing avoidance illusions.

[0049] It should be noted that, in order to adjust the model parameters in the direction of reducing the loss, the negative of the entropy of the output of the large language model can be used as the second sub-loss term, that is, the second sub-loss term is negatively correlated with the entropy of the output of the large language model.

[0050] For example, the calculation of the second sub-loss term can be referenced to the following formula:

[0051] Here, Ygen represents the output of the large language model (i.e., the generated complete answer). The entropy representing the output of a large language model. This represents the second sub-loss term.

[0052] Furthermore, the text prediction result includes the probability distribution of several word elements, which includes the probability of each word element being a value. The calculation steps of the second sub-loss term include: for each word element, obtaining the second sum of the product of the probabilities of each word element being a value, taking the negative of the second sum of each word element and adding them together to obtain the third sum, taking the negative of the third sum as the second sub-loss term, wherein the probability product is the product of the probability of the word element being the corresponding value and its logarithm.

[0053] The calculation steps for the second sub-loss term can be understood by referring to the following formula:

[0054] in, It represents the second sum of the probabilities of each word element as a product of the probabilities of each value (w, which does not represent a number but indicates that it represents different content).

[0055] This represents the third sum obtained by adding the negative values ​​of the second sums of each word element together. It represents the product of the probabilities of a word being a certain value.

[0056] For example, the text prediction results output by the model include the text sequence "The capital of Anhui is Hefei". When the model currently outputs "yes", the next predicted word may be "Hefei", "Nanjing", "Zhengzhou", etc., where "Hefei", "Nanjing", and "Zhengzhou" are the values ​​of the next predicted word.

[0057] Additionally, it should be noted that, as described above regarding the background and knowledge theory, the correlation between two variables X and Y can be represented by mutual information. Therefore, in order to encourage the large language model to improve the correlation between the text prediction results and the current training samples during the model training process, it is necessary to maximize the correlation / relevance between the model input and the model-generated answer (text prediction results) during the model training process, that is, to maximize the mutual information I(X;Y).

[0058] As can be seen from the above, the calculation formula is: I(X;Y)=H(Y)-H(Y|X).

[0059] Here, H(Y) represents the entropy of the model's generated answers, which measures the diversity or richness of information in the model's generated answers; H(Y|X) is the conditional entropy, which measures the predictability / uncertainty of the model's answer Y given input X.

[0060] It should be noted that maximizing I(X;Y) is equivalent to maximizing H(Y) and minimizing H(Y|X).

[0061] Furthermore, as described above regarding the cross-entropy loss term, "minimizing the cross-entropy loss term is equivalent to minimizing the conditional entropy H of the model for the target Y (text prediction result)." θ As can be seen from (Y|X), H(Y|X) can be represented by the cross-entropy loss term mentioned above, while H(Y) actually represents the entropy of the model generating the answer.

[0062] Therefore, in this embodiment, the first sub-loss term in the mutual information enhancement loss term is the aforementioned cross-entropy loss term, and the second sub-loss term is -H(Y).

[0063] Secondly, the entropy calibration loss term in step S12 is described below: In one embodiment, the entropy calibration loss term is determined based on the predicted conditional entropy and the true conditional entropy of the large language model under the current training samples, and is used to quantify the difference between the predicted conditional entropy and the true conditional entropy.

[0064] In one implementation, the entropy calibration loss term can be determined using the following method: First, obtain the true conditional entropy under the current training sample conditions. The true conditional entropy is determined by statistically analyzing the frequency of each labeled text corresponding to the associated samples of the current training sample. The associated samples include the first other training samples in the training set that are the same as the current training sample, and / or the second other training samples that meet the preset similarity requirements with the current training sample.

[0065] For example, for the current training sample, multiple sets of semantically most similar sample pairs (i.e., the associated samples of the current training sample, which contain multiple sample questions and the target answers corresponding to each sample question) are retrieved from the training set in advance. Then, semantic clustering is performed on the target answers (annotated text) in the sample pair set, and the frequency of occurrence of target answers of different semantic categories is counted to obtain the true posterior probability distribution under the conditions of the current training sample. Then, the true conditional entropy under the conditions of the current training sample is calculated using the Shannon entropy formula.

[0066] Second, based on the text prediction results of the current training samples, the prediction conditional entropy under the current training sample conditions is calculated.

[0067] The text prediction results of the current training sample include the probability distribution of several word elements, and the probability distribution of each word element includes the probability of that word element being any of its respective values.

[0068] In one embodiment, the step of obtaining the prediction conditional entropy includes: for each word, obtaining the product of the probabilities of the word as each value, then obtaining the sum of the product of the probabilities of the word as each value (called the sum of probability products), taking the negative of each sum of probability products, adding them together to obtain a total, and then taking the average of the sums corresponding to several words as the prediction conditional entropy under the current training sample conditions.

[0069] The probability product is the product of the probability that a word is the corresponding value and its logarithm.

[0070] It should be noted that in some implementation scenarios, when the model generates the text prediction result (a text sequence containing several words) for the current training sample, it generates each word sequentially. The generation of each word is based on the word generated in the previous step and the current training sample. When generating a word at each step, the model outputs an unnormalized vector, and then applies the softmax function to it to obtain its probability distribution. The Shannon entropy of the probability distribution of the output word at each step can be calculated. Then, the single-step entropy of all words in the generated text sequence is accumulated and averaged to obtain the prediction conditional entropy of the current training sample.

[0071] Third, obtain the mean square error between the true conditional entropy and the predicted conditional entropy, and use it as the entropy calibration loss term.

[0072] For example, the entropy calibration loss term is represented as follows:

[0073] in, Represents the predictive conditional entropy. This represents the true conditional entropy.

[0074] It should be noted that training a model using entropy calibration loss term aims to ensure that the conditional entropy predicted by the model can fit the true conditional entropy in the data. In this way, when there is uncertainty in the data itself, the model can output a flat probability distribution to calibrate the large language model's estimation of uncertainty, rather than making a confident guess.

[0075] Before model training, statistical analysis can be performed on the training dataset or knowledge in related fields (such as finance) to calculate the diversity of the corresponding answers Y in the dataset for similar contexts (Prompt X), thereby estimating the true conditional entropy.

[0076] As can be seen from the above, the true conditional entropy is statistically derived from real data and is an inherent property of the training dataset; while the predictive conditional entropy is generated by the model of the learning data. The degree of fit between the two directly reflects the model's ability to model the uncertainty of the data.

[0077] In the real dataset, the more diverse the Y (indicator annotation text) corresponding to similar X, the higher the uncertainty of the data itself. In this case, the higher the true conditional entropy of the dataset (the flatter the probability distribution), the better. Therefore, by training the model (adjusting the model parameters) by calibrating the loss term to represent the distance between the predicted conditional entropy and the true conditional entropy, when the data itself is uncertain, the trained model can output a flat probability distribution instead of making a confident guess. This can reduce the knowledge illusion of large models.

[0078] S13: Adjust the parameters of the large language model based on at least one loss term.

[0079] In this embodiment, at least one loss term can be weighted and fused to obtain a total loss, and then the parameters of the large language model can be adjusted based on the total loss.

[0080] In one specific embodiment, the mutual information enhancement loss term includes the aforementioned first sub-loss term and second sub-loss term. A weighted fusion of at least one loss term to obtain the total loss includes: taking the sum of the products of the first sub-loss term, the second sub-loss term, and the first weight, and the product of the entropy calibration loss term and the second weight as the total loss, wherein the first weight and the second weight are training hyperparameters.

[0081] For example, the total loss can be determined using the following formula:

[0082] In the formula, This represents the first sub-loss term in the mutual information enhancement term (specifically, the cross-entropy loss term mentioned above). This loss term is used in supervised fine-tuning, where the model learns knowledge from the original data to ensure that the model's output conforms to the factual basis and the true data distribution.

[0083] This represents the second sub-loss term in the mutual information enhancement term (specifically, the negative of the entropy output by the model above). This loss term penalizes uninformative answers, encourages rich and highly relevant responses, and improves the effective transfer of knowledge.

[0084] This represents the entropy calibration loss term. This loss term is responsible for matching the model's confidence level with the certainty of knowledge. When the real knowledge is uncertain, the model also exhibits "uncertainty," outputting a flat probability distribution, thus fundamentally suppressing the illusion of "confidently talking nonsense."

[0085] , This represents the training hyperparameters.

[0086] In this embodiment, compared to the conventional approach of adjusting model parameters by only using cross-entropy loss as the optimization objective, this embodiment uses mutual information enhancement term and entropy calibration loss term as the optimization objectives to adjust model parameters. This approach can train the model to improve the relevance between the output text and the input content, and can also train the model to output a flat probability distribution when the data itself is uncertain, thereby reducing knowledge illusion.

[0087] The above-described scheme, during the training of the large language model, adjusts the model parameters based on at least one loss term. This at least one loss term includes at least one of a mutual information enhancement loss term and an entropy calibration loss term. The mutual information enhancement loss term encourages the large language model to improve the relevance between the text prediction results and the current training samples, while the entropy calibration loss term calibrates the large language model's estimation of uncertainty. Therefore, the method in this application can improve the relevance between the text prediction results and the input content. That is, the model can select effective knowledge for text prediction, or ensure that the knowledge learned by the model does not exceed the knowledge distribution of the original data. Thus, the content output by the model conforms to the true basis, reducing the model's knowledge illusion.

[0088] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the response generation method provided in this application. This embodiment includes: S21: Obtain user input data.

[0089] User input data can be text, speech, or images. When the user input data is not text, it needs to be recognized or converted to obtain the corresponding text, which is then used as new user input data.

[0090] S22: Utilize a large language model to generate response text based on user input data.

[0091] In this embodiment, the large language model is trained using the model training method described above.

[0092] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the large language model training device provided in this application. In this embodiment, the large language model training device 30 includes: a text prediction module 31, a loss determination module 32, and a parameter adjustment module 33. The text prediction module 31 is used to process the current training samples in the training set using the large language model to obtain text prediction results; the loss determination module 32 is used to calculate at least one loss term based on the text prediction results, wherein the at least one loss term includes at least one of a mutual information enhancement loss term and an entropy calibration loss term, the mutual information enhancement loss term is used to encourage the large language model to improve the correlation between the text prediction results and the current training samples, and the entropy calibration loss term is used to calibrate the large language model's estimation of uncertainty; the parameter adjustment module 33 is used to adjust the parameters of the large language model based on at least one loss term.

[0093] In some embodiments, the mutual information enhancement loss term includes a first sub-loss term and a second sub-loss term. The first sub-loss term is used to quantify the difference between the text prediction result and the labeled text, and the second sub-loss term is used to quantify the information content of the text prediction result. The second sub-loss term is negatively correlated with the information content of the text prediction result.

[0094] In some embodiments, the first sub-loss term is the cross-entropy loss term; and / or, the second sub-loss term is determined based on the entropy of the output of the large language model, and the second sub-loss term is negatively correlated with the entropy.

[0095] In some embodiments, the text prediction result includes the probability distribution of several word elements; the calculation steps of the cross-entropy loss term include: obtaining the sum of the logarithms of the probability distributions of each word element as a first sum, and taking the negative of the first sum as the cross-entropy loss term; and / or, the text prediction result includes the probability distribution of several word elements, the probability distribution of each word element including the probability of each word element being a value; the calculation steps of the second sub-loss term include: for each word element, obtaining the second sum of the product of the probabilities of each word element being a value, taking the negative of the second sum of each word element and adding them together to obtain a third sum, taking the negative of the third sum as the second sub-loss term, wherein the probability product is the product of the probability of the word element being a value and its logarithm.

[0096] In some embodiments, the entropy calibration loss term is determined based on the predicted conditional entropy and the true conditional entropy of the large language model under the current training samples, and is used to quantify the difference between the predicted conditional entropy and the true conditional entropy.

[0097] In some embodiments, the calculation steps of the entropy calibration loss term include: obtaining the true conditional entropy under the current training sample conditions, wherein the true conditional entropy is determined by statistically analyzing the frequency of each labeled text corresponding to the associated samples of the current training sample, wherein the associated samples include a first other training sample in the training set that is the same as the current training sample, and / or a second other training sample that meets a preset similarity requirement with the current training sample; inputting the associated samples of the current training sample into the large language model multiple times to obtain the text prediction results output by the large language model multiple times, and calculating the prediction conditional entropy under the current training sample conditions based on the multiple text prediction results; obtaining the mean square error between the true conditional entropy and the prediction conditional entropy as the entropy calibration loss term.

[0098] In some embodiments, adjusting the parameters of a large language model based on at least one loss term includes: performing weighted fusion on at least one loss term to obtain a total loss; and adjusting the parameters of the large language model based on the total loss.

[0099] In some embodiments, the mutual information enhancement loss term includes a first sub-loss term and a second sub-loss term; weighted fusion of at least one loss term to obtain a total loss includes: taking the sum of the product of the first sub-loss term, the second sub-loss term and the first weight, and the product of the entropy calibration loss term and the second weight as the total loss, wherein the first weight and the second weight are training hyperparameters.

[0100] In some embodiments, the training samples in the training set are financial data, and the trained large language model can learn financial knowledge based on the training set.

[0101] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the response generation device provided in this application. In this embodiment, the response generation device 40 includes: a data acquisition module 41 and a response module 42. The data acquisition module 41 is used to acquire user input data; the response module 42 is used to generate response text based on the user input data using a large language model; wherein, the large language model is trained using the method described above.

[0102] Please see Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the electronic device provided in this application. In this embodiment, the electronic device 50 includes a memory 51 and a processor 52 coupled to each other.

[0103] The memory 51 stores program instructions, and the processor 52 executes the program instructions stored in the memory 51 to implement the steps of any of the above-described method implementations. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0104] Specifically, processor 52 controls itself and memory 51 to implement the steps of any of the above embodiments. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 52 may be implemented using integrated circuit chips.

[0105] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework of the computer-readable storage medium provided in this application. The computer-readable storage medium 60 of this application embodiment stores program instructions 61, which, when executed, implement the methods provided in any embodiment or any non-conflicting combination of the above-described methods. The program instructions 61 can form a program file and be stored in the computer-readable storage medium 60 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 60 includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0106] The above-described scheme, during the training of the large language model, adjusts the model parameters based on at least one loss term. This at least one loss term includes at least one of a mutual information enhancement loss term and an entropy calibration loss term. The mutual information enhancement loss term encourages the large language model to improve the relevance between the text prediction results and the current training samples, while the entropy calibration loss term calibrates the large language model's estimation of uncertainty. Therefore, the method in this application can improve the relevance between the text prediction results and the input content. That is, the model can select effective knowledge for text prediction, or ensure that the knowledge learned by the model does not exceed the knowledge distribution of the original data. Thus, the content output by the model conforms to the true basis, reducing the model's knowledge illusion.

[0107] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for training a large language model, characterized in that, The method includes: The current training samples in the training set are processed using a large language model to obtain text prediction results; Based on the text prediction results, at least one loss term is calculated, wherein the at least one loss term includes at least one of a mutual information enhancement loss term and an entropy calibration loss term. The mutual information enhancement loss term is used to encourage the large language model to improve the correlation between the text prediction results and the current training samples, and the entropy calibration loss term is used to calibrate the large language model's estimation of uncertainty. The parameters of the large language model are adjusted based on the at least one loss term.

2. The method according to claim 1, characterized in that, The mutual information enhancement loss term includes a first sub-loss term and a second sub-loss term. The first sub-loss term is used to quantify the difference between the text prediction result and the labeled text, and the second sub-loss term is used to quantify the information content of the text prediction result. The second sub-loss term is negatively correlated with the information content of the text prediction result.

3. The method according to claim 2, characterized in that, The first sub-loss term is a cross-entropy loss term; and / or, the second sub-loss term is determined based on the entropy of the output of the large language model, and the second sub-loss term is negatively correlated with the entropy.

4. The method according to claim 3, characterized in that, The text prediction result includes the probability distribution of several word units; the calculation steps of the cross-entropy loss term include: The sum of the logarithms of the probability distributions of each of the aforementioned words is obtained as the first sum, and the negative of the first sum is taken as the cross-entropy loss term; And / or, the text prediction result includes a probability distribution of several word elements, the probability distribution of the word elements including the probability of each word element being a specific value; the calculation steps of the second sub-loss term include: For each word, obtain the second sum of the product of the probabilities of the word being each value, take the negative of the second sum of each word and add them together to obtain the third sum, take the negative of the third sum as the second sub-loss term, wherein the probability product is the product of the probability of the word being the value and its logarithm.

5. The method according to claim 1, characterized in that, The entropy calibration loss term is determined based on the predicted conditional entropy and the true conditional entropy of the large language model under the current training sample conditions, and is used to quantify the difference between the predicted conditional entropy and the true conditional entropy.

6. The method according to claim 5, characterized in that, The calculation steps for the entropy calibration loss term include: Obtain the true conditional entropy under the current training sample conditions, wherein the true conditional entropy is determined by statistically analyzing the frequency of each labeled text corresponding to the associated samples of the current training sample, wherein the associated samples include a first other training sample in the training set that is the same as the current training sample, and / or a second other training sample that meets a preset similarity requirement with the current training sample; Based on the text prediction results, the prediction conditional entropy under the current training sample conditions is calculated; The mean square error between the true conditional entropy and the predicted conditional entropy is obtained and used as the entropy calibration loss term.

7. The method according to claim 1, characterized in that, Adjusting the parameters of the large language model based on the at least one loss term includes: The total loss is obtained by weighting and fusing the at least one loss term; Based on the total loss, the parameters of the large language model are adjusted.

8. The method according to claim 7, characterized in that, The mutual information enhancement loss term includes a first sub-loss term and a second sub-loss term; the weighted fusion of the at least one loss term to obtain the total loss includes: The total loss is the sum of the product of the first sub-loss term, the second sub-loss term and the first weight, and the product of the entropy calibration loss term and the second weight, where the first weight and the second weight are training hyperparameters.

9. The method according to claim 1, characterized in that, The training samples in the training set are financial data, and the trained large language model can learn financial knowledge based on the training set.

10. A response generation method, characterized in that, include: Obtain user input data; Based on the user input data, a response text is generated using a large language model; The large language model is trained using the method described in any one of claims 1-9.

11. A large language model training device, characterized in that, The training device includes: The text prediction module is used to process the current training samples in the training set using a large language model to obtain text prediction results. The loss determination module is used to calculate at least one loss term based on the text prediction result, wherein the at least one loss term includes at least one of a mutual information enhancement loss term and an entropy calibration loss term, wherein the mutual information enhancement loss term is used to encourage the large language model to improve the correlation between the text prediction result and the current training sample, and the entropy calibration loss term is used to calibrate the large language model's estimation of uncertainty; A parameter adjustment module is used to adjust the parameters of the large language model based on the at least one loss term.

12. A response generation device, characterized in that, The generating apparatus includes: The data acquisition module is used to acquire user input data; The response module is used to generate response text based on the user input data using a large language model; The large language model is trained using the method described in any one of claims 1-9.

13. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store program instructions, and the processor being used to execute the program instructions to implement the method as described in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the method as described in any one of claims 1-10.

Images