Language model training method, device and system based on adaptive cross entropy and medium

By employing an adaptive cross-entropy language model training method, utilizing an improved cross-entropy loss function and dynamically adjusting parameter T, the overfitting problem of the model is solved, thereby improving the model's generalization ability and performance in medical and financial scenarios.

CN119047324APending Publication Date: 2024-11-29PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411235412.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing technologies are prone to overfitting during model training in medical and financial scenarios, making it difficult to generalize on test data and resulting in a decline in model performance.

Method used

An adaptive cross-entropy language model training method is adopted. The loss value between the predicted response and the standard response is calculated by using an improved cross-entropy loss function. The preset parameter T is dynamically adjusted, the weight of the loss function is adjusted according to the diversity, and the model parameters are updated through backpropagation.

Benefits of technology

It alleviates the overfitting problem, improves the model's generalization ability in complex situations, and enhances the model's performance on specific tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047324A_ABST
    Figure CN119047324A_ABST
Patent Text Reader

Abstract

The invention discloses a language model training method, device and system based on adaptive cross entropy and a medium, and the method comprises the steps: obtaining input data of a user, forming a training data set according to the input data and a standard reply, and inputting the training data set into a large model; the large model obtains input data in the training data set and generates a prediction reply; presetting a preset parameter, and multiplying the preset parameter by the cross entropy loss function to form an improved cross entropy loss function; calculating a loss value between the predicted reply and the standard reply using an improved cross entropy loss function; and updating large model parameters according to the loss value, and finishing large model training when a preset number of iterations is reached. The improved cross entropy loss function is used to calculate the loss value of the prediction reply and the standard reply, the model parameters are updated to train the model, the over-fitting phenomenon of the model is relieved during large model training, and the capacity of the large model is improved through fine tuning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing and is applied to dialogue scenarios such as finance or medicine, and in particular to a method, device, system and medium based on adaptive cross-entropy language model training. Background Art

[0002] Existing large-scale supervised training of models usually uses the cross entropy loss function, i.e. supervised fine-tuning (SFT, machine learning technology). Fine-tuning is of great significance in medical and financial scenarios, such as physical examination report extraction, case understanding, knowledge question and answer, and patient management in medical scenarios; business consulting, knowledge retrieval question and answer, smart investment consulting and investment research, insurance technology and other tasks in financial scenarios. Fine-tuning can improve the accuracy and controllability of the model on the task. However, the current training method easily causes the model to fit to the prediction of simple tokens too early, and those tokens that are difficult to predict are not trained sufficiently, so they cannot be generalized well to the test data. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method, device, system and medium based on adaptive cross-entropy language model training that can be applied to data processing or other related fields, aiming to alleviate the overfitting phenomenon of the model during large model training and improve the capability of the large model through fine-tuning.

[0004] The technical solution of the present invention is as follows:

[0005] A method for training a language model based on adaptive cross entropy, the method comprising:

[0006] Obtaining user input data, forming a training data set according to the input data and a standard response, and inputting the training data set into a large model;

[0007] The large model obtains input data in the training data set and generates a predicted response;

[0008] Presetting a preset parameter, and multiplying the preset parameter by the cross entropy loss function to form an improved cross entropy loss function;

[0009] Calculating a loss value between the predicted response and the standard response using an improved cross entropy loss function;

[0010] The large model parameters are updated according to the loss value, and the large model training is completed when a preset number of iterations is reached.

[0011] In one embodiment, obtaining the user's input data, forming a training data set according to the input data and the standard response, and inputting the training data set into the big model includes:

[0012] Obtaining user input data, and preprocessing the input data;

[0013] The preprocessed input data is matched with the standard responses to generate a training data set.

[0014] In one embodiment, the large model obtains input data in the training data set and generates a predicted response, including:

[0015] The response probability of each word in the input data is calculated through the log-likelihood function;

[0016] A predicted response is generated based on the response probability.

[0017] In one embodiment, the calculating the loss value between the predicted response and the standard response using the improved cross entropy loss function includes:

[0018] Set preset parameters based on the diversity of response probabilities;

[0019] The preset parameters are multiplied by the cross entropy loss function to form an improved cross entropy loss function, and the improved cross entropy loss function formula is: Loss = ∑T*t i *log(pi / T);

[0020] Among them, T is the preset parameter, t i represents the value of the standard response, p i is the response probability.

[0021] In one embodiment, the step of setting the preset parameters according to the diversity of the reply probabilities includes:

[0022] The diversity of the reply probabilities is calculated, and the preset parameters are dynamically adjusted according to the diversity. The formula for calculating the diversity is:

[0023]

[0024] Where R is the total number of categories, p i is the reply probability and q is a fixed parameter.

[0025] In one embodiment, the using function to calculate the diversity of reply probabilities and dynamically adjusting the preset parameters according to the diversity include:

[0026] The preset parameters are calculated based on the diversity, and the formula is:

[0027] T = m*(-Diversity(P))+b;

[0028] Among them, T is a preset parameter, m and b are learnable parameters, and Diversity (P) is a measure of diversity. The lower the diversity, the larger the T value, and the higher the diversity, the smaller the T value.

[0029] In one embodiment, updating the large model parameters according to the loss value and reaching a preset number of iterations to complete the large model training includes:

[0030] Calculate the gradient according to the loss value, and update the model parameters through back propagation of the gradient;

[0031] The model parameters are adjusted through the optimization algorithm, and the preset number of iterations is reached to complete the large model training.

[0032] A training device based on an adaptive cross entropy language model, comprising:

[0033] A data collection module, used to obtain user input data, form a training data set according to the input data and standard responses, and input the training data set into the big model;

[0034] A response generation module, used for the large model to obtain input data in the training data set and generate a predicted response;

[0035] A function improvement module, used for presetting preset parameters, multiplying the preset parameters with the cross entropy loss function to form an improved cross entropy loss function;

[0036] A loss calculation module, used for calculating the loss value between the predicted response and the standard response using an improved cross entropy loss function;

[0037] The model training module is used to update the large model parameters according to the loss value and complete the large model training when a preset number of iterations is reached.

[0038] A system for training a language model based on adaptive cross entropy, the system comprising at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned adaptive cross-entropy language model training method.

[0041] A non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the above-mentioned adaptive cross-entropy language model training method.

[0042] Beneficial effect: The present invention discloses a method, device, system and medium based on adaptive cross-entropy language model training. Compared with the prior art, the embodiments of the present invention use an improved cross-entropy loss function to calculate the loss value of the predicted reply and the standard reply, update the model parameters to train the model, and achieve the goal of alleviating the overfitting phenomenon of the model when training a large model, and improving the capability of the large model through fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0044] Figure 1 A flow chart of a method for training a language model based on adaptive cross entropy provided by an embodiment of the present invention;

[0045] Figure 2 Another flow chart of the adaptive cross entropy language model training method provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of functional modules of a device for training an adaptive cross-entropy language model according to an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the hardware structure of the adaptive cross-entropy language model training system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0051] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0052] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] See also Figure 1 , Figure 1 A flowchart of an embodiment of the adaptive cross-entropy language model training method provided by the present invention. The adaptive cross-entropy language model training method provided in this embodiment is applied to a system consisting of a terminal device, a network and a server, wherein the network is a medium for directly providing a communication link between the terminal device and the server, which may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.; the operating system on the terminal device may include a handheld device operating system (iPhone operating system, iOS system), an Android system or other operating system, and the terminal device is connected to the server through the network to achieve interaction, thereby performing operations such as receiving or sending data, and specifically may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, portable computers and desktop servers, etc.

[0054] like Figure 1 As shown, Figure 1 A method for training a language model based on adaptive cross entropy is provided in an embodiment of the present invention. The method specifically comprises the following steps:

[0055] S100 obtains user input data, forms a training data set according to the input data and a standard response, and inputs the training data set into a large model;

[0056] S200 The large model obtains input data in the training data set and generates a predicted response;

[0057] S300 presets preset parameters, multiplies the preset parameters by the cross entropy loss function to form an improved cross entropy loss function

[0058] S400 uses an improved cross entropy loss function to calculate a loss value between the predicted response and the standard response;

[0059] S500 updates the large model parameters according to the loss value and completes the large model training when a preset number of iterations is reached.

[0060] In this embodiment, the user's question or input instruction Prompt is obtained, and the Prompt is paired with a standard reply to form a training data set. In the training stage, Prompt is the input text received by the model, which triggers the model to generate appropriate output. The standard reply is a high-quality reply created manually as part of the training data. These replies are used as "correct answers" or "standard answers" to evaluate and guide the generation quality of the model. The main goal of training a large model is to maximize the probability of generating a reply similar to the manually annotated reply R_g. That is, the model learns how to generate a reply as similar as possible to R_g based on a given Prompt. During the training process, the training data set is input into the large model, and the large model receives the input data in the training data, and uses the log-likelihood function to quantify the probability that the model generates a specific reply. This function calculates the sum of the logarithm of the model output probability and the product of the true label to measure the accuracy of the model prediction. The loss value of the predicted reply and the standard reply is calculated using the improved cross entropy loss function.

[0061] In large models, the cross entropy loss function is usually used to measure the difference between each token generated by the model and the corresponding token in the standard answer R_g. In order to improve the performance of the model, the loss function is adjusted by introducing adaptive parameters. This allows the model to pay more attention to predictions with greater uncertainty during training. By adjusting the loss function, the responses generated by the model have a certain diversity and avoid overfitting caused by excessive certainty. In the improved cross entropy loss function, parameters such as m, b, and T can be learnable and will be adjusted according to the performance of the model during training. The large model parameters are updated according to the loss value and the preset number of iterations is reached to complete the large model training. The training process includes forward propagation, calculating the loss, and updating the model parameters through backpropagation.

[0062] Through continuous iteration, the model learns how to generate text that better matches the manually annotated replies. The model's parameters are updated through the back-propagation algorithm according to the gradient calculated by the loss function. Use an optimizer (such as SGD stochastic gradient descent) to adjust the parameters with the goal of minimizing the loss function. Until the performance of the model on the validation set no longer improves significantly or reaches the predetermined number of iterations. Evaluate the performance of the model on an independent test set to ensure that the model is not overfitting. This method of adaptively adjusting the loss function allows the model to pay more attention to predictions with greater uncertainty during training, thereby improving the model's generalization ability for complex situations, especially in application scenarios that require high accuracy.

[0063] In one embodiment, obtaining the user's input data, forming a training data set according to the input data and the standard response, and inputting the training data set into the big model includes:

[0064] Obtaining user input data, and preprocessing the input data;

[0065] The preprocessed input data is matched with the standard responses to generate a training data set.

[0066] In this embodiment, relevant data is obtained according to the task of fine-tuning the large model. The data may include question-answer pairs, instruction-response pairs, or any form of dialogue data. First, the collected data is preprocessed, which includes cleaning the data, removing irrelevant and redundant information, correcting errors, and deleting duplicates. For text data, operations such as word segmentation, stop word removal, stem extraction, morphological restoration, and standardization are performed as needed. For parts of the data that do not have standard replies, manually annotated replies are performed. These manually annotated replies (denoted as R_g) are manually created high-quality replies that constitute part of the training data. . These replies are used as response pairs or any form of dialogue data) to ensure the accuracy and diversity of the replies. Pair the user's input or instruction (where Prompt is the input text received by the model, which triggers the model to generate appropriate output) with their respective standard replies to ensure that each Prompt has one or more correct standard replies.

[0067] Next, create a corresponding label for each Prompt, which can be an index or one-hot encoding of a standard reply. Format the processed data according to the requirements of the model input. For example, convert it to a format acceptable to the model (such as JSON, CSV, etc.) to form a training dataset, and save the processed data as a file for loading during model training. Check whether the Prompt and reply pairs in the dataset meet the requirements and ensure that there are no errors or inconsistencies. Divide the training dataset into training set, validation set, and test set. The usual ratio can be 70% training set, 15% validation set, and 15% test set. Then construct the input features according to the input requirements of the model. For example, for the Transformer model, it may be necessary to construct input IDs, attention masks, etc. Ensure that the model receives clean, consistent, and correctly formatted training data to improve the effect of model training and the performance of the final model.

[0068] Specifically, determine which type of label to use based on the task requirements. For example, classification tasks may use index labels, while some tasks may use one-hot encoding. In multi-classification problems, each category should be assigned a unique index as a label. If it is a multi-classification problem, each category can be assigned a unique index as a label. For example, in an intent recognition task, "greetings" may be index 0, "request" may be index 1, and so on. For each category, create a binary vector. In this vector, only the position representing the category is set to 1, and the rest of the positions are 0. For each category, create a binary vector in which all positions are 0 except for the position representing the category, which is 1. This encoding method is suitable for the output layer of the model, especially when the softmax activation function is used.

[0069] For sequence generation tasks, such as machine translation or text summarization, the standard response for each prompt can be converted into a sequence of token indexes. For each prompt, create a corresponding label based on its standard response, that is, convert the text into a numerical index or a one-hot encoded vector. And ensure that each prompt is strictly aligned with its corresponding label to facilitate the correct evaluation of the model's predictions during training. Organize the prompt and its corresponding label into a suitable data structure, such as a list, array, or data frame, as input during model training. Use an encoding method to convert the text into a numerical format that the model can understand. Use an encoding method to convert the text into a numerical format that the model can understand. For example, use word embedding or index mapping to convert words into vectors. In sequence tasks, use special start markers (such as <bos>, Begin OfSequence) and end markers (such as

[0070] <eos>, End Of Sequence) to indicate the start and end of a sequence. In a sequence task, use a special start (such as <bos>) and end (such as <eos>) markers to indicate the start and end of the sequence. When the sequences are of different lengths, pad or truncate the sequences to meet the model's input requirements. After creating the labels, verify the accuracy of the labels and make sure they correspond correctly to the prompts. Use these labels during model training and testing to evaluate model performance.

[0071] In one embodiment, the large model obtains input data in the training data set and generates a predicted response, including:

[0072] The response probability of each word in the input data is calculated through the log-likelihood function;

[0073] A predicted response is generated based on the response probability.

[0074] In this embodiment, the training data set generated after the above processing is input into the model, and the model receives the text data Prompt in the training data and converts the text into numerical embeddings, which can capture word meaning information. The embedding is input into the model, and the model may adopt a design based on the Transformer architecture, such as BERT, GPT, etc. The model uses a softmax layer or similar mechanism to calculate the probability distribution for each possible token in the sequence.

[0075] Specifically, the prepared training data set is input into the model and loaded into the model. The input data is processed according to the model input format. Each record in the training data set is processed according to the model input format, including but not limited to word segmentation, adding special tags (such as start tags, end tags), generating input IDs and attention masks, etc. The processed Prompt is fed into the model as input. The model will generate an initial hidden state based on these inputs. When a Prompt is given, the large model predicts the probability distribution of the next token at each time step. This response may be a sequence containing multiple tokens (such as words or characters). For each token in the response, the model outputs a probability distribution indicating the probability that the token is each possible token in the vocabulary.

[0076] The present invention calculates the reply probability of each word segment in the input data by using the log-likelihood function. The probability distribution of the first token is first predicted by the log-likelihood function. For each subsequent token in the sequence, the model uses the generated token and prompt as context to predict the probability distribution of the next token. Assuming that the words in this reply are w_1, w_2, ..., w_L, then before position i of a given sentence, the model predicts the probability of what word w_i is given the previous context and prompt.

[0077] P(w i |w1,w2,…,w i-1 ,Prompt), where w_i is the i-th token in the sequence. The log-likelihood function is:

[0078] Where L is the length of the reply and wi is the i-th token in the reply. The above log-likelihood function is equivalent to the prediction of each token during training. Then the log function in the log-likelihood function is used to convert the multiplication of probabilities into the accumulation of logarithms, and the log-likelihood of the entire sequence is accumulated to obtain the log-likelihood of the entire reply. Based on the calculated probability distribution, the model can generate a predicted reply. This is usually done by sampling or greedily selecting the token with the highest probability.

[0079] Among them, probability sampling is a random process. The model generates a probability distribution for each possible token and randomly selects the next one based on this distribution.

[0080] token. This approach enables the model to generate diverse text because it introduces randomness that allows it to jump out of the prediction of the highest probability token and explore other possibilities. Sampling is often used in tasks such as generating creative writing and conversations to increase the diversity and unpredictability of generated text. Greedy Selection is a method that selects the token with the highest probability as the next token. This method selects the most likely token at each step, so the generated text is usually more fluent and accurate. Greedy selection is often used in tasks that require accurate predictions, such as machine translation, summary generation, etc.

[0081] Furthermore, when the model generates a specific end tag, such as a period or a specific sequence end tag, the generation process is terminated. The generated text is post-processed, including removing redundant tags, grammar correction, etc., to improve readability. The quality of the generated predicted responses is evaluated, and the evaluation strategy can be automatic (such as through indicators such as perplexity) or manual (such as manual review). The large model in the present invention can generate coherent and relevant predicted responses based on the input Prompt, and improve its performance and accuracy through continuous training and optimization.

[0082] In one embodiment, the presetting of preset parameters and multiplying the preset parameters with the cross entropy loss function to form an improved cross entropy loss function include:

[0083] Set preset parameters based on the diversity of response probabilities;

[0084] The preset parameters are multiplied by the cross entropy loss function to form an improved cross entropy loss function, and the improved cross entropy loss function formula is: Loss = ∑T*t i *

[0085] log(pi / T); where T is the preset parameter, t i represents the value of the standard response, p i Yes 20 Complex probability.

[0086] In this embodiment, the loss function (Loss Function), also known as the cost function, is used to measure the gap between the model prediction value and the actual value. The main purpose of the loss function is to provide a quantitative indicator to guide the model to update parameters during training, thereby minimizing the prediction error of the model. The cross entropy loss function measures the difference between the probability distribution of the model output and the probability distribution of the true label. The improved loss function is used to evaluate the degree of match between the probability distribution predicted by the model and the true label. In the context of large models, the probability distribution usually refers to the model's predicted probability for each possible token (such as a word or character). The true label is the standard response given in the training data, and it is the goal that the model attempts to maximize by predicting the probability. The smaller the value of the calculated loss function, the higher the degree of match between the probability distribution predicted by the model and the true label, and the better the performance of the model.

[0087] The standard cross entropy loss function is: loss = ∑t i log(p i ), in order to prevent the model from concentrating the prediction on a certain token, thus causing overfitting. The adaptive parameter T is introduced into the standard cross entropy loss function formula. By adding an adjustable parameter T (T is generally greater than 1), the prediction distribution of the model is adjusted: loss = ∑T*t i *

[0088] log (pi / T), where T is a preset parameter used to adjust the weight of the loss function. pi is the response probability predicted by the model for label i. t i is a binary indicator, taking the value 0 or 1, representing the value of the standard response. If the i-th token is correct, it is 1; otherwise, it is 0. For each category i, if t i =1, then calculate the loss contribution log(p i ); if t i = 0, the loss contribution of this category is 0. Add the loss contributions of all categories to get the cross entropy loss of the entire sample. During the training process, the model parameters are adjusted to minimize the cross entropy loss so that the model's predicted probability p i As close as possible to the target label t i The cross entropy loss can be used to evaluate the "distance" between the model's predicted distribution and the true distribution. The smaller the loss, the more accurate the model's prediction.

[0089] Specifically, i is a binary variable (0 or 1) that indicates the true label. If category i is correct (i.e., the true category of the sample is category i), then t i will be set to 1. If category i is not correct (i.e., the true category of the sample is not category i), then t i will be set to 0. Therefore, t i The value of is not obtained after determining whether the category i is correct, but it directly represents the correct category. When constructing the cross entropy loss function, t i is known, and it represents the true label information.

[0090] Let's take a specific example to illustrate the classification problem with 3 categories. The true label of a sample is category 2. Then for this sample (t1), (t2) and (t3) the values ​​will be (t1 = 0) (because category 1 is not correct), (t2 = 1) (because category 2 is correct), (t3 = 0) (because category 3 is not correct). When calculating the cross entropy loss, only (t2) will contribute to the loss function because only it corresponds to the true label. Therefore, the loss function will be: loss = -(t1log(p1) + t2log(p2) + t3log(p3)) = -(0log(p1) +

[0091] 1log(p2)+0log(p3))=-log(p2). So only p2 (the probability of the true class) will be used to calculate the loss, because only t2 is 1. i If it is 0, it will not affect the loss function.

[0092] In one embodiment, the step of setting the preset parameters according to the diversity of the reply probabilities includes:

[0093] The diversity of the reply probabilities is calculated, and the preset parameters are dynamically adjusted according to the diversity. The formula for calculating the diversity is: Where R is the total number of categories, p i is the reply probability and q is a fixed parameter.

[0094] In this embodiment, usually, the preset parameter T is a fixed artificial setting value. This requires adjusting the parameter during the model training process, thereby increasing the workload. For different predicted distributions of different tokens, the fixed parameter T may be detrimental to model learning. Generally, a distribution with a larger uncertainty should be assigned a smaller parameter, while a distribution with a smaller uncertainty should be assigned a larger parameter.

[0095] Therefore, this solution uses a function to calculate the diversity of reply probabilities and dynamically adjusts the parameter T according to the diversity. Diversity (P) measures the discreteness of the probability distribution. The higher the diversity, the more uncertain the prediction; the lower the diversity, the more concentrated the prediction. The diversity measurement formula is: Where R is the total number of categories, p i is the predicted probability of category i, and q is a fixed parameter, which is set to 2 in this solution. The preset parameter T is dynamically adjusted according to the calculated diversity Diversity(P). A function f can be designed to map Diversity(P) to the value of T. Through this adaptive adjustment mechanism, the model can dynamically adjust the weights in the loss function according to the uncertainty of the prediction, which helps to improve the performance and generalization ability of the model on specific tasks.

[0096] Furthermore, Diversity(P) is used to measure the diversity and dispersion of the probability distribution P. If Diversity(P) is high, it means that the probability distribution is relatively uniform and the uncertainty is large; if Diversity(P) is low, it means that the probability distribution is concentrated and the uncertainty is small. In this way, the model can adaptively adjust the loss contribution of each token, so that the model can learn better during the training process.

[0097] Compared with the traditional cross entropy loss function, the improved function reduces each log(p i ). When the model is overconfident about certain categories, it can reduce the impact of these overconfident predictions on the total loss, prevent the model from being overconfident about certain categories during training, and thus improve the model's generalization ability on unseen data. The preset parameter T is a dynamically adjusted parameter used to control the weights in the loss function and affect the model's prediction probabilities for different categories. The selection of the preset parameter T needs to be adjusted according to the characteristics of the specific problem and data set, and the function is used to calculate the diversity of the reply probability. The preset parameter is dynamically adjusted according to the diversity to ensure that the loss function can effectively guide model training.

[0098] In one embodiment, the using function to calculate the diversity of reply probabilities and dynamically adjusting the preset parameters according to the diversity include:

[0099] The preset parameters are calculated according to the diversity, and the formula is: T=m*(-Diversity(P))+b; wherein T is the preset parameter, m and b are learnable parameters, and Diversity(P) is a measure of diversity. If the diversity is lower, the T value is larger, and if the diversity is higher, the T value is smaller.

[0100] In this embodiment, T is used as a regulating factor to adjust the weight of each category in the cross entropy loss function. By dynamically adjusting T, the model can pay more attention to predictions with higher uncertainty, thereby improving generalization ability. The present invention calculates preset parameters based on diversity, and the formula is: T = m*(-Diversity(P)) + b, where T is a preset parameter, m and b are learnable parameters, and Diversity(P) represents diversity. M and b are learnable parameters of the model. The learnable parameters m and b are updated together with the training of the model during the training process, and m and b determine how T responds to the diversity of probability distributions. M can be regarded as the sensitivity of controlling T to respond to the diversity of probability distributions, and b is the basic value of T. T is inversely proportional to Diversity(P), that is, when the diversity of probability distribution P is higher, the T value is smaller, and when the diversity of probability distribution P is lower, the T value is larger. During the training process, as the model parameters are updated, the probability distribution P will also change, thereby affecting the values ​​of Diversity(P) and T. The adaptive adjustment of T helps the model to automatically balance the degree of attention to each category during the training process.

[0101] In one embodiment, updating the large model parameters according to the loss value and reaching a preset number of iterations to complete the large model training includes:

[0102] Calculate the gradient according to the loss value, and update the model parameters through back propagation of the gradient;

[0103] The model parameters are adjusted through the optimization algorithm, and the preset number of iterations is reached to complete the large model training.

[0104] In this embodiment, in each training iteration, forward propagation is performed first. The input data is passed between the layers of the model until a predicted output is generated. A loss function (such as cross entropy loss or other custom loss) is used to calculate the difference between the model's predicted output and the true label to obtain a loss value. According to the loss function, the gradient of the loss value relative to the model parameters is calculated. Based on the calculated gradient, it is reversely passed back to the input layer starting from the output layer of the model, and the gradient information of each layer is updated along the way. Use optimization algorithms (such as SGD (Stochastic Gradient Descent), Adam (Adaptive Moment Estimation), RMSprop (Root Mean Square Propagation), etc.) to update the parameters of the model according to the gradient information and the preset learning rate.

[0105] Repeat the above steps until the performance of the model on the validation set no longer improves or reaches the preset number of iterations. During the training process, hyperparameters such as learning rate and optimizer parameters need to be adjusted according to the performance of the model. If the performance of the model on the validation set does not improve after multiple consecutive iterations, an early stopping strategy can be used to avoid overfitting. Regularly evaluate the performance of the model on the validation set and the test set to ensure that the model has good generalization ability. During or after the training process, save the parameters of the model for subsequent testing or application. Through these steps, large models can be trained effectively and the performance of the model can be improved through continuous iteration and optimization.

[0106] Specifically, SGD uses only one sample (or a small batch of samples) to calculate the gradient in each iteration, rather than the entire training set. Each iteration may use different samples, so it is called "stochastic gradient descent". Through each sample or small batch of samples, the gradient of the loss function relative to the model parameters is calculated. Use the calculated gradient to update the parameters of the model. Adam combines the advantages of the Momentum and RMSprop algorithms to optimize the model by adaptively adjusting the learning rate of each parameter. Adam uses an exponentially weighted moving average to calculate the first and second moments of the gradient, and updates the parameters based on the estimates of the first and second moments of the gradient. RMSprop uses an exponentially weighted average to calculate the exponentially decaying average of the squared gradient of each parameter, and updates the parameters by the exponentially decaying average of the squared gradient.

[0107] like Figure 2 As shown, Figure 2 is another flow chart of a method for training a language model based on adaptive cross entropy provided by an embodiment of the present invention, the method specifically comprising the following steps:

[0108] (1) Prepare fine-tuning data;

[0109] (2) Model training: Input fine-tuning data, calculate the loss value according to the improved cross entropy loss function, and update the model parameters;

[0110] (3) Deploy the application.

[0111] In this embodiment, the user's input Prompt and the corresponding reply pair (correct answer or expected output) are collected to construct a fine-tuning data set. The data is cleaned and preprocessed, such as word segmentation, noise removal, etc. The Prompt is input into the model, and the model generates a reply based on the input Prompt. In the fine-tuning stage, an improved cross-entropy loss function is used to measure the difference between the reply generated by the model and the annotated reply. The improved cross-entropy loss function evaluates the degree of match between the probability distribution predicted by the model and the true label. The parameters of the model are updated by the back-propagation algorithm according to the gradient calculated by the loss function. The optimizer is used to adjust the parameters in order to minimize the loss function. This scheme proposes an improved cross-entropy loss function for large model training. The overfitting phenomenon of the model can be alleviated during training, thereby avoiding the loss of the model in training because of premature fitting on simple tokens and no longer optimizing on difficult tokens, thereby maximizing the ability of the large model through fine-tuning and improving the average performance of the deployed model on various tasks.

[0112] Another embodiment of the present invention provides a device for training a language model based on an adaptive cross entropy. The device for training a language model based on an adaptive cross entropy is used to perform any embodiment of the aforementioned adaptive cross entropy model training method. Specifically, refer to Figure 3 , Figure 3 A schematic diagram of functional modules of an adaptive cross-entropy language model training device provided in an embodiment of the present invention. The adaptive cross-entropy language model training device can be configured in a management server.

[0113] like Figure 3 As shown, a training device based on an adaptive cross entropy language model includes:

[0114] A data collection module 11 is used to obtain input data from users, form a training data set according to the input data and standard responses, and input the training data set into a large model;

[0115] A response generation module 12, used for the large model to obtain input data in the training data set and generate a predicted response;

[0116] A function improvement module 13, used to pre-set preset parameters, and multiply the preset parameters by the cross entropy loss function to form an improved cross entropy loss function;

[0117] A loss calculation module 14, configured to calculate a loss value between the predicted response and the standard response using an improved cross entropy loss function;

[0118] The model training module 15 is used to update the large model parameters according to the loss value and complete the large model training when a preset number of iterations is reached.

[0119] like Figure 3 As shown, the data collection module 11, the reply generation module 12, the function improvement module 13, the loss calculation module 14 and the model training module 15 are connected in sequence. The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the adaptive cross-entropy language model training method than the program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0120] In one embodiment, the data collection module 11 includes:

[0121] A data processing unit, used to obtain input data from a user and pre-process the input data;

[0122] The data set unit is used to match the preprocessed input data with the standard response to generate a training data set.

[0123] In one embodiment, the reply generation module 12 includes:

[0124] A reply probability unit, used to calculate the reply probability of each word in the input data through a log-likelihood function;

[0125] The prediction reply unit is used to generate a prediction reply according to the reply probability.

[0126] In one embodiment, the function improvement module 13 includes:

[0127] Set preset parameters based on the diversity of response probabilities;

[0128] The preset parameters are multiplied by the cross entropy loss function to form an improved cross entropy loss function, and the improved cross entropy loss function formula is: Loss = ∑T*t i *

[0129] log(pi / T); where T is the preset parameter, t i represents the value of the standard response, p i is the response probability.

[0130] In one embodiment, the step of setting the preset parameters according to the diversity of the reply probabilities includes:

[0131] The diversity of the reply probabilities is calculated, and the preset parameters are dynamically adjusted according to the diversity. The formula for calculating the diversity is: Where R is the total number of categories, p i is the reply probability and q is a fixed parameter.

[0132] In one embodiment, the using function to calculate the diversity of reply probabilities and dynamically adjusting the preset parameters according to the diversity include:

[0133] The preset parameters are calculated according to the diversity, and the formula is: T=m*(-Diversity(P))+b; wherein T is the preset parameter, m and b are learnable parameters, and Diversity(P) is a measure of diversity. If the diversity is lower, the T value is larger, and if the diversity is higher, the T value is smaller.

[0134] In one embodiment, the model training module 14 includes:

[0135] A parameter updating unit, used to calculate the gradient according to the loss value, and update the model parameters by back propagation of the gradient;

[0136] The model training unit is used for model training to adjust model parameters through optimization algorithms and reach a preset number of iterations to complete large model training.

[0137] Unit, used in another embodiment of the present invention to provide a system for training a language model based on adaptive cross entropy, such as Figure 4 As shown, the system 10 includes:

[0138] One or more processors 110 and memory 120, Figure 4 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0139] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0140] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the adaptive cross-entropy language model training method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implementing the adaptive cross-entropy language model training method in the above method embodiment.

[0141] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0143] In one embodiment, obtaining the user's input data, forming a training data set according to the input data and the standard response, and inputting the training data set into the big model includes:

[0144] Obtaining user input data, and preprocessing the input data;

[0145] The preprocessed input data is matched with the standard responses to generate a training data set.

[0146] In one embodiment, the large model obtains input data in the training data set and generates a predicted response, including:

[0147] The response probability of each word in the input data is calculated through the log-likelihood function;

[0148] A predicted response is generated based on the response probability.

[0149] In one embodiment, the calculating the loss value between the predicted response and the standard response using the improved cross entropy loss function includes:

[0150] Set preset parameters based on the diversity of response probabilities;

[0151] The preset parameters are multiplied by the cross entropy loss function to form an improved cross entropy loss function, and the improved cross entropy loss function formula is: Loss = ∑T*t i *log(pi / T);

[0152] Among them, T is the preset parameter, t i represents the value of the standard response, p i is the response probability.

[0153] In one embodiment, the step of setting the preset parameters according to the diversity of the reply probabilities includes:

[0154] The diversity of the reply probabilities is calculated, and the preset parameters are dynamically adjusted according to the diversity. The formula for calculating the diversity is:

[0155] Where R is the total number of categories, p i is the reply probability and q is a fixed parameter.

[0156] In one embodiment, the using function to calculate the diversity of reply probabilities and dynamically adjusting the preset parameters according to the diversity include:

[0157] The preset parameters are calculated according to the diversity, and the formula is: T=m*(-Diversity(P))+b; wherein T is the preset parameter, m and b are learnable parameters, and Diversity(P) is a measure of diversity. If the diversity is lower, the T value is larger, and if the diversity is higher, the T value is smaller.

[0158] In one embodiment, updating the large model parameters according to the loss value and reaching a preset number of iterations to complete the large model training includes:

[0159] Calculate the gradient according to the loss value, and update the model parameters through back propagation of the gradient;

[0160] The model parameters are adjusted through the optimization algorithm, and the preset number of iterations is reached to complete the large model training.

[0161] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions, which are executed by one or more processors, wherein the computer program implements the following steps when executed by the processor: obtaining user input data, forming a training data set based on the input data and standard replies, and inputting the training data set into a large model; the large model obtains the input data in the training data set and generates a predicted reply; uses an improved cross-entropy loss function to calculate the loss value between the predicted reply and the standard reply; updates the large model parameters based on the loss value, and completes the large model training when a preset number of iterations is reached.

[0162] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0163] In summary, the method, device, system and medium based on adaptive cross-entropy language model training disclosed in the present invention include: obtaining user input data, forming a training data set according to the input data and standard replies, and inputting the training data set into a large model; the large model obtains input data in the training data set to generate a predicted reply; pre-setting preset parameters, multiplying the preset parameters with the cross-entropy loss function to form an improved cross-entropy loss function; using the improved cross-entropy loss function to calculate the loss value between the predicted reply and the standard reply; updating the large model parameters according to the loss value, and completing the large model training when the preset number of iterations is reached. The present invention uses the improved cross-entropy loss function to calculate the loss value of the predicted reply and the standard reply, updates the model parameters to train the model, and achieves the alleviation of the overfitting phenomenon of the model during the large model training, and improves the ability of the large model through fine-tuning.

[0164] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0165] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, and units with the same function may be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0166] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the specification described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk.

[0168] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0169] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.< / eos> < / bos> < / eos> < / bos>

Claims

1. A language model training method based on adaptive cross entropy, It is characterized in that include: Obtaining user input data, forming a training data set according to the input data and a standard response, and inputting the training data set into a large model; The large model obtains input data in the training data set and generates a predicted response; Presetting a preset parameter, and multiplying the preset parameter by the cross entropy loss function to form an improved cross entropy loss function; Calculating a loss value between the predicted response and the standard response using an improved cross entropy loss function; The large model parameters are updated according to the loss value, and the large model training is completed when a preset number of iterations is reached.

2. The method for training a language model based on adaptive cross entropy according to claim 1, It is characterized in that The step of obtaining input data from a user, forming a training data set according to the input data and a standard response, and inputting the training data set into a large model includes: Obtaining user input data, and preprocessing the input data; The preprocessed input data is matched with the standard responses to generate a training data set.

3. The method for training a language model based on adaptive cross entropy according to claim 1, It is characterized in that The large model obtains input data in the training data set and generates a predicted response, including: The response probability of each word in the input data is calculated through the log-likelihood function; A predicted response is generated based on the response probability.

4. The method for training a language model based on adaptive cross entropy according to claim 3, It is characterized in that The presetting of the preset parameters and multiplying the preset parameters with the cross entropy loss function to form an improved cross entropy loss function include: Set preset parameters based on the diversity of response probabilities; The preset parameters are multiplied by the cross entropy loss function to form an improved cross entropy loss function, and the improved cross entropy loss function formula is: Loss=∑T*t i *log(p i / T); Among them, T is the preset parameter, t i represents the value of the standard response, p i is the probability of reply.

5. The method for training a language model based on adaptive cross entropy according to claim 4, It is characterized in that The step of setting the preset parameters according to the diversity of the reply probabilities includes: The diversity of the reply probabilities is calculated, and the preset parameters are dynamically adjusted according to the diversity. The formula for calculating the diversity is: Where R is the total number of categories, p i is the reply probability and q is a fixed parameter.

6. The method for training a language model based on adaptive cross entropy according to claim 5, It is characterized in that The calculating of the diversity of the reply probability and dynamically adjusting the preset parameters according to the diversity include: The preset parameters are calculated based on the diversity, and the formula is: T = m*(-Diversity(P))+b; Among them, T is a preset parameter, m and b are learnable parameters, and Diversity (P) is a measure of diversity. The lower the diversity, the larger the T value, and the higher the diversity, the smaller the T value.

7. The method for training a language model based on adaptive cross entropy according to claim 1, It is characterized in that The updating of the large model parameters according to the loss value and reaching a preset number of iterations to complete the large model training includes: Calculate the gradient according to the loss value, and update the model parameters through back propagation of the gradient; The model parameters are adjusted through the optimization algorithm, and the preset number of iterations is reached to complete the large model training.

8. A language model training device based on adaptive cross entropy, It is characterized in that include: A data collection module, used to obtain user input data, form a training data set according to the input data and standard responses, and input the training data set into the big model; A response generation module, used for the large model to obtain input data in the training data set and generate a predicted response; A function improvement module, used to pre-set preset parameters, and multiply the preset parameters by the cross entropy loss function to form an improved cross entropy loss function; A loss calculation module, used for calculating the loss value between the predicted response and the standard response using an improved cross entropy loss function; The model training module is used to update the large model parameters according to the loss value and complete the large model training when a preset number of iterations is reached.

9. A language model training system based on adaptive cross entropy, It is characterized in that The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the adaptive cross-entropy language model training method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, It is characterized in that The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the adaptive cross-entropy language model training method described in any one of claims 1-7.

Citation Information

Cited By

  • Model training method, method for mental health detection, terminal and readable storage medium

    CN119623556A

  • Model training method, method for mental health detection, terminal and readable storage medium

    CN119623556B

  • Training method and device of voice large model, equipment and medium

    CN121438813A

  • Voice large model training method and device, equipment and medium

    CN121438813B

  • Dialogue generation method and device based on improved DPO training and readable medium

    CN121503645A