A language model optimization method based on autoregressive training and related equipment

Abstract

The present application provides a language model optimization method based on autoregressive training and related equipment. The method includes: constructing an autoregressive training framework for the language model, predicting the probability distribution of subsequent words based on the historical context of the input, wherein the probability distribution in each generation of autoregressive training is used to generate the next generation of synthetic data; in each generation of training, mixing real data with synthetic data generated by the previous generation model according to a set ratio as training data, wherein the proportion of real data dynamically decreases as the number of training generations increases; using the mixed data to train the language model, updating the model parameters, and generating the synthetic data required for the next generation of training; after each generation of training, measuring the difference between the data distribution generated by the current model and the real data distribution, and dynamically adjusting the proportion of real data based on the difference; and continuously optimizing the autoregressive training process of the language model through the above steps. This effectively suppresses the distribution drift problem generated by the language model during multiple rounds of autoregressive training.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a language model optimization method based on autoregressive training and related equipment. Background Art

[0002] In recent years, with the widespread adoption of generative pre-trained models (GPT) and other large-scale language models (LLMs), the proportion of synthetic data generated by these models in training corpora has gradually increased. In particular, in natural language processing tasks such as text generation, machine translation, and conversational systems, language models continuously generate new data and incorporate it into the training corpus, forming a train-generate loop.

[0003] However, this cyclical process introduces the potential for performance degradation, known as "model collapse." Model collapse occurs when, during auto-regressive training, the model output gradually deviates from the true data distribution, ultimately losing the diversity or accuracy of the generated language. For example, after many iterations, the model may tend to generate limited, repetitive outputs, reflecting the loss of information at the tail of the original distribution, severely impacting the model's practical application.

[0004] To address this issue, existing techniques have attempted to mitigate the negative impact of autoregressive training to a certain extent by introducing a mixed training strategy of real and synthetic data. For example, they quantitatively described the impact of increasing the proportion of synthetic data on model performance. By introducing a certain proportion of real data into each iteration of autoregressive training, they mitigated distribution shift. However, research results show that when the proportion of synthetic data exceeds a certain threshold, the model still inevitably crashes.

[0005] Existing technologies provide a certain theoretical basis and experimental analysis for understanding the model collapse phenomenon, but there are still many limitations in how to effectively avoid the model collapse problem caused by autoregressive training, making it difficult to fundamentally solve the model collapse problem in autoregressive training. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present application provides a language model optimization method based on autoregressive training and related equipment, which at least solves the problem of serious model output distribution drift and prone distribution collapse in existing autoregressive training, which in turn leads to model degradation.

[0007] In order to achieve the above objectives and other advantages, some embodiments of the present application provide the following aspects:

[0008] In a first aspect, some embodiments of the present application provide a language model optimization method based on autoregressive training, comprising:

[0009] Constructing an autoregressive training framework for a language model that predicts the probability distribution of subsequent words based on the historical context of the input, where the probability distribution in each generation of autoregressive training is used to generate synthetic data for the next generation;

[0010] In each generation of training, real data is mixed with synthetic data generated by the previous generation model in a set ratio as training data, where the proportion of real data in the training data dynamically decreases as the number of training generations increases;

[0011] Using the mixed data to train a language model, update model parameters, and generate synthetic data required for next-generation training, wherein the amount of the synthetic data relative to the amount of the real data is limited to not exceed a set synthetic data ratio control condition;

[0012] After each generation of training, the difference between the data distribution generated by the current model and the real data distribution is measured, and the proportion of the real data is dynamically adjusted according to the difference;

[0013] The autoregressive training process of the language model is continuously optimized based on synthetic data generated by probability distribution, dynamic data ratio control, and synthetic data quantity constraints.

[0014] In a second aspect, some embodiments of the present application further provide an electronic device, comprising:

[0015] One or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor performs any of the above-described language model optimization methods based on autoregressive training.

[0016] In a third aspect, some embodiments of the present application further provide a computer-readable storage medium having stored thereon a computer program and / or instructions, which, when executed by a processor, implements a language model optimization method based on autoregressive training as described above.

[0017] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program and / or instructions, which, when executed by a processor, implements a language model optimization method based on autoregressive training as described above.

[0018] Compared to related technologies, the solution provided in the embodiments of this application achieves adaptive data balancing at different training stages by introducing a control strategy that dynamically adjusts the ratio of real data to synthetic data based on the number of training generations and sets an upper limit on the amount of synthetic data in each generation of training. This effectively reduces the risk of synthetic data excessively affecting model training. This can effectively suppress the distribution drift problem that occurs in the language model during multiple rounds of autoregressive training, prevent the language model output distribution from abnormally shifting or degenerating into a single category output, ensure the controllability of the model training process and the performance reliability of the final model, and significantly improve the stability and robustness of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 1 is a flow chart of a language model optimization method based on autoregressive training provided in an embodiment of the present application;

[0021] Figure 2 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] First embodiment

[0024] The first embodiment of the present application relates to a language model optimization method based on autoregressive training, referring to Figure 1 As shown, the method may include the following steps:

[0025] Step S1: Construct an autoregressive training framework for the language model to predict the probability distribution of subsequent words based on the historical context of the input, where the probability distribution in each generation of autoregressive training is used to generate synthetic data for the next generation.

[0026] Specifically, regarding step S1, a language model is a core foundational model in natural language processing. It is used to model the probabilistic relationship between words or word sequences within a context. Its goal is to predict the probability distribution of the next word given the current context. Common language models include RNN language models such as LSTM and GRU, and language models based on self-attention mechanisms such as Transformer.

[0027] Autoregressive training decomposes a sequence prediction task into multiple, chronologically ordered prediction subtasks. At each step, the model uses historically known context to predict the next word and uses the current output as part of the input for the next step. In this framework, the model predicts the probability of the next word based on historical context and uses the prediction as input for the next step, continuously iterating through training.

[0028] In each generation of training, after being trained on real data and synthetic data, the model will form an understanding of language distribution. This understanding is reflected in the probability distribution generated by the model, that is, each context input will correspond to a probability distribution of candidate words. These probability distributions can be used to sample and generate the next generation of synthetic data, compare it with the real distribution, perform drift measurement analysis, and thus control the risk of model collapse.

[0029] By building a language model with an autoregressive structure, we can achieve generation-by-generation iteration and optimization of generation probability during language model training, providing support for subsequent synthetic data generation.

[0030] Step S2: In each generation of training, real data is mixed with synthetic data generated by the previous generation model in a set ratio as training data, where the proportion of real data in the training data dynamically decreases as the number of training generations increases.

[0031] Regarding step S2, specifically, during each generation of model training, the statically collected real corpus data is mixed with the synthetic text data generated by the previous generation language model in a preset ratio to form the training data input for this generation. The proportion of real data is denoted as α m , this proportion decreases dynamically with the increase of training generations m, gradually guiding the language model to transition from the training stage based on real data to the self-supervised generation training stage based on synthetic data.

[0032] By dynamically adjusting the proportion of real data in mixed data, the distribution drift problem during training can be effectively alleviated, and the stability and robustness of model training as well as the diversity and generalization ability of the final generated results can be improved.

[0033] Step S3: Use the mixed data to train the language model, update the model parameters, and generate the synthetic data required for the next generation of training, wherein the amount of synthetic data relative to the amount of real data is limited to not exceed the set synthetic data ratio control condition.

[0034] Specifically, in step S3, during each training generation, the training data constructed from the hybrid in step S2 is fed into the language model, and the model training process based on the autoregressive structure is executed. The model parameters are updated based on the optimization objective function (e.g., cross-entropy loss). After the current generation of training is completed, the updated model is used to generate several target word sequences using the autoregressive prediction mechanism by inputting the historical context sequence. These serve as the synthetic data generated by the current generation of the model.

[0035] To prevent an excessive proportion of synthetic data in the training process, which could cause the model generation results to deviate from the true corpus distribution or even cause distribution collapse, an upper limit on the amount of synthetic data is required to form a control condition for the proportion of synthetic data. Specifically, based on the total number of real data samples used in the current training and the size of the vocabulary, the upper limit of the allowable synthetic data generation is dynamically calculated. This controls the impact of synthetic data on the training process, avoids problems such as model distribution drift or output degradation caused by an excessive proportion of synthetic data, and ensures the stability of the training process and the effectiveness of the generated data.

[0036] Step S4: After each generation of training, measure the difference between the data distribution generated by the current model and the real data distribution, and dynamically adjust the proportion of the real data based on the difference.

[0037] Specifically, for step S3, distribution difference refers to the statistical deviation between the model-generated data distribution and the true data distribution in the current generation. Distribution difference can be measured using the L1 norm. By introducing a distribution difference measurement and feedback adjustment mechanism based on the L1 norm, the fit between the model's generation capability and the data distribution can be continuously monitored during model training. If the generated distribution deviates too much from the true distribution, the system automatically triggers the true data proportion adjustment mechanism, increasing the true data proportion for correction, guiding training back to a reasonable distribution range, thereby preventing the model from falling into a state of output degradation or collapse.

[0038] Step S5: Based on the synthetic data generated by the probability distribution, dynamic data ratio control, and synthetic data quantity constraints, the autoregressive training process of the language model is continuously optimized.

[0039] Specifically, regarding step S5, after completing each sub-process of each generation of training, the system continuously generates new synthetic data based on the conditional probability distribution generated by the current generation model. Simultaneously, a real data ratio control strategy based on the number of training generations is used to adaptively adjust the mixing ratio between real and synthetic data. Combined with the synthetic data ratio control condition, an upper limit is placed on the total amount of synthetic data that can be used in each generation of training to prevent the risk of distribution collapse during model training. This mechanism forms a recursively optimized training loop, enabling the model to transition from real data-driven to synthetic data-led in a stable and controllable manner during each generation of training. This improves model training efficiency while ensuring the stability and generalization of the model's output distribution.

[0040] It is not difficult to find that compared with related technologies, the solution provided by the embodiment of the present application achieves adaptive data balance of the model at different training stages by introducing a control strategy that dynamically adjusts the ratio between real data and synthetic data based on the number of training generations and sets an upper limit on the amount of synthetic data in each generation of training, effectively reducing the risk of excessive impact of synthetic data on model training. This can effectively suppress the distribution drift problem of the language model during multiple rounds of autoregressive training, prevent the language model output distribution from abnormally shifting or degenerating into a single category output, ensure the controllability of the model training process and the performance reliability of the final model, and significantly improve the stability and robustness of model training.

[0041] Second embodiment

[0042] The second embodiment of this application relates to a language model optimization method based on autoregressive training. The second embodiment is an improvement on the first embodiment. The specific improvement is that: in the second embodiment of this application, a specific implementation method for dynamically adjusting the proportion of real data based on the number of training generations is provided. That is, the process of dynamically reducing the proportion of real data in the training data as the number of training generations increases is as follows:

[0043] As the number of training generations increases, the proportion of real data decreases according to a preset decreasing strategy; the specific decreasing strategy is: the proportion of real data in the initial training stage is close to 1, and as the number of training generations increases, the proportion of real data gradually decreases in a manner inversely proportional to the logarithmic function of the training generations, and never exceeds 1. In addition, the rate of decrease in the proportion of real data is determined by control parameters set according to the scale of training data and the quality of generated data.

[0044] Specifically, in each generation of training, the training data is composed of a mixture of real data and synthetic data. For example, in the mth generation of training, the training data is composed of a mixture of real data and synthetic data generated by the previous generation model, and the ratios are α m and 1-α m , forming the following combination:

[0045]

[0046] Among them, D real Indicates real data; D synthetic Represents synthetic data generated by the previous generation model.

[0047] The proportion of real data in the training data decreases dynamically with the increase of training generations, thus forming a decreasing strategy that can be adaptively evolved. For example, the decreasing strategy of adaptive evolution of real data satisfies the following calculation formula:

[0048]

[0049] Among them, α m represents the proportion of real data in the mth generation of training; β is a control parameter, which is related to the sample size of real data and the quality assessment results of synthetic data; m is the number of generations in the current recursive training process.

[0050] The specific process of dynamically adjusting the proportion of real data is as follows: in the initial training stage, the proportion of real data is close to 1, ensuring that the model establishes a stable language representation ability in the early stage; as the number of training generations gradually increases, the proportion of real data decreases in the manner of the above-mentioned logarithmic function, realizing a natural transition of the training process from being dominated by real data to being dominated by synthetic data; at the same time, by setting the control parameter β, the deceleration rate can be adaptively adjusted according to different data scales and task scenarios to achieve refined control of the model's autoregressive training process.

[0051] It's important to note that the control parameter is a key factor in regulating the rate at which the real data ratio decreases. Its value is related to the size of the real data sample in the current training task and the quality assessment results of the synthetic data. When there are sufficient real data samples or the synthetic data is of high quality, β can be set to a smaller value to accelerate the model's transition to self-supervised training. Conversely, β should be increased to extend the real data-dominated training phase to ensure stability and reliability in the early stages of model training.

[0052] It is not difficult to find that in the solution provided in the embodiment of the present application, the above-mentioned improvements enable the model to balance stability and efficiency throughout the training process, significantly reducing the risks of distribution drift and model crash caused by premature dominance of synthetic data, thereby improving the generalization ability and output quality of the language model.

[0053] Third embodiment

[0054] The third embodiment of the present application relates to a language model optimization method based on autoregressive training. The third embodiment is an improvement on the first embodiment. The specific improvement is that: in the third embodiment of the present application, a specific implementation method for dynamically adjusting the proportion of real data based on distribution drift feedback control is provided. That is, step S4 can further include the following steps:

[0055] Step S401: Based on the data distribution generated by the current generation model and the initial real data distribution, the L1 norm between the two is calculated as a measure of distribution drift. The L1 norm is the sum of the absolute values ​​of the probability differences between the two in all categories;

[0056] Step S402: comparing the measurement value with a preset distribution deviation threshold;

[0057] Step S403: If the metric value exceeds the distribution deviation threshold, the proportion of real data in the next generation of training data is increased, and the proportion of synthetic data is correspondingly reduced to reduce the risk of model distribution drift;

[0058] Step S404: If the metric value does not exceed the distribution deviation threshold, the proportion of real data is adjusted according to the preset decreasing strategy, and the next generation of training is continued.

[0059] Specifically, in autoregressive training, it is assumed that the initial true data distribution is , after m times of autoregressive training, the generation of the model can be expressed as:

[0060]

[0061] Among them, α m is the proportion of the current generation of real data in the mixed training data; is the estimated distribution of the m−1 generation model; The final synthetic data distribution generated for the current generation model.

[0062] Data distribution generated based on the current generation model With the initial real data distribution , calculate the L1 norm between the two as a measure of distribution drift. The norm is defined as the sum of the absolute values ​​of the probability differences between the two distributions in each category over the entire vocabulary space s, that is:

[0063]

[0064] Among them, ‖•‖1 is the L1 norm, which represents the total difference between the two distributions. The larger the value, the more serious the deviation between the two distributions. In the mth generation, the probability of generating the kth word; The probability of the kth word in the initial true distribution; s is the vocabulary size.

[0065] The L1 norm metric is compared with the system-defined distribution deviation threshold to determine whether the current generation model has significant deviations. The distribution deviation threshold is the maximum tolerable range preset by the system to determine whether the current generation model deviates from the true semantic distribution. If the deviation exceeds this threshold, the real data ratio adjustment mechanism is triggered to guide training back to a reasonable distribution range.

[0066] If the metric value exceeds the distribution deviation threshold, the proportion of real data is increased in the next generation of training. m , and correspondingly reduce the proportion of synthetic data to suppress the trend of distribution drift, thereby maintaining the closeness of the model output distribution to the real corpus distribution. If the metric value does not exceed the distribution deviation threshold, the current training status is considered to be within the acceptable range. The proportion of real data will continue to be adjusted according to the original decreasing strategy, and the next generation of training process will be executed.

[0067] Furthermore, the strategy for controlling the degree of distribution drift between the data distribution generated by the current generation model and the initial true data distribution includes: by setting a constant related to the real data scale and recording the training generations of the current autoregressive training, ensuring that after each generation of training, the measurement value of the distribution drift between the data distribution generated by the current generation model and the initial true data distribution shows a decreasing trend, thereby forming a dynamic balance relationship between the training generations and the distribution drift control during the model training process, ensuring that the distribution drift during the training process is always controllable.

[0068] For example, by setting a constant term C related to the actual data scale and combining it with the current training generation m, the following upper bound constraint on the distribution drift is formed:

[0069]

[0070] in, Represents the L1 norm deviation between the generated distribution of the current generation and the initial true distribution.

[0071] Through this control strategy, as the number of training generations increases, the theoretical upper limit of the distribution drift becomes The form gradually converges to form a dynamic drift suppression mechanism driven by training algebra. This mechanism ensures that during the recursive iteration of the model, the deviation between the data distribution generated by the current generation and the initial true data distribution is always within the theoretical controllable range. This strategy can be used with α m The use of data proportional reduction forms a closed-loop control system during the recursive training of the model, thereby effectively preventing the generated distribution from deviating from the true semantic distribution too early, and ensuring that the model maintains stable convergence and semantic consistency in multiple rounds of training.

[0072] It's easy to see that in the solution provided by the embodiments of this application, by introducing the L1 norm as a quantitative indicator of distribution drift, the system can accurately determine whether the model deviates from the true semantic distribution. When the deviation is too large, the proportion of real data is promptly increased to correct it, avoiding the distribution collapse or semantic degradation caused by synthetic data-dominated training. Conversely, continuing to reduce the proportion of real data when the deviation is controllable helps improve training efficiency and enhance the model's self-learning ability.

[0073] It should be noted that the third embodiment of the present application may also be an improvement based on any one or more of the first to second embodiments.

[0074] Fourth embodiment

[0075] The fourth embodiment of this application relates to a language model optimization method based on autoregressive training. The fourth embodiment is an improvement on the first embodiment. The specific improvement is that: in the fourth embodiment of this application, a specific implementation method for controlling the amount of synthetic data is provided. That is, the amount of synthetic data is limited to the amount of real data so that the control conditions are as follows:

[0076] The amount of synthetic data must not exceed a dynamic upper limit determined by the number of samples based on real data and the vocabulary size of the language model. That is, as the number of training data samples increases, the amount of synthetic data is allowed to increase accordingly, and as the vocabulary size increases, the amount of synthetic data allowed is reduced to prevent the risk of probability distribution collapse during model training due to excessive proportion of synthetic data.

[0077] Specifically, during each generation of training, this embodiment limits the amount of synthetic data generated by the current generation model to a dynamic upper limit determined by the total amount of training data samples and the size of the language model vocabulary. For example, the constraint on the amount of synthetic data can be expressed as:

[0078]

[0079] Where n represents the number of samples of synthetic data generated in the current generation; N is the total number of training data samples; and s is the vocabulary size of the language model.

[0080] According to this constraint, the upper limit of synthetic data, n, is proportional to the total amount of training data, N. As the total amount of training data samples, N, increases, the system can moderately increase the upper limit of synthetic data generation, thereby improving training efficiency and self-supervision capabilities. Furthermore, the upper limit of synthetic data, n, is inversely proportional to the logarithm of the vocabulary size, s. This indicates that as the vocabulary size of a language model increases, its generated semantic space becomes sparser and more complex. To prevent rapid shifts in semantic distribution, stricter limits on the amount of synthetic data generated are necessary to avoid problems such as mode collapse and semantic degradation in high-dimensional spaces.

[0081] It is not difficult to find that in the solution provided in the embodiment of the present application, by constraining this proportional relationship, it is possible to effectively prevent the proportion of synthetic data in the training data from being too high, thereby avoiding the model learning process from deviating from the semantic distribution of real data, and thus causing the risk of model collapse or generation quality degradation.

[0082] It should be noted that the fourth embodiment of the present application may also be an improvement based on any one or more of the first to third embodiments.

[0083] Fifth embodiment

[0084] The fifth embodiment of the present application relates to a language model optimization method based on autoregressive training. The fifth embodiment is an improvement on the first embodiment. The specific improvement is that: in the fifth embodiment of the present application, a specific implementation method is provided to extend the effective training period of the model by setting a training stability condition to avoid model crash during the model training process. That is, it includes the following steps:

[0085] In each generation of training, the proportion of real data is dynamically adjusted based on the diversity of the synthetic data and real data generated by the current model, combined with the dispersion measurement value of the initial real data, to reduce the degradation rate of the synthetic data generated by the model and delay the speed at which the data distribution of the model output converges to a single distribution state.

[0086] Specifically, in each generation of model training, this embodiment introduces a theoretical lower bound constraint mechanism for the model collapse time to guide the real data proportion α m Dynamic adjustment of the model. The model collapse time T represents the number of generations in which the model generation distribution converges to a single Dirac distribution (i.e., semantics degenerates to a single output). The lower bound of the model collapse time can be expressed as:

[0087]

[0088] Where T is the time of model collapse, that is, the number of training rounds in which the model generation distribution degenerates into a single point distribution; S0 is the dispersion measure of the initial real data distribution; γ is the degradation rate of the generated data, and its size is proportional to the real data α mand the diversity of real data.

[0089] Based on the theoretical lower bound constraint mechanism of the model collapse time, in each generation of training, the diversity difference between the synthetic data generated by the current language model and the real data is evaluated; combined with the dispersion measurement value S0 of the initial real data, the proportion of real data in the current generation of training data α is dynamically adjusted. m ; If the quality of the synthetic data is detected to be degraded or the semantic concentration is increased, increase α m , enhance the guiding role of real data on training; by controlling the increasing trend of γ, the goal of delaying the model collapse time T is achieved, thereby effectively extending the effective training cycle of the model in a diversity-maintaining state.

[0090] It is not difficult to find that in the solution provided in the embodiment of the present application, a real data ratio control mechanism guided by training stability is constructed. While ensuring the efficiency of model generation, it improves the model's ability to maintain semantic diversity and generation generalization capabilities, and effectively prevents the language model from falling into convergence collapse or content duplication in deep autoregressive training.

[0091] It should be noted that the fifth embodiment of the present application may also be an improvement based on any one or more of the first to fourth embodiments.

[0092] Sixth embodiment

[0093] The sixth embodiment of the present application relates to a language model optimization method based on autoregressive training. The sixth embodiment is an improvement on the first embodiment. The specific improvement is that: in the sixth embodiment of the present application, a specific implementation method of autoregressive training modeling is provided. That is, step S1 can further include the following steps:

[0094] Step S101: The language model receives the context sequence of the training sample as input, and sets the vocabulary size, context length and number of samples;

[0095] Step S102: Based on the currently input context sequence, the conditional probability distribution of all candidate words is calculated using the inner product of the model weight matrix and the context vector. The conditional probability distribution is probability normalized using the softmax function.

[0096] Step S103: using the cross entropy loss function as the objective function of language model training to calculate the error between the conditional probability distribution and the actual target word label;

[0097] Step S104: By optimizing the objective function, iteratively updating the model parameters until the objective function reaches the convergence condition, based on the conditional probability distribution results after the current generation model training is completed, generating the synthetic data required for the next generation training.

[0098] Specifically, we use the statistical language modeling approach to construct a conditional probability distribution framework for predicting the next word based on context. We first perform context modeling and input structure settings. We set the vocabulary size of the language model to s, which represents the number of recognizable words in the language model. We also set the context length to ℓ, which represents the length of the context window used to predict the next word in each training sample. The number of possible context combinations is c. Each training sample consists of a context sequence X and a target word Y, where A discrete random variable representing the historical context, A discrete random variable representing the next generated word.

[0099] The conditional probability distribution between the context of the synthetic data and the target word is defined as:

[0100]

[0101] During the autoregressive training process, the estimates of these conditional probability distributions are recorded as , can be obtained by optimizing the following objective function through the Softmax classifier, which is expressed as follows:

[0102]

[0103] Where M represents the number of training samples; is the weight matrix of the model; Indicates the The context input vector of samples; Indicates the corresponding target word; is the Softmax function, which means converting the linear score into a conditional probability distribution.

[0104] During the autoregressive training process, the conditional probability distribution of the mth generation model can be expressed as:

[0105]

[0106] in, represents the conditional probability of generating word k given context j in the mth generation model; Represents the weight vector of the kth word in the mth generation model; represents the embedding vector representation of context j; s represents the vocabulary size; the Softmax function ensures that all The sum of the probabilities is 1.

[0107] To optimize the model learning effect, this embodiment uses the cross entropy loss function as the training objective to minimize the difference between the probability distribution of the model output and the true target:

[0108]

[0109] in, is the weight matrix of the model; is the Softmax function, and are the sample context vector and target word vector respectively.

[0110] By minimizing this loss function, the fitting ability of the conditional probability distribution can be effectively improved. After the training objective function converges, the current generation language model has learned the conditional probability distribution for term prediction. Based on this probability distribution, the model can generate output terms in two ways: one is to determine the most likely target word through maximum probability selection, and the other is to perform random sampling based on the probability value to enhance generation diversity. Target terms are generated from the vocabulary and concatenated word by word to form new sentences. The data generated in this way can be used as synthetic data in the next generation training process and mixed with real data to form a new training set. Therefore, after each generation of model training, synthetic data for the next generation of training can be generated, constructing a closed-loop mechanism of prediction, generation, and retraining. Combined with various adjustment strategies, the recursive evolution and self-supervised enhancement of the language model can be achieved.

[0111] It is not difficult to find that in the solution provided in the embodiment of the present application, by constructing the conditional probability distribution of the entire vocabulary range based on the inner product operation of the context vector and the term weight matrix, and coordinating it with the Softmax function normalization, it can more accurately express the probability of generating words in the current context, which helps to improve the model's perception of the context and prediction accuracy.

[0112] It should be noted that the sixth embodiment of the present application may also be an improvement based on any one or more of the first to fifth embodiments.

[0113] The step division of the above various methods is only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0114] In addition, some embodiments of the present application further provide an electronic device. The electronic device may be various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device may also be various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0115] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes a language model optimization method based on autoregressive training as provided in any one or more of the above embodiments. Figure 2 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, with each device providing some of the necessary operations. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0116] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 2 The bus connection is taken as an example.

[0117] Input device 1103 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. Examples include a touch screen, keypad, mouse, trackpad, touchpad, pointing stick, one or more mouse buttons, trackball, joystick, and other input devices. Output device 1104 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibration motors). The display device may include, but is not limited to, a liquid crystal display, a light emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0118] To provide user interaction, the electronic device may be a computer. The computer includes a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse) through which the user can provide input to the computer. Other types of devices may also be used to provide user interaction; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback), and input from the user may be received in any form (e.g., voice input or tactile input).

[0119] In an embodiment of the present application, a computer program / instruction is stored on a computer-readable medium. When executed by a processor, the computer program / instruction implements a language model optimization method based on autoregressive training provided in any one or more of the above-described embodiments. The computer-readable medium may be included in the electronic device described in the above-described embodiments, or may exist independently without being incorporated into the device. The computer-readable medium carries one or more computer-readable instructions.

[0120] The memory 1102 can be used as a non-transitory computer-readable storage medium to store non-transitory software programs, non-transitory computer executable programs, and modules. The processor 1101 executes the non-transitory software programs, instructions, and modules stored in the memory 1102 to execute various functional applications and data processing of the server, thereby implementing the program instructions / modules corresponding to the method provided in any one or more of the above embodiments of the present application.

[0121] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely located relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. Computer-readable media may be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0123] Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technology, compact discs, digital versatile discs or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0124] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network or a wide area network, or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] In the above embodiments, all or part of the steps or functions of the present invention may be implemented using software, hardware, firmware, or any combination thereof. For example, implementation may be achieved using a dedicated integrated circuit, a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of the present application may be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) may be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, a floppy disk, or the like. In addition, some steps or functions of the present application may be implemented using hardware, for example, as a circuit that cooperates with a processor to perform the various steps or functions.

[0126] The computer program product provided in the embodiments of the present application includes one or more computer programs / instructions that, when executed by a processor, fully or partially produce the processes or functions described in accordance with the embodiments of the present application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0127] The flowcharts or block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-specific system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The scope of this application is defined by the appended claims rather than the foregoing description and is therefore intended to encompass within this application all changes that come within the meaning and range of equivalents of the claims. Any reference signs in the claims should not be construed as limiting the claims to which they relate. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim may also be implemented by one unit or device through software or hardware. Words such as "first" and "second" are only used to distinguish the description and do not indicate any particular order, nor should they be understood as indicating or implying relative importance.

[0129] The above descriptions are merely specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art may easily propose variations or substitutions within the technical scope disclosed in the present application, and such variations or substitutions shall be encompassed within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims, and the above descriptions shall be regarded as exemplary and non-limiting.

Claims

1. A language model optimization method based on autoregressive training, characterized in that: include: Constructing an autoregressive training framework for a language model that predicts the probability distribution of subsequent words based on the historical context of the input, where the probability distribution in each generation of autoregressive training is used to generate synthetic data for the next generation; In each generation of training, real data is mixed with synthetic data generated by the previous generation model in a set ratio as training data, where the proportion of real data in the training data dynamically decreases as the number of training generations increases; Using the mixed data to train a language model, update model parameters, and generate synthetic data required for next-generation training, wherein the amount of the synthetic data relative to the amount of the real data is limited to not exceed a set synthetic data ratio control condition; After each generation of training, the difference between the data distribution generated by the current model and the real data distribution is measured, and the proportion of the real data is dynamically adjusted based on the difference, including: Based on the data distribution generated by the current generation model and the initial true data distribution, the L1 norm between the two is calculated as a measure of distribution drift. The L1 norm is the sum of the absolute values ​​of the probability differences between the two over all categories; Comparing the metric value with a preset distribution deviation threshold, where the distribution deviation threshold is a preset maximum tolerable range for determining whether the current generation model deviates from the true semantic distribution; If the metric value exceeds the distribution deviation threshold, the proportion of real data in the next generation of training data is increased, and the proportion of synthetic data is correspondingly reduced to reduce the risk of model distribution drift; If the metric value does not exceed the distribution deviation threshold, the proportion of real data is adjusted according to a preset decreasing strategy, and the next generation of training is continued; Strategies for controlling the degree of distribution drift between the data distribution generated by the current generation model and the initial true data distribution include: By setting a constant C related to the actual data scale and recording the number of training generations m of the current autoregressive training, the following upper bound constraint on the distribution drift is formed: ； in, Represents the L1 norm deviation between the current generation distribution and the initial true distribution; As the number of training generations increases, the theoretical upper limit of distribution drift becomes The form gradually converges, ensuring that after each generation of training, the distribution drift measurement between the data distribution generated by the current generation model and the initial real data distribution shows a decreasing trend, thus forming a dynamic balance between the number of training generations and the control of distribution drift during the model training process, ensuring that the distribution drift during the training process is always controllable; The autoregressive training process of the language model is continuously optimized based on synthetic data generated by probability distribution, dynamic data ratio control, and synthetic data quantity constraints.

2. The language model optimization method based on autoregressive training according to claim 1, characterized in that: The process of dynamically reducing the proportion of real data in the training data as the number of training generations increases is as follows: as the number of training generations increases, the proportion of real data decreases according to a preset decreasing strategy; the decreasing strategy is specifically as follows: the proportion of real data in the initial training stage is close to 1, and as the number of training generations increases, the proportion of real data gradually decreases in a manner inversely proportional to the logarithmic function of the number of training generations, and the rate of decrease in the proportion of real data is determined by control parameters set according to the scale of training data and the quality of generated data.

3. The language model optimization method based on autoregressive training according to claim 1, characterized in that: The control condition for limiting the amount of the synthetic data relative to the amount of the real data to no more than a set synthetic data ratio is specifically as follows: the amount of the synthetic data must meet a dynamic upper limit determined jointly based on the number of samples of the real data and the vocabulary size of the language model, that is, as the number of training data samples increases, the amount of synthetic data is allowed to increase accordingly, and at the same time, as the size of the vocabulary expands, the allowed amount of synthetic data is reduced to prevent the risk of probability distribution collapse during model training due to excessive proportion of synthetic data.

4. The language model optimization method based on autoregressive training according to claim 1, characterized in that: The method further includes extending the effective training period of the model by setting a training stability condition to avoid model crash during the model training process, specifically including: In each generation of training, the proportion of real data is dynamically adjusted based on the diversity of the synthetic data and real data generated by the current model, combined with the dispersion measurement value of the initial real data, to reduce the degradation rate of the synthetic data generated by the model and delay the speed at which the data distribution of the model output converges to a single distribution state.

5. The language model optimization method based on autoregressive training according to claim 1, characterized in that: The step of constructing an autoregressive training framework for a language model and predicting the probability distribution of subsequent words based on the historical context of the input includes: The language model receives the context sequence of the training sample as input, and sets the vocabulary size, context length, and number of samples; Based on the current input context sequence, the conditional probability distribution of all candidate words is calculated using the inner product of the model weight matrix and the context vector. The conditional probability distribution is normalized by the softmax function. Using the cross entropy loss function as the objective function for language model training, the error between the conditional probability distribution and the actual target word label is calculated; By optimizing the objective function and iteratively updating the model parameters until the objective function reaches the convergence condition, the synthetic data required for the next generation training is generated based on the conditional probability distribution results after the current generation model training is completed.

6. An electronic device, characterized in that: The electronic device comprises: One or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor performs the language model optimization method based on autoregressive training according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program and / or instructions stored thereon, characterized in that: When the computer program and / or instructions are executed by a processor, the language model optimization method based on autoregressive training as described in any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program and / or instructions, characterized in that When the computer program and / or instructions are executed by a processor, the language model optimization method based on autoregressive training as described in any one of claims 1 to 5 is implemented.

Images