Text generation method and electronic device

Abstract

The application discloses a text generation method and an electronic device, and relates to the technical field of data processing. When a corresponding response text is generated according to a target problem text through a target text generation model, the target text generation model is optimized based on target training samples dynamically and adaptively screened from a plurality of training samples according to the length distribution of a plurality of response texts generated by a current text generation model at a current stage. The screening logic is adaptively and dynamically adjusted based on the real generation behavior of the current stage of the model, so that the target training samples can be continuously optimized during the training process without relying on preset thresholds or manual parameter adjustment, and the method has stronger universality. When the target text generation model is used for text generation, the technical problems of long response text caused by overthinking or logical errors of response text caused by insufficient reasoning can be solved, and the technical effect of generating response text with more stable reasoning quality and more actual needs can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a text generation method and an electronic device. BACKGROUND

[0002] Language models are widely used in natural language processing tasks such as text generation, summarization, and question answering, and are gradually expanding to reasoning and problem solving scenarios. In such tasks, the model needs to generate coherent and rigorous thought chains to ensure the correctness of the output.

[0003] In related technologies, all samples are trained without distinguishing the contribution of response length to reasoning quality, resulting in a waste of computing resources for a large number of low-value samples. Moreover, the length penalty in related technologies can only passively limit the length of the response, making it difficult to guide the model to generate concise and accurate reasoning, and easily leading to overthinking or insufficient reasoning. It cannot adapt to changes in the distribution of the length of the model's response during the training process, and needs to be repeatedly optimized for different models and datasets, lacking universality.

[0004] When generating text, using the model in related technologies will result in unstable reasoning quality and poor efficiency during text generation. Specifically, it may produce lengthy and redundant response text content due to overthinking, or the generated response text may have logical breaks and factual errors due to insufficient reasoning. Moreover, the length of the response text is also uncontrollable, which may slow down the overall efficiency of response text generation. SUMMARY

[0005] The present application provides a text generation method and an electronic device to at least solve the technical problem of overthinking leading to lengthy response text or insufficient reasoning leading to logical errors in response text, achieving the technical effect of generating response text with more stable reasoning quality and more in line with actual needs.

[0006] The present application provides a text generation method, comprising: determining a target question text input; inputting the target question text into a target text generation model to obtain a target response text matched with the target question text; the target text generation model is a model obtained by parameter optimization of a current text generation model based on target training samples, the target training samples are samples dynamically and adaptively selected from a plurality of training samples based on the length distribution of a plurality of response texts generated by the current text generation model at the current stage, each of the training samples comprising a question text and a plurality of response texts generated by the current text generation model based on the question text.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of any of the above text generation methods.

[0008] The application further provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the text generation methods.

[0009] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the steps of any of the text generation methods.

[0010] According to the application, when a corresponding response text is generated according to a target problem text by using a target text generation model, the target text generation model is optimized based on target training samples dynamically and adaptively selected from a plurality of training samples according to a length distribution of a plurality of response texts generated by a current text generation model at a current stage. The selection logic is adaptively and dynamically adjusted based on the real generation behavior of the current stage of the model, so that the target training samples can be continuously optimized during the training process without relying on a preset threshold or manual parameter adjustment, and the application has stronger universality. When the target text generation model is used for text generation, the technical problems of long response text caused by overthinking and logical error of response text caused by insufficient reasoning can be solved, and the technical effect of generating a response text with more stable reasoning quality and more in line with actual needs can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0012] Figure 1 A flowchart of a text generation method provided by an embodiment of the application.

[0013] Figure 2 A training process diagram of a text generation model provided by an embodiment of the application.

[0014] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the application.

[0015] Figure 4 A schematic diagram of a computer-readable storage medium provided by an embodiment of the application. DETAILED DESCRIPTION

[0016] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort should fall within the protection scope of the present application.

[0017] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0018] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0019] In combination with the specific application environment architecture or specific hardware architecture on which the text generation method is executed, the specific application environment architecture or specific hardware architecture is described here. The text generation method can be deployed in a model training and inference architecture, and its execution relies on a distributed computing environment composed of a multi-node server cluster. Each node is equipped with a high-performance GPU (Graphics Processing Unit) / TPU (Tensor Processing Unit) accelerator, a shared memory module and a high-speed interconnection network to support parallel generation of multiple response texts and statistical analysis of their length distribution. In terms of system architecture, the training platform includes a data preprocessing module, a distributed generation module, a dynamic sample selection module and a parameter updating module. Each module works cooperatively through a unified scheduling framework to realize response generation, length feature extraction, target sample selection and iterative optimization of model weights. In addition, the method can also be adapted to special computing hardware with on-chip neural network acceleration units, which can ensure high throughput text generation capability while making the length distribution driven sample selection and model optimization process run efficiently under specific hardware architecture, thereby improving the overall training efficiency and inference quality.

[0020] As Figure 1 In a first aspect, the present application provides a text generation method, comprising: S11: determining an input target question text. S12: inputting the target question text into a target text generation model to obtain a target response text matched with the target question text.

[0021] The target text generation model is a model obtained by optimizing the parameters of the current text generation model through target training samples. The target training samples are samples dynamically and adaptively selected from a plurality of training samples based on the length distribution of a plurality of response texts generated by the current text generation model at the current stage. Each training sample includes a question text and a plurality of response texts generated by the current text generation model based on the question text.

[0022] Specifically, the training data set can be divided into a plurality of information sets containing question texts, and a plurality of question texts are randomly sampled therefrom to obtain a plurality of training samples required for the current round of training. For each sampled question text, the current text generation model generates a plurality of response texts (it should be noted that in the present embodiment, the response text refers to a text containing intermediate reasoning steps generated by the language model for the question text, i.e., a thinking chain. Its role is to show the logical process of the model to reach the final answer, rather than just providing the final conclusion. The length distribution and dynamic adaptive screening described in the present embodiment are also optimized for the length and quality of the thinking chain, in order to guide the model to generate a concise and accurate reasoning path) according to its existing parameters after receiving the question text. These response texts can be obtained by, but not limited to, multiple sampling, changing the sampling temperature, enabling a randomized decoding strategy, etc. By generating a plurality of different response texts for the same question text, the present embodiment can represent the diversified output of the model for the same input under the current parameter state, thereby forming a training sample containing the question text and a plurality of corresponding response texts.

[0023] In specific implementation, the present embodiment does not limit the number of times or the generation strategy of generating response texts, which can be flexibly set according to the computing resources, model size or task complexity. The plurality of response texts contained in each training sample are generated by relying on the understanding ability of the current text generation model for the question text and its generation logic. By generating a set of response texts for each question text, the present embodiment can truly reflect the response behavior of the current model for the input.

[0024] In this embodiment, based on the length distribution of the plurality of response texts generated by the current text generation model at the current stage, it is dynamically and adaptively determined which training samples are selected as target training samples. Specifically, for the plurality of response texts contained in each training sample, the embodiment can count the length information of these response texts, such as the number of tokens, the number of characters, the number of words, or the number of tokens obtained according to a preset tokenization rule. After obtaining the length set of each training sample, the embodiment can perform distribution analysis on these length data, and the analysis method can be, but is not limited to, calculating the mean, variance, quantile, histogram distribution shape, etc. Through this analysis process, the length characteristics of the model when generating responses to different problem texts under the current parameter state can be accurately obtained, and whether there is a difference in the model generation behavior between different training samples can be observed accordingly.

[0025] It needs to be understood that the current stage refers to the specific round or period in which the current parameter optimization is performed in the iterative training process of the target text generation model. It is emphasized that the data relied on by the dynamic adaptive screening (i.e., the length distribution of the plurality of response texts) is the result generated by the model to be optimized (the current text generation model) before the start of this training iteration, rather than data from history or other models.

[0026] It also needs to be understood that the length distribution refers to the statistical distribution of the lengths of the plurality of response texts generated by the current text generation model for a batch (or all) of training problem texts at the current stage. The length distribution is not a single length value, but describes the overall statistical characteristics of the lengths of the response texts in various length intervals (for example, it can be described by a histogram, quantile, mean and variance, etc.).

[0027] It also needs to be understood that the dynamic adaptive screening in this embodiment refers to a screening process in which the standard or threshold for screening target training samples is not fixed and is automatically calculated and adjusted according to the length distribution characteristics calculated at the current stage.

[0028] After completing the above length distribution analysis, the embodiment selects target training samples from the plurality of training samples according to a preset screening logic. The screening logic can be, but is not limited to, selecting training samples with length distribution more consistent with the expected interval, eliminating training samples with significantly short or long lengths, or determining representative training samples based on length variation rules. The screening standard can be pre-set according to the task scenario, or can be dynamically adjusted according to the statistical characteristics of the current training round. By selecting training samples based on length distribution, the embodiment can make the selected target training samples have clear distinguishable properties.

[0029] Based on the target training samples obtained through the foregoing screening, the parameters of the current text generation model are updated. Specifically, in an exemplary embodiment, the generation process of the target text generation model includes: obtaining the reward values of the plurality of response texts corresponding to each question text in the target training samples, and calculating a reward function according to the reward values; calculating a loss function based on the reward function, and updating the parameters of the current text generation model through gradient descent according to the loss function to obtain the target text generation model.

[0030] Each target training sample contains a question text and a plurality of response texts. During the training process, the model can calculate the difference between the current model output and the target output according to these data. The difference can be quantified by a preset loss function. The form of the loss function can be, but is not limited to, cross-entropy loss, sequence modeling loss, or other loss measurement methods suitable for text generation tasks. After obtaining the loss value, the model parameters are updated based on the gradient descent method in this embodiment. The updating method can include single round gradient descent, gradient update with momentum, or parameter adjustment method suitable for specific training framework. By repeatedly calculating the loss and updating the parameters, this embodiment can gradually adapt the model to the language features and response patterns in the target training samples in multiple training rounds.

[0031] In the specific implementation process, the use of target training samples in this embodiment does not limit specific training strategies. Single round training, batch training, or distributed training can be used according to actual task needs. The frequency of parameter updating, learning rate, and iteration number can also be flexibly set according to hardware conditions or task size. When the new parameters are written into the current text generation model, the target text generation model is formed. When the target question text input by the user is received using the target text generation model, the target response text is generated based on the updated parameters, so that the response text is consistent with the corresponding target question text in terms of language structure, content coverage, and expression form. In this way, this embodiment enables the model to gradually acquire the ability to understand the characteristics of the target training samples and adapt to the combination relationship between the question texts and the response texts contained therein.

[0032] The response text obtained based on the target text generation model has more stable reasoning quality. It can avoid excessive thinking of redundancy and also avoid logical loopholes caused by insufficient reasoning, and finally obtain a concise, rigorous, and actual demand response text.

[0033] In an exemplary embodiment, the screening process of the target training samples includes: calculating the average length of the plurality of response texts corresponding to each question text; calculating the length quantile under the current training batch based on the average length; and according to the preset quantile threshold, screening the training samples with the average length located in the predetermined quantile interval as the target training samples.

[0034] In this embodiment, by calculating the average length of multiple response texts corresponding to each question text, the length performance of the current text generation model when facing different question texts can be obtained. The lengths of multiple response texts often fluctuate, and by averaging these lengths, each training sample has a stable and directly comparable length indicator. The calculation method of average length can be, but is not limited to, averaging the token number, averaging the token number after segmentation, or averaging the sequence length according to a specific measurement rule. In this embodiment, the length information of multiple response texts is normalized to a single value, and the length characteristics between training samples are more clearly exhibited.

[0035] After obtaining the average lengths of all training samples, this embodiment further calculates quantiles of these average length data to represent the overall structure of the length distribution under the current training batch. The calculation of quantiles can use common statistical methods, such as determining the length value at a specific percentage position according to the sorting result from small to large. The position of the quantile can be flexibly selected according to the actual training purpose, such as the median, the upper quartile, the lower quartile, or other arbitrary ratio points. This embodiment can reflect the central distribution interval of the average length of response texts in the current training batch and the relative position of the length in the overall distribution through the quantile, so that the subsequent screening based on the quantile has a clear statistical basis.

[0036] After obtaining the length quantile, this embodiment screens target training samples from all training samples according to a pre-set quantile threshold interval. For example, training samples with average lengths in a certain predetermined quantile interval can be selected as target training samples, so that these samples have length characteristics that match the current training purpose. The quantile threshold interval can be fixedly set, or can be adjusted according to the statistical characteristics of the training batch.

[0037] It should be understood that the quantile threshold / predetermined quantile interval refers to a length boundary value calculated based on the length distribution for defining the screening range. For example, the 25th percentile (the first quartile) is a quantile threshold, and the interval between the 25th percentile and the 75th percentile is a predetermined quantile interval. These thresholds or intervals are calculated in real time based on the average length distribution of the current batch of response texts.

[0038] This embodiment screens training samples based on the interval, so that the target training samples have identifiable attributes in the distribution of response lengths, which can be used in subsequent model parameter updating steps.

[0039] In an example embodiment, based on the average length, the length quantile under the current training batch is calculated, including: calculating a cumulative distribution function value according to the average length, the cumulative distribution function value representing the proportion of question texts with an average length less than or equal to a given length; and calculating a quantile function according to the cumulative distribution function value, the quantile function being used to determine the corresponding length value according to a given quantile.

[0040] In this embodiment, by statistically analyzing the average lengths of multiple training samples, the cumulative distribution function value can be calculated based on these length values. Specifically, the average lengths of all training samples can be sorted in ascending order, and the proportion of the number of samples with an average length less than or equal to a given length value to the total number of samples can be calculated in the sorting result. This proportion is the cumulative distribution function value, which represents the proportion of question texts with an average length less than or equal to a given value in the current training batch. The cumulative distribution function can be calculated in a discrete sorting manner, or can be calculated by interpolation after histogram statistics, so that this embodiment can clearly depict the distribution trend of the average length in the current training batch.

[0041] After obtaining the cumulative distribution function value, this embodiment further constructs a quantile function based on these cumulative distribution function values, which is used to obtain the length value corresponding to a given quantile, i.e., to determine the corresponding length value according to a given quantile. The quantile function can be realized by finding the average length corresponding to the target quantile position in the cumulative distribution function. For example, when the target quantile is 0.5, the average length value corresponding to the 50% cumulative proportion can be found in the cumulative distribution as the median. If the target quantile is between two discrete length points, this embodiment can determine its corresponding length value by linear interpolation or other interpolation methods. In this way, this embodiment can map the real distribution structure of the average length in the training batch to the quantile length value that can be directly used, so that the subsequent steps can set the screening interval according to these explicit quantile values.

[0042] In an example embodiment, according to a preset quantile threshold, training samples with an average length in a predetermined quantile interval are selected as target training samples, including: according to a preset first quantile, a second quantile and a maximum quantile, retaining training samples with an average length less than or equal to the first quantile, or with an average length greater than or equal to the second quantile and less than or equal to the maximum quantile, as target training samples; the first quantile < the second quantile < the maximum quantile.

[0043] In this embodiment, when filtering the training samples according to the preset quantile threshold, the average length of all training samples is judged based on the calculated first quantile, second quantile and maximum quantile. Specifically, the training samples with an average length less than or equal to the first quantile are retained in the target training sample set. Such samples generally correspond to shorter responses generated by the model in the current training batch. Shorter responses generally reflect the effective expression ability of the model in the concise reasoning scenario, enabling the embodiment to continuously strengthen the ability of the model to maintain a concise and compact reasoning path during training. At the same time, the embodiment also retains training samples with an average length greater than or equal to the second quantile and less than or equal to the maximum quantile to form another part of the target training samples. This interval generally corresponds to longer responses of a specific length range, which is used to cover the chain thinking features required for complex problems or multi-step reasoning, enabling the model to maintain stability and integrity when facing long-chain reasoning tasks. The first quantile, the second quantile and the maximum quantile satisfy the first quantile < the second quantile < the maximum quantile, so that the target interval division structure is clear and the filtering logic boundary is clear.

[0044] In related technologies, the training process often processes all samples in the same way, and a large number of intermediate length samples are repeatedly used in training, but these samples generally neither have sufficient short-chain reasoning features nor reflect the long-chain reasoning structure required for complex tasks, so their promotion of model reasoning ability is limited. The embodiment maps the average length to the quantile interval, integrates the response length distribution structure of the training batch into the sample filtering logic, and enables the training process to actively focus on the two most valuable length intervals, one corresponding to short responses for stable and correct reasoning, and the other corresponding to specific long responses for effective deep reasoning. In this way, the embodiment avoids a large number of intermediate length samples with limited contribution in the sample distribution, so that training resources are concentrated on samples with reasoning indicative.

[0045] At a further level, the embodiment considers the internal correlation between response length and reasoning quality in the dynamic sampling process of the reinforcement learning training framework. By performing real-time statistics on the average length distribution of the current training batch and dynamically filtering the training samples in the shortest interval and the specific longest interval according to the statistical structure, the embodiment can directly guide the model to focus on high-value samples at the data level. This active filtering method based on length distribution enables the learning goal of the model to be consistent with the evolution of the reasoning structure, and no longer relies on static thresholds or manual parameter tuning to control sample quality, breaking through the bottleneck in the expansion of reasoning ability in related technologies.

[0046] In an exemplary embodiment, after the target question text is input into the target text generation model to obtain the target response text matched with the target question text, the method further comprises: obtaining the accuracy of the target response text output by the target text generation model multiple times; and adjusting the predetermined quantile interval according to the accuracy.

[0047] In the embodiment, after the target text generation model generates the target response text based on the target question text input by the user, the accuracy of multiple outputs of the model can be evaluated. Specifically, the embodiment can generate several response texts for the same question text, and calculate the correctness, completeness or logical consistency of each response text according to a preset accuracy measurement method. The accuracy calculation method can be, but is not limited to, comparing the model output with a verifiable answer, manual annotation or verifiable reference content, and quantifying it into a numerical index or a rating score. Through the accuracy statistics of multiple outputs, the embodiment can obtain the performance distribution of the current target text generation model when processing this type of question, and provide a basis for subsequent parameter adjustment and sample selection.

[0048] After obtaining the accuracy data, the embodiment further adjusts the preset quantile interval according to the accuracy information. Specifically, if the response accuracy of a certain length interval is higher than that of other intervals according to multiple outputs, the embodiment can appropriately expand the coverage of the interval, so that the proportion of target training samples in the length interval increases; on the contrary, if the accuracy of a certain length interval is lower, the interval can be reduced or the training samples of the interval can be temporarily excluded. The adjustment method of the quantile interval can be, but is not limited to, linear or nonlinear adjustment, and can also be dynamically updated in combination with the length distribution of the current training batch. The embodiment adjusts the quantile interval driven by the accuracy, so that the selection of training samples is not only based on the response length, but also combined with the actual generation performance of the model, to realize the dynamic matching of sample selection and model capability in the training process.

[0049] In an exemplary embodiment, after the target question text is input into the target text generation model to obtain the target response text matched with the target question text, the method further comprises: obtaining the accuracy of the target response text output by the target text generation model multiple times; and adjusting the predetermined quantile interval according to the accuracy.

[0050] In this embodiment, when the target text generation model generates a target response text based on the user input target question text, the response rate of multiple outputs of the model can be obtained. Specifically, this embodiment can record the time taken for each response from receiving the question text to generating a complete target response text, and calculate the average response time or response speed in multiple outputs. The response rate can be calculated in terms of the number of tokens, words or tokens generated per second, or can be calculated according to the interval between consecutive outputs. Through the response rate statistics of multiple outputs, this embodiment can reflect the generation efficiency of the model in the current parameter state when processing this type of question, and provide a basis for subsequent adjustment of the quantile interval.

[0051] After obtaining the response rate data, this embodiment further adjusts the preset quantile interval according to the response rate. Specifically, if the response rate of a certain length interval is high, this embodiment can appropriately expand the coverage of the interval, so that the proportion of target training samples in the length interval increases; on the contrary, if the response rate of a certain length interval is low, the interval can be reduced or the training samples of the interval can be temporarily excluded. The adjustment method of the quantile interval can be linear or nonlinear adjustment, or can be dynamically updated in combination with the length distribution and generation behavior of the current training batch. This embodiment adjusts the quantile interval based on the response rate information, so that the training sample selection not only considers the response length, but also takes into account the generation efficiency of the model, so that the target training sample is more consistent with the actual output characteristics of the current model in the length interval distribution.

[0052] In an exemplary embodiment, before the target training sample is screened out, the method further includes: for each question text, calculating the reward value of the corresponding multiple response texts based on the corresponding verifiable response text, filtering the training samples corresponding to the question texts whose reward values are all zero or all one, to obtain a filtered sample set; and the target training sample is specifically selected from the filtered sample set based on the length distribution of the multiple response texts.

[0053] In this embodiment, before the target training samples are screened based on the length distribution of the multiple response texts, the reward value of the response to each question text can be calculated. Specifically, for each question text, the multiple response texts generated by the current text generation model are compared with the verifiable response text corresponding to the question text, and the reward value of each response text is calculated. The reward value can be calculated in a binary manner to indicate whether the response is correct or not, or can be quantified by scoring the accuracy or coverage of the response, thereby forming a reward value set corresponding to the response text in each training sample. Through this process, the performance of the current model on the question text can be determined, and samples that do not need to participate in the training process can be identified accordingly. The verifiable response text refers to a response text that exists for a certain question text and has known correctness or quality (such as a standard answer or a high-quality reply annotated by a human), which is used as a reference benchmark to evaluate the quality of the response text generated by the current text generation model. The reward value is a numerical indicator for quantitatively evaluating the quality of a single response text generated by the current text generation model relative to the verifiable response text. Generally, the higher the reward value, the better the quality of the response text (e.g., more accurate and more relevant).

[0054] After obtaining the reward value, the training samples with all-zero or all-one reward values are filtered out. The training samples corresponding to the question texts with all-zero reward values indicate that the model does not generate correct responses at all under this input, and do not need to participate in the current training round because their contribution to the model parameter update is extremely low. The training samples corresponding to the question texts with all-one reward values indicate that the model stably generates correct responses under this input, and do not need to participate in the training because the model has mastered the generation rule of the question text. By filtering out these samples, a filtered sample set is obtained, which retains the training samples with reward values fluctuating between zero and one. These samples can truly reflect the weaknesses and improvement space of the current model.

[0055] After obtaining the filtered sample set, the target training samples are screened based on the length distribution of the multiple response texts. Specifically, the average length, cumulative distribution, and quantile interval of the multiple response texts of each training sample are calculated according to the length distribution analysis method, and the target training samples are selected according to the pre-set logic. By screening the length distribution on the filtered sample set, the training samples with short responses or specific long responses can be selectively selected based on the elimination of samples with limited contribution to the training or samples that have been mastered, so that the target training samples have clear attributes in terms of length characteristics and can be used for subsequent model parameter update.

[0056] In an example embodiment, the method further comprises: adding the data set corresponding to the target training sample to a training sample pool until the number of samples in the training sample pool is greater than or equal to the training batch size; and if the number of samples in the training sample pool is greater than the training batch size, randomly selecting training samples of the training batch size from the training sample pool to obtain a final training batch.

[0057] In the example embodiment, after the target training samples are selected, the data sets corresponding to the target training samples are added to a training sample pool. The training sample pool is used to temporarily store the high-value training samples that have passed the screening, and the number of samples is accumulated until it is greater than or equal to the preset training batch size. Through this process, the example embodiment can ensure that each training batch meets the model training requirements in terms of quantity, while retaining the characteristics of the target training samples that have passed the length distribution screening. The samples in the training sample pool can be added in the order of screening, but are not limited thereto, and can also be added in other strategies, such as in chronological order or randomly, to form a diversified training sample set.

[0058] When the number of samples in the training sample pool exceeds the training batch size, the example embodiment randomly selects training samples of the training batch size from the training sample pool to form a final training batch. The random selection method can be, but is not limited to, simple random selection, stratified random selection, or random selection according to weights, to ensure that the final training batch is representative in terms of length distribution and reward characteristics, while controlling the sample size of a single training. Through this process, the example embodiment can balance the quality and quantity of training samples during batch generation, and provide appropriate data input for model parameter update.

[0059] Finally, the example embodiment updates the parameters of the current text generation model using the generated final training batch. Specifically, the example embodiment takes each training sample in the training batch as input, calculates the difference between the model output and the target output, and updates the model weights through gradient descent or other applicable parameter adjustment methods. The training can be, but is not limited to, single-round update or multi-round iterative update, to gradually adapt the current text generation model to the characteristics of the target training samples. By using the final training batch that has passed the length distribution screening and has been regulated by the training sample pool, the example embodiment can optimize the parameters while maintaining the clear characteristics of the samples, and form the model state required for the next round of training.

[0060] In an exemplary extended embodiment, when optimizing the parameters of the current text generation model using the target training samples, a dynamic weighting mechanism of the reward values can be combined to further fine-tune the model training direction. Specifically, after obtaining the reward values of the multiple response texts corresponding to each question text, the embodiment no longer directly uses a uniform weight, but instead weights the reward values according to the length, complexity or generation rate of the response texts. The weighting method can be, but is not limited to, assigning a higher weight to short response texts or concise reasoning responses to strengthen the model's performance on concise reasoning tasks, and assigning a moderate weight to long response texts within a specific length interval to maintain the model's ability to handle complex problems. Through this dynamic weighting, the embodiment can enable the reward function to reflect accuracy and response characteristics while taking into account the distribution characteristics of response length or complexity. Subsequently, the embodiment calculates the loss function based on the weighted reward function and updates the model parameters through gradient descent methods. The loss function can be, but is not limited to, the negative expected value of the weighted reward, or the negative gradient of the expected return calculated by combining the policy gradient method. During the update process, single or multi-round iterative training can be used, and momentum, learning rate adjustment or adaptive optimization algorithms can be used to control the parameter update amplitude. After executing the training, the embodiment can form a text generation model that is more sensitive to the characteristics of the target training samples, enabling it to generate concise and accurate short responses as well as long responses covering complex logic when facing different question texts, thereby balancing training efficiency and reasoning performance.

[0061] As Figure 2 , one specific embodiment is: A. Initialization.

[0062] a. Input parameter initialization.

[0063] First, prepare the training data set D, which contains the prompt words and corresponding verifiable answers of the reasoning tasks. Second, select a reinforcement learning method with verifiable rewards such as GRPO (Group Relative Policy Optimization), DAPO (Direct Absolute Policy Optimization), or GSPO (Generalized Scalar Preference Optimization) as the optimization training method, for example, DAPO can be selected.

[0064] b. Hyperparameter initialization.

[0065] Rolling batch size B_r: the number of prompt words sampled each time (i.e. the number of question texts), the embodiment preferably recommends that the value be no less than 256 to ensure the reliability of the length distribution statistics.

[0066] Training batch size B_t: the number of samples used for each model update, the embodiment preferably suggests that the value is not less than 512.

[0067] Number of responses per prompt sample G: To balance sampling diversity and computational cost, the embodiment preferably suggests that the value is between 8 and 16.

[0068] Length filtering threshold: L_low (for example = 0.3, meaning that the shortest 30% of samples are retained), L_high (for example = 0.65, meaning that samples above the 65th percentile are retained), L_max (for example = 0.95, meaning that samples above the 95th percentile are filtered out as too long).

[0069] It should be understood that the application gives the length filtering threshold verified by a large number of experiments, which can improve the performance on different models and datasets. The value range of the quantile threshold L_low is recommended to be 0.2-0.4, the value range of the quantile threshold L_high is recommended to be 0.6-0.7, and the value range of the quantile threshold L_max is recommended to be 0.9-0.98. In addition, it can be seen that the quantile is dynamically calculated based on the average response length distribution of the current training batch, rather than statically set. Of course, the above threshold parameters also need to be adjusted according to the ability of the model to be trained, for example, for a low-capacity model, L_high can be appropriately reduced to increase the retention ratio of long response samples to help the model learn long thought chains of complex problems. However, for a high-capacity model, L_low can be appropriately increased to reduce the retention ratio of short response samples to avoid the model relying too much on simple reasoning and reducing the accuracy on complex problems. Other training parameters, such as maximum response length L_max, learning rate lr, and sampling temperature T, etc.

[0070] c. Model initialization. Select the language model to be trained.

[0071] B. Rolling sampling and preliminary filtering.

[0072] a. Question sampling: randomly sample B_r prompt / questions from the training dataset D ( , where i = 1 ~ B_r).

[0073] b. Response generation: for each prompt / question , generate G responses based on the current model parameters ( , where j = 1 ~ G, G is the number of responses corresponding to the i-th prompt), form the sample set ( , ), where i = 1 ~ B_r, j = 1 ~ G;

[0074] c. Zero-variance sample filtering: Calculate the reward value of each prompt G responses (based on the verifiable answer to judge the correctness, correct is 1, error is 0), filter out the prompts with all 0 or all 1 reward values (this embodiment is called: zero-variance sample), get the filtered sample set , where i=1~B'_r, j=1~G, where B'_r is the number of prompt words / questions retained after filtering.

[0075] C. Length-aware dynamic filtering.

[0076] a. Average response length calculation. Calculate the average length of the G responses of each filtered prompt word / question , Its specific expression is the average length of the G responses corresponding to the ith prompt word), it needs to be mentioned that the corresponding length is measured in token number: .

[0077] b. Length distribution statistics. Based on all the retained prompt words / questions , calculate the cumulative distribution function of length And quantile function .

[0078] Specific explanation as follows: cumulative distribution function: , Indicates the proportion of prompt words / questions with average response length less than or equal to t, where , R is a real number.

[0079] Quantile function: , Indicates the average response length corresponding to the Quantile, inf is the lower bound. For example, , that is, the average response length of the prompt words / questions at the 30% average response length position).

[0080] c. Length filtering. According to the preset L_low (length filtering lower threshold), L_high (length filtering upper threshold), L_max (maximum length filtering threshold), retain the prompt words / questions that meet the following conditions: .

[0081] That is, retain the prompt words / questions with average response length in the shortest L_low interval or L_high~L_max quantile interval, get the final filtered sample set , where i=1~B''_r, j=1~G, where B''_r is the number of prompt words / questions retained after length filtering.

[0082] d. Training sample construction. The current rolling sampling and the filtered sample set are combined to form the training sample pool B. where i = 1 ~ B''_r, j = 1 ~ G, add the training sample pool B, repeat steps a~c until the number of samples in the sample pool B reaches the training batch size B_t; if the sample pool exceeds B_t, randomly select B_t samples to form the final training batch.

[0083] D. Model training update.

[0084] a. Advantage function calculation. Based on the reward value of G responses of each prompt / question in the sample pool, the standardized reward function is calculated, and the calculation of the reward function needs to be combined with the selected reinforcement learning method with verifiable reward, for example, for GRPO and DAPO, the calculation methods of the reward function and the length penalty function are different.

[0085] b. Loss calculation and parameter update. Based on the reward function, the loss function is calculated based on the selected reinforcement learning method with verifiable reward, the loss is calculated using the collected training batch data, and the model parameters are updated by gradient descent.

[0086] c. Training iteration: repeat steps B~D until the preset training time or iteration number is reached, and finally select the checkpoint to evaluate the model performance.

[0087] As Figure 2 , another feature of the length-aware dynamic sampling strategy proposed in this embodiment is that it can be combined with any reinforcement learning training method with verifiable reward. The framework of this method includes: a rolling sampling module: generating multiple responses of the prompt / question; a preliminary filtering module: filtering zero variance samples; a length-aware filtering module: performing dynamic quantile-based length screening; a sample collection module: collecting the screened samples until the training batch size is met; a parameter update module: outputting the collected samples to the reinforcement learning training method with verifiable reward for loss calculation and parameter update. Through the above summary, it can be found that the strategy proposed in this embodiment is completely decoupled from the reinforcement learning training method with verifiable reward, and has universality and practicality.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0089] As Figure 3The embodiment of the present application further provides an electronic device, comprising a memory 101 and a processor 102, the memory 101 stores a computer program, and the processor 102 is configured to execute the computer program to perform the steps in any of the text generation method embodiments.

[0090] As Figure 4 The embodiment of the present application further provides a computer readable storage medium 201, which stores a computer program 202, wherein the computer program 202 is configured to perform the steps in any of the text generation method embodiments when executed.

[0091] In an example embodiment, the computer readable storage medium 201 can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0092] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the text generation method embodiments.

[0093] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the text generation method embodiments.

[0094] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0095] The above describes in detail the text generation method and the electronic device provided by the present application. The principles and implementation manners of the present application are described by using specific examples, and the above description of the embodiments is only applicable to help understand the method of the present application and the core idea thereof. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A text generation method characterized by, The method comprises the following steps: determining input target question text; inputting the target question text into a target text generation model to obtain target response text matched with the target question text; the target text generation model is a model obtained by parameter optimization of a current text generation model through target training samples, the target training samples are samples dynamically and adaptively filtered from a plurality of training samples based on the length distribution of a plurality of response texts generated by the current text generation model at the current stage, each of the training samples comprises a question text and a plurality of response texts generated by the current text generation model based on the question text; the filtering process of the target training samples comprises: calculating the average length of the plurality of response texts corresponding to each question text; based on the average length, calculating the length quantile under the current training batch; according to a preset quantile threshold, filtering the training samples with the average length in the predetermined quantile interval as the target training samples; according to a preset quantile threshold, filtering the training samples with the average length in the predetermined quantile interval as the target training samples, comprising: according to a preset first quantile, a second quantile and a maximum quantile, retaining the training samples with the average length less than or equal to the first quantile, or the average length greater than or equal to the second quantile and less than or equal to the maximum quantile as the target training samples; the first quantile < the second quantile < the maximum quantile.

2. The text generation method of claim 1, wherein, based on the average length, calculating the length quantile under the current training batch, comprising: calculating the cumulative distribution function value according to the average length, the cumulative distribution function value representing the proportion of question texts with an average length less than or equal to a given length; calculating the quantile function according to the cumulative distribution function value, the quantile function being used to determine the corresponding length value according to a given quantile.

3. The text generation method of claim 1, wherein, after inputting the target question text into the target text generation model to obtain the target response text matched with the target question text, further comprising: obtaining the accuracy of the target response text output by the target text generation model multiple times; adjusting the predetermined quantile interval according to the accuracy.

4. The text generation method of claim 1, wherein, after inputting the target question text into the target text generation model to obtain the target response text matched with the target question text, further comprising: obtaining the response rate of the target response text output by the target text generation model multiple times; adjusting the predetermined quantile interval according to the response rate.

5. The text generation method of claim 1, wherein, before dynamically and adaptively filtering the samples from a plurality of training samples based on the length distribution of a plurality of response texts generated by the current text generation model at the current stage, further comprising: for each question text, calculating the reward value of the corresponding plurality of response texts based on the corresponding verifiable response text, filtering the training samples corresponding to the question texts with all zero or all one reward values to obtain a filtered sample set; the target training samples are specifically samples filtered from the filtered sample set based on the length distribution of the plurality of response texts.

6. The text generation method of claim 1, wherein, further comprising: adding a data set corresponding to the target training sample to a training sample pool until a number of samples in the training sample pool is greater than or equal to a training batch size; if the number of samples in the training sample pool is greater than the training batch size, randomly selecting training samples of the training batch size from the training sample pool to obtain a final training batch; optimizing parameters of the current text generation model using the target training sample, including: optimizing the parameters of the current text generation model using the training samples in the final training batch.

7. The text generation method of any one of claims 1-6, wherein, a generation process of the target text generation model, including: obtaining reward values of a plurality of response texts corresponding to each question text in the target training sample, and calculating a reward function according to the reward values; calculating a loss function based on the reward function, and updating the parameters of the current text generation model by gradient descent according to the loss function to obtain a target text generation model.

8. An electronic device, comprising: comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the text generation method according to any one of claims 1 to 7.

Images