Model training method, model fine-tuning method, and device

Abstract

Embodiments of the present specification provide a model training method, a model fine-tuning method and apparatus, wherein the model training method comprises: obtaining output text, wherein the output text is output after input data is processed by a current policy model; performing repetition rate identification on each text unit in the output text to obtain an identification result; determining that the output text is abnormal text if the identification result exceeds a preset threshold; assigning a negative reward of abnormal text to the abnormal text; and performing policy gradient updating on the current policy model based on the negative reward of the abnormal text until a target policy model trained is obtained. In the case of not relying on an external discrimination model or manual annotation, whether the model generates degenerative content is directly detected by using the repetition rate characteristics of the output text itself, and a negative reward punishment mechanism is used to guide the policy model to avoid generating abnormal text with high repetition rate, thereby effectively alleviating the reward error and repetition degeneration problem of the policy model in the reinforcement learning training process.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of large model technology, and in particular to a model training method, a model fine-tuning method, a model training device, and a model fine-tuning device. Background Technology

[0002] In recent years, reinforcement learning-based training methods for large language models have become a key technical path for improving model capabilities. In reinforcement learning, the quality of the reward signal directly determines the evolution direction and final performance of the policy model. Early on, rule-based reward schemes were widely adopted. However, with the increasing complexity of tasks, the coverage of rule-based rewards has become increasingly limited, making it difficult to effectively handle long-tail scenarios such as open-ended question answering, semantic logic derivation, and subjective preference evaluation. Its insufficient semantic expressive power has gradually become apparent.

[0003] To overcome the limitations of rule-based rewards in semantic coverage, the industry has generally shifted to using large language models as referees to provide reward signals. However, this mechanism introduces the extremely challenging problem of reward cheating: agents may exploit logical loopholes or illusions in the referee model to obtain false high scores. Erroneous reward samples constitute an extremely small percentage of massive datasets, completely undetectable by traditional macro-statistical indicators; simultaneously, the high-temperature randomness of the reinforcement learning sampling phase leads to the sporadic triggering of the same logical flaw, making anomalies difficult to reproduce and locate. This makes reward cheating extremely difficult to effectively identify and correct with current technology.

[0004] Therefore, there is an urgent need for a mechanism that can identify abnormal outputs in real time and with lightweight operation and correct reward signals immediately to ensure the stability and reliability of reinforcement learning training. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a model training method. One or more embodiments of this specification also relate to a model fine-tuning method, a model training apparatus, a model fine-tuning device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a model training method is provided, comprising: Obtain the output text, where the output text is generated by the current policy model after processing the input data; The repetition rate of each text unit in the output text is identified to obtain the identification result. If the recognition result exceeds the preset threshold, the output text is determined to be abnormal text; Assign negative rewards to anomalous text; The policy gradient is updated based on the negative reward of the abnormal text until the target policy model is obtained after training.

[0007] According to a second aspect of the embodiments of this specification, a model fine-tuning method is provided, comprising: Obtain the supervised dataset, which is constructed according to the model training method; The current discrimination model is fine-tuned under supervision based on the supervised dataset until a fully trained target discrimination model is obtained.

[0008] According to a third aspect of the embodiments of this specification, a model training apparatus is provided, comprising: The text acquisition module is configured to acquire output text, where the output text is generated by the current policy model after processing the input data. The repetition rate recognition module is configured to identify the repetition rate of each text unit in the output text and obtain the recognition result. The judgment module is configured to determine that if the recognition result exceeds a preset threshold, the output text is abnormal text. The reward allocation module is configured to assign negative rewards to abnormal text. The policy gradient update module is configured to update the policy gradient of the current policy model based on the negative reward of the abnormal text until the target policy model that has been trained is obtained.

[0009] According to a fourth aspect of the embodiments of this specification, a model fine-tuning device is provided, comprising: The dataset acquisition module is configured to acquire supervised datasets, which are constructed according to the model training method. The supervised fine-tuning module is configured to perform supervised fine-tuning of the current discriminant model based on a supervised dataset until a fully trained target discriminant model is obtained.

[0010] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned model training method or model fine-tuning method.

[0011] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method or model fine-tuning method.

[0012] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method or model fine-tuning method.

[0013] The model training method provided in one or more embodiments of this specification includes obtaining output text, wherein the output text is output by the current policy model after processing the input data; performing repetition rate identification on each text unit in the output text to obtain the identification result; if the identification result exceeds a preset threshold, determining that the output text is abnormal text; assigning negative rewards to the abnormal text; and updating the policy gradient of the current policy model based on the negative rewards of the abnormal text until the target policy model that has been trained is obtained.

[0014] In this embodiment, the output text generated by the current policy model is obtained, and the repetition rate of each text unit in the output text is identified to obtain the identification result. When the identification result exceeds a preset threshold, the output text is determined to be abnormal text and a negative reward is assigned to the abnormal text. Then, the policy gradient of the current policy model is updated based on the negative reward until the target policy model is trained. Thus, without relying on external discriminant models or manual annotation, the repetition rate feature of the output text itself can be used to detect whether the model generates degenerate content. The penalty mechanism of negative reward guides the policy model to avoid generating abnormal text with high repetition rate, thereby effectively alleviating the reward error and repetition degradation problems common in the reinforcement learning training process of policy models, improving the generation diversity and output quality of the model. At the same time, since the repetition rate identification only needs to be calculated based on the output text itself, without the need for additional model or label data, it has the beneficial effects of high computational efficiency, easy implementation and strong versatility. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a model training method provided in one embodiment of this specification; Figure 2 A flowchart illustrating the processing steps of a model training method provided in one embodiment of this specification; Figure 3 An anomaly detection flowchart for a model training method provided in one embodiment of this specification; Figure 4 A flowchart illustrating a model fine-tuning method provided in one embodiment of this specification; Figure 5 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification; Figure 6 A schematic diagram of a model fine-tuning device provided in one embodiment of this specification; Figure 7This is a structural block diagram of a computing device provided for one embodiment of this specification. Detailed Implementation

[0016] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0017] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0018] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0019] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0021] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0022] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0023] Large Language Model (LLM): This is an artificial intelligence model based on deep learning (usually based on the Transformer architecture). It is pre-trained on massive amounts of text data and can perform a wide range of tasks such as text generation, reasoning, and question answering.

[0024] Reinforcement learning (RL) is a machine learning paradigm in which an agent optimizes its policy by interacting with its environment and receiving reward signals. In training large language models, RL is used to align the model's output with human preferences or task objectives.

[0025] Reward Model: In reinforcement learning-based language model training, this is a module used to score the model's output, which serves as a reward signal for policy optimization. The quality of the reward model directly affects the training results.

[0026] Policy entropy is a metric that measures the diversity of the output distribution of a language model. Higher entropy indicates greater "uncertainty" in model generation and more dispersed outputs; lower entropy indicates more concentrated and definite model outputs. In reinforcement learning training, policy entropy is an important monitoring metric reflecting the model's behavioral state.

[0027] Garbled output: Abnormal output generated by the language model containing a large number of meaningless characters, repetitive fragments, or non-natural language content. Common causes include: training data containing special Unicode characters (such as `\xa0`), abnormal decoding policies, and policy collapse during reinforcement learning training.

[0028] n-gram duplication detection: a method to determine whether there are abnormal duplication patterns in text by counting the frequency of repetition of n consecutive tokens in the text. It is often used to detect degenerate output of models.

[0029] SAPO / GSPO: Variants of the reinforcement learning algorithm used in the experiments involved in this invention, used to train the mathematical reasoning ability of the language model.

[0030] Stochastic Advantage Policy Optimization (SAPO) is a policy optimization method that combines stochastic policies with advantage function estimation. By introducing an advantage function, it reduces the variance of policy gradient estimation, thereby improving learning stability and sample efficiency.

[0031] Generalized Stochastic Policy Optimization (GSPO) is an extended framework for stochastic policy optimization that unifies various policy gradient algorithms (such as REINFORCE, PPO, and TRPO) and adapts to different task requirements through a generalized objective function or update rule.

[0032] Infinity (inf): In floating-point arithmetic, it represents positive infinity. In this invention, it specifically refers to the "infinity" outlier value that appears in the policy entropy value during training, which is an important indication that the reward signal is erroneous.

[0033] Policy entropy (actor / entropy) is a measure of the uncertainty of the model's output probability distribution during reinforcement learning training. It can be recorded in real time during training without additional inference overhead. This invention, through statistical analysis, reveals that in training steps with abnormal policy entropy (persistently high entropy or becoming inf), the proportion of abnormal reward signals is significantly higher than in training steps with normal policy entropy. This pattern indicates that abnormal policy entropy can serve as an effective online indicator of erroneous reward signals, compressing the problem detection time from "post-evaluation" to "during training."

[0034] The calculation of policy entropy depends on the model's probability distribution output of the entire vocabulary, which requires preserving the complete logit (logistic regression) vector and intruding into the training framework, resulting in certain engineering costs. To address this, this invention further analyzes the model output corresponding to the anomalous steps of policy entropy and finds that it highly co-occurs in temporal distribution with degenerate sequences containing significant repetitive patterns. The essential textual feature of such degenerate sequences is n-gram-level repetition, that is, several consecutive token fragments are repeatedly stacked.

[0035] Based on this discovery, this invention designs a lightweight text pattern detection function that directly analyzes the text content output by the model without accessing the model's internal state. The processing flow is as follows: This specification provides a model training method. One or more embodiments of this specification also relate to a model fine-tuning method, a model training apparatus, a model fine-tuning device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0036] See Figure 1 , Figure 1 A flowchart of a model training method provided in one embodiment of this specification includes the following steps: Step 102: Obtain the output text, where the output text is output by the current policy model after processing the input data.

[0037] The current policy model is a machine learning model used to generate output text from input data. The current policy model can be a model to be trained. Specifically, it can include a generative language model based on the Transformer architecture, a recurrent neural network model, or a convolutional neural network model. The current policy model receives input data and calculates the probability distribution of each text unit in the output text based on the input data, then generates the output text through sampling or greedy search. Furthermore, during training, the current policy model updates its parameters using a policy gradient algorithm to maximize the cumulative reward.

[0038] Output text is a sequence of texts produced by the current policy model after processing the input data. Specifically, output text can include strings, sentences, paragraphs, or complete documents composed of multiple text units in the order they were generated. Output text can be used to represent the generation results of the current policy model given the input data, such as a response to a dialogue context, an answer to an instruction, a continuation paragraph to a prompt, or an analysis of an input question or topic and the output of analytical text. During training, the output text is fed into the reward calculation module or the discriminant model to evaluate its quality and guide model updates.

[0039] In practical applications, there are several ways to obtain the output text in model training methods. One possible approach is to obtain pre-stored input data, use the current policy model to directly perform forward computation on the input data to obtain the probability distribution of each text unit in the output text, then sample the text unit at each position from this probability distribution, and finally concatenate them to form the output text.

[0040] Another alternative approach is to output text unit by unit during the current strategy model's output text generation process, and collect the generated text units in real time until a termination symbol is encountered or the maximum length limit is reached, thereby obtaining the complete output text.

[0041] Another option is to select multiple candidate paths with the highest probabilities from the probability distribution, and then choose one of these paths as the final output text.

[0042] Alternatively, a greedy search method can be used to directly select the text unit with the highest probability at each position and generate the output text in sequence.

[0043] This step obtains the output text generated by the current policy model after processing the input data, providing a basic data source for subsequent repetition rate identification and anomaly detection. This allows the model training process to directly use the model's own generated results for quality assessment without relying on external annotations, thereby achieving real-time monitoring and quantifiable analysis of the policy model's generation behavior. This lays the data foundation for subsequent allocation of negative rewards and execution of policy gradient updates, while avoiding the introduction of additional inference overhead or manual intervention, thus improving the automation level of the training process and the convenience of data acquisition.

[0044] Step 104: Perform repetition rate identification on each text unit in the output text to obtain the identification results.

[0045] A text unit is the smallest unit of granularity that constitutes the output text. Specifically, a text unit can include characters, subwords, lexical units, or vocabulary. Text units can be used to break down the output text into discrete basic elements for purposes such as repetition rate identification, statistical calculation, or sequence modeling. Furthermore, text units are typically obtained by a word segmenter dividing the output text according to a pre-defined vocabulary.

[0046] Repetition rate identification is the process of detecting and quantifying the degree of repetition among text units in an output text. Specifically, repetition rate identification can include statistically analyzing the frequency of identical text units in the output text, detecting continuous or discontinuous repetition patterns, or calculating the proportion of repetitive segments based on a sliding window. Repetition rate identification can be used to determine whether the output text exhibits patterned, cyclical, or degenerate generation phenomena. Repetition rate identification outputs a numerical result that reflects the overall degree of repetition in the output text.

[0047] The identification result is a quantitative value used to measure the degree of repetition in the output text, which can be obtained from the repetition rate identification process. Specifically, the identification result may include the ratio of the number of repetitive segments to the total number of text units, the frequency of occurrence of repetitive text units, or a repetition index calculated based on information entropy. The identification result can be used to compare with a preset threshold to determine whether the output text belongs to anomalous text.

[0048] In practical applications, there are several ways to identify the repetition rate of each text unit in the output text to obtain the recognition result. One possible method is to count the frequency of different text units in the output text, calculate the proportion of repetitive text units to the total number of text units, and use this proportion as the recognition result.

[0049] Another alternative approach is to set a fixed-length sliding window to segment the output text into multiple consecutive text unit fragments, then detect whether each fragment is the same as or highly similar to other fragments, and count the ratio of the number of duplicate fragments to the total number of fragments as the recognition result.

[0050] Another option is to calculate the longest length of consecutive repeating subsequences in the output text, or to calculate the autocorrelation matrix of the output text, and obtain a quantitative indicator of the degree of repetition based on the autocorrelation value. Alternatively, the output text can be identified based on information entropy; the lower the entropy value of the output text, the higher the degree of repetition, and the normalized entropy value is used as the identification result.

[0051] This step identifies the repetition rate of each text unit in the output text, obtaining the identification result used to measure the degree of repetition in the output text. This enables the model training process to automatically detect whether the current policy model generates patterned, cyclical, or degenerate anomalous content. Thus, the generation quality can be quantitatively evaluated without relying on external discriminant models or manual annotation, providing a clear and calculable basis for subsequent anomaly detection and negative reward allocation. At the same time, since the repetition rate identification is based solely on statistical calculations of the output text itself, it has the advantages of low computational cost and strong real-time performance, effectively supporting the efficient penalty mechanism in the policy gradient update process.

[0052] Step 106: If the recognition result exceeds the preset threshold, the output text is determined to be abnormal text.

[0053] A preset threshold is a pre-defined critical value used to determine whether the output text is abnormal. Specifically, the preset threshold can be a fixed real number or an adaptive threshold that is dynamically adjusted based on the recognition results statistically analyzed during training. The preset threshold can be used to divide the recognition results into two intervals: exceeding the threshold corresponds to an abnormal situation, and not exceeding the threshold corresponds to a normal situation. The preset threshold can be adjusted according to the task type, the size of the strategy model, or the length of the generated text.

[0054] Anomalous text refers to output text whose recognition results exceed a preset threshold. It indicates that the current policy model has generated degenerate content with an excessively high repetition rate. Furthermore, anomalous text often exhibits reward anomalies. Specifically, anomalous text can include sentences that cyclically repeat the same phrase, paragraphs that continuously output the same sub-word sequences, or generated results lacking diversity and exhibiting fixed patterns. Anomalous text can be used to trigger negative reward allocation mechanisms, thereby guiding the policy model to avoid generating such degenerate content. Anomalous text can also be referred to as repetitive degenerate samples or pattern collapse samples.

[0055] In practical applications, there are several ways to determine whether the recognition result exceeds a preset threshold and to classify it as abnormal text. One option is to set the preset threshold to a fixed value. When the recognition result is greater than the fixed value, the output text is determined to be abnormal text; otherwise, it is determined to be normal text.

[0056] Another option is to use an adaptive threshold, for example, by calculating the mean and standard deviation of the recognition results in the most recent batches, setting the threshold to the mean plus twice the standard deviation, and determining the current recognition result as abnormal text when it exceeds this dynamic threshold.

[0057] Another option is to set multiple threshold ranges, such as a first threshold and a second threshold. When the recognition result exceeds the first threshold but does not exceed the second threshold, it is judged as slightly abnormal text and a small negative reward absolute value is assigned. When the recognition result exceeds the second threshold, it is judged as severely abnormal text and a larger negative reward absolute value is assigned.

[0058] This step compares the recognition results with a preset threshold and determines that the output text exceeding the threshold is abnormal text. This achieves automated anomaly detection and classification of the content generated by the current policy model, enabling the training process to clearly distinguish between normally generated samples and degenerate samples. This provides clear triggering conditions and judgment criteria for subsequently allocating negative rewards to abnormal text, avoiding the tedious operation of manually reviewing each output text. At the same time, because the threshold judgment method is simple and efficient, it can quickly complete the anomaly judgment in each iteration of reinforcement learning training, effectively improving the real-time performance and automation of quality monitoring during model training.

[0059] Step 108: Assign negative rewards to anomalous text.

[0060] The negative reward for anomalous text is a negative reward signal assigned to it. Specifically, the negative reward for anomalous text can be a negative constant, such as -1, -0.5, or -10; it can also be a negative value that penalizes the degree of repetition, with a larger absolute value of the negative reward for higher repetition rates. The negative reward for anomalous text can be used to reduce the generation probability of anomalous text during policy gradient updates, thereby suppressing the policy model from producing similar outputs. Furthermore, the negative reward for anomalous text can be weighted and combined with other rewards (such as the positive or negative rewards given by the discrimination model for normal text) to form a total reward value for model updates.

[0061] In practical applications, there are several ways to assign negative rewards to anomalous text. One option is to directly assign a fixed negative constant value as the negative reward for the anomalous text.

[0062] Another option is to calculate the negative reward based on the specific numerical value of the recognition result. For example, the negative reward is equal to the negative recognition result multiplied by a penalty coefficient, so that the higher the repetition rate, the larger the absolute value of the negative reward.

[0063] Another option is to set the negative reward as the opposite of the average reward for normal text in the current batch, thereby penalizing abnormal text in a relative sense.

[0064] In addition, the value of the negative reward can be dynamically adjusted according to the number of training steps of the current strategy model. Smaller negative rewards are used in the early stage of training to avoid drastic model jitter, while larger negative rewards are used in the later stage of training to enhance the penalty effect.

[0065] This step assigns a negative reward to output text identified as anomalous, enabling the current policy model to receive a clear penalty signal in subsequent policy gradient updates. This reduces the generation probability of anomalous text and effectively suppresses the generation of degenerate content with excessive repetition. Furthermore, since the allocation of negative rewards is directly linked to the anomaly detection result, no additional training or adjustment of other model parameters is required. This approach enables negative guidance of the policy model's generation behavior with low computational cost within the reinforcement learning framework, improving the convergence stability and output quality of model training.

[0066] Step 110: Update the policy gradient of the current policy model based on the negative reward of the abnormal text until the target policy model is obtained after training.

[0067] Policy gradient update is the process of iteratively optimizing the parameters of the current policy model using a policy gradient algorithm. Specifically, policy gradient update can include calculating the gradient of the reward value relative to the model output, backpropagating the gradient to the model parameters, and updating the parameter values ​​using an optimizer (such as Adam or SGD). Policy gradient update can be used to adjust the behavior of the current policy model, making it more likely to generate high-reward output text while avoiding generating anomalous text corresponding to negative rewards. Furthermore, policy gradient update can be based on policy optimization or on advantage function estimation.

[0068] The target policy model is a trained policy model obtained after the current policy model has undergone policy gradient updates until a stopping condition is met. Specifically, the target policy model can be one that has reached a preset number of training steps, has a policy entropy no lower than a preset entropy threshold, or generates stable models with low repetition rates on the validation set. The target policy model can be deployed in real-world applications, such as dialogue systems, text generation tasks, or content creation aids. Furthermore, compared to the current policy model, the target policy model exhibits a lower probability of generating anomalous text and higher output diversity.

[0069] In practical applications, there are several ways to update the policy gradient of the current policy model based on the negative reward of the abnormal text until the target policy model is obtained after training. One option is to directly use the negative reward of the abnormal text as the reward signal in the policy gradient update, calculate the policy gradient and update the model parameters; then continue to acquire new output text, repeat the recognition and update process until a preset total number of training steps is reached or the repetition rate of the model on the validation set is lower than a preset threshold. At this point, training is stopped and the current policy model is used as the target policy model.

[0070] Another option is to center the negative reward by combining it with the advantage function. That is, the difference between the negative reward and the baseline value is calculated as the advantage value, and the policy gradient is updated based on the advantage value to reduce the variance of the gradient estimation.

[0071] Another option is to collect multiple output texts and their corresponding negative rewards in each training batch, calculate the average reward or cumulative reward, and then use a near-end policy optimization algorithm for multiple rounds of updates, limiting the magnitude of parameter changes in each round of updates.

[0072] In addition, the policy entropy of the policy model can be monitored simultaneously during the policy gradient update process. When the policy entropy is lower than the preset entropy threshold, training can be terminated in advance to avoid the model falling into a degenerate mode too early, and the current model can be used as the target policy model.

[0073] This embodiment of the specification obtains the output text generated by the current policy model and performs repetition rate identification on each text unit in the output text to obtain the identification result. When the identification result exceeds a preset threshold, the output text is determined to be abnormal text and a negative reward is assigned to the abnormal text. Then, the policy gradient of the current policy model is updated based on the negative reward until the target policy model is trained. In this way, without relying on external discriminant models or manual annotation, the repetition rate feature of the output text itself can be used to detect whether the model generates degenerate content. The penalty mechanism of negative reward guides the policy model to avoid generating abnormal text with high repetition rate, thereby effectively alleviating the pattern collapse and repetition degradation problems that are common in the reinforcement learning training process of policy models, improving the generation diversity and output quality of the model. At the same time, since the repetition rate identification only needs to be calculated based on the output text itself, without the need for additional model or label data, it has the beneficial effects of high computational efficiency, easy implementation and strong versatility.

[0074] In one optional embodiment of this specification, repetition rate identification is performed on each text unit in the output text to obtain the identification result, including: The text units in the output text are divided into multiple text unit slices by using a sliding window for text units. Identify duplicate fragments in each text unit fragment, where a duplicate fragment is a text unit fragment that is duplicated with other text unit fragments; The duplication rate of each duplicate fragment is obtained by statistical analysis.

[0075] A text unit sliding window is a fixed-length window used to segment text units in the output text. Specifically, the text unit sliding window can have a preset window size, such as 3, 5, or 10 text characters (i.e., 3-gram, 5-gram, 10-gram, etc.). The text unit sliding window slides along the text unit sequence of the output text according to a step size, segmenting the continuous text units covered by the window each time to form text unit slices. The window size and sliding step size of the text unit sliding window can be set according to the length of the output text, the type of text units, or the accuracy requirements for repetition rate recognition.

[0076] A text unit slice is a continuous subsequence of text units obtained by dividing the output text into individual text units using a sliding window. A text unit slice can include multiple text units arranged sequentially within the window, such as a slice consisting of three consecutive text characters, a slice consisting of five consecutive text characters, and so on. Text unit slices can be used as the basic comparison unit for repetition rate identification, determining the existence of repetitive patterns by comparing the content similarity or completeness between different slices. Furthermore, all text unit slices collectively cover all or part of the text units in the output text.

[0077] A repeating segment is a text unit segment that repeats content from at least one other text unit segment. Specifically, repeating segments can include segments that contain exactly the same content as another segment, segments that are highly similar to another segment (e.g., the edit distance is less than a preset threshold), or segments that have an inclusion relationship with another segment (e.g., one segment is a subsequence of another segment). Repeating segments can be used to count the number and distribution of repeating patterns in the output text, thereby calculating the repetition rate. Furthermore, a text unit segment may repeat multiple other segments, and that segment is still marked as a repeating segment.

[0078] The repetition rate is a numerical value used to quantify the proportion or degree of repetition of each repetitive segment within all text unit segments. Specifically, the repetition rate can include the number of repetitive segments, the ratio of the number of text units covered by repetitive segments to the total number of text units, or a weighted composite index of the number of repetitions of each repetitive segment. The repetition rate can reflect the number of repetitive patterns in the output text or the overall proportion of repetitive patterns; a higher value indicates a more severe degree of repetition degradation in the output text. Furthermore, the repetition rate can be compared with a preset threshold to determine whether the output text belongs to abnormal text.

[0079] In practical applications, there are several ways to identify duplicate fragments in each text unit segment. One possible method is to compare all text unit segments pairwise; if two segments contain identical content, then both segments are marked as duplicate fragments.

[0080] Another option is to calculate the similarity between each pair of text unit segments (e.g., based on edit distance or Jaccard similarity coefficient), and determine that the two segments are duplicate segments when the similarity exceeds a preset similarity threshold. Alternatively, only continuous repetition patterns can be detected, that is, only those segments that are identical to adjacent or fixed-distance text unit segments are identified as duplicate segments.

[0081] In practical applications, there are several ways to obtain the repetition rate of each repeated fragment. One possible method is to count the number of repeated fragments and use that number as the repetition rate.

[0082] Another option is to count the number of repetitions for each repetitive segment separately, then calculate the average, maximum, or sum of the number of repetitions for all repetitive segments, and then normalize the result by combining this with the length of the output text to obtain the repetition rate.

[0083] Another possible approach is to set n as the window size (i.e., each slice contains n text units), count all consecutive slices of length n in the output text, and calculate the number K of repeated slices that appear more than once, or calculate the ratio of the total number of repeated slices to the total number of slices.

[0084] This specification describes an embodiment that segments the output text into multiple text unit fragments by using a sliding window to divide the text unit into segments. It then identifies repetitive fragments within each text unit fragment and calculates the repetition rate of each fragment. This allows for fine-grained detection and quantification of repetitive patterns in the output text at the fragment level. Compared to simply counting the repetition frequency of a single text unit, the sliding window-based fragmentation method can capture repetitive patterns composed of multiple consecutive text units, thus more accurately identifying structured degradation phenomena such as cyclic phrases and repetitive substrings. Furthermore, by calculating the repetition rate by comparing the number of repetitive fragments to the total number of fragments, it provides a stable, comparable, and computationally efficient quantitative indicator for subsequent anomaly detection.

[0085] In one optional embodiment of this specification, after identifying the repetition rate of each text unit in the output text and obtaining the identification result, the method further includes: If the recognition result does not exceed the preset threshold, the output text is determined to be normal text; Using the current discrimination model, normal text is identified, and a reward value for normal text is obtained; The policy gradient is updated based on the negative reward of the abnormal text until the trained target policy model is obtained, including: Based on the reward value of normal text and the negative reward of abnormal text, the policy gradient of the current policy model is updated until the target policy model is obtained after training.

[0086] Normal text refers to the output text whose recognition results do not exceed a preset threshold. Normal text is the non-degenerate content generated by the current policy model, and its reward signal can be considered correct. Specifically, normal text can include semantically fluent and diverse sentences, structurally sound paragraphs without abnormal loops, or natural language generation results that conform to the expected distribution. Normal text can be used to trigger the evaluation mechanism of the discriminative model, thereby obtaining corresponding reward values ​​to guide the policy model to update towards a better generation direction. Furthermore, normal text and abnormal text together constitute all the output text generated by the current policy model, with normal text dominating indicating a good model training state.

[0087] The current discriminant model is a machine learning model used to evaluate the quality of normal text and output a reward value. Specifically, the current discriminant model can include a binary classification model based on the Transformer architecture, a discriminator based on a convolutional neural network, or a reward model based on contrastive learning. The current discriminant model can take normal text as input and output a scalar value as a reward for the normal text, which reflects the quality of the normal text or its degree of conformity to the expected distribution. Furthermore, the current discriminant model can be trained alternately with the current policy model, or it can be pre-trained through supervised fine-tuning.

[0088] The reward value for normal text is a numerical signal output by the current discriminative model after classifying normal text, used to quantitatively evaluate the quality of normal text. Specifically, the reward value for normal text can be a positive number (representing high-quality text), a negative number (representing low-quality text but not reaching the anomaly threshold), or zero. The reward value for normal text can be combined with the negative reward for anomaly text to form a total reward value to guide policy gradient updates. Furthermore, the range of values ​​for the reward value for normal text can be set according to the design of the discriminative model.

[0089] There are several ways to determine whether the output text whose recognition result does not exceed a preset threshold is normal text. One option is to set the preset threshold to a fixed value, and when the recognition result is less than or equal to the threshold, the output text is directly determined to be normal text.

[0090] Another option is to dynamically adjust the preset threshold based on the length of the output text, using a lower threshold for long texts and a higher threshold for short texts, and then compare the recognition result with the dynamic threshold to determine whether it is normal text.

[0091] Another option is to combine policy entropy for comprehensive judgment. When the recognition result does not exceed the preset threshold and the policy entropy is higher than the preset lower limit, it is judged as normal text.

[0092] There are several ways to use the current discriminant model to classify normal text and obtain a reward value for it. One possible approach is to directly input the normal text into the current discriminant model, and the model will output a number as the reward value.

[0093] Another option is to input normal text along with the input data into the current discriminative model, allowing the model to make more accurate evaluations by incorporating context. For example, in a dialogue generation task, input the dialogue history and the generated responses, and the discriminative model outputs a reasonableness score for the responses.

[0094] Another option is to first extract features from the normal text, and then input the feature vector into the output layer of the discriminant model to obtain the reward value.

[0095] This embodiment determines the output text as normal text when the recognition result does not exceed a preset threshold, and uses the current discriminant model to distinguish normal text to obtain a reward value for normal text. Then, based on the reward value of normal text and the negative reward of abnormal text, the current policy model is updated with a policy gradient. This combines the repetition rate self-checking mechanism with the evaluation mechanism of the discriminant model, so that the policy model can obtain fine-grained quality feedback from the discriminant model to improve the semantic rationality and diversity of the generated content, and can effectively suppress the repetition degradation phenomenon through the negative reward of abnormal text. This achieves the synergistic optimization of positive guidance for normal samples and negative punishment for abnormal samples, improving the stability of model training, convergence speed and final generation quality.

[0096] One optional embodiment of this specification utilizes the current discrimination model to distinguish normal text and obtain a reward value for the normal text, including: Using the current discrimination model, normal text is judged, and a positive reward or a negative reward is obtained for normal text. Based on the reward value of normal text and the negative reward of abnormal text, the policy gradient of the current policy model is updated until the target policy model is obtained after training, including: The total reward value is obtained by weighting the positive or negative rewards for normal text and the negative rewards for abnormal text. Based on the total reward value, the policy gradient of the current policy model is updated until the target policy model is obtained after training.

[0097] The positive reward for normal text is a positive reward signal output by the current discriminative model after classifying normal text. Specifically, the positive reward for normal text can be a real number greater than zero, used to characterize that the normal text is of high quality or conforms to the expected distribution. The positive reward for normal text can be used to increase the probability of the corresponding generation path in policy gradient updates, thereby guiding the current policy model to tend to generate high-quality normal text. Furthermore, the positive reward for normal text is usually contrasted with the negative reward for normal text, and both are output by the same current discriminative model based on the quality of the normal text.

[0098] The negative reward for normal text is a negative reward signal output by the current discrimination model after classifying normal text. Specifically, the negative reward for normal text can be a real number less than zero, used to characterize the low quality of normal text. The negative reward for normal text can be used to moderately reduce the probability of the corresponding generation path in policy gradient updates, suppressing the policy model from generating low-quality but not yet degraded content.

[0099] The difference between negative rewards for normal text and negative rewards for abnormal text is that the former applies to normal text with a repetition rate that does not exceed the threshold but has low quality, while the latter applies to abnormal text with a repetition rate that exceeds the threshold.

[0100] Weighting is the process of combining multiple reward values ​​according to their respective weight coefficients to obtain a total reward value. Specifically, weighting may include assigning a first weight to the positive reward of normal text, a second weight to the negative reward of normal text, and a third weight to the negative reward of abnormal text, then multiplying each reward value by its corresponding weight and summing the results. Weighting can be used to adjust the relative importance of different types of reward signals in policy gradient updates. Furthermore, the weight coefficients can be preset hyperparameters or dynamically adjusted according to the training progress.

[0101] The total reward value is a single numerical value used for policy gradient updates, combining the positive or negative rewards of normal text and the negative rewards of anomalous text. Specifically, the total reward value can be equal to the weighted sum, or it can be a value obtained by cropping or normalizing the weighted result. The total reward value can be used to directly replace the original reward value in policy gradient calculation, thereby handling high-quality text, low-quality text, and anomalous text in normal samples simultaneously within a unified framework. Furthermore, the sign of the total reward value determines the overall direction of the policy gradient update for the current output text.

[0102] There are several ways to use the current discriminative model to classify normal text and obtain positive or negative rewards for normal text. One possible approach is to design the current discriminative model as a scorer that outputs real numbers in the range of -1 to 1. When the quality of normal text is high, a positive value is output as a positive reward for normal text, and when the quality of normal text is low, a negative value is output as a negative reward for normal text.

[0103] Another alternative approach is to set a quality threshold. When the raw score output by the current discriminant model exceeds the threshold, it is mapped to a positive reward; when the raw score is below the threshold, it is mapped to a negative reward.

[0104] Another alternative approach is to use two independent discriminant models, one specifically for outputting positive rewards and the other specifically for outputting negative rewards.

[0105] There are several ways to weight the positive rewards, negative rewards, and negative rewards of normal text to obtain the total reward value. One option is to pre-set fixed weight coefficients for the positive rewards, negative rewards, and negative rewards of normal text, and then sum the results after multiplying each reward value by its corresponding weight to obtain the total reward value.

[0106] Another option is to dynamically adjust the weight coefficients according to the current training stage. In the early stage of training, increase the negative reward weight of abnormal text to quickly suppress degradation, and in the later stage of training, increase the positive reward weight of normal text to finely optimize the generation quality, and then perform a weighted summation.

[0107] Another option is to use an adaptive weighting method, which automatically calculates the weights based on the proportion of each type of sample in the current batch. For example, when the proportion of abnormal samples is small, the weight of their negative reward is automatically increased, and vice versa.

[0108] One possible implementation, where the policy gradient is updated based on the reward value, can be referenced by the following formula:

[0109] in, These are the parameters of the policy network (Actor). It is the objective function for the expected cumulative reward. , It is the gradient of the objective function with respect to the parameters. A trajectory, according to the current strategy Sampled, Trajectory length, In state Take action below The probability, The log gradient of the policy, From time step Initial cumulative discount rewards.

[0110] This specification's embodiments utilize the current discriminative model to distinguish normal text, obtaining positive or negative rewards for normal text. The positive or negative rewards for normal text, along with the negative rewards for abnormal text, are weighted to obtain a total reward value. The policy gradient of the current policy model is then updated based on this total reward value. This further distinguishes high-quality and low-quality text within normal samples and flexibly adjusts the relative importance of different types of reward signals through a weighting mechanism. Simultaneously, the strong penalty signal from abnormal text is also included in the same weighting framework. This allows the policy model to receive both fine-grained quality feedback from the discriminative model and explicit penalty signals from repetition rate self-checks during training. This suppresses repetition degradation while improving the quality discriminativeness of generated content. Stable and effective policy optimization can be achieved without introducing additional advantage estimation or baseline correction, reducing algorithm complexity and computational overhead.

[0111] One optional embodiment of this specification describes obtaining output text, including: Get the input data; Using the current policy model, generate the output probability of the input data based on the input data; The output probability is sampled to obtain the output text.

[0112] Input data is the raw information used to power the current policy model and trigger the text generation process. Specifically, input data can include natural language text, user queries, reference images, instruction description text, and prompt prefixes or prefix sequences. Input data can serve as context or conditions for the current policy model, guiding it to generate relevant output text. Input data can be preprocessed, such as through word segmentation, encoding, or the addition of special tokens, to convert it into a tensor form that the model can process.

[0113] Output probability is a probability distribution of each candidate text unit at each position in the output sequence, calculated by the current policy model based on the input data. It can be included in each step of output text generation as a probability vector output by the current policy model, where each element represents the likelihood of the corresponding text unit being selected. Output probability can be used to determine the actual output text unit at each position through sampling or search methods, thereby progressively generating the complete output text. The logarithm of the output probability is typically used in the loss function during policy gradient calculation.

[0114] Sampling is the process of randomly selecting text units based on output probabilities to generate output text. Sampling can be multinomial sampling, where a text unit is randomly selected as the output for the current position according to the output probability distribution; it can also be temperature sampling, where the output probability is first divided by a temperature parameter and then normalized to adjust the randomness of the sampling; or other forms of sampling. Sampling can be used to introduce exploration during policy model training, enabling the model to generate diverse output text and avoiding pattern solidification caused by deterministic outputs. The sampled output text is used for subsequent repetition rate recognition and reward calculation.

[0115] In practical applications, there are multiple ways to sample the output probability to obtain the output text.

[0116] An alternative approach is to randomly sample text units at each position according to the output probability distribution of the current strategy model, such that the probability of each text unit being selected is exactly equal to its output probability value. This sampling is performed sequentially until a termination symbol is generated or the preset maximum length is reached. The text units sampled at each position are then concatenated in sequence to form the output text.

[0117] Another alternative approach is to use temperature sampling. First, obtain the original output probability distribution of the current policy model, then introduce a temperature parameter to adjust the output probabilities. Divide the logarithm of the output probability by the temperature parameter, and then normalize the adjusted value using an activation function to obtain the adjusted probability distribution. Sampling is then performed based on this adjusted probability distribution. When the temperature parameter is greater than 1, the adjusted probability distribution tends to be more uniform, resulting in more diverse and random sampling results. When the temperature parameter is less than 1, the adjusted probability distribution tends to be sharper, further increasing the selection probability of high-probability text units.

[0118] Another option is to use top-k sampling. First, sort the output probability distribution of the current policy model in descending order of probability value, keep only the top k text units with the highest probability, and set the probability of the remaining text units to zero. Then, renormalize the probability of the retained k text units to obtain a new probability distribution. Finally, sample based on the normalized distribution.

[0119] The embodiments in this specification acquire input data, utilize the current policy model to generate output probabilities based on the input data, and then sample the output probabilities to obtain output text. This transforms the input data into a signal that the model can process. By introducing randomness into the generation process through the sampling mechanism, the current policy model can generate diverse output texts during training, avoiding the problem of insufficient exploration caused by deterministic outputs. At the same time, it provides a representative sample basis for subsequent repetition rate identification and policy gradient updates, which helps to improve the model's ability to explore diverse generation patterns and the diversity of the final generated content.

[0120] One optional embodiment of this specification involves updating the policy gradient of the current policy model based on negative rewards until a fully trained target policy model is obtained, including: The policy gradient is updated based on the negative reward to obtain the updated current policy model. Using the updated current policy model, generate the output probability of the input data based on the input data; Based on the output probability, determine the policy entropy of the updated current policy model; Determine whether the policy entropy is less than the preset entropy threshold and whether the number of training steps has reached the preset training step threshold; If the policy entropy is less than the preset entropy threshold and the number of training steps has not reached the preset training step threshold, the output probability is sampled to obtain the output text, and the process is returned to perform the repetition rate recognition of each text unit in the output text to obtain the recognition result. The training continues until the policy entropy is not less than a preset entropy threshold or the number of training steps reaches a preset training step threshold, at which point the target policy model is obtained.

[0121] The updated current policy model is the policy model obtained after performing one or more policy gradient updates on the current policy model. Specifically, the updated current policy model can be a new model instance generated after calculating the gradient based on the negative reward and updating the parameters in one training step. The updated current policy model can be used to replace the original current policy model and continue to perform subsequent output text generation and repetition rate identification processes. The parameters of the updated current policy model have changed compared to the previous one, and its behavior in generating output text will also be adjusted accordingly. Ideally, the probability of generating anomalous text is reduced.

[0122] Policy entropy is a quantitative metric used to measure the degree of uncertainty or randomness in the output probability distribution of an updated policy model given input data. Specifically, policy entropy can be calculated based on the output probability distribution. For example, for each generation position, the policy entropy equals the sum of the negative output probability at that position multiplied by its logarithmic probability. The policy entropy of the entire output sequence can be the average or sum of the policy entropies at each position. Policy entropy can characterize the exploratory nature of a policy model: higher policy entropy indicates that the model assigns a more even probability to multiple candidate text units, tending to generate diverse outputs; lower policy entropy indicates that the model assigns a high probability to a few text units, resulting in more deterministic outputs. Furthermore, excessively low policy entropy usually means that the model may be at risk of pattern degradation or repeated generation. The policy entropy formula can be found here:

[0123] in This indicates that the current policy model is in state. The probability distribution of the output for all candidate text units under the condition that the generated prefix text or the hidden state corresponding to the input data is given. Indicates candidate text units. This represents a vocabulary consisting of all candidate text units. The summation symbol accumulates the sum of each candidate text unit in the vocabulary. It is the logarithm of the probability; the entire formula calculates the information entropy of the probability distribution.

[0124] When generating actual output text, the policy entropy at each generation position can be calculated, and the average or sum of all positions can be taken as the policy entropy for the entire output text. Policy entropy can be used to characterize the generation diversity of the current policy model: when the policy entropy is high, the model assigns a relatively uniform probability to multiple candidate text units, tending to generate diverse outputs; when the policy entropy is low, the model assigns a very high probability to a few text units, the generation results tend to be deterministic, and it is easy to fall into a repetitive degradation mode. In the embodiments of this specification, policy entropy is used to monitor the degradation state of the policy network: when the policy entropy is less than a preset entropy threshold and the number of training steps has not reached a preset training step threshold, the repetition rate identification and policy gradient update process continues; when the policy entropy is not less than the preset entropy threshold or the number of training steps has reached the preset training step threshold, training stops and the current policy model is taken as the target policy model after training is completed.

[0125] A preset entropy threshold is a critical value pre-set to determine whether the updated policy model has entered a low-entropy degenerate state. Specifically, the preset entropy threshold can be a fixed positive real number or a value dynamically adjusted based on model size or task type. The preset entropy threshold can be compared with the policy entropy: when the policy entropy is less than the preset entropy threshold, the model is considered to have insufficient generation diversity and is at risk of degradation; when the policy entropy is not less than the preset entropy threshold, the model is considered to maintain sufficient exploration capability. The preset entropy threshold can be used together with a preset training step threshold as a condition for terminating or continuing training.

[0126] A preset training step threshold is a pre-defined critical value used to limit the maximum number of training steps a model can take. Specifically, the preset training step threshold can be a positive integer or a value that is dynamically adjusted based on model size or task type. The preset training step threshold can be compared with the currently executed training steps: when the training steps reach the preset training step threshold, training stops even if the model has not fully converged or the policy entropy is still low, to prevent overfitting or invalid computation. Furthermore, the preset training step threshold can be used in conjunction with a preset entropy threshold to form a termination condition where "policy entropy is not lower than the threshold or the number of steps reaches the threshold."

[0127] In practical applications, there are several ways to update the policy gradient of the current policy model based on negative rewards until the target policy model is obtained after training.

[0128] One possible approach is to first update the current policy model using policy gradients based on negative rewards to obtain an updated current policy model. Then, the updated current policy model is used to generate output probabilities based on the input data. Policy entropy is calculated based on the output probabilities. It is then determined whether the policy entropy is less than a preset entropy threshold and whether the number of training steps has not reached a preset training step threshold. If both conditions are met simultaneously, i.e., the policy entropy is too low and the number of steps has not reached the threshold, the output probabilities are sampled to obtain the output text, and the process of repetition rate recognition and subsequent steps is returned. Otherwise, training is stopped, and the updated current policy model is used as the target policy model after training is completed.

[0129] Another alternative approach is to use only policy entropy as the stopping condition without considering the number of training steps. That is, training is terminated early when the policy entropy is continuously lower than a preset entropy threshold, and training continues when the policy entropy rebounds above the threshold until the policy entropy is stably higher than the preset entropy threshold.

[0130] Another option is to use only the number of training steps as the stopping condition, that is, to execute a fixed number of iterations for a preset training step threshold, without judging the policy entropy during the process, and finally use the policy model when the number of steps is reached as the target policy model.

[0131] This embodiment of the specification obtains an updated current policy model by updating the policy gradient based on negative rewards. The updated current policy model is then used to generate output probabilities and determine policy entropy. It checks whether the policy entropy is less than a preset entropy threshold and whether the number of training steps has reached a preset training step threshold. If the policy entropy is less than the preset entropy threshold and the number of training steps has not reached the preset training step threshold, the repetition rate identification and update steps continue to be executed repeatedly until the policy entropy is not less than the preset entropy threshold or the number of training steps reaches the preset training step threshold. This allows for dynamic monitoring of the diversity of policy model generation during reinforcement learning training. Policy entropy serves as an early warning indicator of whether the model has fallen into a degenerate mode. Before the model output becomes too deterministic, continuous training guides the model to recover its exploratory capabilities. Combined with the training step threshold as a fallback condition, this avoids infinite training or premature termination due to improper policy entropy threshold settings and effectively curbs the continuous deterioration of repetitive degradation, ultimately obtaining a target policy model with high generation quality and good diversity.

[0132] In one optional embodiment of this specification, before performing repetition rate identification on each text unit in the output text and obtaining the identification result, the method further includes: The output text is normalized to obtain normalized output text. The normalization process includes at least one of the following: noise removal, function word removal, line break removal, and short text filtering.

[0133] Normalization is a process of standardizing the output text to eliminate irrelevant interference. Specifically, normalization may include one or more operations such as noise removal, function word removal, line break removal, and filtering short text units. Normalization can reduce the impact of redundant information and non-semantic factors in the output text on repetition rate detection, improving the accuracy and robustness of repetition rate detection. Normalization is typically performed before repetition rate detection, and its result serves as input for subsequent repetition rate detection steps. Normalization may also include removing redundant whitespace characters, removing duplicate punctuation marks, etc.

[0134] The normalized output text is the result of normalizing the original output text. The normalized output text can be a clean text sequence after removing noise characters, deleting function words, merging line breaks, and removing excessively short phrases. The normalized output text can be used to replace the original output text for subsequent text unit segmentation and repetition rate identification, to eliminate formatting or function word interference unrelated to repetition degradation. It should be noted that the normalized output text retains semantic content units that are meaningful for repetition rate identification.

[0135] Noise removal is an operation to delete interfering characters or symbols that are irrelevant to semantic content from the output text. Specifically, noise removal may include deleting special symbols (such as asterisks, underscores, equal signs), deleting punctuation marks (such as commas, periods, exclamation marks), deleting control characters (such as line breaks, tab characters), or deleting HTML tags and escape sequences. Noise removal can be used to purify the output text and avoid false duplicate recognition caused by noise characters. Further, noise removal can be implemented by regular expression matching or character category-based filtering.

[0136] Function word removal is an operation to delete common functional words from the output text without affecting the core semantics. Specifically, function words may include prepositions (such as "of", "already", "in", "and"), conjunctions (such as "moreover", "but", "because"), auxiliary words (such as " -ing", " -ed", "ah"), as well as articles and pronouns, etc. Function word removal can be used to reduce the dominant role of high-frequency function words in the duplicate rate statistics, making the duplicate rate recognition more focused on the repetition at the content level of content words. Further, function word removal can be performed by matching and deleting based on a preset function word stop list.

[0137] Line break removal is an operation to delete line break characters generated by line breaks or carriage returns in the output text, so as to restore the text to a continuous string. Specifically, line break removal may include replacing line break characters with spaces or directly deleting them, deleting carriage return characters, or merging multiple consecutive line break characters into one delimiter. Line break removal can be used to avoid the offset of the text unit segmentation boundary caused by the physical line break method of the text, and ensure that the sliding window segmentation can correctly reflect the semantic continuity of the output text. Further, line break removal is usually performed after the noise removal operation.

[0138] Filtering short text units is an operation to delete text units that are too short (such as single characters, single words or segments with a length less than a preset threshold) from the output text. Specifically, filtering short text units may include deleting single punctuation marks, deleting single function word characters, or deleting substrings with a length less than 2 characters. Filtering short text units can be used to exclude the interference of isolated short units caused by noise residue or abnormal word segmentation boundaries on the duplicate rate statistics. Further, a minimum length threshold can be set for filtering short text units, and only text units exceeding this threshold are retained.

[0139] Removing redundant white space characters is an operation of merging or deleting multiple consecutive white space characters in the output text so that only necessary delimiters are retained in the text. Removing redundant white space characters can include replacing multiple consecutive spaces with a single space, replacing tab characters with spaces and then merging them, deleting white space characters at the beginning and end of a line, and uniformly processing line break characters together with the surrounding white space characters. Removing redundant white space characters can be used to eliminate redundant intervals caused by non-standard text formatting and avoid misjudging text units that should be the same as different due to differences in white space characters during text unit segmentation.

[0140] Removing duplicate punctuation marks is a processing operation that merges consecutive identical punctuation marks in the output text into a single punctuation mark or deletes the redundant duplicates. Specifically, removing duplicate punctuation marks can include replacing multiple consecutive exclamation marks "!!! " with a single exclamation mark, replacing multiple consecutive question marks "??? " with a single question mark, replacing multiple consecutive commas ",,," with a single comma, or replacing multiple consecutive full stops ".... " with a single full stop. Further, removing duplicate punctuation marks can also include handling cases of mixed duplicate punctuation, such as combinations like "!? " or "?!! ", and one can choose to retain the first punctuation mark or uniformly replace it with a standard form.

[0141] In practical applications, there are various ways to perform normalization processing on the output text to obtain the normalized output text. One optional way is to use regular expressions to match and delete all punctuation marks and special characters to obtain the denoised text; then perform the operation of removing function words on the denoised text, such as deleting function words like "de", "le", "zai", "he", etc. based on a preset stop word list; then perform the operation of removing line breaks, deleting or replacing line break characters and carriage return characters in the text with spaces; finally, perform the operation of filtering short text units, such as deleting character fragments with a length less than 2, thereby obtaining the normalized output text.

[0142] Another optional way is to use a large text cleaning model specifically trained for text cleaning to perform some of the above operations, such as only performing noise removal and line break removal, without performing function word removal and short text unit filtering, in order to retain more complete semantic information.

[0143] Yet another optional way is to adjust the execution order of each operation. For example, first perform line break removal to make the text a continuous string, then perform noise removal and function word removal, and finally perform short text unit filtering; different orders may be applicable to different types of output text.

[0144] The normalization process can be dynamically selected based on the characteristics of the output text. For shorter output text, the filtering of short text units can be omitted, while for output text containing a large number of formatting tags, the removal of HTML tags can be added.

[0145] The embodiments in this specification normalize the output text by performing at least one of the following operations: noise removal, function word removal, line break removal, and filtering of short text units. This normalized output text eliminates the negative impacts of format interference, function word interference, and fragmented units in the output text before repetition rate identification. This allows subsequent text unit sliding window segmentation and repetition fragment detection to focus more on repetition patterns at the substantive content level, avoiding false repetition statistics caused by punctuation marks, line breaks, or high-frequency function words. As a result, the accuracy and reliability of repetition rate identification results are improved, providing a higher quality data foundation for abnormal text judgment and negative reward allocation.

[0146] In one optional embodiment of this specification, the output text includes abnormal text and normal samples other than the abnormal text; If the recognition result exceeds a preset threshold, and the output text is determined to be abnormal text, the following steps are also included: Based on the abnormal text, construct negative example samples; Construct positive examples based on normal samples; A supervised dataset is constructed based on negative and positive examples, which is used to train the discriminative model.

[0147] Normal samples are output text excluding abnormal text, representing generated results whose recognition outcomes do not exceed a preset threshold. Specifically, normal samples can include semantically fluent sentences with acceptable repetition rates, structurally sound paragraphs, or natural language generation results that conform to the expected distribution. Normal samples can be used to construct positive examples, serving as positive examples in the training of the discriminant model.

[0148] Negative examples are negative training data constructed from anomalous text and used to train the discriminative model. Specifically, negative examples can be obtained by directly using anomalous text as input features and labeling it with a negative example; or they can be variations obtained by slightly perturbing the anomalous text. Negative examples can be used to train the discriminative model to learn to recognize feature patterns in low-quality or degraded text. Furthermore, negative examples appear in pairs with positive examples, together forming the negative example portion of the supervised dataset.

[0149] Positive samples are positive training data constructed from normal samples and used to train the discriminative model. Specifically, positive samples can be obtained by directly using normal samples as input features and labeling them with a positive example; or they can be selected from a high-quality subset of normal samples as positive examples. Positive samples can be used to train the discriminative model to learn to recognize feature patterns in high-quality or normal text. Furthermore, positive samples and negative samples together constitute the positive sample portion of the supervised dataset.

[0150] A supervised dataset is a labeled dataset composed of both negative and positive samples used to train a discriminative model. Specifically, a supervised dataset can include multiple sample pairs, each containing an input text and its corresponding label. Supervised datasets can be used to fine-tune the discriminative model under supervision, enabling it to classify or score the input text for quality, or to optimize the model's scoring capabilities. Furthermore, supervised datasets can be dynamically updated during training, continuously collecting new anomalous texts and normal samples as the policy model evolves.

[0151] In practical applications, there are several ways to construct negative sample, positive sample, and supervision datasets. One option is to directly treat each output text judged as an anomalous text as a negative sample and assign it a negative label; at the same time, treat each output text judged as a normal sample as a positive sample and assign it a positive label; then merge all negative and positive samples to construct a supervision dataset, which is used for subsequent training of the discrimination model.

[0152] Another alternative approach is to select only a subset of anomalous texts and normal samples to construct the supervised dataset. This could be achieved by sampling according to a preset ratio (e.g., retaining all anomalous texts and downsampling normal samples at a certain ratio) to balance the number of positive and negative samples and avoid training bias in the discriminant model due to a significant disparity in sample numbers. Alternatively, negative and positive samples can be weighted or graded based on their repetition rates: for example, anomalous texts with higher repetition rates are assigned higher confidence or weights when constructing negative samples; higher-quality normal samples are given higher priority when constructing positive samples. The weighted samples, along with their weight information, are then stored in the supervised dataset for subsequent weighted training of the discriminant model.

[0153] This embodiment divides the output text into anomalous text and normal samples, constructs negative samples based on anomalous text, constructs positive samples based on normal samples, and then constructs a supervised dataset based on both negative and positive samples. This allows for the automatic construction of a labeled dataset for training the discriminative model using the generated results produced by the model itself during reinforcement learning training, eliminating the need for manual annotation and reducing data construction costs. Simultaneously, the negative samples in the supervised dataset directly originate from the actual anomalous degradation phenomena generated by the policy model, while the positive samples come from normal generation results. This ensures a high degree of matching between the training data distribution of the discriminative model and the current state of the policy model, thereby improving the discriminative model's ability to assess the quality of the content generated by the current policy model. This provides a high-quality data foundation for subsequent collaborative training or alternating optimization of the discriminative and policy models.

[0154] Figure 2 This is a flowchart illustrating the processing steps of a model training method provided in one embodiment of this specification. (Refer to...) Figure 2 The process shown in the embodiments of this specification, and a complete implementation of the model training method provided, can be described as follows: First, the input data is processed using the current strategy model, and the output probabilities are sampled to obtain the sequence sampled text as the output text. Then, a repetition rate detection is performed on this output text, that is, the repetition rate of each text unit in the output text is identified to obtain the recognition result. Based on whether the recognition result exceeds a preset threshold, the output text is distinguished into normal text and abnormal text.

[0155] For output text whose recognition results do not exceed the preset threshold, it is classified as normal text. For normal text, the current discrimination model is further used to calculate the normal reward, obtaining the reward value for normal text. For output text whose recognition results exceed the preset threshold, it is classified as abnormal text, and a forced negative reward is assigned to the abnormal text.

[0156] After obtaining the reward value for normal text and the forced negative reward for abnormal text, gradient updates are performed on the current policy model based on these reward values ​​and the forced negative reward. During or after the gradient update process, the policy entropy of the current policy model can be monitored to determine if the model has fallen into a low-entropy degenerate state. Simultaneously, output texts identified as abnormal are further accumulated as an abnormal sample set. Based on this abnormal sample set, combined with normal samples corresponding to normal text, a supervised dataset containing negative and positive samples can be constructed. This supervised dataset is then used to train the reward model (i.e., to train the discriminative model), enabling the reward model to provide more accurate quality assessment signals for subsequent text generation.

[0157] Through the above process, in each iteration of the policy model training, sequence sampling, repetition rate detection, normal and abnormal branch processing, reward / penalty allocation, gradient update and policy entropy monitoring are repeatedly performed. At the same time, abnormal samples are accumulated for the training of the reward model until the current policy model converges or reaches the preset stopping condition, thus obtaining the trained target policy model and target discrimination model.

[0158] Figure 3 This is a flowchart illustrating an anomaly detection method for a model training method provided in one embodiment of this specification. (Refer to...) Figure 3 The process shown in the embodiments of this specification, and a specific implementation of the repetition rate identification and anomaly determination, can be described as follows: First, the input text is obtained, which is the output text generated by the current policy model. Then, the input text is normalized to obtain the normalized output text. The normalization process includes at least one of the following operations: noise removal, function word removal, line break removal, filtering short text units, removing redundant whitespace characters, and removing duplicate punctuation marks. Next, a sliding window scan is performed on the normalized output text: the text units in the output text are segmented according to the sliding window, resulting in multiple text unit slices. The window size can be preset, for example, 3, 5, or 10 text units, and the sliding step can be 1 or equal to the window size. During the sliding window scan, duplicate slices are identified within each text unit slice. Duplicate slices are those whose content overlaps with at least one other text unit slice (e.g., completely identical or highly similar). Then, the duplication rate of each duplicate slice is calculated, for example, by determining the number of duplicate slices. Finally, it is determined whether the repetition rate exceeds a preset threshold: if the repetition rate exceeds the preset threshold, the current output text is determined to be an abnormal sequence (i.e., abnormal text); if the repetition rate does not exceed the preset threshold, the current output text is determined to be a normal sequence (i.e., normal text). Through the above process, a complete automated detection process from input text to abnormal or normal determination is realized, providing a clear basis for subsequent negative reward allocation or positive reward calculation.

[0159] Figure 4 A flowchart of a model fine-tuning method provided in one embodiment of this specification includes the following steps: Step 402: Obtain the supervised dataset, which is constructed according to the model training method.

[0160] A supervised dataset is a labeled dataset constructed according to the model training method, containing positive and negative samples and their corresponding labels. Specifically, a supervised dataset can include negative samples constructed from anomalous and degenerate samples and their negative class labels, and positive samples constructed from normal samples and their positive class labels. Supervised datasets can be used as training data for supervised fine-tuning of discriminative models, enabling the discriminative model to learn the ability to distinguish between normal text and anomalous and degenerate text. Furthermore, supervised datasets can be dynamically collected and updated during the training of the policy model, or they can be permanently stored after a complete training cycle.

[0161] This step obtains a supervised dataset constructed according to the model training method. The supervised dataset is directly derived from the abnormal degenerate samples and normal samples automatically generated during the policy model training process. This eliminates the need for manual annotation to obtain training data with positive and negative labels, significantly reducing the manpower and time costs of data construction. At the same time, the sample distribution in the supervised dataset is highly matched with the current generation state of the policy model, enabling the subsequent supervised fine-tuning of the discriminant model to be based on a data distribution strongly related to the task. This improves the targeting and effectiveness of the discriminant model training and provides a high-quality, low-cost data foundation for training a target discriminant model that can accurately distinguish between normal text and abnormal degenerate text.

[0162] Step 404: Perform supervised fine-tuning of the current discrimination model based on the supervised dataset until a fully trained target discrimination model is obtained.

[0163] The current discriminant model is a machine learning model that undergoes supervised fine-tuning to classify input text as quality and output a reward value. The current discriminant model can be a binary classification model based on the Transformer architecture, a discriminator based on a convolutional neural network, or a reward model based on contrastive learning, etc. The current discriminant model can receive input text and output a scalar value as the quality score or classification probability of that text. The initial parameters of the current discriminant model can be randomly initialized or loaded from a pre-trained model.

[0164] Supervised fine-tuning is the process of training a current discriminative model in a supervised manner using a labeled supervised dataset to optimize its parameters. Specifically, supervised fine-tuning may involve inputting samples from the supervised dataset into the current discriminative model to obtain predicted outputs, calculating the loss function value between the predicted outputs and the true labels, and then updating the parameters of the current discriminative model using a backpropagation algorithm. Supervised fine-tuning can be used to adapt the current discriminative model to specific text quality discrimination tasks, improving its ability to distinguish between normal text and abnormally degraded text. Furthermore, supervised fine-tuning can be performed in multiple rounds until the model converges or reaches a preset stopping condition.

[0165] The target discrimination model is a trained discrimination model obtained after supervised fine-tuning of the current discrimination model until a stopping condition is met. Specifically, the target discrimination model can be a model that has reached a preset accuracy threshold on a supervised dataset, or whose loss function value no longer decreases, or which has reached a preset number of training epochs. The target discrimination model can be deployed in the reinforcement learning training process of the policy model to provide reward values ​​for normal text. Furthermore, compared with the current discrimination model, the target discrimination model has higher discrimination accuracy and stronger generalization ability.

[0166] In practical applications, there are several ways to fine-tune the current discrimination model based on a supervised dataset until a fully trained target discrimination model is obtained.

[0167] An alternative approach is to divide the supervised dataset into a training set and a validation set. In each training round, the training set is used to update the parameters of the current discriminative model, calculate the loss function, and backpropagate. After each round, the model performance is evaluated on the validation set. Training stops when the accuracy or F1 score on the validation set no longer improves for several consecutive rounds, thus obtaining the trained target discriminative model.

[0168] Another alternative approach is to use mini-batch stochastic gradient descent. A fixed number of samples are randomly drawn from the supervised dataset to form a batch. The samples in the batch are then fed into the current discriminant model in sequence. The cross-entropy loss between the predicted label and the true label is calculated, accumulated, and averaged. Then, the model parameters are updated using an optimizer (such as Adam or SGD). The above process is repeated until the entire supervised dataset is traversed multiple times (e.g., 3 to 10 rounds), and the final model is used as the target discriminant model.

[0169] This embodiment of the specification acquires a supervised dataset constructed according to the model training method, and performs supervised fine-tuning of the current discriminative model based on the supervised dataset until a trained target discriminative model is obtained. This allows the discriminative model to be trained in a targeted manner using positive and negative samples automatically generated during the training of the policy model, enabling the discriminative model to learn a discrimination ability that matches the distribution generated by the current policy model, without the need for additional manual annotation or external data. At the same time, the supervised fine-tuning process can adjust the parameters of the discriminative model to the optimal state for predicting reward values ​​for normal text, providing accurate quality feedback signals for the target discriminative model in the subsequent reinforcement learning training of the policy model, thereby forming a collaborative optimization closed loop between the policy model and the discriminative model, improving the overall training efficiency and generation quality.

[0170] Corresponding to the above method embodiments, this specification also provides embodiments of a model training device. Figure 5 This is a schematic diagram of a model training device provided in one embodiment of this specification. Figure 5As shown, the device includes: The text acquisition module 502 is configured to acquire output text, wherein the output text is output by the current policy model after processing the input data; The repetition rate recognition module 504 is configured to perform repetition rate recognition on each text unit in the output text and obtain the recognition result. The judgment module 506 is configured to determine that the output text is abnormal text if the recognition result exceeds a preset threshold. The reward allocation module 508 is configured to assign negative rewards for abnormal text. The policy gradient update module 510 is configured to update the policy gradient of the current policy model based on the negative reward of the abnormal text until the target policy model that has been trained is obtained.

[0171] The repetition rate identification module 504 is further configured to segment each text unit in the output text according to the text unit sliding window to obtain multiple text unit segments; determine the repetitive segments in each text unit segment, wherein the repetitive segment is the text unit segment in which the text unit is repeated with other text unit segments; and perform statistics on each repetitive segment to obtain the repetition rate of each repetitive segment.

[0172] The model training device also includes a normal text reward allocation module, which is configured to determine that the output text is normal text if the recognition result does not exceed a preset threshold; and to use the current discrimination model to discriminate the normal text and obtain the reward value of the normal text. The policy gradient update module 510 is further configured to update the policy gradient of the current policy model based on the reward value of normal text and the negative reward of abnormal text, until the target policy model that has been trained is obtained.

[0173] The judgment module 506 is further configured to use the current discrimination model to judge normal text and obtain a positive reward or a negative reward for normal text. The policy gradient update module 510 is further configured to weight the positive reward or negative reward of normal text and the negative reward of abnormal text to obtain a total reward value; based on the total reward value, the policy gradient is updated on the current policy model until the target policy model that has been trained is obtained.

[0174] The text acquisition module 502 is further configured to acquire input data; generate output probabilities of the input data based on the input data using the current policy model; and sample the output probabilities to obtain the output text.

[0175] The policy gradient update module 510 is further configured to perform policy gradient update on the current policy model based on negative rewards to obtain the updated current policy model; using the updated current policy model, generate the output probability of the input data based on the input data; determine the policy entropy of the updated current policy model based on the output probability; determine whether the policy entropy is less than a preset entropy threshold and whether the number of training steps has reached a preset training step threshold; if the policy entropy is less than the preset entropy threshold and the number of training steps has not reached the preset training step threshold, sample the output probability to obtain the output text, and return to perform the step of repetition rate recognition of each text unit in the output text to obtain the recognition result; until the policy entropy is not less than the preset entropy threshold or the number of training steps has reached the preset training step threshold, the trained target policy model is obtained.

[0176] The model training device also includes a normalization module configured to normalize the output text to obtain normalized output text. The normalization process includes at least one of the following: noise removal, function word removal, line break removal, and short text filtering.

[0177] The model training apparatus also includes a dataset construction module, configured to construct negative examples based on anomalous text; construct positive examples based on normal examples; and construct a supervised dataset based on both negative and positive examples, wherein the supervised dataset is used to train the discriminative model.

[0178] This embodiment of the specification acquires a supervised dataset constructed according to the model training method, and performs supervised fine-tuning of the current discriminative model based on this supervised dataset until a trained target discriminative model is obtained. This allows the construction of a supervised dataset using abnormal degenerate samples and normal samples automatically generated during the policy model training process, obtaining high-quality labeled training data without manual annotation, significantly reducing data construction costs. At the same time, the supervised fine-tuning process enables the discriminative model to learn discriminative capabilities that match the distribution generated by the current policy model, allowing the trained target discriminative model to accurately distinguish between normal text and abnormal degenerate text, providing reliable quality feedback signals for the subsequent reinforcement learning training of the policy model, thereby forming a collaborative optimization closed loop between the policy model and the discriminative model, improving overall training efficiency and the quality of the final generated content.

[0179] The above is an illustrative scheme of a model training device according to this embodiment. It should be noted that the technical solution of this model training device and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training device, please refer to the description of the technical solution of the model training method described above.

[0180] Corresponding to the above method embodiments, this specification also provides embodiments of a model fine-tuning device. Figure 6This is a schematic diagram of a model fine-tuning device provided in one embodiment of this specification. Figure 6 As shown, the device includes: The dataset acquisition module 602 is configured to acquire a supervised dataset, wherein the supervised dataset is constructed according to the model training method; The supervised fine-tuning module 604 is configured to perform supervised fine-tuning on the current discriminant model based on the supervised dataset until the trained target discriminant model is obtained.

[0181] This embodiment of the specification acquires a supervised dataset constructed according to the model training method, and performs supervised fine-tuning of the current discriminative model based on the supervised dataset until a trained target discriminative model is obtained. This allows the discriminative model to be trained in a targeted manner using positive and negative samples automatically generated during the training of the policy model, enabling the discriminative model to learn a discrimination ability that matches the distribution generated by the current policy model, without the need for additional manual annotation or external data. At the same time, the supervised fine-tuning process can adjust the parameters of the discriminative model to the optimal state for predicting reward values ​​for normal text, providing accurate quality feedback signals for the target discriminative model in the subsequent reinforcement learning training of the policy model, thereby forming a collaborative optimization closed loop between the policy model and the discriminative model, improving the overall training efficiency and generation quality.

[0182] The above is a schematic scheme of a model fine-tuning device according to this embodiment. It should be noted that the technical solution of this model fine-tuning device and the technical solution of the model fine-tuning method described above belong to the same concept. For details not described in detail in the technical solution of the model fine-tuning device, please refer to the description of the technical solution of the model fine-tuning method described above.

[0183] Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0184] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0185] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0186] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0187] The processor 720 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned model training method or model fine-tuning method.

[0188] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the above-described model training method or model fine-tuning method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described model training method or model fine-tuning method.

[0189] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method or model fine-tuning method.

[0190] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the above-described model training method or model fine-tuning method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described model training method or model fine-tuning method.

[0191] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method or model fine-tuning method.

[0192] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of the above-described model training method or model fine-tuning method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-described model training method or model fine-tuning method.

[0193] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0194] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0195] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0196] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0197] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, characterized in that, include: Obtain the output text, wherein the output text is generated by the current policy model after processing the input data; The repetition rate of each text unit in the output text is identified to obtain the identification result; If the recognition result exceeds a preset threshold, the output text is determined to be abnormal text; Assign a negative reward to the abnormal text; The policy gradient of the current policy model is updated based on the negative reward of the abnormal text until the target policy model is obtained after training.

2. The method according to claim 1, characterized in that, The process of identifying the repetition rate of each text unit in the output text to obtain the identification result includes: The text units in the output text are segmented according to the sliding window of the text units to obtain multiple text unit fragments; Identify duplicate fragments in each text unit fragment, wherein the duplicate fragment is a text unit fragment in which the text unit is duplicated with other text unit fragments; The duplication rate of each duplicate fragment is obtained by statistical analysis.

3. The method according to any one of claims 1 or 2, characterized in that, After performing repetition rate identification on each text unit in the output text and obtaining the identification result, the method further includes: If the recognition result does not exceed the preset threshold, the output text is determined to be normal text; Using the current discrimination model, the normal text is judged to obtain the reward value of the normal text; The step of updating the policy gradient of the current policy model based on the negative reward of the abnormal text until the trained target policy model is obtained includes: Based on the reward value of the normal text and the negative reward of the abnormal text, the current policy model is updated with policy gradient until the target policy model is obtained after training.

4. The method according to claim 3, characterized in that, The step of using the current discrimination model to distinguish the normal text and obtain a reward value for the normal text includes: Using the current discrimination model, the normal text is judged to obtain a positive reward or a negative reward for the normal text; The step of updating the policy gradient of the current policy model based on the reward value of the normal text and the negative reward of the abnormal text until the trained target policy model is obtained includes: The positive reward or negative reward of the normal text and the negative reward of the abnormal text are weighted to obtain the total reward value; Based on the total reward value, the current policy model is updated with policy gradients until the target policy model is obtained after training.

5. The method according to claim 1, characterized in that, The process of obtaining the output text includes: Get the input data; Using the current policy model, the output probability of the input data is generated based on the input data; The output probability is sampled to obtain the output text.

6. The method according to claim 1, characterized in that, The step of updating the policy gradient of the current policy model based on the negative reward until the trained target policy model is obtained includes: Based on the negative reward, the current policy model is updated by performing a policy gradient update to obtain the updated current policy model; Using the updated current policy model, the output probability of the input data is generated based on the input data; Based on the output probability, the policy entropy of the updated current policy model is determined; Determine whether the policy entropy is less than a preset entropy threshold and whether the number of training steps has reached a preset training step threshold; If the strategy entropy is less than a preset entropy threshold and the number of training steps has not reached a preset training step threshold, the output probability is sampled to obtain the output text, and the process returns to the step of performing repetition rate recognition on each text unit in the output text to obtain the recognition result. The training continues until the policy entropy is not less than a preset entropy threshold or the number of training steps reaches a preset training step threshold, at which point the target policy model is obtained.

7. The method according to claim 1, characterized in that, Before performing repetition rate identification on each text unit in the output text to obtain the identification result, the method further includes: The output text is normalized to obtain normalized output text, wherein the normalization process includes at least one of the following: noise removal, function word removal, line break removal, and short text filtering.

8. The method according to claim 1, characterized in that, The output text includes abnormal text and normal samples other than the abnormal text; After determining that the output text is abnormal text if the recognition result exceeds a preset threshold, the method further includes: Based on the abnormal text, construct negative example samples; Based on the normal samples, construct positive examples; A supervised dataset is constructed based on the negative examples and the positive examples, wherein the supervised dataset is used to train the discriminative model.

9. A model fine-tuning method, characterized in that, include: Obtain a supervised dataset, wherein the supervised dataset is constructed according to the method of claim 8; The current discrimination model is fine-tuned under supervision based on the supervised dataset until the trained target discrimination model is obtained.

10. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium storing a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

Images