Large language model alignment method and device based on decoding probability correction

Abstract

The invention discloses a large language model alignment method and device based on decoding probability correction, and belongs to the field of artificial intelligence. By introducing a lightweight probability correction model, the original generation probability distribution of a large language model is monitored and intervened in real time in the reasoning and decoding process. According to the method, the generation probability distributed to each candidate token by a large language model is analyzed, and the probabilities are dynamically adjusted in real time by utilizing a probability correction model. Meanwhile, a global evaluation mechanism based on a discriminator is also introduced, so that the quality control and the alignment guarantee of the generated content are further enhanced. The discriminator not only has the capability of integrally evaluating the generated text, but also can identify potential improper contents or deviations, so that more comprehensive and accurate feedback is provided for a probability correction model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a large language model alignment method and device based on decoding probability correction. Background Art

[0002] In recent years, pre-trained large language models (LLMs) have demonstrated their powerful capabilities in many fields. These models have learned rich knowledge from large-scale corpora and have become an important component of online services and applications. They are widely used in key fields such as medicine, finance, and law. However, with the wide application of these models, security and robustness issues have become increasingly prominent. In particular, the increase in adversarial attacks has caused models to generate harmful content, which has aroused concerns about social and ethical risks.

[0003] Pre-trained LLMs may reflect the biases in their training data and propagate discriminatory content in some cases, especially when the input contains adversarial cues. This has prompted researchers to explore multiple alignment methods, such as reinforcement learning with human feedback, reinforcement learning with AI feedback, and direct preference optimization, to improve the security and compliance of generated content. However, these methods have significant implementation challenges, including the high labor cost of high-quality data annotation and the huge time and computing resources required during training. In addition, although the aligned models have enhanced their ability to resist adversarial attacks to a certain extent, they can still be bypassed by cleverly designed attacks.

[0004] In addition, during the training process, the instability of the model and its reliance on engineering knowledge make it particularly important to perform alignment without changing the model parameters. To this end, some technicians have proposed some lightweight alignment methods that do not require retraining the model. For example, some methods mitigate the effect of attacks by rewriting the detected adversarial inputs, while others rely on contextual examples to guide the generation process. However, these methods often rely on accurate identification of attacks and have difficulty dealing with more complex and covert attacks.

[0005] Other technologies, such as anomaly detection based on perplexity and dynamic constraint generation, have improved security to some extent, but they are still powerless in the face of complex attacks. Although existing methods have improved in effectiveness, they generally lack real-time intervention capabilities and are difficult to control the generated content in a fine-grained manner. Therefore, they are still insufficient in dealing with complex adversarial attacks. Therefore, studying the alignment method of large language models has important practical significance. Summary of the invention

[0006] The embodiments of the present application provide a large language model alignment method and device based on decoding probability correction to solve the problem of lack of real-time intervention capability when dealing with complex adversarial attacks in the large language model alignment process.

[0007] On the one hand, a large language model alignment method based on decoding probability correction is provided, the method comprising:

[0008] Generating a training set, wherein the training set is composed of possible continuations of a given problem;

[0009] A probability correction model is obtained by training a large language model according to the training set;

[0010] In the text generation process of the large language model, the generation probability of the next text basic unit is adjusted using the trained probability correction model, and the text basic unit is the basic unit for generating text;

[0011] evaluating the generated text by a discriminator;

[0012] In response to the generated text not meeting the alignment standard, a backtracking mechanism is introduced in the large language model to backtrack and adjust the text basic units of the generated text.

[0013] Optionally, the generating of a training set, wherein the training set is formed according to possible continuations of a given problem, includes:

[0014] Constructing harmful samples and useful samples according to historical data sets, wherein the harmful samples and the useful samples respectively contain corresponding given problems;

[0015] Randomly generate different prefixes for each harmful sample and useful sample, the prefixes being used as possible continuations corresponding to a given question;

[0016] Connecting the given questions corresponding to each harmful sample and each useful sample and the corresponding generated prefixes to generate the initial texts corresponding to each harmful sample and each useful sample;

[0017] Generate a variant text corresponding to the initial text by using the large language model;

[0018] The usefulness of each variant text corresponding to the harmful sample is scored through the reward model;

[0019] The harmfulness of each variant sample corresponding to the harmful sample is scored through the content security classification model;

[0020] The usefulness score and the harmfulness score are normalized and weighted to obtain the correction probability of each given question and the corresponding prefix as the training set.

[0021] Optionally, the obtaining of the probability correction model through training the training set includes:

[0022] According to the training set, the input form of the large language model is expressed as Where Prefix represents the generated prefix, t k is the next candidate text basic unit, <s>The large language model is used to aggregate the information of the entire input sequence and extract <s>Labeled feature vector where d m Represents the feature dimension.

[0023] The feature vector h is processed by a multi-layer perceptron CLS The vector is processed and the sigmoid function is used to map the output to a probability value in the [0,1] interval: in, is the quality probability of the model output, σ(x) is the sigmoid function;

[0024] The probability correction model is trained using the mean square error as the loss function, and the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE for:

[0025]

[0026] Optionally, in the text generation process of the large language model, using a trained probability correction model to adjust the generation probability of the next text basic unit includes:

[0027] Use the trained probability correction model to adjust the generation probability of the next basic unit of text, reduce the generation probability of tokens that do not meet human preferences, and increase the probability of tokens that meet human preferences, where tokens are represented as basic units of text;

[0028] Given a prefix prefix, assume that the original probability of the large language model generating the next candidate token is P LLM (t|prefix), P CM (t|prefix) represents the probability of the probability correction model evaluating whether the token conforms to human preferences;

[0029] The generation probability after correction by weighted fusion output is:

[0030]

[0031] Wherein, LLM refers to the large language model, CM refers to the probability correction model, α and β are hyperparameters that control the relative contribution of the large language model and the probability correction model, and satisfy α+β=1;

[0032] Introducing a normalization constant The corrected generation probability is normalized and finally adjusted to obtain the generation probability of the next basic unit of text.

[0033] Optionally, at time step t, the token sequence generated by the large language model is represented as x t ={t0, t1, ..., t t-1 }, and the large language model is used to select from the candidate token set T t ={t t1 , t t2 ,,t tn Generate a new token in} t , the generation probability of each candidate token in the candidate token set is expressed as H(t ti |x t ), the H(t ti |x t ) means in context x t Generate candidate tokens ti probability.

[0034] Optionally, the evaluating the generated text by a discriminator includes:

[0035] Introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0036] when Indicates the added t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0037] Optionally, in response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic units of the generated text by introducing a backtracking mechanism in the large language model includes:

[0038] If at time t, the candidate token set is evaluated as unqualified by the discriminator, a backtracking mechanism is triggered, wherein the candidate token is used to indicate the basic unit of the text;

[0039] The backtracking mechanism is used to return to the previous time step t-1 and the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1;

[0040] The backtracking mechanism reselects tokens that are not in the forbidden list from the candidate token set of the previous time step for generation until a token that meets the alignment criteria is found.

[0041] On the other hand, a large language model alignment device based on decoding probability correction is provided, the device comprising:

[0042] A training generation module, for generating a training set through a large language model, wherein the training set is composed of possible continuations of a given question;

[0043] A model generation module, used for obtaining a probability correction model through training of the training set;

[0044] A probability adjustment module, used to adjust the generation probability of the next text basic unit using the trained probability correction model during the text generation process of the large language model, wherein the text basic unit is the basic unit for generating text;

[0045] an identification and evaluation module, used for evaluating the generated text through a discriminator;

[0046] The backtracking adjustment module is used to backtrack and adjust the text basic units of the generated text by introducing a backtracking mechanism in the large language model in response to the generated text not meeting the alignment standard.

[0047] Optionally, the generating of a training set, wherein the training set is formed according to possible continuations of a given problem, includes:

[0048] Constructing harmful samples and useful samples according to historical data sets, wherein the harmful samples and the useful samples respectively contain corresponding given problems;

[0049] Randomly generate different prefixes for each harmful sample and useful sample, the prefixes being used as possible continuations corresponding to a given question;

[0050] Connecting the given questions corresponding to each harmful sample and each useful sample and the corresponding generated prefixes to generate the initial texts corresponding to each harmful sample and each useful sample;

[0051] Generate a variant text corresponding to the initial text by using the large language model;

[0052] The usefulness of each variant text corresponding to the harmful sample is scored through the reward model;

[0053] The harmfulness of each variant sample corresponding to the harmful sample is scored through the content security classification model;

[0054] The usefulness score and the harmfulness score are normalized and weighted to obtain the correction probability of each given question and the corresponding prefix as the training set.

[0055] Optionally, the obtaining of the probability correction model through training the training set includes:

[0056] According to the training set, the input form of the large language model is expressed as Where Prefix represents the generated prefix, t k is the next candidate text basic unit, <s>The large language model is used to aggregate the information of the entire input sequence and extract <s>Labeled feature vector where d m Represents the feature dimension.

[0057] The feature vector h is processed by a multi-layer perceptron CLS The vector is processed and the sigmoid function is used to map the output to a probability value in the [0,1] interval: in, is the quality probability of the model output, σ(x) is the sigmoid function;

[0058] The probability correction model is trained using the mean square error as the loss function, and the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE for:

[0059]

[0060] Optionally, in the text generation process of the large language model, using a trained probability correction model to adjust the generation probability of the next text basic unit includes:

[0061] Use the trained probability correction model to adjust the generation probability of the next basic unit of text, reduce the generation probability of tokens that do not meet human preferences, and increase the probability of tokens that meet human preferences, where tokens are represented as basic units of text;

[0062] Given a prefix prefix, assume that the original probability of the large language model generating the next candidate token is P LLM (t|prefix), P CM (t|prefix) represents the probability of the probability correction model evaluating whether the token conforms to human preferences;

[0063] The generation probability after correction by weighted fusion output is:

[0064]

[0065] Wherein, LLM refers to the large language model, CM refers to the probability correction model, α and β are hyperparameters that control the relative contribution of the large language model and the probability correction model, and satisfy α+β=1;

[0066] Introducing a normalization constant The corrected generation probability is normalized and finally adjusted to obtain the generation probability of the next basic unit of text.

[0067] Optionally, at time step t, the token sequence generated by the large language model is represented as x t ={t0, t1, ..., t t-1 }, and the large language model is used to select from the candidate token set T t ={t t1 , t t2 ,,t tn Generate a new token in} t , the generation probability of each candidate token in the candidate token set is expressed as H(t ti |X t ), the H(t ti |x t ) means in context x t Generate candidate tokens ti probability.

[0068] Optionally, the evaluating the generated text by a discriminator includes:

[0069] Introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0070] when Indicates the added t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0071] Optionally, in response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic units of the generated text by introducing a backtracking mechanism in the large language model includes:

[0072] If at time t, the candidate token set is evaluated as unqualified by the discriminator, a backtracking mechanism is triggered, wherein the candidate token is used to indicate the basic unit of the text;

[0073] The backtracking mechanism is used to return to the previous time step t-1 and the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1;

[0074] The backtracking mechanism reselects tokens that are not in the forbidden list from the candidate token set of the previous time step for generation until a token that meets the alignment criteria is found.

[0075] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the large language model alignment method based on decoding probability correction as described in the above aspects.

[0076] On the other hand, a computer program product is also provided, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the large language model alignment method based on decoding probability correction described in the above aspect.

[0077] The present application discloses a large language model alignment method and device based on decoding probability correction, which belongs to the field of artificial intelligence. By introducing a lightweight probability correction model, the original generation probability distribution of the large language model is monitored and intervened in real time during the inference decoding process. It analyzes the generation probability assigned by the large language model to each candidate token, and uses the probability correction model to adjust these probabilities in real time and dynamically. At the same time, the present invention also introduces a global evaluation mechanism based on a discriminator to further enhance the quality control and alignment guarantee of the generated content. This discriminator not only has the ability to evaluate the generated text as a whole, but also can identify potential inappropriate content or deviations, thereby providing more comprehensive and accurate feedback for the probability correction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A flowchart of a large language model alignment method based on decoding probability correction provided by an exemplary embodiment of the present application is shown;

[0079] Figure 2 The figure is a flowchart of the steps of a large model alignment method based on decoding probability correction in one embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0081] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0082] First, the terms in the art involved in this application are explained.

[0083] Large Language Model (LLM): A large language model is a natural language processing model trained using deep learning technology that can generate and understand human language. This type of model is usually based on the Transformer architecture, can process large amounts of text data, and perform tasks such as text generation, translation, and question answering by learning the structure and semantics of the language. Common examples include the GPT series, BERT, etc.

[0084] Token: In natural language processing (NLP), a token is the basic unit of text, usually a word, subword, or character. When the model processes text, it splits the text into multiple tokens for analysis and generation. For example, the sentence "I love China" can be split into three tokens: "I", "love", and "China".

[0085] Probability distribution: In this application, the probability distribution of generation refers to the distribution of possibilities that a large language model will generate for the next token given a context or prefix. Each token has a corresponding probability that indicates the likelihood of that token when it is generated. These probabilities can help the model choose the most appropriate next token when generating text.

[0086] Backtracking mechanism: In this application, the backtracking mechanism is used to optimize the generation process or correct harmful output. Specifically, when the model generates text, harmful tokens may be generated. The backtracking mechanism allows the model to re-evaluate previously generated tokens during the generation process. The system is able to backtrack to the previous state or context to make adjustments and corrections to generate safer outputs, thereby improving the harmlessness of the content generated by the large language model and ensuring the quality of the user experience.

[0087] Relationship between the backtracking mechanism and the discriminator: In this application, the discriminator can help evaluate the quality of the generated response. When the generated text is evaluated by the discriminator, if it is judged to be unqualified (for example, the discriminator determines that the generated content is harmful), the backtracking mechanism can be activated to return to the state before the text was generated and regenerate a more harmless response. By analyzing the user's input and the generated response, the discriminator can identify harmful content. The backtracking mechanism allows the system to adjust its context based on the feedback of the discriminator, thereby improving the subsequent generated text. During the training process, the discriminator can provide feedback on the quality of the generated text, prompting the model to learn safer generation strategies. When the quality of the text generated by the model is not high, the backtracking mechanism allows the model to reconsider previous choices, thereby optimizing the generation strategy and improving the overall learning effect. In summary, in this application, the discriminator provides a quality assessment of the generated content, and the backtracking mechanism allows the system to adjust and improve based on these evaluations, thereby improving the harmlessness of the content generated by the large language model.

[0088] This application is based on a large language model and natural language processing. The large language model generates text through the probability distribution of tokens. The backtracking mechanism helps to correct harmful content in the generation, and the discriminator is used to evaluate the quality of the generated content. The combination of these concepts enables the generative model to achieve higher harmlessness when processing and generating text.

[0089] Please refer to Figure 1 , which shows a flowchart of a large language model alignment method based on decoding probability correction provided by an exemplary embodiment of the present application. The method includes:

[0090] Step 101 : Generate a training set, wherein the training set is constructed according to possible continuations of a given problem.

[0091] In the training set construction stage, in order to ensure that the establishment and use of the probability correction model can fully consider the harmfulness and usefulness of information and give a reasonable correction probability distribution, the present invention takes a series of carefully designed steps to build a high-quality training set. This process not only involves the selection and generation of samples, but also includes the evaluation and scoring of the generated text. The ultimate goal is to train a model that can effectively distinguish and optimize the quality of information.

[0092] In a possible implementation, step 101 includes the following contents S11 to S17.

[0093] S11, constructing harmful samples and useful samples according to the historical data set, wherein the harmful samples and the useful samples respectively contain corresponding given problems;

[0094] Potentially harmful harmful samples and highly useful useful samples are carefully selected based on historical data sets, such as 1,000 each. These samples are selected because they are representative in content and can cover a wide range of topics and contexts, thus ensuring that the model can be exposed to diverse data during training.

[0095] S12, randomly generating different prefixes for each harmful sample and useful sample, wherein the prefixes are used as possible continuations corresponding to a given question;

[0096] In order to increase the diversity of each harmful sample and useful sample and ensure the performance of the subsequent probability correction model in different contexts, the present invention randomly generates a plurality of different prefixes, such as 1 to 10, for each sample (hereinafter, for the convenience of description, each harmful sample and useful sample will be referred to as a sample). The prefix is ​​carefully designed to simulate the diversity of questions or instructions that users may ask in the real world, that is, these possible questions are given questions corresponding to each sample, and the prefix serves as a possible continuation of the corresponding given question. In this way, the present invention can simulate more complex and realistic interaction scenarios and provide richer learning materials for subsequent probability correction models.

[0097] S13, connecting the given questions corresponding to each harmful sample and each useful sample and the corresponding generated prefixes to generate initial texts corresponding to each harmful sample and each useful sample;

[0098] The given question and the corresponding generated prefix corresponding to each sample are connected to form a complete text, which is recorded as the initial text corresponding to each harmful sample and each useful sample. The purpose of this is to allow subsequent large language models (such as RoBERTa) to generate different variant texts, such as 5, based on these complete texts. The variant texts corresponding to these initial texts are different not only in language expression, but also in information presentation and organization, which provides more options and possibilities for subsequent evaluation and scoring.

[0099] S14, generating a variant text corresponding to the initial text by using the large language model;

[0100] Furthermore, it is also mentioned in S13 that a variant text corresponding to the initial text will be generated through a large language model.

[0101] S15, scoring the usefulness of each variant text corresponding to the harmful sample through the reward model;

[0102] S16, scoring the harmfulness of each variant sample corresponding to the harmful sample through a content security classification model;

[0103] Furthermore, after generating all possible text variants, the present invention uses two different models for scoring. First, a reward model (such as ArmoRM-Llama3-8B-v0.1) is used to score the usefulness of each useful text. This model can evaluate the effectiveness of the text in providing information, solving problems, or meeting user needs. Secondly, a content safety classification model (such as Llama-Guard-3-8B) is used to score the harmfulness of the text. This model focuses on identifying and evaluating inappropriate content that may exist in the text, such as hate speech, false information, or other harmful information.

[0104] S17, normalizing and weighting the usefulness score and the harmfulness score to obtain the correction probability of each given question and the corresponding prefix as the training set.

[0105] The usefulness score and harmfulness score are normalized to ensure the consistency and comparability of the scores. These scores are weighted averaged to calculate the correction probability of each given question and its corresponding prefix as a training set. This correction probability not only reflects the performance of the text in terms of usefulness, but also takes into account the potential risks of the text in terms of harmfulness, thereby providing a comprehensive evaluation index for the model to guide it to make more reasonable and effective correction decisions in practical applications. Through such data construction and scoring mechanism, the present invention can train a probability correction model that can both identify and optimize useful information and effectively reduce and control harmful information.

[0106] Step 102: A probability correction model is obtained by training a large language model according to the training set.

[0107] The present invention is based on the training set constructed in step 101, and a probability correction model is trained. First of all, let’s talk about the effect of the probability correction model on the large language model in this application. In the text generation process of the large language model, the large language model will first give the generation probability distribution of the next token based on the prefix. However, this distribution may produce some outputs that do not conform to human preferences due to deviations or other reasons in the training data of the large language model. To this end, we train a plug-and-play probability correction model through steps 101 and 102, which adjusts or optimizes the output probability of the large language model by intervening in the reasoning process of the original large language model.

[0108] In order to facilitate the expression of various forms, the present invention takes the large language model RoBERTa as an example to illustrate the content of constructing a probability correction regression model. The large language model predicts the probability of whether the next candidate word will cause the sentence to become harmful under the condition of a given prefix.

[0109] In a possible implementation, step 102 includes the following contents S21 to S23.

[0110] S21, the present invention expresses the input form of the large language model according to the training set as follows Where Prefix represents the generated prefix part, t k is the next candidate token (i.e. the basic unit of text), <s>It is used to aggregate the information of the entire input sequence. The RoBERTa model processes the input and extracts <s>Labeled feature vector where d m Represents the feature dimension.

[0111] S22, through a multi-layer perceptron, the feature vector h CLS The vector is processed and the sigmoid function is used to map the output to a probability value in the [0,1] interval: in, is the quality probability of the model output (including harmless quality and useful quality), σ(x) is the sigmoid function;

[0112] S23, using mean square error as a loss function to train the probability correction model, the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE for:

[0113]

[0114] Step 103, during the text generation process of the large language model, the generation probability of the next text basic unit is adjusted using the trained probability correction model, and the text basic unit is the basic unit for generating text.

[0115] In a possible implementation, step 103 includes the following contents S31 to S34.

[0116] S31, uses the trained probability correction model to adjust the generation probability of the next text basic unit (i.e., candidate token), reduces the generation probability of tokens that do not meet human preferences, and increases the generation probability of tokens that meet human preferences.

[0117] In practical applications, the present invention performs correction evaluation on the top 30 tokens with the highest probability when the large language model generates the next token.

[0118] S32, under the condition of a given prefix prefix, assuming that the original probability of the large language model generating the next candidate token is P LLM (t|prefix), P CM (t|prefix) represents the probability that the probability correction model evaluates whether the token conforms to human preferences.

[0119] S33, the generation probability after correction through weighted fusion output is:

[0120]

[0121] Wherein, LLM refers to the large language model, CM refers to the probability correction model, α and β are hyperparameters that control the relative contributions of the large language model and the probability correction model, and satisfy α+β=1.

[0122] S34, introduction of normalization constant The corrected generation probability is normalized and finally adjusted to obtain the generation probability of the next basic unit of text.

[0123] In order to ensure that the final generation probability of the candidate token is a valid probability distribution, the present invention introduces a normalization constant for normalization processing, and finally adjusts the generation probability of the next basic unit of the text.

[0124] Figure 2 The figure is a flowchart of the steps of a large model alignment method based on decoding probability correction in one embodiment of the present invention.

[0125] Step 104: Evaluate the generated text using a discriminator.

[0126] In order to further ensure the alignment effect of LLM generated content, the present invention introduces a backtracking mechanism based on a discriminator, which can dynamically adjust the generation path during the text generation process. The discriminator evaluates whether the currently generated content meets the security standards from a global perspective. In this application, the content security classification model, namely Llama-Guard-3-8B, used by the discriminator is used as an example for explanation.

[0127] Specifically, when a token is generated, the discriminator evaluates the global context of the current sequence. If the discriminator believes that all the current generation does not meet the alignment criteria, the model will go back to the previous time step, reselect the candidate token and continue generating.

[0128] At time step t, the token sequence generated by the large model is represented as x t ={t0, t1, ..., t t-1 }, and prepare to select T from the candidate token set t ={t t1 , t t2 ,,t tn Generate a new token in} t The generation probability of each candidate token in the candidate token set is H(t ti |x t ), which means that in the context x t Generate candidate tokens ti probability.

[0129] In a possible implementation, step 104 includes the following contents S41 to S42.

[0130] S41, introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0131] S42, when Indicates the added t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0132] where t ti In the example, i = 1, 2, ..., n, and the corresponding candidate token set is T t ={t t1 , t t2 ,,t tn }.

[0133] In a possible implementation, in order to ensure the reliability of the generation process, the present invention introduces a discriminator D(·)∈{0,1} to evaluate the currently generated token sequence x t Whether the alignment standard is met. This means adding t ti The sequence passes the alignment criteria, otherwise t ti Add to the banned list, that is, B t =B t ∪t ti The present invention finally selects the candidate token with the highest generation probability to add to the generation sequence and continues to generate subsequent content, which is expressed as

[0134] Step 105 , in response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic units of the generated text by introducing a backtracking mechanism in the large language model.

[0135] In a possible implementation, step 105 includes the following contents S51 to S53.

[0136] S51, if at time t, the candidate token set is evaluated as unqualified by the discriminator, triggering a backtracking mechanism;

[0137] S52, return to the previous time step t-1 through the backtracking mechanism, and convert the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1;

[0138] S53, reselecting a token that is not in the forbidden list from the candidate token set of the previous time step through the backtracking mechanism to generate a token until a token that meets the alignment criteria is found.

[0139] If at time t, the discriminator evaluates all candidate tokens as unqualified, the backtracking mechanism will be triggered. The backtracking mechanism will go back to the previous time step t-1 and replace the current t t-1 Add to the banned list, that is, B t-1 =B t-1 ∪t t-1 , and reselect tokens that are not in the banned list from the candidate token set of the previous time step for generation until a token that meets the alignment criteria is found, thereby effectively suppressing the generation of harmful content, which is expressed as

[0140] In summary, the present invention designs a plug-and-play probability correction mechanism to address the problem that large language models may deviate from human preference standards during text generation. This mechanism can make fine-grained probability adjustments in real time during the reasoning process of the model, and dynamically optimize the probability distribution of each generation step to ensure that the generated content is not only fluent and natural in language, but also closer to human values ​​and expectations in semantics. Specifically, the present invention integrates a lightweight probability correction model that can analyze and adjust the probability of each word generated by the large language model, thereby improving the quality and harmlessness of the text without sacrificing text diversity and creativity. In addition, in response to the security and ethical issues that may exist in text generation in the prior art, a global evaluation and backtracking mechanism based on a discriminator is designed. This mechanism introduces an advanced discriminator model to perform strict security verification on each generated token, ensuring that the generated text is not only correct in language, but also meets ethical and security standards in content. When the discriminator detects that the generated text may contain harmful, inappropriate or sensitive content, it will immediately trigger the backtracking mechanism, causing the model to return to the previous generation step and reselect safe candidate tokens, thereby avoiding the generation of content that does not meet ethical standards. This discriminator-based security assurance mechanism not only improves the model’s ability to identify and respond to potential risks, but also enhances the reliability and compliance of generated text through real-time monitoring and adjustment.

[0141] An exemplary embodiment of the present application further provides a large language model alignment device based on decoding probability correction, the device comprising:

[0142] A training generation module, for generating a training set through a large language model, wherein the training set is composed of possible continuations of a given question;

[0143] A model generation module, used for obtaining a probability correction model through training of the training set;

[0144] A probability adjustment module, used to adjust the generation probability of the next text basic unit using the trained probability correction model during the text generation process of the large language model, wherein the text basic unit is the basic unit for generating text;

[0145] an identification and evaluation module, used for evaluating the generated text through a discriminator;

[0146] The backtracking adjustment module is used to backtrack and adjust the text basic units of the generated text by introducing a backtracking mechanism in the large language model in response to the generated text not meeting the alignment standard.

[0147] Optionally, the training generation module includes:

[0148] A first generating unit, configured to construct harmful samples and useful samples according to a historical data set, wherein the harmful samples and the useful samples respectively contain corresponding given problems;

[0149] A second generating unit, configured to randomly generate different prefixes for each harmful sample and useful sample, wherein the prefixes are used as possible continuations corresponding to a given question;

[0150] A third generating unit is used to connect the given questions corresponding to each harmful sample and each useful sample with the corresponding generated prefixes to generate initial texts corresponding to each harmful sample and each useful sample;

[0151] A fourth generating unit, configured to generate a variant text corresponding to the initial text by using the large language model;

[0152] A fifth generation unit, used for scoring the usefulness of each variant text corresponding to the harmful sample through a reward model;

[0153] a sixth generation unit, configured to score the harmfulness of each variant sample corresponding to the harmful sample by using a content security classification model;

[0154] The seventh generating unit is used to normalize and weight the usefulness score and the harmfulness score to obtain the correction probability of each given question and the corresponding prefix as the training set.

[0155] Optionally, the model generation module includes:

[0156] The first model unit is used to express the input form of the large language model according to the training set as Where Prefix represents the generated prefix, t k is the next candidate text basic unit, <s>The large language model is used to aggregate the information of the entire input sequence and extract <s>Labeled feature vector where d m Represents the feature dimension.

[0157] The second model unit is used to analyze the feature vector h through a multi-layer perceptron. CLS The vector is processed and the sigmoid function is used to map the output to a probability value in the [0,1] interval: in, is the quality probability of the model output, σ(x) is the sigmoid function;

[0158] The third model unit is used to use the mean square error as a loss function to train the probability correction model, and the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE for:

[0159]

[0160] Optionally, the probability adjustment module includes:

[0161] The first adjustment unit is used to adjust the generation probability of the next text basic unit using the trained probability correction model, reduce the generation probability of tokens that do not meet human preferences, and increase the probability of tokens that meet human preferences, where tokens are represented as text basic units;

[0162] The second adjustment unit is used to assume that the original probability of the large language model generating the next candidate token is P under the condition of a given prefix prefix LLM (t|prefix), P CM (t|prefix) represents the probability of the probability correction model evaluating whether the token conforms to human preferences;

[0163] The third adjustment unit is used to output the corrected generation probability through weighted fusion:

[0164]

[0165] Wherein, LLM refers to the large language model, CM refers to the probability correction model, α and β are hyperparameters that control the relative contribution of the large language model and the probability correction model, and satisfy α+β=1;

[0166] The fourth adjustment unit is used to introduce a normalization constant The corrected generation probability is normalized and finally adjusted to obtain the generation probability of the next basic unit of text.

[0167] Optionally, at time step t, the token sequence generated by the large language model is represented as x t ={t0, t1, ..., t t-1 }, and the large language model is used to select from the candidate token set T t ={t t1 , t t2 ,,t tn Generate a new token in} t , the generation probability of each candidate token in the candidate token set is expressed as H(t ti |x t ), the H(t ti |x t ) means in context x t Generate candidate tokens ti probability.

[0168] Optionally, the identification and evaluation module includes:

[0169] The first evaluation unit is used to introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0170] The second evaluation unit is used when Indicates the added t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0171] Optionally, the retrospective adjustment module includes:

[0172] A first adjustment unit, configured to trigger a backtracking mechanism if the candidate token set is evaluated as unqualified by the discriminator at time t, wherein the candidate token is used to indicate the basic unit of the text;

[0173] The second adjustment unit is used to return to the previous time step t-1 through the backtracking mechanism and adjust the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1;

[0174] The third adjustment unit is used to reselect a token that is not in the forbidden list from the candidate token set of the previous time step through the backtracking mechanism to generate until a token that meets the alignment standard is found.

[0175] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0176] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0177] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.< / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s>

Claims

1. A large language model alignment method based on decoding probability correction, characterized in that: The method comprises: Generating a training set, wherein the training set is composed of possible continuations of a given problem; A probability correction model is obtained by training a large language model according to the training set; In the text generation process of the large language model, the generation probability of the next text basic unit is adjusted using the trained probability correction model, and the text basic unit is the basic unit for generating text; evaluating the generated text by a discriminator; In response to the generated text not meeting the alignment standard, a backtracking mechanism is introduced in the large language model to backtrack and adjust the text basic units of the generated text.

2. The method according to claim 1, characterized in that The generating training set is formed according to possible continuations of a given problem, and includes: Constructing harmful samples and useful samples according to historical data sets, wherein the harmful samples and the useful samples respectively contain corresponding given problems; Randomly generate different prefixes for each harmful sample and useful sample, the prefixes being used as possible continuations corresponding to a given question; Connecting the given questions corresponding to each harmful sample and each useful sample and the corresponding generated prefixes to generate the initial texts corresponding to each harmful sample and each useful sample; Generate a variant text corresponding to the initial text by using the large language model; The usefulness of each variant text corresponding to the harmful sample is scored through the reward model; The harmfulness of each variant sample corresponding to the harmful sample is scored through the content security classification model; The usefulness score and the harmfulness score are normalized and weighted to obtain the correction probability of each given question and the corresponding prefix as the training set.

3. The method according to claim 2, characterized in that The method of obtaining a probability correction model through training the training set includes: According to the training set, the input form of the large language model is expressed as Where Prefix represents the generated prefix, t k is the next candidate text basic unit, <s>The large language model is used to aggregate the information of the entire input sequence and extract <s>Labeled feature vector where d m Represents the feature dimension.< / s> < / s> <s> <s> The feature vector h is processed by a multi-layer perceptron CLS The vector is processed and the sigmoid function is used to map the output to a probability value in the [0,1] interval: in, is the quality probability of the model output, σ(x) is the sigmoid function; The probability correction model is trained using the mean square error as the loss function, and the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE for:

4. The method according to claim 3, characterized in that In the text generation process of the large language model, the trained probability correction model is used to adjust the generation probability of the next text basic unit, including: Use the trained probability correction model to adjust the generation probability of the next basic unit of text, reduce the generation probability of tokens that do not meet human preferences, and increase the probability of tokens that meet human preferences, where tokens are represented as basic units of text; Given a prefix prefix, assume that the original probability of the large language model generating the next candidate token is P LLM (t|prefix), P CM (t|prefix) represents the probability of the probability correction model evaluating whether the token conforms to human preferences; The generation probability after correction by weighted fusion output is: Wherein, LLM refers to the large language model, CM refers to the probability correction model, α and β are hyperparameters that control the relative contribution of the large language model and the probability correction model, and satisfy α+β=1; Introducing a normalization constant The corrected generation probability is normalized and finally adjusted to obtain the generation probability of the next basic unit of text.

5. The method according to claim 1, characterized in that At time step t, the token sequence generated by the large language model is represented as x t ={t0, t1, ..., t t-1 }, and the large language model is used to select from the candidate token set T t ={t t1 , t t2 ,,t tn Generate a new token in} t , the generation probability of each candidate token in the candidate token set is expressed as H(t ti |x t ), the H(t ti |x t ) means in context x t Generate candidate tokens ti probability.

6. The method according to claim 5, characterized in that The step of evaluating the generated text by a discriminator includes: Introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met; when Indicates the added t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

7. The method according to claim 1, characterized in that In response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic units of the generated text by introducing a backtracking mechanism in the large language model, including: If at time t, the candidate token set is evaluated as unqualified by the discriminator, a backtracking mechanism is triggered, wherein the candidate token is used to indicate the basic unit of the text; The backtracking mechanism is used to return to the previous time step t-1 and the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1; The backtracking mechanism reselects tokens that are not in the forbidden list from the candidate token set of the previous time step for generation until a token that meets the alignment criteria is found.

8. A large language model alignment device based on decoding probability correction, characterized in that: The device comprises: A training generation module, for generating a training set through a large language model, wherein the training set is composed of possible continuations of a given question; A model generation module, used to obtain a probability correction model through training of the training set; A probability adjustment module, used to adjust the generation probability of the next text basic unit using the trained probability correction model during the text generation process of the large language model, wherein the text basic unit is the basic unit for generating text; an identification and evaluation module, used for evaluating the generated text through a discriminator; The backtracking adjustment module is used to backtrack and adjust the text basic units of the generated text by introducing a backtracking mechanism in the large language model in response to the generated text not meeting the alignment standard.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the large language model alignment method based on decoding probability correction as described in any one of claims 1 to 7. < / s> < / s>

Images