Large language model alignment method and device based on decoding probability correction

Abstract

The application discloses a large language model alignment method and device based on decoding probability correction, belonging to the field of artificial intelligence. By introducing a lightweight probability correction model, the original generation probability distribution of the large language model is monitored and intervened in real time during the decoding process. It analyzes the generation probability assigned by the large language model to each candidate token and dynamically adjusts these probabilities in real time using the probability correction model. Meanwhile, the application also introduces a global evaluation mechanism based on a discriminator to further enhance the quality control and alignment guarantee of the generated content. This discriminator not only has the ability to evaluate the generated text as a whole, but also can identify potential inappropriate content or bias, thereby providing more comprehensive and accurate feedback to the probability correction model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence, in particular to a large language model alignment method and device based on decoding probability correction. BACKGROUND

[0002] In recent years, pre-trained large language models (LLMs) have demonstrated their powerful capabilities in multiple fields. These models have learned rich knowledge through large-scale corpora and have become an important part of online services and applications, widely used in critical areas such as healthcare, finance, and law. However, as their applications become widespread, issues of security and robustness have become increasingly prominent, particularly the increase in adversarial attacks that cause models to generate harmful content, raising concerns about social and ethical risks.

[0003] Pre-trained LLMs can reflect biases in their training data and propagate discriminatory content in certain cases, especially when the input contains adversarial prompts. This has prompted researchers to explore various alignment methods, such as reinforcement learning based on human feedback, reinforcement learning based on artificial intelligence feedback, and direct preference optimization, to improve the safety and compliance of generated content. However, these methods present significant challenges in implementation, including high human costs for high-quality data labeling and the enormous time and computational resources required during training. Furthermore, although aligned models have enhanced their ability to resist adversarial attacks to some extent, they can still be bypassed by cleverly designed attacks.

[0004] In addition, during the training process, the instability of the model and the reliance on engineering knowledge make it particularly important to align without changing the model parameters. To this end, some lightweight alignment methods have been proposed that do not require retraining the model. For example, some methods rewrite detected adversarial inputs to mitigate attack effects, while others rely on context examples to guide the generation process. However, these methods often depend on accurate identification of attacks and are difficult to deal with more complex and hidden attacks.

[0005] Other techniques such as perplexity-based anomaly detection and dynamic constraint generation, while improving security to some extent, still struggle when faced with complex attacks. Although existing methods have improved in effectiveness, they generally lack real-time intervention capabilities and are difficult to control generated content in fine-grained detail, so there are still deficiencies in dealing with complex adversarial attacks. Therefore, it is of great practical significance to study alignment methods for large language models. SUMMARY

[0006] Embodiments of the present application provide a large language model alignment method and device based on decoding probability correction to address the lack of real-time intervention capabilities when dealing with complex adversarial attacks in the alignment process of large language models.

[0007] In an aspect, a large language model alignment method based on decoding probability correction is provided, the method comprising:

[0008] generating a training set according to possible continuations of a given problem;

[0009] training a probability correction model according to the training set by a large language model;

[0010] adjusting the generation probability of a next text basic unit in a text generation process of the large language model using the trained probability correction model, the text basic unit being a basic unit of the generated text;

[0011] evaluating the generated text by a discriminator;

[0012] in response to the generated text not meeting an alignment standard, backtracking and adjusting the text basic unit of the generated text by introducing a backtracking mechanism in the large language model.

[0013] Optionally, the generated training set according to possible continuations of a given problem comprises:

[0014] constructing harmful samples and useful samples from historical data sets, the harmful samples and the useful samples respectively containing corresponding given problems;

[0015] randomly generating different prefixes for each harmful sample and useful sample, the prefix being a possible continuation of the corresponding given problem;

[0016] connecting the corresponding given problem and the corresponding generated prefix of each harmful sample and each useful sample to generate an initial text corresponding to each harmful sample and each useful sample;

[0017] generating a variant text corresponding to the initial text by the large language model;

[0018] scoring the usefulness of each variant text corresponding to the harmful sample by a reward model;

[0019] scoring the harmfulness of each variant sample corresponding to the harmful sample by a content safety classification model;

[0020] normalizing and weighting the usefulness score and the harmfulness score to obtain a correction probability of each given problem and the corresponding prefix as the training set.

[0021] Optionally, the training of the probability correction model according to the training set by the large language model comprises:

[0022] expressing an input form of the large language model as wherein Prefix represents a generated prefix, t k for the next candidate text base unit, <s>to aggregate information for an entire input sequence, the large language model is configured to extract out <s>The labeled feature vector where d m represents the feature dimension.

[0023] The feature vector h CLS is processed by a multi-layer perceptron, and the output is mapped to a probability value in the interval [0, 1] using a sigmoid function: where, is the quality probability of the model output, and σ(x) is the sigmoid function.

[0024] The mean square error is used as the loss function to train the probability correction model, and the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE is:

[0025]

[0026] Optionally, in the text generation process of the large language model, the trained probability correction model is used to adjust the generation probability of the next text basic unit, comprising:

[0027] The trained probability correction model is used to adjust the generation probability of the next text basic unit, so as to reduce the generation probability of tokens that do not conform to human preferences, and at the same time, to improve the probability of tokens that conform to human preferences, wherein token represents a text basic unit.

[0028] Under the condition of a given prefix prefix, it is assumed that the original probability of the large language model generating the next candidate token is P LLM (t|prefix), P CM (t|prefix) represents the probability that the probability correction model evaluates whether the token conforms to human preferences.

[0029] The output corrected generation probability is:

[0030]

[0031] Where LLM refers to the large language model, CM refers to the probability correction model, and α and β are hyperparameters that control the relative contribution of the large language model and the probability correction model, and satisfy α+β=1.

[0032] A normalization constant is introduced.

[0033] Optionally, at time step t, the token sequence generated by the large language model is represented as x t ={t0, t1, ..., t t-1 }, and the large language model is used to select the candidate token set T t ={t t1 , t t2 ,,t tn Generate a new token in} t , the generation probability of each candidate token in the candidate token set is expressed as H(t ti |x t ), the H(t ti |x t ) means in context x t Generate candidate tokens ti probability.

[0034] Optionally, evaluating the generated text by a discriminator includes:

[0035] Introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0036] when Indicates the addition of t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0037] Optionally, in response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic units of the generated text by introducing a backtracking mechanism in the large language model includes:

[0038] If at time t, the candidate token set is evaluated as unqualified by the discriminator, a backtracking mechanism is triggered, wherein the candidate token is used to indicate the basic unit of the text;

[0039] Go back to the previous time step t-1 through the backtracking mechanism, and set the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1;

[0040] The backtracking mechanism reselects tokens that are not in the forbidden list from the candidate token set of the previous time step for generation until a token that meets the alignment criteria is found.

[0041] In another aspect, a large language model alignment device based on decoding probability correction is provided, the device comprising:

[0042] a training generation module configured to generate a training set by a large language model, the training set being constituted according to possible continuations of a given question;

[0043] a model generation module configured to train a probability correction model by the training set;

[0044] a probability adjustment module configured to adjust the generation probability of a next text basic unit in the text generation process of the large language model using the trained probability correction model, the text basic unit being a basic unit of the generated text;

[0045] a discriminative evaluation module configured to evaluate the generated text by a discriminator;

[0046] a backtracking adjustment module configured to backtrack and adjust the text basic units of the generated text by introducing a backtracking mechanism in the large language model in response to the generated text not meeting the alignment standard.

[0047] Optionally, the training set is generated according to possible continuations of a given question, comprising:

[0048] constructing harmful samples and useful samples according to historical data sets, the harmful samples and the useful samples respectively containing corresponding given questions;

[0049] randomly generating different prefixes for each harmful sample and useful sample, the prefix being a possible continuation of the corresponding given question;

[0050] connecting the corresponding given question and the corresponding generated prefix of each harmful sample and each useful sample to generate an initial text corresponding to each harmful sample and each useful sample;

[0051] generating a variant text corresponding to the initial text by the large language model;

[0052] scoring the usefulness of each variant text corresponding to the harmful sample by a reward model;

[0053] scoring the harmfulness of each variant sample corresponding to the harmful sample by a content security classification model;

[0054] normalizing and weighting the usefulness score and the harmfulness score to obtain a correction probability of each given question and the corresponding prefix as the training set.

[0055] Optionally, the probability correction model is trained by the training set, comprising:

[0056] expressing an input form of the large language model as wherein Prefix represents a generated prefix, t k for the next candidate text elementary unit, <s>to aggregate information for an entire input sequence, the large language model is configured to extract out <s>The labeled feature vector where d m represents the feature dimension.

[0057] The feature vector h CLS is processed by a multi-layer perceptron, and the output is mapped to a probability value in the interval [0, 1] using a sigmoid function: where, is the quality probability of the model output, and σ(x) is the sigmoid function.

[0058] The mean square error is used as the loss function to train the probability correction model, and the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE is:

[0059]

[0060] Optionally, in the text generation process of the large language model, the trained probability correction model is used to adjust the generation probability of the next text basic unit, comprising:

[0061] The trained probability correction model is used to adjust the generation probability of the next text basic unit, so as to reduce the generation probability of tokens that do not conform to human preferences, and to improve the probability of tokens that conform to human preferences, wherein token represents a text basic unit.

[0062] Under the condition of a given prefix prefix, it is assumed that the original probability of the large language model generating the next candidate token is P LLM (t|prefix), P CM (t|prefix) represents the probability that the probability correction model evaluates whether the token conforms to human preferences.

[0063] The output corrected generation probability is obtained by weighted fusion:

[0064]

[0065] wherein LLM refers to the large language model, CM refers to the probability correction model, and α and β are hyperparameters for controlling the relative contribution of the large language model and the probability correction model, and satisfy α+β=1.

[0066] A normalization constant is introduced.

[0067] Optionally, at time step t, the token sequence generated by the large language model is represented as x t ={t0, t1, ..., t t-1 }, and the large language model is used to select the candidate token set T t ={t t1 , t t2 ,,t tn Generate a new token in} t , the generation probability of each candidate token in the candidate token set is expressed as H(t ti |X t ), the H(t ti |x t ) means in context x t Generate candidate tokens ti probability.

[0068] Optionally, evaluating the generated text by a discriminator includes:

[0069] Introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0070] when Indicates the addition of t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0071] Optionally, in response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic units of the generated text by introducing a backtracking mechanism in the large language model includes:

[0072] If at time t, the candidate token set is evaluated as unqualified by the discriminator, a backtracking mechanism is triggered, wherein the candidate token is used to indicate the basic unit of the text;

[0073] Go back to the previous time step t-1 through the backtracking mechanism, and set the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1;

[0074] The backtracking mechanism reselects tokens that are not in the forbidden list from the candidate token set of the previous time step for generation until a token that meets the alignment criteria is found.

[0075] In another aspect, a computer-readable storage medium is provided, the storage medium storing at least one instruction for execution by a processor to implement the decoding probability correction-based large language model alignment method as described in the above aspect.

[0076] In another aspect, a computer program product is also provided, the computer program product storing at least one instruction loaded and executed by the processor to implement the decoding probability correction-based large language model alignment method as described in the above aspect.

[0077] The present application discloses a decoding probability correction-based large language model alignment method and device, belonging to the field of artificial intelligence. By introducing a lightweight probability correction model, the original generation probability distribution of the large language model is monitored and intervened in real time during the decoding process. It analyzes the generation probability assigned by the large language model to each candidate token and dynamically adjusts these probabilities in real time using the probability correction model. Meanwhile, the present application also introduces a global evaluation mechanism based on a discriminator to further enhance the quality control and alignment guarantee of the generated content. This discriminator not only has the ability to evaluate the generated text as a whole, but also can identify potential inappropriate content or bias, thereby providing more comprehensive and accurate feedback to the probability correction model. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 A flowchart of the decoding probability correction-based large language model alignment method provided by an exemplary embodiment of the present application is shown;

[0079] Figure 2 A flowchart of the decoding probability correction-based large model alignment method steps in an embodiment of the present application. DETAILED DESCRIPTION

[0080] To make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0081] "Multiple" referred to herein means two or more. "And / or", describing the association relationship of associated objects, means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0082] First, the terms related to the present application in the art are explained.

[0083] Large Language Model (LLM): A Large Language Model is a natural language processing model trained using deep learning techniques, capable of generating and understanding human language. These models are typically based on the Transformer architecture, allowing them to process large amounts of text data and learn the structure and semantics of language to perform tasks such as text generation, translation, and question answering. Common examples include the GPT series, BERT, and others.

[0084] Token: In natural language processing (NLP), a token is the basic unit of text, usually a word, subword, or character. Models break down text into tokens for analysis and generation when processing text. For example, the sentence "I love China" can be broken down into three tokens: "I", "love", and "China".

[0085] Probability distribution: In this application, the probability distribution generated refers to the likelihood distribution of the next token generated by the large language model given the context or prefix. Each token has a corresponding probability indicating the likelihood of that token being generated. These probabilities can help the model choose the most appropriate next token when generating text.

[0086] Backtracking mechanism: In this application, the backtracking mechanism is used to optimize the generation process or correct harmful output. Specifically, when the model generates text, it may produce harmful tokens. The backtracking mechanism allows the model to re-evaluate previously generated tokens during the generation process, allowing the system to backtrack to previous states or contexts to adjust and correct them to generate safer output, thereby improving the safety of the content generated by the large language model and ensuring the quality of the user experience.

[0087] Relationship between backtracking mechanism and discriminator: In this application, the discriminator can help evaluate the quality of the generated response. When the generated text is evaluated by the discriminator and is determined to be substandard (for example, the discriminator determines that the generated content is harmful), the backtracking mechanism can be activated to return to the state before generating the text and generate a safer response. By analyzing the user's input and the generated response, the discriminator can identify harmful content. The backtracking mechanism allows the system to adjust its context based on the feedback from the discriminator, thereby improving the subsequent generated text. During training, the discriminator can provide feedback on the quality of the generated text, prompting the model to learn safer generation strategies. When the quality of the generated text is not high, the backtracking mechanism allows the model to reconsider previous choices, thereby optimizing the generation strategy and improving the overall learning effect. In summary, in this application, the discriminator provides quality assessments of the generated content, while the backtracking mechanism allows the system to adjust and improve based on these assessments, thereby improving the safety of the content generated by the large language model.

[0088] The present application is based on a large language model and natural language processing. The large language model generates text through the probability distribution of tokens, and the backtracking mechanism helps to correct harmful content in generation, while the discriminator is used to evaluate the quality of the generated content. The combination of these concepts enables the generation model to achieve higher harmlessness when processing and generating text.

[0089] Reference is made to Figure 1 which shows a flowchart of the large language model alignment method based on decoding probability correction provided by an exemplary embodiment of the present application. The method comprises:

[0090] Step 101, generating a training set, the training set is constituted according to the possible continuation of the given problem.

[0091] In the training set construction stage, in order to ensure that the establishment and use of the probability correction model can comprehensively consider the harmfulness and usefulness of information and give a reasonable correction probability distribution, the present application takes a series of carefully designed steps to construct a high-quality training set. This process not only involves the selection and generation of samples, but also includes the evaluation and scoring of generated text, and the ultimate goal is to train a model that can effectively distinguish and optimize information quality.

[0092] In one possible implementation, step 101 includes the following contents S11 to S17.

[0093] S11, constructing harmfulness samples and usefulness samples according to historical data sets, the harmfulness samples and the usefulness samples respectively containing corresponding given problems;

[0094] According to the historical data set, carefully selected harmfulness samples with potential harmfulness and usefulness samples with high usefulness, such as 1000. These samples are selected because they are representative in content and can cover a wide range of topics and contexts, so as to ensure that the model can access diversified data during the training process.

[0095] S12, randomly generating different prefixes for each harmfulness sample and usefulness sample, the prefix as a possible continuation of the corresponding given problem;

[0096] To increase the diversity of each harmful sample and useful sample and ensure the performance of the subsequent probabilistic correction model in different contexts, the present application randomly generates multiple different prefixes for each sample (for the sake of convenience, each harmful sample and useful sample is referred to as a sample hereinafter). The prefixes are carefully designed to simulate the diversity of questions or instructions that users may ask in the real world, i.e. these possible questions are the given questions corresponding to each sample, and the prefixes are the possible continuations of the given questions. In this way, the present application can simulate more complex and realistic interaction scenarios and provide more abundant learning materials for the subsequent probabilistic correction model.

[0097] S13, connecting the given question corresponding to each harmful sample and each useful sample and the corresponding generated prefix to generate an initial text corresponding to each harmful sample and each useful sample;

[0098] The given question corresponding to each sample and the corresponding generated prefix are connected to form a complete text, which is referred to as the initial text corresponding to each harmful sample and each useful sample. The purpose of this is to enable the subsequent large language model (such as RoBERTa) to generate different variant texts, such as 5, based on these complete texts. These initial texts correspond to variant texts that not only differ in language expression, but also differ in information presentation and organization, thereby providing more options and possibilities for subsequent evaluation and scoring.

[0099] S14, generating variant texts corresponding to the initial texts by the large language model;

[0100] Further, in S13, it is mentioned that variant texts corresponding to the initial texts will be generated by the large language model.

[0101] S15, scoring the usefulness of each variant text corresponding to the harmful sample by the reward model;

[0102] S16, scoring the harmfulness of each variant sample corresponding to the harmful sample by the content safety classification model;

[0103] Further, after generating all possible text variants, the present application uses two different models for scoring. First, a reward model (such as ArmoRM-Llama3-8B-v0.1) is used to score the usefulness of each useful text. This model can evaluate the effectiveness of the text in providing information, solving problems or meeting user needs. Second, a content safety classification model (such as Llama-Guard-3-8B) is used to score the harmfulness of the text, which focuses on identifying and evaluating inappropriate content in the text, such as hate speech, false information or other harmful information.

[0104] S17, normalize and weight the usefulness score and the harmfulness score to obtain a correction probability of each given question and the corresponding prefix as the training set.

[0105] The usefulness score and the harmfulness score are normalized to ensure consistency and comparability of the scores. These scores are weighted and averaged to calculate the correction probability of each given question and its corresponding prefix as the training set. This correction probability not only reflects the performance of the text in terms of usefulness, but also considers the potential risks of the text in terms of harmfulness, thereby providing a comprehensive evaluation index for the model to guide it to make more reasonable and effective correction decisions in practical applications. Through such data construction and scoring mechanism, the present application can train a probability correction model that can not only identify and optimize useful information, but also effectively reduce and control harmful information.

[0106] Step 102, training a probability correction model according to the training set by a large language model.

[0107] The present application trains a probability correction model based on the training set constructed in step 101. First, the effect of the probability correction model on the large language model in this application is described. In the text generation process of the large language model, the large language model will first give the generation probability distribution of the next token according to the prefix. However, this distribution may produce some outputs that do not conform to human preferences due to biases in the training data of the large language model or other reasons. Therefore, we train a plug-and-play probability correction model through steps 101 and 102, which adjusts or optimizes the output probability of the large language model by intervening in the reasoning process of the original large language model.

[0108] To facilitate the expression of various forms, the present application takes RoBERTa as an example to illustrate the construction of the probability correction regression model. The large language model predicts the probability of whether the next candidate word will cause the sentence to become harmful under the condition of a given prefix.

[0109] In one possible implementation, step 102 includes the following contents S21 to S23.

[0110] S21, the input form expression of the large language model according to the training set is wherein Prefix represents the generated prefix part, t k is the next candidate token (i.e. the basic unit of text), <s>is used to aggregate information from the entire input sequence. The RoBERTa model extracts <s>Labeled feature vector where d m Represents the feature dimension.

[0111] S22, the feature vector h is processed by a multi-layer perceptron. CLS The vector is processed and the sigmoid function is used to map the output to a probability value in the [0,1] interval: in, is the quality probability of the model output (including harmless quality and useful quality), σ(x) is the sigmoid function;

[0112] S23, using mean square error as a loss function to train the probability correction model, the mean square error is used to optimize the prediction result of the probability correction model, wherein the loss function L MSE for:

[0113]

[0114] Step 103 , during the text generation process of the large language model, the trained probability correction model is used to adjust the generation probability of the next text basic unit, where the text basic unit is the basic unit for generating text.

[0115] In a possible implementation, step 103 includes the following contents S31 to S34.

[0116] S31 uses the trained probability correction model to adjust the generation probability of the next basic unit of text (i.e., candidate token), reducing the generation probability of tokens that do not conform to human preferences, while increasing the generation probability of tokens that conform to human preferences.

[0117] In practical applications, the present invention performs correction evaluation on the top 30 tokens with the highest probability when the large language model generates the next token.

[0118] S32, given the prefix prefix, assuming that the original probability of the large language model generating the next candidate token is P LLM (t|prefix), P CM (t|prefix) represents the probability that the probability correction model assesses whether the token conforms to human preferences.

[0119] S33, the generation probability after correction by weighted fusion output is:

[0120]

[0121] Wherein, LLM refers to the large language model, CM refers to the probability correction model, and alpha and beta are hyperparameters that control the relative contribution of the large language model and the probability correction model, and satisfy alpha+beta=1.

[0122] S34, introducing a normalization constant The corrected generation probability is normalized, and the generation probability of the next text basic unit is finally adjusted.

[0123] In order to ensure that the final generation probability of the candidate token is a valid probability distribution, the present application introduces a normalization constant for normalization processing, and finally adjusts the generation probability of the next text basic unit.

[0124] Figure 2 The flowchart of the large model alignment method based on decoding probability correction in an embodiment of the present application.

[0125] Step 104, evaluating the generated text by the discriminator.

[0126] In order to further ensure the alignment effect of LLM generated content, the present application introduces a backtracking mechanism based on discriminator, which can dynamically adjust the generation path during text generation. The discriminator evaluates whether the current generated content meets the safety standard from a global perspective. In this application, the discriminator used is a content security classification model, i.e. Llama-Guard-3-8B.

[0127] Specifically, when generating a certain token, the discriminator will evaluate the global context of the current sequence. If the discriminator considers that all the generated content does not meet the alignment standard, the model will backtrack to the last time step, reselect the candidate token and continue to generate.

[0128] At time step t, the token sequence generated by the large model is represented as x t ={t0, t1, …, t t-1}, and a new token t t is generated from the candidate token set T t1 ={t t2 , t tn , …, t t}. The generation probability of each candidate token in the candidate token set is H(t ti |x t ), which represents the probability of generating the candidate token t t under the context x ti .

[0129] In a possible implementation, step 104 includes the following contents S41 to S42.

[0130] S41, introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0131] S42, when Indicates the addition of t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0132] where t ti In the example, i=1,2,…,n, and the corresponding candidate token set is T t ={t t1 , t t2 ,,t tn }.

[0133] In one possible implementation, in order to ensure the reliability of the generation process, the present invention introduces a discriminator D(·)∈{0,1} to evaluate the currently generated token sequence x t Whether the alignment standard is met. Means to join t ti The sequence passes the alignment criteria, otherwise t ti Add to the banned list, that is, B t =B t ∪t ti The present invention finally selects the candidate token with the highest generation probability to add to the generation sequence and continues to generate subsequent content, which is expressed as

[0134] Step 105 : In response to the generated text not meeting the alignment standard, a backtracking mechanism is introduced into the large language model to backtrack and adjust the text basic units of the generated text.

[0135] In a possible implementation, step 105 includes the following contents S51 to S53.

[0136] S51, if at time t, the candidate token set is evaluated as unqualified by the discriminator, triggering a backtracking mechanism;

[0137] S52, return to the previous time step t-1 through the backtracking mechanism, and set the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 where t t-1 token at time step t-1;

[0138] S53, through the backtracking mechanism, reselecting tokens not in the disable list from the candidate token set of the previous time step to generate until a token meeting the alignment standard is found.

[0139] If the discriminator evaluates all candidate tokens as unqualified at time t, the backtracking mechanism is triggered. The backtracking mechanism will go back to the previous time step t-1, and the current t t-1 Join the disable list, that is, B t-1 =B t-1 ∪t t-1 And reselect tokens not in the disable list from the candidate token set of the previous time step to generate until a token meeting the alignment standard is found, thereby effectively suppressing the generation of harmful content, expressed as

[0140] In summary, the present application designs a plug-and-play probability correction mechanism to address the problem that large language models may deviate from human preference standards during text generation. This mechanism can perform real-time fine-grained probability adjustment during the reasoning process of the model, dynamically optimize the probability distribution of each generation step, and ensure that the generated content is not only fluent and natural in language, but also closer to human values and expectations in semantics. Specifically, the present application integrates a lightweight probability correction model that can analyze and adjust the probability of each word generated by a large language model, thereby improving the quality and harmlessness of the text without sacrificing text diversity and creativity. In addition, a global evaluation and backtracking mechanism based on a discriminator is designed to address the security and ethical issues that may exist in the text generation of the prior art. This mechanism introduces a high-level discriminator model to strictly verify the safety of each token generated, ensuring that the generated text is not only linguistically correct, but also meets ethical and safety standards in content. When the discriminator detects that the generated text may contain harmful, inappropriate, or sensitive content, the backtracking mechanism is triggered, causing the model to return to the previous generation step and reselect a safe candidate token, thereby avoiding the generation of content that does not meet ethical standards. This security mechanism based on a discriminator not only improves the model's ability to identify and respond to potential risks, but also enhances the reliability and compliance of the generated text through real-time monitoring and adjustment.

[0141] An example embodiment of the present application also provides a large language model alignment device based on decoding probability correction, the device comprising:

[0142] The training generation module is configured to generate a training set by using the large language model, wherein the training set is constituted according to possible continuations of a given question;

[0143] The model generation module is configured to train a probability correction model by using the training set.

[0144] The probability adjustment module is configured to adjust a generation probability of a next text basic unit in a text generation process of the large language model by using the trained probability correction model, wherein the text basic unit is a basic unit of the generated text.

[0145] The discriminative evaluation module is configured to evaluate the generated text by using a discriminator.

[0146] The backtracking adjustment module is configured to backtrack and adjust the text basic units of the generated text by introducing a backtracking mechanism in the large language model in response to the generated text not meeting the alignment standard.

[0147] Optionally, the training generation module comprises:

[0148] The first generation unit is configured to construct harmful samples and useful samples according to historical data sets, wherein the harmful samples and the useful samples respectively contain corresponding given questions.

[0149] The second generation unit is configured to randomly generate different prefixes for each harmful sample and each useful sample, wherein the prefixes are possible continuations of the corresponding given questions.

[0150] The third generation unit is configured to connect the corresponding given questions of each harmful sample and each useful sample and the corresponding generated prefixes to generate initial texts corresponding to each harmful sample and each useful sample.

[0151] The fourth generation unit is configured to generate variant texts corresponding to the initial texts by using the large language model.

[0152] The fifth generation unit is configured to score the usefulness of each variant text corresponding to the harmful sample by using a reward model.

[0153] The sixth generation unit is configured to score the harmfulness of each variant sample corresponding to the harmful sample by using a content security classification model.

[0154] The seventh generation unit is configured to normalize and weight the usefulness score and the harmfulness score to obtain a correction probability of each given question and the corresponding prefix as the training set.

[0155] Optionally, the model generation module comprises:

[0156] a first model unit configured to express an input form of the large language model as wherein Prefix represents a generated prefix, t k for a next candidate text elementary unit, <s>to aggregate information for an entire input sequence, the large language model is configured to extract out <s>The labeled feature vector where d m represents the feature dimension.

[0157] The second model unit is configured to process the feature vector h CLS using a multi-layer perception and map the output to a probability value in the interval [0, 1] using a sigmoid function: where, is the quality probability of the model output, and σ(x) is a sigmoid function.

[0158] The third model unit is configured to train the probability correction model using a mean square error as a loss function, the mean square error being used to optimize the prediction result of the probability correction model, wherein the loss function L MSE is:

[0159]

[0160] Optionally, the probability adjustment module comprises:

[0161] The first adjustment unit is configured to adjust the generation probability of the next text basic unit using the trained probability correction model, reduce the generation probability of tokens that do not conform to human preferences, and improve the probability of tokens that conform to human preferences, wherein a token represents a text basic unit.

[0162] The second adjustment unit is configured to assume that the original probability of the large language model generating the next candidate token is P LLM (t|prefix) under the condition of a given prefix prefix, P CM (t|prefix) represents the probability of the probability correction model evaluating whether the token conforms to human preferences.

[0163] The third adjustment unit is configured to output the corrected generation probability by weighted fusion as:

[0164]

[0165] where LLM represents the large language model, CM represents the probability correction model, and alpha and beta are hyperparameters that control the relative contribution of the large language model and the probability correction model, and satisfy alpha + beta = 1.

[0166] The fourth adjustment unit is configured to introduce a normalization constant to normalize the corrected generation probability, and finally adjust the generation probability of the next text basic unit.

[0167] Optionally, at time step t, the token sequence generated by the large language model is represented as x t ={t0, t1, ..., t t-1 }, and the large language model is used to select the candidate token set T t ={t t1 , t t2 ,,t tn Generate a new token in} t , the generation probability of each candidate token in the candidate token set is expressed as H(t ti |x t ), the H(t ti |x t ) means in context x t Generate candidate tokens ti probability.

[0168] Optionally, the identification and evaluation module includes:

[0169] The first evaluation unit is used to introduce the discriminator to evaluate the currently generated token sequence x t Whether alignment standards are met;

[0170] The second evaluation unit is used when Indicates the addition of t ti The sequence passes the alignment criteria, otherwise the t ti Add to the banned list, expressed as B t =B t ∪t ti .

[0171] Optionally, the retrospective adjustment module includes:

[0172] A first adjustment unit is configured to trigger a backtracking mechanism if, at time t, the candidate token set is evaluated by the discriminator as unqualified, wherein the candidate token is used to indicate the basic unit of text;

[0173] The second adjustment unit is used to return to the previous time step t-1 through the backtracking mechanism and adjust the current t t-1 Add to the banned list, expressed as B t-1 =B t-1 ∪t t-1 , where t t-1 Represented as the token at time step t-1;

[0174] The third adjustment unit is configured to reselect a token that is not in the forbidden list from the candidate token set of the previous time step through the backtracking mechanism to generate tokens until a token that meets the alignment criteria is found.

[0175] The above-mentioned sequence numbers of embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0176] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk.

[0177] The above-mentioned is only optional embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.< / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s>

Claims

1. A large language model alignment method based on decoding probability correction, characterized in that: The method comprises: Generate a training set, wherein the training set is generated according to possible continuation compositions of a given problem; A probability correction model is obtained by training a large language model according to the training set; During the text generation process of the large language model, the trained probability correction model is used to adjust the generation probability of the next text basic unit, where the text basic unit is the basic unit for generating text; The generated text is evaluated by a discriminator; the generated text is evaluated by a discriminator, including: introducing the discriminator to evaluate the currently generated token sequence x t Whether the alignment standard is met; when Instructions to join The sequence passes the alignment criteria, otherwise the Add to the banned list, expressed as ; In response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic unit of the generated text by introducing a backtracking mechanism in the large language model; in response to the generated text not meeting the alignment standard, backtracking and adjusting the text basic unit of the generated text by introducing a backtracking mechanism in the large language model, including: if at time t, the candidate token set is evaluated as unqualified by the discriminator, triggering the backtracking mechanism, wherein the candidate token is used to indicate the text basic unit; returning to the previous time step t-1 by the backtracking mechanism, and changing the current t t-1 Add to the banned list, expressed as , where t t-1 Represented as the token at time step t−1; through the backtracking mechanism, tokens that are not in the banned list are reselected from the candidate token set of the previous time step to generate tokens until a token that meets the alignment criteria is found; Among them, at time step t, the token sequence generated by the large language model is expressed as , and the large language model is used to select from the candidate token set Generate a new token , the generation probability of each candidate token in the candidate token set is expressed as , In the context x t Generate candidate tokens probability.

2. The method according to claim 1, characterized in that The generating of the training set, wherein the training set is generated according to possible continuation compositions of a given problem, includes: Constructing harmful samples and useful samples based on historical data sets, wherein the harmful samples and the useful samples respectively contain corresponding given questions; Randomly generate different prefixes for each harmful sample and useful sample, and the prefixes are used as possible continuations corresponding to a given question; Connecting the given questions and the corresponding generated prefixes corresponding to each harmful sample and each useful sample to generate the initial text corresponding to each harmful sample and each useful sample; Generate a variant text corresponding to the initial text using the large language model; The reward model is used to score the usefulness of each variant text corresponding to the harmful sample; The harmfulness of each variant sample corresponding to the harmful sample is scored using a content security classification model; The usefulness score and the harmfulness score are normalized and weighted to obtain the correction probability of each given question and the corresponding prefix as the training set.

3. The method according to claim 2, characterized in that The probabilistic correction model obtained by training the training set includes: According to the training set, the input form of the large language model is expressed as , where Prefix represents the generated prefix, is the next candidate text basic unit, <s>It is used to aggregate the information of the entire input sequence. The large language model is used to extract <s>Labeled feature vector ,in Represents feature dimension;< / s> < / s> <s> <s> The feature vector is processed by a multi-layer perceptron The vector is processed and the sigmoid function is used to map the output to a probability value in the [0,1] interval: ,in, is the quality probability of the model output, is the sigmoid function; The probability correction model is trained using mean square error as the loss function, and the mean square error is used to optimize the prediction results of the probability correction model, wherein the loss function L MSE for: 。 4. The method according to claim 3, characterized in that In the text generation process of the large language model, the trained probability correction model is used to adjust the generation probability of the next text basic unit, including: Use the trained probability correction model to adjust the generation probability of the next basic unit of text, reducing the generation probability of tokens that do not meet human preferences and increasing the probability of tokens that meet human preferences, where tokens are represented as basic units of text; Given a prefix, assuming that the original probability of the large language model generating the next candidate token is , Indicates the probability of the probability correction model evaluating whether the token conforms to human preferences; The generation probability after correction by weighted fusion output is: ， Wherein, LLM refers to the large language model, CM refers to the probability correction model, is a hyperparameter that controls the relative contributions of the large language model and the probability correction model, and satisfies ; Introducing a normalization constant The corrected generation probability is normalized and finally the generation probability of the next basic unit of text is adjusted.

5. A large language model alignment device based on decoding probability correction, characterized in that: The device comprises: A training generation module, configured to generate a training set using a large language model, wherein the training set is generated based on possible continuation compositions of a given question; A model generation module, configured to obtain a probability correction model through training of the training set; A probability adjustment module is used to adjust the generation probability of the next text basic unit using the trained probability correction model during the text generation process of the large language model, where the text basic unit is the basic unit for generating text; The identification and evaluation module is used to evaluate the generated text through the discriminator; the evaluation of the generated text through the discriminator includes: introducing the discriminator to evaluate the currently generated token sequence x t Whether the alignment standard is met; when Instructions to join The sequence passes the alignment criteria, otherwise the Add to the banned list, expressed as ; A backtracking adjustment module is configured to, in response to the generated text not meeting the alignment criteria, backtrack and adjust the text basic units of the generated text by introducing a backtracking mechanism in the large language model; the backtracking adjustment module is further configured to: if at time t, the candidate token set is evaluated as unqualified by the discriminator, trigger the backtracking mechanism, wherein the candidate token is used to indicate the text basic unit; return to the previous time step t−1 through the backtracking mechanism, and set the current t t-1 Add to the banned list, expressed as , where t t-1 Represented as the token at time step t−1; through the backtracking mechanism, tokens that are not in the banned list are reselected from the candidate token set of the previous time step to generate tokens until a token that meets the alignment criteria is found; Among them, at time step t, the token sequence generated by the large language model is expressed as , and the large language model is used to select from the candidate token set Generate a new token , the generation probability of each candidate token in the candidate token set is expressed as , In the context x t Generate candidate tokens probability.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the large language model alignment method based on decoding probability correction as described in any one of claims 1 to 4 above. < / s> < / s>

Images