Large language model structured preference alignment method and device, electronic equipment and medium

Abstract

Embodiments of the present disclosure disclose a large language model structured preference alignment method, device, electronic equipment and medium. A specific implementation of the method comprises: obtaining a set of triple samples; generating a set of prompt word winning reply sequences and a set of prompt word losing reply sequences; generating a set of winning reply word element probability sequences, a set of losing reply word element probability sequences, a set of reference winning reply word element probability sequences, and a set of reference losing reply word element probability sequences; constructing a semantic loss function corresponding to the large language model to be aligned; constructing a syntax loss function corresponding to the large language model to be aligned; generating gradient correlation information according to the semantic loss function and the syntax loss function; determining an updated gradient vector according to the gradient correlation information; and updating the parameters of the large language model to be aligned using the updated gradient vector. This implementation achieves a Pareto improvement of optimizing the semantic logic capability of the large language model and maintaining the syntax format.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the fields of artificial intelligence and natural language processing technology, specifically to methods, apparatus, electronic devices, and media for aligning structured preferences of large language models. Background Technology

[0002] With the advancement of artificial intelligence technology, especially the development of natural language processing technology, the application of LLM (Large Language Models) in various complex tasks has become increasingly widespread. Among these applications, structured preference alignment (i.e., ensuring that the output of the large language model aligns with human values ​​and intentions), which represents the semantic understanding and logical reasoning capabilities of the large language model, is crucial. Currently, the common approach to performing structured preference alignment on large language models is through Reinforcement Learning from Human Feedback (RLHF) and its variant, Direct Preference Optimization (DPO).

[0003] However, when using the above method, the following technical problems often arise:

[0004] 1) Reinforcement learning based on human feedback requires training and maintaining two models (a reward model and a policy model), leading to unstable model training and easily skewed training results (i.e., it relies on the reward model for fine-tuning, and when the reward model has errors, the policy model has larger errors). Furthermore, maintaining two models simultaneously results in wasted computational resources. 2) Traditional DPO algorithms typically focus on semantics when fine-tuning the model. When negative samples generated by the model are penalized due to semantic logical errors, the optimization algorithm often cannot distinguish the source of the error (semantic error or syntactic error), causing the parameters carrying the correct syntactic structure to be suppressed simultaneously, resulting in structural degradation (improving semantic ability but reducing syntactic ability). Furthermore, some approaches address structural degradation by setting an external parser to forcibly mask grammatically incorrect lexical units. While the output conforms to grammatical rules, this disrupts the hard constraints of the model's original probability distribution, potentially leading to model illusion (the model may be forced to generate seemingly grammatically correct but factually incorrect content to meet hard grammatical rules) and inference latency (each lexical unit generation requires calling the external parser for real-time checking and intervention, introducing additional computational steps and I / O wait time). It also reduces the likelihood of low model creativity and adaptability due to the inability to adapt to dynamically generated unknown grammatical structures (the external parser is based on a fixed, predefined rule base and lacks generalization).

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide a method, apparatus, electronic device, and computer-readable medium for aligning structured preferences of large language models to address the technical problems mentioned in the background section above.

[0008] Firstly, some embodiments of this disclosure provide a structured preference alignment method for a large language model. The method includes: obtaining a triplet sample set, wherein the triplet samples in the triplet sample set include: a prompt word sequence, a winning response sequence, and a losing response sequence; generating a prompt word winning response sequence set and a prompt word losing response sequence set based on the triplet sample set, wherein the prompt word winning response sequence in the prompt word winning response sequence set and the prompt word losing response sequence in the prompt word losing response sequence set correspond one-to-one; generating a winning response lexical probability sequence set, a losing response lexical probability sequence set, a reference winning response lexical probability sequence set, and a reference losing response lexical probability sequence set based on the prompt word winning response sequence set, the prompt word losing response sequence set, the large language model to be aligned, and a reference large language model, wherein the reference large language model has the same model architecture as the large language model to be aligned, and the reference large language model... The model parameters of the language model are kept frozen. Based on the aforementioned set of winning response lexical probability sequences, the set of losing response lexical probability sequences, the set of reference winning response lexical probability sequences, and the set of reference losing response lexical probability sequences, a semantic loss function corresponding to the large language model to be aligned with preferences is constructed. Based on the aforementioned set of winning response lexical probability sequences and the winning response sequences included in the aforementioned triplet sample set, a syntactic loss function corresponding to the large language model to be aligned with preferences is constructed. Based on the aforementioned semantic loss function and the aforementioned syntactic loss function, gradient relevance information corresponding to the large language model to be aligned with preferences is generated, wherein the gradient relevance information represents the correlation between the semantic optimization direction and the syntactic maintenance direction. Based on the aforementioned gradient relevance information, an update gradient vector corresponding to the large language model to be aligned with preferences is determined. Using the aforementioned update gradient vector, the parameters of the large language model to be aligned with preferences are updated to achieve structured preference alignment of the large language model to be aligned with preferences.

[0009] Secondly, some embodiments of this disclosure provide a structured preference alignment apparatus for a large language model. The apparatus includes: an acquisition unit configured to acquire a set of triplet samples, wherein the triplet samples in the triplet sample set include: a prompt word sequence, a winning response sequence, and a losing response sequence; a first generation unit configured to generate a set of prompt word winning response sequences and a set of prompt word losing response sequences based on the triplet sample set, wherein the prompt word winning response sequences in the set of prompt word winning response sequences and the prompt word losing response sequences in the set of prompt word losing response sequences correspond one-to-one; and a second generation unit configured to generate a set of winning response lexical probability sequences, a set of losing response lexical probability sequences, a set of reference winning response lexical probability sequences, and a set of reference losing response lexical probability sequences based on the set of prompt word winning response sequences, the set of prompt word losing response sequences, the large language model to be aligned, and a reference large language model, wherein the reference large language model has the same model architecture as the large language model to be aligned, and the model parameters of the reference large language model are maintained. The system is divided into three phases: a frozen state; a first construction unit configured to construct a semantic loss function corresponding to the large language model to be aligned with preferences based on the set of winning response word probability sequences, the set of losing response word probability sequences, the set of reference winning response word probability sequences, and the set of reference losing response word probability sequences; a second construction unit configured to construct a syntactic loss function corresponding to the large language model to be aligned with preferences based on the set of winning response word probability sequences and the winning response sequences included in the set of triplet samples; a third generation unit configured to generate gradient relevance information corresponding to the large language model to be aligned with preferences based on the semantic loss function and the syntactic loss function, wherein the gradient relevance information represents the correlation between the semantic optimization direction and the syntactic maintenance direction; a determination unit configured to determine the update gradient vector corresponding to the large language model to be aligned with preferences based on the gradient relevance information; and a parameter update unit configured to update the parameters of the large language model to be aligned with preferences using the update gradient vector to achieve structured preference alignment of the large language model to be aligned with preferences.

[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0012] The above embodiments of this disclosure have the following beneficial effects: Through the structured preference alignment method for large language models in some embodiments of this disclosure, Pareto improvement can be achieved in optimizing the semantic logic capability and maintaining the grammatical format of large language models, while reducing the occurrence of model illusion problems, inference delay problems, and poor robustness problems. Specifically, the reasons for the prevalence of model structure degradation problems, model illusion problems, inference delay problems, and poor robustness problems are as follows: 1) Reinforcement learning based on human feedback requires training and maintaining two models (reward model and policy model), which leads to unstable model training and the training results are prone to deviation (i.e., it needs to rely on the reward model for model fine-tuning, and when the reward model has errors, the policy model has larger errors). Furthermore, because two models need to be maintained at the same time, it leads to a waste of computational resources. 2) When traditional DPO algorithms fine-tune the model, they usually focus on semantics. When the negative samples generated by the model are penalized due to semantic logical errors, the optimization algorithm often cannot distinguish the source of the error (semantic error or grammatical error), which leads to the suppression of the parameter direction carrying the correct grammatical structure at the same time, that is, the structure degradation problem (improving semantic capability, reducing grammatical capability). Furthermore, some methods address structural degradation by setting an external parser to forcibly mask grammatically incorrect lexical units. While the output conforms to grammatical rules, this disrupts the hard constraints of the model's original probability distribution, potentially leading to model illusion (the model may be forced to generate seemingly grammatically correct but factually incorrect content to meet rigid grammatical rules) and inference latency (each lexical unit generation requires real-time checking and intervention from the external parser, introducing additional computational steps and I / O latency). It also reduces the likelihood of poor model robustness due to difficulty adapting to dynamically generated unknown grammatical structures (the external parser, based on a fixed, predefined rule base, lacks generalization ability). Based on this, some embodiments of the large language model structured preference alignment method disclosed herein first obtain a triplet sample set. This triplet sample set includes: a prompt word sequence, a winning response sequence, and a losing response sequence. Based on this triplet sample set, a prompt word winning response sequence set and a prompt word losing response sequence set are generated. Specifically, there is a one-to-one correspondence between the winning response sequence in the set of winning response sequences and the losing response sequence in the set of losing response sequences. This allows us to obtain a set of winning response sequences representing superior response content preferences and a set of losing response sequences representing inferior response content preferences. Then, based on the aforementioned set of winning response sequences, the aforementioned set of losing response sequences, the large language model for preference alignment, and the reference large language model, we generate a set of winning response word probability sequences, a set of losing response word probability sequences, a set of reference winning response word probability sequences, and a set of reference losing response word probability sequences.The reference large language model has the same model architecture as the aforementioned large language model for preference alignment, and its model parameters are frozen. Therefore, we can obtain the set of winning response lexical probability sequences generated by the large language model for preference alignment corresponding to the winning response sequence set, the set of reference winning response lexical probability sequences generated by the reference large language model, and the set of losing response lexical probability sequences generated by the large language model for preference alignment corresponding to the losing response sequence set, as well as the set of reference losing response lexical probability sequences generated by the reference large language model. Then, based on the aforementioned sets of winning response lexical probability sequences, losing response lexical probability sequences, reference winning response lexical probability sequences, and reference losing response lexical probability sequences, we construct the semantic loss function corresponding to the aforementioned large language model for preference alignment. This semantic loss function allows us to measure the difference between the prediction results and the actual results of the large language model and guide it towards results that better align with preferences. Based on the aforementioned set of winning response lexical probability sequences and the winning response sequences included in the aforementioned triplet sample set, we construct the syntactic loss function corresponding to the aforementioned large language model for preference alignment. Therefore, a grammatical loss function representing grammatical structure constraints can be obtained, allowing for more accurate identification and punishment of grammatical biases, thus guiding model optimization. Secondly, based on the aforementioned semantic and grammatical loss functions, gradient relevance information corresponding to the large language model to be aligned with preferences is generated. This gradient relevance information represents the correlation between the semantic optimization direction and the grammatical maintenance direction. This gradient relevance information allows for the determination of whether gradient interference (grammatical loss) exists during the optimization process. Then, based on the gradient relevance information, the update gradient vector corresponding to the large language model to be aligned with preferences is determined. This yields the update gradient vector in the absence of grammatical loss. Finally, using the updated gradient vector, the parameters of the large language model to be aligned with preferences are updated to achieve structured preference alignment. This results in a large language model with structured preference alignment. Furthermore, because gradient relevance information can be obtained by constructing semantic and grammatical loss functions, it is possible to determine whether grammatical loss exists during semantic optimization. When grammatical loss exists, gradient interference elimination operations (i.e., orthogonal projection operations) can be performed to maintain the grammatical structure. This approach achieves Pareto improvements in optimizing the semantic logic capabilities and maintaining the grammatical format of large language models. Furthermore, it eliminates the need for an external parser, reducing issues such as model illusion, inference latency, and poor robustness. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the large language model structured preference alignment method disclosed herein;

[0015] Figure 2 This is a schematic diagram of the structure of some embodiments of the large language model structured preference alignment device according to the present disclosure;

[0016] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0020] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0021] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0022] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1A flow 100 is shown, illustrating some embodiments of a large language model structured preference alignment method according to this disclosure. This large language model structured preference alignment method includes the following steps:

[0024] Step 101: Obtain the triplet sample set.

[0025] In some embodiments, the execution entity (e.g., a computing device) of the large language model structured preference alignment method can obtain a set of triplet samples from a triplet sample information database via a wireless or wired connection. The triplet sample information database can be any information database storing the triplet sample set. The specific limitations of the triplet sample information database are not specified here. For example, the triplet sample information database can be a local database or any server capable of storing the triplet sample set. The triplet samples in the triplet sample set can include: a prompt word sequence, a winning response sequence, and a losing response sequence. The prompt word sequence can be a sequence of tokens that can ask questions or execute instructions for the large language model. The winning response sequence can be a sequence of tokens corresponding to a response judged as better by an annotator (human or AI) for a given prompt word sequence. The losing response sequence can be a sequence of tokens corresponding to a response judged as worse by an annotator that appears in conjunction with the winning response sequence for the same prompt word sequence. For example, the above suggestion sequence could be "Query all employee names greater than 25 years old.". The corresponding tokens for this suggestion sequence could be ["query", "all", "age", "greater than", "25", "years old", "of", "employees", "name", "."]. The winning response sequence could be "SELECT name FROM employees WHERE age > 25;". The corresponding tokens for this winning response sequence could be ["SELECT", "name", "FROM", "employees", "WHERE", "age",">", "25", ";"]. The losing response sequence could be "SELECT name FROM employees WHERE age25;". The corresponding tokens for this losing response sequence could be ["SELECT", "name", "FROM", "employees", "WHERE", "age", "25", ";"]. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other currently known or future wireless connection methods.

[0026] Step 102: Based on the triplet sample set, generate a set of winning response sequences and a set of losing response sequences for the prompt words.

[0027] In some embodiments, the executing entity can generate a set of winning prompt word response sequences and a set of losing prompt word response sequences based on the triplet sample set. The winning prompt word response sequences in the winning prompt word response sequence set and the losing prompt word response sequences in the losing prompt word response sequence set correspond one-to-one.

[0028] In some optional implementations of certain embodiments, based on the aforementioned triplet sample set, the executing entity can generate a set of winning prompt word response sequences and a set of losing prompt word response sequences through the following steps:

[0029] The first step is to perform the following processing steps for each triplet sample in the above triplet sample set:

[0030] The first step is to combine the prompt word sequence and winning response sequence included in the above triplet sample to generate a prompt word winning response sequence. Here, the combination method can be concatenation. For example, the above prompt word winning response sequence could be "Query all employee names older than 25.\nSELECT name FROM employees WHEREage > 25;".

[0031] The second step involves combining the prompt word sequence and the failed response sequence included in the above triplet sample to generate a prompt word failed response sequence. This combination can be done by concatenation. For example, the prompt word failed response sequence could be: "Query all employee names older than 25.\nSELECT name FROM employeesWHERE age 25;".

[0032] The second step is to determine the winning response sequences of each generated prompt word as the set of winning response sequences of prompt words.

[0033] The third step is to determine the generated sequence of failed responses for each prompt word as a set of failed response sequences for prompt words.

[0034] Step 103: Based on the set of winning response sequences with prompt words, the set of losing response sequences with prompt words, the large language model to be aligned with preferences, and the reference large language model, generate the set of winning response word probability sequences, the set of losing response word probability sequences, the set of reference winning response word probability sequences, and the set of reference losing response word probability sequences.

[0035] In some embodiments, the execution entity can generate a set of winning response lexical probability sequences, a set of losing response lexical probability sequences, a set of reference winning response lexical probability sequences, and a set of reference losing response lexical probability sequences based on the set of winning response sequences, the set of losing response sequences, the large language model to be aligned with preferences, and a reference large language model. The reference large language model has the same model architecture as the large language model to be aligned with preferences. Specifically, the model of the reference large language model can be an initial copy of the large language model to be aligned with preferences. The model parameters of the reference large language model are kept frozen. The large language model to be aligned with preferences can be a large language model that takes the winning response sequence (or the losing response sequence) as input and the winning response lexical probability sequence (or the losing response lexical probability sequence) as output. The large language model to be aligned with preferences can be a large language model to be subjected to structured preference alignment. The aforementioned reference large language model can be a large language model that takes the winning response sequence (or losing response sequence) of the prompt word as input and outputs the probability sequence of the winning response lexicon (or the probability sequence of the losing response lexicon). This reference large language model can be used to provide a probability baseline, ensuring that the update of the large language model to be aligned during the alignment process does not deviate excessively from the original language distribution and basic generative capabilities. For example, the aforementioned large language model to be aligned is not specifically limited here. For example, the aforementioned large language model to be aligned can be GPT (Generative Pre-trained Transformer) or LLaMA (Large Language Model Meta AI).

[0036] In some optional implementations of certain embodiments, based on the aforementioned set of winning response sequences, the aforementioned set of losing response sequences, the large language model to be aligned with preferences, and the reference large language model, the executing entity can generate the set of winning response lexical probability sequences, the set of losing response lexical probability sequences, the reference set of winning response lexical probability sequences, and the reference set of losing response lexical probability sequences through the following steps:

[0037] The first step involves inputting each winning response sequence from the aforementioned set of winning response sequences into the aforementioned large language model for preference alignment, thereby obtaining a set of winning response lexical probability sequences. Each winning response lexical probability sequence in this set corresponds one-to-one with the winning response sequences in the aforementioned set of winning response sequences. The winning response lexical probability sequences in this set represent the probability of generating each winning response lexical in the winning response sequence. In practice, firstly, for each winning response sequence in the aforementioned set of winning response sequences, the executing entity can input the winning response sequence into the aforementioned large language model for preference alignment to obtain a winning response lexical probability sequence. Then, the obtained winning response lexical probability sequences can be defined as the set of winning response lexical probability sequences.

[0038] The second step involves inputting each of the failed response sequences from the aforementioned set of failed response sequences into the aforementioned large language model for preference alignment, thereby obtaining a set of failed response token probability sequences. Each failed response token probability sequence in this set corresponds one-to-one with the failed response sequences in the aforementioned set of failed response sequences. The failed response token probability sequences in this set represent the probability of generating each failed response token in the failed response sequence. In practice, firstly, for each failed response sequence in the aforementioned set of failed response sequences, the executing entity can input the failed response sequence into the aforementioned large language model for preference alignment to obtain a failed response token probability sequence. Then, the obtained failed response token probability sequences can be defined as the set of failed response token probability sequences.

[0039] The third step involves inputting each winning response sequence from the aforementioned set of winning response sequences into the aforementioned reference large language model to obtain a set of reference winning response word probability sequences. Each reference winning response word probability sequence in this set corresponds one-to-one with a winning response sequence from the aforementioned set of winning response sequences. These reference winning response word probability sequences represent the reference probability of each winning response word in the generated winning response sequence. In practice, firstly, for each winning response sequence in the aforementioned set of winning response sequences, the executing entity can input the winning response sequence into the aforementioned reference large language model to obtain a reference winning response word probability sequence. Then, the obtained reference winning response word probability sequences can be defined as the set of reference winning response word probability sequences.

[0040] The fourth step involves inputting each of the failed prompt word response sequences in the aforementioned failed prompt word response sequence set into the aforementioned reference large language model to obtain a reference failed prompt word probability sequence set. Each reference failed prompt word probability sequence in this set corresponds one-to-one with the failed prompt word response sequences in the aforementioned failed prompt word response sequence set. The reference failed prompt word probability sequences in this set represent the reference probability of each failed prompt word in the generated failed prompt response sequence. In practice, firstly, for each failed prompt word response sequence in the aforementioned failed prompt word response sequence set, the executing entity can input the failed prompt word response sequence into the aforementioned reference large language model to obtain a reference failed prompt word probability sequence. Then, the obtained reference failed prompt word probability sequences can be defined as the reference failed prompt word probability sequence set.

[0041] Step 104: Based on the set of probability sequences of winning response terms, the set of probability sequences of losing response terms, the set of probability sequences of reference winning response terms, and the set of probability sequences of reference losing response terms, construct the semantic loss function corresponding to the large language model to be aligned with preferences.

[0042] In some embodiments, the execution entity may construct the semantic loss function corresponding to the large language model to be aligned with preferences based on the set of winning response token probabilities, the set of losing response token probabilities, the set of reference winning response token probabilities, and the set of reference losing response token probabilities.

[0043] In some optional implementations of certain embodiments, based on the aforementioned set of winning response token probability sequences, set of losing response token probability sequences, set of reference winning response token probability sequences, and set of reference losing response token probability sequences, the aforementioned execution entity can construct the semantic loss function corresponding to the aforementioned large language model to be aligned with preferences through the following steps:

[0044] The first step is to generate a set of winning response probabilities based on the aforementioned set of winning response term probability sequences. In practice, for each winning response term probability sequence in the aforementioned set of winning response term probability sequences, the executing entity can determine the winning response probability by multiplying the probabilities of each winning response term included in the winning response term probability sequence. Then, the obtained winning response probabilities can be used to determine the set of winning response probabilities.

[0045] The second step is to generate a set of failed response probabilities based on the aforementioned set of failed response token probability sequences. In practice, for each failed response token probability sequence in the aforementioned set of failed response token probability sequences, the executing entity can determine the failed response probability as the product of the probabilities of each failed response token included in the aforementioned failed response token probability sequence. Then, the obtained set of failed response probabilities can be determined.

[0046] The third step is to generate a set of reference winning response probabilities based on the aforementioned set of reference winning response term probability sequences. Each reference winning response probability in this set corresponds one-to-one with the winning response probabilities in the aforementioned set of winning response probabilities. In practice, for each reference winning response term probability sequence in the aforementioned set of reference winning response term probability sequences, the executing entity can determine the reference winning response probability as the product of the probabilities of each reference winning response term included in that sequence. Then, the resulting set of reference winning response probabilities can be defined as a reference winning response probability set.

[0047] The fourth step involves generating a set of reference failed response probabilities based on the aforementioned set of reference failed response token probability sequences. Each reference failed response probability in this set corresponds one-to-one with the failed response probabilities in the overall failed response probability set. In practice, for each reference failed response token probability sequence in the aforementioned set, the executing entity can determine the reference failed response probability as the product of the probabilities of each reference failed response token included in that sequence. Then, the resulting set of reference failed response probabilities can be defined as a reference failed response probability set.

[0048] Step 5: Based on the aforementioned set of winning response probabilities and the aforementioned set of reference winning response probabilities, generate a set of winning logarithmic ratios. In practice, based on the aforementioned set of winning response probabilities and the aforementioned set of reference winning response probabilities, the executing entity can generate the set of winning logarithmic ratios through the following steps:

[0049] The first step involves performing the following processing steps for each winning response probability in the aforementioned set of winning response probabilities:

[0050] The first sub-step is to determine the reference winning response probability corresponding to the above winning response probability in the above set of reference winning response probabilities as the target reference winning response probability.

[0051] The second sub-step is to determine the ratio of the above-mentioned winning response probability to the above-mentioned target reference winning response probability as the winning response ratio information.

[0052] The third sub-step involves determining the logarithm of the aforementioned winning response ratio information as the winning logarithm ratio.

[0053] The second step is to determine the obtained winning log ratios as a set of winning log ratios.

[0054] Step 6: Based on the aforementioned set of defeat response probabilities and the aforementioned set of reference defeat response probabilities, generate a set of defeat logarithmic ratios. The defeat logarithmic ratios in this set correspond one-to-one with the winning logarithmic ratios in the aforementioned set of winning logarithmic ratios. In practice, the executing entity can generate the set of defeat logarithmic ratios using the following steps:

[0055] The first step is to perform the following processing steps for each failure response probability in the above set of failure response probabilities:

[0056] The first sub-step is to determine the reference failure response probability corresponding to the above failure response probability in the above set of reference failure response probabilities as the target reference failure response probability.

[0057] The second sub-step is to determine the ratio of the aforementioned failure response probability to the aforementioned target reference failure response probability as the failure response ratio information.

[0058] The third sub-step is to determine the logarithm of the aforementioned loser response ratio information as the loser logarithm ratio.

[0059] The second step is to determine the obtained logarithm ratios as a set of logarithm ratios.

[0060] Step 7: For each winning log ratio in the above set of winning log ratios, perform the following processing steps:

[0061] The first step is to determine the target log-loss ratio as the log-loss ratio corresponding to the log-loss ratio of the winning ratio in the above set of log-loss ratios.

[0062] The second step is to determine the difference between the above-mentioned winning log ratio and the above-mentioned target losing log ratio as the winning-losing difference information;

[0063] The third step involves multiplying the win / loss difference information by a preset preference parameter to determine the win / loss preference information. This preset preference parameter can be a pre-defined preference parameter. It is used to scale the magnitude of the preference difference. A larger preset preference parameter value amplifies the distinction between win and loss, resulting in a larger update magnitude; a smaller preset preference parameter value results in a smaller update magnitude.

[0064] The fourth step involves using a preset activation function to map the win / loss preference information, resulting in mapped preference information. Here, the preset activation function can be the Sigmoid activation function. This function maps the win / loss preference information to the (0, 1) interval. That is, the mapped preference information lies within the interval (0, 1).

[0065] The fifth step is to determine the logarithm of the above mapping preference information as the logarithmic preference information.

[0066] Step 8: The negative of the mean of each determined logarithmic preference information is used as the loss function value of the semantic loss function corresponding to the large language model to be aligned with preferences. Here, the semantic loss function can be generated by the following formula:

[0067] .

[0068] Among them, the above This can represent a semantic loss function. (The above...) It can represent a sequence of cue words. (The above...) This can represent the winning response sequence. (The above...) This can represent a losing response sequence. (The above...) This can represent a set of triplet samples. (The above...) It can express expectation. (The above) This can represent the average value of the semantic loss function values ​​corresponding to the triplet sample set, i.e., the expected value. The above... This can represent the minimum expected loss value. That is, adding a negative sign indicates that the maximization problem is transformed into a minimization problem. The above... This can represent the large language model to be aligned with preferences in a given sequence of prompt words. The probability of generating a winning response sequence under the given premise. The above. This can represent a reference large language model in a given sequence of prompt words. The probability of generating a reference winning response sequence under the premise of [the above]. This can represent a large language model to be aligned with preferences. (The above...) This can represent a reference to a large language model. (The above...) This can represent the large language model to be aligned with preferences in a given sequence of prompt words. The probability of generating a losing response sequence under the given condition. The above. This can represent a reference large language model in a given sequence of prompt words. The probability of generating a reference failed response sequence under the premise of [the above]. This ratio can characterize the degree of preference of the large language model to be aligned with over the reference large language model in generating winning response sequences. If the ratio is greater than 1, it indicates that the large language model to be aligned with is more inclined to generate winning response sequences than the reference large language model. If it is less than 1, the opposite is true. This ratio can characterize the degree of preference of the large language model to be aligned with over the reference large language model in generating defeated response sequences. If the ratio is greater than 1, it indicates that the large language model to be aligned with is more inclined to generate defeated response sequences than the reference large language model. If it is less than 1, the opposite is true. and Taking the logarithm can be used to transform the ratio into a more manageable numerical scale, intuitively representing the strength of the preference. The above... Preset preference parameters can be provided to control the intensity of preference learning. (The above...) This can be expressed as the difference between the preference strength of the target large language model relative to the reference large language model for the winning response sequence and the preference strength of the target large language model relative to the reference large language model for the losing response sequence. The larger the difference, the stronger the model's preference for the winning response sequence compared to the losing response sequence. This can represent the Sigmoid activation function, used to map the preference difference from the previous step to probability values ​​in the interval (0, 1). The above... It can represent the maximum likelihood estimation.

[0069] Step 105: Based on the set of winning response word probability sequences and the set of triplet samples, construct the syntax loss function corresponding to the large language model to be aligned with preferences.

[0070] In some embodiments, the execution entity may construct the syntax loss function corresponding to the large language model to be aligned with preferences based on the set of winning response lexical probability sequences and the set of triplet samples.

[0071] In some optional implementations of certain embodiments, based on the set of winning response lexical probability sequences and the winning response sequences included in the set of triplet samples, the execution entity can construct the syntax loss function corresponding to the large language model to be aligned with preferences through the following steps:

[0072] The first step is to logarithmize the winning response term probability sequences included in the aforementioned winning response term probability sequence set to generate a set of logarithmic winning response term probability sequences. In practice, firstly, for each winning response term probability sequence in the aforementioned set of winning response term probability sequences, the executing entity can determine the logarithm of each winning response term probability included in the aforementioned winning response term probability sequence as the winning response term logarithmic probability sequence. Then, the resulting set of winning response term logarithmic probability sequences can be determined.

[0073] The second step involves performing binary masking on each winning response sequence included in the aforementioned triplet sample set to generate a masked winning response sequence set. The masked winning response sequences in this set correspond one-to-one with the logarithmic probability sequences of winning response terms in the aforementioned set of winning response term logarithmic probability sequences. Here, the binary masking process can be performed by predefining a list of grammatical terms and matching each winning response term in the winning response sequence against each grammatical term in that list. If a match is successful (i.e., the winning response term exists in the grammatical term list), the winning response term at that position is marked as 1. If a match is unsuccessful (i.e., the winning response term does not exist in the grammatical term list), the winning response term at that position is marked as 0. For example, a winning response sequence could be ["SELECT", "name", "FROM", "employees", "WHERE", "age", ">", "25",";"]. The masked winning response sequence corresponding to the above winning response sequence can be [1, 0, 1, 0, 1, 0, 1, 0, 1]. Here, the winning response tokens "SELECT", "FROM", and "WHERE" are SQL keywords, and the winning response token ">" is an operator. The winning response token ";" is a structure terminator. All the winning response tokens are syntactic tokens from a predefined list of syntactic tokens, and the winning response tokens at their corresponding positions are marked with 1. Thus, by using 1 to mark winning response tokens representing syntactic content and 0 to mark winning response tokens representing semantic content, the distinction between syntactic and semantic content is achieved.

[0074] Third, for each winning response word log probability sequence in the above set of winning response word log probability sequences, perform the following processing steps:

[0075] The first step is to determine the winning mask response sequence in the aforementioned winning mask response sequence set that corresponds to the logarithmic probability sequence of the winning response tokens as the target mask information sequence. Specifically, the target mask information in the target mask information sequence corresponds one-to-one with the logarithmic probabilities of the winning response tokens in the aforementioned logarithmic probability sequence.

[0076] The second step is to determine the mask probability information by multiplying the target mask information with the corresponding log probability of the winning response word in the above target mask information sequence.

[0077] The third step is to sum the obtained mask probability information to determine the total mask probability information.

[0078] The fourth step is to determine the negative of the mean of the total probability information of each mask as the loss function value of the syntax loss function corresponding to the large language model to be aligned with preferences. Here, the syntax loss function can be expressed by the following formula:

[0079] .

[0080] Among them, the above This can represent a syntax loss function. (The above...) This can represent a set of triplet samples. (The above...) This can represent the average loss of the syntax loss function values ​​for each triplet sample in the triplet sample set. (The above...) It can represent a sequence of cue words. (The above...) This can represent the winning response sequence. The winning response term. (The above) This can represent the winning response sequence. The winning response sequence is a prefix sequence composed of all winning response words preceding the winning response word. The above... This can represent the winning response sequence. The masked response tokens, after being masked, can be either 0 or 1. The above... This can represent the sequence number corresponding to each winning response term in the winning response sequence. The above... This can be represented in a given sequence of prompt words. The winning response sequence of the prefix Generate the first under the premise The probability of a winning response term. The above. It can represent the first The masked response word and the first The product of the probabilities corresponding to the winning response tokens. It can represent the number of winning response tokens in the winning response sequence.

[0081] Step 106: Generate gradient correlation information corresponding to the large language model to be aligned with preferences based on the semantic loss function and the syntactic loss function.

[0082] In some embodiments, the execution entity can generate gradient relevance information corresponding to the large language model to be aligned with preferences, based on the semantic loss function and the syntactic loss function. The gradient relevance information represents the correlation between the semantic optimization direction and the syntactic maintenance direction. Specifically, the gradient relevance information represents whether the semantic optimization direction and the syntactic maintenance direction are the same or opposite. That is, when the gradient relevance information is greater than or equal to 0, it indicates that the semantic optimization direction and the syntactic maintenance direction are the same. When the gradient relevance information is less than 0, it indicates that the semantic optimization direction and the syntactic maintenance direction are opposite.

[0083] In some optional implementations of certain embodiments, based on the semantic loss function and the syntactic loss function described above, the execution entity can generate the gradient relevance information corresponding to the large language model to be aligned with preferences through the following steps:

[0084] The first step is to generate a semantic gradient vector based on the semantic loss function described above. This semantic gradient vector represents the parameter update direction that increases the probability of a winning response and decreases the probability of a losing response. In practice, the executing entity can utilize the chain rule and backpropagation algorithm to differentiate the semantic function and generate the semantic gradient vector. This semantic gradient vector can be generated using the following formula:

[0085] .

[0086] Among them, the above This can represent a semantic gradient vector. (The above...) It can represent the model parameters Finding the gradient is equivalent to finding the derivative.

[0087] The second step is to generate a syntactic gradient vector based on the aforementioned syntactic loss function. This syntactic gradient vector represents the parameter update direction for the probability of generating lexical units in the enhanced syntax. In practice, the execution entity can utilize the chain rule and backpropagation algorithm to differentiate the syntactic loss function to generate the syntactic gradient vector. Here, the syntactic gradient vector can be generated by the following formula:

[0088] .

[0089] Among them, the above This can represent the syntactic gradient vector. (The above...) This can represent model parameters. (The above...) It can represent the model parameters Find the gradient. (The above...) This can represent a syntax loss function. Here, Adding absolute values ​​can be understood as only caring about the direction sensitivity of the model's gradient vector, regardless of the sign. The direction here is also the direction that strengthens the grammatical structure.

[0090] The third step is to generate gradient relevance information based on the semantic gradient vector and the syntactic gradient vector. In practice, the executing entity can determine the gradient relevance information as the inner product of the semantic gradient vector and the syntactic gradient vector.

[0091] Step 107: Determine the update gradient vector corresponding to the large language model to be aligned with preferences based on gradient correlation information.

[0092] In some embodiments, the execution entity may determine the update gradient vector corresponding to the large language model to be aligned with preferences based on the gradient correlation information.

[0093] In some optional implementations of certain embodiments, based on the aforementioned gradient correlation information, the executing entity can determine the update gradient vector corresponding to the large language model to be aligned with preferences through the following steps:

[0094] The first step is to determine, in response to the determination that the semantic optimization direction represented by the aforementioned gradient relevance information is the same as the grammar maintenance direction, to use the aforementioned semantic gradient vector as the update gradient vector. In practice, in response to the determination that the semantic optimization direction represented by the aforementioned gradient relevance information is the same as the grammar maintenance direction, the aforementioned execution entity can use the aforementioned semantic gradient vector as the update gradient vector.

[0095] The second step, in response to determining that the aforementioned gradient relevance information represents a semantic optimization direction opposite to the grammar maintenance direction, is to generate an updated gradient vector based on the aforementioned semantic gradient vector and grammar gradient vector. In practice, in response to determining that the aforementioned gradient relevance information represents a semantic optimization direction opposite to the grammar maintenance direction, the executing entity can generate the updated gradient vector based on the aforementioned semantic gradient vector and grammar gradient vector through the following steps:

[0096] The first step is to determine the square of the Euclidean norm of the above grammatical gradient vector as the grammatical gradient norm information.

[0097] The second step is to determine the sum of the above syntax gradient norm information and the preset denominator threshold as the updated syntax gradient norm information.

[0098] The third step is to determine the ratio of the gradient correlation information to the updated syntax gradient norm information as the syntax update coefficient information.

[0099] The fourth step is to determine the syntax update gradient vector by combining the above syntax update coefficient information with the above syntax gradient vector.

[0100] The fifth step is to determine the update gradient vector as the difference between the semantic gradient vector and the syntax update gradient vector. The update gradient vector can be generated using the following formula:

[0101] .

[0102] Among them, the above This can represent a semantic gradient vector. (The above...) This can represent the syntactic gradient vector. (The above...) This can represent updating the gradient vector. (The above...) This can represent a preset denominator threshold. Here, the preset denominator threshold can be a small constant used to prevent the denominator from being zero. This can represent the square of the L2 norm (Euclidean norm) of the syntax gradient vector, i.e., the syntax gradient norm information. The above... It can represent the inner product of the semantic gradient vector and the syntactic gradient vector. That is, gradient correlation information.

[0103] Step 108: Update the parameters of the large language model to be aligned with preferences by updating the gradient vector, so as to achieve structured preference alignment of the large language model to be aligned with preferences.

[0104] In some embodiments, the execution entity can use the updated gradient vector to update the parameters of the large language model to be aligned with preferences, thereby achieving structured preference alignment of the large language model. In practice, the execution entity can update the parameters of the large language model to be aligned with preferences using the following formula:

[0105] .

[0106] Among them, the above This can represent the updated model parameters. (The above...) This can represent the original mode parameters. (The above...) This can represent the preset learning rate. (The above...) This can represent updating the gradient vector.

[0107] The above embodiments of this disclosure have the following beneficial effects: Through the structured preference alignment method for large language models in some embodiments of this disclosure, Pareto improvement can be achieved in optimizing the semantic logic capability and maintaining the grammatical format of large language models, while reducing the occurrence of model illusion problems, inference delay problems, and poor robustness problems. Specifically, the reasons for the prevalence of model structure degradation problems, model illusion problems, inference delay problems, and poor robustness problems are as follows: 1) Reinforcement learning based on human feedback requires training and maintaining two models (reward model and policy model), which leads to unstable model training and the training results are prone to deviation (i.e., it needs to rely on the reward model for model fine-tuning, and when the reward model has errors, the policy model has larger errors). Furthermore, because two models need to be maintained at the same time, it leads to a waste of computational resources. 2) When traditional DPO algorithms fine-tune the model, they usually focus on semantics. When the negative samples generated by the model are penalized due to semantic logical errors, the optimization algorithm often cannot distinguish the source of the error (semantic error or grammatical error), which leads to the suppression of the parameter direction carrying the correct grammatical structure at the same time, that is, the structure degradation problem (improving semantic capability, reducing grammatical capability). Furthermore, some methods address structural degradation by setting an external parser to forcibly mask grammatically incorrect lexical units. While the output conforms to grammatical rules, this disrupts the hard constraints of the model's original probability distribution, potentially leading to model illusion (the model may be forced to generate seemingly grammatically correct but factually incorrect content to meet rigid grammatical rules) and inference latency (each lexical unit generation requires real-time checking and intervention from the external parser, introducing additional computational steps and I / O latency). It also reduces the likelihood of poor model robustness due to difficulty adapting to dynamically generated unknown grammatical structures (the external parser, based on a fixed, predefined rule base, lacks generalization ability). Based on this, some embodiments of the large language model structured preference alignment method disclosed herein first obtain a triplet sample set. This triplet sample set includes: a prompt word sequence, a winning response sequence, and a losing response sequence. Based on this triplet sample set, a prompt word winning response sequence set and a prompt word losing response sequence set are generated. Specifically, there is a one-to-one correspondence between the winning response sequence in the set of winning response sequences and the losing response sequence in the set of losing response sequences. This allows us to obtain a set of winning response sequences representing superior response content preferences and a set of losing response sequences representing inferior response content preferences. Then, based on the aforementioned set of winning response sequences, the aforementioned set of losing response sequences, the large language model for preference alignment, and the reference large language model, we generate a set of winning response word probability sequences, a set of losing response word probability sequences, a set of reference winning response word probability sequences, and a set of reference losing response word probability sequences.The reference large language model has the same model architecture as the aforementioned large language model for preference alignment, and its model parameters are frozen. Therefore, we can obtain the set of winning response lexical probability sequences generated by the large language model for preference alignment corresponding to the winning response sequence set, the set of reference winning response lexical probability sequences generated by the reference large language model, and the set of losing response lexical probability sequences generated by the large language model for preference alignment corresponding to the losing response sequence set, as well as the set of reference losing response lexical probability sequences generated by the reference large language model. Then, based on the aforementioned sets of winning response lexical probability sequences, losing response lexical probability sequences, reference winning response lexical probability sequences, and reference losing response lexical probability sequences, we construct the semantic loss function corresponding to the aforementioned large language model for preference alignment. This semantic loss function allows us to measure the difference between the prediction results and the actual results of the large language model and guide it towards results that better align with preferences. Based on the aforementioned set of winning response lexical probability sequences and the winning response sequences included in the aforementioned triplet sample set, we construct the syntactic loss function corresponding to the aforementioned large language model for preference alignment. Therefore, a grammatical loss function representing grammatical structure constraints can be obtained, allowing for more accurate identification and punishment of grammatical biases, thus guiding model optimization. Secondly, based on the aforementioned semantic and grammatical loss functions, gradient relevance information corresponding to the large language model to be aligned with preferences is generated. This gradient relevance information represents the correlation between the semantic optimization direction and the grammatical maintenance direction. This gradient relevance information allows for the determination of whether gradient interference (grammatical loss) exists during the optimization process. Then, based on the gradient relevance information, the update gradient vector corresponding to the large language model to be aligned with preferences is determined. This yields the update gradient vector in the absence of grammatical loss. Finally, using the updated gradient vector, the parameters of the large language model to be aligned with preferences are updated to achieve structured preference alignment. This results in a large language model with structured preference alignment. Furthermore, because gradient relevance information can be obtained by constructing semantic and grammatical loss functions, it is possible to determine whether grammatical loss exists during semantic optimization. When grammatical loss exists, gradient interference elimination operations (i.e., orthogonal projection operations) can be performed to maintain the grammatical structure. This approach achieves Pareto improvements in optimizing the semantic logic capabilities and maintaining the grammatical format of large language models. Furthermore, it eliminates the need for an external parser, reducing issues such as model illusion, inference latency, and poor robustness.

[0108] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a large language model structured preference alignment device, which are similar to... Figure 1 Corresponding to the method embodiments shown, this large language model structured preference alignment device can be specifically applied to various electronic devices.

[0109] like Figure 2 As shown, the large language model structured preference alignment device 200 in some embodiments includes: an acquisition unit 201, a first generation unit 202, a second generation unit 203, a first construction unit 204, a second construction unit 205, a third generation unit 206, a determination unit 207, and a parameter update unit 208. The acquisition unit 201 is configured to acquire a set of triplet samples, wherein the triplet samples in the triplet sample set include: prompt word sequence, winning response sequence, and losing response sequence; the first generation unit 202 is configured to generate a set of prompt word winning response sequences and a set of prompt word losing response sequences based on the triplet sample set, wherein the prompt word winning response sequences in the set of prompt word winning response sequences and the prompt word losing response sequences in the set of prompt word losing response sequences correspond one-to-one; the second generation unit 203 is configured to generate a set of winning response word probability sequences, a set of losing response word probability sequences, a set of reference winning response word probability sequences, and a set of reference losing response word probability sequences based on the set of prompt word winning response sequences, the set of prompt word losing response sequences, the large language model to be aligned with preferences, and the reference large language model, wherein the reference large language model has the same model architecture as the large language model to be aligned with preferences, and the model parameters of the reference large language model are kept frozen; the first construction unit 204 is configured to... The aforementioned set of winning response word probability sequences, set of losing response word probability sequences, set of reference winning response word probability sequences, and set of reference losing response word probability sequences are used to construct the semantic loss function corresponding to the aforementioned large language model to be aligned with preferences. The second construction unit 205 is configured to construct the syntactic loss function corresponding to the aforementioned large language model to be aligned with preferences based on the aforementioned set of winning response word probability sequences and the winning response sequences included in the aforementioned triplet sample set. The third generation unit 206 is configured to generate gradient relevance information corresponding to the aforementioned large language model to be aligned with preferences based on the aforementioned semantic loss function and the aforementioned syntactic loss function, wherein the aforementioned gradient relevance information represents the correlation between the semantic optimization direction and the syntactic maintenance direction. The determination unit 207 is configured to determine the updated gradient vector corresponding to the aforementioned large language model to be aligned with preferences based on the aforementioned gradient relevance information. The parameter update unit 208 is configured to use the aforementioned updated gradient vector to update the parameters of the aforementioned large language model to be aligned with preferences, so as to achieve the structured preference alignment of the aforementioned large language model to be aligned with preferences.

[0110] It is understandable that the units and references described in the structured preference alignment device 200 of this large language model are... Figure 1The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the large language model structured preference alignment device 200 and the units contained therein, and will not be repeated here.

[0111] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0112] like Figure 3 As shown, the electronic device 300 may include a processing unit 301 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0113] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0114] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0115] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0116] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0117] The aforementioned computer-readable medium may be included within the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a set of triplet samples, wherein the triplet samples in the aforementioned triplet sample set include: a prompt word sequence, a winning response sequence, and a losing response sequence; generate a set of prompt word winning response sequences and a set of prompt word losing response sequences based on the aforementioned triplet sample set, wherein the prompt word winning response sequences in the aforementioned set of prompt word winning response sequences and the prompt word losing response sequences in the aforementioned set of prompt word losing response sequences correspond one-to-one; and generate a set of winning response word element probability sequences, a set of losing response word element probability sequences, a set of reference winning response word element probability sequences, and a set of reference losing response word element probability sequences based on the aforementioned set of prompt word winning response sequences, the aforementioned set of prompt word losing response sequences, the large language model to be aligned with preferences, and a reference large language model, wherein the aforementioned reference large language model has the same model architecture as the aforementioned large language model to be aligned with preferences. The model parameters of the aforementioned reference large language model are kept frozen. Based on the aforementioned set of winning response lexical probability sequences, the set of losing response lexical probability sequences, the set of reference winning response lexical probability sequences, and the set of reference losing response lexical probability sequences, a semantic loss function corresponding to the aforementioned large language model to be aligned with preferences is constructed. Based on the aforementioned set of winning response lexical probability sequences and the winning response sequences included in the aforementioned triplet sample set, a syntactic loss function corresponding to the aforementioned large language model to be aligned with preferences is constructed. Based on the aforementioned semantic loss function and the aforementioned syntactic loss function, gradient relevance information corresponding to the aforementioned large language model to be aligned with preferences is generated, wherein the aforementioned gradient relevance information represents the correlation between the semantic optimization direction and the syntactic maintenance direction. Based on the aforementioned gradient relevance information, an update gradient vector corresponding to the aforementioned large language model to be aligned with preferences is determined. Using the aforementioned update gradient vector, the parameters of the aforementioned large language model to be aligned with preferences are updated to achieve structured preference alignment of the aforementioned large language model to be aligned with preferences. Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server.In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0120] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A structured preference alignment method for large language models, characterized in that, include: Obtain a triplet sample set, wherein the triplet samples in the triplet sample set include: prompt word sequence, winning response sequence, and losing response sequence; Based on the triplet sample set, a set of winning response sequences and a set of losing response sequences are generated, wherein the winning response sequences in the set of winning response sequences and the losing response sequences in the set of losing response sequences correspond one-to-one. Based on the set of winning response sequences for the prompt words, the set of losing response sequences for the prompt words, the large language model to be aligned with preferences, and the reference large language model, generate a set of probability sequences for winning response terms, a set of probability sequences for losing response terms, a set of probability sequences for reference winning response terms, and a set of probability sequences for reference losing response terms. The reference large language model has the same model architecture as the large language model to be aligned with preferences, and the model parameters of the reference large language model are kept frozen. Based on the set of winning response word probability sequences, the set of losing response word probability sequences, the set of reference winning response word probability sequences, and the set of reference losing response word probability sequences, construct the semantic loss function corresponding to the large language model to be aligned with preferences. Based on the set of winning response word probability sequences and the set of triplet samples, construct the grammatical loss function corresponding to the large language model to be aligned with preferences. Based on the semantic loss function and the syntactic loss function, gradient relevance information corresponding to the large language model to be aligned with preferences is generated, wherein the gradient relevance information represents the correlation between the semantic optimization direction and the syntactic maintenance direction; Based on the gradient correlation information, determine the update gradient vector corresponding to the large language model to be aligned with preferences; The updated gradient vector is used to update the parameters of the large language model to be aligned with preferences, so as to achieve structured preference alignment of the large language model to be aligned with preferences.

2. The method according to claim 1, characterized in that, The step of generating a set of winning response sequences and a set of losing response sequences based on the triplet sample set includes: For each triplet sample in the triplet sample set, perform the following processing steps: The prompt word sequence and winning response sequence included in the triplet sample are combined to generate a prompt word winning response sequence; The prompt word sequence and the failed response sequence included in the triplet sample are combined to generate a prompt word failed response sequence; The generated winning response sequences for each prompt word are defined as the set of winning response sequences for each prompt word. The generated sequence of failed responses for each prompt word is defined as the set of failed response sequences for prompt words.

3. The method according to claim 1, characterized in that, The step of generating gradient relevance information corresponding to the large language model to be aligned with preferences based on the semantic loss function and the syntactic loss function includes: Based on the semantic loss function, a semantic gradient vector is generated, wherein the semantic gradient vector represents the parameter update direction that increases the probability of winning response and decreases the probability of losing response; Based on the syntactic loss function, a syntactic gradient vector is generated, wherein the syntactic gradient vector represents the parameter update direction of the probability of strengthening the generation of lexical units; Gradient relevance information is generated based on the semantic gradient vector and the syntactic gradient vector.

4. The method according to claim 3, characterized in that, The step of determining the update gradient vector corresponding to the large language model to be aligned with preferences based on the gradient correlation information includes: In response to determining that the gradient correlation information represents the semantic optimization direction and the syntax maintenance direction are the same, the semantic gradient vector is determined as the update gradient vector; In response to determining that the gradient correlation information represents a semantic optimization direction opposite to the grammar maintenance direction, an updated gradient vector is generated based on the semantic gradient vector and the grammar gradient vector.

5. The method according to claim 1, characterized in that, The step of generating a set of probability sequences for winning responses, a set of probability sequences for losing responses, a set of probability sequences for reference winning responses, and a set of probability sequences for reference losing responses based on the set of winning response sequences for prompt words, the set of losing response sequences for prompt words, the large language model to be aligned with preferences, and the reference large language model includes: Each of the winning response sequences included in the set of winning response sequences is input into the large language model to be aligned with preferences to obtain a set of winning response word probability sequences. The winning response word probability sequences in the set of winning response word probability sequences correspond one-to-one with the winning response sequences in the set of winning response sequences. The winning response word probability sequences in the set of winning response word probability sequences represent the probability of generating each winning response word in the winning response sequence. Each of the failed response sequences in the set of failed response sequences is input into the large language model to be aligned with preferences to obtain a set of failed response token probability sequences. The failed response token probability sequences in the set of failed response token probability sequences correspond one-to-one with the failed response sequences in the set of failed response sequences. The failed response token probability sequences in the set of failed response token probability sequences represent the probability of generating each failed response token in the failed response sequence. Each winning response sequence of the prompt word included in the set of winning response sequences is input into the reference large language model to obtain a set of reference winning response word probability sequences. The reference winning response word probability sequences in the set of reference winning response word probability sequences correspond one-to-one with the winning response sequences of the prompt word included in the set of winning response sequences. The reference winning response word probability sequences in the set of reference winning response word probability sequences represent the reference probability of each winning response word in the winning response sequence. Each of the failed prompt word response sequences in the set of failed prompt word response sequences is input into the reference large language model to obtain a set of reference failed prompt word probability sequences. The reference failed prompt word probability sequences in the set of reference failed prompt word probability sequences correspond one-to-one with the failed prompt word response sequences in the set of failed prompt word response sequences. The reference failed prompt word probability sequences in the set of reference failed prompt word probability sequences represent the reference probability of each failed prompt word in the failed prompt response sequence.

6. The method according to claim 1, characterized in that, The step of constructing the semantic loss function corresponding to the large language model to be aligned with preferences based on the set of winning response word probability sequences, the set of losing response word probability sequences, the set of reference winning response word probability sequences, and the set of reference losing response word probability sequences includes: Based on the set of winning response word probability sequences, generate a set of winning response probabilities; Based on the set of probability sequences of defeated response terms, generate a set of probability sequences of defeated responses; Based on the set of reference winning response word probability sequences, a set of reference winning response probabilities is generated, wherein the reference winning response probabilities in the set of reference winning response probabilities correspond one-to-one with the winning response probabilities in the set of winning response probabilities; Based on the set of reference defeated response token probabilities, a set of reference defeated response probabilities is generated, wherein the reference defeated response probabilities in the set of reference defeated response probabilities correspond one-to-one with the defeated response probabilities in the set of defeated response probabilities. A set of winning logarithmic ratios is generated based on the winning response probability set and the reference winning response probability set; Based on the set of lost response probabilities and the set of reference lost response probabilities, a set of lost log ratios is generated, wherein the lost log ratios in the set of lost log ratios correspond one-to-one with the winning log ratios in the set of winning log ratios. For each winning log ratio in the set of winning log ratios, perform the following processing steps: The loser log ratio value in the set of loser log ratio values ​​that corresponds to the winr log ratio value is determined as the target loser log ratio value; The difference between the winning log ratio and the target losing log ratio is determined as the winning-losing difference information; The product of the win / loss difference information and the preset preference parameter is determined as the win / loss preference information; The win / loss preference information is mapped using a preset activation function to obtain mapped preference information. The logarithm of the mapping preference information is determined as the logarithmic preference information; The negative of the mean of each determined logarithmic preference information is used as the loss function value of the semantic loss function corresponding to the large language model to be aligned with preferences.

7. The method according to claim 1, characterized in that, The step of constructing the syntactic loss function corresponding to the large language model to be aligned with preferences based on the set of winning response lexical probability sequences and the set of triplet samples includes: Logarithmic processing is performed on each winning response word probability sequence included in the winning response word probability sequence set to generate a winning response word logarithmic probability sequence set. The winning response sequences included in the triplet sample set are subjected to binary masking to generate a masked winning response sequence set, wherein the masked winning response sequences in the masked winning response sequence set correspond one-to-one with the winning response word log probability sequences in the winning response word log probability sequence set; For each winning response term log probability sequence in the set of winning response term log probability sequences, perform the following processing steps: The mask winning response sequence in the set of mask winning response sequences that corresponds to the log probability sequence of the winning response token is determined as the target mask information sequence, wherein the target mask information in the target mask information sequence corresponds one-to-one with the log probability of the winning response token in the log probability sequence of the winning response token; For each target mask information in the target mask information sequence, the product of the target mask information and the corresponding log probability of the winning response word is determined as the mask probability information; The sum of the obtained mask probability information is used to determine the total mask probability information; The negative of the mean of the total probability information of each mask is determined as the loss function value of the syntax loss function corresponding to the large language model to be aligned with preferences.

8. A structured preference alignment device for a large language model, characterized in that, include: The acquisition unit is configured to acquire a set of triplet samples, wherein the triplet samples in the set of triplet samples include: prompt word sequence, winning response sequence and losing response sequence; The first generation unit is configured to generate a set of winning response sequences and a set of losing response sequences based on the triplet sample set, wherein the winning response sequences in the set of winning response sequences and the losing response sequences in the set of losing response sequences correspond one-to-one. The second generation unit is configured to generate a set of probability sequences of winning response terms, a set of probability sequences of losing response terms, a set of probability sequences of reference winning response terms, and a set of probability sequences of reference losing response terms, based on the set of winning response sequences of prompt words, the set of losing response sequences of prompt words, the large language model to be aligned with preferences, and the reference large language model. The reference large language model has the same model architecture as the large language model to be aligned with preferences, and the model parameters of the reference large language model are kept frozen. The first construction unit is configured to construct the semantic loss function corresponding to the large language model to be aligned with preferences based on the set of winning response word probability sequences, the set of losing response word probability sequences, the set of reference winning response word probability sequences, and the set of reference losing response word probability sequences. The second construction unit is configured to construct the syntax loss function corresponding to the large language model to be aligned with preferences based on the set of winning response word probability sequences and the set of triplet samples. The third generation unit is configured to generate gradient relevance information corresponding to the large language model to be aligned with preferences based on the semantic loss function and the syntactic loss function, wherein the gradient relevance information characterizes the correlation between the semantic optimization direction and the syntactic maintenance direction. The determining unit is configured to determine the update gradient vector corresponding to the large language model to be aligned with preferences based on the gradient correlation information. The parameter update unit is configured to update the parameters of the large language model to be aligned with preferences using the updated gradient vector, so as to achieve structured preference alignment of the large language model to be aligned with preferences.

9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images