Large language model value alignment method and device, equipment and storage medium

Abstract

The invention discloses a large language model value alignment method and device, equipment and a storage medium. The method comprises the following steps: according to a standard preference data set, target preference distribution is set; output distribution of the target fine adjustment alignment algorithm in the fine adjustment process of the target large language model is obtained and serves as distribution to be optimized; extracting a token embedding table of the target large language model, calculating a semantic distance between tokens according to a norm of an embedded vector in the token embedding table, and constructing an optimal transmission cost matrix fusing semantic information; a preference alignment problem of a distribution object to be optimized and target preference distribution is converted into an optimal transmission problem, the minimum transmission distance between the distribution object to be optimized and the target preference distribution is calculated, and a PLOT loss item is obtained; and constructing a joint loss function according to the PLOT loss item and a basic loss item of the target fine tuning method, and performing end-to-end fine tuning optimization on the model. According to the scheme, the token-level preference learning is converted into the optimal transmission problem, so that the spanning from local optimization to global distribution alignment is realized, and the integrity of the preference learning is improved.

Description

Technical Field

[0001] This application relates to the field of large language model technology, specifically to a method, apparatus, device, and storage medium for aligning the value of a large language model. Background Technology

[0002] The rapid development of large language models has enabled them to demonstrate powerful capabilities in numerous fields. However, unaligned models may generate harmful, unhelpful, or logically inconsistent content, severely limiting their practical applications. Value alignment techniques, which enable models to learn human judgment standards through preference learning, have become one of the core directions of research in large language models. Currently, alignment techniques for large language models are mainly divided into two categories: inference-stage alignment methods and fine-tuning-stage alignment methods. Inference-stage alignment methods regulate model behavior through constraints during the decoding process (such as controllable decoding, output sampling filtering, etc.), without modifying model parameters. However, they suffer from increased computational overhead during inference, vulnerability to adversarial attacks, and insufficient long-term alignment reliability. Fine-tuning-stage alignment methods, on the other hand, internalize preferences into model capabilities by adjusting model parameters, resulting in more stable performance. This has become the mainstream technical approach. Its core is to optimize the model's output distribution based on preference data by designing reasonable loss functions and training strategies. The core requirement of preference learning is to enable models to accurately capture differences in human preferences for output (such as the distinction between "high-quality answers" and "low-quality answers") and transform these differences into learnable signals for the model.

[0003] Current mainstream methods for preference learning in the fine-tuning stage can be divided into two categories: reinforcement learning base classes and pure fine-tuning classes. For example, reinforcement learning based on human feedback (RLHF), a representative method in the alignment stage, constructs a dataset of positive and negative samples reflecting preferences using human annotations. First, a reward model is trained on this dataset to quantify output quality, and then a proximal policy optimization (PPO) algorithm is used for reinforcement learning. Its core mechanism is to limit the parameter update magnitude by introducing the KL divergence with the original model, preventing the model from deviating too much from its original capabilities. However, this method has significant drawbacks: it requires constructing a multi-model architecture including a reward model, a reference model, and a critique model, resulting in extremely high computational costs; the training process relies on the design of the reward model, making it prone to instability; and it requires a large amount of human-annotated data, leading to high annotation costs.

[0004] Furthermore, to address the complexity of RLHF, existing technologies have proposed Direct Preference Optimization (DPO). DPO eliminates variational function dependencies through mathematical derivation of the PPO loss term, transforming preference learning into a supervised contrastive learning form. This method requires only a language model for parameter updates and a fixed reference model, eliminating the need to train reward and critic models, significantly reducing computational costs while achieving performance similar to RLHF, making it a mainstream alternative. However, DPO still focuses on the contrastive optimization of local token probabilities, failing to fully consider the global semantic relationships between tokens, resulting in a lack of holistic preference learning and limited performance improvement in complex preference tasks (such as logical reasoning and multi-dimensional value judgment). Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for aligning the value of a large language model, which can solve the problem that the preference learning process of a large language model in the prior art does not fully consider the global semantic relationship between tokens, resulting in a lack of holistic preference learning and limited performance improvement in complex preference tasks.

[0006] Firstly, embodiments of this application provide a method for aligning the value of a large language model, which employs the following approach: A large language model value alignment method, which will be integrated into the target fine-tuning alignment algorithm and used to enhance the value alignment performance of the target large language model, includes the following steps: Based on a pre-collected standard preference dataset, a target preference distribution reflecting human preferences is constructed; wherein, the standard preference dataset includes at least user query data, preferred answer data, and non-preferred answer data; The output distribution of the target fine-tuning alignment algorithm during the fine-tuning of the target large language model is obtained as the distribution to be optimized, and is used as the distribution object to be optimized; Extract the token embedding table of the target large language model, and embed the vectors in the token embedding table accordingly. The norm is used to calculate the semantic distance between tokens, and an optimal transmission cost matrix that integrates semantic information is constructed to replace the original cost matrix of the target large language model. The preference alignment problem between the distribution to be optimized and the target preference distribution is transformed into an optimal transmission problem. The minimum transmission distance between the two is calculated to obtain the PLOT loss term. ; According to PLOT loss item A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function.

[0007] In conjunction with the first aspect, in one implementation, the step of constructing a target preference distribution reflecting human preferences based on a pre-collected standard preference dataset... The target preference distribution is obtained by transforming the reward model output distribution or the positive and negative sample token frequency difference distribution to ensure that the target preference distribution meets the set mathematical distribution requirements.

[0008] In conjunction with the first aspect, in one implementation, when the target preference distribution is obtained by transforming the output distribution of the reward model, the following steps are taken: A pre-trained, mature reward model is used to score preference responses. The output distribution of the reward model is used as the initial preference distribution, and the target preference distribution is obtained by normalizing the initial preference distribution.

[0009] In conjunction with the first aspect, in one implementation, when the target preference distribution is obtained by transforming the positive and negative sample token frequency difference distribution, the following steps are taken: Statistically analyze the token frequency distribution of preferred and non-preferred responses in the standard preference dataset, and calculate the difference distribution. The target preference distribution is obtained by performing non-negative transformation and normalization on the difference distribution.

[0010] In conjunction with the first aspect, in one implementation, the step of extracting the token embedding table of the target large language model, calculating the semantic distance between tokens based on the L2 norm of the embedding vectors in the token embedding table, and constructing an optimal transmission cost matrix that fuses semantic information to replace the original cost matrix of the target large language model includes the following steps: Obtain the token embedding table of the target large language model. ,in Let be the embedding vector of the i-th token, with dimension d; Calculate the L2 norm (Euclidean norm) of each embedding vector to transform the high-dimensional embedding into one-dimensional semantic features: ,in Let be the value of the k-th dimension of e.

[0011] Construct an n×n dimensional cost matrix C using the absolute difference between the embedding norms of two tokens as the transmission cost, where elements .

[0012] In conjunction with the first aspect, in one implementation, the problem of aligning the distribution to be optimized with the target preference distribution is transformed into an optimal transmission problem, and the minimum transmission distance between the two is calculated to obtain the PLOT loss term. The following formula is used: in: and These are the cumulative distribution functions (CDF) of Q and P, respectively. , which is the distance interval between adjacent tokens; The W distance is calculated by integrating the difference in CDF.

[0013] In conjunction with the first aspect, in one implementation, the step of calculating the loss term based on the PLOT... A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function, employing the following formula: Here, α is a hyperparameter used to balance the weights of basic fine-tuning and preference enhancement.

[0014] Secondly, embodiments of this application provide a large language model value alignment device, which adopts the following scheme: A large language model value alignment device, the device comprising: A preference distribution construction module is configured to construct a target preference distribution reflecting human preferences based on a pre-collected standard preference dataset; wherein the standard preference dataset includes at least user query data, preferred answer data, and non-preferred answer data; The model output distribution extraction module is configured to obtain the output distribution of the target fine-tuning alignment algorithm during the fine-tuning of the target large language model as the distribution to be optimized, and as the distribution object to be optimized; The semantic-aware cost matrix construction module is configured to extract the token embedding table of the target large language model, calculate the semantic distance between tokens based on the L2 norm of the embedding vectors in the token embedding table, and construct the optimal transmission cost matrix that integrates semantic information to replace the original cost matrix of the target large language model. The optimal transmission loss calculation module is configured to transform the preference alignment problem between the distribution to be optimized and the target preference distribution into an optimal transmission problem, calculate the minimum transmission distance between the two, and obtain the PLOT loss term. ; The joint optimization module is configured to optimize based on the PLOT loss term. A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function.

[0015] Thirdly, embodiments of this application provide a large language model value alignment device, which adopts the following scheme: A large language model value alignment device includes a processor, a memory, and a large language model value alignment program stored in the memory and executable by the processor, wherein when the large language model value alignment program is executed by the processor, it implements the steps of the large language model value alignment method as described in the claims.

[0016] Fourthly, embodiments of this application provide a storage medium, employing the following technical solution: A storage medium storing a large language model value alignment program of an optimal transfer algorithm, wherein when the large language model value alignment program of the optimal transfer algorithm is executed by a processor, it implements the steps of the large language model value alignment method as described above.

[0017] The beneficial effects of the technical solutions provided in this application include at least the following: The method, apparatus, device, and storage medium for value alignment of large language models provided in this application transform token-level preference learning into an optimal transmission problem, achieving a leap from local optimization to global distribution alignment and improving the overall effectiveness of preference learning. Simultaneously, the cost matrix is ​​constructed using base token semantic embedding, enabling preference optimization to consider both probability distribution and semantic association, achieving optimal transmission computation that integrates semantic information, enhancing the model's ability to capture fine-grained preference differences, and addressing the insufficient accuracy of preference capture in traditional methods. Furthermore, the original distribution characteristics of the model are preserved during the optimization process, and a joint loss function design balances preference alignment effectiveness with generality, avoiding the "alignment tax" problem present in related technologies. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall process of an embodiment of the large language model value alignment method of this application; Figure 2 This is a schematic diagram of the functional modules in one embodiment of the large language model value alignment device of this application; Figure 3 This is a schematic diagram of the hardware structure of the large language model value alignment device involved in the embodiments of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0020] The present application provides a method, apparatus, device, and storage medium for aligning the value of a large language model. The key inventive point is that by transforming token-level preference learning into an optimal transmission problem, it achieves a leap from local optimization to global distribution alignment, improving the overall holistic nature of preference learning. Simultaneously, by constructing a cost matrix based on token semantic embedding, preference optimization considers both probability distribution and semantic association, achieving optimal transmission computation that integrates semantic information. This enhances the model's ability to capture fine-grained preference differences and solves the problem of insufficient preference capture accuracy in traditional methods. Furthermore, the original distribution characteristics of the model are preserved during the optimization process. Through the design of a joint loss function, the preference alignment effect and generality are balanced, avoiding the "alignment tax" problem existing in related technologies.

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0022] Firstly, embodiments of this application provide a method for aligning the value of a large language model.

[0023] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the large language model value alignment method of this application. The large language model value alignment method provided by this invention targets a preference learning task based on a standard preference dataset. ,in For user queries, To provide answers that align with human preferences, For responses that deviate from the preferred approach, N is the total number of samples. Let the model output distribution during fine-tuning be... (Depending on model parameters θ), the target preference distribution is as follows: The core of this invention is to calculate the difference between the two through optimal transmission and transform it into an optimizable loss term.

[0024] Specifically, the method provided by this invention includes the following steps: S100, Target Preference Distribution Construction: Based on a pre-collected standard preference dataset, a target preference distribution reflecting human preferences is constructed; wherein, the standard preference dataset includes at least user query data, preferred answer data, and non-preferred answer data; It should be noted that the target preference distribution It must possess strict mathematical distribution characteristics (non-negativity, normalization). This specific implementation provides two construction methods: obtaining it through the output distribution of the reward model or by transforming the distribution of positive and negative sample token frequency differences, ensuring that the target preference distribution meets the set mathematical distribution requirements.

[0025] In some embodiments, target preference distribution When transforming the output distribution of the reward model, a pre-trained and mature reward model is used to score the preference responses. The output distribution of the reward model is used as the initial preference distribution, and the target preference distribution is obtained by normalizing the initial preference distribution. .

[0026] In other embodiments, the target preference distribution When transforming the token frequency difference distribution of positive and negative samples, the token frequency distribution of preferred and non-preferred answer data in the standard preference dataset is statistically analyzed, and the difference distribution is calculated using the following formula: However, since Q may take values ​​in the range [-1, 1], it needs to be further transformed into a strict distribution through nonnegativity transformation and normalization: Nonnegative transformation: The range after transformation is ; Normalization: Where p is the value of Q on the i-th token, n is the vocabulary size, and the target preference distribution P is finally obtained.

[0027] S200, Model Output Distribution Extraction: Obtain the output distribution of the target fine-tuning alignment algorithm during the fine-tuning of the target large language model. (θ is the model parameter) is used as the distribution to be optimized, and as the distribution object to be optimized; S300, Semantic Awareness Cost Matrix Construction: Extract the token embedding table of the target large language model, and based on the embedding vectors in the token embedding table... The norm is used to calculate the semantic distance between tokens, and an optimal transmission cost matrix that integrates semantic information is constructed to replace the original cost matrix of the target large language model. Specifically, traditional optimal transmission calculation uses a 0-1 cost matrix (0 for identical tokens, 1 for different tokens), without considering semantic association. This invention constructs a cost matrix C that incorporates semantic information through token embedding vectors. The specific steps are as follows: I. Token Embedding Extraction: Obtaining the token embedding table of the large language model ,in Let be the embedding vector of the i-th token, with dimension d.

[0028] II. Embedding Vector Normalization: Calculate the normalization of each embedding vector. The Euclidean norm transforms high-dimensional embeddings into one-dimensional semantic features: ,in Let be the value of the k-th dimension of e.

[0029] III. Cost Matrix Calculation: Using the absolute difference between the embedding norms of two tokens as the transmission cost, construct an n×n dimensional cost matrix C, where elements... This matrix ensures lower transmission costs between tokens with similar semantics, allowing preference optimization to balance probability and semantics.

[0030] S400. Optimal Transmission Loss Calculation: The preference alignment problem between the distribution to be optimized and the target preference distribution is transformed into an optimal transmission problem. The minimum transmission distance between the two is calculated to obtain the PLOT loss term. ; Specifically, the PLOT loss term is essentially the model output distribution. With target preference distribution To determine the optimal transmission distance between language models, considering the large vocabulary size (typically hundreds of thousands), directly solving the linear programming problem would be computationally too costly. Therefore, this invention uses a one-dimensional Wasserstein distance (W) to simplify the calculation. The specific formula is as follows: in: and These are the cumulative distribution functions (CDF) of Q and P, respectively. , which is the distance interval between adjacent tokens; The W distance is calculated by integrating the differences in CDF (or summing in discrete scenarios), which preserves the global optimization characteristics of optimal transmission while reducing the computational complexity to zero. Suitable for scenarios with large vocabulary.

[0031] S500, Joint Optimization: Based on PLOT Loss Term A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function.

[0032] Specifically, this invention combines the PLOT loss term with the basic loss term of existing fine-tuning methods to construct a joint loss function: Where α is a hyperparameter, the value of which ranges from 5 to 10, with an optimal value of 8, used to balance the weights of basic fine-tuning and preference enhancement.

[0033] Based on the above steps, this embodiment further proposes a complete training process for a large-scale target language model, including the following steps: F1. Data Preparation: Load the preference dataset, preprocess it, and then construct sample pairs. ; F2. Load the pre-trained large language model (such as Llama3, Qwen2.5 series) and set the parameters of the basic fine-tuning method (SFT, DPO, PRO, etc.); F3. Distribution Construction: Calculate the target preference distribution P based on the dataset, and extract the model token embedding table to construct the cost matrix C; F4. Fine-tuning Training: Calculating the Base Loss With PLOT loss The model parameters are updated by a joint loss function. The hardware environment is 4×NVIDIA A100 80GB GPU with a batch size of 4. The training can converge after 1 epoch. F5. Model Output: After training, save the model parameters to obtain a large value alignment model that incorporates the optimal transmission preference enhancement.

[0034] Compared with the current mainstream DPO method and the similar AOT method, the advantages of this invention are mainly reflected in the following aspects: Comparison Dimensions Existing technologies (DPO / AOT) This invention (PLOT enhancement method) Optimize perspective DPO: Local token probability comparison; AOT: Batch-level distribution alignment Token-level global distribution alignment, balancing probability and semantics. Harmlessness performance (ASR index) DPO achieved an ASR of 30.08% and an AOT of 28.93% on Llama 3.2-3B using GCG attacks. After PLOT enhancement, the success rates dropped to 26.83% and 25.54% respectively, significantly reducing the attack success rate. Logical reasoning performance (GSM8K) The accuracy rate of DPO was 65.49%. After PLOT enhancement, the percentage increased to 68.36%, an increase of 2.87 percentage points. Hyperparameter stability When the DPO hyperparameter changes by ±50%, the ASR fluctuation reaches 1.32%. PLOT fluctuation was only 0.04%, significantly improving robustness. Calculation cost The training time for a single epoch of DPO is T; the training time for AOT is 1.2T due to batch distribution. PLOT has a computational complexity of O(n log n) and a training time of 1.05T, which is close to DPO. General capabilities retained DPO has a 7.80% win rate in AlpacaEval 2.0 (LC). After the PLOT enhancement, the efficiency reached 9.15%, and its versatility actually increased instead of decreasing. In summary, compared with existing technologies, this invention has significantly improved preference learning performance, stability, and semantic capture accuracy, while controlling computational costs and achieving a balance of "high performance, high stability, and low cost".

[0035] To verify the feasibility and superiority of this invention, this application further uses three large language models with different architectures and parameter scales as experimental subjects: Llama3.2-3B-Instruct, Llama3.1-8B-Instruct, Qwen2.5-7B-Instruct. Select two types of core preference tasks: human values ​​(including harmlessness, helpfulness) The datasets and evaluation methods are as follows: (The datasets include three sub-preferences: social assistance, humanistic care, and logic problem solving, which includes four sub-preferences: mathematics, reasoning, programming, and STEM). Training data: The harmless task used the Harmless base subset (42,536 items) of the HH-RLHF dataset; the helpful task used the Helpful subset (118,257 items) of HH-RLHF; and the logical task used the Magpie subset (59,539 items) of INFORM-Preference-Magnitude-80K. 4,000 samples were randomly sampled for training for each task class.

[0036] Test data: Harmlessness was tested using the HarmBench benchmark, which included six red team attack methods such as ZS, PEZ, and GCG; Helpfulness was tested using the HH-RLHF test set (6,240 items) and GPT-4 human scoring; Logical tasks were tested using GSM8K (1,000 items), MATH (5,000 items), and GPT-4 scoring from MT-Bench.

[0037] Baseline methods: SFT, DPO, PRO, AOT. The PLOT loss of this invention is integrated into the above methods respectively, and the performance differences before and after enhancement are compared.

[0038] Experimental results: Harmlessness preference learning outcomes Using attack success rate (ASR) as the core metric, a lower ASR indicates better harmlessness. Experimental results show that the ASR of all baseline methods significantly decreased after PLOT enhancement: Llama 3.2-3B-Instruct + DPO combination: GCG attack ASR decreased from 30.08% to 26.83%, a decrease of 3.25 percentage points; ZS attack ASR decreased from 8.46% to 4.39%, a decrease of 4.07 percentage points; The Qwen2.5-7B-Instruct+AOT combination significantly reduced the ASR of GCG attacks from 38.59% to 35.27% and the SFS attack from 26.86% to 20.21%.

[0039] Human values ​​and logic task results The PLAT augmentation method performed best in helpfulness, humanistic care, and logical reasoning tasks. Helpful: The reward model score improved from 70.63 for DPO to 72.14, and the GPT-4 score improved from 7.87 to 8.74; Mathematical reasoning: GSM8K accuracy improved from 65.49 to 68.36, and MATH accuracy improved from 43.89 to 48.28. STEM tasks: The GPT-4 score improved from 7.45 to 8.25, demonstrating its adaptability to complex preference tasks.

[0040] Stability and general capability verification Hyperparameter sensitivity experiments show that when α varies in the range of 5-12, the ASR fluctuation of the PLOT augmentation method does not exceed 0.5%, while the fluctuation of DPO reaches 1.8%; the LC win rate of AlpacaEval 2.0 increases from 7.80% to 9.15%, proving that the model's generality is not lost but enhanced.

[0041] Experimental conclusions Experimental results fully demonstrate that the PLOT method of this invention can effectively enhance the preference learning performance of existing fine-tuning alignment techniques, achieving performance improvement in multiple dimensions of tasks such as harmlessness, helpfulness, and logical reasoning. At the same time, it has excellent hyperparameter stability and general capability retention characteristics, controllable computational cost, and fully meets the needs of practical applications.

[0042] Secondly, embodiments of this application also provide a large language model value alignment device.

[0043] A large language model value alignment device, the device comprising: A preference distribution construction module is configured to construct a target preference distribution reflecting human preferences based on a pre-collected standard preference dataset; wherein the standard preference dataset includes at least user query data, preferred answer data, and non-preferred answer data; The model output distribution extraction module is configured to obtain the output distribution of the target fine-tuning alignment algorithm during the fine-tuning of the target large language model as the distribution to be optimized, and as the distribution object to be optimized; The semantic-aware cost matrix construction module is configured to extract the token embedding table of the target large language model, calculate the semantic distance between tokens based on the L2 norm of the embedding vectors in the token embedding table, and construct the optimal transmission cost matrix that integrates semantic information to replace the original cost matrix of the target large language model. The optimal transmission loss calculation module is configured to transform the preference alignment problem between the distribution to be optimized and the target preference distribution into an optimal transmission problem, calculate the minimum transmission distance between the two, and obtain the PLOT loss term. ; The joint optimization module is configured to optimize based on the PLOT loss term. A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function.

[0044] The functions of each module in the above-mentioned large language model value alignment device based on the optimal transmission algorithm correspond to the steps in the above-mentioned large language model value alignment method embodiment, and their functions and implementation processes will not be described in detail here.

[0045] Thirdly, embodiments of this application provide a large language model value alignment device based on an optimal transmission algorithm. The large language model value alignment device based on an optimal transmission algorithm can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0046] Reference Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of a large language model value alignment device based on an optimal transmission algorithm, as described in an embodiment of this application. In this embodiment, the large language model value alignment device based on the optimal transmission algorithm may include a processor, a memory, a communication interface, and a communication bus.

[0047] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interfaces. The communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the large language model value alignment device based on the optimal transmission algorithm, as well as interfaces used for interconnecting the large language model value alignment device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0048] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0049] The processor can be a general-purpose processor, which can call a large language model value alignment program based on the optimal transfer algorithm stored in memory and execute the large language model value alignment method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the large language model value alignment program based on the optimal transfer algorithm is called can refer to the various embodiments of the large language model value alignment method of this application, and will not be repeated here.

[0050] Fourthly, embodiments of this application also provide a storage medium.

[0051] This application stores a large language model value alignment program based on an optimal transmission algorithm on its storage medium, wherein... When the large language model value alignment program based on the optimal transmission algorithm is executed by the processor, it implements the steps of the large language model value alignment method described above.

[0052] The method implemented when the large language model value alignment procedure based on the optimal transmission algorithm is executed can be referred to in various embodiments of the large language model value alignment method of this application, and will not be repeated here.

[0053] It should be noted that the sequence numbers of the embodiments in this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not represent a sequential order, nor do they limit "first," "second," and "third" to different types.

[0054] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0055] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0056] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of operation 5 is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0057] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0058] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for aligning the value of a large language model, characterized in that, It will be integrated into the target fine-tuning alignment algorithm and used to enhance the value alignment performance of the target large language model, which includes the following steps: Based on a pre-collected standard preference dataset, a target preference distribution reflecting human preferences is constructed; wherein, the standard preference dataset includes at least user query data, preferred answer data, and non-preferred answer data; The output distribution of the target fine-tuning alignment algorithm during the fine-tuning of the target large language model is obtained as the distribution to be optimized, and is used as the distribution object to be optimized; Extract the token embedding table of the target large language model, and embed the vectors in the token embedding table accordingly. The norm is used to calculate the semantic distance between tokens, and an optimal transmission cost matrix that integrates semantic information is constructed to replace the original cost matrix of the target large language model. The preference alignment problem between the distribution to be optimized and the target preference distribution is transformed into an optimal transmission problem. The minimum transmission distance between the two is calculated to obtain the PLOT loss term. ; According to PLOT loss item A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function.

2. The large language model value alignment method as described in claim 1, characterized in that, The process involves constructing a target preference distribution that reflects human preferences based on a pre-collected standard preference dataset. The target preference distribution is obtained by transforming the reward model output distribution or the positive and negative sample token frequency difference distribution to ensure that the target preference distribution meets the set mathematical distribution requirements.

3. The large language model value alignment method as described in claim 2, characterized in that, When the target preference distribution is obtained by transforming the output distribution of the reward model, the following steps are taken: A pre-trained, mature reward model is used to score preference responses. The output distribution of the reward model is used as the initial preference distribution, and the target preference distribution is obtained by normalizing the initial preference distribution.

4. The large language model value alignment method as described in claim 2, characterized in that, When the target preference distribution is obtained by transforming the positive and negative sample token frequency difference distribution, the following steps are taken: Statistically analyze the token frequency distribution of preferred and non-preferred responses in the standard preference dataset, and calculate the difference distribution. The target preference distribution is obtained by performing non-negative transformation and normalization on the difference distribution.

5. The large language model value alignment method as described in claim 1, characterized in that, The process of extracting the token embedding table of the target large language model, calculating the semantic distance between tokens based on the L2 norm of the embedding vectors in the token embedding table, and constructing an optimal transmission cost matrix that fuses semantic information to replace the original cost matrix of the target large language model includes the following steps: Obtain the token embedding table of the target large language model. ,in Let be the embedding vector of the i-th token, with dimension d; Calculate the L2 norm (Euclidean norm) of each embedding vector to transform the high-dimensional embedding into one-dimensional semantic features: ,in Let e ​​be the value of its k-th dimension. Construct an n×n dimensional cost matrix C using the absolute difference between the embedding norms of two tokens as the transmission cost, where elements .

6. The large language model value alignment method as described in claim 1, characterized in that, The problem of aligning the distribution to be optimized with the target preference distribution is transformed into an optimal transmission problem. The minimum transmission distance between the two is calculated to obtain the PLOT loss term. The following formula is used: in: and , respectively, are the cumulative distribution functions (CDF) of Q and P; , which is the distance interval between adjacent tokens; The W distance is calculated by integrating the difference in CDF.

7. The large language model value alignment method as described in claim 6, characterized in that, The loss term based on PLOT A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function, employing the following formula: Here, α is a hyperparameter used to balance the weights of basic fine-tuning and preference enhancement.

8. A large language model value alignment device, characterized in that, The device includes: A preference distribution construction module is configured to construct a target preference distribution reflecting human preferences based on a pre-collected standard preference dataset; wherein the standard preference dataset includes at least user query data, preferred answer data, and non-preferred answer data; The model output distribution extraction module is configured to obtain the output distribution of the target fine-tuning alignment algorithm during the fine-tuning of the target large language model as the distribution to be optimized, and as the distribution object to be optimized; The semantic-aware cost matrix construction module is configured to extract the token embedding table of the target large language model, calculate the semantic distance between tokens based on the L2 norm of the embedding vectors in the token embedding table, and construct the optimal transmission cost matrix that integrates semantic information to replace the original cost matrix of the target large language model. The optimal transmission loss calculation module is configured to transform the preference alignment problem between the distribution to be optimized and the target preference distribution into an optimal transmission problem, calculate the minimum transmission distance between the two, and obtain the PLOT loss term. ; The joint optimization module is configured to optimize based on the PLOT loss term. A joint loss function is constructed using the base loss term of the target fine-tuning method, and the model is then fine-tuned end-to-end using the joint loss function.

9. A large language model value alignment device, characterized in that, The large language model value alignment device includes a processor, a memory, and a large language model value alignment program stored in the memory and executable by the processor, wherein when the large language model value alignment program is executed by the processor, it implements the steps of the large language model value alignment method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a large language model value alignment program with an optimal transfer algorithm, wherein when the large language model value alignment program with the optimal transfer algorithm is executed by the processor, it implements the steps of the large language model value alignment method as described in claims 1 to 7.

Images