Large language model adaptive multi-preference alignment method and system based on hybrid expert architecture

Abstract

The invention discloses a large language model adaptive multi-preference alignment method and system based on a hybrid expert architecture, and relates to the field of artificial intelligence. The problem that in the prior art, a multi-preference mixing method with self-adaptive capacity is needed to promote multi-preference research is solved. The method comprises the following steps: adding a designed preference mixing module based on a mixed expert architecture on a large language model; freezing backbone network parameters in the large language model, and adjusting a preference mixing module to obtain synthesized preference data; determining a preference alignment algorithm, and based on the synthesized preference data, combining the grouping soft loss with a loss function in the preference alignment algorithm to obtain a target loss function; and training the big language model based on the obtained target loss function by using a collected data set containing multiple preferences to obtain a big language model with balanced preferences. The method is also suitable for the fields of hardware computing power enhancement and algorithm improvement.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model adaptive multi-preference alignment method and system based on a hybrid expert architecture. Background Art

[0002] The development of large language models is an important progress in the field of artificial intelligence in recent years. With the increase in hardware computing power and the improvement of algorithms, researchers have developed larger and larger neural network models with more parameters. These large models perform well in tasks such as natural language processing (NLP) and computer vision (CV), providing a powerful tool for solving complex problems. Large models require alignment training to align large models with human preferences. However, human preferences are difficult to generalize and need to be classified into multiple preferences for description. This multi-preference description method has the problem of conflicts between preferences. Although traditional preference alignment methods can perform well on one preference, when extended to multi-preference alignment, these methods treat all preferences as a unified preference and train them uniformly, resulting in problems such as preferences cannot be obtained at the same time. When one preference performs well, it will harm the performance of other preferences.

[0003] Subsequently, researchers developed a variety of preference mixing schemes, including reward model fusion before training and manual data screening and filtering; using EM algorithm fusion during training, training multiple preference reward models simultaneously, and improving algorithms to integrate multiple reward models; model fusion after training and parameter editing, etc.

[0004] However, these solutions usually require a lot of manual participation, with the model or data processing personnel specifying the preference boundaries for the reward model or the large model being trained, and cannot be adaptively mixed. At the same time, the high additional training overhead makes these methods only remain in the experimental stage and cannot be implemented. Therefore, a multi-preference hybrid method with adaptive capabilities is needed to promote the implementation of multi-preference research. Summary of the invention

[0005] The present invention aims to solve the problem that multiple preference mixing schemes in the prior art usually require a lot of manual participation, and the model or data processing personnel specify the preference boundaries for the reward model or the trained large model, and cannot be adaptively mixed. At the same time, the high additional training cost makes these methods only stay in the experimental stage and cannot be implemented.

[0006] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] Solution 1: The present invention proposes a large language model adaptive multi-preference alignment method based on a hybrid expert architecture, the method comprising the following steps:

[0008] Step 1: Add the designed preference mixture module based on the hybrid expert architecture to the large language model;

[0009] Step 2: Freeze the backbone network parameters in the large language model, adjust the preference mixing module, and obtain synthesized preference data;

[0010] Step 3: Determine the preference alignment algorithm, and based on the synthesized preference data, combine the grouping soft loss with the loss function in the preference alignment algorithm to obtain the target loss function;

[0011] Step 4: Use a data set collected containing multiple preferences, train the large language model based on the obtained target loss function, and obtain a large language model with balanced preferences.

[0012] Furthermore, a preferred implementation is provided, wherein the preference mixing module in step 1 includes multiple low-rank adaptive modules and a router.

[0013] Furthermore, a preferred implementation is provided, in step 3, based on the synthesized preference data, the group soft loss is combined with the preference learning algorithm to obtain the target loss function as follows:

[0014] Step 3.1: Determine the number of experts assigned to each preference, expressed as:

[0015] T i =(h i ,s i )

[0016] Among them, i is the preference of the allocation expert, h i ,s i Indicates the start and end corner marks of the assigned expert range;

[0017] Step 3.2: Calculate the ideal preference distribution D for preference i i , the calculation method is as follows:

[0018] D i ={0}×K

[0019] sc d =sc / (s i -h i )

[0020] D i [h i :s i ]=sc d

[0021] Among them, sc∈(0,1] represents the soft constraint coefficient, which indicates the degree of attention paid by the preference mixing module to preference i, and h i ,s i The range of experts determined in step 3.1;

[0022] Step 3.3, for a piece of training data from preference i, its loss is expressed as:

[0023] L egs =KL(D i ||w router [:K])

[0024] Among them, w router is the calculated router activation value; only the first K activation values ​​are used to participate in the loss calculation, and the activation value corresponding to the empty expert does not participate in the loss function calculation;

[0025] Step 3.4: Combined with the preference learning algorithm, the large language model is adjusted, and the total loss function is expressed as:

[0026] L=L XXX +βL egs

[0027] Among them, L XXX represents the loss of the selected preference learning algorithm and β is the scaling factor.

[0028] Furthermore, a preferred implementation is provided, wherein the preference alignment algorithm described in step three is implemented by any one of the preference learning algorithms including DPO, PPO, and KTO.

[0029] Further, a preferred implementation is provided, in which the method of using the target loss function obtained in the DPO reinforcement learning algorithm in step 3 is:

[0030]

[0031] where y w With y l The selection response and rejection response of the preference data pairs in the multi-preference dataset are respectively DPO is a hyperparameter,

[0032] The final objective loss function is expressed as:

[0033] L=L DPO +βL egs .

[0034] Furthermore, a preferred implementation is provided, in step 4, the large language model is trained based on the obtained target loss function, and the target loss function is optimized using the ADAM algorithm to obtain the final model.

[0035] Furthermore, a preferred implementation is provided, in which a data set containing multiple preferences is collected in step 4, and a parallel processing method is used to train a large language model based on the obtained target loss function to accelerate the training process.

[0036] Solution 2: A large language model adaptive multi-preference alignment system based on a hybrid expert architecture, the system comprising:

[0037] Preference mixture module, which is used to add a designed preference mixture module based on a mixture of experts architecture to the large language model;

[0038] An adjustment module, used to freeze the backbone network parameters in the large language model, adjust the preference mixing module, and obtain synthesized preference data;

[0039] The target loss function determination module is used to determine the preference alignment algorithm. Based on the synthesized preference data, the grouping soft loss is combined with the preference alignment algorithm to obtain the target loss function.

[0040] The preference-balanced large language model determination module is used to use a data set collected containing multiple preferences, train the large language model based on the obtained target loss function, and obtain a preference-balanced large language model.

[0041] Solution three: A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes any one of the methods described in Solution one.

[0042] Solution 4: A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of Solution 1.

[0043] The present invention is beneficial in that:

[0044] The invention discloses a large language model adaptive multi-preference alignment method based on a hybrid expert architecture, which aims to improve the ability of the large model to learn and adapt to multiple conflicting human preferences during post-training, while reducing the training cost. Through this method, multiple preferences can be more efficiently mixed during the large model training process, so that the output results of the large model are more in line with human preferences.

[0045] The method described in the present invention explicitly models the learning process and storage form of multiple preference knowledge by adding a preference mixing module based on a hybrid expert architecture to a large language model. Through the router in the hybrid expert architecture, the preference mixing module can save the parameter changes generated by multiple conflicting preferences during the training process, and adaptively activate different parameter changes during the large model reasoning process. The preference mixing module is a parameter-efficient fine-tuning module that can significantly reduce the training cost of the model. Even if data that has not appeared in the training process appears, there is no need to manually specify the preference type. In addition, this method can also be expanded to any alignment algorithm and number of preferences, or other methods can be used to obtain preference data.

[0046] The present invention is also applicable to the field of hardware computing capability enhancement and algorithm improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of a large language model adaptive multi-preference alignment method based on a hybrid expert architecture as described in implementation mode 1.

[0048] Figure 2 This is a schematic diagram of the process of the preference mixing module described in Implementation Method 1.

[0049] Figure 3 This is a schematic diagram of the structure of the preference mixing module based on the hybrid expert architecture described in the first implementation mode.

[0050] Figure 4 This is a schematic diagram of the activation of the big language model at different layers and by different experts when the empathy instruction is input into the big model as described in the eleventh implementation mode.

[0051] Figure 5 This is a schematic diagram of the activation of the large language model at different layers and among different experts when helpful instructions are input into the large model as described in the eleventh embodiment.

[0052] Figure 6 This is a schematic diagram of the large model for inputting harmless instructions as described in Implementation Method 11, and the activation of different layers and different experts in the large model. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the implementation methods of the present application clearer, the technical solutions in the implementation methods of the present application will be clearly and completely described below in conjunction with the drawings in the implementation methods of the present application. Obviously, the described implementation methods are only part of the implementation methods of the present application, not all of the implementation methods.

[0054] Implementation 1: This implementation provides a large language model adaptive multi-preference alignment method based on a hybrid expert architecture, the method comprising the following steps:

[0055] Step 1: Add the designed preference mixture module based on the hybrid expert architecture to the large language model;

[0056] Step 2: Freeze the backbone network parameters in the large language model and only adjust the preference mixing module to obtain synthesized preference data;

[0057] Step 3: Determine the preference alignment algorithm, and based on the synthesized preference data, combine the grouping soft loss with the loss function in the preference learning algorithm to obtain the target loss function;

[0058] Step 4: Use a data set collected containing multiple preferences, train the large language model based on the obtained target loss function, and obtain a large language model with balanced preferences.

[0059] Implementation method 2: This implementation method is a further limitation of the large language model adaptive multi-preference alignment method based on a hybrid expert architecture described in implementation method 1. The preference mixing module in step 1 includes multiple low-rank adaptive modules and a router.

[0060] Implementation method 3: This implementation method is a further limitation of the large language model adaptive multi-preference alignment method based on the hybrid expert architecture described in implementation method 1. In step 3, based on the synthesized preference data, the group soft loss is combined with the preference learning algorithm to obtain the target loss function as follows:

[0061] Step 3.1: Determine the number of experts assigned to each preference, expressed as:

[0062] T i =(h i ,s i )

[0063] Among them, i is the preference of the allocation expert, h i ,s i Indicates the start and end corner marks of the assigned expert range;

[0064] Step 3.2: Calculate the ideal preference distribution D for preference i i , the calculation method is as follows:

[0065] D i ={0}×K

[0066] sc d =sc / (s i -h i )

[0067] D i [h i :s i ]=sc d

[0068] Among them, sc∈(0,1] represents the soft constraint coefficient, which indicates the degree of attention paid by the preference mixing module to preference i, and h i ,s i The range of experts determined in step 3.1;

[0069] Step 3.3, for a piece of training data from preference i, its loss is expressed as:

[0070] L egs =KL(D i ||w router [:K])

[0071] Among them, w router is the calculated router activation value; only the first K activation values ​​are used to participate in the loss calculation, and the activation value corresponding to the empty expert does not participate in the loss function calculation;

[0072] Step 3.4: Combined with the preference learning algorithm, the large language model is adjusted, and the total loss function is expressed as:

[0073] L=L XXX +βL egs

[0074] Among them, L XXX represents the loss of the selected preference learning algorithm and β is the scaling factor.

[0075] Implementation method 4: This implementation method is a further limitation of the large language model adaptive multi-preference alignment method based on a hybrid expert architecture described in implementation method 3. The preference alignment algorithm described in step 3 is implemented by any one of the preference learning algorithms including DPO, PPO, and KTO.

[0076] Implementation mode 5: This implementation mode is a further limitation of the large language model adaptive multi-preference alignment method based on the hybrid expert architecture described in Implementation mode 4. The method of using the target loss function obtained in the DPO reinforcement learning algorithm in step 3 is:

[0077]

[0078] where y w With y l The selection response and rejection response of the preference data pairs in the multi-preference dataset are respectively DPO is a hyperparameter,

[0079] The final objective loss function is expressed as:

[0080] L=L DPO +βL egs .

[0081] Implementation method six: This implementation method is a further limitation of the large language model adaptive multi-preference alignment method based on the hybrid expert architecture described in implementation method one. In step 4, the large language model is trained based on the obtained target loss function, and the target loss function is optimized using the ADAM algorithm to obtain the final model.

[0082] Implementation method seven: This implementation method is a further limitation of the large language model adaptive multi-preference alignment method based on the hybrid expert architecture described in implementation method one. In step 4, a data set containing multiple preferences is collected, and a parallel processing method is used to train the large language model based on the obtained target loss function to accelerate the training process.

[0083] Embodiment 8: This embodiment proposes a large language model adaptive multi-preference alignment system based on a hybrid expert architecture, the system comprising:

[0084] Preference mixture module, which is used to add a designed preference mixture module based on a mixture of experts architecture to the large language model;

[0085] An adjustment module is used to freeze the backbone network parameters in the large language model and only adjust the preference mixing module to obtain synthesized preference data;

[0086] The target loss function determination module is used to determine the preference alignment algorithm. Based on the synthesized preference data, the grouping soft loss is combined with the preference learning algorithm to obtain the target loss function.

[0087] The preference-balanced large language model determination module is used to use a data set collected containing multiple preferences, train the large language model based on the obtained target loss function, and obtain a preference-balanced large language model.

[0088] Embodiment 9. This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in any one of embodiments 1 to 7.

[0089] Embodiment 10: This embodiment proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method described in any one of embodiments 1 to 7 are implemented.

[0090] Implementation eleven: This implementation provides an example, which is used to explain the above implementations one to eight. The specific example is as follows:

[0091] See also Figures 1 to 6To illustrate this embodiment, the method described in this embodiment includes the following steps:

[0092] First, we need to add a specially designed preference mixture module based on a hybrid expert architecture to the fine-tuned large language model, freeze the parameters of the large model backbone network, and only fine-tune the preference mixture module.

[0093] Secondly, select a preference alignment algorithm, such as DPO, PPO, etc. Combine the group soft loss with the loss function such as the preference alignment algorithm to obtain the target loss function.

[0094] Finally, using a dataset containing multiple preferences, we try to train the model using the report target loss function to obtain a large language model with balanced preferences.

[0095] Since the Transformer architecture commonly used in large language models does not have the ability to adapt to multiple preferences, and fine-tuning all parameters is too resource-intensive, a preference mixing module based on a hybrid expert architecture is designed in step 1, namely, the preference mixing module. Figure 2 As shown in the figure, before training, the preference mixing module needs to be added to the second linear layer of the feedforward layer of the large language model, and the other parameters of the model are frozen during the training process. In order to reduce the training cost, the preference mixing module parallelizes the calculation process. This step is one of the pre-steps, which aims to add the ability to express multiple preference mixtures to the large language model. The large language model with the added preference mixing module has low training cost and the ability to automatically mix multiple conflicting preferences.

[0096] like Figure 3 As shown in the figure, the preference mixing module consists of multiple low-rank adaptive modules (Low-RankAdaptor, LoRA) and a router. Multiple LoRAs are divided into several groups, and each group of LoRAs acts as an expert and focuses on a preference. The router is a linear layer that maps the context hidden state to the LoRA activation value. The role of LoRA is to independently fit different human preferences. The role of the router is to determine the activation value for each LoRA. The activation value determines the extent to which LoRA affects the decoding process, and then determines the degree of activation of each human preference in the decoding process. The calculation process of the preference mixing module is:

[0097]

[0098] w router =softmax(W r x+b)

[0099] Where K represents the number of LoRA experts, x is the context hidden state, W0 is the linear layer parameter with PMoL added, and W r and b are the parameters of the linear layer of the router, wrouter is the activation value calculated by the router, with a length of K+1. i Represents a LoRA expert, K ​​in total. The preference mixture module organizes multiple LoRAs in the form of a mixture of experts (MoE). The softmax function ensures that the sum of the activation values ​​of all experts is 1. The number of activation values ​​obtained by the router is one more than the number of LoRAs. This additional activation value represents an empty expert, indicating the proportion of content that is not related to preferences. The number of LoRA experts K and the hyperparameters of each LoRA are hyperparameters that need to be specified.

[0100] Grouped soft loss is one of the keys to large language models learning human preferences. It is necessary to select a suitable preference learning algorithm based on the acceptable training calculation cost, use the preference data synthesized in the previous step, and add an auxiliary loss function to the loss function: grouped soft loss, to help the preference mixing module better fit a variety of human preferences.

[0101] Step 3.1: Determine the number of experts assigned to each preference, expressed as:

[0102] T i =(h i ,s i )

[0103] Among them, i is the preference of the allocation expert, h i ,s i Indicates the start and end subscripts of the expert range to be assigned.

[0104] Step 3.2: Calculate the ideal preference distribution D for preference i i , the calculation method is as follows:

[0105] D i ={0}×K

[0106] sc d =sc / (s i -h i )

[0107] D i [h i :s i ]=sc d

[0108] Among them, sc∈(0,1] represents the soft constraint coefficient, which represents the degree of attention required for the preference mixing module to the preference i, and is a hyperparameter that needs to be specified. i ,s i is the range of experts determined in step 1.

[0109] Step 3.3, for a piece of training data from preference i, its loss is expressed as:

[0110] L egs =KL(D i ||w router [:K])

[0111] Among them, w router is the router activation value. Only the first K activation values ​​are used to participate in the loss calculation, and the activation value corresponding to the empty expert does not participate in the loss function calculation.

[0112] Step 3.4: Combined with the preference learning algorithm, fine-tune the large model, and the total loss is expressed as:

[0113] L=L XXX +βL egs

[0114] Among them, L XXX represents the loss of the selected preference learning algorithm, and β is the scaling factor, which is a hyperparameter that needs to be specified.

[0115] During the training process, the collected multi-preference datasets, the selected preference learning algorithm, and the large model with the preference mixing module added need to be used for training to obtain the final model. The model parameters need to be frozen during the training process, and only the preference mixing module part needs to be fine-tuned. In order to speed up the training process, this design also parallelizes the above calculation process to speed up the reasoning and training process.

[0116] The multi-preference alignment algorithm can adopt various reinforcement learning algorithms such as PPO, DPO, KTO, etc. We give an example based on DPO implementation.

[0117] The loss of DPO is expressed as:

[0118]

[0119] where y w With y l The selection and rejection responses in the preference data pairs from the multi-preference dataset, respectively. DPO is a hyperparameter that controls the training intensity of the DPO algorithm on the preferred data pairs. The final training loss is expressed as:

[0120] L=L DPO +βL egs

[0121] During the training process, the ADAM algorithm is used to optimize the above losses to obtain the final model.

[0122] Embodiment 1, this embodiment is as follows Figure 1The process shown is carried out, and Google's Gemma2-2B is selected as the basic large model. First, it is necessary to add a preference mixing module to the second linear layer in the feedforward layer of the large model to learn multiple human preferences while freezing other parameters of the model. Then, this embodiment selects three human preferences, helpfulness, harmlessness, and empathy as the three preferences to be mixed. The first two have open source data sets available. The third preference, empathy, uses the open source large model sampling as a negative example to synthesize preference data. Finally, this embodiment assigns 4 experts to each of the three preferences, and one empty expert, a total of 13 experts. The DPO algorithm is used as the preference learning algorithm to complete the final training process.

[0123] Figures 3 to 5 The preference mixing module activates different experts when using questions related to three preferences. The preference mixing module always activates experts related to the preferences of the input data. This shows that the scheme has the ability to automatically mix and select preferences.

[0124] Table 1 compares the results with the full parameter fine-tuning and LoRA fine-tuning methods. The reward model is trained with a preferred preference data. The large model is evaluated using GPT-4o, with a score range of 0-5. The higher the score, the higher the degree of conformity between the model and the preference.

[0125]

[0126] The comparison of evaluation results with other baseline methods is shown in Table 1. When using three preferences, the present invention achieved the highest average score when using the reward model and GPT-4o evaluation, which shows that the present invention has better preference mixing ability than the baseline.

[0127] Table 2 compares the training cost with the full parameter fine-tuning and LoRA fine-tuning methods. Calculated as time (hours) multiplied by the amount of graphics memory consumed (GB) multiplied by the number of graphics cards used.

[0128] method Training cost Fine-tuning all parameters 58.63 LoRA 11.30 This method 10.21

[0129] The training cost comparison results with other baseline methods are shown in Table 2. When using 30,000 data, the training cost of this method is slightly higher than that of the LoRA method, but it achieves results that significantly exceed the LoRA method.

[0130] Figure 1Any process or method description in the flowchart described in or otherwise described herein can be understood as representing a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a custom logic function or process, and the scope of the preferred embodiment of the present invention includes other implementations, in which the functions may be performed in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present invention. The logic and / or steps represented in the flowchart or otherwise described herein illustrate the possible implementation architecture, functions and operations of the apparatus and methods according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It is also noted that each block in the block diagram or flowchart, and combinations of blocks in the block diagram or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions. For example, what may be considered as an ordered list of executable instructions for implementing a logical function may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions in an instruction execution system, apparatus, or device).

[0131] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0132] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A large language model adaptive multi-preference alignment method based on a hybrid expert architecture, characterized in that: The method comprises the following steps: Step 1: Add the designed preference mixture module based on the hybrid expert architecture to the large language model; Step 2: Freeze the backbone network parameters in the large language model, adjust the preference mixing module, and obtain synthesized preference data; Step 3: Determine the preference alignment algorithm, and based on the synthesized preference data, combine the grouping soft loss with the loss function in the preference alignment algorithm to obtain the target loss function; Step 4: Use a data set collected containing multiple preferences, train the large language model based on the obtained target loss function, and obtain a large language model with balanced preferences.

2. The large language model adaptive multi-preference alignment method based on hybrid expert architecture according to claim 1 is characterized in that: The preference mixing module in step 1 includes multiple low-rank adaptive modules and a router.

3. The large language model adaptive multi-preference alignment method based on hybrid expert architecture according to claim 1 is characterized in that: In step 3, based on the synthesized preference data, the group soft loss is combined with the loss function in the preference learning algorithm to obtain the target loss function as follows: Step 3.1: Determine the number of experts assigned to each preference, expressed as: T i =(h i ,s i ) Among them, i is the preference of the allocation expert, h i ,s i Indicates the start and end corner marks of the assigned expert range; Step 3.2: Calculate the ideal preference distribution D for preference i i , the calculation method is as follows: D i ={0}×K sc d =sc / (s i -h i ) D i [h i :s i ]=sc d Among them, sc∈(0,1] represents the soft constraint coefficient, which indicates the degree of attention paid by the preference mixing module to preference i, and h i ,s i The range of experts determined in step 3.1; Step 3.3, for a piece of training data from preference i, its loss is expressed as: L egs =KL(D i ||w router [:K]) Among them, w router is the calculated router activation value; only the first K activation values ​​are used to participate in the loss calculation, and the activation value corresponding to the empty expert does not participate in the loss function calculation; Step 3.4: Combined with the preference learning algorithm, the large language model is adjusted, and the total loss function is expressed as: L=L XXX +βL egs Among them, L XXX represents the loss of the selected preference learning algorithm and β is the scaling factor.

4. The large language model adaptive multi-preference alignment method based on hybrid expert architecture according to claim 1, characterized in that: The preference alignment algorithm described in step 3 is implemented by any preference learning algorithm including DPO, PPO, and KTO.

5. The large language model adaptive multi-preference alignment method based on hybrid expert architecture according to claim 4 is characterized in that: The method of using the target loss function obtained in the DPO reinforcement learning algorithm in step 3 is: where y w With y l The selection response and rejection response of the preference data pairs in the multi-preference dataset are respectively DPO is a hyperparameter, The final objective loss function is expressed as: L=L DPO +βL egs 。 6. The large language model adaptive multi-preference alignment method based on hybrid expert architecture according to claim 1, characterized in that: In step 4, the large language model is trained based on the obtained target loss function, and the target loss function is optimized using the ADAM algorithm to obtain the final model.

7. The large language model adaptive multi-preference alignment method based on hybrid expert architecture according to claim 1, characterized in that: In step 4, a data set containing multiple preferences is collected, and a parallel processing method is used to train the large language model based on the obtained target loss function to accelerate the training process.

8. A large language model adaptive multi-preference alignment system based on a hybrid expert architecture, characterized in that: The system comprises: Preference mixture module, which is used to add a designed preference mixture module based on a mixture of experts architecture to the large language model; An adjustment module, used to freeze the backbone network parameters in the large language model, adjust the preference mixing module, and obtain synthesized preference data; The target loss function determination module is used to determine the preference alignment algorithm. Based on the synthesized preference data, the grouping soft loss is combined with the preference alignment algorithm to obtain the target loss function. The preference-balanced large language model determination module is used to use a data set collected containing multiple preferences, train the large language model based on the obtained target loss function, and obtain a preference-balanced large language model.

9. A computer device comprising a memory and a processor, characterized in that A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images