Large model multi-preference alignment method and device based on hierarchical hybrid expert model

Abstract

The invention provides a large model multi-preference alignment method and device based on a hierarchical hybrid expert model, and relates to the technical field of natural language processing. The method comprises the following steps: acquiring a pre-trained single-target fine tuning model; extracting a target vector of each single-target strategy in the model, decomposing the target vector through a task vector singular value decomposition method, and generating a low-rank adapter as a LoRA expert of each single target; a PCB-merging and Free-merging fusion model is adopted for processing, and a multi-target LoRA expert is obtained; generating a linear routing layer, and constructing a reward loss function; optimizing the loss function by adopting mirror gradient descent and smooth Chebyshev scaling to obtain a multi-target routing expert; designing a weight router; constructing a hierarchical hybrid expert model according to the multi-target LoRA experts, the multi-target routing experts and the weight routers; and inputting the obtained prompt words and preference vectors input by the user into the hierarchical hybrid expert model, and outputting preferences conforming to the user. According to the invention, the reasoning efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a large-model multi-preference alignment method and device based on a hierarchical hybrid expert model. Background Art

[0002] With the rapid development of artificial intelligence, generative large language models (LLMs) have made significant progress in the field of natural language processing with their powerful language understanding and generation capabilities. LLMs models are pre-trained on massive text data to learn speech patterns and rules, and can generate natural, fluent and creative texts. However, in practical applications, LLMs models often need to be adjusted according to the personalized needs of users and different usage scenarios to meet diverse task requirements. When user preferences involve multiple target dimensions, single target alignment methods often cannot balance conflicting needs, so the model is difficult to cope with diverse user needs and preferences. Therefore, in large model alignment, how to balance multiple goals and meet the preferences of different users at the same time has become a major challenge at present.

[0003] Most traditional multi-target alignment methods are based on reinforcement learning with human feedback or direct preference optimization techniques, which linearly weight different targets into a single target according to given preferences for RL optimization one by one. At present, a multi-target decoding method and a multi-target parameter fusion method during reasoning have been proposed, which are more flexible and efficient than reinforcement learning for target alignment methods. At present, a multi-target correction model or a reward model that guides decoding is used to solve the multi-target alignment problem. However, the above methods still face various shortcomings. The above methods have long reasoning time, high storage cost, and difficulty in fully handling conflicts and dynamic changes between different targets. Therefore, how to balance the trade-offs between multiple targets while maintaining efficiency is still a difficult problem that needs to be solved urgently in current technology. Summary of the invention

[0004] In order to solve the technical problems of high training cost, large model inference time and system complexity in the prior art, and the inability to capture the connection between objectives when facing multiple objective functions that may conflict with different objectives, the embodiment of the present invention provides a large model multi-preference alignment method and device based on a hierarchical hybrid expert model. The technical solution is as follows:

[0005] On the one hand, a large model multi-preference alignment method based on a hierarchical hybrid expert model is provided, the method is implemented by a large model multi-preference alignment device based on a hierarchical hybrid expert model, and the method includes:

[0006] S1. Obtain a pre-trained single-target fine-tuned large language model; extract the target vector of each single-target strategy in the model, decompose the target vector of each single target by task vector singular value decomposition, and generate a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, use PCB-merging and Free-merging fusion methods to construct a multi-target LoRA expert model;

[0007] S2. Generate a linear routing layer and construct a reward loss function corresponding to the linear routing layer; optimize the loss function using a mirror gradient descent method and a smoothed Chebyshev scalarization method to obtain a multi-objective routing expert;

[0008] S3. Design weighted routers; construct a hierarchical hybrid expert model based on the multi-objective LoRA expert model, multi-objective routing experts and weighted routers;

[0009] S4. Obtain the prompt word and preference vector input by the user, input the prompt word and preference vector input by the user into the hybrid expert model for alignment processing, and output a signal that meets the user's preference.

[0010] Optionally, the S1 extracts the target vector of each single-target strategy in the model, performs sparse processing on the target vector by task vector singular value decomposition method, and generates a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, the PCB-merging fusion model and the Free-merging fusion model are used for processing to obtain a multi-target LoRA expert model, including:

[0011] S11, defining parameters of the pre-trained single-objective fine-tuned large language model; according to the parameters of the pre-trained single-objective fine-tuned large language model, extracting the target vector of each single-objective strategy in the model;

[0012] S12. Use the task vector singular value decomposition method to perform sparse processing, and obtain a low-rank adapter by decomposing the target vector of each single target; use the low-rank adapter as the LoRA expert of each single target;

[0013] S13. Set the preference vector, and use the PCB-merging fusion model and the Free-merging fusion model to process each single-target LoRA expert to obtain a multi-target LoRA expert model.

[0014] Optionally, the reward loss function of S2 is expressed by the following formula (1):

[0015] (1)

[0016] in, Reward signals representing N targets are preferred A linear weighted combination of Indicates hidden state; Represents the parameters of the pre-trained model; represents routing expert; Represents a hierarchical mixture of experts model; Represents all LoRA experts.

[0017] Optionally, the process of optimizing the loss function by using the mirror gradient descent method and the smoothed Chebyshev scalar quantization method in S2 is expressed by the following formula (2):

[0018] (2)

[0019] in, Indicate the reference point for each objective, i.e. the desired level of performance; represents the preference weight of the i-th target; Represents the trainable parameters of the model.

[0020] Optionally, the weight router is used to map the user's continuous preference vector to the nearest expert and assign a voting weight to each selected expert; wherein the nearest experts include: LoRA experts and routing experts;

[0021] The calculation process of the weighted router is expressed by the following formula (3):

[0022] (3)

[0023] in, Represents the preference vector given by the user; Indicates the numbers of the N experts closest to the preference vector given by the user.

[0024] Optionally, the step S4 inputs the prompt word and the preference vector input by the user into the hybrid expert model for alignment processing, and outputs a signal that meets the user's preference, including:

[0025] S41, inputting the prompt word and preference vector input by the user into the hierarchical hybrid expert model, and the weight router selects multiple routing experts to activate according to the preference vector, and assigns a corresponding weight to each routing expert;

[0026] S42, assigning a corresponding weight to each routing expert and inputting it into the multi-target routing expert, dynamically voting for each single-target LoRA expert according to the received hidden layer state input, assigning a corresponding weight to each routing expert, and weighted summing the voting values ​​of all routing experts to obtain the final voting summary result;

[0027] S43. Input the final voting summary result into the multi-objective LoRA expert model. Through the final voting summary result, dynamically activate the LoRA expert parameters, and combine the parameters of the pre-trained single-objective fine-tuned large language model to generate a preference that meets the user.

[0028] Optionally, the process of the output of S4 conforming to the user's preference is expressed by the following formula (4):

[0029] (4)

[0030] in, Indicate in preference The final output for a given input x; represents the raw output of the pre-trained parameters; Indicates the voting weight given to each LoRA expert; The B matrix represents each LoRA expert; A matrix representing each LoRA expert; Represents the hidden state of the input.

[0031] On the other hand, a large model multi-preference alignment device based on a hierarchical hybrid expert model is provided, and the device is applied to a large model multi-preference alignment method based on a hierarchical hybrid expert model, and the device comprises:

[0032] The first construction unit is used to obtain a pre-trained single-target fine-tuned large language model; extract the target vector of each single-target strategy in the model, perform sparse processing on the target vector by the task vector singular value decomposition method, and generate a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, the PCB-merging fusion model and the Free-merging fusion model are used for processing to obtain a multi-target LoRA expert model;

[0033] An acquisition unit is used to generate a linear routing layer and construct a reward loss function corresponding to the linear routing layer; the loss function is optimized by using a mirror gradient descent method and a smoothed Chebyshev scalarization method to obtain a multi-objective routing expert;

[0034] The second construction unit is used to design a weighted router; a hierarchical hybrid expert model is constructed based on the multi-objective LoRA expert model, the multi-objective routing experts and the weighted router;

[0035] The output unit is used to obtain the prompt word and preference vector input by the user, input the prompt word and preference vector input by the user into the hybrid expert model for alignment processing, and output in accordance with the user's preference.

[0036] Optionally, the first construction unit is used to:

[0037] Defining parameters of the pre-trained single-objective fine-tuned large language model; extracting a target vector for each single-objective strategy in the model based on the parameters of the pre-trained single-objective fine-tuned large language model;

[0038] The task vector singular value decomposition method is used for sparse processing. By decomposing the target vector of each single target, a low-rank adapter is obtained; the low-rank adapter is used as the LoRA expert of each single target;

[0039] The preference vector is set, and the PCB-merging fusion model and the Free-merging fusion model are used to process the LoRA experts of each single target to obtain the multi-target LoRA expert model.

[0040] Optionally, the reward loss function is expressed by the following formula (1):

[0041] (1)

[0042] in, Reward signals representing N targets are preferred A linear weighted combination of Indicates hidden state; Represents the parameters of the pre-trained model; represents routing expert; represents a hierarchical mixture of experts model; Represents all LoRA experts.

[0043] Optionally, the process of optimizing the loss function by using the mirror gradient descent method and the smoothed Chebyshev scalar quantization method is expressed by the following formula (2):

[0044] (2)

[0045] in, Indicate the reference point for each objective, i.e. the desired level of performance; represents the preference weight of the i-th target; Represents the trainable parameters of the model.

[0046] Optionally, the weight router is used to map the user's continuous preference vector to the nearest expert and assign a voting weight to each selected expert; wherein the nearest experts include: LoRA experts and routing experts;

[0047] The calculation process of the weighted router is expressed by the following formula (3):

[0048] (3)

[0049] in, Represents the preference vector given by the user; Indicates the numbers of the N experts closest to the preference vector given by the user.

[0050] Optionally, the output unit is used to:

[0051] The prompt words and preference vectors input by the user are input into the hierarchical hybrid expert model. The weighted router selects multiple routing experts to activate according to the preference vectors and assigns corresponding weights to each routing expert.

[0052] Assign corresponding weights to each routing expert and input them into the multi-target routing expert. According to the received hidden layer state input, dynamically vote for each single-target LoRA expert. According to the corresponding weights assigned to each routing expert, the voting values ​​of all routing experts are weighted summed to obtain the final voting summary result.

[0053] The final voting summary results are input into the multi-objective LoRA expert model. Through the final voting summary results, the LoRA expert parameters are dynamically activated, and the parameters of the pre-trained single-objective fine-tuned large language model are combined to generate a model that meets the user's preferences.

[0054] Optionally, the process of outputting the information in accordance with the user's preference is expressed by the following formula (4):

[0055] (4)

[0056] in, Indicate in preference The final output for a given input x; Represents the raw output of the pre-trained parameters; Indicates the voting weight given to each LoRA expert; The B matrix represents each LoRA expert; A matrix representing each LoRA expert; Represents the hidden state of the input.

[0057] On the other hand, a large-model multi-preference alignment device based on a hierarchical hybrid expert model is provided, and the large-model multi-preference alignment device based on a hierarchical hybrid expert model comprises: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the large-model multi-preference alignment methods based on the hierarchical hybrid expert model is implemented.

[0058] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned large-model multi-preference alignment methods based on hierarchical hybrid expert models.

[0059] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0060] The embodiment of the present invention first obtains a pre-trained single-target fine-tuned large language model; extracts the target vector of each single-target strategy in the model, decomposes the target vector of each single target by using the task vector singular value decomposition method, and generates a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, a multi-target LoRA expert model is constructed by using the PCB-merging and Free-merging fusion methods; a linear routing layer is generated, and a reward loss function corresponding to the linear routing layer is constructed; the loss function is optimized by using the mirror gradient descent method and the smoothed Chebyshev scalarization method to obtain the routing expert of the multi-target; a weight router is designed; a hierarchical hybrid expert model is constructed according to the multi-target LoRA expert model, the multi-target routing expert and the weight router; the prompt word and preference vector input by the user are obtained, the prompt word and the preference vector input by the user are input into the hierarchical hybrid expert model for alignment processing, and the output meets the user's preference.

[0061] The embodiment of the present invention firstly generates a low-rank adapter to construct a multi-target LoRA expert, and reduces the parameter scale of the model by designing a routing expert, thereby reducing the storage and computing costs; secondly, the embodiment of the present invention remains frozen during reasoning, and only a small amount of training is required for the routing expert, thereby reducing the training cost; thirdly, the embodiment of the present invention only activates a small number of expert models during reasoning, thereby reducing the computing overhead and improving the reasoning efficiency; finally, through the hierarchical expert model design, it can dynamically adapt to different user preferences and realize the optimal Pareto frontier of multi-target alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 It is a flow chart of a large model multi-preference alignment method based on a hierarchical hybrid expert model provided by an embodiment of the present invention;

[0064] Figure 2It is a structural schematic diagram of a hierarchical hybrid expert framework provided by an embodiment of the present invention;

[0065] Figure 3 It is an overall flow chart of a hierarchical hybrid expert model provided by an embodiment of the present invention;

[0066] Figure 4 It is a block diagram of a large model multi-preference alignment device based on a hierarchical hybrid expert model provided by an embodiment of the present invention;

[0067] Figure 5 It is a structural schematic diagram of a large model multi-preference alignment device based on a hierarchical hybrid expert model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0069] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0070] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0071] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0072] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0073] The embodiment of the present invention provides a large model multi-preference alignment method based on a hierarchical hybrid expert model, which can be implemented by a large model multi-preference alignment device based on a hierarchical hybrid expert model, and the large model multi-preference alignment device based on a hierarchical hybrid expert model can be a terminal or a server. Figure 1 The flowchart of the large model multi-preference alignment method based on the hierarchical hybrid expert model is shown. The processing flow of the method may include the following steps:

[0074] S1. Obtain a pre-trained single-target fine-tuned large language model; extract the target vector of each single-target strategy in the model, perform sparse processing on the target vector through the task vector singular value decomposition method, and generate a low-rank adapter as the LoRA expert for each single target; according to the LoRA expert of each target, use the PCB-merging fusion model and the Free-merging fusion model for processing to obtain a multi-target LoRA expert model.

[0075] Among them, the multi-objective LoRA expert model is the first-level basic component of the hierarchical hybrid expert model, and each target LoRA expert is responsible for dealing with the alignment problem of a single preference.

[0076] Optionally, the specific implementation process of S1 includes S11-S13:

[0077] S11, defining parameters of the pre-trained single-objective fine-tuned large language model; extracting the target vector of each single-objective strategy in the model according to the parameters of the pre-trained single-objective fine-tuned large language model;

[0078] Among them, each single-objective strategy is obtained by fine-tuning the objective corresponding to each strategy.

[0079] In a feasible implementation, each single-target fine-tuning model is initialized with a same set of pre-trained parameters, and is fine-tuned on its respective corresponding targets to obtain the parameters of the pre-trained single-target fine-tuned large language model.

[0080] In a feasible implementation, the process of extracting the target vector of each single-target strategy is expressed by the following formula (1):

[0081] (1)

[0082] in, represents the goal vector for each single-goal policy; represents the fine-tuned parameters; Represents the parameters of the pre-trained model.

[0083] Among them, the ability of the model on a single target can be captured by extracting the target vector.

[0084] S12. Use the task vector singular value decomposition method to perform sparse processing, and obtain a low-rank adapter by decomposing the target vector of each single target; use the low-rank adapter as the LoRA expert of each single target;

[0085] Each LoRA expert specializes in a specific goal. In order to enhance the preference coverage of LoRA experts, the present application samples preferences from the inside of the N-dimensional simplex to construct LoRA experts dedicated to the corresponding preferences.

[0086] S13. Set the preference vector, and use the PCB-merging fusion model and the Free-merging fusion model to process each single-target LoRA expert to obtain a multi-target LoRA expert model.

[0087] Among them, the PCB-merging fusion model uses internal balance to measure the importance of parameters in a single task, and uses external balance to evaluate the parameter similarity between different tasks, and selects significant parameters that are beneficial to all tasks.

[0088] Among them, the Free-merging fusion model uses the frequency domain information of parameters to effectively identify and filter specific parameters that are harmful to multiple tasks, thereby minimizing the impact of task conflicts on backbone parameters at the lowest cost.

[0089] Among them, the PCB-merging and Free-merging fusion methods can identify and delete redundant parameters within each task vector, and at the same time eliminate parameter competition between multiple task vectors; therefore, a set of model parameters can simultaneously improve the performance of multiple tasks without sacrificing the performance on a specific task; without introducing additional training, performance comparable to multi-task training can be obtained.

[0090] In a feasible implementation, target vectors of all targets and a specific preference vector are obtained, and the PCB-merging fusion model and the Free-merging fusion model are used for processing to obtain a multi-target LoRA expert for a specific preference.

[0091] In a feasible implementation, the present application extracts N single-target LoRA experts and constructs L multi-target LoRA experts; by linearly combining the outputs of multiple LoRA experts and appending the original outputs of pre-trained parameters, the final output of each module with mixed experts in the hierarchical mixed expert model is used to generate an output that meets user preferences.

[0092] S2. Generate a linear routing layer and construct a reward loss function corresponding to the linear routing layer; use the mirror gradient descent method and the smoothed Chebyshev scalarization method to optimize the loss function and obtain a multi-objective routing expert.

[0093] Among them, the multi-objective routing expert is the second-level intermediate layer component of the hierarchical hybrid expert model, which is used to dynamically select and combine multiple LoRA experts according to the user's preference vector to generate output that meets the user's preferences. This application uses a router that focuses on specific preferences to act as an expert, which can achieve a balance between parameter quantity and performance.

[0094] Among them, each routing expert is represented by a linear layer, the input is the hidden state, and the output is the voting weight of all LoRA experts, which is expressed by the following formula (2):

[0095] (2)

[0096] in, A first parameter representing a routing expert; A second parameter representing a routing expert; Represents the voting weight of all LoRA experts; Represents the hidden layer state.

[0097] The routing expert is located in each Transformer trainable module, and can make fine-grained voting choices at the Transformer module level and Transformer layer level. The routing expert can fully analyze the input hidden state and dynamically select and combine LoRA experts according to different inputs. Each routing expert is only responsible for a specific preference, and its optimization goal is to minimize the reward loss of its corresponding preference, so it is called an expert.

[0098] Optionally, the reward loss function of S2 is expressed by the following formula (3):

[0099] (3)

[0100] in, Reward signals representing N targets are preferred A linear weighted combination of Indicates hidden state; Represents the parameters of the pre-trained model; represents routing expert; Represents a hierarchical mixture of experts model; Represents all LoRA experts.

[0101] In order to capture the non-convex area of ​​the Pareto front and eliminate the inherent shortcomings of linear scalarization, this application adopts the Chebyshev scalarization method to calibrate the current strategy through a reference point for further optimization.

[0102] Optionally, the process of optimizing the loss function using the mirror gradient descent method and the smoothed Chebyshev scalar quantization method in S2 is expressed by the following formula (4):

[0103] (4)

[0104] in, Indicate the reference point for each objective, i.e. the desired level of performance; represents the preference weight of the i-th target; Represents the trainable parameters of the model.

[0105] S3. Design weighted routers; construct a hierarchical hybrid expert model based on the multi-objective LoRA expert model, multi-objective routing experts and weighted routers.

[0106] Among them, the weighted router is the third-level intermediate layer component of the hierarchical hybrid expert model. In order to ensure the computational efficiency during reasoning, for N target alignments, this application only selects to activate the N nearest experts including LoRA experts and routing experts.

[0107] Among them, the weight router is used to decompose the multi-objective Pareto optimization problem into multiple multi-objective sub-problems and map the user's continuous preference vector to a discrete single preference expert.

[0108] Optionally, a weight router is used to map the user's continuous preference vector to the nearest expert and assign a voting weight to each selected expert; wherein the nearest experts include: LoRA experts and routing experts;

[0109] The calculation process of the weighted router is expressed by the following formula (5):

[0110] (5)

[0111] in, Represents the preference vector given by the user; Indicates the numbers of the N experts closest to the preference vector given by the user.

[0112] In order to assign voting weights to each selected expert, this application uses the representative vector of the LoRA expert to divide the N-dimensional simplex into several areas, and uses the representative vector of the routing expert to further divide the area into smaller areas. Since the preference vector given by the user is regarded as a vector sampled from the N-dimensional simplex with a sum of 1, each given preference will fall into one of the fine-grained N-dimensional simplexes divided by the expert, which can be regarded as a finer-grained N-target alignment, in which N sub-targets are supported by N experts as basis vectors.

[0113] S4. Obtain the prompt word and preference vector input by the user, input the prompt word and preference vector input by the user into the hybrid expert model for alignment processing, and output in accordance with the user's preference.

[0114] in, Figure 2 It is a structural diagram of a hierarchical hybrid expert framework provided by an embodiment of the present invention; in a feasible implementation, the hierarchical hybrid expert framework includes: the zeroth level pre-trained original parameters, the first level LoRA experts, the second level routing experts and the third level weight router. Obtain the preference weight and hidden layer state input given by the user; input the preference weight given by the user into the third level weight router, activate the nearest N experts according to the preference weight, wherein the activated N experts form an N-dimensional simplex, determine the number of votes for each expert, obtain the voting score for all experts, and input the voting score for all experts into the second level routing expert; input the hidden layer input state into the second level routing expert, dynamically select and combine the voting weight of each LoRA expert according to the fine granularity of the input hidden layer state, and obtain the voting score for the LoRA expert; wherein each routing expert specializes in a pre-specific preference; input the voting score and hidden layer input state of the LoRA expert into the first level LoRA expert, and generate a specific user preference according to the input hidden layer state and the original parameters of the pre-trained model.

[0115] Optionally, the specific implementation process of S4 includes S41-S43:

[0116] S41, inputting the prompt word and preference vector input by the user into the hierarchical hybrid expert model, and the weight router selects multiple routing experts to activate according to the preference vector, and assigns a corresponding weight to each routing expert;

[0117] S42, assigning a corresponding weight to each routing expert and inputting it into the multi-target routing expert, dynamically voting for each single-target LoRA expert according to the received hidden layer state input, assigning a corresponding weight to each routing expert, and weighted summing the voting values ​​of all routing experts to obtain the final voting summary result;

[0118] S43. Input the final voting summary result into the multi-objective LoRA expert model. Through the final voting summary result, dynamically activate the LoRA expert parameters, and combine the parameters of the pre-trained single-objective fine-tuned large language model to generate a preference that meets the user.

[0119] Optionally, the process of the output of S4 conforming to the user's preference is expressed by the following formula (6):

[0120] (6)

[0121] in, Indicate in preference The final output for a given input x; represents the raw output of the pre-trained parameters; Indicates the voting weight given to each LoRA expert; The B matrix represents each LoRA expert; A matrix representing each LoRA expert; Represents the hidden state of the input.

[0122] in, Figure 3 It is an overall flow chart of a hierarchical hybrid expert model provided by an embodiment of the present invention; in a feasible implementation manner, taking three targets as an example, N trained single-target models are obtained, and the task vector singular value decomposition method is used for decomposition and sparse processing to obtain N single-target LoRA experts; the PCB-merging fusion method and the Free-merging fusion method are used for processing to obtain L multi-target LoRA experts; multi-target reinforcement learning is performed based on the L multi-target LoRA experts to obtain R multi-target routing experts; a hierarchical hybrid expert model is constructed based on N single-target LoRA experts, L multi-target LoRA experts and R multi-target routing experts; the prompt word input by the user and the preference weight given by the user are obtained; the prompt word input by the user and the preference weight given by the user are input into the hierarchical hybrid expert model for processing to obtain an aligned output, that is, in line with the user's preference.

[0123] Among them, in order to verify the effectiveness of this application, this application conducted experiments on multiple benchmark data sets, covering 14 different goals and 200 user preferences. The hierarchical hybrid expert framework proposed in this application outperforms 15 existing baseline methods in the multi-target alignment task. Specifically, the hierarchical hybrid expert has achieved the optimal Pareto front in multiple tasks, especially in the two-target, three-target and multi-target alignment tasks, the hierarchical hybrid expert model has outstanding performance. For example, in the Helpful Assistant task, the Pareto front of the hierarchical hybrid expert model is better than the existing baseline methods, and the best performance is achieved under multiple weight settings. In addition, the hierarchical hybrid expert model also performed well in the five-target alignment task, with an average score significantly higher than other methods. On the HelpSteer dataset, the average score of the hierarchical hybrid expert model reached 61.74, which is higher than the MOD and RiC baseline methods.

[0124] Among them, the hierarchical hybrid expert model proposed in this application only needs to store 7.64B of parameters, reducing storage and computing costs; the training parameters of the hierarchical hybrid expert model proposed in this application are only 8M, which is much lower than other methods, such as 0.64B of RiC and 0.16B of PAD, reducing training costs. During the reasoning stage of this application, the hierarchical hybrid expert framework only activates a small number of expert models, reducing computing overhead. The reasoning cost of the hierarchical hybrid expert model proposed in this application is only 1.23 times that of normal reasoning, which is much lower than other methods, such as 3.10 of MOD and 2.98 of PAD, improving reasoning efficiency.

[0125] The embodiment of the present invention first obtains a pre-trained single-target fine-tuned large language model; extracts the target vector of each single-target strategy in the model, decomposes the target vector of each single target by using the task vector singular value decomposition method, and generates a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, a multi-target LoRA expert model is constructed by using the PCB-merging and Free-merging fusion methods; a linear routing layer is generated, and a reward loss function corresponding to the linear routing layer is constructed; the loss function is optimized by using the mirror gradient descent method and the smoothed Chebyshev scalarization method to obtain the routing expert of the multi-target; a weight router is designed; a hierarchical hybrid expert model is constructed according to the multi-target LoRA expert model, the multi-target routing expert and the weight router; the prompt word and preference vector input by the user are obtained, the prompt word and the preference vector input by the user are input into the hierarchical hybrid expert model for alignment processing, and the output meets the user's preference.

[0126] This application decomposes the multi-objective alignment problem into a series of single-preference sub-problems and uses specialized experts to handle each sub-problem, which can effectively cover the entire Pareto frontier while maintaining low training overhead. This application can not only balance the trade-offs between different objectives, but also achieve superior Pareto optimal results on multiple benchmarks. This application uses a lightweight and parameter-efficient LoRA adapter as a specific target expert, reducing the number of trainable parameters and storage costs required. In addition, the routing expert mechanism further optimizes the model selection process, improves the reasoning efficiency, and achieves a cost-effective balance between the number of parameters, reasoning cost and performance. This application uses model sparsification and model fusion to extract LoRA experts without any training. At the same time, the hierarchical hybrid expert model proposed in this application can be dynamically balanced at multiple granular levels of model, target vector, Transformer layer and Transformer module. This application can flexibly adapt to diverse user preferences and provide more personalized output results. The three basic components in the hierarchical hybrid expert model of this application only focus on specific goals and preferences, so that the model can dynamically switch between different preferences. This application can insert new experts at any time to adapt to new goals, so as to better meet the diverse needs of user growth.

[0127] The embodiment of the present invention firstly generates a low-rank adapter to construct a multi-target LoRA expert, and reduces the parameter scale of the model by designing a routing expert, thereby reducing the storage and computing costs; secondly, the embodiment of the present invention remains frozen during reasoning, and only a small amount of training is required for the routing expert, thereby reducing the training cost; thirdly, the embodiment of the present invention only activates a small number of expert models during reasoning, thereby reducing the computing overhead and improving the reasoning efficiency; finally, through the hierarchical expert model design, it can dynamically adapt to different user preferences and realize the optimal Pareto frontier of multi-target alignment.

[0128] Figure 4 1 is a block diagram of a large model multi-preference alignment device based on a hierarchical hybrid expert model according to an exemplary embodiment. The device is used in a large model multi-preference alignment method based on a hierarchical hybrid expert model. Figure 4 The device includes a first construction unit 410, an acquisition unit 420, a second construction unit 430 and an output unit 440. Wherein:

[0129] The first construction unit 410 is used to obtain a pre-trained single-target fine-tuned large language model; extract the target vector of each single-target strategy in the model, perform sparse processing on the target vector by task vector singular value decomposition method, and generate a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, use PCB-merging fusion model and Free-merging fusion model for processing to obtain a multi-target LoRA expert model;

[0130] The acquisition unit 420 is used to generate a linear routing layer and construct a reward loss function corresponding to the linear routing layer; the loss function is optimized by using a mirror gradient descent method and a smoothed Chebyshev scalarization method to obtain a multi-objective routing expert;

[0131] The second construction unit 430 is used to design a weighted router; a hierarchical hybrid expert model is constructed according to the multi-objective LoRA expert model, the multi-objective routing experts and the weighted router;

[0132] The output unit 440 is used to obtain the prompt word and preference vector input by the user, input the prompt word and preference vector input by the user into the hybrid expert model for alignment processing, and output the information that meets the user's preference.

[0133] Optionally, the first construction unit 410 is used to:

[0134] Defining parameters of the pre-trained single-objective fine-tuned large language model; extracting a target vector for each single-objective strategy in the model based on the parameters of the pre-trained single-objective fine-tuned large language model;

[0135] The task vector singular value decomposition method is used for sparse processing. By decomposing the target vector of each single target, a low-rank adapter is obtained; the low-rank adapter is used as the LoRA expert of each single target;

[0136] The preference vector is set, and the PCB-merging fusion model and the Free-merging fusion model are used to process the LoRA experts of each single target to obtain the multi-target LoRA expert model.

[0137] Optionally, the reward loss function is expressed by the following formula (1):

[0138] (1)

[0139] in, Reward signals representing N targets are preferred A linear weighted combination of Indicates hidden state; Represents the parameters of the pre-trained model; represents routing expert; Represents a hierarchical mixture of experts model; Represents all LoRA experts.

[0140] Optionally, the process of optimizing the loss function by using the mirror gradient descent method and the smoothed Chebyshev scalar quantization method is expressed by the following formula (2):

[0141] (2)

[0142] in, Indicate the reference point for each objective, i.e. the desired level of performance; represents the preference weight of the i-th target; Represents the trainable parameters of the model.

[0143] Optionally, the weight router is used to map the user's continuous preference vector to the nearest expert and assign a voting weight to each selected expert; wherein the nearest experts include: LoRA experts and routing experts;

[0144] The calculation process of the weighted router is expressed by the following formula (3):

[0145] (3)

[0146] in, Represents the preference vector given by the user; Indicates the numbers of the N experts closest to the preference vector given by the user.

[0147] Optionally, the output unit 440 is used to:

[0148] The prompt words and preference vectors input by the user are input into the hierarchical hybrid expert model. The weighted router selects multiple routing experts to activate according to the preference vectors and assigns corresponding weights to each routing expert.

[0149] Assign corresponding weights to each routing expert and input them into the multi-target routing expert. According to the received hidden layer state input, dynamically vote for each single-target LoRA expert. According to the corresponding weights assigned to each routing expert, the voting values ​​of all routing experts are weighted summed to obtain the final voting summary result.

[0150] The final voting summary results are input into the multi-objective LoRA expert model. Through the final voting summary results, the LoRA expert parameters are dynamically activated, and the parameters of the pre-trained single-objective fine-tuned large language model are combined to generate a model that meets the user's preferences.

[0151] Optionally, the process of outputting the information in accordance with the user's preference is expressed by the following formula (4):

[0152] (4)

[0153] in, Indicate in preference The final output for a given input x; Represents the raw output of the pre-trained parameters; Indicates the voting weight given to each LoRA expert; The B matrix represents each LoRA expert; A matrix representing each LoRA expert; Represents the hidden state of the input.

[0154] The embodiment of the present invention first obtains a pre-trained single-target fine-tuned large language model; extracts the target vector of each single-target strategy in the model, decomposes the target vector of each single target by using the task vector singular value decomposition method, and generates a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, a multi-target LoRA expert model is constructed by using the PCB-merging and Free-merging fusion methods; a linear routing layer is generated, and a reward loss function corresponding to the linear routing layer is constructed; the loss function is optimized by using the mirror gradient descent method and the smoothed Chebyshev scalarization method to obtain the routing expert of the multi-target; a weight router is designed; a hierarchical hybrid expert model is constructed according to the multi-target LoRA expert model, the multi-target routing expert and the weight router; the prompt word and preference vector input by the user are obtained, the prompt word and the preference vector input by the user are input into the hierarchical hybrid expert model for alignment processing, and the output meets the user's preference.

[0155] The embodiment of the present invention firstly generates a low-rank adapter to construct a multi-target LoRA expert, and reduces the parameter scale of the model by designing a routing expert, thereby reducing the storage and computing costs; secondly, the embodiment of the present invention remains frozen during reasoning, and only a small amount of training is required for the routing expert, thereby reducing the training cost; thirdly, the embodiment of the present invention only activates a small number of expert models during reasoning, thereby reducing the computing overhead and improving the reasoning efficiency; finally, through the hierarchical expert model design, it can dynamically adapt to different user preferences and realize the optimal Pareto frontier of multi-target alignment.

[0156] Figure 5 is a structural diagram of a large model multi-preference alignment device based on a hierarchical hybrid expert model provided by an embodiment of the present invention, such as Figure 5 As shown, the large model multi-preference alignment device based on the hierarchical hybrid expert model may include the above Figure 4The large model multi-preference alignment device based on hierarchical mixture of expert models is shown. Optionally, the large model multi-preference alignment device 510 based on hierarchical mixture of expert models may include a first processor 2001 .

[0157] Optionally, the large model multi-preference alignment device 510 based on hierarchical hybrid expert model may further include a memory 2002 and a transceiver 2003 .

[0158] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0159] Combine the following Figure 5 The components of the large model multi-preference alignment device 510 based on the hierarchical hybrid expert model are specifically introduced:

[0160] The first processor 2001 is the control center of the large model multi-preference alignment device 510 based on the hierarchical hybrid expert model, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0161] Optionally, the first processor 2001 can perform various functions of the large model multi-preference alignment device 510 based on the hierarchical hybrid expert model by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0162] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0163] In a specific implementation, as an embodiment, the large model multi-preference alignment device 510 based on the hierarchical hybrid expert model may also include multiple processors, such as Figure 5The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0164] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0165] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0166] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0167] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0168] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0169] It should be noted that Figure 5 The structure of the large model multi-preference alignment device 510 based on the hierarchical hybrid expert model shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0170] In addition, the technical effects of the large model multi-preference alignment device 510 based on the hierarchical hybrid expert model can refer to the technical effects of the large model multi-preference alignment method based on the hierarchical hybrid expert model described in the above method embodiment, and will not be repeated here.

[0171] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0172] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0173] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0174] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0175] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0176] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0177] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0178] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0179] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0182] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0183] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A large model multi-preference alignment method based on a hierarchical hybrid expert model, characterized in that: The method comprises: S1. Obtain a pre-trained single-target fine-tuned large language model; extract the target vector of each single-target strategy in the model, perform sparse processing on the target vector by task vector singular value decomposition method, and generate a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, use PCB-merging fusion model and Free-merging fusion model for processing to obtain a multi-target LoRA expert model; S2. Generate a linear routing layer and construct a reward loss function corresponding to the linear routing layer; optimize the loss function using a mirror gradient descent method and a smoothed Chebyshev scalarization method to obtain a multi-objective routing expert; S3. Design weighted routers; construct a hierarchical hybrid expert model based on the multi-objective LoRA expert model, multi-objective routing experts and weighted routers; S4, obtaining the prompt word and preference vector input by the user; inputting the prompt word and preference vector input by the user into the hybrid expert model for alignment processing, and outputting the information in accordance with the user's preference.

2. The large model multi-preference alignment method based on hierarchical hybrid expert model according to claim 1 is characterized in that: The S1 extracts the target vector of each single-target strategy in the model, performs sparse processing on the target vector by the task vector singular value decomposition method, and generates a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, the PCB-merging fusion model and the Free-merging fusion model are used for processing to obtain a multi-target LoRA expert model, including: S11, defining parameters of the pre-trained single-objective fine-tuned large language model; according to the parameters of the pre-trained single-objective fine-tuned large language model, extracting the target vector of each single-objective strategy in the model; S12. Use the task vector singular value decomposition method to perform sparse processing, and obtain a low-rank adapter by decomposing the target vector of each single target; use the low-rank adapter as the LoRA expert of each single target; S13. Set the preference vector, and use the PCB-merging fusion model and the Free-merging fusion model to process each single-target LoRA expert to obtain a multi-target LoRA expert model.

3. The large model multi-preference alignment method based on hierarchical hybrid expert model according to claim 1 is characterized in that: The reward loss function of S2 is expressed by the following formula (1): （1） in, Reward signals representing N targets are preferred A linear weighted combination of Indicates hidden state; Represents the parameters of the pre-trained model; represents routing expert; represents a hierarchical mixture of experts model; Represents all LoRA experts.

4. The large model multi-preference alignment method based on hierarchical hybrid expert model according to claim 1 is characterized in that: The process of optimizing the loss function using the mirror gradient descent method and the smoothed Chebyshev scalar quantization method in S2 is expressed by the following formula (2): （2） in, Indicate the reference point for each objective, i.e. the desired level of performance; represents the preference weight of the i-th target; Represents the trainable parameters of the model.

5. The large model multi-preference alignment method based on hierarchical hybrid expert model according to claim 1 is characterized in that: The weight router is used to map the user's continuous preference vector to the nearest expert and assign a voting weight to each selected expert; wherein the nearest experts include: LoRA experts and routing experts; The calculation process of the weighted router is expressed by the following formula (3): （3） in, Represents the preference vector given by the user; Indicates the numbers of the N experts closest to the preference vector given by the user.

6. The large model multi-preference alignment method based on hierarchical hybrid expert model according to claim 1 is characterized in that: The step S4 inputs the prompt word and the preference vector input by the user into the hybrid expert model for alignment processing, and outputs an alignment that meets the user's preference, including: S41, inputting the prompt word and preference vector input by the user into the hierarchical hybrid expert model, and the weight router selects multiple routing experts to activate according to the preference vector, and assigns a corresponding weight to each routing expert; S42, assigning a corresponding weight to each routing expert and inputting it into the multi-target routing expert, dynamically voting for each single-target LoRA expert according to the received hidden layer state input, assigning a corresponding weight to each routing expert, and weighted summing the voting values ​​of all routing experts to obtain the final voting summary result; S43. Input the final voting summary result into the multi-objective LoRA expert model. Through the final voting summary result, dynamically activate the LoRA expert parameters, and combine the parameters of the pre-trained single-objective fine-tuned large language model to generate a preference that meets the user.

7. The large model multi-preference alignment method based on hierarchical hybrid expert model according to claim 1 is characterized in that: The process of the output of S4 conforming to the user's preference is expressed by the following formula (4): （4） in, Indicate in preference The final output for a given input x; represents the raw output of the pre-trained parameters; Indicates the voting weight given to each LoRA expert; The B matrix represents each LoRA expert; A matrix representing each LoRA expert; Represents the hidden state of the input.

8. A large model multi-preference alignment device based on a hierarchical hybrid expert model, the large model multi-preference alignment device based on a hierarchical hybrid expert model is used to implement the large model multi-preference alignment method based on a hierarchical hybrid expert model as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The first construction unit is used to obtain a pre-trained single-target fine-tuned large language model; extract the target vector of each single-target strategy in the model, perform sparse processing on the target vector by the task vector singular value decomposition method, and generate a low-rank adapter as the LoRA expert of each single target; according to the LoRA expert of each target, the PCB-merging fusion model and the Free-merging fusion model are used for processing to obtain a multi-target LoRA expert model; An acquisition unit is used to generate a linear routing layer and construct a reward loss function corresponding to the linear routing layer; the loss function is optimized by using a mirror gradient descent method and a smoothed Chebyshev scalarization method to obtain a multi-objective routing expert; The second construction unit is used to design a weighted router; a hierarchical hybrid expert model is constructed based on the multi-objective LoRA expert model, the multi-objective routing experts and the weighted router; The output unit is used to obtain the prompt word and preference vector input by the user, input the prompt word and preference vector input by the user into the hybrid expert model for alignment processing, and output in accordance with the user's preference.

9. A large model multi-preference alignment device based on a hierarchical hybrid expert model, characterized in that: The large model multi-preference alignment device based on the hierarchical hybrid expert model includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images