Medical hybrid expert fine tuning method based on multi-view learning

Abstract

The invention discloses a medical hybrid expert fine tuning method based on multi-view learning. The method comprises the following steps that firstly, a model carries out representation enhancement training on a plurality of target task classifications; the attention-feedforward neural network adapter is used for efficient fine tuning of parameters, expert modules of different tasks are trained respectively, and the characterization ability of the model to various tasks is enhanced. Secondly, extracting a model increment parameter generated by training in the previous step, and regarding the model increment parameter as an expert module; and connecting different classification expert modules obtained by multiple times of mixed training through a router network, and carrying out secondary training by using a mixed classification downstream task data set. And finally, a dense router network is adopted. Meanwhile, the model can learn potential relations in different tasks in a joint multi-task expert, so that the model can solve different types of problems by using multiple perspectives.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing technology, specifically to the field of large language model training technology, and more specifically to providing a medical hybrid expert fine-tuning method based on multi-perspective learning. Background Art

[0002] As large language models gradually penetrate the medical field, their comprehensive capabilities in multi-task scenarios have become a research focus. Unlike general domains, medical texts possess distinct specialized characteristics, with high terminology density and a specific semantic space. Furthermore, different task types, including contextual understanding and knowledge reasoning, exhibit strong heterogeneity. Furthermore, medical decision-making places stringent demands on the accuracy and interpretability of model outputs.

[0003] While traditional full-parameter fine-tuning methods can improve domain adaptability, they face high computational costs and data requirements. Mainstream efficient parameter fine-tuning methods are limited by the limited parameter adjustment space and struggle to simultaneously capture commonalities and specific knowledge across multiple tasks. Specifically, independent adapter modules with fixed structures struggle to dynamically adapt to the differences in semantic distribution across tasks, leading to representation conflicts during knowledge transfer. This phenomenon is primarily due to the inherent contradiction of efficient parameter fine-tuning methods in task-diversity scenarios: the mismatch between limited adjustable parameters and the complex knowledge representation requirements.

[0004] In this context, the hybrid expert mechanism offers a new solution to this problem. By assigning specialized "expert" modules to different tasks and controlling them through routers, only experts relevant to the current task participate in reasoning. This approach can control the model's computational cost and provide flexible adjustment capabilities. Although recent studies have attempted to combine hybrid experts with efficient parameter fine-tuning methods, their routers typically rely on a single linear layer as a sparse expert network, failing to fully capture the fine-grained semantic features in medical text. This results in low expert selection accuracy, which in turn affects the model's knowledge representation capabilities.

[0005] In summary, this field urgently needs to design a medical hybrid expert fine-tuning method based on multi-perspective learning to solve the above problems. Summary of the Invention

[0006] In view of this, the present invention provides a medical mixed expert fine-tuning method based on multi-perspective learning (Mixture of Attention-FFN Adapter, MAF Adapter). First, a medical mixed expert fine-tuning method based on multi-perspective learning is proposed, which enhances the model's knowledge representation ability in multi-task scenarios through two-stage fine-tuning of multi-perspective representation enhancement and mixed learning. Second, a model knowledge injection method based on incremental fine-tuning is proposed, which integrates the Attention-Feedforward Neural Network Adapter (Attention-FFN Adapter, AF Adapter) with the modern generative large language model structure and applies it to multi-perspective representation enhancement. Third, a multi-perspective hybrid router network is proposed to learn the semantic information contained in different tasks, isolate the knowledge conflicts between different tasks, and utilize the potential connections between different tasks.

[0007] The technical route of the present invention is implemented in the following form: First, in the multi-perspective representation enhancement stage, the model is trained for representation enhancement of multiple target task categories. By using AF Adapter to efficiently fine-tune parameters, expert modules for different tasks are trained separately to enhance the model's representation capabilities for various tasks. Secondly, entering the multi-perspective hybrid learning stage, the model incremental parameters generated by the first stage of training are extracted and regarded as expert modules. The different classification expert modules obtained by multiple mixed trainings are connected through a router network, and the hybrid classification downstream task data set is used for secondary training. Finally, in order to enable the model to better perform multi-perspective learning, MAF Adapter adopts a dense router network. At the same time, the model can learn the potential connections in different tasks in the joint multi-task experts, so that the model can use multiple perspectives to solve different types of problems.

[0008] This invention takes medical hybrid expert fine-tuning as an example, and the specific steps are:

[0009] S1. Design a multi-perspective representation enhancement mechanism to classify downstream tasks into multiple different types. Pre-train multiple task expert modules for each type of task and extract the incremental model parameters generated by the training.

[0010] S2, introduces a multi-perspective hybrid router network, treats the incremental model parameters generated by S1 training as expert modules, connects different classification expert modules obtained by multiple training sessions through the router network, and uses the hybrid classification downstream task dataset for secondary training;

[0011] S3: A two-stage progressive training strategy is adopted. In S1, the expert module is trained separately to enhance the task capability; in S2, the matching between the expert module and the task is optimized through joint training.

[0012] Furthermore, step S1 includes:

[0013] S11. Freeze the original model parameters, add i attention heads to the original model, and expand the input hidden state x of the attention layer. After three expanded linear layers, the query, key, and value vectors Q′, K′, and V′ are calculated respectively. The calculation formula is as follows:

[0014] Q′(x)=x[W Q :W Q ′]+[b Q :b′ Q ],

[0015] K′(x)=x[W K :W K ′]+[b K :b′ K ],

[0016] V′(x)=x[W V :W V ′]+[b V :b′ V ],

[0017] in, represents the weight matrix and bias, d k =d v =d model / h,d model is the dimension size of the model, h = nh kv is the number of attention heads, n is the number of queries in each group;

[0018] S12, cut Q', K', V', cut Q' into h+i parts, cut K', V' into h kv +i part, calculates the multi-head attention of the original model and the newly added expert module j as follows:

[0019]

[0020] Among them, K′ jmod n , V′ jmod n Represents the attention head part of the base model; Indicates the newly added expert attention head part;

[0021] S13, calculation expert module output GQA AF (x), the formula is as follows:

[0022] GQA AF (x)=[head1:…:head h :head′1:…:head i ′][WO ⊥W O ′]+b O +b′ O ,

[0023] Among them, head1:…:head h is the original attention head, head′1:…:head′ i To add attention heads, represents the weight matrix and bias output from the expert module, Represents the weight matrix and bias of the original module;

[0024] S14, add gating layer and feedforward network, add size d a In the feedforward network layer with a gating layer, the input hidden state x first passes through the extended gating layer to calculate the linear unit weight, and then is calculated in sequence with the extended upper projection and extended lower projection layers, and the final output result FFN AF (x), the calculation formula is as follows:

[0025] G AF (x)=Act(x[W gate :W′ gate ]+[b gate :b′ gate ]),

[0026]

[0027] FFN AF (x)=I AF (x)[W down :W′ down ]+b down +b′ down ,

[0028] in, are the weight matrices and biases of the newly added expert modules, is the weight matrix and bias of the original module;

[0029] S15. Extract the parameters of the expert module. After obtaining the expert model for different types of task training, extract the added parameters for the second stage of hybrid training. The extracted parameters are W′ Q , b′ Q , W′ K , b′ K , W′ V , b′ V , W′ O , b′ O , W′gate , b′ gate , W′ up , b′ up , W′ down , b′ down .

[0030] Furthermore, step S2 includes:

[0031] S21. In multi-view learning, the router takes the hidden state of each layer of Transformer as input and outputs the original model state weight R model And the weight of each expert status R expert , the routers added in different layers are independent of each other, and the calculation formula is as follows:

[0032] S(x)=xW S +b S ,

[0033] R model =Sigmoid(SW model +b model ),

[0034]

[0035] in, and Corresponding to the weight matrix and bias respectively, and are the state weights of the original model and each expert, d i is a hyperparameter, indicating the size of the shared linear layer;

[0036] S22. Apply the expert weights to the multi-view multi-head attention layer and the feedforward network layer. For the multi-head attention layer, add the corresponding weights of the n experts in the first stage to the original model. The formula is as follows:

[0037]

[0038] For the feedforward network layer, the corresponding weights of the n experts in the first stage are also added to the original model. The formula is as follows:

[0039]

[0040] Among them, W′ {i} represents the corresponding added weight of the i-th expert, b′ {i} Indicates the corresponding added bias of the i-th expert, gate represents the gating layer, up represents the upper projection layer, and down represents the lower projection layer;

[0041] S23, adjust the multi-head attention layer and feedforward network layer according to the router weight. For the multi-head attention layer, the router weight is weighted in its attention output part to obtain GQA MAF (x), the calculation formula is as follows:

[0042]

[0043] Among them, R model It needs to be expanded according to the number and size of the original model attention heads R expert Then we need to add attention heads i according to the number of attention heads added by each expert {j} and the attention head size are expanded to head1,…head h is the multi-head dot product attention result, head′1,...head′ i Calculated for the added i-th expert attention head; for the feedforward network layer, the router weight is applied to the gated part output FFN MAF (x), as shown in the following formula:

[0044]

[0045] Among them, R model It needs to be expanded according to the hidden layer size of the original model feedforward network layer R expert The size of the hidden units added by each expert needs to be Expand to I MAF (x) represents the hidden layer calculation results in the extended feedforward network layer.

[0046] Furthermore, step S3 includes:

[0047] S31. During the S1 training process, the target task is divided into multiple types of task sets. The representation ability of the model is improved for different types of task sets. The training adopts an efficient parameter fine-tuning method, freezing all parameters inherited from the original model. Only the added expert module can be updated.

[0048] In S32, the expert modules trained in S1 are spliced ​​and combined, and a core multi-view hybrid router network is added. The training uses a multi-classification hybrid dataset. Only the added expert modules and router modules can be updated, while all parameters inherited from the original general domain language model are frozen.

[0049] S33. The training dataset adopts the instruction fine-tuning paradigm. The input data format is consistent with the original model input template. The instruction data can be formally represented as (I, X, Y), where I is the task instruction, X is the task optional input, and Y is the task output.

[0050] After adopting the above strategy, the positive effects of the present invention are:

[0051] (1) Aiming at the problems in the medical field, such as high term density, strong semantic specificity and task diversity of medical texts, this paper proposes a medical hybrid expert fine-tuning method based on multi-perspective learning, which enhances the knowledge representation ability of large language models in multi-task scenarios and can effectively improve the accuracy and interpretability of medical decisions.

[0052] (2) In order to solve the problem of knowledge transfer difficulties in traditional parameter efficient fine-tuning methods when dealing with multi-task scenarios, this paper proposes a model knowledge injection method based on incremental fine-tuning, and effectively isolates knowledge conflicts between tasks through a multi-perspective hybrid router network, which brings about performance improvement of the model in complex medical tasks, especially in knowledge sharing between tasks and accuracy of expert selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the medical hybrid expert fine-tuning method based on multi-perspective learning of the present invention.

[0054] Figure 2 This is the overall structure diagram of the medical hybrid expert fine-tuning method based on multi-view learning of the present invention

[0055] Figure 3 This is the structural diagram of the present invention's attention-feedforward neural network adapter combined with group query attention and gated linear unit

[0056] Figure 4 This is the network structure diagram of the multi-view hybrid router of the present invention DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more obvious and understandable, the present invention is further described in detail below in conjunction with embodiments and accompanying drawings.

[0058] See Figure 1 , Figure 1This is a flow chart of the medical hybrid expert fine-tuning method based on multi-perspective learning of the present invention, which includes: first, designing a multi-perspective representation enhancement mechanism, dividing downstream tasks into multiple different types, and "pre-training" multiple task expert modules for each different type of task, and extracting the model incremental parameters generated by the training; second, introducing a multi-perspective hybrid router network, treating the model incremental parameters generated by S1 training as expert modules, connecting different classification expert modules obtained by multiple trainings through the router network, and using the hybrid classification downstream task data set for secondary training; finally, adopting a two-stage progressive training strategy, training the expert module separately in S1 to enhance the task capability; and optimizing the matching between the expert module and the task through joint training in S2.

[0059] See Figure 2 、 Figure 3 、 Figure 4 , Figure 2 This is the overall structure diagram of the medical hybrid expert fine-tuning method based on multi-view learning of the present invention. Figure 3 This is the structural diagram of the present invention's attention-feedforward neural network adapter combined with grouped query attention and gated linear units. Figure 4 This is a diagram of the network structure of a multi-perspective hybrid router of the present invention, and the specific steps include:

[0060] S1. Design a multi-perspective representation enhancement mechanism to divide downstream tasks into multiple different types, and "pre-train" multiple task expert modules for each different type of task, and extract the model incremental parameters generated by the training.

[0061] S11. Freeze the original model parameters, add i attention heads to the original model, and expand the input hidden state x of the attention layer. After three expanded linear layers, the query, key, and value vectors Q′, K′, and V′ are calculated respectively. The calculation formula is as follows:

[0062] Q′(x)=x[W Q :W Q ′]+[b Q :b′ Q ],

[0063] K′(x)=x[W K :W K ′]+[b K :b′ K ],

[0064] V′(x)=x[W V :W V ′]+[b V :b′ V ],

[0065] in, represents the weight matrix and bias, d k =d v =d model / h,d model is the dimension size of the model, h = nh kv is the number of attention heads, n is the number of queries in each group;

[0066] S12, cut Q', K', V', cut Q' into h+i parts, cut K', V' into h kv +i part, calculates the multi-head attention of the original model and the newly added expert module j as follows:

[0067]

[0068] Among them, K′ jmod n , V′ jmod n Represents the attention head part of the base model; Indicates the newly added expert attention head part;

[0069] S13, calculation expert module output GQA AF (x), the formula is as follows:

[0070] GQA AF (x)=[head1:…:head h :head′1:…:head′ i ][W O ⊥W′ O ]+b O +b′ O ,

[0071] Among them, head1:…:head h is the original attention head, head′1:…:head′ i To add attention heads, represents the weight matrix and bias output from the expert module, Represents the weight matrix and bias of the original module;

[0072] S14, add gating layer and feedforward network, add size d a In the feedforward network layer with a gating layer, the input hidden state x first passes through the extended gating layer to calculate the linear unit weight, and then is calculated in sequence with the extended upper projection and extended lower projection layers, and the final output result FFN AF (x), the calculation formula is as follows:

[0073] G AF (x)=Act(x[W gate :W′gate ]+[b gate :b′ gate ]),

[0074]

[0075] FFN AF (x)=I AF (x)[W down :W′ down ]+b down +b′ down ,

[0076] in, are the weight matrices and biases of the newly added expert modules, is the weight matrix and bias of the original module;

[0077] S15. Extract the parameters of the expert module. After obtaining the expert model for different types of task training, extract the added parameters for the second stage of hybrid training. The extracted parameters are W′ Q , b′ Q , W′ K , b′ K , W′ V , b′ V , W′ O , b′ O , W′ gate , b′ gate , W′ up , b′ up , W′ down , b′ down .

[0078] S2, introduces a multi-perspective hybrid router network, regards the model incremental parameters generated by S1 training as expert modules, connects different classification expert modules obtained by multiple training sessions through the router network, and uses the hybrid classification downstream task dataset for secondary training.

[0079] S21. In multi-view learning, the router takes the hidden state of each layer of Transformer as input and outputs the original model state weight R model And the weight of each expert status R expert , the routers added in different layers are independent of each other, and the calculation formula is as follows:

[0080] S(x)=xW S +b S ,

[0081] R model =Sigmoid(SWmodel +b model ),

[0082]

[0083] in, and Corresponding to the weight matrix and bias respectively, and are the state weights of the original model and each expert, d i is a hyperparameter, indicating the size of the shared linear layer;

[0084] S22. Apply the expert weights to the multi-view multi-head attention layer and the feedforward network layer. For the multi-head attention layer, add the corresponding weights of the n experts in the first stage to the original model. The formula is as follows:

[0085]

[0086]

[0087] For the feedforward network layer, the corresponding weights of the n experts in the first stage are also added to the original model. The formula is as follows:

[0088]

[0089] Among them, W′ {i} represents the corresponding added weight of the i-th expert, b′ {*} Indicates the corresponding added bias of the i-th expert, gate represents the gating layer, up represents the upper projection layer, and down represents the lower projection layer;

[0090] S23, adjust the multi-head attention layer and feedforward network layer according to the router weight. For the multi-head attention layer, the router weight is weighted in its attention output part to obtain GQA MAF (x), the calculation formula is as follows:

[0091]

[0092] Among them, R model It needs to be expanded according to the number and size of the original model attention heads R expert Then we need to add attention heads i according to the number of attention heads added by each expert {j} and the attention head size are expanded to head1,…head h is the multi-head dot product attention result, head′1,...head i′ is calculated by adding the i-th expert attention head; for the feedforward network layer, the router weight is applied to the gated part output FFN MAF (x), as shown in the following formula:

[0093]

[0094] Among them, R model It needs to be expanded according to the hidden layer size of the original model feedforward network layer R expert The size of the hidden units added by each expert needs to be Expand to I MAF (x) represents the hidden layer calculation results in the extended feedforward network layer.

[0095] S3: A two-stage progressive training strategy is adopted. In S1, the expert module is trained separately to enhance the task capability; in S2, the matching between the expert module and the task is optimized through joint training.

[0096] S31. During the S1 training process, the target task is divided into multiple types of task sets. The representation ability of the model is improved for different types of task sets. The training adopts an efficient parameter fine-tuning method, freezing all parameters inherited from the original model. Only the added expert module can be updated.

[0097] In S32, the expert modules trained in S1 are spliced ​​and combined, and a core multi-view hybrid router network is added. The training uses a multi-classification hybrid dataset. Only the added expert modules and router modules can be updated, while all parameters inherited from the original general domain language model are frozen.

[0098] S33, the training data set adopts the instruction fine-tuning paradigm. The input data format is consistent with the original model input template. The instruction data can be formally represented as (I, X, Y), where I is the task instruction, X is the task optional input, and Y is the task output. The router shared linear layer size is 1024, that is, d i =1024, the parameter initialization adopts the normal distribution centered on 0, that is: The training objective loss function loss(Y) is to minimize the log-likelihood probability of the model output Y.

[0099] Specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, those skilled in the art will appreciate that various modifications and substitutions may be made to the specific embodiments of the present invention without departing from the spirit and scope of the present invention. Such modifications and substitutions are intended to fall within the scope of the claims.

Claims

1. A medical hybrid expert fine-tuning method based on multi-view learning, characterized by The following steps are involved: S1. Design a multi-perspective representation enhancement mechanism to classify downstream tasks into multiple different types. Pre-train multiple task expert modules for each type of task and extract the incremental model parameters generated by the training. S2, introduces a multi-perspective hybrid router network, treats the incremental model parameters generated by S1 training as expert modules, connects different classification expert modules obtained by multiple training sessions through the router network, and uses the hybrid classification downstream task dataset for secondary training; S3: A two-stage progressive training strategy is adopted. In S1, the expert module is trained separately to enhance the task capability; in S2, the matching between the expert module and the task is optimized through joint training.

2. The medical hybrid expert fine-tuning method based on multi-view learning according to claim 1 is characterized in that: The step S1 specifically includes: S11. Freeze the original model parameters, add i attention heads to the original model, and expand the input hidden state x of the attention layer. After three expanded linear layers, the query, key, and value vectors Q′, K′, and V′ are calculated respectively. The calculation formula is as follows: Q′(x)=x[W Q :W′ Q ]+[b Q :b′ Q ], N′(x)=x[W K :W′ K ]+[b K :b′ K ], V′(x)=x[W V :W′ V ]+[b V :b′ V ], in, represents the weight matrix and bias, d k =d v =d model / h,d model is the dimension size of the model, h = nh kv is the number of attention heads, n is the number of queries in each group; S12, cut Q', K', V', cut Q' into h+i parts, cut K', V' into h kv +i part, calculates the multi-head attention of the original model and the newly added expert module j as follows: Among them, K′ j mod n , V′ j mod n Represents the attention head part of the base model; Indicates the newly added expert attention head part; S13, calculation expert module output GQA AF (x), the formula is as follows: Among them, head1:…:head h is the original attention head, head′1:…:head′ i To add attention heads, represents the weight matrix and bias output from the expert module, Represents the weight matrix and bias of the original module; S14, add gating layer and feedforward network, add size d a In the feedforward network layer with a gating layer, the input hidden state x first passes through the extended gating layer to calculate the linear unit weight, and then is calculated in sequence with the extended upper projection and extended lower projection layers, and the final output result FFN AF (x), the calculation formula is as follows: G AF (x)=Act(x[W gate :W′ gate ]+[b gate :b′ gate ]), FFN AF (x)=I AF (x)[W down :W′ down ]+b down +b′ down , in, are the weight matrices and biases of the newly added expert modules, is the weight matrix and bias of the original module; S15. Extract the parameters of the expert module. After obtaining the expert model for different types of task training, extract the added parameters for the second stage of hybrid training. The extracted parameters are W′ Q , b′ Q , W′ K , b′ K , W′ V , b′ V , W′ O , b′ O , W′ gate , b′ gate , W′ up , b′ up , W′ down , b′ down .

3. The medical hybrid expert fine-tuning method based on multi-view learning according to claim 1 is characterized in that: The step S2 specifically includes: S21. In multi-view learning, the router takes the hidden state of each layer of Transformer as input and outputs the original model state weight R model And the weight of each expert status R expert , the routers added in different layers are independent of each other, and the calculation formula is as follows: S(x)=xW S +b s , R mode =Sigmoid(SW model +b model ), in, and Corresponding to the weight matrix and bias respectively, and are the state weights of the original model and each expert, d * is a hyperparameter, indicating the size of the shared linear layer; S22. Apply the expert weights to the multi-view multi-head attention layer and the feedforward network layer. For the multi-head attention layer, add the corresponding weights of the n experts in the first stage to the original model. The formula is as follows: For the feedforward network layer, the corresponding weights of the n experts in the first stage are also added to the original model. The formula is as follows: Among them, W′ {i} represents the corresponding added weight of the i-th expert, b′ {i} Indicates the corresponding added bias of the i-th expert, gate represents the gating layer, up represents the upper projection layer, and down represents the lower projection layer; S23, adjust the multi-head attention layer and feedforward network layer according to the router weight. For the multi-head attention layer, the router weight is weighted in its attention output part to obtain GQA MAF (x), the calculation formula is as follows: Among them, R model It needs to be expanded according to the number and size of the original model attention heads R expert Then we need to add attention heads i according to the number of attention heads added by each expert {j} and the attention head size are expanded to head1:…:head h is the multi-head dot product attention result, head′1:…:head′ i Calculated for the added i-th expert attention head; for the feedforward network layer, the router weight is applied to the gated part output FFN MAF (x), as shown in the following formula: Among them, R model It needs to be expanded according to the hidden layer size of the original model feedforward network layer R expert The size of the hidden units added by each expert needs to be Expand to I MAF (x) represents the hidden layer calculation results in the extended feedforward network layer.

4. The medical hybrid expert fine-tuning method based on multi-view learning according to claim 1 is characterized in that: The step S3 specifically includes: S31. During the S1 training process, the target task is divided into multiple types of task sets. The representation ability of the model is improved for different types of task sets. The training adopts an efficient parameter fine-tuning method, freezing all parameters inherited from the original model. Only the added expert module can be updated. In S32, the expert modules trained in S1 are spliced ​​and combined, and a core multi-view hybrid router network is added. The training uses a multi-classification hybrid dataset. Only the added expert modules and router modules can be updated, while all parameters inherited from the original general domain language model are frozen. S33. The training dataset adopts the instruction fine-tuning paradigm. The input data format is consistent with the original model input template. The instruction data can be formally represented as (I, X, Y), where I is the task instruction, X is the task optional input, and Y is the task output.

Images