A large language model adaptive fusion method, device and equipment

Abstract

This invention relates to the field of large language model fusion technology, and discloses an adaptive fusion method, apparatus, and device for large language models. The method includes: fine-tuning a base model based on multiple vertical domain datasets to obtain multiple vertical domain models; calculating the increment vector of the model parameters of each vertical domain model relative to the base model parameters, denoted as the task vector; sampling from the union of the multiple vertical domain datasets to obtain multiple batch training sets; fixing the base model and performing batch training on the gating parameters; during training, for each training sample, extracting the semantic feature vector of the input text and calculating the gating probability matrix; calculating the fusion weight of each task vector; updating the fusion model based on the base model parameters, task vectors, and corresponding fusion weights; and finally updating the gating parameters through model feedforward and backpropagation, and repeating the training. This invention can obtain a fusion model applicable to multiple vertical domains.

Description

Technical Field

[0001] This invention belongs to the field of large language model fusion technology, and particularly relates to an adaptive fusion method, apparatus and device for large language models. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, large language models (LLMs) have demonstrated remarkable general capabilities across multiple domains, such as question answering, code generation, and mathematical computation. However, as the application of LLMs in specific domains continues to increase, there is a need to support tasks that incorporate both general and domain-specific knowledge. To achieve this goal, performance optimization techniques based on model fusion have been proposed. Specifically, multiple task models obtained by further fine-tuning the same base model (such as the general large model, CLIP, ResNet-IN1K, etc.) can be merged to integrate the capabilities of models from multiple domains and improve overall efficiency. However, although model fusion techniques effectively enhance the multi-task capabilities of models, existing methods still have limitations.

[0004] Currently, common model fusion methods include linear averaging (such as TIES-Merging) and task vector distillation. Linear averaging achieves fusion by weighting the model weights, but this method cannot resolve nonlinear conflicts between task vectors. Task vector distillation distills knowledge from multiple models into a single model, but this process leads to the loss of high-frequency features and a significant increase in training costs. Summary of the Invention

[0005] In view of this, the present invention provides a large language model adaptive fusion method, apparatus and device to obtain a fusion model applicable to multiple vertical domains.

[0006] One aspect of the present invention provides an adaptive fusion method for large language models, comprising the following steps:

[0007] The base model was fine-tuned based on multiple vertical domain datasets to obtain multiple vertical domain models;

[0008] Calculate the increment vector of the model parameters of each vertical domain model relative to the base model parameters, denoted as the task vector;

[0009] The training set is obtained by sampling from the union of the multiple vertical domain datasets;

[0010] The base model is fixed, and the gating parameters are trained. During the training process, for each training sample, the semantic feature vector of the input text of the training sample is extracted based on the base model, and the gating probability matrix is ​​calculated according to the current gating parameters. The fusion weight of each task vector is calculated based on the gating probability matrix. The fusion model parameters are obtained based on the base model parameters, task vectors and corresponding fusion weights.

[0011] The parameters of the fusion model are assigned to the base model to obtain a new fusion model. For the new fusion model, the gating parameters are updated through model feedforward and backpropagation, and the training process is repeated until the termination condition is met.

[0012] In some embodiments, after acquiring multiple vertical domain datasets, each sample data is preprocessed, specifically: each sample data includes input text and output text, and both the input text and output text are segmented and converted into numerical sequences.

[0013] In some embodiments, multiple batch training sets are obtained by sampling from the union of the multiple vertical domain datasets. Each batch training set includes sample data from multiple vertical domains, and each round of training is based on one batch training set.

[0014] In some embodiments, during the training process, for each training sample, intermediate semantic feature vectors are extracted based on each feature extraction layer of the base model to obtain multi-layer semantic feature vectors;

[0015] For each layer of semantic feature vectors, calculate the gating probability matrix based on the current gating parameters, and average the gating probability matrices corresponding to each layer to obtain the overall gating probability matrix.

[0016] The fusion weights for each task vector are calculated based on the overall gating probability matrix;

[0017] Based on the base model parameters, task vectors, and corresponding fusion weights, the fusion model parameters for each layer are obtained.

[0018] In some embodiments, after obtaining the gating probability matrix, Bernoulli sampling is first performed on the gating probability matrix to generate a binary gating vector, and the fusion weight for each task vector is calculated based on the gating vector.

[0019] In some embodiments, a normalized fusion weight is calculated for each task vector based on the conflict factor and gating vector between vertical domain tasks, wherein the normalized fusion weight for the i-th vertical domain task vector is... The calculation formula is:

[0020]

[0021] in, Let be the value of the gating vector for the i-th vertical task. This represents the conflict factor between the i-th vertical domain task and other vertical domain tasks. is a constant, and N is the number of vertical domains.

[0022] In some embodiments, the conflict factor between the i-th vertical domain task and other vertical domain tasks The calculation formula is:

[0023]

[0024] in, S ij It represents the similarity between the task vectors of the i-th vertical domain and the j-th vertical domain, where N is the number of vertical domains.

[0025] In some embodiments, updating the gating parameters via model feedforward and backpropagation includes: predicting the output of the corresponding new fusion model based on the input text of each training sample; calculating the cross-entropy loss between the predicted output distribution and the actual output distribution; estimating the gating gradient based on the cross-entropy loss; and updating the gating parameters based on the gating gradient.

[0026] A second aspect of the present invention provides a large language model adaptive fusion apparatus, comprising:

[0027] The vertical model fine-tuning module is configured to fine-tune the base model based on multiple vertical datasets to obtain multiple vertical models.

[0028] The task vector calculation module is configured to calculate the incremental vector of the model parameters of each vertical domain model relative to the base model parameters, denoted as the task vector.

[0029] The training data acquisition module is configured to obtain a training set by sampling from the union of the multiple vertical domain datasets.

[0030] The model training module is configured to: fix the base model and train the gating parameters; during the training process, for each training sample, extract the semantic feature vector of the task text based on the base model, calculate the gating probability matrix according to the current gating parameters; calculate the fusion weight of each task vector according to the gating probability matrix; and obtain the fusion model parameters based on the base model parameters, task vectors and corresponding fusion weights.

[0031] The gating parameter update module is configured to: assign the fusion model parameters to the base model to obtain a new fusion model; update the gating parameters for the new fusion model through model feedforward and backpropagation; repeat the training process until the termination condition is met.

[0032] A third aspect of the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer instructions that, when executed by the processor, cause the electronic device to perform the method described thereon.

[0033] The above one or more technical solutions obtain multiple vertical domain models through fine-tuning and calculate task vectors, which can accurately capture the unique features and differences of each vertical domain. Through the gating mechanism, the model can dynamically adjust the fusion weights of the task vectors according to the semantic features of the input text, thereby achieving adaptive prediction for different vertical domains and improving the flexibility and accuracy of the model in multi-vertical domain scenarios. Attached Figure Description

[0034] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0035] Figure 1 An architectural diagram of a computer system provided in an exemplary embodiment of this application is shown;

[0036] Figure 2 A flowchart illustrating an exemplary embodiment of the large language model adaptive fusion method provided in this application is shown.

[0037] Figure 3 A structural block diagram of a large language model adaptive fusion apparatus provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0038] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0039] In the description of the embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on".

[0040] As described in the background section, among common model fusion methods, linear averaging is a relatively simple fusion strategy that achieves fusion by weighted averaging of the weights of different task models. The advantage of this method is its relatively simple computational process and ease of implementation. However, linear averaging has significant limitations; it cannot effectively resolve nonlinear conflicts between task vectors. When the weights of different task models conflict in direction in the vector space, simple weighted averaging may lead to mutual cancellation between weights, thereby reducing the performance of the fused model. This phenomenon is particularly common in multi-task scenarios because the optimal weights for different tasks often point in different directions, and direct linear combination struggles to capture these complex relationships.

[0041] Task vector distillation takes a different approach, fusing knowledge from multiple task models into a single model. The core idea is to compress and integrate the features and knowledge of multiple models to obtain a superior fusion model. However, task vector distillation faces several challenges in practical applications. First, the distillation process can lead to the loss of high-frequency features. High-frequency features often contain subtle but crucial information that is essential for a model's performance on certain tasks. During distillation, due to information compression and simplification, these high-frequency features may be ignored or lost, affecting the accuracy of the fused model. Second, task vector distillation significantly increases training costs. The distillation process requires additional training steps and computational resources, making the entire fusion process more complex and time-consuming. Furthermore, the information loss and increased training costs during distillation often result in the fused model performing worse than individually trained models on certain tasks, thus limiting its widespread adoption in practical applications.

[0042] Figure 1 A structural block diagram of a computer system 100 provided in an exemplary embodiment of this application is shown. The computer system 100 includes a server 110 and a terminal 120.

[0043] Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0044] Terminal 120 can be an electronic device such as a mobile phone, tablet computer, vehicle terminal (vehicle system), wearable device, PC (Personal Computer), unmanned reservation terminal, smart speaker, etc. Terminal 120 is used to collect task data from different vertical domains and build vertical domain datasets.

[0045] Server 110 and terminal 120 can communicate via a network, such as a wired or wireless network.

[0046] Those skilled in the art will understand that the number of terminals 120 described above can be more or less. For example, there may be only one terminal 120, or there may be dozens or hundreds, or more. This application does not limit the number or type of terminals 120 in its embodiments.

[0047] Figure 2 This invention illustrates a flowchart of an exemplary embodiment of a large language model adaptive fusion method provided in this application. The method is executed by a server or terminal, which may be... Figure 1 The server or terminal shown. The method specifically includes the following steps:

[0048] S101: Fine-tuning the base model based on multiple vertical domain datasets to obtain multiple vertical domain models;

[0049] S102: Calculate the incremental vector of the model parameters of each vertical domain model relative to the base model parameters, denoted as the task vector;

[0050] S103: The training set is obtained by sampling from the union of the multiple vertical domain datasets;

[0051] S104: Fix the base model and train the gating parameters; during the training process, for each training sample, extract the semantic feature vector of the input text of the training sample based on the base model, and calculate the gating probability matrix according to the current gating parameters; calculate the fusion weight of each task vector according to the gating probability matrix; obtain the fusion model parameters based on the base model parameters, task vectors and corresponding fusion weights.

[0052] S105: Assign the fusion model parameters to the base model to obtain a new fusion model. For the new fusion model, update the gating parameters through model feedforward and backpropagation. Repeat steps S104 and S105 until the termination condition is met.

[0053] This method achieves efficient fusion and adaptive prediction of multiple vertical domain datasets by fine-tuning the base model and combining it with a gating mechanism. First, by fine-tuning multiple vertical domain models and calculating task vectors, the unique features and differences of each vertical domain are accurately captured, providing rich semantic information for subsequent fusion. Simultaneously, the introduction of the gating mechanism allows the model to dynamically adjust the fusion weights of the task vectors based on the semantic features of the input text, thereby achieving adaptive prediction for different vertical domains and improving the model's flexibility and accuracy in multi-vertical domain scenarios. Furthermore, this method fixes the base model during training and only trains the gating parameters, significantly reducing the computational cost and time complexity of training.

[0054] In step S101, the base model M can be a general-purpose large language model such as GPT, or a language model such as BERT. N vertical domain datasets D={D1,D2,...,D...} N This corresponds to N different target domains, i.e., vertical domains. Different vertical domains include intelligent question answering, code generation, mathematical calculation, etc., in fields such as healthcare, law, and finance.

[0055] In some embodiments, after acquiring multiple vertical domain datasets, each sample data in each vertical domain dataset is preprocessed. Each sample data includes input text and output text, representing the requirements and solutions of a task. Both the input and output texts are segmented and tokenized. Tokenization converts the text into a numerical token ID, typically an integer sequence, which is convenient for use as input and output of the model.

[0056] Based on each preprocessed vertical domain dataset, the base model M is fine-tuned to obtain multiple vertical domain models {M1, M2, ..., M}. N}

[0057] In step S102, the incremental vector of the model parameters of each vertical domain model relative to the base model parameters is calculated, denoted as the task vector {Δθ1,Δθ2,...,Δθ}. N}. Task vector Δθ i This represents the changes in model parameters over the task in vertical domain i, which reflect the task-specific knowledge of vertical domain i.

[0058] In step S103, the union of the multiple vertical domain datasets is... Multiple batches of training sets are obtained through sampling, with each batch containing sample data from multiple vertical domains. Each training round is based on one batch of training sets. By sampling from the union of multiple vertical domain datasets, the diversity of cross-domain data can be fully utilized, enhancing the generalization ability of the final fusion model. It can be understood that in the training set, the input and output text of each sample data item are preprocessed numerical token IDs.

[0059] In step S104, the base model is fixed, and the gating parameters are trained. During the training process, the following steps are performed for each training sample to obtain a new fusion model. Specifically, these include:

[0060] S1041: Based on each feature extraction layer of the base model, extract intermediate semantic feature vectors to obtain multi-layer semantic feature vectors;

[0061] S1042: For each layer of semantic feature vectors, calculate the gating probability matrix based on the current gating parameters, and average the gating probability matrices corresponding to each layer to obtain the overall gating probability matrix;

[0062] S1043: Calculate the fusion weights for each task vector based on the overall gating probability matrix;

[0063] S1044: Based on the base model parameters, task vectors, and corresponding fusion weights, obtain the fusion model parameters for each layer.

[0064] By introducing a gating mechanism, the model can dynamically adjust the fusion weights of the task vectors based on the semantic features of the input text. Since the training set covers the inputs and outputs of different tasks, the model can learn the differences and similarities between tasks in different target domains, thereby achieving adaptive prediction for different vertical domains. The resulting gating probability matrix... G (G∈R) N Each column vector of ) G i Indicates the first i Vertical tasks and current input x The correlation between these probabilities allows for the dynamic selection of the vertical region most relevant to the current input.

[0065] In step S1041, intermediate semantic feature vectors are extracted from each feature extraction layer of the base model to obtain multi-layer semantic feature vectors. Let the input text be... x Multi-layer semantic feature vectors are represented as h x =Encoder( x ), h x ∈R dd is the number of feature extraction layers in the base model.

[0066] In step S1042, the formula for calculating the gating probability matrix is: G=σ(W•h x +b), where W∈R d×N It is a learnable weight matrix, b∈R N It is a learnable bias vector, and σ is the Sigmoid function that maps the output value to the interval [0,1].

[0067] In step S1043, after obtaining the gated probability matrix, Bernoulli sampling is first performed on the gated probability matrix to generate a binary gated vector, converting the continuous probability values ​​in the gated probability matrix into discrete values, thereby more clearly defining the relevant vertical region. Specifically, Bernoulli sampling is performed on the gated probability matrix to generate a binary gated vector g=[g1,g2,...,g N ], i.e., g i ~Bernoulli(G i ), where Bernoulli denotes Bernoulli sampling. For example, if the input text is about the medical field, the gating probability matrix G has a higher gating probability for the medical task. Through Bernoulli sampling, the g of the medical task... i More likely to be 1, g for other tasks i It is more likely to be 0.

[0068] It is understandable that for each layer of semantic feature vector, a gating vector is obtained. Based on the correlation between the current task text and each vertical domain task reflected in the gating vector, the fusion weight corresponding to each task vector can be calculated.

[0069] To reduce conflicts between task vectors, some embodiments also consider the degree of conflict between tasks when calculating the fusion weights. Specifically, for each vertical task, the similarity between that vertical task vector and every other vertical task vector is calculated. Based on the cumulative value of these similarities, a conflict factor is calculated. The formula for calculating the conflict factor between the i-th vertical task vector and other vertical task vectors is as follows:

[0070]

[0071] in, S ij This refers to the similarity between task vectors. If cosine similarity is used for calculation, the formula is: , where Δ θ i and Δ θ j It is the task vector, and cos represents the cosine similarity.

[0072] The formula for calculating the normalized weights corresponding to the task vector of the i-th vertical domain is:

[0073]

[0074] Here, ε is a constant to prevent the denominator from being zero.

[0075] The conflict factor is used to quantify the degree of conflict, making the calculation of fusion weights more accurate. It can dynamically adjust the weight allocation according to the degree of conflict between tasks, thereby better balancing the contributions of different tasks during the fusion process and avoiding performance degradation caused by conflicts between task vectors.

[0076] In step S1044, based on the base model parameters, task vector, and corresponding fusion weights, the calculation formula for the fusion model parameters is obtained as follows:

[0077]

[0078] in, Indicates the parameters of the fusion model. This represents the parameters of the base model.

[0079] It is understandable that, after the above steps, a set of fusion model parameters is obtained for each training sample, including the fusion model parameters calculated from the semantic feature vectors and gating vectors of each layer, and thus multiple new fusion models can be obtained.

[0080] In step S105, multiple sets of fusion model parameters are assigned to the base model to obtain multiple new fusion models. Then, model feedforward is performed to predict the output based on the input text of each training sample and the corresponding new fusion model. The cross-entropy loss between the predicted output distribution and the actual output distribution is calculated. L The model feedforward evaluates the model's performance using a loss function, providing gradient information for backpropagation. Then, through backpropagation, the gated gradient is estimated based on the cross-entropy loss, and the gated parameters are updated based on the gated gradient.

[0081] Here, the gate gradient is the gradient of the gate probability matrix, since the gate vector... g Since the gradients are discrete values ​​obtained through Bernoulli sampling, direct calculation of the gradient is impossible. Therefore, a pass-through estimation is used to approximate the gradient; that is, the gradient of the gate vector g is used to approximate the gradient of the gate probability matrix G. .

[0082] For the new fusion model, the gating parameters are updated through model feedforward and backpropagation. Based on other batches of training sets, steps S104 and S105 are repeated until the termination condition is met. The termination condition can be understood as the training sample size reaching a preset upper limit, or the loss function converging to a certain value.

[0083] This gated parameter update mechanism based on cross-entropy loss can effectively improve the model's adaptability to different inputs, ensure that the model can adapt to multi-domain tasks, and improve the model's generalization ability and stability.

[0084] Compared to traditional model fusion methods, the methods provided in one or more of the above embodiments have the following advantages: In terms of nonlinear conflict handling, the fusion weights are normalized through conflict awareness, avoiding parameter cancellation caused by linear averaging; in terms of information preservation, the risk of losing high-frequency features due to model distillation is avoided through direct parameter manipulation and dynamic gating mechanisms; in terms of domain adaptability, the gating probability matrix is ​​updated through Bernoulli sampling and model training, enabling adaptive adaptation to the semantics of the input text; and in terms of computational efficiency, since only the gating parameters are trained, the number of parameters is much smaller than the total number of model parameters, greatly improving computational efficiency.

[0085] Based on the above method, one or more embodiments of the present invention also provide a large language model adaptive fusion device, comprising: a vertical domain model fine-tuning module 201, configured to: fine-tune a base model based on multiple vertical domain datasets to obtain multiple vertical domain models; a task vector calculation module 202, configured to: calculate the incremental vector of the model parameters of each vertical domain model relative to the parameters of the base model, denoted as the task vector; a training data acquisition module 203, configured to: sample from the union of the multiple vertical domain datasets to obtain a training set; and a model training module 204, configured to: fix the base model, The gating parameters are trained. During training, for each training sample, the semantic feature vector of the task text is extracted based on the base model, and the gating probability matrix is ​​calculated according to the current gating parameters. The fusion weight of each task vector is calculated based on the gating probability matrix. The fusion model parameters are obtained based on the base model parameters, the task vectors and the corresponding fusion weights. The gating parameter update module 205 is configured to: assign the fusion model parameters to the base model to obtain a new fusion model; update the gating parameters for the new fusion model through model feedforward and backpropagation; repeat the training process until the termination condition is met.

[0086] The vertical model fine-tuning module 201 is also configured to perform the following: acquire multiple vertical datasets and preprocess each sample data in them, specifically: each sample data includes input text and output text, and both input text and output text are segmented and converted into numerical sequences.

[0087] In the training data acquisition module 203, multiple batch training sets are obtained by sampling from the union of the multiple vertical domain datasets. Each batch training set includes sample data from multiple vertical domains, and each round of training is based on a batch training set.

[0088] In the model training module 204, for each training sample, intermediate semantic feature vectors are extracted based on each feature extraction layer of the base model to obtain multi-layer semantic feature vectors; for each layer of semantic feature vectors, a gating probability matrix is ​​calculated based on the current gating parameters, and the gating probability matrices corresponding to each layer are averaged to obtain the overall gating probability matrix; the fusion weights for each task vector are calculated based on the overall gating probability matrix; and the fusion model parameters for each layer are obtained based on the base model parameters, task vectors, and corresponding fusion weights.

[0089] In addition, after obtaining the overall gating probability matrix, Bernoulli sampling is first performed on the gating probability matrix to generate binary gating vectors, and the fusion weights for each task vector are calculated based on the gating vectors.

[0090] One or more embodiments of the present invention also provide an electronic device that can be used to implement the methods in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0091] The memory in this embodiment of the invention is used to store various types of data to support, for example... Figure 2 The execution of the method shown.

[0092] It is understood that the memory can be volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The memory in this embodiment of the invention is capable of storing, for example... Figure 2 The computer programs corresponding to each step in the method shown are as follows. The operating system contains various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. Application programs can contain various other applications.

[0093] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where a general-purpose processor can be a microprocessor or any conventional processor, etc.

[0094] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 2The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0095] in, Figure 2 The computer program instructions corresponding to the method shown may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive fusion method for large language models, characterized in that, Includes the following steps: The base model was fine-tuned based on multiple vertical domain datasets to obtain multiple vertical domain models; Calculate the increment vector of the model parameters of each vertical domain model relative to the base model parameters, denoted as the task vector; The training set is obtained by sampling from the union of the multiple vertical domain datasets; The base model is fixed, and the gating parameters are trained. During training, for each training sample, the semantic feature vector of the input text of the training sample is extracted based on the base model, and the gating probability matrix is ​​calculated according to the current gating parameters; the fusion weight of each task vector is calculated according to the gating probability matrix. Based on the base model parameters, task vectors, and corresponding fusion weights, the fusion model parameters are obtained; The parameters of the fusion model are assigned to the base model to obtain a new fusion model. For the new fusion model, the gating parameters are updated through model feedforward and backpropagation. The training process is repeated until the termination condition is met. After obtaining the gated probability matrix, Bernoulli sampling is first performed on the gated probability matrix to generate binary gated vectors, and the fusion weights for each task vector are calculated based on the gated vectors. Each column vector G of the gated probability matrix G i This indicates the relevance between the i-th vertical task and the current input; Based on the conflict factor and gating vector between vertical domain tasks, the normalized fusion weight is calculated for each task vector, where the normalized fusion weight for the i-th vertical domain task vector is... The calculation formula is: in, Let be the value of the gating vector for the i-th vertical task. This represents the conflict factor between the i-th vertical domain task and other vertical domain tasks. The constant is N, and the number of vertical domains is N. The conflict factor between the i-th vertical domain task and other vertical domain tasks The calculation formula is: in, S ij It represents the similarity between the task vectors of the i-th vertical domain and the j-th vertical domain, where N is the number of vertical domains.

2. The adaptive fusion method for large language models as described in claim 1, characterized in that, After obtaining multiple vertical domain datasets, each sample data is preprocessed. Specifically, each sample data includes input text and output text. Both input and output texts are segmented and converted into numerical sequences.

3. The adaptive fusion method for large language models as described in claim 1, characterized in that, Multiple batch training sets are obtained by sampling from the union of the multiple vertical domain datasets. Each batch training set includes sample data from multiple vertical domains, and each round of training is based on one batch training set.

4. The adaptive fusion method for large language models as described in claim 1, characterized in that, During training, for each training sample, intermediate semantic feature vectors are extracted based on each feature extraction layer of the base model to obtain multi-layer semantic feature vectors; For each layer of semantic feature vectors, calculate the gating probability matrix based on the current gating parameters, and average the gating probability matrices corresponding to each layer to obtain the overall gating probability matrix. The fusion weights for each task vector are calculated based on the overall gating probability matrix; Based on the base model parameters, task vectors, and corresponding fusion weights, the fusion model parameters for each layer are obtained.

5. The adaptive fusion method for large language models as described in claim 1, characterized in that, Updating the gating parameters through model feedforward and backpropagation includes: predicting the output of the corresponding new fusion model based on the input text of each training sample; calculating the cross-entropy loss between the predicted output distribution and the actual output distribution; estimating the gating gradient based on the cross-entropy loss; and updating the gating parameters based on the gating gradient.

6. An adaptive fusion device for large language models, characterized in that, include: The vertical model fine-tuning module is configured to fine-tune the base model based on multiple vertical datasets to obtain multiple vertical models. The task vector calculation module is configured to calculate the incremental vector of the model parameters of each vertical domain model relative to the base model parameters, denoted as the task vector. The training data acquisition module is configured to obtain a training set by sampling from the union of the multiple vertical domain datasets. The model training module is configured to: fix the base model and train the gating parameters; During training, for each training sample, the semantic feature vector of the input text of the training sample is extracted based on the base model, and the gating probability matrix is ​​calculated according to the current gating parameters; the fusion weight of each task vector is calculated according to the gating probability matrix. Based on the base model parameters, task vectors, and corresponding fusion weights, the fusion model parameters are obtained; The gating parameter update module is configured to: assign the fusion model parameters to the base model to obtain a new fusion model; update the gating parameters for the new fusion model through model feedforward and backpropagation; repeat the training process until the termination condition is met. After obtaining the gated probability matrix, Bernoulli sampling is first performed on the gated probability matrix to generate binary gated vectors, and the fusion weights for each task vector are calculated based on the gated vectors. Each column vector G of the gated probability matrix G i This indicates the relevance between the i-th vertical task and the current input; Based on the conflict factor and gating vector between vertical domain tasks, the normalized fusion weight is calculated for each task vector, where the normalized fusion weight for the i-th vertical domain task vector is... The calculation formula is: in, Let be the value of the gating vector for the i-th vertical task. This represents the conflict factor between the i-th vertical domain task and other vertical domain tasks. The constant is N, and the number of vertical domains is N. The conflict factor between the i-th vertical domain task and other vertical domain tasks The calculation formula is: in, S ij It represents the similarity between the task vectors of the i-th vertical domain and the j-th vertical domain, where N is the number of vertical domains.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 5.

Images