Multi-source heterogeneous large language model fusion method and system

Abstract

The invention discloses a multi-source heterogeneous large language model fusion method and system. The method comprises the following steps: obtaining a group of probability distribution matrixes through pre-trained K-source large language models; aligning the group of probability distribution matrixes by using a preset alignment strategy to obtain an aligned group of probability distribution matrixes; fusing the aligned group of probability distribution matrixes to obtain a fused probability distribution matrix; and continuously training a preset target large language model according to the fused probability distribution matrix by using a preset fusion target to obtain a final model. According to the method, high-quality large language model fusion is realized.

Description

Technical Field

[0001] The present invention relates to the field of large language models, and in particular to a multi-source heterogeneous large language model fusion method and system. Background Art

[0002] With the continued success of large language models (LLMs) such as GPT and LLaMA for natural language processing (NLP) tasks, it has become a strategic imperative for companies to develop their own LLMs. However, developing LLMs is extremely expensive. In addition to requiring large amounts of training data, advanced technologies, extensive computing resources, and skilled labor, the development process also places a huge strain on energy consumption and the environment. Despite differences in structure and functionality, these LLMs have similar capabilities across the range of NLP tasks. In addition to the traditional approach of training LLMs from scratch, another option is to combine existing LLMs into a new, more powerful model, which we call Knowledge Fusion of LLMs. If successful, this fusion not only reduces the cost of initial training, but also enables the integrated model to benefit from the strengths of all LLMs. This new model can also be fine-tuned and adapted for a variety of downstream tasks. In addition, fusion can also occur between LLMs that have been fine-tuned specifically for specific tasks.

[0003] Integrating the capabilities of diverse models has been a long-standing goal of researchers, and existing methods can be divided into two main categories. First, traditional model integration techniques combine the outputs of multiple models to improve overall system performance. It should be noted that this technique does not involve explicitly merging multiple models into a new model. Commonly used model integration methods usually use weighted averaging or majority voting to integrate predictions from various models. Recently, Jiang et al. introduced an integration framework designed to leverage the different strengths of multiple open source LLMs. The framework first uses a pairwise comparison method to detect subtle differences between candidate outputs. It then combines the top-ranked candidates to produce an enhanced output that fully exploits their strengths while mitigating their weaknesses.

[0004] Secondly, weight fusion provides another way to achieve model fusion at the parameter level. Gupta et al. and Wortsman et al. fused the weights of models with the same structure, which were obtained through different training strategies or configurations to achieve overall performance improvement. Similarly, Cha et al., Rame et al. and Arpit explored the weighted average of models obtained from different configurations to improve out-of-distribution generalization. In addition, Jin et al. fused models designed for specific domains or tasks to create a general model that can handle all domains or tasks. In addition to the full parameter fusion of the model, Wang et al., Huang et al. and Zhang et al. applied linear mathematical operations to the adapter parameters to achieve better generalization performance.

[0005] The Chinese invention application with application number 202410263277.7, "A speculative reasoning acceleration method based on large model cache", has the following technical solutions: S1. Model architecture design: first design the basic architecture of the small language model; S2. Data preparation: collect and preprocess data for training the model; S3. Training process: use the deep learning framework to train the model; S4. Integrate the large language model: integrate the trained small language model with the pre-trained large language model so that it can use the KV cache of the large language model for speculative decoding. Its technical solution uses a model integration method, which requires maintaining multiple training models and executing each model during the reasoning process, which is impractical for LLMs because LLMs require a lot of memory and reasoning time. Similarly, this method is not convenient for fine-tuning, which is essential for many LLMs. Summary of the invention

[0006] In order to solve the technical problems of high video memory occupancy and long inference time in the existing large language model fusion, the present invention provides a multi-source heterogeneous large language model fusion method and system. The technical solution adopted by the present invention is:

[0007] A first aspect of the present invention provides a multi-source heterogeneous large language model fusion method, the method comprising:

[0008] Obtain a set of probability distribution matrices through pre-trained K source large language models;

[0009] Aligning the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices;

[0010] Fusing the aligned set of probability distribution matrices to obtain a fused probability distribution matrix;

[0011] The preset target large language model is continuously trained according to the fused probability distribution matrix with a preset fusion target to obtain a final model.

[0012] As a preferred solution, the training method of the K-source large language model includes:

[0013] Let t represent the The text sequence of length N sampled from ＜i =(t1,t2,…,t i-1 ) represents the sequence before the i-th token;

[0014] The K source large language models are trained respectively using the text sequence by minimizing the negative log likelihood; the expression for minimizing the negative log likelihood is: Among them, P t θ ∈R N×V represents the probability distribution matrix created by aggregating token-level predictions, where the i-th row of the probability distribution matrix represents the predicted distribution of the model for the i-th token in a vocabulary of size V, O t ∈{0,1} N×V represents a one-hot label matrix, where each row is the one-hot representation of the corresponding gold label, and D(·,·) represents the difference function between the two matrices. When it is implemented as KL divergence, D(·,·) is equivalent to the cross entropy loss at the label level.

[0015] As a preferred solution, the method of aligning the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices includes:

[0016] The words from the word segmenters corresponding to different probability distribution matrices are mapped by the minimum edit distance strategy, so as to achieve the alignment of the set of probability distribution matrices and obtain an aligned set of probability distribution matrices.

[0017] As a preferred solution, a method for fusing the aligned set of probability distribution matrices to obtain a fused probability distribution matrix includes:

[0018] When processing text t, the cross entropy loss between the probability distribution matrix and the correct label is used as an indicator of the prediction quality of the large language model;

[0019] The aligned set of probability distribution matrices are fused through a preset fusion function to obtain a fused probability distribution matrix.

[0020] As a preferred solution, the preset fusion function is any one of the following:

[0021] MinCE function, AvgCE function; where: MinCE means selecting the fused probability distribution matrix from a set of probability distribution matrices by selecting the minimum cross entropy loss, AvgCE means taking the weighted average of the set of probability distribution matrices to obtain the fused probability distribution matrix, and the weight is determined by the cross entropy loss of each matrix.

[0022] As a preferred solution, the preset fusion target is defined as:

[0023]

[0024] Among them, P t represents the fused probability distribution matrix, Fusion(·) represents a function that combines multiple matrices together, Q t represents the output distribution matrix of the target large language model on text t, and D(·,·) represents the matrix P t and the matrix Q t The difference function between .

[0025] A second aspect of the present invention provides a multi-source heterogeneous large language model fusion system, the system comprising a probability distribution matrix acquisition module, an alignment module, a fusion module and a training module;

[0026] The probability distribution matrix acquisition module is used to obtain a set of probability distribution matrices through pre-trained K source large language models;

[0027] The alignment module is used to align the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices;

[0028] The fusion module is used to fuse the aligned set of probability distribution matrices to obtain a fused probability distribution matrix;

[0029] The training module is used to continuously train the preset target large language model according to the fused probability distribution matrix with a preset fusion target to obtain a final model.

[0030] As a preferred solution, the preset alignment strategy is a minimum edit distance strategy.

[0031] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned multi-source heterogeneous large language model fusion method.

[0032] A fourth aspect of the present invention provides a computer device, comprising a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the aforementioned multi-source heterogeneous large language model fusion method.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention obtains a set of probability distribution matrices through pre-trained K source large language models, aligns the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices, fuses the aligned set of probability distribution matrices to obtain a fused probability distribution matrix, and continuously trains a preset target large language model according to the fused probability distribution matrix with a preset fusion target to obtain a final model, thereby avoiding direct fusion from the model parameter level. Therefore, it can be applicable to any model structure or training data to obtain knowledge fusion of LLMs, and since only one model is retained after fusion, low video memory and time costs can be maintained during reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of a multi-source heterogeneous large language model fusion method provided in this embodiment;

[0036] Figure 2 A schematic diagram of the application of the multi-source heterogeneous large language model fusion method provided in this embodiment. DETAILED DESCRIPTION

[0037] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;

[0038] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0039] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0040] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0041] In addition, in the description of this application, unless otherwise specified, "multiple" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0042] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0043] Example 1

[0044] Please refer to Figure 1 as well as Figure 2 This embodiment provides a multi-source heterogeneous large language model fusion method (FuseLLM), the method comprising:

[0045] S1: Obtain a set of probability distribution matrices through pre-trained K source large language models;

[0046] In a specific embodiment, the training method of the K source large language models includes:

[0047] Let t represent the The text sequence of length N sampled from ＜i =(t1,t2,…,t i-1 ) represents the sequence before the i-th token;

[0048] The K source large language models are trained respectively using the text sequence by minimizing the negative log likelihood; the expression for minimizing the negative log likelihood is: Among them, P t θ ∈R N×Vrepresents the probability distribution matrix created by aggregating token-level predictions, where the i-th row of the probability distribution matrix represents the predicted distribution of the model for the i-th token in a vocabulary of size V, O t ∈{0,1} N×V represents a one-hot label matrix, where each row is the one-hot representation of the corresponding gold label, and D(·,·) represents the difference function between the two matrices. When it is implemented as KL divergence, D(·,·) is equivalent to the cross entropy loss at the label level.

[0049] It should be noted that, through the set of probability distribution matrices, the knowledge of each model can be externalized into a unified space, essentially creating a unified probabilistic representation for the text.

[0050] S2: aligning the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices;

[0051] In a specific embodiment, the method of aligning the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices includes:

[0052] The words from the word segmenters corresponding to different probability distribution matrices are mapped by the minimum edit distance strategy, so as to achieve the alignment of the set of probability distribution matrices and obtain an aligned set of probability distribution matrices.

[0053] It should be noted that ensuring word segmentation alignment between multiple LLMs is crucial for effective knowledge fusion because it ensures the correct mapping of the probability distribution matrix. Dynamic programming is used to recursively minimize the total cost of editing a word segmentation sequence to match another sequence. If there is a one-to-one mapping between two words, the corresponding distribution will be perfectly mapped. Otherwise, the mapped distribution will degenerate into a one-hot vector. Since the words of the same sequence generated by different word segmenters usually have only limited differences, the present invention proposes to improve the success rate of word segmentation alignment by replacing the exact match (EM) constraint in the above method with a minimum edit distance (MinED) strategy, which maps words from different word segmenters based on MinED. This relaxation of the alignment constraint helps to retain a large amount of information in the distribution matrix while introducing smaller errors.

[0054] S3: fusing the aligned set of probability distribution matrices to obtain a fused probability distribution matrix;

[0055] It should be noted that the probability distribution matrix can reflect some inherent knowledge in understanding the text. Therefore, different probability distribution matrices can be used to represent the diverse knowledge contained in the same text from different LLMs. Based on this understanding, the FuseLLM method proposed in the present invention solves the LLMs fusion problem through probabilistic modeling, aiming to create a unified LLM by merging the probability distributions of the source LLMs. To achieve this goal, when a set of LLMs are used for fusion, FuseLLM performs lightweight continuous training on the target LLM, and the training data comes from the original text corpus similar to the pre-training dataset. Unlike relying solely on the CLM target, FuseLLM pays more attention to minimizing the difference between the probability distribution of the target LLM and the probability distribution of the source LLM.

[0056] In a specific embodiment, the method of fusing the aligned set of probability distribution matrices to obtain a fused probability distribution matrix includes:

[0057] When processing text t, the cross entropy loss between the probability distribution matrix and the correct label is used as an indicator of the prediction quality of the large language model;

[0058] The aligned set of probability distribution matrices are fused through a preset fusion function to obtain a fused probability distribution matrix.

[0059] In a specific embodiment, the preset fusion function is any one of the following:

[0060] MinCE function, AvgCE function; where: MinCE means selecting the fused probability distribution matrix from a set of probability distribution matrices by selecting the minimum cross entropy loss, AvgCE means taking the weighted average of the set of probability distribution matrices to obtain the fused probability distribution matrix, and the weight is determined by the cross entropy loss of each matrix.

[0061] It should be noted that the MinCE function can output a distribution matrix with a minimum cross entropy score; the AvgCE function can calculate the weighted average of the distribution matrix based on the cross entropy score.

[0062] S4: continuously training a preset target large language model according to the fused probability distribution matrix with a preset fusion target to obtain a final model;

[0063] The preset fusion target is defined as:

[0064]

[0065] Among them, P t represents the fused probability distribution matrix, Fusion(·) represents a function that combines multiple matrices together, Q t represents the output distribution matrix of the target large language model on text t, and D(·,·) represents the matrix P t and the matrix Q t The difference function between .

[0066] It should be noted that the overall goal of the continuous training of the present invention is the causal language modeling goal and fusion target The weighted combination of is as follows:

[0067] We demonstrate that the target large language model trained by our approach outperforms each source model and the baseline model in most tasks through evaluation on three benchmarks, including a total of 42 tasks spanning reasoning, common sense, and code generation. In addition, we simulate the existence of LLMs with the same structure but different functions by continuously training a single base model on several domain-specific corpora. Our approach also shows superior potential when fusing LLMs with the same structure compared to traditional model ensemble and weight merging methods.

[0068] Example 2

[0069] This embodiment provides a multi-source heterogeneous large language model fusion system, the system includes a probability distribution matrix acquisition module, an alignment module, a fusion module and a training module;

[0070] The probability distribution matrix acquisition module is used to obtain a set of probability distribution matrices through pre-trained K source large language models;

[0071] The alignment module is used to align the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices;

[0072] The fusion module is used to fuse the aligned set of probability distribution matrices to obtain a fused probability distribution matrix;

[0073] The training module is used to continuously train the preset target large language model according to the fused probability distribution matrix with a preset fusion target to obtain a final model.

[0074] In a specific embodiment, the preset alignment strategy is a minimum edit distance strategy.

[0075] In a specific embodiment, the preset fusion target is defined as:

[0076]

[0077] Among them, Pt represents the fused probability distribution matrix, Fusion(·) represents a function that combines multiple matrices together, Q t represents the output distribution matrix of the target large language model on text t, and D(·,·) represents the matrix P t and the matrix Q t The difference function between .

[0078] Example 3

[0079] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of a multi-source heterogeneous large language model fusion method described in Example 1 are implemented.

[0080] Example 4

[0081] A computer device includes a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor. When the computer program is executed by the processor, the steps of a multi-source heterogeneous large language model fusion method described in Example 1 are implemented.

[0082] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-source heterogeneous large language model fusion method, characterized in that: The method comprises: Obtain a set of probability distribution matrices through pre-trained K source large language models; Aligning the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices; Fusing the aligned set of probability distribution matrices to obtain a fused probability distribution matrix; The preset target large language model is continuously trained according to the fused probability distribution matrix with a preset fusion target to obtain a final model.

2. According to the multi-source heterogeneous large language model fusion method of claim 1, it is characterized in that: The training method of the K-source large language model includes: Let t represent the The text sequence of length N sampled from ＜i =(t1,t2,…,t i-1 ) represents the sequence before the i-th token; The K source large language models are trained respectively using the text sequence by minimizing the negative log likelihood; the expression for minimizing the negative log likelihood is: Among them, P t θ ∈R N×V represents the probability distribution matrix created by aggregating token-level predictions, where the i-th row of the probability distribution matrix represents the predicted distribution of the model for the i-th token in a vocabulary of size V, O t ∈{0,1} N×V represents a one-hot label matrix, where each row is the one-hot representation of the corresponding gold label, and D(·,·) represents the difference function between the two matrices. When it is implemented as KL divergence, D(·,·) is equivalent to the cross entropy loss at the label level.

3. The multi-source heterogeneous large language model fusion method according to claim 1, characterized in that: The method of aligning the set of probability distribution matrices by using a preset alignment strategy to obtain an aligned set of probability distribution matrices includes: The words from the word segmenters corresponding to different probability distribution matrices are mapped by the minimum edit distance strategy, so as to achieve the alignment of the set of probability distribution matrices and obtain an aligned set of probability distribution matrices.

4. The multi-source heterogeneous large language model fusion method according to claim 1, characterized in that: The method of fusing the aligned set of probability distribution matrices to obtain a fused probability distribution matrix includes: When processing text t, the cross entropy loss between the probability distribution matrix and the correct label is used as an indicator of the prediction quality of the large language model; The aligned set of probability distribution matrices are fused through a preset fusion function to obtain a fused probability distribution matrix The fused probability distribution matrix.

5. The multi-source heterogeneous large language model fusion method according to claim 4, characterized in that: The preset fusion function is any one of the following: MinCE function, AvgCE function; where: MinCE means selecting the fused probability distribution matrix from a set of probability distribution matrices by selecting the minimum cross entropy loss, AvgCE means taking the weighted average of the set of probability distribution matrices to obtain the fused probability distribution matrix, and the weight is determined by the cross entropy loss of each matrix.

6. The multi-source heterogeneous large language model fusion method according to claim 1, characterized in that: The preset fusion target is defined as: Among them, P t represents the fused probability distribution matrix, Fusion(·) represents a function that combines multiple matrices together, Q t represents the output distribution matrix of the target large language model on text t, and D(·,·) represents the matrix P t and the matrix Q t The difference function between .

7. A multi-source heterogeneous large language model fusion system, characterized by: The system includes a probability distribution matrix acquisition module, an alignment module, a fusion module and a training module; The probability distribution matrix acquisition module is used to obtain a set of probability distribution matrices through pre-trained K source large language models; The alignment module is used to align the set of probability distribution matrices using a preset alignment strategy to obtain an aligned set of probability distribution matrices; The fusion module is used to fuse the aligned set of probability distribution matrices to obtain a fused probability distribution matrix; The training module is used to continuously train the preset target large language model according to the fused probability distribution matrix with a preset fusion target to obtain a final model.

8. The multi-source heterogeneous large language model fusion system according to claim 7, characterized in that: The preset alignment strategy is the minimum edit distance strategy.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a multi-source heterogeneous large language model fusion method as described in any one of claims 1 to 6 are implemented.

10. A computer device, characterized in that: The method comprises a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein when the computer program is executed by the processor, the steps of a multi-source heterogeneous large language model fusion method as described in any one of claims 1 to 6 are implemented.

Images