Large language model training method based on scalable residual head structure

Abstract

The invention discloses a large language model training method based on a scalable residual head structure, and the method comprises the steps: S1, preparing training data which comprises first training data and second training data; s2, adding a scalable residual head structure on the original large language model; s3, defining a discriminator routing network, wherein the discriminator routing network selects whether to start a scalable residual head structure according to an output value of a sigmoid function; s4, performing fine tuning training on the large language model added with the scalable residual head structure through the first training data to obtain a first model weight; and S5, taking the first model weight as an initial weight, and performing fine tuning training on the large language model added with the scalable residual head structure by using the second training data to obtain a final scalable residual head structure model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large language models, in particular to a large language model training method based on a scalable residual head structure. BACKGROUND

[0002] Large Language Model (LLM) is one of the core technologies in the field of Natural Language Processing (NLP) in recent years. Its applications are widespread, from text generation, translation, dialogue systems to information retrieval and other scenarios. In 2017, Google published a paper "Attention is all you need", which proposed the Attention mechanism and the Transformer architecture based on this mechanism, which quickly became the mainstream model architecture for most NLP tasks. In particular, the two-stage training mode of pre-training and fine-tuning plays a key role in the development of large language models. This mode allows the model to be pre-trained on large-scale unsupervised data, and then fine-tuned on supervised task-specific data, making the model better adapt to different downstream tasks.

[0003] In recent years, more and more models such as GPT, BERT, T5 have been proposed, which have promoted the progress of language generation and understanding technology. GPT series models are known for their powerful generation capabilities, and BERT is outstanding in text understanding. These models rely on hundreds of billions or even thousands of billions of parameters, and show strong generalization ability when processing complex language tasks. However, with the increase of model size, the computing resources and memory requirements for training and inference also rise sharply, and how to efficiently fine-tune large language models and adapt to practical application scenarios has become the focus of research.

[0004] In order to improve the performance of large language models on specific tasks, various parameter-efficient fine-tuning methods have been proposed in recent years, such as LoRA, from the paper LoRA: Low-Rank Adaptation of Large Language Models. Using LORA, the training parameters are only one ten-thousandth of the overall parameters, the GPU memory usage is reduced by 2 / 3, and no additional inference time is introduced. When dealing with large pre-trained models (such as large language models, multi-modal models, etc.), directly adjusting all parameters is costly and computationally complex. LoRA provides an efficient parameter fine-tuning method.

[0005] Although the LoRA technology greatly reduces the training cost and improves the efficiency in fine-tuning large models by reducing the scale of parameter fine-tuning, this method still has some limitations when dealing with complex tasks, and these shortcomings are the technical problems that the present application wants to solve:

[0006] 1. Insufficient adaptability to complex tasks: LoRA only adjusts part of the original model's weights through low-rank matrices, which is suitable for scenarios with fewer parameters and simple tasks. However, when the model needs to handle tasks such as multi-turn dialogue, complex reasoning, and strong context association, the low-rank approximation of LoRA cannot fully capture the complex data features, resulting in suboptimal performance of the fine-tuned model. The present invention introduces a scalable residual head structure and multi-stage fine-tuning training, further enhancing the model's ability to handle complex tasks while maintaining the original model's performance.

[0007] 2. Trade-off between fine-tuning efficiency and accuracy: LoRA reduces the training amount of parameters through low-rank decomposition, but this strategy also leads to a decrease in accuracy. In tasks with high accuracy requirements, LoRA may not achieve the expected results. The present invention designs a scalable residual head, adding additional adjustment layers to the model structure, maintaining the efficiency of fine-tuning and improving the generation accuracy of the model in complex tasks.

[0008] 3. Limitations of model parameter updates: LoRA freezes most model parameters, only fine-tuning the inserted low-rank matrix and a small number of parameters, resulting in insufficient flexibility in model adjustment for certain tasks. The present invention improves the flexibility of model parameter updates by focusing on training the scalable residual head and the last layer of the Transformer decoder, enabling the model to better adapt to new data and task scenarios during fine-tuning.

[0009] 4. Large training data requirements: Although LoRA reduces the number of parameters during model fine-tuning, it still requires a large amount of high-quality training data to ensure the effectiveness and generalization ability of the fine-tuned model in actual tasks. Especially when dealing with complex tasks, insufficient data may result in poor fine-tuning results. The present invention introduces a multi-stage fine-tuning strategy and a new scalable residual head structure, pre-training with general data at the initial stage, and then fine-tuning with small-scale, high-precision data at the later stage, further reducing the dependence on large amounts of data while improving the model's performance in specific tasks.

[0010] Another related prior art is the Adapter fine-tuning technology. Adapter technology is a lightweight fine-tuning method commonly used to reduce the fine-tuning overhead of large models in different tasks and improve the model's performance in specific domains or tasks.

[0011] Adapter Tuning (Paper: Parameter-Efficient Transfer Learning for NLP), which designs an Adapter structure and embeds it into the structure of the Transformer. For each Transformer layer, two Adapter structures are added (after the projection of multi-head attention and after the second feed-forward layer). During training, the parameters of the original pre-trained model are fixed, and only the newly added Adapter structure and Layer Norm layer are fine-tuned, thereby ensuring the efficiency of training.

[0012] Whenever a new downstream task appears, an easily expandable downstream model is generated by adding an Adapter module, thereby avoiding full fine-tuning and the problem of catastrophic forgetting.

[0013] Although Adapter reduces the number of parameters required for fine-tuning through modular design, it has certain limitations when facing complex multi-stage tasks, especially for dialog, reasoning, and Retrieval Augmented Generation (RAG) tasks. In addition, since Adapter only inserts additional modules in local layers, the overall model's expression ability may be limited, making it difficult to effectively fine-tune and optimize in complex scenarios.

[0014] These shortcomings are the key problems that the present application aims to solve by designing a scalable residual head structure and a multi-stage fine-tuning training method.

[0015] With the development of RAG (Retrieval-Augmented Generation) technology, models can also dynamically retrieve content from external knowledge bases to further enhance the accuracy and diversity of generated content. In this context, how to effectively improve the effectiveness of RAG systems and further alleviate the illusion problem is an important issue for the application of large models. SUMMARY

[0016] The present application provides a scalable residual head structure-based large language model training method to solve the technical problems existing in the prior art.

[0017] To achieve the above purpose, the present application provides a scalable residual head structure-based large language model training method, which comprises:

[0018] S1: Prepare training data, including first training data and second training data;

[0019] S2: Add a scalable residual head structure to the original large language model;

[0020] S3: defining a discriminator routing network, the discriminator routing network selecting whether to enable the scalable residual head structure according to an output value of a sigmoid function;

[0021] S4: performing fine-tuning training on the large language model added with the scalable residual head structure by using first training data to obtain first model weights;

[0022] S5: using the first model weights as initial weights, performing fine-tuning training on the large language model added with the scalable residual head structure by using second training data to obtain a final scalable residual head structure model.

[0023] In an embodiment of the present application, step S1 comprises:

[0024] All collected data are divided into question and answer pair data and RAG system data. The question and answer pair data refers to fixed knowledge data arranged in advance according to the content of the knowledge base. These data are arranged in the form of questions and answers as part of the first training data. The RAG system data refers to real chat data generated by the RAG system. The RAG system data includes question and answer chat data and historical chat records. The question and answer chat data is used as another part of the first training data, and the historical chat records are used as second training data.

[0025] In an embodiment of the present application, the scalable residual head structure in step S2 comprises a dimension increasing layer, a dimension decreasing layer, a residual connection layer and a normalization layer connected in sequence, wherein:

[0026] In the dimension increasing layer, an initial vector is obtained as a first vector after passing through a first full connection layer. The dimension of the initial vector is D, and the dimension of the first vector is increased to 2D. The initial vector is a vector output from a multi-layer Transformer decoder in the original large language model. Then, the first vector is processed using a silu activation function, wherein the silu activation function f(x) is as follows:

[0027] f(x) = x * sigmoid(x)

[0028]

[0029] wherein x is the first vector;

[0030] In the dimension decreasing layer, the first vector is obtained as a second vector after passing through a second full connection layer. The dimension of the second vector is reduced to D.

[0031] In the residual connection layer, the second vector is added to the initial vector to obtain a third vector.

[0032] In the normalization layer, the third vector is normalized using RMSNorm,

[0033] The formula corresponding to RMSNorm is as follows:

[0034]

[0035] wherein RMS(a) is the root mean square value of the input vector, used for normalization, a i is the input of the model or layer, n is the number of inputs of the model or layer, g i is a learnable scaling vector, representing an adjustable weight for each dimension.

[0036] In an embodiment of the present application, the discriminator routing network of step S3 comprises an input layer, a hidden layer and an output layer, wherein:

[0037] The input layer is used to receive a three-dimensional vector, and the three dimensions of the three-dimensional vector respectively represent the input length, the task type and the context length,

[0038] The hidden layer uses a 64-dimensional fully connected layer, and the activation function of the hidden layer is a relu function, which is used to obtain a nonlinear transformation, and the relu function is f(y) = max(0, y), wherein y is the input of the relu function,

[0039] The output layer outputs a scalar, and the result of the scalar is the probability value output by the discriminator routing network, and the activation function of the output layer is a sigmoid function, which is used to generate a value between 0 and 1,

[0040] The output logic of the discriminator routing network is to binarize the probability value, and if the probability value is greater than or equal to 0.5, the residual head is activated, otherwise the residual head is not activated.

[0041] In an embodiment of the present application, steps S4 and S5 comprise:

[0042] Step 1: loading the pre-training weight of the original large model of the open source to the scalable residual head structure,

[0043] Step 2: training the scalable residual head structure loaded with the pre-training weight by using the first training data, and obtaining the intermediate model weight after completing the training,

[0044] The training method and parameters in steps 1 and 2 include the following (1) to (3):

[0045] (1) freezing all model parameters except the scalable residual head structure, the discriminator routing network and the last layer of the multi-layer Transformer decoder, and only training the parameters of the scalable residual head, the discriminator routing network and the last layer of the multi-layer Transformer decoder;

[0046] (2) Use deepspeed for single machine multi-card training for 5 rounds, with an initial learning rate of 1e-4;

[0047] (3) Train in combination with lora, unfreeze the parameters of the scalable residual head structure on the basis of lora, and train the parameters of lora and the scalable residual head at the same time;

[0048] Step 3: Load the intermediate model weight to the scalable residual head structure, at this time the scalable residual head structure and the discriminator routing network are endowed with the trained model weight;

[0049] Step 4: Through the second training data, the scalable residual head structure loaded with the intermediate model weight is trained, and the final model weight is obtained after training,

[0050] Among them, the training method and parameters in step 3 and step 4 include the following (1)~(3):

[0051] (1) Freeze all model parameters except the scalable residual head structure, the discriminator routing network and the last layer of the multi-layer Transformer decoder, only train the parameters of the scalable residual head, the discriminator routing network and the last layer of the multi-layer Transformer decoder,

[0052] (2) Use deepspeed for single machine multi-card training for 5 rounds, with an initial learning rate of 5e-5,

[0053] (3) Train in combination with lora, unfreeze the parameters of the scalable residual head structure and the discriminator routing network on the basis of lora, and train the parameters of lora and the scalable residual head and the discriminator routing network.

[0054] The scalable residual head structure-based large language model training method provided by the application has the following significant beneficial effects compared with the prior art:

[0055] 1. Improve model performance in complex tasks:

[0056] Technical scheme: By adding a scalable residual head structure to the large language model, additional adjustment layers are added based on the original model capabilities. The scalable residual head can effectively enhance information transmission and feature expression when processing complex conversations, multi-round reasoning and other tasks, improving the processing capacity for complex tasks.

[0057] Beneficial effect: The model can better understand and generate complex conversation content, improving accuracy and effectiveness in multi-round conversation and reasoning tasks.

[0058] 2. Improve adaptability to complex tasks:

[0059] Technical solution: By adding a discriminator routing network in a large language model, it can dynamically determine whether to activate the scalable residual head, and adjust the model's behavior according to different task types (such as question answering, generation, and summary) and context information.

[0060] Beneficial effect: This dynamic adaptation mechanism allows the model to balance simple and complex tasks, avoiding unnecessary activation of additional network structures and reducing waste of computing resources.

[0061] 3. Enhance the support ability for long context tasks:

[0062] Technical solution: In multi-stage fine-tuning training, RAG system data (including retrieval and generation content) are combined, and two-stage training strategy is used to process question and answer pair data and data with historical chat records.

[0063] Beneficial effect: When processing long context tasks that combine text and knowledge base retrieval content, the model can better integrate and utilize different types of data, improving the understanding and generation ability of long context information and enhancing the overall task performance.

[0064] 4. Reduce the redundancy of the model:

[0065] Technical solution: The discriminator routing network can avoid meaningless residual head activation by determining task requirements, reducing redundant calculations and potential overfitting problems of the model.

[0066] Beneficial effect: This way effectively utilizes computing resources, enabling additional modules only when necessary, and by only activating part of the path (such as fine-tuning the weights of residual head and discriminator), the discriminator reduces the dependence on the entire model parameters, reducing training and inference costs.

[0067] 5. Improve the global optimization ability of the model:

[0068] Technical solution: The invention focuses on training the scalable residual head and the last layer of the Transformer decoder, breaking through the limitations of local optimization and optimizing the overall performance of the model. Compared with traditional local fine-tuning methods, it balances global and local fine-tuning strategies, and can better adjust model parameters.

[0069] Beneficial effect: When facing tasks that require global information integration, the model can adjust and optimize more comprehensively, improve the adaptability to complex data and tasks, and significantly improve the overall performance.

[0070] 6. Improve the generalization ability and efficiency of fine-tuning training:

[0071] Technical solution: Adopt a multi-stage training strategy, divide the training data into question and answer pair data and RAG system data, fine-tune them respectively, and combine them to improve the model's generalization ability and adaptability.

[0072] Beneficial effects: The model can better adapt to new data and task changes, reducing the risk of training overfitting and improving the model's performance in new tasks and data environments. In addition, the multi-stage training process optimizes training efficiency, allowing the model to maintain high performance while also reducing training time.

[0073] 7. Reduce dependence on large amounts of training data:

[0074] Technical solution: By introducing a multi-stage fine-tuning strategy and a new scalable residual head structure, in different training stages, different training structured data, and the last layer of the scalable parameter residual head and the connected multi-layer Transformer decoder are trained, reducing the dependence on large amounts of data, and the training and migration effect is better than inserting independent small network modules in each layer of Transformer, improving the model's performance in specific tasks

[0075] Beneficial effects: Data annotation costs are significantly reduced, as this technology does not require large amounts of manually annotated data to achieve high-quality fine-tuning results. Through multi-stage fine-tuning, the model can still adapt to task requirements and improve performance with a small amount of data. This not only shortens the training time, but also reduces the economic cost of obtaining large-scale labeled data, improving the model's practicality in data-deficient scenarios. At the same time, this method also improves the model's migration ability in different tasks and domains, allowing the model to achieve good generalization ability even in resource-constrained situations.

[0076] In summary, the technical solution of the present application introduces a scalable residual head structure and a multi-stage fine-tuning strategy to optimize the performance of large language models in complex tasks, long context data processing, small sample data migration, and global optimization, improving the model's processing capacity and generalization ability, reducing the cost of large language model projects, and enhancing the effectiveness and efficiency in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0078] Figure 1A schematic diagram of a scalable residual head structure-based large language model training method according to an embodiment of the present application;

[0079] Figure 2 A design example diagram of a scalable residual head structure according to an embodiment of the present application.

[0080] Figure 3 A design example of a discriminator routing network according to an embodiment of the present application. DETAILED DESCRIPTION

[0081] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0082] The present application proposes a large language model training method based on a scalable residual head structure to address the limitations of LoRA and Adapter fine-tuning methods in the prior art, aiming to solve the following technical problems:

[0083] 1. Improve model performance in complex tasks: Existing LoRA and Adapter technologies effectively reduce fine-tuning parameters, but they cannot fully capture and express complex data features in complex dialogues, multi-round reasoning, and other tasks, leading to decreased model performance. The present application enhances the model's expression ability when dealing with complex tasks by adding a scalable residual head structure to the original model, making the model more adaptable and accurate in complex scenarios.

[0084] 2. Improve model support for complex and long context tasks: Existing technologies such as the Adapter method have weak processing capabilities for complex and long context data, making it difficult to handle tasks that combine text and knowledge base retrieval content. The present application effectively improves the model's performance in multi-modal tasks by introducing RAG system data (retrieval and generation enhancement) during training and using a multi-stage training mechanism, enabling better coordination and integration between different types of data.

[0085] 3. Enhance the global optimization ability of the model: The local update strategy of LoRA and Adapter limits the global optimization of the model, especially in tasks that require more extensive adjustments to the entire model. To address this, the present application proposes a multi-level fine-tuning scheme that focuses on training the scalable residual head and the last layer of the Transformer decoder, breaking through the limitations of local optimization by amplifying the parameters of the scalable residual head, and improving the adaptability and flexibility of the model as a whole.

[0086] 4. Improve the generalization ability and training efficiency of the model: the existing fine-tuning method has weak generalization ability of the model in dealing with new data or frequent knowledge update tasks, and is prone to overfitting or performance decline. The present application designs a set of phased training process by combining question and answer pair data and RAG system data, so that the model can better adapt to new data and task changes, and improve the generalization ability and efficiency of fine-tuning training.

[0087] 5. Reduce dependence on large amount of training data: the present application reduces the dependence on large amount of data by introducing a multi-stage fine-tuning strategy and a new scalable residual head structure, and training the scalable parameter residual head and the last layer of the connected multi-layer Transformer decoder in different training stages and different structured data, which is better than inserting an independent small network module in each layer of Transformer, and improves the performance of the model in specific tasks.

[0088] By solving the above technical problems, the present application aims to develop a large language model fine-tuning method that can perform well in complex and long context tasks, and has efficient fine-tuning and global optimization ability, which is suitable for various natural language processing task scenarios.

[0089] The core of the technical scheme of the present application includes three parts: scalable residual head design, multi-stage training process and discriminator routing network. The scalable residual head structure is introduced based on the large language model, and the discriminator routing network is used to determine whether to activate the scalable residual head. The scalable residual head uses multiple fully connected layers to process the output of the model, and uses activation functions and normalization techniques to improve the learning ability of the model.

[0090] Figure 1 The schematic diagram of the large language model training method based on the scalable residual head structure of an embodiment of the present application is shown in Figure 1 The present application provides a large language model training method based on a scalable residual head structure, which aims to improve the generation quality and question and answer accuracy of the model through phased data training, which includes:

[0091] S1: Prepare training data, which includes first training data and second training data;

[0092] S2: Add a scalable residual head structure to the original large language model;

[0093] S3: Define a discriminator routing network, which selects whether to enable the scalable residual head structure according to the output value of the sigmoid function;

[0094] S4: Fine-tune the large language model with the added scalable residual head structure with the first training data to obtain the first model weight;

[0095] S5: using the first model weight as the initial weight, fine-tuning training the large language model added with the scalable residual head structure using the second training data to obtain a final scalable residual head structure model.

[0096] In an embodiment of the present application, step S1 comprises:

[0097] All collected data is divided into question and answer pair data and RAG system data. The question and answer pair data refers to fixed knowledge data arranged in advance according to the knowledge base content. These data are arranged in the form of questions and answers as part of the first training data. The RAG system data refers to real chat data generated through the RAG system data. The RAG system data includes question and answer chat data and historical chat records. The question and answer chat data is another part of the first training data, and the historical chat records are the second training data.

[0098] It should be noted that the RAG system data also includes the knowledge base content retrieved through the system as prior knowledge. These retrieved knowledge base contents may be correct or incorrect.

[0099] I. First training data example:

[0100] instruction: question 1

[0101] system: system prompt word

[0102] output: answer 1

[0103] instruction: question 2

[0104] system: system prompt word

[0105] output: answer 2

[0106] instruction: question 3

[0107] system: system prompt word

[0108] output: answer 3

[0109] ...

[0110] II. Second training data example:

[0111] instruction: question 1

[0112] system: system prompt word + retrieved knowledge base content 1

[0113] history: historical chat question and answer record 1

[0114] output: Answer 1

[0115] instruction: Question 2

[0116] system: System prompt words + Retrieved knowledge base content 2

[0117] history: Historical chat question and answer record 2

[0118] output: Answer 2

[0119] instruction: Question 3

[0120] system: System prompt words + Retrieved knowledge base content 3

[0121] history: Historical chat question and answer record 3

[0122] output: Answer 3

[0123] …

[0124] In an embodiment of the present application, the scalable residual head structure in step S2 includes a dimension increasing layer, a dimension reducing layer, a residual connection layer and a normalization layer connected in turn, wherein:

[0125] In the dimension increasing layer, an initial vector is obtained after passing through a first fully connected layer, and the dimension of the initial vector is D, and the dimension of the first vector is increased to 2D, the initial vector is a vector output from a multi-layer Transformer decoder in the original large language model, and then the first vector is processed using a silu activation function, the silu activation function is used to enhance the expression ability of the model in the current residual head structure, while maintaining the integrity of the information, wherein the silu activation function f(x) is as follows:

[0126] f(x) = x * sigmoid(x)

[0127]

[0128] where x is the first vector;

[0129] In the dimension reducing layer, a second vector is obtained after the first vector passes through a second fully connected layer, and the dimension of the second vector is reduced to D;

[0130] In the residual connection layer, the second vector is added to the initial vector to obtain a third vector;

[0131] In the normalization layer, the third vector is normalized using RMSNorm,

[0132] The formula corresponding to RMSNorm is as follows:

[0133]

[0134] wherein RMS(a) is the root mean square value of the input vector, used for normalization, a i is the input of the model or layer, n is the number of inputs of the model or layer, g i is a learnable scaling vector, representing an adjustable weight for each dimension.

[0135] Figure 2 is a design example of the scalable residual head structure of an embodiment of the present application, as Figure 2 shown, the output obtained from the multi-layer Transformer decoder of the original model is denoted as x, x passes through the Fc full connection layer with a dimension of hidden_size, to complete the preliminary information mapping in the dimension of hidden_size, and then passes through the Fc full connection layer with a dimension of hidden_sizx2 for dimension increase, wherein 2 times hidden_size can continue to be enlarged as a scaling parameter n to balance local fine-tuning and full parameter fine-tuning, here only as a minimum example to minimize the training parameters. After transformation by the activation function silu, it again passes through the Fc full connection layer with a dimension of hidden_size for dimension reduction, and is added to the x obtained at the beginning after transformation by the activation function silu, and the output obtained after RMSNorm normalization is returned to the Lm_head of the original model, that is, the final output layer of the original model.

[0136] In an embodiment of the present application, the discriminator routing network of step S3 includes an input layer, a hidden layer and an output layer, wherein:

[0137] The input layer is used to receive a three-dimensional vector, and the three dimensions of the three-dimensional vector respectively represent the input length, the task type and the context length,

[0138] The hidden layer uses a 64-dimensional full connection layer, and the activation function of the hidden layer is a relu function, which is used to obtain a nonlinear transformation, and the relu function is f(y) = max(0, y), wherein y is the input of the relu function,

[0139] The output layer outputs a scalar, and the result of the scalar is the probability value output by the discriminator routing network, and the activation function of the output layer is a sigmoid function, which is used to generate a value between 0 and 1,

[0140] The output logic of the discriminator routing network is to binarize the probability value, and if the probability value is greater than or equal to 0.5, the residual head is activated, otherwise the residual head is not activated.

[0141] As Figure 3A design example of a discriminator routing network of an embodiment of the application is shown.

[0142] In an embodiment of the application, steps S4 and S5 include:

[0143] Step 1: load the pre-training weights of the original large model of the open source to the scalable residual head structure,

[0144] Step 2: model training of the scalable residual head structure loaded with the pre-training weights is performed by using the first training data, and the intermediate model weight is obtained after the training is completed,

[0145] Among them, the training method and parameters in step 1 and step 2 include the following (1)~(3):

[0146] (1) freeze all model parameters except the scalable residual head structure, the discriminator routing network and the last layer of the multi-layer Transformer decoder, and only train the parameters of the scalable residual head, the discriminator routing network and the last layer of the multi-layer Transformer decoder;

[0147] (2) use deepspeed for single-machine multi-card training for 5 rounds, and the initial learning rate is 1e-4;

[0148] (3) combine lora for training, unfreeze the parameters of the scalable residual head structure based on lora, and train the parameters of lora and the scalable residual head at the same time;

[0149] Step 3: load the intermediate model weight to the scalable residual head structure, at this time the scalable residual head structure and the discriminator routing network are assigned with the trained model weight;

[0150] Step 4: model training of the scalable residual head structure loaded with the intermediate model weight is performed by using the second training data, and the final model weight is obtained after the training is completed,

[0151] Among them, the training method and parameters in step 3 and step 4 include the following (1)~(3):

[0152] (1) freeze all model parameters except the scalable residual head structure, the discriminator routing network and the last layer of the multi-layer Transformer decoder, and only train the parameters of the scalable residual head, the discriminator routing network and the last layer of the multi-layer Transformer decoder,

[0153] (2) use deepspeed for single-machine multi-card training for 5 rounds, and the initial learning rate is 5e-5,

[0154] (3) Training combined with LORA, unfreezing the parameters of the scalable residual head structure and discriminator routing network based on LORA, and training the parameters of LORA, the scalable residual head, and the discriminator routing network.

[0155] The method of scalable residual head structure and multi-stage fine-tuning training proposed in the present application aims to improve the performance of large language models and solves some deficiencies in the prior art. In addition to the present application scheme, the following are several possible alternative schemes, which can also achieve similar purposes of the invention:

[0156] 1. Graph Neural Network (GNN) combined with Transformer

[0157] Scheme description: Introduce graph neural network (GNN) structure into the original Transformer model to enhance the model's understanding ability of graph structure data or complex relationship data. GNN can handle complex relationships between nodes and integrate this information into the representation of Transformer.

[0158] Implementation method:

[0159] Insert GNN modules in each layer of Transformer to process the graph structure features of input data.

[0160] GNN modules extract relationship information between nodes through graph convolution operations and integrate it with the representation of Transformer.

[0161] Joint training of graph neural network and Transformer to improve the model's understanding ability of complex data relationships.

[0162] Effect: This scheme can improve the model's ability to handle complex data relationships, especially in tasks that require in-depth understanding of graph structure or relationship data.

[0163] 2. Adaptive weight adjustment

[0164] Scheme description: Dynamically adjust the learning rate of different layers of the model during training, and adaptively adjust according to the importance of the layer and the requirements of the task. This method can make the model more flexible in adjusting the weights of different layers during fine-tuning to adapt to the requirements of complex tasks.

[0165] Implementation method:

[0166] Introduce an adaptive mechanism to dynamically adjust the learning rate of each layer according to task requirements.

[0167] Automatically adjust the weight update strategy by monitoring the model's performance and gradient changes during training.

[0168] Optimize the learning rate adjustment strategy using a feedback mechanism to achieve the best training results.

[0169] Effect: This approach can improve the flexibility and adaptability of the model during fine-tuning, making it better optimized for complex tasks.

[0170] 3. Mixture of Experts

[0171] Approach Description: Use a mixture of experts model, where the model is composed of multiple expert networks, each focusing on handling different types of data or tasks. Based on the characteristics of the input data, select the appropriate expert for processing.

[0172] Implementation Method:

[0173] Construct multiple expert networks in the model, each focusing on different tasks or data types.

[0174] Through a gating mechanism, select the most suitable expert network for calculation based on the characteristics of the input data.

[0175] Optimize the gating mechanism and expert networks during training to improve the overall performance of the model.

[0176] Effect: This approach allows the model to be more targeted when handling multiple tasks, improving the effectiveness of processing specific tasks, while allowing the model to refine and optimize in complex tasks.

[0177] The large language model training method based on the scalable residual head structure provided by the present application has the following significant beneficial effects compared with the prior art:

[0178] 1. Improve model performance in complex tasks:

[0179] Technical Solution: By adding a scalable residual head structure to the large language model, additional adjustment layers are added while maintaining the original model's capabilities. The scalable residual head can effectively enhance information transmission and feature expression when the model handles complex conversations, multi-round reasoning, and other tasks, improving the ability to handle complex tasks.

[0180] Beneficial Effects: The model can better understand and generate complex conversation content, improving accuracy and effectiveness in multi-round conversations and reasoning tasks.

[0181] 2. Improve adaptability to complex tasks:

[0182] Technical solution: By adding a discriminator routing network in the large language model, it can dynamically determine whether to activate the scalable residual head, and adjust the model's behavior according to different task types (such as question and answer, generation, and summary) and context information.

[0183] Beneficial effects: This dynamic adaptation mechanism allows the model to balance simple and complex tasks, avoiding unnecessary activation of additional network structures and reducing waste of computing resources.

[0184] 3. Enhance the support ability for long context tasks:

[0185] Technical solution: In the multi-stage fine-tuning training, RAG system data (including retrieval and generation content) are combined, and two-stage training strategy is used to process question and answer pair data and data with historical chat records.

[0186] Beneficial effects: When processing long context tasks that combine text and knowledge base retrieval content, the model can better integrate and utilize different types of data, improving the understanding and generation ability of long context information, and enhancing the overall task performance.

[0187] 4. Reduce the redundancy of the model:

[0188] Technical solution: The discriminator routing network can avoid meaningless residual head activation by determining task requirements, reducing redundant calculations and potential overfitting problems of the model.

[0189] Beneficial effects: This way effectively utilizes computing resources, enabling additional modules only when necessary, and reducing the dependence on the entire model parameters by only activating part of the path (such as fine-tuning the weights of residual head and discriminator), reducing training and inference costs.

[0190] 5. Improve the global optimization ability of the model:

[0191] Technical solution: The invention focuses on training the scalable residual head and the last layer of the Transformer decoder, breaking through the limitations of local optimization and optimizing the overall performance of the model. Compared with traditional local fine-tuning methods, it balances global and local fine-tuning strategies, and can better adjust model parameters.

[0192] Beneficial effects: When facing tasks that require global information integration, the model can adjust and optimize more comprehensively, improve the adaptability to complex data and tasks, and significantly improve the overall performance.

[0193] 6. Improve the generalization ability and efficiency of fine-tuning training:

[0194] Technical solution: Adopting a multi-stage training strategy, the training data is divided into question and answer pair data and RAG system data, fine-tuning is performed respectively, and the two are combined to improve the model generalization ability and adaptability.

[0195] Beneficial effects: The model can better adapt to new data and task changes, reducing the risk of training overfitting and improving the model's performance in new tasks and data environments. In addition, the multi-stage training process optimizes training efficiency, allowing the model to maintain high performance while also reducing training time.

[0196] 7. Reduce dependence on large amounts of training data:

[0197] Technical solution: By introducing a multi-stage fine-tuning strategy and a new scalable residual head structure, in different training stages, different training structured data, the last layer of the scalable parameter residual head and the connected multi-layer Transformer decoder are trained, reducing the dependence on large amounts of data, and the training and migration effect is better than inserting independent small network modules in each layer of Transformer, improving the model's performance in specific tasks

[0198] Beneficial effects: Data annotation costs are significantly reduced, as this technology does not require large amounts of manually annotated data to achieve high-quality fine-tuning results. Through multi-stage fine-tuning, the model can still adapt to task requirements and improve performance with a small amount of data. This not only shortens the training time, but also reduces the economic cost of obtaining large-scale labeled data, improving the model's practicality in data-deficient scenarios. At the same time, this method also improves the model's migration ability in different tasks and domains, so that the model can still achieve good generalization ability even in resource-constrained situations.

[0199] In summary, the technical solution of the present application optimizes the performance of large language models in complex tasks, long context data processing, small sample data migration, and global optimization by introducing scalable residual head structures and multi-stage fine-tuning strategies, improving the model's processing capacity and generalization ability, reducing the landing cost of large language model projects, and enhancing the effectiveness and efficiency in practical applications.

[0200] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily required to implement the present application.

[0201] Those skilled in the art can understand that the modules in the device in the embodiments can be distributed in the device of the embodiments as described in the embodiments, or can be changed and located in one or more devices different from the present embodiment. The modules of the above-mentioned embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0202] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features therein can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A large language model training method based on a scalable residual head structure, characterized in that: include: S1: Prepare training data, where the training data includes first training data and second training data; S2: Add a scalable residual head structure to the original large language model; S3: Define the discriminator routing network, which chooses whether to enable the scalable residual head structure based on the output value of the sigmoid function; S4: Fine-tune the large language model with the scalable residual head structure using the first training data to obtain a first model weight. S5: Using the first model weight as the initial weight, fine-tune the large language model with the scalable residual head structure using the second training data to obtain the final scalable residual head structure model.

2. The large language model training method based on a scalable residual head structure according to claim 1, characterized in that: Step S1 includes: All collected data are divided into question-answer pair data and RAG system data. Question-answer pair data refers to fixed knowledge data organized in advance according to the content of the knowledge base. These data are organized in the form of questions and answers as part of the first training data. RAG system data refers to real chat data generated by RAG system data. RAG system data includes question-answer chat data and historical chat records. Question-answer chat data is used as another part of the first training data, and historical chat records are used as the second training data.

3. The large language model training method based on a scalable residual head structure according to claim 1, characterized in that: The scalable residual head structure in step S2 includes a dimensionality increase layer, a dimensionality reduction layer, a residual connection layer, and a normalization layer connected in sequence, wherein: In the dimension-raising layer, the initial vector passes through the first fully connected layer to obtain the first vector. The dimension of the initial vector is D. The dimension of the first vector is increased to 2D. The initial vector is the vector output from the multi-layer Transformer decoder in the original large language model. The first vector is then processed using the silu activation function, where the silu activation function f(x) is as follows: f(x)=x*sigmoid(x) Where x is the first vector; In the dimensionality reduction layer, the first vector passes through the second fully connected layer to obtain the second vector, and the dimension of the second vector is reduced to D; In the residual connection layer, the second vector is added to the initial vector to obtain the third vector; In the normalization layer, RMSNorm is used to normalize the third vector. The formula corresponding to RMSNorm is as follows: Where RMS(a) is the root mean square value of the input vector, used for normalization, a i is the input of the model or layer, n is the number of inputs of the model or layer, g i is a learnable scaling vector representing the adjustable weights for each dimension.

4. The large language model training method based on a scalable residual head structure according to claim 1, characterized in that: The discriminator routing network of step S3 includes an input layer, a hidden layer, and an output layer, wherein: The input layer is used to receive a three-dimensional vector. The three dimensions of the three-dimensional vector represent the input length, task type, and context length respectively. The hidden layer uses a 64-dimensional fully connected layer. The activation function of the hidden layer is the relu function, which is used to obtain nonlinear transformation. The relu function is: f(y) = max(0, y), where y is the input of the relu function. The output layer outputs a scalar, the result of which is the probability value output by the discriminator routing network. The activation function of the output layer is the sigmoid function, which is used to generate a value between 0 and 1. The output logic of the discriminator routing network is to binarize the probability value. If the probability value is greater than or equal to 0.5, the residual head is activated; otherwise, the residual head is not activated.

5. The large language model training method based on a scalable residual head structure according to claim 1, characterized in that: Steps S4 and S5 include: Step 1: Load the pre-trained weights of the open-source original large model into the scalable residual head structure. Step 2: Use the first training data to train the scalable residual head structure loaded with pre-trained weights, and obtain the intermediate model weights after training. The training methods and parameters in steps 1 and 2 include the following (1) to (3): (1) Freeze all model parameters except the scalable residual head structure, the discriminator routing network, and the last layer of the multi-layer Transformer decoder, and only train the parameters of the scalable residual head, the discriminator routing network, and the last layer of the multi-layer Transformer decoder; (2) Use deepspeed to perform single-machine multi-GPU training for 5 rounds with an initial learning rate of 1e-4; (3) Combined with LoRa for training, the parameters of the scalable residual head structure are unfrozen based on LoRa, and the parameters of LoRa and the scalable residual head are trained at the same time; Step 3: Load the intermediate model weights into the scalable residual head structure. At this time, the scalable residual head structure and the discriminator routing network are assigned the trained model weights. Step 4: Use the second training data to train the scalable residual head structure loaded with the intermediate model weights. After the training is completed, the final model weights are obtained. The training methods and parameters in steps 3 and 4 include the following (1) to (3): (1) Freeze all model parameters except the scalable residual head structure, the discriminator routing network, and the last layer of the multi-layer Transformer decoder, and only train the parameters of the scalable residual head, the discriminator routing network, and the last layer of the multi-layer Transformer decoder. (2) Use deepspeed to perform single-machine multi-GPU training for 5 rounds, with an initial learning rate of 5e-5. (3) Combined with LoRa for training, the parameters of the scalable residual head structure and the discriminator routing network are unfrozen based on LoRa, and the parameters of LoRa, the scalable residual head, and the discriminator routing network are trained at the same time.

Images