Large language model training method, system and device and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a large language model training method, system and device and a storage medium, and the method comprises the steps: collecting task samples based on task difficulty and task diversity demands; obtaining a support set and a query set of the task samples, and performing data enhancement processing on the support set; performing internal loop training on the meta-model by using the support set, and performing external loop training on the meta-model subjected to the internal loop training by using the query set to obtain a meta-gradient; and summarizing the element gradients, and updating the global model based on the element gradient summarized data by using an adaptive optimizer. The adaptive capacity of the large language model to the task type is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a training method, system, device and storage medium for a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have made significant progress in the field of natural language processing. They are able to generate coherent text, understand complex instructions, and show strong context adaptation capabilities. However, despite the powerful capabilities of these models, existing technologies still face many challenges in handling new tasks and domain adaptability.

[0003] The current state of the art is not optimistic. On the one hand, traditional transfer learning methods usually require a large amount of labeled data for fine-tuning, which is particularly difficult in new tasks or data-scarce situations, and it is difficult to quickly adapt to the needs of new tasks. On the other hand, although meta-learning methods such as Model-Agnostic Meta-Learning (MAML) have shown excellent performance in small sample learning, their application is still subject to many limitations when dealing with large language models, making it difficult to fully realize their potential.

[0004] Therefore, how to maintain the powerful performance of large language models while improving their ability to adapt to new tasks and fields is an important issue that needs to be urgently addressed in the field of natural language processing. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a large-scale language model training method, system, device and storage medium to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for training a large language model, comprising: Collect task samples based on task difficulty and task diversity requirements; Obtain the support set and query set of the task sample, and perform data augmentation on the support set; The meta-model is trained in an inner loop using the support set, and the meta-model trained in an inner loop is trained in an outer loop using the query set to obtain the meta-gradient. Aggregate meta-gradients and use the adaptive optimizer to update the global model based on the meta-gradient summary data.

[0007] In an optional implementation, collecting task samples based on task difficulty and task diversity requirements includes: Obtaining a task difficulty, where the task difficulty is the loss value of the model on the task during historical training; Determine task diversity, where the task diversity is the average distance between the task and a sampled task set; The sampling probability of the task is determined based on the task difficulty and task diversity. The sampling probability is calculated as follows:

[0008] in, To balance the difficulty, is the diversity hyperparameter, is the task difficulty, For task diversity.

[0009] In an optional implementation, obtaining the task difficulty includes: The calculation formula for task difficulty is:

[0010] in, It is a model On Task The validation set The accuracy of It is the highest accuracy among all tasks; The task difficulty is normalized to obtain the quantitative value of the task difficulty.

[0011] In an optional implementation, confirming the diversity of tasks includes: The TF-IDF method is used to represent the task as a feature vector:

[0012] in, Representative words On Task The frequency of words in ; Inverse Document Frequency , The total number of tasks representing candidates; Computational tasks With the sampled task set The cosine similarity of each task in:

[0013] in, It's a task The TF-IDF vector of It's a task TF-IDF vector of Calculate task diversity based on the cosine similarity:

[0014] The task diversity is the task The average distance to the sampled task set.

[0015] In an optional implementation, performing data enhancement processing on the support set includes: Replace the words in the sample data with their synonyms with a probability of 10%; Randomly insert context-related words into sentences in sample data with a probability of 10%; Randomly delete words from sentences in sample data with a probability of 10%; Translate the text data into Chinese and then back to English with a probability of 20%.

[0016] In an optional implementation, inner loop training is performed on the meta-model using the support set, and outer loop training is performed on the meta-model after inner loop training using the query set to obtain a meta-gradient, including: Initialize the metamodel parameters, let ; Set the optimization function: ; Based on the optimization function, the support set of data enhancement is used Perform inner loop training on the meta-model: for For each task in , perform K steps of gradient descent to adapt to the task: ; Compute the gradient: ; Update metamodel parameters: ; Use the query set to train the metamodel in an outer loop:

[0017] in, is the outer loop learning rate, is the task batch size, is an adaptive optimizer, represent The loss function is It is through Task-specific parameters after in-loop adaptation, is the i-th sample in the query set, represents the inner loop learning rate, represent The gradient of Represents the distribution of tasks;

[0018] in Represents the first samples and their labels, Represents the number of query set samples.

[0019] In an optional embodiment, the meta-gradients are aggregated, and the global model is updated based on the meta-gradient aggregate data using an adaptive optimizer, including: The update rule is defined, and the gradient for time step t is , the update rules include:

[0020] in are the shared first and second order momentum, represents the current gradient, Represents a small constant to prevent division by zero errors, are the bias-corrected first- and second-order momentum estimates, are different values ​​of the momentum decay rate, is the learning rate, is the weight decay coefficient.

[0021] In a second aspect, the present invention provides a large language model training system, comprising: Sampling module, used to collect task samples based on task difficulty and task diversity requirements; The processing module is used to obtain the support set and query set of the task sample and perform data enhancement processing on the support set; A training module, used for performing inner loop training on the meta-model using the support set, and performing outer loop training on the meta-model trained by the inner loop using the query set to obtain a meta-gradient; The update module is used to summarize the meta-gradients and use the adaptive optimizer to update the global model based on the meta-gradient summary data.

[0022] In a third aspect, a device is provided, comprising: Memory, used to store training programs for large language models; A processor is used to implement the steps of the large language model training method provided in the first aspect when executing the large language model training program.

[0023] In a fourth aspect, a computer-readable storage medium is provided, on which a training program for a large language model is stored. When the training program for a large language model is executed by a processor, the steps of the training method for a large language model provided in the first aspect are implemented.

[0024] The beneficial effects of the present invention are that the training method, system, device and storage medium of the large language model provided by the present invention perform task sampling on the basis of considering the difficulty and diversity of tasks, thereby improving the sample comprehensiveness of the data set, and increasing the sample size of the data set required for inner loop training through data enhancement processing, thereby improving the training effect; further, by combining the inner and outer loop training and the adaptive optimizer, the momentum state sharing and adaptive parameter adjustment of the inner and outer loop training are realized, thereby improving the adaptability of the large language model to the task type.

[0025] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0028] Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.

[0029] Figure 3 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0032] The key terms appearing in the present invention are explained below.

[0033] Large Language Models (LLMs) are natural language processing models based on deep learning that can understand and generate natural language text. They learn the grammar, semantics, and various language features of a language by training on large-scale text data, so that they can perform a variety of language tasks such as text generation, translation, summarization, question answering, etc. LLMs use neural network-based models and often use natural language processing (NLP) techniques to process and calculate their outputs.

[0034] Traditional transfer learning methods usually require a large amount of labeled data for fine-tuning and are difficult to quickly adapt to new tasks.

[0035] Meta-learning methods such as Model-Agnostic Meta-Learning (MAML) have shown excellent performance in few-shot learning, but their application to large language models is still limited.

[0036] Existing LLM meta-training methods such as MetaICL and MetaICT mainly adopt the multi-task learning paradigm and do not fully utilize the advantages of meta-learning.

[0037] Data augmentation technology is effective in improving model robustness, but its combination with meta-learning is not yet deep enough.

[0038] The optimization algorithm for LLM usually uses Adam or AdamW, but how to effectively share the optimizer state in the meta-learning scenario remains a problem.

[0039] In view of the technical problems existing in large-scale language models, the present invention provides a training method for a large-scale language model.

[0040] The large language model training method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the large language model training system runs in the computer device.

[0041] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a training system for a large language model. According to different requirements, the order of the steps in the flowchart may be changed, and some may be omitted.

[0042] like Figure 1 As shown, the method includes: S1. Collect task samples based on task difficulty and task diversity requirements.

[0043] Build a comprehensive and challenging task sample library. This requires us to not only consider the difficulty level of the tasks when collecting task samples, ensuring that tasks from simple to complex are covered, but also pay attention to the diversity of tasks, including but not limited to text types, subject areas, language styles, etc. By deeply analyzing the needs of the target application scenarios, we can select or design task samples in a targeted manner to ensure that they can fully reflect the challenges and complexities in actual applications.

[0044] S2. Obtain the support set and query set of the task sample, and perform data augmentation on the support set.

[0045] After collecting enough task samples, we divide them into two independent data sets: support set and query set. The purpose of this is to be able to use the two data sets for inner and outer loop training in the subsequent meta-learning process, thereby effectively improving the generalization ability of the model. For the support set, we perform data enhancement processing, such as synonym replacement, sentence reorganization, and adding noise, to increase the diversity and richness of the data, so as to improve the robustness of the model to noise and changes.

[0046] S3. Use the support set to perform inner loop training on the meta-model, and use the query set to perform outer loop training on the meta-model trained by the inner loop to obtain the meta-gradient.

[0047] The meta-learning strategy is used to train the model. First, the meta-model is trained in the inner loop using the support set, that is, the model is trained on one or more tasks and its loss and gradient information is collected. Then, we use this information to update the parameters of the meta-model so that it can better adapt to new tasks. Next, we use the query set to perform outer loop training on the meta-model that has been trained in the inner loop, that is, evaluate the performance of the meta-model on new tasks and calculate the meta-gradient. The meta-gradient reflects the impact of the meta-model parameters on the performance of the new task and is the key to our subsequent update of the global model.

[0048] S4. Aggregate meta-gradients and use the adaptive optimizer to update the global model based on the meta-gradient summary data.

[0049] After obtaining the meta-gradient, we aggregate it and use an adaptive optimizer (such as Adam, RMSprop, etc.) to update the parameters of the global model. The adaptive optimizer can dynamically adjust the learning rate according to the direction and size of the meta-gradient, thereby more effectively optimizing the performance of the global model. By continuously training the inner and outer loops, and updating the global model based on the meta-gradient, we can gradually improve the model's generalization ability and adaptability to new tasks.

[0050] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0051] S101. Obtain task difficulty, where the task difficulty is the loss value of the model on the task in historical training.

[0052] In a specific example, the calculation formula for task difficulty is:

[0053] in, It is a model On Task The validation set The accuracy of It is the highest accuracy among all tasks; Normalize the task difficulty to get the task difficulty quantization value. Normalize the task difficulty to the interval [0, 1]. The higher the difficulty, the closer the value is to 1.

[0054] S102. Confirming task diversity, where the task diversity is the average distance between the task and a sampled task set.

[0055] The TF-IDF method is used to represent the task as a feature vector:

[0056] in, Representative words On Task The frequency of words in ; Inverse Document Frequency , The total number of tasks representing candidates; Computational tasks With the sampled task set The cosine similarity of each task in:

[0057] in, It's a task The TF-IDF vector of It's a task TF-IDF vector of Calculate task diversity based on the cosine similarity:

[0058] The task diversity is the task The average distance to the sampled task set.

[0059] S103. Determine the sampling probability of the task based on the task difficulty and task diversity, and the sampling probability is calculated as follows:

[0060] in, To balance the difficulty, is the diversity hyperparameter, is the task difficulty, For task diversity.

[0061] It can be understood that the dynamic task sampling strategy provides diverse and challenging task batches for the two-level optimization.

[0062] The performance feedback of the double-layer optimized model is used to update the task difficulty , affecting the subsequent sampling probability.

[0063] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0064] S201. Sampling to obtain training task set , where each task Contains a set of support set samples ,in Representatives support the concentration of samples and their labels, Represents the number of support set samples and query set samples ,in Represents the first samples and their labels, Represents the number of query set samples.

[0065] S202. Perform data enhancement processing on the support set.

[0066] Original task sample: Sentence: The quick brown fox jumps over the lazy dog. Data augmentation example: Replace the words in the sample data with their synonyms with a probability of 10%: Assuming this process is triggered (10% probability), we can replace "quick" with its synonym "swift": Enhanced sentence: The swift brown fox jumps over the lazy dog. Randomly insert context-related words into sentences in sample data with 10% probability: Assuming this processing is triggered (with 10% probability), we can insert a context-related word into the sentence, such as "agile" (related to "quick"): Enhanced sentence: The agile quick brown fox jumps over the lazy dog. Randomly delete words from sentences in sample data with a probability of 10%: Assuming this process is triggered (with 10% probability), a random word, such as "brown", is deleted: Enhanced sentence: The quick fox jumps over the lazy dog. Translate text data into Chinese and then back to English with a probability of 20%: Assume this process is triggered (20% probability): Chinese translation: The quick brown fox jumps over the lazy dog.

[0067] Translated back into English: The agile brown fox jumps over the lazy dog. Due to the uncertainty of translation, "agile" here may not be the direct equivalent of the original English, but it conveys a similar meaning. In addition, since "agile" does not appear directly in the original English sentence, the enhancement here introduces a new expression.

[0068] Comprehensive example: Assume that all the above enhancement methods are applied to the original sentence at the same time (of course, in actual applications, only one or several methods are randomly selected each time), and each method is triggered (only for example, the triggering of each method is random in practice): Original sentence: The quick brown fox jumps over the lazy dog. Apply synonym replacement: "quick" -> "swift"; Insert related words: "agile" (assuming it is inserted before "swift"); Randomly delete words: "brown" (hypothetically deleted); Translate back to English (changes that may have been introduced during the translation from Chinese); Possible enhanced sentences: The agile swift fox jumps over the lazy dog. In an embodiment of the present invention, based on step S3, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0069] A two-level optimization framework of MAML is adopted, including inner-loop adaptation and outer-loop meta-update.

[0070] S301. Initialize the metamodel parameters, ; Before training begins, we first initialize the parameters of the meta-model. These parameters are the objects that need to be optimized in the subsequent training process, and they determine the initial state and performance of the meta-model.

[0071] S302. Set optimization function: ; The optimization function is used to guide the update of metamodel parameters.

[0072] S303. Based on the optimization function, using the support set of data enhancement Perform inner loop training on the meta-model: for For each task in , perform K steps of gradient descent to adapt to the task: ; Compute the gradient: ; Based on the current task-specific parameters and the support set data, the gradient of the loss function with respect to the task-specific parameters is calculated.

[0073] Update metamodel parameters: ; Using the computed gradients, we update the task-specific parameters according to the inner loop learning rate.

[0074] For each extracted task, we perform K steps of gradient descent to adapt the task. In each step, we compute the gradient of the loss function on the support set and use these gradients to update the task-specific parameters.

[0075] S304. Perform outer loop training on the meta-model using the query set.

[0076] For each example in the query set, the loss function is calculated using the task-specific parameters adapted by the inner loop.

[0077] Based on the calculated loss and outer loop learning rate, an adaptive optimizer is used to update the meta-model parameters. This updating process aims to make the meta-model better adaptable to different types of tasks, thereby improving its generalization ability on unseen tasks.

[0078]

[0079] in, is the outer loop learning rate, is the task batch size, is an adaptive optimizer, represent The loss function is It is through Task-specific parameters after in-loop adaptation, is the i-th sample in the query set, represents the inner loop learning rate, represent The gradient of Represents the distribution of tasks;

[0080] in Represents the first samples and their labels, Represents the number of query set samples.

[0081] In an embodiment of the present invention, based on step S4, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0082] As can be seen from step S4, both the inner loop training and the outer loop training use adaptive optimizers. The same AdamW optimizer is used in the inner and outer loops, and their momentum states are shared.

[0083] The update rule is defined, and the gradient for time step t is , the update rules include:

[0084] in are the shared first and second order momentum, represents the current gradient, Represents a small constant to prevent division by zero errors, are the bias-corrected first- and second-order momentum estimates, are different values ​​of the momentum decay rate, is the learning rate, is the weight decay coefficient. Usually .

[0085] The adaptive optimizer is shared between the inner loop and the outer loop to maintain consistent optimization behavior. The sharing of momentum helps to transfer useful optimization information between tasks.

[0086] The performance of the optimizer may affect the assessment of task difficulty, thereby indirectly affecting the sampling strategy.

[0087] It is understandable that the training steps of a complete large language model mainly include: Initialization: Randomly initialize metamodel parameters , initialize the state of the adaptive optimizer.

[0088] Outer loop (meta-training): while convergence condition is not met do; Dynamic task sampling: Calculate the difficulty of each task and diversity ; according to Sampling task batches; For each sampled task Execute the inner loop: Partition support set and queryset ;

[0089] Applying Data Augmentation: ;

[0090] initialization: ;

[0091] Internal loop adaptation: for k = 1 to K do; Compute the gradient: ;

[0092] Update parameters: ; end for; Compute the meta-gradient: ; Meta-update: Compute the average meta-gradient: , update the metamodel: ; Updated mission statistics: Updated mission difficulty and task diversity

[0093] end while.

[0094] On the basis of the above embodiment, in order to further enhance the understanding of the training method of the large language model of the present invention, the following is a specific implementation equation: 1. Data Preparation First, prepare two sets of data: a) Meta-training dataset: Contains 50,000 historical customer question-answer pairs from multiple existing product categories. Each data includes a customer question and the corresponding standard answer.

[0095] b) Meta-test dataset: Contains 1,000 customer question-answer pairs in the newly launched smart home product category. This is used to evaluate the model’s ability to adapt to new domains.

[0096] Data structure example: Each record contains two fields: "Question" and "Answer", corresponding to customer inquiry and customer service response respectively.

[0097] 2. Model selection Select the pre-trained Qwen2 13B model as the base language model.

[0098] 3. MAMA-LLM Implementation 3.1 MAML-based two-layer optimization Implement the Model-Agnostic Meta-Learning (MAML) framework, including inner-loop adaptation and outer-loop meta-update.

[0099] Inner loop adaptation: For each task, perform 5 steps of gradient descent with a learning rate of 0.01. Use task-specific support set data for adaptation.

[0100] Outer loop meta-update: Based on the query set performance of multiple tasks, calculate meta-gradients and update the global model.

[0101] 3.2 Dynamic Task Sampling Strategy Implement sampling strategies based on task difficulty and diversity: Task difficulty calculation: Use the model’s loss on the task as the difficulty indicator.

[0102] Task diversity calculation: Convert each task’s description into a TF-IDF (Term Frequency-Inverse Document Frequency) vector.

[0103] Compute the cosine similarity between tasks.

[0104] The diversity of a task is defined as 1 minus the average similarity with other tasks.

[0105] Sampling probability calculation: Combining difficulty and diversity, an exponential function is used to calculate the sampling probability of each task.

[0106] Dynamic update: After each training cycle, the task difficulty is updated based on the model performance.

[0107] 3.3 Data Augmentation Module Implement four NLP-specific data augmentation techniques: Synonym Replacement: Replace a word with its synonym with a probability of 10%.

[0108] Random insertion: insert context-related words at random positions in the sentence with a probability of 10%.

[0109] Random deletion: Randomly delete words in the sentence with a probability of 10%.

[0110] Back translation: Translate the text into Chinese and then back to English with a probability of 20%.

[0111] For each support set sample, one or more of the above enhancement methods are randomly applied to generate enhanced data.

[0112] 3.4 Adaptive Optimizer Implementing the AdamW optimizer with momentum sharing: Initialization: Create momentum and adaptive learning rate states for each parameter.

[0113] Update rules: Compute the exponential moving average (momentum) of the gradients and the exponential moving average of the squared gradients.

[0114] Use these statistics to adjust the learning rate for each parameter.

[0115] Weight decay is applied to prevent overfitting.

[0116] Momentum sharing: Momentum state is shared between inner and outer loops to keep the optimization continuous.

[0117] The learning rate is set to 1e-5 and the weight decay coefficient is 0.01.

[0118] 4. Training Process The training process is carried out for 100 cycles, and the steps of each cycle are as follows: Five training tasks are selected using a dynamic task sampling strategy.

[0119] For each task: a. Sample 10 examples as support set and apply data augmentation. b. Perform inner loop adaptation (5 steps of gradient descent). c. Compute loss and meta-gradient on query set.

[0120] Aggregate meta-gradients and update the global model using an adaptive optimizer.

[0121] Update task difficulty and sampling probability.

[0122] During training, the meta-training loss is recorded and monitored to assess learning progress.

[0123] 5. Apply to new tasks (smart home product customer service) Prepare data: Select 20 samples from the new smart home product customer service data as the support set.

[0124] Fast adaptation: a. Load the trained MAMA-LLM model. b. Perform 10 steps of inner loop adaptation using the support set data.

[0125] Evaluation: Calculate the loss on the remaining query set to evaluate the model performance.

[0126] Real-world application: Use the adapted model to answer new questions from users about smart home products.

[0127] 6. Effect display Here is an example of this in action: User question: "How do I connect a smart light bulb to my mobile app?" Model answer: "To connect a smart bulb to the mobile app, follow these steps: 1. Make sure you have the latest version of our smart home app installed on your phone. 2. Open the app and tap 'Add a device'. 3. Select the 'Lighting devices' category and then select your smart bulb model. 4. Plug in the bulb and make sure it is in pairing mode (usually a fast blinking state). 5. Follow the instructions in the app to connect to the temporary Wi-Fi network created by the bulb. 6. Enter the password for your home Wi-Fi and the app will automatically complete the pairing process. 7. Name the bulb and assign it to the corresponding room. Now you can control the smart bulb through the app. If you encounter any problems, please don't hesitate to ask for further help." This answer demonstrates that the model is able to generate detailed and targeted responses that cover specific installation steps for smart home devices, indicating that the large language model adaptively trained using the training method provided by the present invention has successfully adapted to new product areas.

[0128] In some embodiments, the large language model training system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the large language model training system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Function for training large language models.

[0129] In this embodiment, the large language model training system can be divided into multiple functional modules according to the functions it performs, such as Figure 2As shown. The functional modules of the system 200 may include: a sampling module 210, a processing module 220, a training module 230 and an updating module 240. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0130] Sampling module, used to collect task samples based on task difficulty and task diversity requirements; The processing module is used to obtain the support set and query set of the task sample and perform data enhancement processing on the support set; A training module, used for performing inner loop training on the meta-model using the support set, and performing outer loop training on the meta-model trained by the inner loop using the query set to obtain a meta-gradient; The update module is used to summarize the meta-gradients and use the adaptive optimizer to update the global model based on the meta-gradient summary data.

[0131] Optionally, as an embodiment of the present invention, collecting task samples based on task difficulty and task diversity requirements includes: Obtaining a task difficulty, where the task difficulty is the loss value of the model on the task during historical training; Determine task diversity, where the task diversity is the average distance between the task and a sampled task set; The sampling probability of the task is determined based on the task difficulty and task diversity. The sampling probability is calculated as follows:

[0132] in, To balance the difficulty, is the diversity hyperparameter, is the task difficulty, For task diversity.

[0133] Optionally, as an embodiment of the present invention, obtaining the task difficulty includes: The calculation formula for task difficulty is:

[0134] in, It is a model On Task The validation set The accuracy of It is the highest accuracy among all tasks; The task difficulty is normalized to obtain the quantitative value of the task difficulty.

[0135] Optionally, as an embodiment of the present invention, confirming the diversity of tasks includes: The TF-IDF method is used to represent the task as a feature vector:

[0136] in, Representative words On Task The frequency of words in ; Inverse Document Frequency , The total number of tasks representing candidates; Computational tasks With the sampled task set The cosine similarity of each task in:

[0137] in, It's a task The TF-IDF vector of It's a task TF-IDF vector of Calculate task diversity based on the cosine similarity:

[0138] The task diversity is the task The average distance to the sampled task set.

[0139] Optionally, as an embodiment of the present invention, performing data enhancement processing on the support set includes: Replace the words in the sample data with their synonyms with a probability of 10%; Randomly insert context-related words into sentences in sample data with a probability of 10%; Randomly delete words from sentences in sample data with a probability of 10%; Translate the text data into Chinese and then back to English with a probability of 20%.

[0140] Optionally, as an embodiment of the present invention, inner loop training is performed on the meta-model using the support set, and outer loop training is performed on the meta-model after inner loop training using the query set to obtain a meta-gradient, including: Initialize the metamodel parameters, let ; Set the optimization function: ; Based on the optimization function, the support set of data enhancement is used Perform inner loop training on the meta-model: for For each task in , perform K steps of gradient descent to adapt to the task: ; Compute the gradient: ; Update metamodel parameters: ; Use the query set to train the metamodel in an outer loop:

[0141] in, is the outer loop learning rate, is the task batch size, is an adaptive optimizer, represent The loss function is It is through Task-specific parameters after in-loop adaptation, is the i-th sample in the query set, represents the inner loop learning rate, represent The gradient of Represents the distribution of tasks;

[0142] in Represents the first samples and their labels, Represents the number of query set samples.

[0143] Optionally, as an embodiment of the present invention, the meta-gradients are summarized, and the global model is updated based on the meta-gradient summary data using an adaptive optimizer, including: The update rule is defined, and the gradient for time step t is , the update rules include:

[0144] in are the shared first and second order momentum, represents the current gradient, Represents a small constant to prevent division by zero errors, are the bias-corrected first- and second-order momentum estimates, are different values ​​of the momentum decay rate, is the learning rate, is the weight decay coefficient.

[0145] Figure 3 The training method of the large language model provided for the embodiment of the present application can be applied to the device. It will be appreciated by those skilled in the art that the device structure involved in the embodiment of the present invention does not constitute a limitation on the device, and the device may include more or less components than shown, or combine certain components, or different component arrangements. In an embodiment of the present invention, the device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the embodiments of the present application described herein and / or required.

[0146] The device 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0147] The memory 320 may be used to store the execution instructions of the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is enabled to perform some or all of the steps in the following method embodiments.

[0148] The processor 310 is the control center of the storage device, and uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 310 can include only a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0149] The communication unit 330 is used to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0150] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0151] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0152] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0153] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0154] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0156] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A method for training a large language model, characterized in that: include: Collect task samples based on task difficulty and task diversity requirements; Obtain the support set and query set of the task sample, and perform data enhancement on the support set; The meta-model is trained in an inner loop using the support set, and the meta-model trained in an inner loop is trained in an outer loop using the query set to obtain the meta-gradient. Aggregate meta-gradients and use the adaptive optimizer to update the global model based on the meta-gradient summary data.

2. The method according to claim 1, characterized in that Task samples are collected based on task difficulty and task diversity requirements, including: Obtaining a task difficulty, where the task difficulty is the loss value of the model on the task during historical training; Determine task diversity, where the task diversity is the average distance between the task and a sampled task set; The sampling probability of the task is determined based on the task difficulty and task diversity. The sampling probability is calculated as follows: in, To balance the difficulty, is the diversity hyperparameter, is the task difficulty, For task diversity.

3. The method according to claim 2, characterized in that Get the difficulty of the task, including: The calculation formula for task difficulty is: in, It is a model On Task The validation set The accuracy of It is the highest accuracy among all tasks; The task difficulty is normalized to obtain the quantitative value of the task difficulty.

4. The method according to claim 2, characterized in that: Identify task diversity, including: The TF-IDF method is used to represent the task as a feature vector: in, Representative words On Task term frequency in ; inverse document frequency , The total number of tasks representing candidates; Computational tasks With the sampled task set The cosine similarity of each task in: in, It's a task The TF-IDF vector of It's a task TF-IDF vector of Calculate task diversity based on the cosine similarity: The task diversity is the task The average distance to the sampled task set.

5. The method according to claim 1, characterized in that Perform data augmentation on the support set, including: Replace the words in the sample data with their synonyms with a probability of 10%; Randomly insert context-related words into sentences in sample data with a probability of 10%; Randomly delete words from sentences in sample data with a probability of 10%; Translate the text data into Chinese and then back to English with a probability of 20%.

6. The method according to claim 1, characterized in that The meta-model is trained in an inner loop using the support set, and the meta-model trained in an inner loop is trained in an outer loop using the query set to obtain the meta-gradient, including: Initialize the metamodel parameters, let ; Set the optimization function: ; Based on the optimization function, the support set of data enhancement is used Perform inner loop training on the meta-model: for For each task in , perform K steps of gradient descent to adapt to the task: ； Compute the gradient: ; Update metamodel parameters: ; Use the query set to train the metamodel in an outer loop: in, is the outer loop learning rate, is the task batch size, is an adaptive optimizer, represent The loss function is It is through Task-specific parameters after in-loop adaptation, is the i-th sample in the query set, represents the inner loop learning rate, represent The gradient of Represents the distribution of tasks; in Represents the first samples and their labels, Represents the number of query set samples.

7. The method according to claim 6, characterized in that Aggregate meta-gradients and use the adaptive optimizer to update the global model based on the meta-gradient summary data, including: The update rule is defined, and the gradient for time step t is , the update rules include: in are the shared first and second order momentum, represents the current gradient, Represents a small constant to prevent division by zero errors, are the bias-corrected first- and second-order momentum estimates, are different values ​​of the momentum decay rate, is the learning rate, is the weight decay coefficient.

8. A large language model training system, characterized in that: include: Sampling module, used to collect task samples based on task difficulty and task diversity requirements; The processing module is used to obtain the support set and query set of the task sample and perform data enhancement processing on the support set; A training module, used for performing inner loop training on the meta-model using the support set, and performing outer loop training on the meta-model trained by the inner loop using the query set to obtain a meta-gradient; The update module is used to summarize the meta-gradients and use the adaptive optimizer to update the global model based on the meta-gradient summary data.

9. A large language model training device, characterized in that: include: Memory, used to store training programs for large language models; A processor, used to implement the steps of the large language model training method as described in any one of claims 1-7 when executing the training program of the large language model.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a training program for a large language model, and when the training program for the large language model is executed by a processor, the steps of the training method for a large language model as described in any one of claims 1 to 7 are implemented.

Images