Large model continuous knowledge migration method based on sample playback

Abstract

The invention discloses a large model continuous knowledge migration method based on sample playback, belongs to the technical field of large language model continuous learning, and is applied to a large model continuous learning scene. The implementation method comprises the following steps of: 1, training single subtask sample structure data by utilizing a large language model of maximum likelihood estimation; 2, using a Rouge-L index to screen sample data higher than a threshold in a single subtask, and storing the sample data in a memory pool; 3, performing iteration instruction fine tuning on the large language model on multiple tasks by playing back the data in the memory pool; compared with the prior art, through large model continuous learning based on sample playback of the model, the complexity of a method for relieving catastrophic forgetting of the large language model is reduced, and the current state of the model and the mastering degree of the model on a specific task are fused into data playback; the stability of the large language model is enhanced through data matching and playback of samples in a memory pool.

Description

TECHNICAL FIELD

[0001] The application relates to a large model continuous knowledge transfer method based on sample playback and belongs to the technical field of large language model continuous learning. BACKGROUND

[0002] In the process of multi-task continuous learning, large language models inevitably face the challenge of catastrophic forgetting. One of the main reasons for catastrophic forgetting of large language models is their limited memory capacity. Large language models usually rely on parameter weights to store and call knowledge. However, the number of parameters of the model is fixed, and when the model learns new data, in order to adapt to new tasks or knowledge, the old weights may be overwritten by new weights of subsequent tasks. This overwriting process is often irreversible, causing the model to gradually lose important information learned previously while learning new knowledge. In addition, the model lacks a mechanism to automatically identify which parameters are associated with a specific task or knowledge. In multi-task learning, different tasks may involve different knowledge domains and skill requirements. However, large language models currently cannot accurately identify which parameters are closely related to a specific task and which are general.

[0003] To alleviate this problem, continuous learning methods based on parameter regularization suppress changes to key parameters of previous tasks when learning new tasks by adding penalty terms or regularization. At the same time, architecture-based methods assign different parameter sets to different tasks to prevent interference from subsequent tasks. Although these methods provide mechanisms to prevent forgetting from the perspective of parameter management, they face challenges such as high computational demand, complex hyperparameter adjustment, and unstable learning process. A simpler and more effective method to alleviate catastrophic forgetting of large language models is data playback. Existing data playback-based methods either use random data selection strategies, resulting in the neglect of certain important samples or specific parts of data distribution, or only focus on intra-task data relationships without considering inter-task data relationships, or only focus on the diversity and complexity of data itself without considering the current state of the model and its mastery of a specific task, which cannot effectively strengthen knowledge in the target domain.

[0004] Therefore, how to reduce the complexity of methods to alleviate catastrophic forgetting of large language models and integrate the current state of the model and its mastery of a specific task into data playback has become a problem to be solved. SUMMARY

[0005] The purpose of the present application is to solve the technical problem of stability decline caused by memory forgetting of large language models, and a large model continuous knowledge transfer method based on sample playback is proposed.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] A large model continuous knowledge transfer method based on sample playback of the application is applied to the large model continuous learning scene, comprising the following steps:

[0008] Step 1: training a single subtask sample structure data by using a maximum likelihood estimation large language model;

[0009] Step 1.1: constructing a data sample for large language model instruction fine-tuning;

[0010] Step 1.2: constructing the sample format structure as shown in formula (1) to form a sample structure;

[0011] {system,query,response} (1)

[0012] Wherein, system represents the system instruction of the large language model; query represents the question input to the large language model; response represents the response of the large language model to the input question;

[0013] Step 1.3: training the sample structure by using a maximum likelihood estimation large language model;

[0014] Step 1.3.1: inputting the sample structure data as a training target into the maximum likelihood estimation large language model;

[0015] Step 1.3.2: optimizing the large language model by using the loss function shown in formula (2);

[0016]

[0017] Wherein, x represents query, y represents response, and N is the length of the response sequence; p θ represents the probability of the large language model predicting y i ;

[0018] Step 2: screening sample data higher than the threshold value in a single subtask into a memory pool by using a Rouge-L index;

[0019] Step 2.1: taking the response of the large language model to query as test data after training; taking the response corresponding to query as comparison data;

[0020] Step 2.2: comparing the test data and the comparison data by using the Rouge-L index through the longest common subsequence to obtain a single subtask response score;

[0021] Step 2.3: using the single subtask response score to sort the sample data in descending order; setting a screening threshold, and storing the sample data higher than the threshold in the memory pool;

[0022] Step 3: iteratively fine-tuning the large language model on multiple tasks by replaying the data in the memory pool;

[0023] Step 3.1: constructing a mapping relationship between multiple tasks and the large language model for multiple tasks, forming a task set and a model set;

[0024] Step 3.2: fine-tuning the i-1 model in the model set on the i task in the task set;

[0025] Step 3.3: using the Rouge-L index to obtain the response score of the large language model on the i task;

[0026] Step 3.4: using the response score on the i task to perform descending sorting on the sample data; set a filtering threshold, and store the sample data higher than the threshold in the memory pool;

[0027] Step 3.5: combining the sample data of the current task with all sample data in the memory pool in the manner shown in formula (3) to fine-tune the previous large language model, and then obtain the current large language model;

[0028]

[0029] wherein i represents the current large language model; D i represents the sample data of the i task; D' j represents the sample data of the j task stored in the memory pool;

[0030] Step 3.6: executing steps 3.2 to 3.5 in a cyclic iteration manner until the execution of the multiple tasks is completed;

[0031] A computer-readable storage device, the storage device stores a computer program, characterized in that the computer program is executed to realize the large model continuous knowledge transfer method based on sample replay.

[0032] The present application has the following beneficial effects compared with the prior art:

[0033] 1. The present application solves the problems of high complexity and not considering the current state of the model and the mastery of the model for a specific task in the existing large model continuous learning method by using the large model continuous learning based on model sample replay, and improves the forgetting phenomenon of the large language model;

[0034] 2. The present application improves the generalization of the large language model in multiple tasks through Rouge-L measurement and backward transfer learning;

[0035] 3. The present application enhances the stability of the large language model by data matching and sample replay in the memory pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process of the present invention;

[0037] Figure 2 This is the instruction fine-tuning diagram of the present invention. DETAILED DESCRIPTION

[0038] In order to better illustrate the purpose and advantages of the present invention, the following is a further description of the invention in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the scope of protection of the present invention.

[0039] Five tasks from the SuperNI dataset: task192 hotpotqa sentence generation, task1219 tedtranslation, task177 para-nmt paraphrasing, task024 cosmosqa answer generation, and task511 reddit tifu long text summarization are used as the basic data for this example.

[0040] Example

[0041] like Figure 1 As shown, the present invention is a large model continuous knowledge transfer method based on sample playback, and the specific implementation steps are as follows:

[0042] Step 1: Use the maximum likelihood estimation large language model to train the single subtask sample structure data;

[0043] Step 1.1: Construct data samples for fine-tuning large language model instructions;

[0044] Step 1.2: Construct the data sample into a sample formatting structure as shown in formula (1) to form a sample structure;

[0045] {system,query,response} (1)

[0046] Among them, system represents the system instructions of the large language model; query represents the question input to the large language model; response represents the response of the large language model to the input question;

[0047] Step 1.3: Use the large language model with maximum likelihood estimation to train the sample structure;

[0048] Step 1.3.1: input the sample structure data as a training target into a large language model of maximum likelihood estimation;

[0049] Step 1.3.2: optimize the large language model by using a loss function shown in formula (2);

[0050]

[0051] wherein x represents a query, y represents a response, and N is the length of the response sequence; p θ represents the probability of the large language model predicting y i ;

[0052] In the embodiment, as shown in formula (1), the sample structure data of the task 192 hotpot qa sentence generation task is input into the large language model for training; Figure 2

[0053] Step 2: store the sample data higher than a threshold value in a memory pool by using a Rouge-L index to screen the single subtask;

[0054] Step 2.1: take the response of the query made by the trained large language model as test data; and take the response corresponding to the query as comparison data;

[0055] Step 2.2: compare the test data and the comparison data by using a Rouge-L index and the longest common subsequence to obtain a single subtask response score;

[0056] Step 2.3: sort the sample data in descending order by using the single subtask response score; set a screening threshold value; and store the sample data higher than the threshold value in a memory pool;

[0057] ​In the embodiment, compared with the task 192 hotpot QA sentence generation task, if one of the comparison data is: Rugosa; also called sugar corn and pole corn is a variety of maize with a high sugar content. And the corresponding test data is: Rugosa, which is also known as sugar corn or pole corn, is a type of maize characterized by its high sugar content. According to the longest common subsequence of the two sequences, the Rouge-L index is calculated, and the average value of the Rouge-L index of all test data is finally taken as the response score of the task sample data in the multi-task large language model;

[0058] Step 3: iteratively fine-tune the large language model on multiple tasks by replaying the data in the memory pool;

[0059] Step 3.1: build a mapping relationship between the multi-task and the multi-task large language model, form a task set and a model set;

[0060] Step 3.2: fine-tune the i-1th model in the model set on the i th task in the task set;

[0061] Step 3.3: use the Rouge-L index to obtain the response score of the large language model on the i th task;

[0062] Step 3.4: use the response score on the i th task to perform descending sorting on the sample data; set a filtering threshold, and store the sample data higher than the threshold in the memory pool;

[0063] Step 3.5: combine the sample data of the current task with all the sample data in the memory pool to fine-tune the previous large language model in the manner shown in formula (3), and then obtain the current large language model;

[0064]

[0065] Wherein, i represents the current large language model; D i represents the sample data of the i th task; D' j represents the sample data of the j th task stored in the memory pool;

[0066] Step 3.6: execute steps 3.2 to 3.5 in a cyclic iteration manner until the execution of the multi-task ends;

[0067] In the embodiment, the multi-task task1219 tedtranslation, task177 para-nmtparaphrasing, task024 cosmosqa answer generation, and task511 reddit tifu long text summarization are used to iteratively fine-tune the instruction of the large language model.

[0068] A computer-readable storage device, the storage device storing a computer program, characterized in that the computer program, when executed, implements the large model continuous knowledge transfer method based on sample playback.

[0069] To further embody the superiority of the present application, a comparative experiment is used to verify the effect of the present application.

[0070] As shown in Table 1, from the AR and BWT indicators, the average value of AR of the present application is 61.53, and the average value of BWT is 1.08. Compared with other methods, it is more conducive to reducing the degree of catastrophic forgetting of the large language model, and can also enhance the backward and forward moving ability of the large language model.

[0071] Table 1: Sample playback comparison experiment results

[0072]

[0073] The above specific description further details the purpose, technical solution and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A large-model continuous knowledge transfer method based on sample playback, characterized by: The following steps are included: Step 1: Use the maximum likelihood estimation large language model to train the single subtask sample structure data; Step 2: Use the Rouge-L indicator to filter sample data above the threshold in a single subtask and store them in the memory pool; Step 2.1: Use the trained large language model’s response to the query as test data; use the response corresponding to the query as comparison data; Step 2.2: Use the Rouge-L metric to compare the test data with the comparison data using the longest common subsequence to obtain a single subtask response score. Step 2.3: Sort the sample data in descending order using the single subtask response score; set a screening threshold and store the sample data above the threshold in the memory pool; Step 3: Iteratively fine-tune the large language model on multiple tasks by replaying the data in the memory pool; Step 3.1: Construct a mapping relationship between multiple tasks and large language models to form a task set and a model set; Step 3.2: Fine-tune the instruction of the i-1th model in the model set on the i-th task in the task set; Step 3.3: Use the Rouge-L indicator to obtain the response score of the large language model on the i-th task; Step 3.4: Sort the sample data in descending order using the response score on the i-th task; Set a screening threshold and store sample data above the threshold in the memory pool; Step 3.5: Combine the sample data of the current task with all the sample data in the memory pool using the method shown in formula (3) to fine-tune the instructions of the previous large language model, and then obtain the current large language model; Among them, i represents the current large language model; D i represents the sample data of the i-th task; D′ j Represents the sample data stored in the memory pool for the jth task; Step 3.6: Execute steps 3.2 to 3.5 in a loop iteration manner until the multi-task execution is completed.

2. The large-model continuous knowledge transfer method based on sample playback according to claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Construct data samples for fine-tuning large language model instructions; Step 1.2: Construct the data sample into a sample formatting structure as shown in formula (1) to form a sample structure; {system,query,response} (1) Among them, system represents the system instructions of the large language model; query represents the question input to the large language model; response represents the response of the large language model to the input question; Step 1.3: Use the large language model estimated by maximum likelihood to train the sample structure.

3. The large-model continuous knowledge transfer method based on sample playback according to claim 2, characterized in that: Step 1.3 is implemented as follows: Step 1.3.1: Input the sample structure data as training targets into the large language model with maximum likelihood estimation; Step 1.3.2: Optimize the large language model using the loss function shown in formula (2); Where x represents the query, y represents the response, N is the length of the response sequence; p θ Represents the large language model prediction y i probability.

4. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed, a large model continuous knowledge transfer method based on sample playback is implemented as described in any one of claims 1 to 3.

Images