Self-adaptive learning method for fine tuning of large model in financial field based on loss value

Abstract

The invention relates to a self-adaptive learning method for fine tuning of a large model in the financial field based on a loss value, and the method comprises the following steps: 1, collecting a full-amount financial data set of each open source, and carrying out the fine tuning of a selected pre-training model, and obtaining a pre-fine tuning model A; 2, predicting each sample to obtain a loss function value of each sample by using the pre-fine tuning model A obtained in the step 1; 3, the loss function values are sorted in an ascending order, and a Qwen72B pre-training model is trained to obtain a model B; and 4, carrying out adaptive learning by adopting a dynamic data sampling method based on a loss function value, and adjusting the selection of training data according to the performance of the model in each round of training. According to the method, the problem of low learning efficiency of fine tuning of the large model in a financial data training process is solved, the learning efficiency and performance of the financial large model in processing tasks with different difficulties can be effectively optimized, and meanwhile, the risk of over-fitting and unstable training is reduced.

Description

Technical Field

[0001] The present invention relates to a large model fine-tuning technology, and in particular to an adaptive learning method for fine-tuning a large model in the financial field based on loss value. Background Art

[0002] In recent years, with the rapid development of deep learning and big models, the application of artificial intelligence in various fields has made significant progress. Especially in the financial field, the application based on big models has become a trend. These big models can provide strong support in risk assessment, investment strategy, market forecasting, etc. by processing and analyzing massive data. However, although big models have shown great potential in the financial field, their learning efficiency in the fine-tuning process still faces many challenges.

[0003] Currently, fine-tuning of large models usually relies on a large amount of labeled data. During the fine-tuning process, large models need to continuously adjust learning parameters to minimize the loss function value, thereby improving the accuracy of predictions. However, financial data is highly complex and diverse, and large models often encounter tasks of different difficulty levels during training. The difference in difficulty of these tasks can cause large models to have low learning efficiency during training, the so-called seesaw phenomenon (large models may converge quickly on simple tasks, but do not learn sufficiently on complex tasks, that is, only some of the complex tasks are learned). This unbalanced learning process not only wastes computing resources, but may also cause large models to perform poorly in practical applications. Summary of the invention

[0004] In view of the problem of low learning efficiency in the fine-tuning of large models in the face of financial data training, the present invention proposes an adaptive learning method for fine-tuning large models in the financial field based on loss values, which can effectively optimize the training process of large financial models. This method systematically identifies which data or tasks are the most challenging for the large model by analyzing the loss function values ​​of each training sample of the large model in real time during the training process. In this way, the large model can automatically adjust the subsequent learning path and optimize the learning progress. This adaptive learning path optimization method is particularly important in large model training, and can greatly improve the training efficiency and large model effect.

[0005] Specifically, through the adaptive learning mechanism, the large model can focus on learning data or tasks with high loss function values ​​that are more difficult for the large model during the training process. At the same time, simpler tasks are introduced in a timely manner to help the large model consolidate existing knowledge and avoid overfitting problems caused by excessive concentration on one aspect. This adaptive learning path optimization method is particularly suitable for the training of large models in the financial field, and can improve the stability and prediction accuracy of large models in the complex and changing financial market environment.

[0006] The technical solution of the present invention is:

[0007] An adaptive learning method for fine-tuning a large model in the financial field based on loss value improves the learning efficiency and performance of the model when processing tasks of different difficulty levels by optimizing the sorting mechanism of the loss function value of the large model and the adaptive learning path. The method can automatically identify the difficulty level in the training data and adjust the training strategy accordingly. It includes the following steps:

[0008] Step 1. Pre-fine-tuning based on the open source Qwen72B pre-training model: Collect all open source financial data sets, select 1,000 pre-fine-tuning training data with good diversity to fine-tune the selected pre-training model, and obtain the pre-fine-tuning model A;

[0009] Step 2. Financial data difficulty assessment: Use the pre-fine-tuning model A obtained in step 1 to predict each sample in the full financial data set; through the forward propagation process of the pre-fine-tuning model A, the loss function value of each sample can be obtained;

[0010] Step 3. Sequential learning: When fine-tuning the Qwen72B pre-trained model, sort the data in the full financial dataset in ascending order according to the loss function values ​​obtained in step 2, and train the Qwen72B pre-trained model in this order; based on this sorted dataset, perform the first round of fine-tuning training on the Qwen72B pre-trained model to obtain model B;

[0011] Step 4. Use a dynamic data sampling method based on the loss function value to perform adaptive learning: This method adjusts the selection of training data according to the performance of the model in each round of training; data that performs poorly on model B will be sampled more so that the model can better learn these difficulties in subsequent training, thereby improving the overall learning effect.

[0012] Furthermore, step 1 specifically includes the following steps:

[0013] Step 1.1. First, use the Qwen72B pre-trained model to perform forward inference on all samples in the full financial dataset, and obtain the embedding vector of each sample in the last layer of the Qwen72B pre-trained model;

[0014] Step 1.2. Perform k-means clustering based on the embedding vectors of each sample, divide the entire financial dataset into 100 categories, and randomly obtain 10 samples from each category to form 1000 pre-fine-tuning training data;

[0015] Step 1.3. Based on the above 1000 pre-fine-tuning training data, train an epoch on the Qwen72B pre-training model so that the pre-training model can obtain basic instruction following capabilities while avoiding destroying the knowledge distribution of the original pre-training model; the model generated in this step is called pre-fine-tuning model A.

[0016] Furthermore, step 4 specifically includes the following steps:

[0017] Step 4.1. First, repeat step 2 based on model B to obtain new loss function values ​​for all data in the training set; that is, use the current model B to predict each data sample in the training set and calculate the loss function value of each sample. These loss function values ​​reflect the performance of the model on these data samples;

[0018] Step 4.2. In order to accelerate the learning efficiency of the model, a difficulty biased sampling technique is implemented; this technique performs corresponding data sampling according to the gradients of different difficulties: more frequent sampling or even full sampling is performed for data with high loss function values, while less sampling is performed for data with low loss function values; this is because data with high loss function values ​​indicate that the model has not learned enough and needs more training; at the same time, data with low loss function values ​​are appropriately sampled to maintain the learning balance of the model on tasks of different difficulty levels, and to prevent the model from being over-optimized on difficult data and degrading performance on simple tasks;

[0019] Specifically, the set of loss function values ​​of each sample obtained in step 4.1 is represented as L, and the difficulty percentile d of each sample is defined based on formula 1;

[0020]

[0021] Where L i Represents the loss function value of the i-th sample, L max Represents the maximum loss function value, L min Represents the minimum loss function value, d i represents the difficulty percentile of the i-th sample;

[0022] Then, the set L is sampled using Formula 2 to create the difficulty bias set L ′ , where samples with lower difficulty are given a higher discard rate:

[0023] L ′ ={L i ∈L∶U(0,1)<2*(d i ) 4} (2)

[0024] In this formula, U(0,1) represents a uniformly distributed random number between 0 and 1; this formula means that for each sample, the corresponding loss function value L i , generate a random number U(0,1), if this random number is less than 2*(d i ) 4 , then this sample will be retained in the difficulty bias set L ′ In; due to d i For samples with higher loss function values, the value is also higher, so 2*(d i ) 4 is also higher, which means that samples with higher loss function values ​​are more likely to be retained. On the contrary, samples with lower loss function values ​​are more likely to be discarded, but they will not be completely discarded; this achieves the goal of difficulty biased sampling;

[0025] Through this difficulty biased sampling technique, all samples with the highest loss function values ​​can be retained, and for the remaining samples, they will be sampled with a certain probability according to their loss function values. In this way, not only the number of training samples is reduced, but also the large model can pay more attention to those financial data with higher loss function values ​​during training, so as to learn more efficiently in the iterative process. Finally, the data set L obtained after difficulty biased sampling is ′ Sort the loss function values ​​in ascending order and fine-tune one epoch based on model B;

[0026] Step 4.3. Based on the model obtained after fine-tuning in step 4.2, repeat steps 4.1-4.2 and iterate continuously. Fine-tune only one epoch in each iteration until the average loss function value of the fine-tuned model on the entire financial data set reaches one-third of the loss function value of the model in the pre-training stage or reaches the actual number of training samples set. In this way, through adaptive learning, the model can gradually improve its understanding and processing capabilities of complex data, thereby improving the performance of the model as a whole.

[0027] Optionally, the difficulty biased sampling technique in step 4.2 can use the reward model of the RLHF part of the large model to score the training sample and the generation result of the training sample on the fine-tuned model respectively, and use the interpolation of the former score and the latter score as the judgment model for the learning of the sample. If the score is large, it means that the model has not learned enough for the training sample and needs to learn again.

[0028] The beneficial effects of the present invention are:

[0029] The present invention manages the training process more finely based on the method of loss function value sorting and adaptive learning path. This method can more reasonably and automatically identify the difficulty of training data from the perspective of the model through the loss function value, and simulate the human learning process. At the same time, combined with dynamic data sampling and adaptive learning path optimization, it can effectively improve the learning efficiency and performance of large models when processing tasks of different difficulty levels, while reducing the risk of overfitting and training instability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a specific flow chart of the adaptive learning method for fine-tuning a large model in the financial field based on loss value. DETAILED DESCRIPTION

[0031] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0032] The present invention aims to improve the learning efficiency and performance of the model when processing tasks of different difficulty levels by optimizing the sorting mechanism of the loss function value of the large model and the adaptive learning path. The method can automatically identify the difficulty level in the training data and adjust the training strategy accordingly. The following is the specific implementation process:

[0033] Step 1. Pre-fine-tuning based on the open source Qwen72B pre-trained model: Select 1,000 pre-fine-tuning training data with good diversity to fine-tune the selected pre-trained model. The specific method is as follows:

[0034] Step 1.1. First, use the Qwen72B pre-trained model to perform forward inference on all samples in our full financial dataset (obtained by collecting various open source financial datasets) to obtain the embedding vector of each sample in the last layer of the Qwen72B pre-trained model.

[0035] Step 1.2. Perform k-means clustering based on the embedding vectors of each sample, divide the entire financial dataset into 100 categories, and randomly obtain 10 samples from each category to form 1,000 pre-fine-tuning training data.

[0036] Step 1.3. Based on the above 1000 pre-fine-tuning training data, train one epoch (i.e., a complete dataset training cycle) on the Qwen72B pre-training model so that the pre-training model can obtain basic instruction following capabilities while avoiding destroying the knowledge distribution of the original pre-training model. The model generated in this step is called pre-fine-tuning model A.

[0037] Step 2. Financial data difficulty assessment: We will use the pre-fine-tuning model A obtained in step 1 to predict each sample in our full financial data set. Through the forward propagation process of the pre-fine-tuning model A, we can get the loss function value of each sample. These loss function values ​​reflect the difficulty of each sample for the pre-fine-tuning model A: the higher the loss function value, the more difficult the sample is for the pre-fine-tuning model A; the lower the loss function value, the easier the sample is for the pre-fine-tuning model A.

[0038] Step 3. Sequential learning: When formally fine-tuning the Qwen72B pre-trained model, we sort the data in the full financial dataset in ascending order according to the loss function value obtained in step 2, and train the Qwen72B pre-trained model in this order. In this way, the large model is trained by simulating the learning path of humans from shallow to deep, starting with simple knowledge points and gradually transitioning to more complex content. Based on this sorted dataset, we perform the first round of fine-tuning training on the Qwen72B pre-trained model to obtain model B. This learning method helps to stabilize the model training process, so that the model can gradually adapt to simpler data in the early stage, and then remain calm when facing more difficult data.

[0039] Step 4. Adaptive learning: When performing alignment and fine-tuning training on large models, especially in the financial field where the data is highly complex and diverse, it is usually difficult for the model to fully master all the knowledge in the training set after only one round of training. Therefore, we usually perform multiple rounds of training (called multiple epochs). In order to ensure that the model can fully learn, we use a dynamic data sampling method based on the loss function value for adaptive learning. This method adjusts the selection of training data based on the performance of the model in each round of training. Data that performs poorly on model B (that is, data with higher loss function values) will be sampled more so that the model can better learn these difficulties in subsequent training, thereby improving the overall learning effect. The specific method is as follows:

[0040] Step 4.1. First, repeat step 2 based on model B to obtain new loss function values ​​for all data in the training set. That is, we will use the current model B to predict each data sample in the training set and calculate the loss function value of each sample. These loss function values ​​reflect the performance of the model on these data samples.

[0041] Step 4.2. In order to accelerate the learning efficiency of the model, we implemented a difficulty biased sampling technique. This technique will perform corresponding data sampling according to the gradients of different difficulties: more frequent sampling or even full sampling for data with high loss function values ​​(high difficulty), and less sampling for data with low loss function values ​​(low difficulty). This is because data with high loss function values ​​indicates that the model has not learned enough and needs more training. At the same time, we also appropriately sample data with low loss function values ​​to maintain the learning balance of the model on tasks of different difficulty levels, and prevent the model from over-optimizing on difficult data and degrading performance on simple tasks.

[0042] Specifically, we denote the set of loss function values ​​of each sample obtained in step 4.1 as L, and define the difficulty percentile d of each sample based on Formula 1 (the lowest loss function value is represented as 0, and the highest loss function value is represented as 1).

[0043]

[0044] Where L i Represents the loss function value of the i-th sample, L max Represents the maximum loss function value, L min Represents the minimum loss function value, d i Represents the difficulty percentile of the i-th sample.

[0045] We then sample the set L using Formula 2 to create our difficulty bias set L ′ , where samples with lower difficulty are given a higher discard rate:

[0046] L ′ ={L i ∈L∶U(0,1)<2*(d i ) 4} (2)

[0047] In this formula, U(0,1) represents a uniformly distributed random number between 0 and 1. This formula means that for each sample, the corresponding loss function value L i , we generate a random number U(0,1), if this random number is less than 2*(d i ) 4 , then this sample will be retained in the difficulty bias set L ′ In. Due to d i For samples with higher loss function values, the value is also higher, so 2*(d i ) 4is also higher, which means that samples with higher loss function values ​​are more likely to be retained. On the contrary, samples with lower loss function values ​​are more likely to be discarded, but they will not be completely discarded. In this way, we achieve the goal of difficulty biased sampling.

[0048] Through this difficulty biased sampling technique, we can keep all the samples with the highest loss function values, about 16%, and for the remaining samples, we will sample them with a certain probability according to their loss function values. In this way, we not only reduce the number of training samples, but also allow the large model to pay more attention to those financial data with higher loss function values ​​during training, so as to learn more efficiently in the iterative process. Finally, we sample the data set L obtained after difficulty biased sampling. ′ Sort the loss function values ​​in ascending order and fine-tune model B for one epoch.

[0049] Step 4.3. Based on the model obtained after fine-tuning in step 4.2, repeat steps 4.1-4.2, and iterate continuously. Each iteration fine-tunes only one epoch until the average loss function value of the fine-tuned model on the full financial data set reaches one-third of the loss function value of the model in the pre-training stage or reaches the actual number of training samples (defined according to the specific scenario, such as training about 10,000 samples in the financial scenario). In this way, through adaptive learning, the model can gradually improve its ability to understand and process complex data, thereby improving the performance of the model as a whole.

[0050] There are many methods for the difficulty bias sampling technology in step 4.2 above. For example, the reward model of the RLHF part of the large model can be used to score the training sample and the generation result of the training sample on the fine-tuned model respectively, and the interpolation of the former score and the latter score is used to judge the learning situation of the model for the sample. If the score is large, it means that the model has not learned enough for the training sample and needs to learn again. However, this method is relatively complex and inefficient, that is, it requires a separate training of a reward model, and it is necessary to perform generative tasks on the fine-tuned large model, which is far less simple and effective than the method directly using the loss function value.

[0051] The present invention ensures that the large model can efficiently learn and adapt to tasks of different difficulty levels through a refined training strategy, thereby improving its generalization ability in practical applications while maintaining the model performance.

[0052] (1) The ranking mechanism of the loss function value can more reasonably and automatically identify the difficulty of the training data from the perspective of the large model;

[0053] (2) The adaptive learning path optimization method from easy to difficult based on the loss function value can adjust the learning strategy according to the real-time performance of the large model, thereby improving the training efficiency;

[0054] These key points work together to significantly improve the training efficiency and final performance of large language models.

[0055] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. An adaptive learning method for fine-tuning a large model in the financial field based on loss value, characterized in that: By optimizing the sorting mechanism and adaptive learning path of the large model loss function value, the learning efficiency and performance of the model when processing tasks of different difficulty levels are improved; this method can automatically identify the difficulty level in the training data and adjust the training strategy accordingly; it includes the following steps: Step 1. Pre-fine-tuning based on the open source Qwen72B pre-training model: Collect all open source financial data sets, select 1,000 pre-fine-tuning training data with good diversity to fine-tune the selected pre-training model, and obtain the pre-fine-tuning model A; Step 2. Financial data difficulty assessment: Use the pre-fine-tuning model A obtained in step 1 to predict each sample in the full financial data set; through the forward propagation process of the pre-fine-tuning model A, the loss function value of each sample can be obtained; Step 3. Sequential learning: When fine-tuning the Qwen72B pre-trained model, sort the data in the full financial dataset in ascending order according to the loss function values ​​obtained in step 2, and train the Qwen72B pre-trained model in this order; based on this sorted dataset, perform the first round of fine-tuning training on the Qwen72B pre-trained model to obtain model B; Step 4. Use a dynamic data sampling method based on the loss function value to perform adaptive learning: This method adjusts the selection of training data according to the performance of the model in each round of training; data that performs poorly on model B will be sampled more so that the model can better learn these difficulties in subsequent training, thereby improving the overall learning effect.

2. The adaptive learning method for fine-tuning a large financial model based on loss value according to claim 1 is characterized in that: Step 1 specifically includes the following steps: Step 1.

1. First, use the Qwen72B pre-trained model to perform forward inference on all samples in the full financial dataset, and obtain the embedding vector of each sample in the last layer of the Qwen72B pre-trained model; Step 1.

2. Perform k-means clustering based on the embedding vectors of each sample, divide the entire financial dataset into 100 categories, and randomly obtain 10 samples from each category to form 1000 pre-fine-tuning training data; Step 1.

3. Based on the above 1000 pre-fine-tuning training data, train an epoch on the Qwen72B pre-training model so that the pre-training model can obtain basic instruction following capabilities while avoiding destroying the knowledge distribution of the original pre-training model; the model generated in this step is called pre-fine-tuning model A.

3. The adaptive learning method for fine-tuning a large financial model based on loss value according to claim 2 is characterized in that: Step 4 specifically includes the following steps: Step 4.

1. First, repeat step 2 based on model B to obtain new loss function values ​​for all data in the training set; that is, use the current model B to predict each data sample in the training set and calculate the loss function value of each sample. These loss function values ​​reflect the performance of the model on these data samples; Step 4.

2. In order to accelerate the learning efficiency of the model, a difficulty biased sampling technique is implemented; this technique performs corresponding data sampling according to the gradients of different difficulties: more frequent sampling or even full sampling is performed for data with high loss function values, while less sampling is performed for data with low loss function values; this is because data with high loss function values ​​indicate that the model has not learned enough and needs more training; at the same time, data with low loss function values ​​are appropriately sampled to maintain the learning balance of the model on tasks of different difficulty levels, and to prevent the model from being over-optimized on difficult data and degrading performance on simple tasks; Specifically, the set of loss function values ​​of each sample obtained in step 4.1 is represented as L, and the difficulty percentile d of each sample is defined based on formula 1; Where L i Represents the loss function value of the i-th sample, L max Represents the maximum loss function value, L min Represents the minimum loss function value, d i represents the difficulty percentile of the i-th sample; Then, the set L is sampled using Formula 2 to create the difficulty bias set L ′ , where samples with lower difficulty are given a higher discard rate: L ′ ={L i ∈L∶U(0,1)<2*(d i ) 4 } (2) In this formula, U(0,1) represents a uniformly distributed random number between 0 and 1; this formula means that for each sample, the corresponding loss function value L i , generate a random number U(0,1), if this random number is less than 2*(d i ) 4 , then this sample will be retained in the difficulty bias set L ′ In; due to d i For samples with higher loss function values, the value is also higher, so 2*(d i ) 4 is also higher, which means that samples with higher loss function values ​​are more likely to be retained. On the contrary, samples with lower loss function values ​​are more likely to be discarded, but they will not be completely discarded; this achieves the goal of difficulty biased sampling; Through this difficulty biased sampling technique, all samples with the highest loss function values ​​can be retained, and for the remaining samples, they will be sampled with a certain probability according to their loss function values. In this way, not only the number of training samples is reduced, but also the large model can pay more attention to those financial data with higher loss function values ​​during training, so as to learn more efficiently in the iterative process. Finally, the data set L obtained after difficulty biased sampling is ′ Sort the loss function values ​​in ascending order and fine-tune one epoch based on model B; Step 4.

3. Based on the model obtained after fine-tuning in step 4.2, repeat steps 4.1-4.2 and iterate continuously. Fine-tune only one epoch in each iteration until the average loss function value of the fine-tuned model on the entire financial data set reaches one-third of the loss function value of the model in the pre-training stage or reaches the actual number of training samples set. In this way, through adaptive learning, the model can gradually improve its understanding and processing capabilities of complex data, thereby improving the performance of the model as a whole.

4. The adaptive learning method for fine-tuning a large financial model based on loss value according to claim 3 is characterized in that: The difficulty biased sampling technique in step 4.2 can use the reward model of the RLHF part of the large model to score the training sample and the generation result of the training sample on the fine-tuned model respectively, and use the interpolation of the former score and the latter score as the judgment of the model's learning of the sample. If the score is large, it means that the model has not learned enough for the training sample and needs to learn again.

Images