Large model field adaptive fine tuning method based on hybrid task and hierarchical fine tuning

Abstract

The invention provides a large-model domain self-adaptive fine tuning method based on mixed tasks and hierarchical fine tuning, which is characterized in that a method for dynamically adjusting a sampling proportion is adopted, and the sampling proportion of a domain data set and a general data set is dynamically adjusted according to the loss reduction condition in the fine tuning process; freeze, LoRA and other fine adjustment methods are combined to carry out fine adjustment process design, the cosine similarity of a sampled general data set and a constructed field data set is calculated by using Sension Bert, and the difference between the general data set and existing data distribution of a base large model is measured by using the confusion degree; and selecting a proper general data set according to the cosine similarity and the confusion degree. Compared with the prior art, the method has good generalization, large model field self-adaptive fine adjustment is carried out from the aspects of data set construction, fine adjustment process and the like, the field capability of the large model after specific field fine adjustment is improved, the general capability of the large model is kept as much as possible, and the field fine adjustment requirement of the large model in production practice can be met.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and large language model technology, and in particular to a domain-adaptive fine-tuning method for large models based on hybrid tasks and hierarchical fine-tuning. Background Technology

[0002] In recent years, large language models (LMs), exemplified by ChatGPT, have become a significant driving force in the field of natural language processing. From models with 100 million parameters like BERT and GPT to massive models with hundreds of billions of parameters like GPT-3, the development of large models has not only greatly advanced machines' ability to understand and generate natural language but also provided strong support for various application scenarios, achieving remarkable success, especially in a wide range of general tasks. Large language models, through massive amounts of pre-trained data and parameters, exhibit a variety of capabilities, including mathematical reasoning, code generation, and instruction following. These capabilities are further enhanced after supervised fine-tuning (SFT). While the open-source community has enhanced certain single capabilities through specific supervised fine-tuning methods, proprietary large models often need to demonstrate greater versatility across multiple skills. Therefore, exploring how supervised fine-tuning can promote the development of multiple capabilities has become particularly important.

[0003] However, the surge in model parameters has created an urgent need for large-scale, high-quality corpora and computational resources, undoubtedly raising the entry barrier for researching and developing large models and posing a challenge to the popularization and development of artificial intelligence. In particular, when faced with situations requiring fine-tuning of models to meet specific industry needs, how to customize these general-purpose models to adapt to specific industries and application scenarios has become a common problem encountered by both academia and industry. This process, known as domain-specific foundational model customization, aims to address the issue that general-purpose models may not be able to fully capture the unique patterns and requirements of domain-specific data. Despite its undeniable importance, while resources on general-purpose models are abundant, review papers on constructing domain-specific foundational models are relatively scarce.

[0004] In summary, existing techniques for fine-tuning large models suffer from several limitations. First, there is insufficient research on the methods and proportions of multi-task training on domain datasets. Second, multi-task training may introduce catastrophic forgetting into large models. Third, the fine-tuning process often uses a single method and performs single-stage fine-tuning, which limits performance improvement. Specifically, this is reflected in the following aspects:

[0005] 1) In practical fine-tuning, training with mixed data from multiple tasks is a common strategy, aiming to leverage complementary information and transfer effects between tasks. Extensive experimental evidence shows that multi-task joint training often significantly improves the performance of individual sub-tasks, a phenomenon known as the "cocktail effect." This strategy is similar to mixed instruction tuning, unifying diverse tasks into instruction-response pairs for training, helping the model learn more general answering and reasoning abilities. The method of combining different task data significantly impacts the fine-tuning results. A common approach is sampling proportionally to the task data volume: allocating the number of samples in each batch according to the size of the task training set to balance the learning of high-resource and low-resource tasks. In contrast, uniform sampling (drawing the same number of samples per task) can easily lead to overfitting or neglect on tasks with small datasets. Furthermore, there are dynamic sampling strategies such as inverse temperature sampling and dynamically adjusting sampling probabilities based on performance, as well as methods that directly weight different tasks in the loss function; these all fall under the category of task mixing control. However, for different tasks in different specific domains, the above simple data mixing strategies often fail to achieve better results. How to more effectively enable LLMs to solve multiple tasks still requires further exploration.

[0006] 2) While multi-task training can promote knowledge sharing, it can also introduce catastrophic forgetting, causing the model to lose performance on previously learned tasks when learning new ones. A recent instruction tuning study showed that adding a small step of pre-training unsupervised corpus (such as C4 data) after every few steps of training with task data can significantly reduce forgetting and unexpectedly improve the model's generalization ability to tasks. This means that continuing to expose the model to a large-scale corpus it is familiar with helps maintain its basic language comprehension ability, thereby mitigating knowledge loss in multi-task learning. Although various strategies exist for improving data quality and mixing multi-task data, challenges remain. On the one hand, constructing large-scale, high-quality fine-tuning datasets is costly, requiring efficient automated cleaning and consistency evaluation techniques. Currently, there is limited systematic research on annotation consistency, domain coverage, etc., and the reliability of data for cross-domain, multilingual, or emerging tasks requires further validation. On the other hand, the theoretical foundation of multi-task fine-tuning is still incomplete: how to automatically determine the weight of each task, how to dynamically adjust the task order, and how to quantify the positive and negative effects of transfer between tasks remain open questions. Furthermore, with the rapid growth in model size, maintaining computational and sample efficiency in multi-task training has become crucial. For example, how to combine the parameter efficient fine-tuning (PEFT) method with multi-task architecture requires further research.

[0007] 3) Optimizing the fine-tuning process not only affects model performance but also directly impacts the resource costs and engineering feasibility of actual deployment. LoRA has gained widespread use in recent years. It represents the weight matrix in the Transformer as a low-rank matrix factorization, fine-tuning only two low-rank matrices (A and B) while freezing the original parameters. This method can significantly reduce training overhead while preserving the original model structure.

[0008] 4) To achieve excellent performance on domain-specific tasks while minimizing the reduction in the model's general capabilities, some studies have proposed special fine-tuning strategies. For example, this involves sequentially learning specialized and general capabilities while retaining a small subset of specialized data for two-stage hybrid fine-tuning. Compared to the commonly used single-stage hybrid dataset fine-tuning, this approach does improve domain-specific capabilities while reducing the decline in the model's general capabilities after fine-tuning. However, a drawback is that the domain-specific capabilities of the fine-tuned model are not as good as those achieved using traditional methods. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by providing a large-scale model-based domain adaptive fine-tuning method based on hybrid tasks and hierarchical fine-tuning. This method employs a dynamic adjustment of the sampling ratio, dynamically adjusting the sampling proportion of the domain dataset and the general dataset based on the loss reduction during fine-tuning. It also incorporates the characteristics of fine-tuning methods such as Freeze and LoRA in its fine-tuning process design. Based on the fine-tuning dataset used in a specific domain, this method obtains a question-answer pair dataset through data preprocessing and annotation, constructs a three-segment prompt word template, and transforms it into a standard Alpaca format dataset, thus obtaining the domain data for fine-tuning. For different tasks in the domain dataset, the large model labels different domain tasks and then scores the complexity of different domain tasks. Weights are assigned to the number of labels and complexity, with a larger sampling ratio given to task datasets with more labels and higher complexity. Sentence BERT is used to calculate the cosine similarity between the sampled general dataset and the constructed domain dataset, and perplexity is used to measure the difference between the general dataset and the existing data distribution of the base large model. A suitable general dataset is selected based on cosine similarity and perplexity. This invention explores a domain-specific adaptive fine-tuning method for large models from the perspectives of fine-tuning dataset construction and fine-tuning process optimization. It primarily focuses on evaluating the domain-specific and general capabilities of a general model after fine-tuning in a specific domain. By addressing dataset construction and fine-tuning processes, it achieves domain-specific adaptive fine-tuning of large models, improving their domain-specific capabilities while preserving their general capabilities as much as possible. It also exhibits good generalization ability, effectively improving model performance and significantly meeting the domain-specific fine-tuning needs of large models in production practice, thus demonstrating broad engineering application prospects.

[0010] The specific technical solution to achieve the purpose of this invention is: a large-scale model domain adaptive fine-tuning method based on hybrid tasks and hierarchical fine-tuning. Its characteristic is the use of a dynamic adjustment method for the sampling ratio. Based on the loss reduction during fine-tuning, the sampling ratio of the domain dataset and the general dataset is dynamically adjusted. Furthermore, the fine-tuning process is designed in conjunction with fine-tuning methods such as Freeze and LoRA, specifically including:

[0011] Step 1: Fine-tuning data collection and preprocessing

[0012] Collect fine-tuning data for a specific domain, deduplicate and enhance the data through manual sorting and automated scripts, and filter out data that may involve privacy and security issues to obtain preprocessed fine-tuning domain data.

[0013] Step 2: Question-answer pair dataset

[0014] The preprocessed fine-tuned domain data is labeled to obtain a dataset in the form of question-answer pairs;

[0015] Step 3: Fine-tune the data

[0016] Text extraction and integration were performed on the question-and-answer pair dataset to construct a three-segment prompt word template, resulting in a dataset in standard Alpaca format.

[0017] Step 4: Determine the proportion of domain tasks

[0018] We sampled data from different domain tasks in the standard alpaca format dataset, used large models including ChatGPT to score the complexity of the sampled data and assign labels, and assigned a larger sampling ratio to task datasets with more labels and higher complexity to obtain the sampling weights for different domain tasks.

[0019] Step 5: Determine the common dataset to use

[0020] Using Sentence Bert sampling, the cosine similarity between the domain dataset obtained in step 4 and commonly used candidate general datasets is calculated. The general dataset with lower cosine similarity and higher perplexity is selected as the general dataset to be used.

[0021] Step 6: Expert Large Model

[0022] A two-stage hierarchical fine-tuning process is performed on the determined general dataset to obtain a large expert model for a specific domain.

[0023] The data preparation for fine-tuning the large model domain in step 1 specifically includes:

[0024] 1.1: Obtain the required domain-specific data through methods such as system logs, web crawlers, or manual collection. In order to improve the fine-tuning effect, try to collect and organize fine-tuning data of different domain tasks in this domain.

[0025] 1.2: Data Preprocessing: The data collected in step 1.1 is preprocessed, and similar or identical data are filtered out. Data augmentation is performed using methods such as sentence rewriting and translation.

[0026] The data format conversion and question-answer pair construction in step 2 specifically include:

[0027] 2.1: Text Formatting and Transformation Method Based on Structured Question-Answer Data: For the collected domain-specific fine-tuning data, a Python script is used for processing. First, the data is read and the "question" and "answer" fields are extracted. Then, a formatting module combines the content into structured text, adding necessary connecting statements to enhance semantic coherence, and using a large language model (such as GPT-4o) to optimize awkward text. Finally, the processed content is saved in a standard question-answer pair format for subsequent use.

[0028] Step 3, which involves constructing a three-segment prompt word template to obtain the final fine-tuning data, specifically includes:

[0029] 3.1: Dataset Transformation Process: A Python script is used to convert the TXT file containing question-answer pairs into Alpaca dataset format; "questions" and "answers" are extracted using regular expressions or string splitting, with "questions" stored in the `instruction` field and "answers" stored in the `output` field, and a structured JSON object is generated. Finally, all question-answer pairs are saved as JSON files conforming to Alpaca format for easy use in subsequent fine-tuning tasks.

[0030] The construction of the domain dataset in step 4 specifically includes:

[0031] 4.1: Based on the fine-tuning data obtained in step 3, samples are taken from different domain tasks in this field. Large models such as ChatGPT are used to score the complexity of the sampled domain task data and assign labels. A larger sampling ratio is assigned to task datasets with more labels and higher complexity to obtain the final domain dataset.

[0032] 4.2: Based on the constructed domain dataset, determine the selection of a general dataset, as follows:

[0033] 4.2.1: In a domain dataset with a determined mixing ratio of domain tasks, use Sentence Bert to calculate the cosine similarity between the domain dataset and a series of alternative general datasets, and select the general dataset with lower cosine similarity and higher perplexity for the final mixed dataset.

[0034] 4.3: During fine-tuning, dynamically adjust the sampling ratio between the domain dataset and the general dataset:

[0035] 4.4: Based on the domain dataset and the general dataset, the final dataset for fine-tuning is obtained. In the subsequent fine-tuning process, the sampling ratio of the domain dataset and the general dataset is dynamically adjusted according to the loss decrease in each epoch.

[0036] The two-stage layered fine-tuning in step 6 is as follows:

[0037] 6.1: Use the fine-tuning dataset for the first stage of fine-tuning. Write a Freeze fine-tuning script based on the open-source deep learning framework PyTorch, specify the last few trainable_layers in the large model to fine-tune, freeze the other layers of the large model, and set the requires_grad parameter of these layers to False so that the gradients of these layers will not be calculated during backpropagation during fine-tuning.

[0038] 6.2: Loading the Qwen3 model using the transformer library and writing the NEFTune script;

[0039] 6.3: Load the training data and Qwen3 model into the GPU, and run the fine-tuning script to perform parameter freezing and fine-tuning;

[0040] 6.4: Second-stage fine-tuning was performed using a general dataset with a small amount of domain data. A LoRA fine-tuning script was written based on the open-source deep learning framework PyTorch, specifying the fine-tuning parameters rank r and weight alpha.

[0041] 6.5: Loading the Qwen3 model using the transformer library and writing the NEFTune script;

[0042] 6.6: Load the training data and Qwen3 model into the GPU and run the fine-tuning script for efficient parameter fine-tuning.

[0043] The fine-tuning training process of the domain-specific expert large model introduces the distributed training framework DeepSpeed, and adopts the ZeRO2 scheme in DeepSpeed.

[0044] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0045] 1) Significantly reduce the amount of annotation and improve efficiency: Large language models can be fine-tuned with a small amount of high-quality data, which significantly improves the domain capability performance of large models, reduces the catastrophic forgetting of the original knowledge and other domain capabilities of large models, and can improve the practical application performance of large models in specific domains.

[0046] 2) Generalization and Adaptability: Large language models are trained on extensive datasets during both the pre-training and post-training phases, resulting in better generalization capabilities and the ability to adapt to different natural language tasks. By fine-tuning and introducing retrieval-enhanced generation modules to adapt to domain-specific knowledge, their application effectiveness in specialized fields can be improved, demonstrating superior performance compared to traditional machine learning models and small language models.

[0047] 3) Improve training efficiency: By introducing model optimization modules such as DeepSpeed, the training efficiency of the model is greatly improved, enhancing the practical application value of the model. Attached Figure Description

[0048] Figure 1 This is a flowchart of the present invention;

[0049] Figure 2 This is a schematic diagram of the dataset structure during the training of the Qwen3 expert large model.

[0050] Figure 3 This is a schematic diagram of the loss function curve during the training process of the Qwen3 expert large model. Detailed Implementation

[0051] See Figure 1 This invention employs a method of dynamically adjusting the sampling ratio. Based on the loss reduction during fine-tuning, the sampling ratio of the domain dataset and the general dataset is dynamically adjusted. The fine-tuning process is designed in conjunction with fine-tuning methods such as Freeze and LoRA, and specifically includes the following steps:

[0052] Step 1: Construct domain fine-tuning data

[0053] See Figure 2 In collecting fine-tuning data for specific domains, methods such as system logs and web crawlers are often used. However, this data may contain a large amount of duplicate, synonymous, and low-quality data, as well as potential privacy and security issues. By using a combination of manual processing and automated scripts, this type of data suitable for fine-tuning large language models is deduplicated and augmented, and data that may involve privacy and security issues is filtered out, resulting in preprocessed fine-tuning domain data.

[0054] Step 2: Label the preprocessed fine-tuned domain data obtained in Step 1 to obtain a dataset in a unified question-answer pair format.

[0055] Step 3: Extract and integrate the text from the question-and-answer format dataset obtained in Step 2, construct a three-segment prompt word template, and obtain a standard alpaca format dataset for subsequent fine-tuning.

[0056] Step 4: Using the fine-tuned data obtained in Step 3, sample from different domain tasks, use large models such as ChatGPT to score the complexity of the sampled data and assign labels, and assign a larger sampling ratio to task datasets with more labels and higher complexity.

[0057] Step 5: After obtaining the sampling weights for different domain tasks in Step 4, use Sentence Bert to calculate the cosine similarity between the domain dataset and the candidate general dataset, and select the general dataset with lower cosine similarity and higher perplexity for the final mixed dataset.

[0058] Step 6: During the fine-tuning process, dynamically adjust the sampling ratio of the domain dataset and the general dataset based on the loss decrease in each epoch.

[0059] Step 7: Perform two-stage, layered fine-tuning to optimize the fine-tuning process and avoid catastrophic omissions;

[0060] Step 8: In the fine-tuning training process in Step 7, the distributed training framework DeepSpeed ​​and the zero-redundancy optimizer ZeRO are introduced to reduce the resources required for model and data parallelism. After fine-tuning training, a large expert model for a specific domain is obtained.

[0061] The data preparation for fine-tuning the large model domain in step 1 specifically includes:

[0062] 1.1: Use methods such as system logs, web crawlers, or manual collection to obtain the required domain-specific data. In order to improve the fine-tuning effect, try to collect and organize fine-tuning data of different domain tasks in this domain.

[0063] 1.2: Data Preprocessing: Data preprocessing is performed on the data collected in step 1.1. Data that is similar or identical is filtered out, and data augmentation is performed using methods such as sentence rewriting and translation.

[0064] The data format conversion and question-answer pair construction in step 2 specifically include:

[0065] 2.1: Text Formatting and Transformation Method Based on Structured Question-Answer Data: For the collected domain-specific fine-tuning data, a Python script is used for processing. First, the data is read and the "question" and "answer" fields are extracted. Then, a formatting module combines the content into structured text, adding necessary connecting statements to enhance semantic coherence, and using a large language model (such as GPT-4o) to optimize awkward text. Finally, the processed content is saved in a standard question-answer pair format for subsequent use.

[0066] Step 3, which involves constructing a three-segment prompt word template to obtain the final fine-tuning data, specifically includes:

[0067] 3.1: Dataset Transformation Process: A Python script is used to convert the TXT file containing question-answer pairs to the Alpaca dataset format. "Questions" and "Answers" are extracted using regular expressions or string splitting. "Questions" are stored in the `instruction` field, and "Answers" are stored in the `output` field, generating a structured JSON object. Finally, all question-answer pairs are saved as JSON files conforming to the Alpaca format for later fine-tuning.

[0068] The process of constructing the domain dataset in step 4 specifically includes:

[0069] 4.1: Based on the fine-tuning data obtained in step 3, samples are taken from different domain tasks in this field. Large models such as ChatGPT are used to score the complexity of the sampled domain task data and assign labels. A larger sampling ratio is assigned to task datasets with more labels and higher complexity to obtain the final domain dataset.

[0070] Step 5, based on the domain dataset constructed in step 4, determines the selection of a general dataset, as follows:

[0071] 5.1: In step 4, a domain dataset with a determined domain task mixing ratio was obtained. SentenceBert was used to calculate the cosine similarity between the domain dataset and a series of alternative general datasets. The general dataset with lower cosine similarity and higher perplexity was selected for the final mixed dataset.

[0072] In step 6, during the fine-tuning process, the sampling ratio of the domain dataset and the general dataset is dynamically adjusted, as follows:

[0073] 6.1: Based on the domain dataset and general dataset obtained in step 5, the final dataset for fine-tuning is obtained. In the subsequent fine-tuning process, the sampling ratio of the domain dataset and the general dataset is dynamically adjusted according to the loss decrease in each epoch.

[0074] The two-stage layered fine-tuning in step 7 is as follows:

[0075] 7.1: Use the fine-tuning dataset obtained in step 5 to perform the first stage of fine-tuning. Write a Freeze fine-tuning script based on the open-source deep learning framework PyTorch, specify the last few trainable_layers in the large model to fine-tune, freeze the other layers of the large model, and set the requires_grad parameter of these layers to False. During fine-tuning, the gradients of these layers will no longer be calculated in backpropagation.

[0076] 7.2: Loading the Qwen3 model using the transformer library and writing the NEFTune script;

[0077] 7.3: Load the training data and Qwen3 model into the GPU, and run the fine-tuning script to freeze the parameters.

[0078] 7.4: Second-stage fine-tuning was performed using a general dataset with a small amount of domain data. A LoRA fine-tuning script was written based on the open-source deep learning framework PyTorch, specifying the fine-tuning parameters rank r and weight alpha.

[0079] 7.5: Loading the Qwen3 model using the transformer library and writing the NEFTune script;

[0080] 7.6: Load the training data and Qwen3 model into the GPU and run the fine-tuning script for efficient parameter fine-tuning.

[0081] Step 8 introduces the distributed training framework DeepSpeed ​​during the fine-tuning training of the Qwen expert large model constructed in step 7, and adopts the ZeRO2 scheme in DeepSpeed.

[0082] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0083] Example 1

[0084] See Figure 1 The present invention specifically includes:

[0085] (a) Data collection

[0086] 1) Domain-specific data

[0087] We collect fine-tuning data for specific domains using common methods such as system logs and web crawlers. Through manual processing and automated scripts, we deduplicate and enhance this type of data suitable for fine-tuning large language models, and filter out data that may involve privacy and security issues, thus obtaining preliminary fine-tuning domain data.

[0088] 2) Data preprocessing

[0089] Data processing is performed using Python scripts. First, the initial data file is read, and the "question" and "answer" fields for each line are extracted using a data parsing module. Then, a formatting module combines the extracted content into a text structure according to a predetermined format, maintaining the semantic integrity of the original question-answer pairs and adding semantically coherent connecting statements to enhance readability when necessary. A large language model with more parameters, such as GPT-4o, is used to process and optimize some awkward text while preserving semantics. Finally, a text output module saves the formatted content as a standard question-answer pair format for subsequent data processing.

[0090] 3) High-quality tabular data filtering

[0091] The initial data is used to select high-quality data that is representative and logical in the field, while also paying attention to the semantic diversity of different data.

[0092] (ii) Data integration

[0093] 1) Extract table information: Extract the preprocessed table data into text format, denoted as "content information".

[0094] 2) Prompt word template sampling: To avoid the classification results output by the expert model being too sensitive to prompt words, a prompt word candidate set was designed. For each sample, a template "prompt word" is randomly sampled, which can effectively alleviate the sensitivity of the expert model to prompt words.

[0095] 3) Input text construction

[0096] See Figure 3 When constructing the input text, "examples" can be selectively added to enhance the context learning ability of the expert large model. The final constructed input text = {prompt words} + {content information} + {examples}.

[0097] (III) Fine-tuning and organizing data

[0098] 1) Alpaca Format Data and Three-Part Prompt Template: Alpaca is a fine-tuning dataset format built on Meta's open-source LLaMA model, specifically designed for instruction-tuning. Its key feature is the provision of a three-part prompt template, containing a clear task description (instruction), input, and output. To convert a TXT file containing question-answer pairs to the Alpaca dataset format, a script was written using Python. First, the Python script reads the TXT file, with each record formatted as "Question: <Question Content>" and "Answer: <Answer Content>". Each question-answer pair is then extracted using regular expressions or simple string splitting methods.

[0099] 2) The script first reads each line of data, extracting the corresponding text content by searching for keywords in "question" and "answer". Then, it saves the extracted "question" and "answer" in Alpaca format, storing the "question" in the `instruction` field and the "answer" in the `output` field. For each question-answer pair, a JSON object is generated using an appropriate structured approach.

[0100] 3) The processed data is saved as a new JSON file according to the Alpaca dataset format. Each question-answer pair is stored as an independent JSON object, which eventually forms a complete Alpaca format dataset for subsequent fine-tuning and other work.

[0101] (iv) Fine-tuning training of expert large model

[0102] 1) After data integration, efficient parameter fine-tuning techniques such as Freeze, LoRA, and NEFTune are combined to improve the performance and adaptability of large language models for specific tasks. Efficient parameter fine-tuning enables efficient downstream task adaptation while maintaining a low parameter count, thereby improving the platform's flexibility and efficiency. NEFTune enhances the generalization ability and robustness of expert large models by adding noise to the training data, further ensuring the quality and reliability of the expert large model's output.

[0103] 2) Freeze (Freeze Fine-tuning): During training, only a small portion of the model's weights are updated, while most of the original model's parameters are frozen. Only a subset of parameters are trained, allowing large models to be trained on a single GPU or without TP or PP operations. This reduces the demand on GPU memory. This fine-tuning method only requires fine-tuning high-level features, accelerating model convergence. By freezing the parameters of the first few dozen layers and only training and updating the parameters of the last few layers, it maximizes the preservation of common language features learned during the pre-training of large language models, resulting in strong interpretability.

[0104] 3) Directly fine-tuning all parameters of a large model may lead to overfitting on downstream tasks. The Freeze method reduces the risk of overfitting by fixing the underlying parameters and preserving the general characteristics of the pre-trained model. This method is particularly suitable for fine-tuning datasets with smaller amounts of data, reducing the data collection requirements of enterprise payment platforms and shifting the focus from data quantity to data quality. It allows for the collection of more diverse and logically sound question-and-answer data.

[0105] 4) LoRA fine-tuning: By training two low-rank matrices A and B in a specified module of a large model, knowledge of the fine-tuning dataset is injected. The number of parameters is usually only 0.8%-1.5% of the number of parameters in the pre-trained model. Compared with full fine-tuning, it greatly saves the storage space occupied by the pre-trained model.

[0106] For newly added tasks, developers only need to annotate a small amount of data to fine-tune the corresponding tasks, and only need to change the LoRA parameters during deployment to adapt to the new tasks, which greatly improves the model's reusability and flexibility.

[0107] 5) Noise Embedding Finetuning (NEFTune) is used to enhance and perturb the input training data, improving the generalization ability of expert large models. This alleviates overfitting during the finetuning stage of expert large models and the "illusion problem" that occurs in their application. It is an innovative fine-tuning technique. Its core algorithm introduces uniformly distributed noise into word vectors to achieve data diversity and robustness. Specifically, in each training iteration, the algorithm adds uniformly distributed noise to the word vectors. This technique is not only simple to implement but also does not introduce new model parameters, requiring no additional computational resources or data overhead. NEFTune not only improves the expert large model's tolerance to noise but also enhances its adaptability to unseen samples, thus achieving better performance in practical applications. In actual experiments, introducing NEFTune can improve accuracy by approximately 8%.

[0108] (V) Optimization of Expert Large Model Training

[0109] As the scale of deep learning models continues to increase, the computational and memory requirements during training also rise. Therefore, this invention introduces DeepSpeed ​​to accelerate training. DeepSpeed ​​is a distributed training framework open-sourced by Microsoft. It significantly reduces the resources required for model parallelism and data parallelism through a novel parallel optimizer, ZeRO (Zero Redundancy Optimizer), thus accelerating model training. This invention adopts the ZeRO2 scheme from DeepSpeed ​​to optimize the storage of optimizer states and gradients, enabling large expert models to be trained in parallel in a distributed environment, thereby maximizing the utilization of GPU computing resources. After applying DeepSpeed, the training speed of large expert models is significantly improved. Compared with traditional training methods, training time is reduced by more than 50%, while handling larger-scale models. This improvement not only enhances training efficiency but also reduces the demand for computational resources, making the training of large-scale models more feasible.

[0110] (vi) Expert large-scale model evaluation

[0111] 1) Evaluation Dataset Preparation: Before fine-tuning the large model, the organized fine-tuning dataset is divided into a training set and a test set in a ratio of 4:1. Stratified sampling is used to ensure that the training set and the test set can cover the samples of each task.

[0112] 2) Evaluation Results

[0113] Hyperparameters play a crucial role in the fine-tuning of expert large models, and their settings directly affect the performance and convergence speed of these models. The fine-tuning process of this invention includes several important hyperparameters, which are described below:

[0114] `freeze_trainable_layers` indicates the range of parameters for Freeze fine-tuning. Generally, the last few layers of the model are used for fine-tuning, while the parameters of other layers remain unchanged.

[0115] , r, represent the rank of LoRA fine-tuning, which affects the number of parameters of low-rank matrices A and B and the computational cost of model fine-tuning in LoRA fine-tuning;

[0116] alpha represents the scaling hyperparameter for the calculation results of the low-rank matrix in LoRA fine-tuning;

[0117] , lr, represent the learning rate for fine-tuning the expert model, which affects the rate at which the loss function decreases during model fine-tuning and the final convergence of the fine-tuning;

[0118] num_train_epochs represents the number of training epochs on the dataset during fine-tuning, with 1 epoch representing one round of training of the model on the entire dataset;

[0119] neftune_alpha represents the noise scaling parameter used by NEWTune.

[0120] After trying various parameter combinations, the hyperparameters were finally determined to be: freeze_trainable_layers = 8, r = 64, alpha = 128, lr = 5e-5, num_train_epochs = 5, neftune_alpha = 5.

[0121] See Figure 3 The decrease in the loss function during fine-tuning is shown in the figure. Other large-scale model inference settings were based on the official Qwen3 GitHub settings, and all experiments were completed using a single A100 graphics card. Testing was conducted using an enterprise expense reporting question-and-answer dataset. The BLEU score quantifies the quality of the generated text by comparing the n-gram (n consecutive words) matching degree between the generated text and the reference text; generally, 4-grams are used to measure the generation of short sentences in the generated text. The Rouge-N score splits the generated text and the reference text into n-grams and evaluates the text generation quality by calculating the recall of the reference text. The Rouge-L score comprehensively evaluates the text generation quality by calculating the recall and precision of the longest common subsequence (LCS) of the generated text and the reference text. Testing was conducted using the fine-tuned Qwen3 expert large-scale model, with BLEU-4 = 54.56%, Rogue-1 = 66.23%, Rogue-2 = 55.05%, and Rogue-L = 60.29%. It is evident that the expert large-scale model of this invention performs well on multiple evaluation metrics.

[0122] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A large model field adaptation fine-tuning method based on mixed tasks and hierarchical fine-tuning, characterized in that, Adopting dynamic adjustment of sampling ratio and combining the fine-tuning methods of Freeze and LoRA, the large model field adaptive fine-tuning is realized, which specifically includes the following steps: Step 1: Collection and preprocessing of fine-tuning data Collect fine-tuning data in a specific field, and use manual sorting and automated scripts to remove duplicate data and enhance it. Data that may involve privacy and security issues are excluded to obtain preprocessed fine-tuning domain data. Step 2: Question and answer pair data set Label the preprocessed fine-tuning domain data to obtain a question and answer pair data set. Step 3: Fine-tuning data Extract and integrate the question and answer pair data set into a three-part prompt word template to obtain a standard alpaca format data set. Step 4: Determine the proportion of domain tasks Sample the standard alpaca format data set from different domain tasks, use the ChatGPT large model to score the complexity of the sampled data and assign labels, and assign a larger sampling ratio to task data sets with more labels and higher complexity. Get the sampling weight of different domain tasks. Step 5: Determine the use of general data set Use Sentence Bert to sample the cosine similarity between the domain data set and the commonly used alternative general data set, and select the general data set with lower cosine similarity and higher perplexity as the used general data set. Step 6: Expert large model Two-stage hierarchical fine-tuning of the determined general data set to obtain a specific domain expert large model.

2. The large model domain self-adaption fine-tuning method based on mixed task and hierarchical fine-tuning according to claim 1, characterized in that, The step 1 specifically includes: Step 1-1: Use system logs, crawlers or manual collection methods to obtain the required specific domain data. Step 1-2: Filter similar or identical data from the collected specific domain data, and use sentence rewriting and translation methods for data enhancement preprocessing.

3. The large model domain self-adaption fine-tuning method based on mixed task and hierarchical fine-tuning according to claim 1, characterized in that, The step 2 uses a structured question and answer data text formatting conversion method to process the collected domain fine-tuning data using Python scripts, which specifically includes: Step 2-1: Read the domain fine-tuning data and extract the "question" and "answer" fields. Step 2-2: Combine the extracted "question" and "answer" fields into structured text through the formatting module. Step 2-3: Add connection statements to enhance semantic coherence, optimize the text using a large language model, and save it as a standard question and answer pair format.

4. The large model domain self-adaption fine-tuning method based on mixed task and hierarchical fine-tuning according to claim 1, characterized in that, The step 3 specifically includes: Step 3-1: Convert the TXT file containing the question and answer pair to the Alpaca data set format using Python scripts. Step 3-2: Extract "question" and "answer" through regular expressions or string splitting, store "question" in the instruction field, and "answer" in the output field, and generate JSON objects in a structured manner. Step 3-3: Save all question and answer pairs as JSON files in Alpaca format.

5. The large model domain self-adaption fine-tuning method based on mixed task and hierarchical fine-tuning according to claim 1, characterized in that, The step 6 specifically includes: Step 6-1: Dynamically adjust the sampling ratio of the domain data set and the general data set according to the loss of each epoch. Step 6-2: Two-stage hierarchical fine-tuning training is performed using the distributed training framework DeepSpeed and the zero-redundancy optimizer ZeRO to obtain a domain expert large model. During the fine-tuning training process, the distributed training framework DeepSpeed and the zero-redundancy optimizer ZeRO are introduced to reduce the resources required for model and data parallelism.

6. The large model domain self-adaption fine-tuning method based on mixed task and hierarchical fine-tuning according to claim 5, characterized in that, The two-stage hierarchical fine-tuning training in step 6-2 specifically includes: Step 6-2-1: Write a Freeze fine-tuning script based on the open-source deep learning framework pytorch, specify the last several layers trainable_layers in the fine-tuning large model, freeze the other layers of the large model, and set the requires_grad parameter of these layers to False. Perform the first stage of fine-tuning on the fine-tuning dataset. Step 6-2-2: Load the Qwen3 model based on the transformer library and write the NEFTune script. Step 6-2-3: Load the training data and Qwen3 model into the GPU, run the fine-tuning script for parameter freezing fine-tuning Freeze, and the training data uses the final dataset mixed with domain data and general data. Step 6-2-4: Write a LoRA fine-tuning script based on the open-source deep learning framework pytorch, specify the fine-tuning parameter rank r and weight alpha, and use the general dataset mixed with a small amount of domain data to perform the first stage of fine-tuning on the first stage of fine-tuning dataset. Step 6-2-5: Load the Qwen3 model based on the transformer library and write the NEFTune script. Step 6-2-6: Load the above training data and Qwen3 model into the GPU, and run the fine-tuning script for parameter fine-tuning.

Images