Large language model fine tuning method, device and equipment and storage medium

Abstract

The invention discloses a large-scale language model fine tuning method, device and equipment and a storage medium, and relates to the technical field of large-scale language model fine tuning. According to the method, a preset check point is firstly subjected to large-scale language model fine tuning based on training loss performance, sample embedding space distribution and sample reply score performance of a current model; and performing multi-dimensional self-reference diagnosis on the training data set, identifying a suboptimal sample which is not matched with the capability of the current model, and then processing the suboptimal sample through an adaptive optimization engine to complete dynamic evolution of the training data set. Therefore, dynamic evolution of the data set adaptation model can be realized, the problem of redundancy or insufficient adaptation of static data is avoided, and the training efficiency is improved; meanwhile, data waste is reduced, data diversity is reserved, the model generalization ability is promoted, manual intervention is not needed, and the data optimization cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model fine-tuning, and particularly relates to a large language model fine-tuning method, device, equipment and storage medium. BACKGROUND

[0002] Supervised fine-tuning (SFT) of large language models (LLMs) is a core technology in the field of artificial intelligence to improve the adaptation capability of models to downstream tasks, and its effect is highly dependent on the quality of training data. Existing data optimization methods are all one-time static processing before fine-tuning, mainly including three categories: data set filtering selection based on heuristic rules or model indicators, such as Alpaca-clean removing low-quality instructions; data set enhancement synthesis with the help of teacher models, such as WizardLM generating diverse samples through instruction evolution; and mixed general data or regularization disaster forgetting alleviation methods.

[0003] However, these static open-loop operations have significant defects: the fixed data set is inconsistent with the ability of dynamic evolution of the model, leading to data redundancy or hindering learning in the later training period, and low efficiency; traditional "hard filtering" strategies easily lose data diversity and "difficult examples" containing valuable information, damaging the generalization ability of the model; disaster forgetting alleviation methods either rely on pre-training data that are difficult to obtain or sacrifice downstream task performance, and it is difficult to balance. Therefore, there is an urgent need for an optimization technology that realizes the co-evolution of data and models, efficiently utilizes data, and alleviates forgetting, breaking through the limitations of existing static processing. SUMMARY

[0004] To solve the above problems, the present application provides a large language model fine-tuning method, device, equipment and storage medium, including the following contents:

[0005] In a first aspect, the present application provides a large language model fine-tuning method, which comprises:

[0006] At a preset checkpoint of the large language model fine-tuning, based on the training loss performance of the current model, the sample embedding space distribution and the sample reply score performance, a multi-dimensional self-referential diagnosis is performed on the training data set to identify suboptimal samples that do not match the current model capability.

[0007] The suboptimal samples are processed by an adaptive optimization engine to complete the dynamic evolution of the training data set.

[0008] Optionally, the multi-dimensional self-referential diagnosis of the training data set based on the training loss performance of the current model, the sample embedding space distribution and the sample reply score performance to identify suboptimal samples that do not match the current model capability comprises:

[0009] record the training loss of each sample in the training data set at each training stage corresponding to the current preset checkpoint, and if the training loss of the sample is greater than a preset loss threshold in continuous multiple training stages, mark the sample as a complexity mismatch sample;

[0010] Encode the instruction part of all samples in the training data set into embedding vectors respectively, and perform clustering processing on the embedding vectors using a clustering algorithm, and mark the samples located in the sparse area after clustering as samples with insufficient diversity;

[0011] Use the current model to score the clarity, factuality and helpfulness of each sample in the training data set, and if the score is lower than a preset threshold, mark the corresponding sample as a quality mismatch sample.

[0012] Optionally, the processing of the identified suboptimal sample includes:

[0013] For the complexity mismatch sample, a large language model is called to simplify and rewrite the complexity mismatch sample;

[0014] For the sample with insufficient diversity, the sample with insufficient diversity is used as a seed to generate new samples that are semantically similar but have different sentence patterns;

[0015] For the quality mismatch sample, according to the low score dimension of the quality mismatch sample, the quality mismatch sample is rewritten accordingly.

[0016] Optionally, the method further includes:

[0017] At each training step of the large language model fine-tuning, a statistical table of the running mean and standard deviation of the historical loss of each dialogue turn is recorded;

[0018] Based on the deviation of the loss value of each sample from the historical statistical data of the corresponding dialogue turn, the reliability of the supervision signal of the sample is evaluated;

[0019] If the loss value of the sample exceeds the sum of the running mean and twice the standard deviation of the dialogue turn corresponding to the sample, it is determined that the supervision signal of the sample is an unreliable signal, and the weight of the sample in the loss calculation process is dynamically reduced.

[0020] Optionally, the method further includes:

[0021] Before training starts, a copy of the original base model is loaded, a parameter freezing operation is performed on the copy of the original base model, and the copy of the original base model after parameter freezing is configured as a reference model;

[0022] A regularization term is added to a loss function used for model training, input parts of samples in a training data set are respectively input into a current model and a reference model, a first group of logits distributions are output by the current model, a second group of logits distributions are output by the reference model, and a KL divergence between the first group of logits distributions and the second group of logits distributions is calculated.

[0023] A total loss calculation formula is set as L total =L task +λ×L KL , wherein L task is a standard cross-entropy loss, L KL is a KL divergence loss corresponding to the KL divergence, λ is a regularization term weight coefficient, and the model training process is constrained through the total loss calculation formula, so that the model retains a basic semantic representation.

[0024] In a second aspect, the present application provides a large language model fine-tuning device, which comprises:

[0025] An identification unit is configured to perform multi-dimensional self-referential diagnosis on a training data set based on training loss performance of a current model, sample embedding space distribution, and sample reply score performance at a preset checkpoint of large language model fine-tuning, and identify suboptimal samples that do not match the capability of the current model.

[0026] A processing unit is configured to process the suboptimal samples through an adaptive optimization engine to complete dynamic evolution of the training data set.

[0027] Optionally, the identification unit is specifically configured to record training losses of each sample in the training data set at each training stage corresponding to the current preset checkpoint, and mark the sample as a complexity mismatch sample if the training loss of the sample is greater than a preset loss threshold in continuous multiple training stages.

[0028] The instruction part of each sample in the training data set is respectively encoded into an embedding vector, and a clustering algorithm is used to perform clustering processing on the embedding vector, and samples located in a sparse area after clustering are marked as diversity insufficient samples.

[0029] The reply of each sample in the training data set is scored for clarity, factuality, and helpfulness by using the current model, and the corresponding sample is marked as a quality mismatch sample if the score is lower than a preset threshold.

[0030] Optionally, the processing unit is specifically configured to call a large language model to simplify and rewrite the complexity mismatch sample.

[0031] For the samples with insufficient diversity, new samples with similar semantics but different sentence structures are generated using the samples with insufficient diversity as seeds;

[0032] For the quality mismatched samples, the samples are rewritten specifically based on their low-scoring dimensions.

[0033] Optionally, the device further includes a dynamic weighting unit for recording a statistical table of the running mean and standard deviation of the historical loss for each dialogue turn at each training step of fine-tuning the large language model.

[0034] The reliability of the supervision signal for each sample is evaluated based on the degree of deviation between the loss value of each sample and the historical statistics of the corresponding dialogue round.

[0035] If the loss value of the sample exceeds the sum of the running mean of the corresponding dialogue round and twice the standard deviation, the supervision signal of the sample is determined to be unreliable, and the weight of the sample in the loss calculation process is dynamically reduced.

[0036] Optionally, the device further includes: a semantic preservation unit, used to load a copy of the original base model before training starts, perform a parameter freezing operation on the copy of the original base model, and configure the parameter-frozen copy of the original base model as a reference model.

[0037] Add a regularization term to the loss function used for model training, input the input part of the samples in the training dataset into the current model and the reference model respectively, output the first set of logits distribution through the current model, output the second set of logits distribution through the reference model, and calculate the KL divergence between the first set of logits distribution and the second set of logits distribution;

[0038] Let the total loss calculation formula be L total =L task +λ×L KL L task For standard cross-entropy loss, L KL The KL divergence loss is the KL divergence loss, where λ is the weight coefficient of the regularization term. The total loss calculation formula constrains the model training process, ensuring that the model retains its basic semantic representation.

[0039] Thirdly, this application provides an apparatus comprising a memory and a processor, the memory for storing instructions or code, and the processor for executing the instructions or code to cause the apparatus to perform the method described in any of the implementations of the first aspect.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium, which stores codes, when the codes are executed, a device executing the codes implements the method described in any implementation manner of the first aspect.

[0041] The present application provides a large language model fine-tuning method. In the execution of the method, at a preset checkpoint of large language model fine-tuning, based on the training loss performance of the current model, the sample embedding space distribution and the sample reply score performance, a multi-dimensional self-referential diagnosis is performed on the training data set to identify suboptimal samples that do not match the current model capability, and then the suboptimal samples are processed through an adaptive optimization engine to complete the dynamic evolution of the training data set. In this way, the training data set can be adapted to the dynamic evolution process of the model, avoiding the redundancy or insufficient adaptation problem of the static data set in the later training stage, improving the model training efficiency; the identification basis of the suboptimal samples is consistent with the real-time capability state of the model, ensuring that the data optimization is more targeted; through the processing of the suboptimal samples instead of directly discarding, the effective information in the original data set is retained, the loss of data diversity is reduced, and the generalization ability of the model is improved; the entire data diagnosis and optimization process does not require human intervention, reducing the operation cost of data optimization in large language model fine-tuning. BRIEF DESCRIPTION OF DRAWINGS

[0042] To make the technical solutions of the embodiments or the prior art clearer, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 A flowchart of a large language model fine-tuning method provided by an embodiment of the present application;

[0044] Figure 2 A structural schematic diagram of a large language model fine-tuning device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the following will describe the technical solutions in the embodiments of the present application clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] Figure 1A flowchart of a large language model fine-tuning method provided by an embodiment of the present application. In combination with Figure 1 As shown in the figure, the large language model fine-tuning method provided by the embodiment of the present application can include:

[0047] S101, at a preset checkpoint of large language model fine-tuning, based on the training loss performance, sample embedding space distribution and sample reply score performance of the current model, multi-dimensional self-referential diagnosis is performed on the training data set to identify suboptimal samples that do not match the current model capability.

[0048] The preset checkpoint here refers to a preset key evaluation node in the large language model fine-tuning process, and in the embodiment, it is specifically after each training cycle (epoch) is completed. Its role is to provide a fixed time node for data diagnosis and ensure that data optimization is synchronized with model training progress; multi-dimensional self-referential diagnosis is a comprehensive evaluation of the training data set from three dimensions of training loss performance, sample embedding space distribution and sample reply score performance based on the real-time capability state of the current model. The core purpose is to screen out suboptimal samples that cannot adapt to the current learning needs of the model and provide a clear direction for subsequent data optimization; suboptimal samples include complexity mismatch samples, diversity deficiency samples and quality mismatch samples. Such samples are either too complex to hinder model learning or lack diversity or have quality defects and cannot provide effective training signals for the model.

[0049] This step drives data diagnosis by the model's own state, breaking the limitations of traditional static data processing, making the identification of suboptimal samples more in line with the real-time capability of the model, solving the problem of disconnection between data and model capability in the prior art, laying a foundation for subsequent precise optimization and helping to improve the pertinence and efficiency of model training.

[0050] The specific diagnosis operation process is as follows:

[0051] Loss pattern analysis: training loss can intuitively reflect the learning difficulty and mastery of the model for the sample. Loss pattern analysis judges the adaptability of the sample to the model capability by tracking the loss change of the sample. The system first records the training loss of each sample in the training data set at the current epoch, and then continuously tracks the loss change of each sample. If the training loss of a sample is greater than the preset loss threshold (i.e., the loss value is high) for consecutive multiple epochs, it indicates that the complexity of the sample exceeds the processing capability of the current model, and it is marked as a "complexity mismatch sample".

[0052] Embedding space analysis: This analysis is used to evaluate the semantic diversity of the training dataset, ensuring that the samples cover a sufficiently wide range of application scenarios. First, the instruction portions of all samples in the training dataset are encoded into embedding vectors. These embedding vectors transform textual semantics into quantifiable mathematical representations, facilitating distribution analysis using clustering algorithms. Subsequently, the k-means or DBSCAN algorithm is used to cluster all embedding vectors. Samples located in sparse regions after clustering are marked as "insufficient diversity samples." These samples correspond to semantic scenarios that are less covered in the dataset, and their absence would limit the model's generalization ability.

[0053] Self-alignment scoring: This method assesses the quality of sample responses using the current model's own judgment capabilities, without requiring additional manual annotation or third-party tools. Using the model currently undergoing fine-tuning, each sample's response in the training dataset is scored across three dimensions: "clarity" (assessing the clarity of expression), "factual accuracy" (assessing the truthfulness and accuracy of the content), and "helpfulness" (assessing the practical value in solving the problem). The scoring range is 1-5 points. If a sample's overall score or any dimension score falls below a preset threshold, it is marked as a "quality mismatch sample."

[0054] S102. The suboptimal samples are processed by an adaptive optimization engine to complete the dynamic evolution of the training dataset.

[0055] The adaptive optimization engine is a core processing module with functions such as sample rewriting, simplification, and diversity generation. Its role is to perform targeted optimization operations on different types of suboptimal samples, so that the training dataset can keep up with the evolution of the model's capabilities and achieve dynamic updates.

[0056] This step maximizes the preservation of effective information in the original dataset by optimizing suboptimal samples instead of discarding them directly. It avoids loss of data diversity, solves the data waste problem caused by traditional "hard filtering" strategies, and enables the co-evolution of the dataset and the model, ensuring that the model can obtain suitable training signals at each learning stage.

[0057] The specific handling method is as follows: For samples with mismatched complexity, a large-scale language model (LLM) is used for simplification and rewriting, reducing the learning difficulty while preserving the core semantics. For example, the original sample instruction is "Please explain the joint liability exemption clause in the insurance policy and its applicable conditions in multi-vehicle rear-end collisions," which is rewritten as "What is the joint liability exemption clause? How does it take effect in a rear-end collision?"

[0058] Diversity-deficient sample processing: With diversity-deficient samples as semantic seeds, new samples with similar semantics but different sentence patterns and expression methods are generated by means of an adaptive optimization engine to fill the semantic sparse areas in the data set, enrich the sample types in this semantic scenario, and improve the diversity of the training data.

[0059] Quality mismatch sample processing: Rewriting according to the low-score dimensions of self-alignment scoring. If marked due to low "clarity" score, optimize expression logic to make the reply more clear; if marked due to low "factuality" score, correct the false information to ensure the content is true and accurate; if marked due to low "helpfulness" score, supplement useful information to improve the value of problem solving.

[0060] To intelligently suppress noise interference without discarding data and improve training stability, the method further includes a dynamic supervision reliability weighting step in each training step of large language model fine-tuning. Specifically, in each training step of large language model fine-tuning, a statistical table of running mean and standard deviation of historical loss of each dialogue turn is recorded; based on the deviation of the loss value of each sample from the historical statistical data of the corresponding dialogue turn, the supervision signal reliability of the sample is evaluated; if the loss value of the sample exceeds the sum of the running mean and twice the standard deviation of the dialogue turn corresponding to the sample, it is determined that the supervision signal of the sample is an unreliable signal, and the weight of the sample in the loss calculation process is dynamically reduced.

[0061] The dynamic supervision reliability weighting is a mechanism for evaluating the reliability of the supervision signal based on the historical statistical data of the sample loss and dynamically adjusting the sample weight. The core is to achieve a soft filtering effect, taking into account data utilization and training stability. This step does not rely on expensive offline labeling or third-party evaluation tools, and can achieve noise suppression through lightweight online statistics. Compared with the traditional "hard filtering" strategy, it can better preserve data diversity and improve the robustness of model training.

[0062] The specific process is as follows:

[0063] Online statistics: The system maintains a statistical table recording the historical loss of each dialogue turn (such as the first turn, the second turn, etc.). The table contains the running mean (μ turn ) and standard deviation (σ turn ) of the historical loss of each dialogue turn, which is used to reflect the normal loss fluctuation range of different dialogue turn samples and provide a benchmark for reliability evaluation.

[0064] Weight calculation: For the samples belonging to the kth dialogue in the current batch, calculate the loss value (l i ) of the current training step, if the loss value exceeds the sum of the running mean (μ k ) and twice the standard deviation (σ k ) of the dialogue turn (i.e. li > μ k + 2σ k ), the supervision signal of the sample is determined as an unreliable signal, and the weight (w i ) of the sample in loss calculation is set to a value less than 1, so as to reduce the interference of the noise signal on the model parameter update.

[0065] To prevent the model from forgetting the general knowledge accumulated in the pre-training phase (i.e., catastrophic forgetting) during the fine-tuning process, the method further includes a knowledge preservation step. Without introducing pre-training data, the step achieves knowledge preservation by constraining the semantic representation consistency between the current model and the original base model, and balances the downstream task performance and general knowledge reserve. That is, the method further includes: before starting training, loading a copy of the original base model, performing a parameter freezing operation on the copy of the original base model, and configuring the copy of the original base model after parameter freezing as a reference model; adding a regularization term to the loss function used for model training, inputting the input part of each sample in the training data set into the current model and the reference model respectively, outputting a first group of logits distribution through the current model, outputting a second group of logits distribution through the reference model, and calculating the KL divergence between the first group of logits distribution and the second group of logits distribution; setting the total loss calculation formula as L total =L task +λ×L KL , wherein L task is a standard cross-entropy loss, L KL is a KL divergence loss corresponding to the KL divergence, and λ is a regularization term weight coefficient. The model training process is constrained by the total loss calculation formula, so that the model retains the basic semantic representation.

[0066] The specific operation is as follows:

[0067] Reference model loading: before starting training, a copy of the original base model (π ref ) is loaded and its parameters are frozen, and the frozen model copy is used as a reference model to provide an original semantic representation benchmark and provide a reference standard for knowledge preservation of the current model.

[0068] KL divergence calculation: the KL divergence (i.e., relative entropy) is used to measure the difference between two groups of distributions. In each training step, for each sample in the current batch, the input part (i.e., the user's question) is input into the current fine-tuning model (π θ ) and the reference model (π ref ) respectively, the current model outputs a first group of logits distribution, the reference model outputs a second group of logits distribution, and the KL divergence (L KL ) between the two groups of logits distribution is calculated, which reflects the deviation of the current model from the original model in semantic representation.

[0069] Total loss calculation: set the total loss calculation formula as L total =L task +λ×L KL , where L task is the standard cross-entropy loss, used to ensure the downstream task adaptation effect of the model; L KL is the KL divergence loss, used to constrain the semantic representation of the current model to be consistent with the reference model, avoiding forgetting of general knowledge; 0.1 is the regularization term weight coefficient, which can be adjusted according to actual fine-tuning requirements. Through the total loss formula, the optimization of downstream task performance is realized while the basic semantic representation is preserved, effectively alleviating the catastrophic forgetting.

[0070] Through the outer loop (data diagnosis and optimization loop) composed of S101 and S102 above, and the inner loop (training process optimization loop) composed of the dynamic supervision reliability weighting and knowledge preservation steps, the present application constructs a complete "data-model" co-evolution closed-loop learning framework, so that the training data set can evolve with the growth of the model, maximize the use of original data information, improve the model performance and generalization ability, and at the same time reduce the threshold of high-quality fine-tuning.

[0071] The above is some specific implementation modes of the large language model fine-tuning method provided by the embodiments of the present application. Based on this, the present application also provides a corresponding device. The device provided by the embodiments of the present application will be introduced from the perspective of functional modularization.

[0072] Figure 2 The structure diagram of the large language model fine-tuning device provided by the embodiments of the present application. In combination with Figure 2 , the large language model fine-tuning device 200 provided by the embodiments of the present application comprises:

[0073] The recognition unit 210 is configured to perform multi-dimensional self-reference diagnosis on the training data set based on the training loss performance of the current model, the sample embedding space distribution and the sample reply score performance at the preset checkpoint of the large language model fine-tuning, and identify suboptimal samples that do not match the current model capability.

[0074] The processing unit 220 is configured to process the suboptimal samples through an adaptive optimization engine to complete the dynamic evolution of the training data set.

[0075] In an implementation mode of the embodiments of the present application, the recognition unit is specifically configured to record the training loss of each sample in the training data set at each training stage corresponding to the current preset checkpoint, and if the training loss of the sample is greater than a preset loss threshold in continuous multiple training stages, the sample is marked as a complexity mismatch sample.

[0076] Encode instruction parts of all samples in the training data set into embedding vectors respectively, and perform clustering processing on the embedding vectors by using a clustering algorithm, and mark samples located in sparse areas after clustering as samples with insufficient diversity;

[0077] Score the reply of each sample in the training data set by using the current model in terms of clarity, factuality and helpfulness, and if the score is lower than a preset threshold, mark the corresponding sample as a quality mismatch sample.

[0078] In an implementation manner of the embodiment of the application, the processing unit is specifically configured to, for the complexity mismatch sample, invoke a large language model to simplify and rewrite the complexity mismatch sample.

[0079] For the sample with insufficient diversity, a new sample with similar semantics and different sentence patterns is generated by taking the sample with insufficient diversity as a seed.

[0080] For the quality mismatch sample, the quality mismatch sample is rewritten according to the low-score dimension of the quality mismatch sample.

[0081] In an implementation manner of the embodiment of the application, the device further includes a dynamic weighting unit configured to record a statistical table of running mean and standard deviation of historical loss of each dialogue turn at each training step of fine-tuning of the large language model.

[0082] Based on the deviation degree of the loss value of each sample and the historical statistical data of the corresponding dialogue turn, the reliability of the supervision signal of the sample is evaluated.

[0083] If the loss value of the sample exceeds the sum of the running mean and twice the standard deviation of the dialogue turn corresponding to the sample, it is determined that the supervision signal of the sample is an unreliable signal, and the weight of the sample in the loss calculation process is dynamically reduced.

[0084] In an implementation manner of the embodiment of the application, the device further includes a semantic preservation unit configured to, before training is started, load a copy of an original base model, perform a parameter freezing operation on the copy of the original base model, and configure the copy of the original base model after parameter freezing as a reference model.

[0085] A regularization term is added to the loss function used for model training, the input part of the sample in the training data set is respectively input into the current model and the reference model, a first group of logits distributions is output by the current model, a second group of logits distributions is output by the reference model, and the KL divergence between the first group of logits distributions and the second group of logits distributions is calculated.

[0086] The total loss calculation formula is set as L total =L task+ λ x L KL wherein L task is a standard cross-entropy loss, L KL is a KL divergence loss corresponding to the KL divergence, and λ is a regularization term weight coefficient. The total loss calculation formula constrains the model training process, so that the model retains the basic semantic representation.

[0087] The embodiments of the present application also provide a corresponding device and a computer storage medium for implementing the scheme provided by the embodiments of the present application.

[0088] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes to enable the device to execute the method described in any embodiment of the present application.

[0089] The computer storage medium stores codes, and when the codes are executed, the device executing the codes implements the method described in any embodiment of the present application.

[0090] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (English: read-only memory, ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the method described in each embodiment or some parts of the embodiments of the present application.

[0091] It can be understood that in the specific embodiments of the present application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved need to obtain user permission or consent when the above embodiments of the present application are applied to specific products or technologies, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0092] It should be noted that, in the specification, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0093] It should also be noted that each of the embodiments described in the specification illustrate aspects of the application and are not meant to be an exhaustive list of all possible embodiments. The scope of the application is therefore intended to cover all possible embodiments that are within the scope of the claims. Furthermore, the description of the embodiments is not meant to limit the scope of the application to the exact embodiments described. The scope of the application is intended to cover all possible embodiments that are within the scope of the claims.

[0094] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art without any creative effort, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for fine-tuning a large-scale language model, characterized in that, The method includes: At the preset checkpoints for fine-tuning large language models, based on the current model's training loss performance, sample embedding space distribution, and sample response score performance, a multi-dimensional self-reference diagnosis is performed on the training dataset to identify suboptimal samples that do not match the current model's capabilities. The suboptimal samples are processed by an adaptive optimization engine to achieve dynamic evolution of the training dataset.

2. The method according to claim 1, characterized in that, Based on the training loss performance, sample embedding spatial distribution, and sample response score performance of the current model, a multi-dimensional self-referenced diagnosis is performed on the training dataset to identify suboptimal samples that do not match the capabilities of the current model, including: Record the training loss of each sample in the training dataset at each training stage corresponding to the current preset checkpoint. If the training loss of the sample is greater than the preset loss threshold in multiple consecutive training stages, the sample is marked as a complexity mismatch sample. The instruction portions of all samples in the training dataset are encoded into embedding vectors, and the embedding vectors are clustered using a clustering algorithm. Samples located in sparse regions after clustering are labeled as samples with insufficient diversity. The current model is used to score the clarity, factuality, and helpfulness of the responses to each sample in the training dataset. If the scores are lower than a preset threshold, the corresponding sample is marked as a quality mismatch sample.

3. The method according to claim 1, characterized in that, The processing of the identified suboptimal samples includes: For the samples with mismatched complexity, a large language model is invoked to simplify and rewrite the samples. For the samples with insufficient diversity, new samples with similar semantics but different sentence structures are generated using the samples with insufficient diversity as seeds; For the quality mismatched samples, the samples are rewritten specifically based on their low-scoring dimensions.

4. The method according to claim 1, characterized in that, The method further includes: At each training step of fine-tuning the large language model, a statistical table is created to record the running mean and standard deviation of the historical loss for each dialogue turn; The reliability of the supervision signal for each sample is evaluated based on the degree of deviation between the loss value of each sample and the historical statistics of the corresponding dialogue round. If the loss value of the sample exceeds the sum of the running mean of the corresponding dialogue round and twice the standard deviation, the supervision signal of the sample is determined to be unreliable, and the weight of the sample in the loss calculation process is dynamically reduced.

5. The method according to claim 1, characterized in that, The method further includes: Before training starts, a copy of the original base model is loaded, and a parameter freeze operation is performed on the copy of the original base model. The copy of the original base model after parameter freeze is then configured as the reference model. Add a regularization term to the loss function used for model training, input the input part of the samples in the training dataset into the current model and the reference model respectively, output the first set of logits distribution through the current model, output the second set of logits distribution through the reference model, and calculate the KL divergence between the first set of logits distribution and the second set of logits distribution; Let the total loss calculation formula be L total =L task +λ×L KL L task For standard cross-entropy loss, L KL The KL divergence loss is the KL divergence loss, where λ is the weight coefficient of the regularization term. The total loss calculation formula constrains the model training process, ensuring that the model retains its basic semantic representation.

6. A large-scale language model fine-tuning device, characterized in that, The device includes: The identification unit is used to perform multi-dimensional self-reference diagnosis on the training dataset based on the current model's training loss performance, sample embedding space distribution, and sample response score performance at preset checkpoints for fine-tuning of large language models, and to identify suboptimal samples that do not match the current model's capabilities. The processing unit is used to process the suboptimal samples through an adaptive optimization engine to complete the dynamic evolution of the training dataset.

7. The apparatus according to claim 6, characterized in that, The device also includes a dynamic weighting unit for recording a statistical table of the running mean and standard deviation of the historical loss for each dialogue turn at each training step of fine-tuning the large language model. The reliability of the supervision signal for each sample is evaluated based on the degree of deviation between the loss value of each sample and the historical statistics of the corresponding dialogue round. If the loss value of the sample exceeds the sum of the running mean of the corresponding dialogue round and twice the standard deviation, the supervision signal of the sample is determined to be unreliable, and the weight of the sample in the loss calculation process is dynamically reduced.

8. The apparatus according to claim 6, characterized in that, The device further includes: a semantic preservation unit, used to load a copy of the original base model before training starts, perform a parameter freezing operation on the copy of the original base model, and configure the parameter-frozen copy of the original base model as a reference model. Add a regularization term to the loss function used for model training, input the input part of the samples in the training dataset into the current model and the reference model respectively, output the first set of logits distribution through the current model, output the second set of logits distribution through the reference model, and calculate the KL divergence between the first set of logits distribution and the second set of logits distribution; Let the total loss calculation formula be L total =L task +λ×L KL L task For standard cross-entropy loss, L KL The KL divergence loss is the KL divergence loss, where λ is the weight coefficient of the regularization term. The total loss calculation formula constrains the model training process, ensuring that the model retains its basic semantic representation.

9. A computing device, characterized in that, The computing device includes: a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the steps of the method as described in any one of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 5.

Images