Model fine-tuning training method, system, device and medium based on data sorting

Abstract

The application discloses a large model fine-tuning training method, system, device and medium based on data sorting, which comprises the following steps: selecting multiple preheating training data from fine-tuning training data, preheating a target large model through the preheating training data to obtain a preheating large model; determining an input question vector and a quality score of the fine-tuning training data according to the preheating large model; dividing the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then performing vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorting data and complex sorting data; and performing multi-stage fine-tuning training on the preheating large model through the simple sorting data and the complex sorting data in sequence to obtain a target fine-tuning large model. The application improves the efficiency and model performance of large model fine-tuning training and can be applied to the field of artificial intelligence technology.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device, and medium for fine-tuning training of large models based on data sorting. Background Technology

[0002] In recent years, with the development of technology, large models have become a hot topic in the field of artificial intelligence. Large models achieve better learning and generalization capabilities by increasing the model's size and complexity; however, this also brings new challenges to computing resources, energy consumption, and algorithm optimization. The fine-tuning stage of large models typically uses less data than the pre-training stage to adapt the model to specific downstream tasks. How to learn information from the fine-tuning data is a key step in improving model performance.

[0003] Currently, fine-tuning data for large models faces challenges such as complex data domains, difficulties in data classification, the potential for related data to positively influence each other, and the possibility of irrelevant data having a negative impact. While the amount of data is relatively small compared to pre-training data, it is still typically in the hundreds of thousands or millions, making it difficult to distinguish between good and bad data. Manual cleaning is too costly, and there are no good screening methods for some cleaning processes. Because it is difficult to differentiate between easy and difficult data, training both types of data together may result in repetitive learning on easy data and failure to learn from difficult data, impacting the efficiency and performance of large model fine-tuning training.

[0004] Terminology Explanation:

[0005] Fine-tuning is an optimization technique that aims to improve the performance of a pre-trained large model on a specific task by using a small amount of sample data from the target domain. The goal of fine-tuning is to adapt the large model to a specific task and data distribution, thereby improving its performance.

[0006] Large-scale language models (LLMs) have demonstrated remarkable in-context learning (ICL) capabilities. However, a significant gap exists between the pre-training objectives of the language model and its downstream ICL usage, impacting ICL performance. Some works have inserted a module between language model pre-training and downstream ICL inference to bridge this gap, a process known as warmup. Warmup involves targeted training on ICL data, updating or adding language model parameters to adapt the model to the downstream ICL task. Summary of the Invention

[0007] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0008] Therefore, one objective of this invention is to provide a large model fine-tuning training method based on data sorting, which improves the efficiency and performance of large model fine-tuning training.

[0009] Another objective of this invention is to provide a large model fine-tuning training system based on data sorting.

[0010] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include:

[0011] On one hand, embodiments of the present invention provide a method for fine-tuning training of a large model based on data ranking, comprising the following steps:

[0012] Multiple preheating training data are selected from the fine-tuning training data, and the target large model is preheated using the preheating training data to obtain the preheated large model;

[0013] The input question vector and quality score of the fine-tuning training data are determined based on the preheated large model.

[0014] The fine-tuning training data is divided into simple training data and complex training data based on the quality score. Then, vector clustering and sorting are performed on the simple training data and the complex training data respectively based on the input question vector to obtain simple sorted data and complex sorted data.

[0015] The preheating large model is then subjected to multi-stage fine-tuning training using the simple sorted data and the complex sorted data in sequence to obtain the target fine-tuned large model.

[0016] Furthermore, in one embodiment of the present invention, the step of determining the input question vector and quality score of the fine-tuning training data based on the preheated large model specifically includes:

[0017] The preheated large model is used to infer the fine-tuned training data to obtain the input question vector of each fine-tuned training data and to determine the model fitting difficulty of each fine-tuned training data.

[0018] Determine the answer length of each of the fine-tuned training data, and determine the quality score of each of the fine-tuned training data based on the model fitting difficulty and the answer length;

[0019] The quality score is positively correlated with the difficulty of model fitting, and negatively correlated with the length of the answer.

[0020] Furthermore, in one embodiment of the present invention, the step of dividing the fine-tuned training data according to the quality score to obtain simple training data and complex training data specifically includes:

[0021] The fine-tuned training data are sorted according to the quality score.

[0022] The sorted fine-tuned training data is divided according to the preset first percentile to obtain the simple training data and the complex training data.

[0023] The quality score of the simple training data is lower than the quality score of the complex training data.

[0024] Furthermore, in one embodiment of the present invention, the step of performing vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data specifically includes:

[0025] Based on the input question vector, the simple training data / complex training data are subjected to vector clustering to obtain multiple first cluster data groups and corresponding multiple first cluster centers;

[0026] The average quality score of each first cluster data group is determined based on the quality score, and the data group distance between each first cluster data group is determined based on the distance between each first cluster center.

[0027] The first cluster data group with the lowest average quality score is selected as the initial current cluster data group;

[0028] The first unsorted cluster data group with the smallest distance from the current cluster data group is selected as the new current cluster data group to obtain the simple sorted data / the complex sorted data.

[0029] Furthermore, in one embodiment of the present invention, the step of sequentially performing multi-stage fine-tuning training on the preheating large model using the simple sorting data and the complex sorting data to obtain the target fine-tuned large model specifically includes:

[0030] The simple sorted data is used as the initial current training data, and the preheated large model is used as the initial current fine-tuned large model.

[0031] The current fine-tuned large model is fine-tuned using the current training data to obtain the updated current fine-tuned large model, and the loss value for each current training data is determined.

[0032] Based on the loss values, select multiple low-loss data points and multiple high-loss data points from the current training data, and perform data cleaning on the high-loss data points;

[0033] The updated current training data is generated based on the complex sorted data, the low-loss data, and the high-loss data after data cleaning. The current training data is then used to fine-tune the current large-scale model until the fine-tuning training reaches a preset number of times, thus obtaining the target large-scale model.

[0034] Furthermore, in one embodiment of the present invention, the step of selecting multiple low-loss data points and multiple high-loss data points from the current training data based on the loss value specifically includes:

[0035] The current training data are reordered from smallest to largest based on the loss values;

[0036] The low-loss data is obtained by truncating the reordered current training data downwards based on the preset second percentile.

[0037] Multiple second cluster data groups are determined based on the current training data, and the current training data in each second cluster data group are reordered in ascending order of the loss value;

[0038] The high-loss data is obtained by truncating the second clustered data group after reordering it according to the preset third percentile.

[0039] Furthermore, in one embodiment of the present invention, the data cleaning of the high-loss data specifically includes:

[0040] The high-loss data is classified to obtain incorrect question-answer pairs and correct question-answer pairs;

[0041] Correct the questions or answers in the aforementioned incorrect question-and-answer pairs;

[0042] The correct question-and-answer pair is reconstructed, and a thought process is added.

[0043] On the other hand, embodiments of the present invention provide a large model fine-tuning training system based on data ranking, comprising:

[0044] The large model warm-up module is used to select multiple warm-up training data from the fine-tuning training data, and use the warm-up training data to warm up the target large model to obtain the warmed large model;

[0045] The large model inference module is used to determine the input question vector and quality score of the fine-tuned training data based on the preheated large model.

[0046] The data partitioning and sorting module is used to partition the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then perform vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data.

[0047] The multi-stage fine-tuning training module is used to perform multi-stage fine-tuning training on the preheated large model sequentially using the simple sorted data and the complex sorted data to obtain the target fine-tuned large model.

[0048] On the other hand, embodiments of the present invention provide an electronic device, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory, wherein when the program is executed by the processor, it implements the large model fine-tuning training method based on data sorting as described above.

[0049] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the large model fine-tuning training method based on data sorting as described above.

[0050] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0051] In this embodiment of the invention, multiple preheating training data are selected from the fine-tuning training data. The target large model is preheated using the preheating training data to obtain a preheated large model. The input question vector and quality score of the fine-tuning training data are determined based on the preheated large model. The fine-tuning training data is divided into simple training data and complex training data based on the quality score. Then, vector clustering and sorting are performed on the simple training data and complex training data respectively based on the input question vector to obtain simple sorted data and complex sorted data. The preheated large model is then fine-tuned in multiple stages using the simple sorted data and complex sorted data to obtain the target fine-tuned large model. In this embodiment of the invention, the input question vector and quality score of the fine-tuning training data are determined based on the preheating large model. The training data is divided into simple and complex training data according to the quality score. Vector clustering and sorting are performed on the simple and complex training data according to the input question vector. Thus, the preheating large model can be fine-tuned in multiple stages using the obtained simple and complex sorted data. This not only allows for staged training of the large model based on the learning difficulty of the training data, but also considers the data correlation within the simple and complex training data, enabling the large model to converge faster and improving the efficiency and performance of the large model fine-tuning training. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating the steps of a large model fine-tuning training method based on data sorting provided in an embodiment of the present invention;

[0054] Figure 2 A flowchart of step S102 provided in an embodiment of the present invention;

[0055] Figure 3 A flowchart of step S103 provided in an embodiment of the present invention;

[0056] Figure 4 Another flowchart of step S103 provided in an embodiment of the present invention;

[0057] Figure 5 A flowchart of step S104 provided in an embodiment of the present invention;

[0058] Figure 6 A flowchart of step S1043 provided in an embodiment of the present invention;

[0059] Figure 7 Another flowchart of step S1043 provided in the embodiment of the present invention

[0060] Figure 8 This is a schematic diagram of the structure of a large model fine-tuning training system based on data sorting provided in an embodiment of the present invention;

[0061] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention;

[0062] Figure 10 This is a schematic diagram of the structure of the storage medium provided in an embodiment of the present invention. Detailed Implementation

[0063] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are only set for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0064] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] The large model fine-tuning training method based on data ranking provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, set-top box, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the large model fine-tuning training method based on data ranking, etc., but is not limited to the above forms.

[0066] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0067] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0068] Current large-scale model learning methods include course-based training in the pre-training domain, which trains data by subject. However, this approach is lacking in the fine-tuning training domain. Two-stage learning only divides general data and downstream task data, without considering data difficulty for distributed learning, and it doesn't utilize the loss information obtained in the first stage to optimize the data, nor does it analyze the problems of data with poor model performance. Existing technologies suffer from the following problems:

[0069] 1) The fine-tuning data is diverse, and unrelated data may conflict with each other, resulting in large fluctuations in loss and slow convergence;

[0070] 2) Data quality assessment is difficult, manual full-scale annotation is costly, some annotations lack directionality, and difficult data lacks focused learning.

[0071] 3) The problem of forgetting in two-stage fine-tuning of learning.

[0072] Therefore, a method is needed to fine-tune and train large models that can sort data according to their relevance and then learn sequentially, learn data of varying difficulty in sequence, focus on strengthening learning on difficult data, and optimize learning after discovering erroneous data.

[0073] This invention employs sequential learning based on relevance ranking, with easy and difficult data learned sequentially. Difficult data is optimized using a thought chain approach, and erroneous data can be optimized manually or through a model. Furthermore, the second-order optimization process can be extended to multiple stages because it includes some data from the previous stage and challenging data, thus better avoiding the problem of information forgetting.

[0074] The embodiments of the present invention solve the problems of large loss fluctuations, slow convergence, high manual labeling costs, lack of training order for difficult and easy data due to the variety of fine-tuning data, and serious forgetting in the second-stage training when divided into two-stage learning. By adding a small amount of specific data in the first stage, the memory ability of the model is enhanced.

[0075] like Figure 1 The diagram shows a flowchart of a large model fine-tuning training method based on data sorting provided in an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a method for fine-tuning training a large model based on data ranking, specifically including the following steps:

[0076] S101. Select multiple preheating training data from the fine-tuning training data, and preheat the target large model using the preheating training data to obtain the preheated large model.

[0077] Specifically, the full-scale fine-tuning training data is D. train In the initial training phase, the system warms up the model using x% (e.g., 1%) of fine-tuned training data as warm-up training data D. warmup D warmup ∈Dtrain The model is trained for X rounds (usually 1 to 4 times) to allow it to learn from the current overall data, facilitating scoring of fine-tuned training data. Suitable target models include llama2, telechart-12b, and chatglm.

[0078] S102. Determine the input problem vector and quality score of the fine-tuning training data based on the preheated large model.

[0079] like Figure 2 The diagram shown is a flowchart of step S102 provided in an embodiment of the present invention. (Refer to...) Figure 2 As an optional implementation, the input question vector and quality score of the fine-tuning training data are determined based on the preheated large model, specifically including:

[0080] S1021. By preheating the large model, inference is performed on the fine-tuned training data to obtain the input question vector of each fine-tuned training data, and the model fitting difficulty of each fine-tuned training data is determined.

[0081] S1022. Determine the answer length of each fine-tuning training data, and determine the quality score of each fine-tuning training data based on the model fitting difficulty and the answer length.

[0082] Among them, the quality score is positively correlated with the difficulty of model fitting, and negatively correlated with the length of the answer.

[0083] Specifically, the large model warmed up after warmup is used for D. train The data is used for inference, and a quality score is calculated for each data point. This score takes into account the difficulty of model fitting for the query and answer, and reduces the impact of length (when the difficulty is the same, the longer the answer, the more difficult it is to fit, and the higher the loss will be).

[0084] In some optional embodiments, the quality score is calculated using the following formula:

[0085]

[0086] Among them, ppl query&answer The length indicates the difficulty of model fitting when fine-tuning the training data. answer This indicates the length of the answer to fine-tuning the training data.

[0087] Additionally, warming up the large model after using warmup affects D. train When performing reasoning on the data, the input question vector (i.e., the embedding vector, which can be the output of a certain layer of the neural network, such as the vector of the last layer of the last word in the question and answer) is obtained.

[0088] S103. Divide the fine-tuning training data according to the quality score to obtain simple training data and complex training data. Then, perform vector clustering and sorting on the simple training data and complex training data according to the input question vector to obtain simple sorted data and complex sorted data.

[0089] like Figure 3 The diagram shown is a flowchart of step S103 provided in an embodiment of the present invention. (Refer to...) Figure 3 As an optional implementation, the fine-tuning training data is further divided according to the quality score to obtain simple training data and complex training data, specifically including:

[0090] S1031. Sort the fine-tuning training data according to the quality score;

[0091] S1032. Divide the sorted fine-tuning training data according to the preset first percentile to obtain simple training data and complex training data.

[0092] The quality score of simple training data is lower than that of complex training data.

[0093] Specifically, the preset first percentile can be set manually, for example, the 60th percentile. Then, the fine-tuning training data is sorted from smallest to largest according to the quality score. The top 60% and bottom 40% of the fine-tuning training data are selected respectively, thus dividing the fine-tuning training data into the 60% with the lowest score, denoted as D. train_easy And the 40% higher score is represented as D. train_hard The purpose of dividing the data into two parts (or multiple parts) of varying difficulty in this embodiment of the invention is to facilitate subsequent step-by-step learning, enabling learning from easy to difficult.

[0094] like Figure 4 The diagram shown is another flowchart of step S103 provided in an embodiment of the present invention. (Refer to...) Figure 4 As an optional implementation, simple training data and complex training data are further subjected to vector clustering and sorting based on the input question vector to obtain simple sorted data and complex sorted data, specifically including:

[0095] S1033. Perform vector clustering on simple / complex training data based on the input question vector to obtain multiple first cluster data groups and corresponding multiple first cluster centers;

[0096] S1034. Determine the average quality score of each first cluster data group based on the quality score, and determine the data group distance between each first cluster data group based on the distance between each first cluster center.

[0097] S1035. Select the first cluster data group with the lowest average quality score as the initial current cluster data group;

[0098] S1036. Select the first unsorted cluster data group with the smallest distance from the current cluster data group as the new current cluster data group to obtain simple sorted data / complex sorted data.

[0099] Specifically, the k-means clustering algorithm is used to cluster the data D based on the input question vector obtained in the preceding steps. train_easy D train_hard Perform vector clustering separately, and remove duplicates from data that are very close in distance.

[0100] Calculate the distance between different cluster centroids and the mean score of different cluster data groups:

[0101]

[0102] Where avg_score represents the average quality score, and score i This represents the quality score of the i-th training data (simple or complex) in a clustered data set.

[0103] Starting with the cluster center with the lowest mean score, the cluster data group corresponding to the shortest distance center point is selected for sorting in each iteration to obtain the data learning order. This embodiment of the invention groups highly correlated data together, enhancing the data effect, and the learning of correlated data is smoother, making the loss less prone to large fluctuations and facilitating rapid convergence. For example, with four center points A, B, C, and D, the cluster data group corresponding to A has the lowest average quality score. The distance between A and B is 0.1, A and C is 0.2, A and D is 0.3, B and C is 0.4, B and D is 0.6, and C and D is 0.25. Therefore, the cluster data group corresponding to A is first selected as the initial current cluster data group, and the closest to it is B. This process continues, selecting the cluster data groups corresponding to B, C, and D in sequence as the new current cluster data groups, ultimately resulting in a learning order of A, B, C, D.

[0104] It should be noted that the above process can be used to sort simple and complex training data respectively, resulting in simple sorted data rank(D). train_easy ) and complex sorted data rank(D train_hard ).

[0105] S104. The preheated large model is then fine-tuned in multiple stages using simple sorted data and complex sorted data to obtain the target fine-tuned large model.

[0106] like Figure 5 The diagram shown is a flowchart of step S104 provided in an embodiment of the present invention. (Refer to...) Figure 5 As an optional implementation, the preheating large model is further fine-tuned through multiple stages using simple and complex sorted data to obtain the target fine-tuned large model, which specifically includes:

[0107] S1041. Use the simple sorted data as the initial current training data, and use the preheated large model as the initial current fine-tuning large model;

[0108] S1042. Fine-tune the current fine-tuned large model using the current training data to obtain the updated current fine-tuned large model, and determine the loss value for each current training data.

[0109] S1043. Select multiple low-loss data points and multiple high-loss data points from the current training data based on the loss values, and perform data cleaning on the high-loss data points.

[0110] S1044. Generate updated current training data based on complex sorted data, low loss value data, and high loss value data after data cleaning, and return to fine-tune the current fine-tuned large model using the current training data until the fine-tuning training reaches the preset number of times to obtain the target fine-tuned large model.

[0111] like Figure 6 The diagram shown is a flowchart of step S1043 provided in an embodiment of the present invention. (Refer to...) Figure 6 As an optional implementation, multiple low-loss data points and multiple high-loss data points are selected from the current training data based on the loss values, specifically including:

[0112] S10431. Reorder the current training data according to the loss value from smallest to largest;

[0113] S10432. Based on the preset second percentile, the current training data after reordering is truncated downwards to obtain low-loss data.

[0114] S10433. Determine multiple second cluster data groups based on the current training data, and reorder the current training data in each second cluster data group according to the loss value from smallest to largest.

[0115] S10434. Based on the preset third percentile, the second cluster data group after reordering is truncated upwards to obtain high loss value data.

[0116] Specifically, using the rank(D) after clustering and sorting train_easyFine-tuning training is performed, and the loss of each data point in the last round of fine-tuning is recorded. y% (e.g., 1%) of the training data with the lowest loss in the first stage are randomly selected as low-loss data D. random Take the top z percent (e.g., 5%) of training data with the largest loss in each cluster as the high-loss data D. loss_high , and D loss_high The answers are processed (either manually or through a model).

[0117] This invention provides reinforcement learning for difficult-to-learn data and directionally seeks out and corrects erroneous data. Because the loss difference between different classes of data can be large, this invention selects the training data with the largest loss value for each cluster for data cleaning and adds it to the fine-tuning training in subsequent stages.

[0118] The updated current training data is rank(D) train_hard )+D random +D loss_high In this embodiment of the invention, based on the model's learning of simple data in the first stage, complex data is learned in the second stage based on the updated current training data. In order to prevent forgetting of the first stage, a small amount of random knowledge is retained to reduce the forgetting problem. The difficult data in the first stage is reinforced, and the erroneous data in the first stage is corrected.

[0119] In the second stage of the present invention, fine-tuning can be repeated multiple times as needed. Each time, low-loss data and high-loss data are selected based on the loss value of the previous training round. After processing, the data is reassembled into the current training data for a new round of training. After reaching the preset number of training times, the target fine-tuned large model can be obtained.

[0120] like Figure 7 The diagram shown is another flowchart of step S1043 provided in an embodiment of the present invention. (Refer to...) Figure 7 As an optional implementation method, data cleaning is performed on the high-loss data, which specifically includes:

[0121] S10435. Classify the high-loss data to obtain incorrect question-answer pairs and correct question-answer pairs;

[0122] S10436. Correct the questions or answers for incorrect question-and-answer pairs;

[0123] S10437. Reconstruct the answers to correct questions and answers and add a thought process.

[0124] Specifically, for erroneous data, corrections are made (in cases where high loss is caused by poor data quality); for correct data, but with high loss, it indicates that the model is learning with difficulty, so the answer is reconstructed, and more thought processes are added (either manually or with the assistance of a large model) to break down and analyze the answer, so as to facilitate further learning by the model.

[0125] The method flow and implementation process of the embodiments of the present invention have been described above. It can be understood that the embodiments of the present invention determine the input question vector and quality score of the fine-tuning training data based on the preheated large model, divide the training data into simple and complex training data based on the quality score, and perform vector clustering and sorting on the simple and complex training data respectively based on the input question vector. Thus, the preheated large model can be fine-tuned in multiple stages using the obtained simple and complex sorted data. This approach not only allows for staged training of the large model based on the learning difficulty of the training data, but also considers the data correlation within the simple and complex training data, enabling the large model to converge faster and improving the efficiency and performance of the large model fine-tuning training.

[0126] Compared with the prior art, the embodiments of the present invention also have the following advantages:

[0127] 1) This invention innovatively proposes a large model fine-tuning training method based on data sorting and mental chain reinforcement. This technology clusters and divides the data after the model is scored, so that related data learns sequentially and difficult data learns in stages, thereby reducing the fluctuation of loss value and accelerating convergence.

[0128] 2) This invention presents a unique and simple method for scoring data difficulty. It combines the difficulty of questions and answers with a reduction in the impact of length bias.

[0129] 3) Combining the loss values ​​of clustering and the first-stage model, select the data that needs to be focused on. For erroneous data, correct it (for cases where the loss is high due to poor data quality); for correct data (for cases where the loss is high due to data difficulty), reconstruct the answer and add more thought processes (which can be done manually or with the assistance of a large model) to facilitate further learning by the model; save manpower for manual annotation and build data in a targeted manner.

[0130] 4) The model fine-tuning is divided into two stages (or multiple stages, because it is scalable). Based on the learning of simple data in the first stage, the model learns complex data. To prevent forgetting of the first stage, a small amount of random knowledge from the first stage is retained to reduce the forgetting problem. The difficult data in the first stage is relearned, and a thought chain step is added to help the model learn better. The erroneous data with high loss values ​​in the first stage are corrected.

[0131] like Figure 8 The diagram shown is a structural schematic of a large model fine-tuning training system based on data sorting provided in an embodiment of the present invention. (Refer to...) Figure 8 This invention provides a large model fine-tuning training system based on data ranking, comprising:

[0132] The large model warm-up module is used to select multiple warm-up training data from the fine-tuning training data, and warm up the target large model with the warm-up training data to obtain the warmed large model;

[0133] The large model inference module is used to determine the input question vector and quality score of the fine-tuning training data based on the preheated large model.

[0134] The data partitioning and sorting module is used to partition the fine-tuning training data according to the quality score to obtain simple training data and complex training data. Then, based on the input question vector, the simple training data and complex training data are clustered and sorted respectively to obtain simple sorted data and complex sorted data.

[0135] The multi-stage fine-tuning training module is used to perform multi-stage fine-tuning training on the preheated large model through simple sorted data and complex sorted data in sequence to obtain the target fine-tuned large model.

[0136] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0137] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned large model fine-tuning training method based on data sorting. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0138] like Figure 9 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 9 This invention provides an electronic device, comprising:

[0139] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0140] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the large model fine-tuning training method based on data sorting according to the embodiments of this invention.

[0141] The input / output interface 903 is used to implement information input and output;

[0142] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0143] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0144] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0145] like Figure 10 The diagram shown is a structural schematic of the storage medium provided in an embodiment of the present invention. (Refer to...) Figure 10 The present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 1001, which can be executed by one or more processors to implement the above-described large model fine-tuning training method based on data sorting.

[0146] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0147] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0148] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0149] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0152] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or, if necessary, processing in other suitable ways, and then stored in computer memory.

[0153] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0154] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0155] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0156] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for fine-tuning training a large model based on data ranking, characterized in that, Includes the following steps: Multiple preheating training data are selected from the fine-tuning training data, and the target large model is preheated using the preheating training data to obtain the preheated large model; The input question vector and quality score of the fine-tuning training data are determined based on the preheated large model. The fine-tuning training data is divided into simple training data and complex training data based on the quality score. Then, vector clustering and sorting are performed on the simple training data and the complex training data respectively based on the input question vector to obtain simple sorted data and complex sorted data. The preheating large model is then subjected to multi-stage fine-tuning training using the simple sorted data and the complex sorted data in sequence to obtain the target fine-tuned large model. The step of determining the input question vector and quality score of the fine-tuning training data based on the preheated large model specifically includes: The preheated large model is used to infer the fine-tuned training data to obtain the input question vector of each fine-tuned training data and to determine the model fitting difficulty of each fine-tuned training data. Determine the answer length of each of the fine-tuned training data, and determine the quality score of each of the fine-tuned training data based on the model fitting difficulty and the answer length; Wherein, the quality score is positively correlated with the model fitting difficulty, the quality score is negatively correlated with the answer length, the target fine-tuning large model is used to input question text and output answer text, the input question vector is the embedding vector of the question text in the fine-tuning training data, and the answer length is the length of the answer text in the fine-tuning training data.

2. The method for fine-tuning training a large model based on data sorting according to claim 1, characterized in that, The step of dividing the fine-tuned training data according to the quality score to obtain simple training data and complex training data specifically includes: The fine-tuned training data are sorted according to the quality score. The sorted fine-tuned training data is divided according to the preset first percentile to obtain the simple training data and the complex training data. The quality score of the simple training data is lower than the quality score of the complex training data.

3. The method for fine-tuning training a large model based on data sorting according to claim 1, characterized in that, The step of performing vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data specifically includes: Based on the input question vector, the simple training data / complex training data are subjected to vector clustering to obtain multiple first cluster data groups and corresponding multiple first cluster centers; The average quality score of each first cluster data group is determined based on the quality score, and the data group distance between each first cluster data group is determined based on the distance between each first cluster center. The first cluster data group with the lowest average quality score is selected as the initial current cluster data group; The first unsorted cluster data group with the smallest distance from the current cluster data group is selected as the new current cluster data group to obtain the simple sorted data / the complex sorted data.

4. A method for fine-tuning training a large model based on data sorting according to any one of claims 1 to 3, characterized in that, The process of sequentially performing multi-stage fine-tuning training on the preheated large model using the simple sorted data and the complex sorted data to obtain the target fine-tuned large model specifically includes: The simple sorted data is used as the initial current training data, and the preheated large model is used as the initial current fine-tuned large model. The current fine-tuned large model is fine-tuned using the current training data to obtain the updated current fine-tuned large model, and the loss value for each current training data is determined. Based on the loss values, select multiple low-loss data points and multiple high-loss data points from the current training data, and perform data cleaning on the high-loss data points; The updated current training data is generated based on the complex sorted data, the low-loss data, and the high-loss data after data cleaning. The current training data is then used to fine-tune the current large-scale model until the fine-tuning training reaches a preset number of times, thus obtaining the target large-scale model.

5. The large model fine-tuning training method based on data sorting according to claim 4, characterized in that, The step of selecting multiple low-loss data points and multiple high-loss data points from the current training data based on the loss value specifically includes: The current training data are reordered from smallest to largest based on the loss values; The low-loss data is obtained by truncating the reordered current training data downwards based on the preset second percentile. Multiple second cluster data groups are determined based on the current training data, and the current training data in each second cluster data group are reordered in ascending order of the loss value; The high-loss data is obtained by truncating the second clustered data group after reordering it according to the preset third percentile.

6. The method for fine-tuning training a large model based on data sorting according to claim 4, characterized in that, The data cleaning process for the high-loss data specifically includes: The high-loss data is classified to obtain incorrect question-answer pairs and correct question-answer pairs; Correct the questions or answers in the aforementioned incorrect question-and-answer pairs; The correct question-and-answer pair is reconstructed, and a thought process is added.

7. A large model fine-tuning training system based on data sorting, characterized in that, include: The large model warm-up module is used to select multiple warm-up training data from the fine-tuning training data, and use the warm-up training data to warm up the target large model to obtain the warmed large model; The large model inference module is used to determine the input question vector and quality score of the fine-tuned training data based on the preheated large model. The data partitioning and sorting module is used to partition the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then perform vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data. A multi-stage fine-tuning training module is used to perform multi-stage fine-tuning training on the preheated large model sequentially using the simple sorted data and the complex sorted data to obtain the target fine-tuned large model. The step of determining the input question vector and quality score of the fine-tuning training data based on the preheated large model specifically includes: The preheated large model is used to infer the fine-tuned training data to obtain the input question vector of each fine-tuned training data and to determine the model fitting difficulty of each fine-tuned training data. Determine the answer length of each of the fine-tuned training data, and determine the quality score of each of the fine-tuned training data based on the model fitting difficulty and the answer length; Wherein, the quality score is positively correlated with the model fitting difficulty, the quality score is negatively correlated with the answer length, the target fine-tuning large model is used to input question text and output answer text, the input question vector is the embedding vector of the question text in the fine-tuning training data, and the answer length is the length of the answer text in the fine-tuning training data.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the large model fine-tuning training method based on data sorting as described in any one of claims 1 to 6.

9. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the large model fine-tuning training method based on data sorting as described in any one of claims 1 to 6.

Images