Large model fine-tuning training method, system and equipment based on data sorting and medium
By preheating training data and quality score division, the multi-stage clustering and sorting training method is adopted to solve the problem of data complexity and low efficiency in fine-tuning training of large models, and improve the model performance and training efficiency.
Patent Information
- Application Number
- CN202411155152.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-21
AI Technical Summary
In the fine-tuning training of large models, there are problems such as complex data fields, difficulty in data classification, possible positive effects of related data, and possible negative effects of irrelevant data, resulting in reduced model performance and low training efficiency.
Preheat the big model by preheating the training data, determine the input problem vector and quality score of the fine-tuning training data, divide the simple and complex training data, and perform multi-stage fine-tuning training based on vector clustering and sorting to optimize the data learning order and clean process.
The efficiency and performance of fine-tuning training of large models is improved, and through phased training and data correlation optimization, the model converges faster, reducing loss fluctuations and manual labeling costs.
Smart Images

Figure CN118965048A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model fine-tuning training method, system, device and medium based on data sorting. Background Art
[0002] In recent years, with the development of science and technology, large models have become a hot topic in the field of artificial intelligence. Large models achieve better learning and generalization capabilities by increasing the scale and complexity of the model. However, this also brings new challenges to computing resources, energy consumption and algorithm optimization. Among them, the fine-tuning stage of large models usually uses less data than the pre-training stage to adapt the model to specific downstream tasks. How to learn the information in the fine-tuning data is a key step to improve model performance.
[0003] At present, there are problems with large model fine-tuning data, such as complex data fields, difficult data classification, possible positive effects of related data on each other, and possible negative effects of irrelevant data. Although the data is relatively small compared to pre-training, the amount is generally in the hundreds of thousands or millions. It is difficult to distinguish between good and bad data, and the cost of direct manual cleaning is too high. There is no good screening method for some cleaning. Since it is difficult to distinguish between difficult and easy data, learning difficult and easy data together may cause repeated learning of simple data and failure to learn difficult data, which affects the efficiency of large model fine-tuning training and model performance.
[0004] Terminology explanation:
[0005] Fine-tuning is an optimization technique that aims to improve the performance of a pre-trained large model on a specific task by fine-tuning it using a small amount of sample data from the target domain. The purpose of fine-tuning is to adapt the large model to a specific task and data distribution, thereby improving the performance of the model.
[0006] Large model warmup: Currently, large-scale language models (LLMs) have demonstrated amazing in-context learning (ICL) capabilities. However, there is a significant gap between the language model pre-training objectives and downstream ICL usage, which affects the model's ICL performance. Therefore, some work inserts a module between the language model pre-training and downstream ICL inference, attempting to narrow the gap between the language model and ICL usage, which is called warmup. By performing targeted training on ICL data, updating or increasing language model parameters, the language model can be adapted to downstream ICL tasks. Summary of the invention
[0007] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.
[0008] To this end, an object of an embodiment of the present invention is to provide a large model fine-tuning training method based on data sorting, which improves the efficiency of large model fine-tuning training and model performance.
[0009] Another object of an embodiment of the present invention is to provide a large model fine-tuning training system based on data sorting.
[0010] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:
[0011] On the one hand, an embodiment of the present invention provides a large model fine-tuning training method based on data sorting, comprising the following steps:
[0012] Selecting a plurality of preheating training data from the fine-tuning training data, and preheating the target large model by using the preheating training data to obtain a preheating large model;
[0013] Determining an input question vector and a quality score of the fine-tuning training data according to the preheated large model;
[0014] Dividing the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then performing vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data;
[0015] The preheated large model is subjected to multi-stage fine-tuning training using the simple sorting data and the complex sorting data in turn to obtain a target fine-tuning large model.
[0016] Further, in one embodiment of the present invention, determining the input question vector and quality score of the fine-tuning training data according to the preheated large model specifically includes:
[0017] Inferring the fine-tuning training data through the preheated large model, obtaining the input problem vector of each of the fine-tuning training data, and determining the model fitting difficulty of each of the fine-tuning training data;
[0018] Determine the answer length of each of the fine-tuning training data, and determine the quality score of each of the fine-tuning training data according to the model fitting difficulty and the answer length;
[0019] The quality score is positively correlated with the difficulty of model fitting, and the quality score is negatively correlated with the length of the answer.
[0020] Further, in an embodiment of the present invention, the fine-tuning training data is divided according to the quality score to obtain simple training data and complex training data, which specifically includes:
[0021] sorting the fine-tuning training data according to the quality scores;
[0022] Dividing the sorted fine-tuning training data according to a preset first percentile to obtain the simple training data and the complex training data;
[0023] The quality score of the simple training data is lower than the quality score of the complex training data.
[0024] Further, in one embodiment of the present invention, the vector clustering and sorting of the simple training data and the complex training data are performed according to the input question vector to obtain simple sorted data and complex sorted data, which specifically includes:
[0025] Performing vector clustering on the simple training data / the complex training data according to the input question vector to obtain a plurality of first clustering data groups and a corresponding plurality of first clustering centers;
[0026] Determine an average quality score of each of the first cluster data groups according to the quality score, and determine a corresponding data group distance between each of the first cluster data groups according to the distance between each of the first cluster centers;
[0027] Selecting the first clustering data group with the lowest average quality score as the initial current clustering data group;
[0028] The unsorted first clustering data group with the smallest data group distance from the current clustering data group is selected in sequence as a new current clustering data group to obtain the simple sorting data / the complex sorting data.
[0029] Further, in one embodiment of the present invention, the preheating large model is subjected to multi-stage fine-tuning training by sequentially using the simple sorting data and the complex sorting data to obtain a target fine-tuning large model, which specifically includes:
[0030] Using the simple sorting data as initial current training data, and using the preheated large model as the initial current fine-tuning large model;
[0031] Fine-tune the current fine-tuned large model using the current training data to obtain an updated current fine-tuned large model, and determine the loss value of each current training data;
[0032] Selecting a plurality of low-loss value data and a plurality of high-loss value data from the current training data according to the loss value, and performing data cleaning on the high-loss value data;
[0033] Generate updated current training data based on the complex sorting data, the low loss value data and the high loss value data after data cleaning, and return to fine-tune the current fine-tuning large model through the current training data until the fine-tuning training reaches a preset number of times to obtain the target fine-tuning large model.
[0034] Further, in one embodiment of the present invention, the step of selecting a plurality of low loss value data and a plurality of high loss value data from the current training data according to the loss value specifically includes:
[0035] Reorder the current training data from small to large according to the loss value;
[0036] According to the preset second percentile, the reordered current training data is intercepted downward to obtain the low loss value data;
[0037] Determine a plurality of second clustering data groups according to the current training data, and reorder the current training data in each of the second clustering data groups from small to large according to the loss value;
[0038] The reordered second cluster data group is intercepted upward according to a preset third percentile to obtain the high loss value data.
[0039] Furthermore, in one embodiment of the present invention, the data cleaning of the high loss value data specifically includes:
[0040] Classifying the high loss value data to obtain incorrect question-answer pairs and correct question-answer pairs;
[0041] Correcting the question or answer of the incorrect question-answer pair;
[0042] The correct question and answer pairs are reconstructed and a thought chain process is added.
[0043] On the other hand, an embodiment of the present invention provides a large model fine-tuning training system based on data sorting, comprising:
[0044] A large model preheating module, used to select a plurality of preheating training data from the fine-tuning training data, and preheat the target large model through the preheating training data to obtain a preheated large model;
[0045] A large model reasoning module, used to determine an input question vector and a quality score of the fine-tuning training data according to the preheated large model;
[0046] A data partitioning and sorting module, used to partition the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then perform vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data;
[0047] The multi-stage fine-tuning training module is used to perform multi-stage fine-tuning training on the preheated large model through the simple sorting data and the complex sorting data in sequence to obtain a target fine-tuning large model.
[0048] On the other hand, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein the program, when executed by the processor, realizes the large model fine-tuning training method based on data sorting as described above.
[0049] On the other hand, an embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium for computer-readable storage, and which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the large model fine-tuning training method based on data sorting as described above.
[0050] The advantages and beneficial effects of the present invention will be partly given in the following description, partly become apparent from the following description, or be understood through the practice of the present invention:
[0051] The embodiment of the present invention selects multiple preheating training data from the fine-tuning training data, preheats the target large model through the preheating training data to obtain the preheating large model, determines the input problem vector and quality score of the fine-tuning training data according to the preheating large model, divides the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then performs vector clustering and sorting on the simple training data and the complex training data according to the input problem vector to obtain simple sorted data and complex sorted data, and performs multi-stage fine-tuning training on the preheating large model through the simple sorted data and the complex sorted data in turn to obtain the target fine-tuning large model. The embodiment of the present invention determines the input problem vector and quality score of the fine-tuning training data according to the preheated large model, divides the simple training data and the complex training data according to the quality score, and respectively performs vector clustering and sorting on the simple training data and the complex training data according to the input problem vector, so that the preheated large model can be subjected to multi-stage fine-tuning training through the obtained simple sorted data and complex sorted data. The large model can be trained in stages according to the learning difficulty of the training data, and the data correlation within the simple training data and the complex training data is considered, so that the large model can converge faster, thereby improving the efficiency of the large model fine-tuning training and the model performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0053] Figure 1 A flowchart of a method for fine-tuning a large model based on data sorting provided in an embodiment of the present invention;
[0054] Figure 2 A step flow chart of step S102 provided in an embodiment of the present invention;
[0055] Figure 3 A step flow chart of step S103 provided in an embodiment of the present invention;
[0056] Figure 4 Another step flow chart of step S103 provided in an embodiment of the present invention;
[0057] Figure 5 A step flow chart of step S104 provided in an embodiment of the present invention;
[0058] Figure 6 A step flow chart of step S1043 provided in an embodiment of the present invention;
[0059] Figure 7 Another step flow chart of step S1043 provided in an embodiment of the present invention
[0060] Figure 8 A schematic diagram of the structure of a large model fine-tuning training system based on data sorting provided in an embodiment of the present invention;
[0061] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention;
[0062] Fig.10 A schematic diagram of the structure of a storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0063] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limitations on the present application. It should be noted that, although the functional module division is performed in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order from the module division in the system schematic diagram or the flow chart. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0064] In the description of the present invention, the meaning of "a plurality" is two or more. If there is a description of the first or the second, it is only for the purpose of distinguishing the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this document are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.
[0065] The large model fine-tuning training method based on data sorting provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the large model fine-tuning training method based on data sorting, etc., but is not limited to the above forms.
[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0067] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0068] In the current large-scale model learning, there is course learning in the pre-training field, which divides the data into subjects for training, but it has not yet appeared in the field of fine-tuning training. The second-stage learning also only divides the general data and downstream task data, without considering the data difficulty for distribution learning, and does not use the loss information obtained in the first stage to optimize the data, nor does it analyze the problems of data with poor model learning effects. The existing technology has the following problems:
[0069] 1) There are many types of fine-tuning data, and irrelevant data may conflict with each other, resulting in large loss fluctuations and slow convergence;
[0070] 2) It is difficult to judge the data quality, the cost of manual full-data labeling is high, some labeling has no directionality, and difficult data lacks focus on learning;
[0071] 3) The forgetting problem of two-stage fine-tuning learning.
[0072] Therefore, a large model fine-tuning training method is needed that can sort data according to relevance and then learn them sequentially, learn difficult and easy data sequentially, focus on strengthening the learning of difficult data, and discover and optimize erroneous data before learning.
[0073] The embodiment of the present invention learns sequentially after sorting according to relevance, learns difficult and easy data sequentially, and uses thinking chain optimization to learn difficult data. Error data can be optimized manually or by model, and the second-order optimization process can be expanded to multiple stages. Because it includes part of the data and difficult data of the previous stage, it can better avoid the problem of information forgetting.
[0074] The embodiments of the present invention solve the problems of large loss fluctuations, slow convergence, high manual labeling costs, lack of training order for difficult and easy data, and serious forgetting in the two-stage training when the learning is divided into two stages. The memory capacity of the model is enhanced by adding a small amount of specific data from the previous stage.
[0075] like Figure 1 FIG. 1 is a flowchart of a method for fine-tuning a large model based on data sorting provided by an embodiment of the present invention, referring to FIG. Figure 1 The embodiment of the present invention provides a large model fine-tuning training method based on data sorting, which specifically includes the following steps:
[0076] S101. Select multiple preheating training data from the fine-tuning training data, and preheat the target large model through the preheating training data to obtain the preheating large model.
[0077] Specifically, the full amount of fine-tuning training data is D train In the initial stage of training, the system preheats the model and uses x% (e.g. 1%) of the fine-tuning training data as the preheating training data D warmup ,D warmup ∈Dtrain , perform X (usually 1 to 4) rounds of training, so that the model can learn the current overall data to a certain extent, so as to score the fine-tuning training data. The target large models that can be used include llama2, telechat-12b, chatglm, etc.
[0078] S102. Determine the input question vector and quality score of the fine-tuning training data based on the preheated large model.
[0079] like Figure 2 FIG. 1 is a flow chart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 2 As an optional implementation, the input question vector and quality score of the fine-tuning training data are determined based on the preheated large model, which specifically includes:
[0080] S1021, inferring the fine-tuning training data by preheating the large model, obtaining an input question vector of each fine-tuning training data, and determining the model fitting difficulty of each fine-tuning training data;
[0081] S1022, determining the answer length of each fine-tuning training data, and determining the quality score of each fine-tuning training data according to the model fitting difficulty and the answer length;
[0082] Among them, the quality score is positively correlated with the difficulty of model fitting, and the quality score is negatively correlated with the answer length.
[0083] Specifically, the warmed-up model is used to train The system infers the data and calculates the quality score of each data. It takes into account the difficulty of model fitting of query and answer and reduces the impact of length (when the difficulty is the same, the longer the answer, the harder it is to fit and the higher the loss).
[0084] In some optional embodiments, the quality score is calculated as follows:
[0085]
[0086] Among them, ppl query&answer Indicates the difficulty of fine-tuning the model fitting of the training data, length answer Represents the answer length of the fine-tuning training data.
[0087] In addition, the warm-up model has a significant effect on D train When the data is inferred, the input question vector (i.e., the embedding vector, which can be the output result of a certain layer of the neural network, such as the vector of the last layer of the last word of the question and answer) is obtained.
[0088] S103. Divide the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then perform vector clustering and sorting on the simple training data and the complex training data respectively according to the input question vector to obtain simple sorted data and complex sorted data.
[0089] like Figure 3 FIG. 1 is a flow chart of step S103 provided in an embodiment of the present invention, referring to FIG. Figure 3 As an optional implementation, the fine-tuning training data is further divided according to the quality score to obtain simple training data and complex training data, which specifically includes:
[0090] S1031. Sort the fine-tuning training data according to the quality scores;
[0091] S1032, dividing the sorted fine-tuning training data according to a preset first percentile to obtain simple training data and complex training data;
[0092] Among them, the quality score of simple training data is lower than that of complex training data.
[0093] Specifically, the preset first percentile can be set manually, for example, the 60% percentile, and then the fine-tuning training data is sorted from small to large according to the quality score, and the first 60% of the fine-tuning training data and the last 40% of the fine-tuning training data are selected respectively, so that the fine-tuning training data can be divided into 60% with low scores, which is expressed as D train_easy , and the 40% with the highest score, represented by D train_hard The purpose of dividing the data into two parts (or multiple parts) of difficult and easy in the embodiment of the present invention is to facilitate subsequent step-by-step learning and realize step-by-step learning from easy to difficult.
[0094] like Figure 4 FIG. 1 is another step flow chart of step S103 provided in an embodiment of the present invention, referring to FIG. Figure 4 As an optional implementation, vector clustering and sorting are performed on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data, which specifically includes:
[0095] S1033, performing vector clustering on the simple training data / complex training data according to the input question vector to obtain a plurality of first cluster data groups and a corresponding plurality of first cluster centers;
[0096] S1034, determining an average quality score of each first cluster data group according to the quality score, and determining a data group distance between corresponding first cluster data groups according to the distance between the first cluster centers;
[0097] S1035, selecting the first cluster data group with the lowest average quality score as the initial current cluster data group;
[0098] S1036. Select the unsorted first cluster data group with the smallest data group distance from the current cluster data group in turn as a new current cluster data group to obtain simple sorting data / complex sorting data.
[0099] Specifically, the k-means clustering algorithm is used to cluster the data D according to the input question vector obtained in the previous steps. train_easy , D train_hard Vector clustering is performed separately, and data with particularly close distances are deduplicated.
[0100] Count the distances between different cluster center points and the score means of different cluster data groups:
[0101]
[0102] Among them, avg_score represents the average quality score, score i Represents the quality score of the i-th training data (simple training data or complex training data) in the clustering data group.
[0103] Starting from the cluster center with the lowest score mean, each time the cluster data group corresponding to the center point with the shortest distance is selected for sorting to obtain the order of data learning. The embodiment of the present invention puts data with strong correlation together to increase the data effect, and the learning of related data is over-smoothed, and the loss is not prone to large fluctuations, which is conducive to rapid convergence. For example, there are four center points ABCD, and the average quality score of the cluster data group corresponding to A is the smallest. The distance between A and B is 0.1, the distance between A and C is 0.2, the distance between A and D is 0.3, the distance between B and C is 0.4, the distance between B and D is 0.6, and the distance between C and D is 0.25. Then, the cluster data group corresponding to A is first selected as the initial current cluster data group, and the closest to it is B. By analogy, the cluster data groups corresponding to B, C, and D are selected in turn as the new current cluster data group, and the final learning order is A, B, C, and D.
[0104] It should be noted that the simple training data and the complex training data can be sorted through the above process to obtain the simple sorting data rank (D train_easy ) and complex sorting data rank(D train_hard ).
[0105] S104, performing multi-stage fine-tuning training on the preheated large model using simple sorting data and complex sorting data in turn to obtain a target fine-tuning large model.
[0106] like Figure 5 FIG. 1 is a flowchart of step S104 provided in an embodiment of the present invention, referring to FIG. Figure 5 As an optional implementation, the preheated large model is subjected to multi-stage fine-tuning training using simple sorting data and complex sorting data in turn to obtain a target fine-tuning large model, which specifically includes:
[0107] S1041, using the simple sorting data as the initial current training data, and using the preheated large model as the initial current fine-tuning large model;
[0108] S1042, fine-tuning the current fine-tuning large model through the current training data to obtain an updated current fine-tuning large model, and determining the loss value of each current training data;
[0109] S1043, selecting a plurality of low-loss value data and a plurality of high-loss value data from the current training data according to the loss value, and performing data cleaning on the high-loss value data;
[0110] S1044. Generate updated current training data based on complex sorting data, low loss value data, and high loss value data after data cleaning, and return to fine-tune the current fine-tuning large model through the current training data until the fine-tuning training reaches a preset number of times to obtain the target fine-tuning large model.
[0111] like Figure 6 FIG. 1 is a flow chart of step S1043 provided in an embodiment of the present invention, referring to FIG. Figure 6 As an optional implementation, multiple low-loss value data and multiple high-loss value data are selected from the current training data according to the loss value, which specifically includes:
[0112] S10431. Reorder the current training data from small to large according to the loss value;
[0113] S10432, intercepting downward the reordered current training data according to a preset second percentile to obtain low loss value data;
[0114] S10433, determining a plurality of second clustering data groups according to the current training data, and reordering the current training data in each second clustering data group from small to large according to the loss value;
[0115] S10434. The reordered second cluster data group is intercepted upward according to a preset third percentile to obtain high loss value data.
[0116] Specifically, using the rank after clustering sorting (D train_easy), perform fine-tuning training, and record the loss of each data in the last round of fine-tuning; randomly select y% (e.g. 1%) of the training data from the top 95% with the smallest loss in the first stage as the low loss value data D random ; Take the topz% (such as 5%) of training data with the largest loss for each cluster as the high loss value data D loss_high , and D loss_high The answer is processed (manually or by model).
[0117] The embodiment of the present invention performs enhanced learning on difficult-to-learn data and searches for erroneous data in a targeted manner for correction. Because the loss difference between different types of data may be large, the embodiment of the present invention selects the training data with the largest loss value for each cluster for data cleaning and adds it to the fine-tuning training in the subsequent stage.
[0118] The updated current training data is rank(D train_hard )+D random +D loss_high . The embodiment of the present invention learns simple data in the first stage of the model, and then learns complex data based on the updated current training data in the second stage. In order to prevent forgetting of the first stage, a small amount of random knowledge is retained to reduce the forgetting problem, and the difficult data in the first stage is strengthened and the erroneous data in the first stage is corrected.
[0119] The fine-tuning of the second stage in the embodiment of the present invention can be repeated multiple times as needed. Each time, low loss value data and high loss value data are selected according to the loss value of the previous round of training. After processing, the current training data is recomposed to perform a new round of training. After reaching the preset number of training times, the target fine-tuned large model can be obtained.
[0120] like Figure 7 FIG. 1 is another step flow chart of step S1043 provided in an embodiment of the present invention, referring to FIG. Figure 7 As an optional implementation, data cleaning is performed on the high loss value data, which specifically includes:
[0121] S10435. Classify the high loss value data to obtain incorrect question-answer pairs and correct question-answer pairs;
[0122] S10436. Correct the question or answer for incorrect question and answer pairs;
[0123] S10437. Reconstruct the answers to correct question and answer pairs and add a chain of thought processes.
[0124] Specifically, for erroneous data, corrections are made (due to high loss caused by poor data quality); for correct data but high loss, it means that the model is difficult to learn, so the answer is reconstructed and more thinking chain processes are added (manually or with the assistance of large models) to disassemble and analyze the answer to facilitate further learning of the model.
[0125] The above is an explanation of the method flow and implementation process of the embodiment of the present invention. It can be understood that the embodiment of the present invention determines the input question vector and quality score of the fine-tuning training data according to the preheated large model, divides the simple training data and the complex training data according to the quality score, and performs vector clustering and sorting on the simple training data and the complex training data according to the input question vector, so that the preheated large model can be fine-tuned in multiple stages through the obtained simple sorted data and complex sorted data, which can not only train the large model in stages according to the learning difficulty of the training data, but also consider the data correlation within the simple training data and the complex training data, so that the large model can converge faster, and the efficiency of the large model fine-tuning training and the model performance are improved.
[0126] Compared with the prior art, the embodiments of the present invention also have the following advantages:
[0127] 1) The present invention innovatively proposes a large model fine-tuning training method based on data sorting and thinking chain reinforcement. This technology clusters and divides the data after the model is scored, so that the relevant data can be learned sequentially and the difficult and easy data can be learned in stages, thereby reducing the fluctuation of the loss value and accelerating the convergence.
[0128] 2) This invention has created a simple data difficulty scoring method It combines the difficulty of questions and answers and reduces the length bias effect.
[0129] 3) Combine the clustering and the loss value of the first-stage model to select the data that need to be focused on. Correct the wrong data (due to high loss caused by poor data quality); reconstruct the answer to the correct data (due to high loss caused by high data difficulty), add more thinking chain processes (can be done manually or with the assistance of large models) to facilitate further learning of the model; save manpower for manual labeling and construct data in a targeted manner.
[0130] 4) Divide the model fine-tuning into two stages (or multiple stages, because it is scalable). Based on the simple data learned in the first stage of the model, the complex data is learned; in order to prevent forgetting of the first stage, a small amount of random knowledge in the first stage is retained to reduce the forgetting problem; the difficult data in the first stage is re-learned, and the thinking chain steps are added to help the model learn better; the erroneous data with high loss values in the first stage are corrected.
[0131] like Figure 8 FIG. 1 is a schematic diagram of a large model fine-tuning training system based on data sorting provided by an embodiment of the present invention, referring to FIG. Figure 8 , an embodiment of the present invention provides a large model fine-tuning training system based on data sorting, comprising:
[0132] A large model preheating module is used to select a plurality of preheating training data from the fine-tuning training data, and preheat the target large model through the preheating training data to obtain a preheated large model;
[0133] The large model inference module is used to determine the input question vector and quality score of the fine-tuning training data based on the pre-warmed large model;
[0134] The data partitioning and sorting module is used to partition the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then perform vector clustering and sorting on the simple training data and the complex training data respectively according to the input question vector to obtain simple sorted data and complex sorted data;
[0135] The multi-stage fine-tuning training module is used to perform multi-stage fine-tuning training on the preheated large model through simple sorting data and complex sorting data in sequence to obtain the target fine-tuning large model.
[0136] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0137] The embodiment of the present invention also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned large model fine-tuning training method based on data sorting. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0138] like Fig. 9 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention, referring to FIG. Fig. 9 , an embodiment of the present invention provides an electronic device, including:
[0139] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;
[0140] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the large model fine-tuning training method based on data sorting in the embodiment of the present invention;
[0141] Input / output interface 903, used to implement information input and output;
[0142] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0143] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0144] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0145] like Fig.10 FIG. 1 is a schematic diagram of a storage medium according to an embodiment of the present invention. Fig.10 An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs 1001, and the one or more programs 1001 can be executed by one or more processors to implement the above-mentioned large model fine-tuning training method based on data sorting.
[0146] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0147] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.
[0148] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0149] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified to the contrary, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0150] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0151] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0152] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the above-mentioned program is printed, since the above-mentioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in other suitable ways as necessary, and then stored in a computer memory.
[0153] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0154] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0155] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0156] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A large model fine-tuning training method based on data sorting, characterized in that: The following steps are involved: Selecting a plurality of preheating training data from the fine-tuning training data, and preheating the target large model by using the preheating training data to obtain a preheating large model; Determining an input question vector and a quality score of the fine-tuning training data according to the preheated large model; Dividing the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then performing vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data; The preheated large model is subjected to multi-stage fine-tuning training using the simple sorting data and the complex sorting data in turn to obtain a target fine-tuning large model.
2. According to the method of fine-tuning a large model based on data sorting according to claim 1, it is characterized in that: The step of determining the input question vector and quality score of the fine-tuning training data according to the preheated large model specifically includes: Inferring the fine-tuning training data through the preheated large model, obtaining the input problem vector of each of the fine-tuning training data, and determining the model fitting difficulty of each of the fine-tuning training data; Determine the answer length of each of the fine-tuning training data, and determine the quality score of each of the fine-tuning training data according to the model fitting difficulty and the answer length; The quality score is positively correlated with the difficulty of model fitting, and the quality score is negatively correlated with the length of the answer.
3. According to the method of fine-tuning a large model based on data sorting according to claim 1, it is characterized in that: The fine-tuning training data is divided according to the quality score to obtain simple training data and complex training data, which specifically includes: sorting the fine-tuning training data according to the quality scores; Dividing the sorted fine-tuning training data according to a preset first percentile to obtain the simple training data and the complex training data; The quality score of the simple training data is lower than the quality score of the complex training data.
4. The large model fine-tuning training method based on data sorting according to claim 1 is characterized in that: The step of performing vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data specifically includes: Performing vector clustering on the simple training data / the complex training data according to the input question vector to obtain a plurality of first clustering data groups and a corresponding plurality of first clustering centers; Determine an average quality score of each of the first cluster data groups according to the quality score, and determine a corresponding data group distance between each of the first cluster data groups according to the distance between each of the first cluster centers; Selecting the first clustering data group with the lowest average quality score as the initial current clustering data group; The unsorted first clustering data group with the smallest data group distance from the current clustering data group is selected in sequence as a new current clustering data group to obtain the simple sorting data / the complex sorting data.
5. A large model fine-tuning training method based on data sorting according to any one of claims 1 to 4, characterized in that: The step of sequentially performing multi-stage fine-tuning training on the preheated large model through the simple sorting data and the complex sorting data to obtain a target fine-tuned large model specifically includes: Using the simple sorting data as initial current training data, and using the preheated large model as the initial current fine-tuning large model; Fine-tune the current fine-tuned large model using the current training data to obtain an updated current fine-tuned large model, and determine the loss value of each current training data; Selecting a plurality of low-loss value data and a plurality of high-loss value data from the current training data according to the loss value, and performing data cleaning on the high-loss value data; Generate updated current training data based on the complex sorting data, the low loss value data and the high loss value data after data cleaning, and return to fine-tune the current fine-tuning large model through the current training data until the fine-tuning training reaches a preset number of times to obtain the target fine-tuning large model.
6. A large model fine-tuning training method based on data sorting according to claim 5, characterized in that: The step of selecting a plurality of low loss value data and a plurality of high loss value data from the current training data according to the loss value specifically includes: Reorder the current training data from small to large according to the loss value; According to the preset second percentile, the reordered current training data is intercepted downward to obtain the low loss value data; Determine a plurality of second clustering data groups according to the current training data, and reorder the current training data in each of the second clustering data groups from small to large according to the loss value; The reordered second cluster data group is intercepted upward according to a preset third percentile to obtain the high loss value data.
7. The large model fine-tuning training method based on data sorting according to claim 5 is characterized in that: The data cleaning of the high loss value data specifically includes: Classifying the high loss value data to obtain incorrect question-answer pairs and correct question-answer pairs; Correcting the question or answer of the incorrect question-answer pair; The correct question and answer pairs are reconstructed and a thought chain process is added.
8. A large model fine-tuning training system based on data sorting, characterized in that: include: A large model preheating module, used to select a plurality of preheating training data from the fine-tuning training data, and preheat the target large model through the preheating training data to obtain a preheated large model; A large model reasoning module, used to determine an input question vector and a quality score of the fine-tuning training data according to the preheated large model; A data partitioning and sorting module, used to partition the fine-tuning training data according to the quality score to obtain simple training data and complex training data, and then perform vector clustering and sorting on the simple training data and the complex training data according to the input question vector to obtain simple sorted data and complex sorted data; The multi-stage fine-tuning training module is used to perform multi-stage fine-tuning training on the preheated large model through the simple sorting data and the complex sorting data in sequence to obtain a target fine-tuning large model.
9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the large model fine-tuning training method based on data sorting are realized as described in any one of claims 1 to 7.
10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the large model fine-tuning training method based on data sorting as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Road damage detection method
CN117292287A
Statistics graph automatic question answering method based on weak supervision alignment mechanism
CN118132720A
System and method for generating a multi-task machine learning model
US20240257209A1
