Large model training method and device, equipment, storage medium and product
By dynamically adjusting the sampling ratio during the training process of large models, the problem of large models getting stuck in local optima is solved, improving model performance and data utilization efficiency, and ensuring that the data is fully learned.
Patent Information
- Application Number
- CN202510297896.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-13
- Publication Date
- 2026-02-10
AI Technical Summary
In existing technologies, large model training relies on expert experience or sampling ratios obtained directly from proxy models, which can easily lead to large models getting stuck in local optima, affecting performance, and may result in insufficient training with high-quality data or over-training with homogeneous data.
By acquiring secondary corpora from multiple different domains, the sampling ratio is calculated based on word-level features. The sampling ratio is then dynamically adjusted using the loss and gain of the trained proxy model and the large model, thereby dynamically adjusting the proportion of different domain corpora in the training process and avoiding local optima.
It achieves dynamic adjustment of the sampling ratio based on the learning degree of the large model on different corpora, avoids getting trapped in local optima, improves model performance, and ensures that the data is fully learned, avoiding insufficient training on high-quality data and over-training on homogeneous data.
Smart Images
Figure CN121502329A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of model training, and particularly relates to a large model training method and device, equipment, a storage medium and a product. BACKGROUND
[0002] The large model training process mainly involves confirming a sampling ratio in the large model training process based on expert experience or a proxy model.
[0003] (1) Based on expert experience: the sampling ratio of each type of data is determined through expert experience. However, this solution is relatively subjective, and may lead to insufficient training of high-quality data and excessive training of homogeneous data.
[0004] (2) Based on a proxy model: a small-scale proxy model is first trained to obtain a sampling ratio, and the sampling ratio is directly used as a fixed sampling ratio in the large model training process. However, this solution directly uses the sampling ratio obtained by the proxy model as a fixed sampling ratio in the large model training process, which may easily cause the large model to fall into a local optimal solution and affect the performance of the large model. SUMMARY
[0005] The present application provides a large model training method, device, equipment, a storage medium and a product to solve the problem that the prior art depends on expert experience and directly uses the sampling ratio obtained by a proxy model as a fixed sampling ratio in the large model training process, which may easily cause the large model to fall into a local optimal solution and affect the performance of the large model.
[0006] To achieve the above object, the embodiments of the present application provide a large model training method, comprising:
[0007] obtaining a plurality of second corpus sets; wherein each second corpus set belongs to a different field;
[0008] performing word segmentation on corpus samples of each second corpus set, and obtaining a first sampling ratio of each second corpus set according to the word-level features of each second corpus set;
[0009] sampling the corresponding second corpus set based on each first sampling ratio to obtain a first training sample set, training a proxy model based on the first training sample set, and obtaining a trained proxy model;
[0010] sample each of the second corpus sets according to the current second sampling ratio of the second corpus set, to obtain a current second training sample set, train the large model based on the current second training sample set, and in the training process, adjust the next second sampling ratio according to the current loss gain of the trained proxy model and the current loss gain of the large model; wherein the proxy model and the large model have the same structure, and the parameter quantity of the proxy model is less than that of the large model.
[0011] As an improvement of the above scheme, the current second sampling ratio of the second corpus set is determined by the current repetition degree of the second corpus set and the current basic sampling ratio;
[0012] The sampling each of the second corpus sets according to the current second sampling ratio of the second corpus set, to obtain a current second training sample set, train the large model based on the current second training sample set, and in the training process, adjust the next second sampling ratio according to the current loss gain of the trained proxy model and the current loss gain of the large model, comprises:
[0013] Initialize the repetition degree of each of the second corpus sets;
[0014] Take each of the first sampling ratios as the initial basic sampling ratio of the corresponding second corpus set;
[0015] According to the current repetition degree of each of the second corpus sets and the current basic sampling ratio, obtain the current second sampling ratio of each of the second corpus sets, and sample each of the second corpus sets according to the current second sampling ratio of the second corpus set, to obtain a current second training sample set;
[0016] Train the large model using the current second training sample set;
[0017] After this training, adjust the current basic sampling ratio of each of the second corpus sets according to the current loss gain of the trained proxy model and the current loss gain of the large model;
[0018] According to the current second training sample set, update the current repetition degree of each of the second corpus sets;
[0019] If the preset condition is not met, return to the step of obtaining the current second sampling ratio of each of the second corpus sets according to the current repetition degree of each of the second corpus sets and the current basic sampling ratio, and sampling each of the second corpus sets according to the current second sampling ratio of the second corpus set, to obtain a current second training sample set, until the preset condition is met, and stop training.
[0020] As an improvement of the above-mentioned scheme, the obtaining a plurality of second corpus sets comprises:
[0021] obtaining a full corpus dataset; the full corpus dataset comprises a plurality of original corpus sets, each of which belongs to different fields;
[0022] For each of the original corpus sets, clustering a plurality of corpus samples sampled from the original corpus set to obtain a plurality of first corpus sets;
[0023] Based on the first Euclidean distance between each corpus sample of the full corpus dataset and the center corpus sample of each first corpus set, each corpus sample of the full corpus dataset is divided into the first Euclidean distance closest first corpus set to form a plurality of second corpus sets.
[0024] As an improvement of the above-mentioned scheme, after the above-mentioned scheme, before the above-mentioned scheme, the large model training method comprises:
[0025] For each first corpus set, calculate the second Euclidean distance between each corpus sample of the first corpus set and the center corpus sample of the first corpus set, and delete the corpus sample with the second Euclidean distance greater than the preset distance from the first corpus set.
[0026] As an improvement of the above-mentioned scheme, the word-level feature comprises at least one of adjacent word correlation, first information entropy of word in the second corpus set, and second information entropy of word in all second corpus sets.
[0027] As an improvement of the above-mentioned scheme, the above-mentioned scheme comprises:
[0028] Tokenizing the corpus sample of each of the second corpus sets to obtain a plurality of words, calculating the first probability of the word of the second corpus set appearing in the second corpus set, and the second probability of the word of the second corpus set appearing in all second corpus sets;
[0029] According to the first probability, the adjacent word correlation between two adjacent words in the second corpus set is calculated;
[0030] According to the first probability, the first information entropy of the word of the second corpus set in the second corpus set is calculated;
[0031] According to the first probability and the second probability, a second information entropy of a word in the second corpus set in all the second corpus sets is calculated;
[0032] The adjacent word correlation degree, the first information entropy and the second information entropy are weighted to obtain a first sampling proportion of the second corpus set.
[0033] To achieve the above object, the embodiment of the present application further provides a large model training device, comprising:
[0034] An acquisition module is configured to acquire a plurality of second corpus sets; wherein each second corpus set belongs to a different field;
[0035] A word segmentation module is configured to perform word segmentation on corpus samples of each second corpus set, and obtain a first sampling proportion of each second corpus set according to word-level features of each second corpus set;
[0036] A first training module is configured to sample each second corpus set based on the first sampling proportion of the second corpus set, to obtain a first training sample set, train a proxy model based on the first training sample set, and obtain a trained proxy model;
[0037] A second training module is configured to sample each second corpus set based on a current second sampling proportion of the second corpus set, to obtain a current second training sample set, train a large model based on the current second training sample set, and adjust a next second sampling proportion according to a current loss gain of the trained proxy model and a current loss gain of the large model during the training process; wherein the proxy model and the large model have the same structure, and the parameter amount of the proxy model is less than that of the large model.
[0038] To achieve the above object, the embodiment of the present application further provides a large model training device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the large model training method as described above when executing the computer program.
[0039] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, comprising a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the large model training method as described above when running.
[0040] To achieve the above object, the embodiment of the present application further provides a computer program product, comprising computer programs / instructions, which implement the large model training method as described above when executed by a processor.
[0041] Compared with existing technologies, the large model training method, apparatus, device, storage medium, and product provided in this application adjust the second sampling ratio based on the current loss gain of the trained surrogate model and the current loss gain of the large model during the training process. This dynamically adjusts the second sampling ratio of the second corpus in future training according to the learning degree of the large model on different corpora, avoiding the large model from getting trapped in local optima and improving its performance. Furthermore, this application does not rely on expert experience, avoiding insufficient training with high-quality data and over-training with homogeneous data. In addition, this application determines the first sampling ratio during the surrogate model training process based on the complexity of the second corpus, allowing the data to learn fully and improving model performance. Attached Figure Description
[0042] Figure 1 This is a flowchart of a large model training method provided in an embodiment of this application;
[0043] Figure 2 This is a structural block diagram of a large model training device provided in an embodiment of this application;
[0044] Figure 3 This is a structural block diagram of a large model training device provided in an embodiment of this application. Detailed Implementation
[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0046] See Figure 1 , Figure 1 This is a flowchart of a large model training method provided in an embodiment of this application. The large model training method includes:
[0047] S1. Obtain multiple second corpora; wherein each second corpus belongs to a different domain;
[0048] S2. Segment the corpus samples of each second corpus into words, and obtain the first sampling ratio of each second corpus based on the word-level features of each second corpus.
[0049] S3. Based on each of the first sampling ratios, sample the corresponding second corpus to obtain the first training sample set, and train the proxy model based on the first training sample set to obtain the trained proxy model.
[0050] S4, sampling the corresponding second corpus based on a current second sampling ratio of each of the second corpora, obtaining a current second training sample set, training the large model based on the current second training sample set, and adjusting a next second sampling ratio according to a current loss gain of the trained proxy model and a current loss gain of the large model during the training process; wherein the proxy model and the large model have the same structure, and a parameter quantity of the proxy model is less than a parameter quantity of the large model.
[0051] In the present application, the learning degree of the model for different domain second corpora is measured by the loss gain of the model, and the large model uses a dynamically adjusted method during the training process to dynamically adjust the second sampling ratio of the second corpus in future training according to the learning degree of the second corpus in different domains. The trained large model can be applied to artificial intelligence, medical treatment, natural language processing, computer vision and other fields.
[0052] Because the data quantity and learning difficulty of the second corpora in different domains are different, a proxy model is trained in advance as a benchmark for corpus difficulty. Specifically, a proxy model with a parameter scale much smaller than the large model is constructed and trained: in the proxy model training, based on a first sampling ratio of each of the second corpora The corresponding second corpus is sampled, and the loss function is used for training. The present application embodiment does not limit the loss function, for example, the following autoregressive language model loss function formula is used for training of the proxy model to obtain the proxy model Model proxy :
[0053]
[0054] In the formula, z represents the zth word of the second corpus D‘ j , and N1 represents the number of words of the second corpus. The word, i.e., token of the second corpus refers to the smallest semantic unit of the corpus sample of the second corpus. Because the second corpora in different domains have different learning difficulties, the second corpus with a larger learning difficulty should be allocated a relatively high second sampling ratio during the training process, and for the same reason, the second sampling ratio of the second corpus for which the large model has obtained learning ability should be reduced during the training process.
[0055] Because the second corpora in different domains have different learning difficulties, the second corpus with a larger learning difficulty should be allocated a relatively high second sampling ratio during the training process, and for the same reason, the second sampling ratio of the second corpus for which the large model has obtained learning ability should be reduced during the training process.
[0056] In the present application, in order to improve the training effect of the large model, a dynamic sampling scheme is adopted for model training. By dynamically adjusting the proportion of the second corpus set in the next training batch, i.e. the next second sampling ratio, according to the learning degree of the large model on different fields of the second corpus set, the optimization of the model training effect is realized. Specifically:
[0057] By comparing the learning degree of the current large model on the second corpus set with the performance of the trained proxy model on the same data, if the learning degree of the second corpus set is significantly weaker than the performance of the trained proxy model on the same data, it is considered that the second corpus set has not been fully learned, and the second sampling ratio of the second corpus set should be increased; otherwise, the second sampling ratio of the second corpus set should be reduced. For example, directly taking each first sampling ratio as the initial second sampling ratio of the corresponding second corpus set, and in the training process, adjusting the next second sampling ratio according to the current loss gain of the trained proxy model and the current loss gain of the large model.
[0058] In an optional embodiment, the obtaining a plurality of second corpus sets comprises:
[0059] Obtaining a full corpus data set; the full corpus data set comprises a plurality of original corpus sets, each original corpus set belonging to a different field;
[0060] For each original corpus set, clustering a plurality of corpus samples sampled from the original corpus set to obtain a plurality of first corpus sets;
[0061] Based on the first Euclidean distance between each corpus sample of the full corpus data set and the center corpus sample of each first corpus set, each corpus sample of the full corpus data set is divided into the first corpus set closest to the first Euclidean distance, forming a plurality of second corpus sets.
[0062] The embodiments of the present application can divide the full corpus data set in the pre-training process of the large model into original corpus sets of different fields based on a classification system of data sources, semantics, etc.: D0, D1, …, Dn. n wherein D i represents the original corpus set of the i-th field, i = 0, 1, …, n. For example, the full corpus data set is divided into n clustering clusters by a semantic clustering method, and each clustering cluster is an original corpus set. One original corpus set belongs to one field.
[0063] Then, in order to compress the subdivision calculation cost, a plurality of corpus samples are sampled from each original corpus set D i to obtain a plurality of corpus samples sampled from each original corpus set D i i The sampled corpus samples were clustered to obtain multiple first corpus sets: in, This indicates the original corpus D. i The k-th first corpus set is obtained by clustering multiple sampled corpus samples, where k = 0, 1, ..., K. In other words, this step clusters each original corpus set D... i The original corpus D, which covers a broad range of fields, was further subdivided. i The first corpus is further subdivided into multiple domains and scopes.
[0064] Finally, in order to achieve the subdivision of the full corpus dataset, each corpus sample in the full corpus dataset is compared with each first corpus set. The first Euclidean distance between the central corpus samples (i.e., cluster centers) is used to assign each corpus sample in the full corpus dataset to the first corpus set with the closest first Euclidean distance. Multiple second corpora are formed: D'0, D'1, ..., D' m ; where D' j Let j represent the j-th second corpus; j = 0, 1, ..., m.
[0065] This embodiment first subdivides multiple corpus samples from the original corpus to obtain multiple first corpus sets. Then, each corpus sample from the full corpus dataset is assigned to a suitable first corpus set. This achieves the subdivision of all corpus samples in the full corpus dataset, reducing the internal correlation of the data in the full corpus dataset, improving the accuracy of the sampling ratio, enabling the model to more accurately capture and understand features, learn better, and improve model performance. Moreover, by clustering the sampled corpus samples, it avoids directly clustering and subdividing all corpus samples in the full corpus dataset, thus reducing the computational cost of subdividing the full corpus dataset.
[0066] In an optional embodiment, after clustering multiple corpus samples sampled from each of the original corpus sets to obtain multiple first corpus sets, and before assigning each corpus sample of the full corpus dataset to the first corpus set closest in the first Euclidean distance, the large model training method includes:
[0067] For each first corpus, calculate the second Euclidean distance between each corpus sample in the first corpus and the central corpus sample in the first corpus, and delete corpus samples whose second Euclidean distance is greater than a preset distance from the first corpus.
[0068] In order to improve the accuracy of segmentation of all corpus samples in the full corpus dataset, this embodiment of the application removes corpus samples with a second Euclidean distance greater than a preset distance from the first corpus. Deleting it makes the first corpus... Only corpus samples with typical characteristics are retained, namely, corpus samples with a second Euclidean distance less than or equal to a preset distance. The second Euclidean distance is the Euclidean distance between corpus samples in the first corpus and the central corpus sample (i.e., cluster center) of the first corpus. The typical characteristic is that the second Euclidean distance is less than or equal to the preset distance. Specifically, the preset distance is determined based on the average value of the second Euclidean distances in the first corpus.
[0069] In other words, the first corpus is obtained through clustering. Then, corpus samples with a second Euclidean distance greater than the preset distance are removed from the first corpus. Delete, and you will get the final first corpus. At this point, we recalculate the relationship between each corpus sample in the full corpus dataset and each final first corpus set. The first Euclidean distance between the central corpus samples is used to assign each corpus sample in the full corpus dataset to the final first corpus set that has the closest first Euclidean distance.
[0070] In one specific embodiment, for each original corpus D i Multiple sampled corpus data are clustered to obtain multiple first corpus sets. Next, perform the following steps:
[0071] Calculate the first corpus Central corpus sample:
[0072] Calculate the first corpus Each corpus sample and the first corpus The second Euclidean distance between the central corpus samples:
[0073] Calculate the first corpus The average value of the second Euclidean distance:
[0074] Set preset distance
[0075] in, Represents the first corpus The central corpus sample, Represents the first corpus A certain corpus sample, N2 represents the first corpus set. The number of corpus samples; N3 represents the first corpus. The second Euclidean distance; distance() represents the second Euclidean distance. Represents the first corpus The average of the second Euclidean distance, where alpha represents a preset parameter, typically alpha < 1, used to narrow down the first corpus. The number of feature points.
[0076] Next, the first corpus Medium to large The remaining corpus samples are discarded, and a seed sample set is formed, which is the final first corpus set. That is, the final first corpus It only retains less than or equal to The corpus sample.
[0077] Finally, for each corpus sample in the full corpus dataset, calculate its relationship with each final first corpus. The first Euclidean distance between the central corpus samples is used to classify them into the first corpus set with the closest first Euclidean distance. Multiple second corpora D' were formed j That is, the second corpus D' j domain j = The final first corpus The domain k; where x f This represents one of the corpus samples in the full corpus dataset; This represents the final first corpus. One of the corpus samples, the first Euclidean distance is the distance between the corpus sample in the full corpus dataset and the final first corpus set. The Euclidean distance between the central corpus samples.
[0078] In one optional embodiment, the word-level features include at least one of the following: adjacent word correlation degree, first information entropy of the word in the second corpus, and second information entropy of the word in all the second corpus.
[0079] It is worth noting that word-level features can measure the complexity of a corpus, i.e. the difficulty of training and learning. By analyzing the word-level features of the second corpus, the first sampling ratio of the second corpus can be obtained. The higher the complexity, the larger the first sampling ratio.
[0080] In an optional embodiment, the step of segmenting the corpus samples of each second corpus and obtaining the first sampling ratio of each second corpus based on the word-level features of each second corpus includes:
[0081] For each corpus sample in the second corpus, perform word segmentation to obtain multiple words, calculate the first probability of a word in the second corpus appearing in the second corpus, and the second probability of a word in the second corpus appearing in all the second corpus.
[0082] Based on the first probability, calculate the adjacency degree between two adjacent words in the second corpus;
[0083] Based on the first probability, calculate the first information entropy of the words in the second corpus in the second corpus;
[0084] Based on the first probability and the second probability, calculate the second information entropy of the words in the second corpus across all the second corpus sets;
[0085] The adjacent word correlation degree, the first information entropy, and the second information entropy are weighted to obtain the first sampling ratio of the second corpus.
[0086] For example, according to Calculate the adjacency degree between two adjacent words in the second corpus; where, This represents the second corpus D' j The z-th word, This represents the second corpus D' j The (z+1)th word, express Second Corpus D' j The first probability of occurrence, Indicates t z+1 Second Corpus D' j The first probability of occurrence. This formula is based on the current word t. z Determine the next word t z+1 The entropy is used to calculate the correlation between two adjacent words; the higher the value, the more complex the corpus.
[0087] according to Calculate the first information entropy of the words in the second corpus; the higher the value, the more complex the corpus.
[0088] For example, according to: Calculate the second information entropy of the words in the second corpus across all words in the second corpus; Indicates t z The second probability of occurrence in the entire second corpus. The second information entropy is used to correct the first information entropy to offset words that are particularly complex in the full dataset.
[0089] In an optional embodiment, the weighting of the adjacent word correlation, the first information entropy, and the second information entropy to obtain the first sampling ratio of the second corpus includes:
[0090] The adjacent word correlation degree, the first information entropy, and the second information entropy are weighted to obtain the weighted result of the second corpus;
[0091] The weighted result of the second corpus is normalized to obtain the first sampling ratio of the second corpus.
[0092] For example, according to Calculate the weighted result of the second corpus. The weighted result This can be used to characterize the complexity of the second corpus. Here, sum1 represents the first summation function, sum2 represents the second summation function, a represents the first preset parameter, b represents the second preset parameter, and c represents the third preset parameter.
[0093] To better understand the above formula, assume the second corpus is D' j There are two corpus samples: Corpus Sample 1: We haven't seen each other for a long time; Corpus Sample 2: We Chinese are hospitable. After word segmentation, the words in Corpus Sample 1 are: we, good, long, not, see, gone; the words in Corpus Sample 2 are: we, Chinese, good, hospitable.
[0094] For us:
[0095] For good:
[0096]
[0097] The embodiments of this application do not limit the specific method of normalization. For example, the weighted result can be normalized using the softmax method: Second corpus D' j First sampling ratio
[0098] In one optional embodiment, the current second sampling ratio of the second corpus is determined by the current repetition rate of the second corpus and the current base sampling ratio;
[0099] The step of sampling the corresponding second corpus based on the current second sampling ratio of each second corpus to obtain the current second training sample set, training the large model based on the current second training sample set, and adjusting the next second sampling ratio according to the current loss gain of the trained proxy model and the current loss gain of the large model during the training process includes:
[0100] The repetition rate of each second corpus set is initialized;
[0101] Each of the first sampling ratios is used as the initial base sampling ratio of the corresponding second corpus;
[0102] Based on the current repetition rate and the current basic sampling ratio of each second corpus, the current second sampling ratio of each second corpus is obtained. Based on the current second sampling ratio of each second corpus, the corresponding second corpus is sampled to obtain the current second training sample set.
[0103] The large model is trained using the current second training sample set;
[0104] After this training, the base sampling ratio of each second corpus is adjusted based on the current loss gain of the trained proxy model and the current loss gain of the large model.
[0105] Based on the current second training sample set, update the current repetition rate of each second corpus;
[0106] If the preset conditions are not met, return to the step of obtaining the current second sampling ratio of each second corpus based on the current repetition and the current basic sampling ratio of each second corpus, and sampling the corresponding second corpus based on the current second sampling ratio of each second corpus to obtain the current second training sample set, until the preset conditions are met and training stops.
[0107] It is worth noting that since the number of corpus samples in each second corpus is limited, increasing the second sampling ratio of some second corpus sets will cause the corpus samples of that second corpus set to be repeatedly fed into the training data. The derivative model will become overly dependent on the repeated data and reduce its generalization. Therefore, in this application, repetition is also introduced as one of the factors of the second sampling probability.
[0108] The embodiments of this application are for training to obtain the final large model, and the specific training process is as follows:
[0109] (1) Initialize the second corpus D' j Repeatability
[0110] (2) The second corpus D' j First sampling ratio As the second corpus D' j Initial base sampling ratio
[0111] (3) According to Calculate the second corpus D j ‘ Current second sampling ratio Based on the second corpus D j ‘ Current second sampling ratio For the second corpus D j ‘ Sampling is performed to obtain the current second training sample set, which includes multiple second corpus sets obtained based on multiple second sampling ratios. In the formula, This represents the second corpus D' j Current baseline sampling ratio;
[0112] (4) Train the large model using the current second training sample set;
[0113] (5) Calculate the loss and gain of the large model. and Proxy Model proxy Loss and gain according to Update the second corpus D' j Current base sampling ratio In the formula, η is the update step size. From the second corpus The validation set extracted from the data.
[0114] (6) According to Update the current repetition rate; where count(D' j ) is the second corpus D' in the current second training sample set. j The total number of corpus samples used, It is the second corpus D' j The total number of corpus samples.
[0115] (7) Repeat steps (3) to (6) until the preset conditions are met to complete the training.
[0116] In this embodiment, the first sampling ratio is used as the initial base sampling ratio to reduce the computational and training complexity of the second sampling ratio; at the same time, repetition is introduced to ensure that while the second sampling ratio of high-quality data increases, data oversampling is suppressed.
[0117] The large model training method provided in this application adjusts the second sampling ratio of the second corpus in future training based on the current loss and gain of the trained surrogate model and the current loss and gain of the large model during the training process. This dynamically adjusts the second sampling ratio of the second corpus in future training according to the learning degree of the large model on different corpora, which can avoid the large model getting stuck in local optima and improve the performance of the large model. Moreover, this application embodiment does not rely on expert experience, which can avoid the situation of insufficient training with high-quality data and over-training with homogeneous data. In addition, this application embodiment determines the first sampling ratio in the surrogate model training process according to the complexity of the second corpus, so that the data can be fully learned and the model performance can be improved.
[0118] See Figure 2 , Figure 2 This is a structural block diagram of a large model training device 10 provided in an embodiment of this application. The large model training device 10 includes:
[0119] The acquisition module 11 is used to acquire multiple second corpora; wherein each second corpus belongs to a different domain;
[0120] The word segmentation module 12 is used to segment words into each corpus sample of the second corpus set, and to obtain the first sampling ratio of each second corpus set based on the word-level features of each second corpus set.
[0121] The first training module 13 is used to sample the corresponding second corpus based on each of the first sampling ratios to obtain a first training sample set, and to train the proxy model based on the first training sample set to obtain a trained proxy model.
[0122] The second training module 14 is used to sample the corresponding second corpus based on the current second sampling ratio of each second corpus to obtain the current second training sample set, train the large model based on the current second training sample set, and adjust the next second sampling ratio according to the current loss gain of the trained surrogate model and the current loss gain of the large model during the training process; wherein, the surrogate model and the large model have the same structure, and the number of parameters of the surrogate model is smaller than the number of parameters of the large model.
[0123] Optionally, the current second sampling ratio of the second corpus is determined by the current repetition rate and the current base sampling ratio of the second corpus;
[0124] The second training module 14 is specifically used for:
[0125] The repetition rate of each second corpus set is initialized;
[0126] Each of the first sampling ratios is used as the initial base sampling ratio of the corresponding second corpus;
[0127] Based on the current repetition rate and the current basic sampling ratio of each second corpus, the current second sampling ratio of each second corpus is obtained. Based on the current second sampling ratio of each second corpus, the corresponding second corpus is sampled to obtain the current second training sample set.
[0128] The large model is trained using the current second training sample set;
[0129] After this training, the base sampling ratio of each second corpus is adjusted based on the current loss gain of the trained proxy model and the current loss gain of the large model.
[0130] Based on the current second training sample set, update the current repetition rate of each second corpus;
[0131] If the preset conditions are not met, return to the step of obtaining the current second sampling ratio of each second corpus based on the current repetition and the current basic sampling ratio of each second corpus, and sampling the corresponding second corpus based on the current second sampling ratio of each second corpus to obtain the current second training sample set, until the preset conditions are met and training stops.
[0132] Optionally, the acquisition module 11 is specifically used for:
[0133] Obtain the full corpus dataset; the full corpus dataset includes multiple original corpus sets, each belonging to a different domain;
[0134] For each of the original corpus sets, multiple corpus samples sampled from the original corpus set are clustered to obtain multiple first corpus sets;
[0135] Based on the first Euclidean distance between each corpus sample in the full corpus dataset and the central corpus sample of each first corpus set, each corpus sample in the full corpus dataset is assigned to the first corpus set with the closest first Euclidean distance, forming multiple second corpus sets.
[0136] Optionally, the acquisition module 11 is further configured to:
[0137] For each first corpus, calculate the second Euclidean distance between each corpus sample in the first corpus and the central corpus sample in the first corpus, and delete corpus samples whose second Euclidean distance is greater than a preset distance from the first corpus.
[0138] Optionally, the word-level features include at least one of the following: the correlation degree between adjacent words, the first information entropy of the word in the second corpus, and the second information entropy of the word in all the second corpus.
[0139] Optionally, the word segmentation module 12 is further configured to:
[0140] For each corpus sample in the second corpus, perform word segmentation to obtain multiple words, calculate the first probability of a word in the second corpus appearing in the second corpus, and the second probability of a word in the second corpus appearing in all the second corpus.
[0141] Based on the first probability, calculate the adjacency degree between two adjacent words in the second corpus;
[0142] Based on the first probability, calculate the first information entropy of the words in the second corpus in the second corpus;
[0143] Based on the first probability and the second probability, calculate the second information entropy of the words in the second corpus across all the second corpus sets;
[0144] The adjacent word correlation degree, the first information entropy, and the second information entropy are weighted to obtain the first sampling ratio of the second corpus.
[0145] It is worth noting that the working process of each module in the large model training device 10 described in this application embodiment can refer to the working process of the large model training method described in the above embodiment, and will not be repeated here.
[0146] The large model training device 10 provided in this application adjusts the second sampling ratio during the large model training process based on the current loss gain of the trained surrogate model and the current loss gain of the large model. This dynamically adjusts the second sampling ratio of the second corpus in future training according to the large model's learning degree on different corpora, preventing the large model from getting trapped in local optima and improving its performance. Furthermore, this application embodiment does not rely on expert experience, avoiding insufficient training with high-quality data and over-training with homogeneous data. In addition, this application embodiment determines the first sampling ratio during surrogate model training based on the complexity of the second corpus, ensuring sufficient data learning and improving model performance.
[0147] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the large model training method as described in any of the above embodiments.
[0148] Furthermore, this application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the large model training method as described in any of the above embodiments.
[0149] See Figure 3 , Figure 3 This is a structural block diagram of a large model training device 20 provided in an embodiment of this application. The large model training device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described large model training method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.
[0150] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the large model training device 20.
[0151] The large model training device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the large model training device 20 and does not constitute a limitation on the large model training device 20. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components. For example, the large model training device 20 may also include input / output devices, network access devices, buses, etc.
[0152] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the large model training device 20, connecting all parts of the large model training device 20 via various interfaces and lines.
[0153] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the large model training device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0154] The modules / units integrated into the large model training device 20, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0155] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0156] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.
Claims
1. A method for training large models, characterized in that, include: Multiple second corpora are obtained; each of the second corpora belongs to a different domain. For each corpus sample in the second corpus, word segmentation is performed, and the first sampling ratio of each second corpus is obtained based on the word-level features of each second corpus. Based on each of the first sampling ratios, the corresponding second corpus is sampled to obtain the first training sample set. The proxy model is then trained based on the first training sample set to obtain the trained proxy model. Based on the current second sampling ratio of each second corpus set, the corresponding second corpus set is sampled to obtain the current second training sample set. The large model is trained based on the current second training sample set. During the training process, the next second sampling ratio is adjusted according to the current loss gain of the trained surrogate model and the current loss gain of the large model. The surrogate model and the large model have the same structure, but the number of parameters of the surrogate model is smaller than that of the large model.
2. The large model training method as described in claim 1, characterized in that, The current second sampling ratio of the second corpus is determined by the current repetition rate and the current base sampling ratio of the second corpus; The step of sampling the corresponding second corpus based on the current second sampling ratio of each second corpus to obtain the current second training sample set, training the large model based on the current second training sample set, and adjusting the next second sampling ratio according to the current loss gain of the trained proxy model and the current loss gain of the large model during the training process includes: The repetition rate of each second corpus set is initialized; Each of the first sampling ratios is used as the initial base sampling ratio of the corresponding second corpus; Based on the current repetition rate and the current basic sampling ratio of each second corpus, the current second sampling ratio of each second corpus is obtained. Based on the current second sampling ratio of each second corpus, the corresponding second corpus is sampled to obtain the current second training sample set. The large model is trained using the current second training sample set; After this training, the base sampling ratio of each second corpus is adjusted based on the current loss gain of the trained proxy model and the current loss gain of the large model. Based on the current second training sample set, update the current repetition rate of each second corpus; If the preset conditions are not met, return to the step of obtaining the current second sampling ratio of each second corpus based on the current repetition and the current basic sampling ratio of each second corpus, and sampling the corresponding second corpus based on the current second sampling ratio of each second corpus to obtain the current second training sample set, until the preset conditions are met and training stops.
3. The large model training method as described in claim 1, characterized in that, The acquisition of multiple second corpora includes: Obtain the full corpus dataset; the full corpus dataset includes multiple original corpus sets, each belonging to a different domain; For each of the original corpus sets, multiple corpus samples sampled from the original corpus set are clustered to obtain multiple first corpus sets; Based on the first Euclidean distance between each corpus sample in the full corpus dataset and the central corpus sample of each first corpus set, each corpus sample in the full corpus dataset is assigned to the first corpus set with the closest first Euclidean distance, forming multiple second corpus sets.
4. The large model training method as described in claim 3, characterized in that, After clustering multiple corpus samples from each original corpus set to obtain multiple first corpus sets, and before assigning each corpus sample from the full corpus dataset to the first corpus set closest in Euclidean distance, the large model training method includes: For each first corpus, calculate the second Euclidean distance between each corpus sample in the first corpus and the central corpus sample in the first corpus, and delete corpus samples whose second Euclidean distance is greater than a preset distance from the first corpus.
5. The large model training method as described in claim 1, characterized in that, The word-level features include: the correlation degree between adjacent words, the first information entropy of the word in the second corpus, and at least one of the second information entropy of the word in all the second corpus.
6. The large model training method as described in claim 1, characterized in that, The step of segmenting the corpus samples of each second corpus into words, and obtaining the first sampling ratio of each second corpus based on the word-level features of each second corpus, includes: For each corpus sample in the second corpus, perform word segmentation to obtain multiple words, calculate the first probability of a word in the second corpus appearing in the second corpus, and the second probability of a word in the second corpus appearing in all the second corpus. Based on the first probability, calculate the adjacency degree between two adjacent words in the second corpus; Based on the first probability, calculate the first information entropy of the words in the second corpus in the second corpus; Based on the first probability and the second probability, calculate the second information entropy of the words in the second corpus across all the second corpus sets; The adjacent word correlation degree, the first information entropy, and the second information entropy are weighted to obtain the first sampling ratio of the second corpus.
7. A large-scale model training device, characterized in that, include: The acquisition module is used to acquire multiple second corpora; each of the second corpora belongs to a different domain. The word segmentation module is used to segment words into each corpus sample of the second corpus and obtain the first sampling ratio of each second corpus based on the word-level features of each second corpus. The first training module is used to sample the corresponding second corpus based on each of the first sampling ratios to obtain the first training sample set, and to train the proxy model based on the first training sample set to obtain the trained proxy model. The second training module is used to sample the corresponding second corpus based on the current second sampling ratio of each second corpus to obtain the current second training sample set. The large model is trained based on the current second training sample set. During the training process, the next second sampling ratio is adjusted according to the current loss gain of the trained surrogate model and the current loss gain of the large model. The surrogate model and the large model have the same structure, but the number of parameters of the surrogate model is smaller than that of the large model.
8. A large-scale model training device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the large model training method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the large model training method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the large model training method as described in any one of claims 1 to 6.