Model training method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a model training method and device, computer equipment and a storage medium, and the method comprises the following steps: respectively setting a duplication removing function according to a first preset number of preset parameters, processing initial corpus data according to the set duplication removing function, and according to the length of word data in a processing result, calculating the length of the word data; the word data are put into corresponding partition files, and the partition files are used for recording the word data of the corresponding length; according to a model training requirement, obtaining a second preset number of target word data from partition files contained in each partition file set, and summarizing the target word data to obtain a third preset number of training data sets; and training the initial model by using the training data set to obtain a third preset number of trained models. According to the method, the problems that only a single training data set can be obtained and different functional models cannot be trained and the training quality cannot be ensured when various text corpora are processed by using a unified processing scheme are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training method, device, computer equipment and storage medium. Background Art

[0002] Currently, when processing a large and diverse text corpus for Large Language Model (LLM) training, we often process the text corpus through a fixed deduplication and length screening scheme based on existing experience to form a training data set, and then use the training data set to train the model. For example: the maximum text length supported by the model is 32,000, and we directly use Python's built-in unordered collection to deduplicate the text, and then delete the data with a length exceeding 32,000 to form a training data set.

[0003] However, due to the different impacts of text corpora from different sources on model training, the training data set formed by the above solution is only a preliminary screening, and it is impossible to screen out the best training data set from the text corpus. The quality of the trained model cannot be guaranteed to be the best, and may even cause the pre-trained model score to be lower than expected. In addition, using a unified processing solution, it is impossible to train models with different functions. For example, if a large amount of conversational text data is collected, and the current model only supports a maximum input length of 8,000, then according to the existing technology, only data with a length of less than 8,000 will be left after preprocessing. When the subsequent model text length can support a maximum of 16,000 or 32,000, it needs to be preprocessed again, which increases the time cost.

[0004] Therefore, the related technology has the problem that a unified processing solution is used to process various text corpora, which can only produce a single training data set and cannot train different functional models and ensure the training quality. Summary of the invention

[0005] In view of this, the present invention provides a model training method, apparatus, computer equipment and storage medium to solve the problem that using a unified processing solution to process various text corpora can only produce a single training data set, and it is impossible to train different functional models and ensure the training quality.

[0006] In a first aspect, the present invention provides a model training method, comprising:

[0007] Deduplication functions are respectively set according to a first preset number of preset parameters, the initial corpus data is processed according to the set deduplication functions, and the word data is placed into corresponding partition files according to the length of the word data in the processing result, wherein the partition files are used to record word data of corresponding lengths, the partition files are included in a first preset number of partition file sets, and the partition file sets correspond to the preset parameters;

[0008] According to the model training requirements, a second preset number of target word data is obtained from the partition files included in each partition file set, and the target word data are aggregated to obtain a third preset number of training data sets;

[0009] The initial models are trained using the training data sets respectively to obtain a third preset number of trained models.

[0010] The model training method provided in this embodiment sets different preset parameters for the deduplication function, processes the initial corpus data according to the set deduplication function, and obtains processing results with various deduplication strengths. The word data of different length intervals in the processing results are classified and stored in different partition files respectively. According to the model training requirements, the word data is flexibly proportioned, and the initial corpus data is fully utilized to build a high-quality training data set, so that different training data sets can be generated using one initial corpus data. Different functional models can be trained using different training data sets, which improves the training quality and reduces the requirements for the quantity of word data to a certain extent. It solves the problem that using a unified processing solution to process various text corpora can only produce a single training data set, and it is impossible to train different functional models and ensure the training quality.

[0011] In some optional implementations, after the initial models are trained using the training data sets to obtain a third preset number of trained models, the method further includes:

[0012] Using a preset test data set to test a third preset number of trained models, and determining a test score corresponding to the trained models according to the test results;

[0013] The trained model with the highest test score is used as the target model, and the training dataset corresponding to the target model is saved.

[0014] In this embodiment, a preset test data set is used to evaluate models trained with different data sets, so as to select the target model with the best performance and expand the initial corpus data according to the training data set corresponding to the target model.

[0015] In some optional implementations, deduplication functions are respectively set according to the first preset number of preset parameters, the initial corpus data is processed according to the set deduplication functions, and the word data is placed in a corresponding partition file according to the length of the word data in the processing result, including:

[0016] Determining a target setting parameter from a first preset number of preset parameters;

[0017] Use the target setting parameters to set the deduplication function and obtain the set deduplication function;

[0018] The initial corpus data is processed according to the set deduplication function to obtain the processing result;

[0019] Using the length processing function to perform word segmentation processing on the data in the processing result to obtain a fourth preset number of word data;

[0020] The partition file set corresponding to the target setting parameters is used as the target partition file set, and the word data is put into the corresponding partition file in the target partition file set according to the length of the word data;

[0021] Determine new target setting parameters from a first preset number of preset parameters, use the new target setting parameters as target setting parameters, and execute subsequent steps starting from setting a deduplication function using the target setting parameters until each preset parameter is used to set the target setting parameters.

[0022] In this embodiment, the first preset number of preset parameters are sequentially used as target setting parameters, and the target setting parameters are used to set the deduplication function to process the initial corpus data, and various deduplication intensity processing results are obtained. The word data of different length intervals in the processing results are classified and stored in different partition files, which is convenient for flexible matching of word data to form different training data sets, so as to make full use of the initial corpus data to build a high-quality training data set.

[0023] In some optional implementations, the data in the processing result is segmented using a length processing function to obtain a fourth preset number of word data, including:

[0024] Obtaining row data contained in the processing result, and deleting row data whose data length is less than a preset threshold, to obtain remaining row data;

[0025] The remaining line data is used as word data to obtain a fourth preset number of word data.

[0026] In some optional implementations, before putting the word data into the corresponding partition file, the method further includes:

[0027] Creating a first preset number of partition file sets, and generating a first identifier for each partition file set according to the first preset number of preset parameters;

[0028] Creating a partition file corresponding to each preset output file partition in each partition file set, and generating a second identifier for each partition file according to the preset output file partition;

[0029] Create a file directory containing the first identifier and the second identifier.

[0030] In this embodiment, a first identifier for each partition file set is generated according to a first preset number of preset parameters, a second identifier for each partition file is generated according to a preset output file partition, and a file directory containing the first identifier and the second identifier is created, so as to facilitate the management of the partition file set and the partition files, and to facilitate the classification of the word data in the deduplication results and save them in the corresponding partition files in the corresponding partition file set.

[0031] In some optional implementations, according to the model training requirements, a second preset number of target word data is obtained from the partition files included in each partition file set, and the target word data is aggregated to obtain a third preset number of training data sets, including:

[0032] Determine the first target length of word data according to model training requirements;

[0033] In each partition file set, the partition file corresponding to the first target length is used as the first target partition file, and the word data contained in the first target partition file is used as the target word data;

[0034] The target word data in each partition file set are respectively aggregated to obtain a third preset number of training data sets.

[0035] In this embodiment, according to the model training requirements, the first target length of the word data is determined, and the word data of the partition files corresponding to the first target length in each partition file set are aggregated to obtain multiple training data sets. According to the model training requirements, the word data can be flexibly matched to form different training data sets, and the initial corpus data can be fully utilized to build a high-quality training data set to meet the requirements of training different functional models and ensuring the training quality.

[0036] In some optional implementations, according to the model training requirements, a second preset number of target word data is obtained from the partition files included in each partition file set, and the target word data is aggregated to obtain a third preset number of training data sets, including:

[0037] According to the model training requirements, determine the second target length and target number of word data;

[0038] Determine the text length threshold supported by the initial model;

[0039] When the text length threshold is greater than the second target length, the length between the second target length and the text length threshold is used as the third target length;

[0040] In each partition file set, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file;

[0041] Determine a preset ratio, and determine a first quantity and a second quantity according to the preset ratio and the target quantity;

[0042] Selecting a first number of word data from the word data included in the second target partition file as target word data, and selecting a second number of word data from the word data included in the third target partition file as target word data;

[0043] The target word data in each partition file set are respectively aggregated to obtain a third preset number of training data sets.

[0044] In this embodiment, a first number of word data is selected from the word data contained in the second target partition file as the target word data, and a second number of word data is selected from the word data contained in the third target partition file as the target word data, and the target word data in each partition file set is respectively summarized to obtain a third preset number of training data sets. According to the model training requirements, the word data is flexibly matched to form different training data sets, and the initial corpus data is fully utilized to construct a high-quality training data set to meet the requirements of training different functional models and ensuring the training quality.

[0045] In a second aspect, the present invention provides a model training device, comprising:

[0046] A processing module, used to set deduplication functions according to a first preset number of preset parameters, respectively, process the initial corpus data according to the set deduplication functions, and put the word data into corresponding partition files according to the length of the word data in the processing result, wherein the partition file is used to record word data of corresponding length, the partition file is included in a first preset number of partition file sets, and the partition file set corresponds to the preset parameters;

[0047] A summarizing module, used to obtain a second preset number of target word data from the partition files included in each partition file set according to model training requirements, and summarize the target word data to obtain a third preset number of training data sets;

[0048] The training module is used to train the initial models using the training data sets to obtain a third preset number of trained models.

[0049] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the model training method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the model training method of the first aspect or any corresponding embodiment thereof.

[0051] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the model training method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a schematic diagram of the current processing text prediction and model training process according to an embodiment of the present invention;

[0054] Figure 2 is a flow chart of a model training method according to an embodiment of the present invention;

[0055] Figure 3 is a flowchart of a method for preprocessing a large language model dataset according to an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of a file directory according to an embodiment of the present invention;

[0057] Figure 5 is a structural block diagram of a model training device according to an embodiment of the present invention;

[0058] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] The current process of processing text prediction and model training is as follows Figure 1As shown, it includes: preparing raw data, obtaining open source data sets or collecting expected data by yourself, and using these data as raw data. Deduplication, screening and other processing, using Python's unordered set (set) method or simhash algorithm and other deduplication methods to delete duplicate data from the raw data. The deduplicated data is length-screened according to the maximum text length supported by the model. For example, if the model supports a maximum text length of 8000, then data with a length exceeding 8000 needs to be deleted. In addition, Set is an unordered set of Python, and each element inside it is unique (no duplicates) and immutable (cannot be changed); the simhash algorithm reduces the dimension of the text and generates a simhash value. By comparing the distance between the simhash values ​​of different texts, the similarity of the two texts is determined to achieve data deduplication. Pre-training data set, after the raw data is pre-processed by deduplication, length screening, etc., a training data set is formed. Model training, converting the training data set into a file that the model can recognize, and performing model training. Evaluating the quality of the model, after the model training is completed, the trained model is evaluated using the test data set to obtain the model score.

[0061] The above scheme has the problem of not being able to obtain the best quality data set and being inflexible. The above scheme uses the existing processing strategy to preprocess the text corpus from different sources, and finally obtains the training data set. The effect of using the training data set to train the model is likely not to be optimal. For example, to train the article writing ability of a large language model, the data set obtained for traditional books and the data set obtained for online articles are used. The text of traditional book data is refined and of high quality, while the online article data has a large amount of water text and low quality. If the data from the two sources uses the same set of preprocessing solutions, the quality of the final data set will be affected. Using a unified preprocessing strategy, different functional models cannot be trained. For example, a large amount of conversational text data is collected, and the current model only supports a maximum input length of 8000. Then, according to the existing technology, only data with a length of less than 8000 will be left after preprocessing. When the subsequent model text length can support a maximum of 16000 or 32000, the original data needs to be reprocessed and the training data set needs to be regenerated, which increases the time cost. A pre-training dataset is generated from a piece of raw data, but it is not possible to make detailed internal proportions of the raw data. For example, a conversation dataset contains data of various lengths, and it is not possible to configure the proportion of data of each length, which is not flexible enough.

[0062] Based on the above content, the embodiment of the present invention provides a model training method, using the simhash algorithm, by configuring different parameters, to obtain data with multiple deduplication strengths; calculating the length of the deduplication data, and then classifying the data in different length intervals, and storing them in different files respectively; flexibly matching the data to form different training data sets, evaluating the models trained by different training data sets, and selecting the model with the highest score and the corresponding training data set. The best of the best is selected for the original data, and the data is partitioned by artificially controlling the diversity (different sets obtained by using different simhash parameters) and partitioning the data according to the token length, and finally generating multiple different training data sets, and training the best model according to these training data sets, and determining the optimal training data set. The original data can be fully utilized to build high-quality pre-training data sets and pre-training models, and the requirements for the number of pre-training data can be reduced to a certain extent. In order to achieve the effect of generating different data sets from one original data, pre-training the models using different data sets respectively, and then evaluating the trained models, so as to obtain the effect of the pre-training data set that can train the best model.

[0063] According to an embodiment of the present invention, a model training embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device with data processing capabilities, such as a computer, a server, etc., and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0064] In this embodiment, a model training method is provided, which can be used for the above-mentioned computer device. Figure 2 is a flow chart of a model training method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0065] Step S201, respectively set deduplication functions according to a first preset number of preset parameters, process the initial corpus data according to the set deduplication functions, and put the word data into corresponding partition files according to the length of the word data in the processing results, wherein the partition files are used to record word data of corresponding lengths, and the partition files are included in a first preset number of partition file sets, and the partition file sets correspond to the preset parameters.

[0066] Specifically, a large-scale text corpus of diversity for large language model training is obtained as the initial corpus data, for example, the text prediction in the pile data set, which is a large-scale text corpus of diversity for large language model training. The deduplication function is, for example, the simhash algorithm of python, and the corresponding preset parameters are the number of digits f and the distance threshold k. The first preset number of preset parameters are, for example, f=32, k=2; f=64, k=10; f=128, k=2; f=32, k=5, etc. The above k and f are examples for the pile data set. If it is other data sets, the values ​​of the k and f parameters can be adjusted according to the actual content of the data set. By setting different preset parameters for the deduplication function, different levels of deduplication processing are performed on each initial corpus data. The number of digits f is 32 / 64 / 128, and a higher number of digits will provide higher accuracy. The distance threshold k indicates that when searching for similar simhash, a maximum of k digits are allowed to be different, and a higher value will allow more differences. The first preset number represents multiple, and no specific number limit is made here. According to the first preset number of preset parameters, the deduplication function (the number of bits f and the distance threshold k) is set respectively, and the initial corpus data is deduplicated according to the set deduplication function to obtain the processing result. It should be noted that, based on the characteristics of simhash, for long texts (length greater than 1000), settings such as f=32, k=2, f=64, k=10 are adopted; for short texts, settings such as the number of bits f=128, k=2, f=32, k=5 are adopted. That is, different combinations of f and k parameters are used to perform deduplication processing of different intensities on the original data, and finally multiple data sets are obtained. A large language model refers to a language model that can generate language models that conform to the regular content of the language by training on a large amount of text data and learning the grammar and semantics of natural language. The training data set generated in this embodiment can train various large prediction models and is not limited to a certain large language model. In addition, when obtaining a diverse large-scale text corpus for training a large language model, the trained large language model A can be used to generate a diverse large-scale text corpus as initial corpus data, and then the initial corpus data can be processed using steps S101 and S102, and the obtained training data set can be used to train the large language model B.

[0067] The partition file is included in the first preset number of partition file sets, and the partition file set corresponds to the preset parameters. The partition file set can be named with the corresponding preset parameters, for example: f_128_k_2, f_64_k_5, etc. The partition files include: "20-100", "101-200", "201-300", etc., which are used to save word data with a length of 20-100, save word data with a length of 101-200, and save word data with a length of 201-300. Each line in the processing result is regarded as a word data. According to the length of the word data in the processing result, the word data is put into the corresponding partition file, for example: the simhash parameter is set to f=128, k=2, and the token length is 51, which is written into the partition file 20-100.txt in the partition file set f_128_k_2. The token is a single unit element, that is, word data, which is obtained after the processing result is segmented.

[0068] Step S202: According to the model training requirements, a second preset number of target word data are obtained from the partition files included in each partition file set, and the target word data are aggregated to obtain a third preset number of training data sets.

[0069] Specifically, model training requirements include: only using data with a token length of less than 1000, using 10 million data to train short text output capabilities, and the text length requirement cannot exceed 100, etc. According to the model training requirements, a second preset number of target word data is obtained from the partition files contained in each partition file set, and the target word data is summarized to obtain a third preset number of training data sets. For example: only using data with a token length of less than 1000, then only the word data in 20-100.txt to 901-1000.txt contained in each partition file set is used as the target word data, and the target word data is summarized to obtain a third preset number of training data sets. The third preset number is set according to actual needs and can be equal to the number of training data sets.

[0070] It should be noted that the deduplication scheme and the method of partitioning data according to token length can be used to fine-tune the processing of data, for example: the few-shot data set can be freely matched using the files after token partitioning; if this scheme is not used, then when constructing multiple shot data sets, such as 20shot, 30shot, etc., each construction needs to calculate the length of the original data once (if it is not calculated, the data length will exceed the maximum text length supported by the model), which will cause a lot of time waste. When using the method of the present invention, the data is stored in different partition files according to different lengths when the original data is processed for the first time. Then, when constructing data sets of different shots, only one addition and subtraction calculation is required to obtain how many different partitions need to use, and then directly obtain the splicing, avoiding the process of passing the data to the tokenizer to calculate the token length, greatly reducing the construction time of the few-shot data set, and the more few-shot data sets that need to be constructed, the more obvious the time cost of the scheme is reduced. Among them, few-shot is a small number of samples given to the model during reasoning, but weight updates are not allowed. For a typical dataset, few-shot has context and examples (such as an English sentence and its French translation), and shot is sample data.

[0071] Step S203, training the initial models using the training data sets respectively to obtain a third preset number of trained models.

[0072] Specifically, the initial model is trained using the training data sets to obtain a third preset number of trained models, where the initial model is a large language model to be trained. In addition, the models trained using different training data sets can be evaluated to select the model with the highest score and the corresponding training data set.

[0073] The model training method provided in this embodiment sets different preset parameters for the deduplication function, processes the initial corpus data according to the set deduplication function, and obtains processing results with various deduplication strengths. The word data of different length intervals in the processing results are classified and stored in different partition files respectively. According to the model training requirements, the data is flexibly allocated, and the initial corpus data is fully utilized to build a high-quality training data set, so that different training data sets can be generated using one initial corpus data. Different functional models can be trained using different training data sets, which improves the training quality and reduces the requirements for the number of word data to a certain extent. It solves the problem that using a unified processing solution to process various text corpora can only produce a single training data set, and it is impossible to train different functional models and ensure the training quality.

[0074] In some optional implementations, after the initial models are trained using the training data sets to obtain a third preset number of trained models, the method further includes:

[0075] Using a preset test data set to test a third preset number of trained models, and determining a test score corresponding to the trained models according to the test results;

[0076] The trained model with the highest test score is used as the target model, and the training dataset corresponding to the target model is saved.

[0077] Specifically, after the initial models are trained using the training data sets respectively to obtain the third preset number of trained models, the models trained using the different training data sets can be evaluated using the preset test data sets, and the test scores corresponding to the trained models are determined according to the test results. The preset test data sets include, for example, the input data and reference output data of the model selected in advance according to the functions of the model, and the test score of the model is obtained by using the similarity between the output results of the model and the reference output data during the test.

[0078] The model with the highest score is the best model. The trained model with the highest score is used as the target model, and the training data set corresponding to the target model is saved. The initial corpus data can be expanded by referring to the training data set corresponding to the target model. For example, there are only 1 million word data in the initial corpus data that can be used to train the model. However, in order to ensure the training effect, 2 million word data are needed to train the model. The initial corpus data can be supplemented with word data of the same data type as that in the training data set corresponding to the target model, such as conversation data, online text data, traditional book data, etc.

[0079] In this embodiment, a preset test data set is used to evaluate models trained with different data sets, so as to select the target model with the best performance and expand the initial corpus data according to the training data set corresponding to the target model.

[0080] In some optional implementations, deduplication functions are respectively set according to the first preset number of preset parameters, the initial corpus data is processed according to the set deduplication functions, and the word data is placed in a corresponding partition file according to the length of the word data in the processing result, including:

[0081] Determining a target setting parameter from a first preset number of preset parameters;

[0082] Use the target setting parameters to set the deduplication function and obtain the set deduplication function;

[0083] The initial corpus data is processed according to the set deduplication function to obtain the processing result;

[0084] Using the length processing function to perform word segmentation processing on the data in the processing result to obtain a fourth preset number of word data;

[0085] The partition file set corresponding to the target setting parameters is used as the target partition file set, and the word data is put into the corresponding partition file in the target partition file set according to the length of the word data;

[0086] Determine new target setting parameters from a first preset number of preset parameters, use the new target setting parameters as target setting parameters, and execute subsequent steps starting from setting a deduplication function using the target setting parameters until each preset parameter is used to set the target setting parameters.

[0087] Specifically, the first preset number of preset parameters are, for example, f=32, k=2; f=64, k=10; f=128, k=2; f=32, k=5, etc., and the target setting parameters are determined from the first preset number of preset parameters, for example, f=32, k=2 are used as the target setting parameters. The deduplication function is, for example, the simhash algorithm of python.

[0088] The number of digits f and the distance threshold k in the simhash algorithm are set as target setting parameters to obtain the set deduplication function. By setting different number of digits f and distance threshold k, deduplication functions with different deduplication strengths can be obtained. The initial corpus data is processed according to the set deduplication function to obtain the processing result.

[0089] The length processing function is, for example, a tokenizer, which is used to divide the data in the processing result into a single unit element token, and convert the token into a unique code recognizable by the model, namely word data. The length processing function is used to perform word segmentation processing on the data in the processing result to obtain a fourth preset number of word data, where the fourth preset number represents multiple, and no specific quantity limit is made here.

[0090] The partition file set corresponding to the target setting parameters is used as the target partition file set. For example, the partition file set is named according to the f and k parameter values ​​of simhash. The partition file set corresponding to the target setting parameters is the partition file set with the same name as the target setting parameters. According to the length of the word data, the word data is put into the corresponding partition file in the target partition file set. For example, the data with simhash parameters set to f=128, k=2, and a token length of 51 is written into the partition file 20-100.txt in the partition file set f_128_k_2.

[0091] Determine new target setting parameters from the first preset number of preset parameters, for example, f=64, k=10 as new target setting parameters. Use the new target setting parameters as target setting parameters, and perform subsequent steps starting from setting a deduplication function using the target setting parameters until each preset parameter is used to set the target setting parameters.

[0092] The above process is as follows Figure 3 As shown in the figure, read the original data; perform format filtering, sensitive word filtering and other processing; use different simhash parameters to remove duplicates and obtain multiple data sets; traverse each set and perform length processing; calculate the token length obtained after tokenizer segmentation; record data of different lengths into corresponding partition files.

[0093] In this embodiment, the first preset number of preset parameters are sequentially used as target setting parameters, and the target setting parameters are used to set the deduplication function to process the initial corpus data, and various deduplication intensity processing results are obtained. The word data of different length intervals in the processing results are classified and stored in different partition files, which is convenient for flexible matching of word data to form different training data sets, so as to make full use of the initial corpus data to build a high-quality training data set.

[0094] In some optional implementations, the data in the processing result is segmented using a length processing function to obtain a fourth preset number of word data, including:

[0095] Obtaining row data contained in the processing result, and deleting row data whose data length is less than a preset threshold, to obtain remaining row data;

[0096] The remaining line data is used as word data to obtain a fourth preset number of word data.

[0097] Specifically, length processing functions such as tokenizers are used to extract row data contained in the processing results. Length processing functions such as tokenizers are used to divide the data in the processing results into a separate unit of element tokens, and convert tokens into unique codes that can be recognized by the model, i.e., word data. For pile datasets, row data with a data length of less than 20 is generally considered to be low-quality data and is directly deleted. Other datasets need to be analyzed according to actual conditions. Therefore, a preset threshold value such as 20 or other values ​​that meet actual needs is used.

[0098] The length processing function is used to delete the row data whose data length is less than the preset threshold, and the remaining row data is obtained. The remaining row data is used as word data to obtain a fourth preset number of word data. The fourth preset number represents a plurality, and no specific number limitation is made here.

[0099] In some optional implementations, before putting the word data into the corresponding partition file, the method further includes:

[0100] Creating a first preset number of partition file sets, and generating a first identifier for each partition file set according to the first preset number of preset parameters;

[0101] Creating a partition file corresponding to each preset output file partition in each partition file set, and generating a second identifier for each partition file according to the preset output file partition;

[0102] Create a file directory containing the first identifier and the second identifier.

[0103] Specifically, a first preset number of partition file sets are created, and a first identifier of each partition file set is generated according to the first preset number of preset parameters, for example: the preset parameters corresponding to the partition file set are used as the first identifier of the partition file set, such as: f_128_k_2, f_32_k_2, f_32_k_5, f_64_k_10, etc.

[0104] Currently, the maximum text length of most models supports up to 32000, and a few support 64000 / 128000. In addition, the length of most open source datasets or self-collected datasets is basically within 1000. Therefore, when creating partition files, the unit of segmentation distance within 1000 is 100, and the unit of segmentation distance above 1000 is set to 1000. Therefore, the preset output file partitions are as follows: "20-100", "101-200", "201-300", "301-400", "401-500", "501-600", "601-700", "701-800", "801-900", "901-1000", "1001-2000", "2001-3000", ... "31001-32000". In each partition file set, a partition file corresponding to each preset output file partition is created, for example, a partition file corresponding to "20-100" is created to store word data with a data length between 20 and 100; a partition file corresponding to "101-200" is created to store word data with a data length between 101 and 200, and so on. A second identifier of each partition file is generated according to the preset output file partition, for example, the preset output file partition corresponding to the partition file is used as the first identifier of the partition file, such as: 20-100.txt, 101-200.txt, etc.

[0105] Create a file directory containing the first identifier and the second identifier, such as Figure 4As shown: the first identifier includes: f_128_k_2, f_32_k_2, f_32_k_5, f_64_k_10, the second identifier includes: 20-100.txt, ..., 31001-32000.txt, that is, the partition file set with the first identifier f_128_k_2 includes the partition files with the second identifiers 20-100.txt, ..., 31001-32000.txt respectively, and the remaining partition file sets refer to Figure 4 , I will not go into details here.

[0106] In this embodiment, a first identifier for each partition file set is generated according to a first preset number of preset parameters, a second identifier for each partition file is generated according to a preset output file partition, and a file directory containing the first identifier and the second identifier is created, so as to facilitate the management of the partition file set and the partition files, and to facilitate the classification of the word data in the deduplication results and save them in the corresponding partition files in the corresponding partition file set.

[0107] In some optional implementations, according to the model training requirements, a second preset number of target word data is obtained from the partition files included in each partition file set, and the target word data is aggregated to obtain a third preset number of training data sets, including:

[0108] Determine the first target length of word data according to model training requirements;

[0109] In each partition file set, the partition file corresponding to the first target length is used as the first target partition file, and the word data contained in the first target partition file is used as the target word data;

[0110] The target word data in each partition file set are respectively aggregated to obtain a third preset number of training data sets.

[0111] Specifically, the model training requirement is, for example, to use only data with a token length of less than 1000. According to the model training requirement, the first target length of the word data is determined, and the first target length is, for example, 1000.

[0112] In each partition file set, the partition file corresponding to the first target length is used as the first target partition file. For example, if the first target length is 1000, the partition file used to store the word vector whose length is less than the first target length is used as the first target partition file, such as the partition file with the second identifier 20-100.txt, ..., 900-1000.txt as the first target partition file. The word data contained in the first target partition file is used as the target word data. The target word data in each partition file set are summarized respectively to obtain a third preset number of training data sets. The third preset number is set according to actual needs and can be equal to the number of training data sets.

[0113] In this embodiment, according to the model training requirements, the first target length of the word data is determined, and the word data of the partition files corresponding to the first target length in each partition file set are aggregated to obtain multiple training data sets. According to the model training requirements, the word data can be flexibly matched to form different training data sets, and the initial corpus data can be fully utilized to build a high-quality training data set to meet the requirements of training different functional models and ensuring the training quality.

[0114] In some optional implementations, according to the model training requirements, a second preset number of target word data is obtained from the partition files included in each partition file set, and the target word data is aggregated to obtain a third preset number of training data sets, including:

[0115] According to the model training requirements, determine the second target length and target number of word data;

[0116] Determine the text length threshold supported by the initial model;

[0117] When the text length threshold is greater than the second target length, the length between the second target length and the text length threshold is used as the third target length;

[0118] In each partition file set, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file;

[0119] Determine a preset ratio, and determine a first quantity and a second quantity according to the preset ratio and the target quantity;

[0120] Selecting a first number of word data from the word data included in the second target partition file as target word data, and selecting a second number of word data from the word data included in the third target partition file as target word data;

[0121] The target word data in each partition file set are respectively aggregated to obtain a third preset number of training data sets.

[0122] Specifically, the model training requirements are as follows: using 10 million pieces of data to train the short text output capability, the text length requirement cannot exceed 100. According to the model training requirements, the second target length and target number of word data are determined, the second target length is 100, and the target number is 10 million.

[0123] Determine the text length threshold supported by the initial model, for example, if the maximum text length supported by the model is 4000, the text length threshold is 4000. If the text length threshold is greater than the second target length, the length between the second target length and the text length threshold is used as the third target length, for example, if the second target length is 100, the text length threshold is 4000, and the third target length is 100-4000.

[0124] In each partition file set, the partition file corresponding to the second target length is used as the second target partition file, for example, the partition file used to store word vectors with a length less than the second target length is used as the second target partition file, such as the partition file with the second identifier 20-100.txt, ..., 90-100.txt is used as the second target partition file. The partition file corresponding to the third target length is used as the third target partition file, for example, the partition file used to store word vectors with a length greater than or equal to the second target length and less than or equal to the text length threshold is used as the third target partition file, such as the partition file with the second identifier 100-200.txt, ..., 3001-4000.txt is used as the third target partition file.

[0125] Determine a preset ratio, for example, 80% and 20%. Determine a first quantity and a second quantity according to the preset ratio and the target quantity, for example, if the target quantity is 10 million, the first quantity is 10 million×80%=8 million, and the second quantity is 10 million×20%=2 million.

[0126] A first number of word data is selected from the word data included in the second target partition file as the target word data, and a second number of word data is selected from the word data included in the third target partition file as the target word data. For example, 8 million word data are selected from the second target partition file as the target word data, and then the remaining 2 million data are selected and evenly acquired from the word data included in the third target partition file. The acquisition method can be uniform acquisition or other methods that meet actual needs.

[0127] In this embodiment, a first number of word data is selected from the word data contained in the second target partition file as the target word data, and a second number of word data is selected from the word data contained in the third target partition file as the target word data, and the target word data in each partition file set is respectively summarized to obtain a third preset number of training data sets. According to the model training requirements, the word data is flexibly matched to form different training data sets, and the initial corpus data is fully utilized to construct a high-quality training data set to meet the requirements of training different functional models and ensuring the training quality.

[0128] In some optional implementations, another method for performing deduplication processing on initial corpus data includes steps A1 to A4.

[0129] Step A1: divide at least two corpus data with word overlap greater than a preset ratio threshold in the initial corpus data into one bucket, and obtain a plurality of bucket data subsets.

[0130] Specifically, the text overlap between any two corpus data in the initial corpus data is equal to the number of overlapping characters divided by the number of characters included in the corpus data with more characters in the two corpus data. The bucketing process performed in this way can ensure that a bucketed data subset includes the same text information, has strong text correlation, and has a high probability of including corpus data with the same semantics, so it is necessary to deduplicate the corpus data.

[0131] Step A2: for each bucket data subset, obtain multiple candidate data pairs.

[0132] Specifically, for each bucket data subset, any two corpus data in the corresponding data subset are combined into a pair of candidate data pairs, and a total of multiple pairs of candidate data pairs are obtained, which can effectively ensure the comprehensiveness of the obtained candidate data pairs.

[0133] Step A3: using a semantic similarity scoring model to predict the semantic similarity of the two corpus data in each candidate data pair.

[0134] Specifically, for each candidate data pair, two corpus data in the candidate data pair are input into the semantic similarity scoring model, and the semantic similarity scoring model can predict and output the semantic similarity between the two.

[0135] Step A4: based on the semantic similarity of each candidate data pair among the plurality of candidate data pairs, deduplicate the corpus data in the corresponding data subset.

[0136] Specifically, first, from multiple pairs of candidate data pairs, at least one pair of target data pairs with semantic similarity greater than or equal to a preset similarity threshold is obtained; then, at least two corpus data corresponding to at least one pair of target data pairs in the corresponding data subset are aggregated as duplicate corpus data to achieve deduplication processing. It is also possible to first remove candidate data pairs with semantic similarity less than a preset similarity threshold from multiple pairs of candidate data pairs, and the remaining ones are candidate data pairs that need to be deduplicated. Deduplication processing can be achieved by directly aggregating the corpus data of the remaining candidate data pairs. When specifically aggregating, a mapping table can be established to map all these duplicate corpus data to a target corpus data, so that the duplicate corpus data can be identified by a unified target corpus data when it is promoted and applied downstream, such as when a knowledge base is created. When multiple duplicate corpus data are aggregated, which target corpus data they are aggregated on can also be set according to actual needs. For example, if there is corpus data in the knowledge base among multiple duplicate corpus data, the corpus data in the knowledge base can be selected as the target corpus data. If there is no corpus data in the knowledge base, you can select the most concise corpus data, such as the one with fewer words and numbers, as the target corpus data. If there are multiple most concise corpus data, you can randomly select one. Or you can further refer to industry literature to obtain the one with the highest frequency. Or you can train a neural network model to select the most appropriate target corpus data from multiple repeated corpus data as the aggregated target corpus data.

[0137] In this embodiment, a semantic similarity scoring model is used to implement deduplication processing, which strengthens the learning of difficult negative sample data and enables the semantic similarity scoring model to improve the accuracy of identifying repeated and non-repeated corpora, thereby effectively improving the deduplication efficiency of corpus data.

[0138] In the present embodiment, a model training device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0139] This embodiment provides a model training device, such as Figure 5 As shown, including:

[0140] The processing module 501 is used to set deduplication functions according to the first preset number of preset parameters, process the initial corpus data according to the set deduplication functions, and put the word data into corresponding partition files according to the length of the word data in the processing result, wherein the partition file is used to record the word data of the corresponding length, the partition file is included in the first preset number of partition file sets, and the partition file set corresponds to the preset parameters;

[0141] A summarizing module 502 is used to obtain a second preset number of target word data from the partition files included in each partition file set according to model training requirements, and summarize the target word data to obtain a third preset number of training data sets;

[0142] The training module 503 is used to train the initial models using the training data sets respectively to obtain a third preset number of trained models.

[0143] In some optional implementations, after the initial models are trained using the training data sets to obtain a third preset number of trained models, the device is used to:

[0144] Using a preset test data set to test a third preset number of trained models, and determining a test score corresponding to the trained models according to the test results;

[0145] The trained model with the highest test score is used as the target model, and the training dataset corresponding to the target model is saved.

[0146] In some optional implementations, the processing module 501 sets deduplication functions according to the first preset number of preset parameters, processes the initial corpus data according to the set deduplication functions, and puts the word data into corresponding partition files according to the length of the word data in the processing result, including:

[0147] Determining a target setting parameter from a first preset number of preset parameters;

[0148] Use the target setting parameters to set the deduplication function and obtain the set deduplication function;

[0149] The initial corpus data is processed according to the set deduplication function to obtain the processing result;

[0150] Using the length processing function to perform word segmentation processing on the data in the processing result to obtain a fourth preset number of word data;

[0151] The partition file set corresponding to the target setting parameters is used as the target partition file set, and the word data is put into the corresponding partition file in the target partition file set according to the length of the word data;

[0152] Determine new target setting parameters from a first preset number of preset parameters, use the new target setting parameters as target setting parameters, and execute subsequent steps starting from setting a deduplication function using the target setting parameters until each preset parameter is used to set the target setting parameters.

[0153] In some optional implementations, the processing module 501 performs word segmentation processing on the data in the processing result using a length processing function to obtain a fourth preset number of word data, including:

[0154] Obtaining row data contained in the processing result, and deleting row data whose data length is less than a preset threshold, to obtain remaining row data;

[0155] The remaining line data is used as word data to obtain a fourth preset number of word data.

[0156] In some optional implementations, before putting the word data into the corresponding partition file, the device is further used to:

[0157] Creating a first preset number of partition file sets, and generating a first identifier for each partition file set according to the first preset number of preset parameters;

[0158] Creating a partition file corresponding to each preset output file partition in each partition file set, and generating a second identifier for each partition file according to the preset output file partition;

[0159] Create a file directory containing the first identifier and the second identifier.

[0160] In some optional implementations, the aggregation module 502 obtains a second preset number of target word data from the partition files included in each partition file set according to the model training requirements, and aggregates the target word data to obtain a third preset number of training data sets, including:

[0161] Determine the first target length of word data according to model training requirements;

[0162] In each partition file set, the partition file corresponding to the first target length is used as the first target partition file, and the word data contained in the first target partition file is used as the target word data;

[0163] The target word data in each partition file set are respectively aggregated to obtain a third preset number of training data sets.

[0164] In some optional implementations, the aggregation module 502 obtains a second preset number of target word data from the partition files included in each partition file set according to the model training requirements, and aggregates the target word data to obtain a third preset number of training data sets, including:

[0165] According to the model training requirements, determine the second target length and target number of word data;

[0166] Determine the text length threshold supported by the initial model;

[0167] When the text length threshold is greater than the second target length, the length between the second target length and the text length threshold is used as the third target length;

[0168] In each partition file set, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file;

[0169] Determine a preset ratio, and determine a first quantity and a second quantity according to the preset ratio and the target quantity;

[0170] Selecting a first number of word data from the word data included in the second target partition file as target word data, and selecting a second number of word data from the word data included in the third target partition file as target word data;

[0171] The target word data in each partition file set are respectively aggregated to obtain a third preset number of training data sets.

[0172] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0173] The model training device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0174] The embodiment of the present invention also provides a computer device having the above Figure 5 The model training apparatus shown.

[0175] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0176] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0177] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0178] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0180] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0181] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0182] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0183] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined in this application.

Claims

1. A model training method, characterized in that: The method comprises: Deduplication functions are respectively set according to a first preset number of preset parameters, initial corpus data is processed according to the set deduplication functions, and word data is placed into corresponding partition files according to the length of word data in the processing result, wherein the partition files are used to record word data of corresponding lengths, the partition files are included in a first preset number of partition file sets, and the partition file sets correspond to the preset parameters; According to the model training requirements, a second preset number of target word data are obtained from the partition files included in each of the partition file sets, and the target word data are aggregated to obtain a third preset number of training data sets; The initial models are trained using the training data sets respectively to obtain a third preset number of trained models.

2. The method according to claim 1, characterized in that After the initial models are trained using the training data sets to obtain a third preset number of trained models, the method further includes: Using a preset test data set to test the third preset number of trained models, and determining a test score corresponding to the trained models according to the test results; The trained model with the highest test score is used as the target model, and the training data set corresponding to the target model is saved.

3. The method according to claim 1, characterized in that The step of setting a deduplication function according to the first preset number of preset parameters, processing the initial corpus data according to the set deduplication function, and placing the word data into a corresponding partition file according to the length of the word data in the processing result includes: Determining a target setting parameter from the first preset number of preset parameters; Using the target setting parameters, setting the deduplication function to obtain the set deduplication function; Processing the initial corpus data according to the set deduplication function to obtain the processing result; Using a length processing function to perform word segmentation processing on the data in the processing result to obtain a fourth preset number of word data; Taking the partition file set corresponding to the target setting parameter as the target partition file set, and putting the word data into the corresponding partition file in the target partition file set according to the length of the word data; Determine a new target setting parameter from the first preset number of preset parameters, use the new target setting parameter as the target setting parameter, and execute subsequent steps starting from setting the deduplication function using the target setting parameter until each of the preset parameters is used to set the target setting parameter.

4. The method according to claim 3, characterized in that The using of the length processing function to perform word segmentation processing on the data in the processing result to obtain a fourth preset number of word data includes: Obtaining the row data contained in the processing result, and deleting the row data whose data length is less than a preset threshold, to obtain the remaining row data; The remaining row data is used as the word data to obtain a fourth preset number of the word data.

5. The method according to claim 1, characterized in that: Before putting the word data into the corresponding partition file, the method further includes: Creating a first preset number of partition file sets, and generating a first identifier for each of the partition file sets according to the first preset number of preset parameters; Creating a partition file corresponding to each preset output file partition in each partition file set, and generating a second identifier for each partition file according to the preset output file partition; Create a file directory containing the first identifier and the second identifier.

6. The method according to claim 1, characterized in that According to the model training requirements, obtaining a second preset number of target word data from the partition files included in each of the partition file sets, and aggregating the target word data to obtain a third preset number of training data sets, including: Determining a first target length of word data according to the model training requirements; In each of the partition file sets, taking the partition file corresponding to the first target length as the first target partition file, and taking the word data contained in the first target partition file as the target word data; The target word data in each of the partition file sets are respectively aggregated to obtain a third preset number of the training data sets.

7. The method according to claim 1, characterized in that According to the model training requirements, obtaining a second preset number of target word data from the partition files included in each of the partition file sets, and aggregating the target word data to obtain a third preset number of training data sets, including: Determining a second target length and a target number of word data according to the model training requirements; Determining a text length threshold supported by the initial model; When the text length threshold is greater than the second target length, the length between the second target length and the text length threshold is used as the third target length; In each of the partition file sets, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file; Determine a preset ratio, and determine a first quantity and a second quantity according to the preset ratio and the target quantity; Selecting a first number of word data from the word data included in the second target partition file as target word data, and selecting a second number of word data from the word data included in the third target partition file as target word data; The target word data in each of the partition file sets are respectively aggregated to obtain a third preset number of the training data sets.

8. A model training device, characterized in that: The device comprises: A processing module, used to set deduplication functions according to a first preset number of preset parameters, respectively, process the initial corpus data according to the set deduplication functions, and put the word data into corresponding partition files according to the length of the word data in the processing result, wherein the partition file is used to record word data of corresponding length, the partition file is included in a first preset number of partition file sets, and the partition file set corresponds to the preset parameters; A summarizing module, used for obtaining a second preset number of target word data from the partition file according to model training requirements, and summarizing the target word data to obtain a third preset number of training data sets; The training module is used to train the initial model using the training data set to obtain a third preset number of trained models.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model training method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the model training method described in any one of claims 1 to 7.

Images