Model training method and device, computer device, and storage medium

Abstract

The application relates to the technical field of computers and discloses a model training method and device, a computer device and a storage medium, the method comprising the following steps: setting a deduplication function according to a first preset number of preset parameters, processing initial corpus data according to the set deduplication function, and putting word data into corresponding partition files according to the length of the word data in the processing result, wherein the partition files are used for recording word data with corresponding lengths; obtaining a second preset number of target word data from the partition files contained in each partition file set according to a model training requirement, and obtaining a third preset number of training data sets by summarizing the target word data; and training an initial model by using the training data sets respectively to obtain a third preset number of trained models. The application solves the problem that using a unified processing scheme to process various text corpus can only obtain a single training data set, and different function models cannot be trained and the training quality cannot be ensured.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a model training method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Currently, when dealing with diverse and large-scale text corpora for training Large Language Models (LLMs), existing experience often leads to the use of fixed deduplication and length filtering schemes to process the text corpora, forming a training dataset, which is then used to train the model. For example, if the model supports a maximum text length of 32,000 characters, Python's built-in unordered set function can be used to deduplicate the text corpora, and then data exceeding 32,000 characters in length can be deleted to form a training dataset.

[0003] However, because text corpora from different sources have varying impacts on model training, the training dataset generated by the above approach only performs preliminary screening and cannot select the optimal training dataset from the text corpus. Therefore, the quality of the trained model cannot be guaranteed to be optimal, and may even lead to lower-than-expected scores. Furthermore, using a uniform processing method cannot train models with different functionalities. For example, if a large amount of dialogue text data is collected, and the current model only supports a maximum input length of 8000 characters, then according to current technology, preprocessing will only leave data shorter than 8000 characters. When subsequent models support text lengths of 16000 or 32000 characters, reprocessing is required, increasing time costs.

[0004] Therefore, the related technologies have the problem that using a uniform processing scheme to process various text corpora can only produce a single training dataset, making it impossible to train different functional models and guarantee training quality. Summary of the Invention

[0005] In view of this, the present invention provides a model training method, apparatus, computer equipment and storage medium to solve the problem that using a unified processing scheme to process various text corpora can only produce a single training dataset, and cannot train different functional models and ensure training quality.

[0006] In a first aspect, the present invention provides a model training method, comprising:

[0007] Deduplication functions are set according to a first preset number of preset parameters. The initial corpus data is processed according to the set deduplication functions. The word data is put into the corresponding partition file according to the length of the word data in the processing result. The partition file is used to record word data of the corresponding length. The partition file is contained in a first preset number of partition file sets, and the partition file sets correspond to the preset parameters.

[0008] Based on the model training requirements, a second preset number of target word data are obtained from the partition files contained in each partition file set, and the target word data are summarized to obtain a third preset number of training datasets.

[0009] The initial model is trained using the training dataset to obtain a third preset number of trained models.

[0010] The model training method provided in this embodiment sets different preset parameters for the deduplication function and processes the initial corpus data according to the set deduplication function to obtain processing results with various deduplication strengths. Word data of different length ranges in the processing results are classified and stored in different partition files. According to the model training requirements, the word data is flexibly allocated, making full use of the initial corpus data to construct a high-quality training dataset. This allows for the generation of different training datasets from a single initial corpus data, enabling the training of different functional models using different training datasets, improving training quality, and to some extent reducing the requirement for a large number of word data points. This solves the problem that using a uniform processing scheme to process various text corpora can only produce a single training dataset, making it impossible to train different functional models and guarantee training quality.

[0011] In some alternative implementations, after training the initial model using the training dataset to obtain a third preset number of trained models, the method further includes:

[0012] The third preset number of trained models are tested using a preset test dataset, and the test score corresponding to the trained model is determined based on the test results.

[0013] The model with the highest test score is selected as the target model, and the corresponding training dataset is saved.

[0014] In this embodiment, a preset test dataset is used to evaluate models trained on different datasets, which facilitates the selection of the best-performing target model and the expansion of the initial corpus data based on the training dataset corresponding to the target model.

[0015] In some optional implementations, deduplication functions are set according to a first preset number of preset parameters, the initial corpus data is processed according to the set deduplication functions, and the word data is placed into the corresponding partition file according to the length of the word data in the processing result, including:

[0016] The target setting parameter is determined from the first preset number of preset parameters;

[0017] By setting the target parameters, the deduplication function is set, and the deduplication function after setting is obtained;

[0018] The initial corpus data is processed according to the set deduplication function to obtain the processing result;

[0019] The length processing function is used to segment the data in the processing result to obtain a fourth preset number of words.

[0020] Use the partition file set corresponding to the target setting parameters as the target partition file set, and put the word data into the corresponding partition file in the target partition file set according to the length of the word data;

[0021] A new target setting parameter is determined from the first preset number of preset parameters. The new target setting parameter is used as the target setting parameter. The subsequent steps are executed starting from setting the deduplication function using the target setting parameter, until each preset parameter is used to set the target setting parameter.

[0022] In this embodiment, a first preset number of preset parameters are sequentially used as target setting parameters, and the initial corpus data is processed using a deduplication function set by the target setting parameters to obtain processing results with various deduplication strengths. Word data of different length ranges in the processing results are classified and stored in different partition files, facilitating flexible allocation of word data to form different training datasets, thus fully utilizing the initial corpus data to construct a high-quality training dataset.

[0023] In some optional implementations, a length processing function is used to segment the data in the processing result to obtain a fourth preset number of word data, including:

[0024] Retrieve the rows of data contained in the processing result, and delete the rows of data whose length is less than a preset threshold to obtain the remaining rows of data;

[0025] The remaining rows of data are used as word data to obtain a fourth preset number of word data.

[0026] In some alternative implementations, the method further includes, before placing the word data into the corresponding partition file:

[0027] Create a first preset number of partition file sets, and generate a first identifier for each partition file set based on the first preset number of preset parameters;

[0028] Create a partition file corresponding to each preset output file partition in each partition file set, and generate a second identifier for each partition file based on the preset output file partition;

[0029] Create a file directory containing the first identifier and the second identifier.

[0030] In this embodiment, a first identifier for each partition file set is generated according to a first preset number of preset parameters, and a second identifier for each partition file is generated according to a preset output file partition. A file directory containing the first identifier and the second identifier is created to facilitate the management of the partition file sets and partition files. Furthermore, it facilitates the classification of word data in the deduplication results and saves them to the corresponding partition files in the corresponding partition file sets.

[0031] In some optional implementations, based on model training requirements, a second preset number of target word data points are obtained from the partition files contained in each partition file set, and the target word data are aggregated to obtain a third preset number of training datasets, including:

[0032] Determine the first target length of the word data based on the model training requirements;

[0033] In each set of partitioned files, the partition file corresponding to the first target length is taken as the first target partition file, and the word data contained in the first target partition file is taken as the target word data;

[0034] The target word data in each partition file set is summarized to obtain a third preset number of training datasets.

[0035] In this implementation, based on the model training requirements, a first target length for the word data is determined. The word data from the partition files corresponding to the first target length in each partition fileset are then aggregated to obtain multiple training datasets. This allows for flexible allocation of word data according to model training needs, forming different training datasets. It fully utilizes the initial corpus data to construct high-quality training datasets, meeting the requirements for training different functional models while ensuring training quality.

[0036] In some optional implementations, based on model training requirements, a second preset number of target word data points are obtained from the partition files contained in each partition file set, and the target word data are aggregated to obtain a third preset number of training datasets, including:

[0037] Based on the model training requirements, determine the length and number of the second target in the word data;

[0038] Determine the text length threshold supported by the initial model;

[0039] If the text length threshold is greater than the second target length, the length between the second target length and the text length threshold shall be used as the third target length.

[0040] In each set of partition files, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file;

[0041] Determine the preset ratio, and based on the preset ratio and the target quantity, determine the first quantity and the second quantity;

[0042] Select a first number of words from the word data contained in the second target partition file as target word data, and select a second number of words from the word data contained in the third target partition file as target word data;

[0043] The target word data in each partition file set is summarized to obtain a third preset number of training datasets.

[0044] In this embodiment, a first number of words are selected as target words from the word data contained in the second target partition file, and a second number of words are selected as target words from the word data contained in the third target partition file. The target word data in each partition file set are then aggregated to obtain a third preset number of training datasets. This allows for flexible allocation of word data according to model training requirements, forming different training datasets. It fully utilizes the initial corpus data to construct high-quality training datasets, meeting the needs of training different functional models and ensuring training quality.

[0045] Secondly, the present invention provides a model training apparatus, comprising:

[0046] The processing module is used to set deduplication functions according to a first preset number of preset parameters, process the initial corpus data according to the set deduplication functions, and put the word data into the corresponding partition file according to the length of the word data in the processing result. The partition file is used to record word data of the corresponding length. The partition file is contained in a first preset number of partition file sets, and the partition file sets correspond to the preset parameters.

[0047] The aggregation module is used to obtain a second preset number of target word data from the partition files contained in each partition file set according to the model training requirements, and aggregate the target word data to obtain a third preset number of training datasets.

[0048] The training module is used to train the initial model using the training dataset to obtain a third preset number of trained models.

[0049] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the model training method of the first aspect or any corresponding embodiment described above.

[0050] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the model training method of the first aspect or any corresponding embodiment thereof.

[0051] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the model training method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the current text processing and model training process according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic flowchart of a model training method according to an embodiment of the present invention;

[0055] Figure 3 This is a flowchart of a method for preprocessing large language model datasets according to an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of a file directory according to an embodiment of the present invention;

[0057] Figure 5 This is a structural block diagram of a model training device according to an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The current text processing and model training process is as follows: Figure 1The process includes: preparing raw data by acquiring open-source datasets or collecting custom data; deduplication and filtering using Python's unordered set (Set) method or the simhash algorithm to remove duplicate data; filtering the deduplicated data based on the model's maximum supported text length (e.g., if the model supports a maximum text length of 8000 characters, data exceeding 8000 characters must be removed); `Set` is an unordered collection in Python, where each element is unique (no duplicates) and immutable; the simhash algorithm reduces the dimensionality of the text, generating a simhash value, and comparing the distances between different texts' simhash values ​​to determine their similarity; preparing a pre-training dataset; converting the raw data into a file recognizable by the model for training; and evaluating the model's quality using a test dataset to determine its score.

[0061] The above approach suffers from several drawbacks, including the inability to obtain optimal quality datasets and a lack of flexibility. It uses existing processing strategies to preprocess text corpora from different sources to obtain a training dataset. However, training a model using this dataset is unlikely to yield optimal results. For example, when training a large language model's article writing capabilities, using datasets from traditional books and online articles presents challenges. Traditional book data is concise and of high quality, while online article data often contains low-quality, low-quality text. Using the same preprocessing approach for both types of data negatively impacts the quality of the final dataset. Furthermore, a uniform preprocessing strategy cannot train models with different functionalities. For instance, if a large amount of dialogue text data is collected, and the current model's maximum input length is only 8000 characters, preprocessing with current technology will only leave data shorter than 8000 characters. When subsequent models support maximum text lengths of 16000 or 32000 characters, the original data must be reprocessed to generate a new training dataset, increasing time costs. A pre-training dataset is generated from a single set of raw data. However, it is impossible to perform detailed internal allocation of the raw data. For example, in a dialogue dataset, there are data of various lengths, and it is impossible to configure the proportion of data of each length, which is not flexible enough.

[0062] Based on the above, this invention provides a model training method that uses the simhash algorithm to obtain data with varying deduplication levels by configuring different parameters. The length of the deduplicated data is calculated, and then data within different length ranges is classified and stored in different files. The data is flexibly allocated to form different training datasets. Models trained on different training datasets are evaluated, and the model with the highest score and its corresponding training dataset are selected. The method selects the best from the best based on the original data by manually controlling diversity (using different sets obtained using different simhash parameters) and partitioning the data according to token length. Finally, multiple different training datasets are generated, and the best model is trained using these datasets, determining the optimal training dataset. This method can fully utilize the original data to construct high-quality pre-trained datasets and pre-trained models, and to some extent reduces the requirement for a large amount of pre-trained data. It achieves the effect of generating different datasets from a single set of original data, using different datasets to pre-train the model, and then evaluating the trained model to obtain the pre-trained dataset that yields the best model.

[0063] According to an embodiment of the present invention, a model training embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer device with data processing capabilities, such as a computer, server, etc. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0064] This embodiment provides a model training method that can be used with the aforementioned computer equipment. Figure 2 This is a flowchart of a model training method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0065] Step S201: Set deduplication functions according to the first preset number of preset parameters, process the initial corpus data according to the set deduplication functions, and put the word data into the corresponding partition file according to the length of the word data in the processing result. The partition file is used to record word data of the corresponding length. The partition file is contained in the first preset number of partition file sets, and the partition file sets correspond to the preset parameters.

[0066] Specifically, a diverse and large-scale text corpus for training large language models is obtained as the initial corpus data, such as text samples from the Pile dataset, which is a diverse and large-scale text corpus used for training large language models. The deduplication function is, for example, Python's SimHash algorithm, with preset parameters of bit length *f* and distance threshold *k*. The first preset number of preset parameters is as follows: *f* = 32, *k* = 2; *f* = 64, *k* = 10; *f* = 128, *k* = 2; *f* = 32, *k* = 5, etc. The *k* and *f* values ​​mentioned above are examples for the Pile dataset; for other datasets, the values ​​of *k* and *f* can be adjusted according to the actual content of the dataset. By setting different preset parameters for the deduplication function, different levels of deduplication are applied to each initial corpus data. The bit length *f* can be 32 / 64 / 128, with higher bit lengths providing higher accuracy. The distance threshold *k* indicates that when searching for similar SimHash values, a maximum of *k* bits of difference are allowed; higher values ​​allow for more differences. The first preset number can be multiple, but no specific limit is set here. Based on a first preset number of preset parameters, deduplication functions (number of bits f and distance threshold k) are set respectively. The initial corpus data is then deduplicated using these functions to obtain the processing results. It should be noted that, based on the characteristics of simhash, for long texts (length greater than 1000 characters), settings such as f=32, k=2, f=64, k=10 are used; for short texts, settings such as f=128, k=2, f=32, k=5 are used. That is, different combinations of f and k parameters are used to perform deduplication processing on the original data with varying degrees of intensity, ultimately resulting in multiple datasets. Large language models refer to models that learn the syntax and semantics of natural language through training on large amounts of text data, thereby generating language models that conform to linguistic rules. The training dataset generated in this embodiment can be used to train various large language models and is not limited to any particular large language model. In addition, when obtaining diverse large-scale text corpora for training large language models, the large-scale diverse text corpora can be generated using the already trained large language model A as the initial corpus data. Subsequently, the initial corpus data is processed using steps S101 and S102, and the obtained training dataset is used to train the large language model B.

[0067] The partition files are contained within a first preset number of partition file sets. These partition file sets correspond to preset parameters and can be named using those parameters, such as f_128_k_2, f_64_k_5, etc. The partition files include "20-100", "101-200", and "201-300", respectively used to store word data with lengths between 20-100, 101-200, and 201-300. Each line in the processing result is treated as a word data entry. Based on the length of the word data in the processing result, the word data is placed into the corresponding partition file. For example, data with simhash parameters set to f=128, k=2, and a token length of 51 is written to the partition file 20-100.txt within the partition file set f_128_k_2. A token is a single unit of element, i.e., word data, obtained after word segmentation of the processing result.

[0068] Step S202: According to the model training requirements, obtain a second preset number of target word data from the partition files contained in each partition file set, and summarize the target word data to obtain a third preset number of training datasets.

[0069] Specifically, model training requirements may include, for example, using only data with token lengths under 1000, and using 10 million data points to train short text output capabilities with a text length not exceeding 100. Based on these requirements, a second preset number of target word data points are obtained from the partition files contained in each partition file set. These target word data points are then aggregated to obtain a third preset number of training datasets. For example, if only data with token lengths under 1000 is used, then only the word data in .txt files from 20-100 to 901-1000 contained in each partition file set are used as target word data, and the target word data is aggregated to obtain a third preset number of training datasets. The third preset number is set according to actual needs and can be equal to the number of training datasets.

[0070] It's important to note that the deduplication scheme and the method of partitioning data based on token length can be used for fine-tuning data processing. For example, the few-shot dataset can be freely allocated using the partitioned files. Without this scheme, when building multiple shot datasets, such as 20-shot and 30-shot datasets, the length of the original data needs to be calculated each time (otherwise, the data length would exceed the maximum text length supported by the model), resulting in significant time waste. Using the method of this invention, the data is already stored in different partitioned files according to different lengths during the initial processing of the original data. Therefore, when building datasets for different shots, only one addition or subtraction calculation is needed to determine how many records are needed for each partition, and then the data can be directly concatenated. This avoids the process of sending the data to the tokenizer to calculate the token length, greatly reducing the construction time of the few-shot dataset. Furthermore, the more few-shot datasets that need to be built, the more significant the time cost reduction becomes. Here, "few-shot" refers to a small number of samples given to the model during inference, but weight updates are not allowed. For a typical dataset, few-shots have context and examples (e.g., an English sentence and its French translation), while shot is the sample data.

[0071] Step S203: Train the initial model using the training dataset to obtain a third preset number of trained models.

[0072] Specifically, the initial model is trained using the training dataset to obtain a third predetermined number of trained models, where the initial model is the large language model to be trained. Furthermore, the models trained using different training datasets can be evaluated, and the model with the highest score and its corresponding training dataset can be selected.

[0073] The model training method provided in this embodiment sets different preset parameters for the deduplication function, processes the initial corpus data according to the set deduplication function, and obtains processing results with various deduplication strengths. Word data of different length ranges in the processing results are classified and stored in different partition files. According to the model training requirements, the data is flexibly allocated, making full use of the initial corpus data to construct a high-quality training dataset. This allows for the generation of different training datasets from a single initial corpus data, enabling the training of different functional models using different training datasets, improving training quality, and to some extent reducing the requirement for a large number of word data. This solves the problem that using a uniform processing scheme to process various text corpora can only produce a single training dataset, making it impossible to train different functional models and guarantee training quality.

[0074] In some alternative implementations, after training the initial model using the training dataset to obtain a third preset number of trained models, the method further includes:

[0075] The third preset number of trained models are tested using a preset test dataset, and the test score corresponding to the trained model is determined based on the test results.

[0076] The model with the highest test score is selected as the target model, and the corresponding training dataset is saved.

[0077] Specifically, after training the initial model using the training dataset to obtain a third preset number of trained models, a preset test dataset can be used to evaluate the models trained on different training datasets. The test score corresponding to the trained model is determined based on the test results. The preset test dataset, for example, consists of the model's input data and reference output data selected in advance according to the model's functionality. The test score of the model is obtained by using the similarity between the model's output and the reference output during the testing process.

[0078] The model with the highest score is the optimal model. This highest-scoring trained model is used as the target model, and the corresponding training dataset is saved. The initial corpus can be expanded using the target model's training dataset. For example, if the initial corpus only contains 1 million words suitable for training, but 2 million words are needed to ensure effective training, word data of the same type as those in the target model's training dataset can be added to the initial corpus, such as dialogue data, online articles, or traditional books.

[0079] In this embodiment, a preset test dataset is used to evaluate models trained on different datasets, which facilitates the selection of the best-performing target model and the expansion of the initial corpus data based on the training dataset corresponding to the target model.

[0080] In some optional implementations, deduplication functions are set according to a first preset number of preset parameters, the initial corpus data is processed according to the set deduplication functions, and the word data is placed into the corresponding partition file according to the length of the word data in the processing result, including:

[0081] The target setting parameter is determined from the first preset number of preset parameters;

[0082] By setting the target parameters, the deduplication function is set, and the deduplication function after setting is obtained;

[0083] The initial corpus data is processed according to the set deduplication function to obtain the processing result;

[0084] The length processing function is used to segment the data in the processing result to obtain a fourth preset number of words.

[0085] Use the partition file set corresponding to the target setting parameters as the target partition file set, and put the word data into the corresponding partition file in the target partition file set according to the length of the word data;

[0086] A new target setting parameter is determined from the first preset number of preset parameters. The new target setting parameter is used as the target setting parameter. The subsequent steps are executed starting from setting the deduplication function using the target setting parameter, until each preset parameter is used to set the target setting parameter.

[0087] Specifically, a first preset number of preset parameters are used, for example: f=32, k=2; f=64, k=10; f=128, k=2; f=32, k=5, etc. The target setting parameter is determined from this first preset number of preset parameters; for example, f=32, k=2 is used as the target setting parameter. The deduplication function is, for example, Python's simhash algorithm.

[0088] By setting the number of bits *f* and the distance threshold *k* in the simhash algorithm as target parameters, a deduplication function with varying deduplication strengths can be obtained. The initial corpus data is then processed using this deduplication function to obtain the final result.

[0089] Length processing functions, such as the tokenizer, are used to divide the processed data into individual unit elements (tokens) and convert each token into a unique code that the model can recognize—that is, word data. By using the length processing function to segment the processed data, a fourth preset number of words are obtained. This fourth preset number represents multiple words; no specific limit is set here.

[0090] The partition file set corresponding to the target setting parameters is used as the target partition file set. For example, if the partition file set is named according to the f and k parameter values ​​of simhash, the partition file set corresponding to the target setting parameters will have the same name as the target setting parameters. Based on the length of the word data, the word data is placed into the corresponding partition file in the target partition file set. For example, if the simhash parameters are set to f=128, k=2, and the token length is 51, the data will be written to the partition file 20-100.txt in the partition file set f_128_k_2.

[0091] A new target setting parameter is determined from a first set number of preset parameters. For example, f=64 and k=10 are used as the new target setting parameter. The new target setting parameter is then used as the target setting parameter, and subsequent steps are executed starting from setting the deduplication function using the target setting parameter, until each preset parameter has been used to set the target setting parameter.

[0092] The above process is as follows Figure 3 As shown, the process involves reading the raw data; performing format filtering and sensitive word filtering; deduplicating data using different simhash parameters to obtain multiple data sets; iterating through each set and processing its length; calculating the length of the token obtained after tokenizer segmentation; and recording data of different lengths into the corresponding partition files.

[0093] In this embodiment, a first preset number of preset parameters are sequentially used as target setting parameters, and the initial corpus data is processed using a deduplication function set by the target setting parameters to obtain processing results with various deduplication strengths. Word data of different length ranges in the processing results are classified and stored in different partition files, facilitating flexible allocation of word data to form different training datasets, thus fully utilizing the initial corpus data to construct a high-quality training dataset.

[0094] In some optional implementations, a length processing function is used to segment the data in the processing result to obtain a fourth preset number of word data, including:

[0095] Retrieve the rows of data contained in the processing result, and delete the rows of data whose length is less than a preset threshold to obtain the remaining rows of data;

[0096] The remaining rows of data are used as word data to obtain a fourth preset number of word data.

[0097] Specifically, length processing functions, such as the tokenizer, are used to extract line data from the processed results. The tokenizer divides the data into individual token units and converts each token into a unique code that the model can recognize—a word. For the pile dataset, lines shorter than 20 are generally considered low-quality data and are deleted. Other datasets require analysis based on specific needs; therefore, a preset threshold of 20 or other values ​​that meet practical requirements is used.

[0098] The length processing function is used to delete rows of data whose length is less than a preset threshold, resulting in the remaining rows of data. These remaining rows are then used as word data to obtain a fourth preset number of word data. The fourth preset number represents multiple terms; no specific limit is set here.

[0099] In some alternative implementations, the method further includes, before placing the word data into the corresponding partition file:

[0100] Create a first preset number of partition file sets, and generate a first identifier for each partition file set based on the first preset number of preset parameters;

[0101] Create a partition file corresponding to each preset output file partition in each partition file set, and generate a second identifier for each partition file based on the preset output file partition;

[0102] Create a file directory containing the first identifier and the second identifier.

[0103] Specifically, a first preset number of partition file sets are created, and a first identifier for each partition file set is generated based on the first preset number of preset parameters. For example, the preset parameters corresponding to the partition file set are used as the first identifier of the partition file set, such as f_128_k_2, f_32_k_2, f_32_k_5, f_64_k_10, etc.

[0104] Most current models support a maximum text length of 32,000 characters, with a few supporting 64,000 / 128,000 characters. Furthermore, most open-source or self-collected datasets have a length of less than 1,000 characters. Therefore, when creating partitioned files, the unit of distance for partitions within 1,000 characters is 100, and the unit for partitions greater than 1,000 characters is 1,000. Thus, the default output file partitions are as follows: "20-100", "101-200", "201-300", "301-400", "401-500", "501-600", "601-700", "701-800", "801-900", "901-1000", "1001-2000", "2001-3000", ... "31001-32000". Create a partition file corresponding to each preset output file partition in each partition file set. For example, create a partition file corresponding to "20-100" to store word data with a length between 20 and 100; create a partition file corresponding to "101-200" to store word data with a length between 101 and 200, and so on. Generate a second identifier for each partition file based on the preset output file partition. For example, use the preset output file partition corresponding to the partition file as the first identifier for that partition file, such as 20-100.txt, 101-200.txt, etc.

[0105] Create a file directory containing a first identifier and a second identifier, such as Figure 4As shown: The first identifier includes: f_128_k_2, f_32_k_2, f_32_k_5, f_64_k_10; the second identifier includes: 20-100.txt, ..., 31001-32000.txt. That is, the partition file set with the first identifier f_128_k_2 includes the partition files with the second identifiers 20-100.txt, ..., 31001-32000.txt. See [link to other partition file sets] for details. Figure 4 This will not be elaborated upon here.

[0106] In this embodiment, a first identifier for each partition file set is generated according to a first preset number of preset parameters, and a second identifier for each partition file is generated according to a preset output file partition. A file directory containing the first identifier and the second identifier is created to facilitate the management of the partition file sets and partition files. Furthermore, it facilitates the classification of word data in the deduplication results and saves them to the corresponding partition files in the corresponding partition file sets.

[0107] In some optional implementations, based on model training requirements, a second preset number of target word data points are obtained from the partition files contained in each partition file set, and the target word data are aggregated to obtain a third preset number of training datasets, including:

[0108] Determine the first target length of the word data based on the model training requirements;

[0109] In each set of partitioned files, the partition file corresponding to the first target length is taken as the first target partition file, and the word data contained in the first target partition file is taken as the target word data;

[0110] The target word data in each partition file set is summarized to obtain a third preset number of training datasets.

[0111] Specifically, the model training requirements include, for example, using only data with token lengths of 1000 or less. Based on these requirements, a first target length for the word data is determined, for example, 1000.

[0112] In each partition file set, the partition file corresponding to the first target length is designated as the first target partition file. For example, if the first target length is 1000, the partition file used to store word vectors shorter than the first target length is designated as the first target partition file, such as the partition files identified as 20-100.txt, ..., 900-1000.txt. The word data contained in the first target partition file is designated as the target word data. The target word data in each partition file set are then aggregated to obtain a third preset number of training datasets. This third preset number is set according to actual needs and can be equal to the number of training datasets.

[0113] In this implementation, based on the model training requirements, a first target length for the word data is determined. The word data from the partition files corresponding to the first target length in each partition fileset are then aggregated to obtain multiple training datasets. This allows for flexible allocation of word data according to model training needs, forming different training datasets. It fully utilizes the initial corpus data to construct high-quality training datasets, meeting the requirements for training different functional models while ensuring training quality.

[0114] In some optional implementations, based on model training requirements, a second preset number of target word data points are obtained from the partition files contained in each partition file set, and the target word data are aggregated to obtain a third preset number of training datasets, including:

[0115] Based on the model training requirements, determine the length and number of the second target in the word data;

[0116] Determine the text length threshold supported by the initial model;

[0117] If the text length threshold is greater than the second target length, the length between the second target length and the text length threshold shall be used as the third target length.

[0118] In each set of partition files, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file;

[0119] Determine the preset ratio, and based on the preset ratio and the target quantity, determine the first quantity and the second quantity;

[0120] Select a first number of words from the word data contained in the second target partition file as target word data, and select a second number of words from the word data contained in the third target partition file as target word data;

[0121] The target word data in each partition file set is summarized to obtain a third preset number of training datasets.

[0122] Specifically, the model training requirements are as follows: The ability to output short texts is trained using 10 million data points, and the text length cannot exceed 100. Based on the model training requirements, the second target length and target quantity of the word data are determined; the second target length is 100, and the target quantity is 10 million.

[0123] Determine the initial text length threshold supported by the model. For example, if the maximum text length supported by the model is 4000, then the text length threshold is 4000. If the text length threshold is greater than the second target length, use the length between the second target length and the text length threshold as the third target length. For example, if the second target length is 100 and the text length threshold is 4000, then the third target length is 100-4000.

[0124] In each set of partitioned files, the partition file corresponding to the second target length is designated as the second target partition file. For example, the partition file used to store word vectors with a length less than the second target length is designated as the second target partition file, such as the partition files identified as 20-100.txt, ..., 90-100.txt. The partition file corresponding to the third target length is designated as the third target partition file. For example, the partition file used to store word vectors with a length greater than or equal to the second target length and less than or equal to the text length threshold is designated as the third target partition file, such as the partition files identified as 100-200.txt, ..., 3001-4000.txt.

[0125] Determine the preset ratio, for example, 80% and 20%. Based on the preset ratio and the target quantity, determine the first quantity and the second quantity. For example, if the target quantity is 10 million, the first quantity is 10 million × 80% = 8 million, and the second quantity is 10 million × 20% = 2 million.

[0126] Select a first number of words from the word data contained in the second target partition file as target words, and select a second number of words from the word data contained in the third target partition file as target words. For example, select 8 million words from the second target partition file as target words, and then select the remaining 2 million words from the word data contained in the third target partition file on an average basis. The acquisition method can be uniform acquisition or other methods that meet actual needs.

[0127] In this embodiment, a first number of words are selected as target words from the word data contained in the second target partition file, and a second number of words are selected as target words from the word data contained in the third target partition file. The target word data in each partition file set are then aggregated to obtain a third preset number of training datasets. This allows for flexible allocation of word data according to model training requirements, forming different training datasets. It fully utilizes the initial corpus data to construct high-quality training datasets, meeting the needs of training different functional models and ensuring training quality.

[0128] In some alternative implementations, another method for deduplicating the initial corpus data includes steps A1 to A4.

[0129] Step A1: Divide at least two pieces of corpus data in the initial corpus data with a text overlap greater than a preset ratio threshold into one bucket, resulting in multiple bucket data subsets.

[0130] Specifically, the text overlap between any two pieces of initial corpus data is equal to the number of overlapping characters divided by the number of characters in the corpus with more overlapping characters. This binning process ensures that a subset of data within a bin contains the same textual information, exhibits strong textual relevance, and has a high probability of including corpus data with the same semantics. Therefore, deduplication of the corpus data is necessary.

[0131] Step A2: For each subset of data in each bucket, obtain multiple pairs of candidate data.

[0132] Specifically, for each data subset, any two corpus data within the corresponding data subset are combined into a candidate data pair, resulting in multiple candidate data pairs, which effectively ensures the comprehensiveness of the obtained candidate data pairs.

[0133] Step A3: Use a semantic similarity scoring model to predict the semantic similarity between the two corpus data in each candidate data pair.

[0134] Specifically, for each candidate data pair, the two corpus data in the candidate data pair are input into the semantic similarity scoring model, which can predict and output the semantic similarity between the two.

[0135] Step A4: Based on the semantic similarity of each candidate data pair in multiple candidate data pairs, the corpus data in the corresponding data subset is deduplicated.

[0136] Specifically, first, from multiple candidate data pairs, at least one target data pair with a semantic similarity greater than or equal to a preset similarity threshold is selected. Then, the at least two corpus data corresponding to the at least one target data pair within the corresponding data subset are treated as duplicate corpus data and aggregated to achieve deduplication. Alternatively, candidate data pairs with semantic similarity less than a preset similarity threshold can be removed from multiple candidate data pairs, leaving only the candidate data pairs requiring deduplication. The remaining candidate data pairs are then aggregated directly to achieve deduplication. During aggregation, a mapping table can be established to map these duplicate corpus data to a single target corpus data, thus ensuring that duplicate corpus data is identified using a unified target corpus data when promoted and applied downstream, such as during knowledge base creation. The target corpus data to which multiple duplicate corpus data are aggregated can be set according to actual needs. For example, if multiple duplicate corpus data contain corpus data from a knowledge base, that corpus data from the knowledge base can be selected as the target corpus data. If no corpus data exists in the knowledge base, the most concise corpus data, such as the one with the fewest words, can be selected as the target corpus data. If multiple concise corpus data exist, one can be selected randomly. Alternatively, industry literature can be consulted to obtain the one with the highest frequency of occurrence. Alternatively, a neural network model can be trained to filter the most appropriate target corpus data from multiple repetitive corpus data, which can then be used as the aggregated target corpus data.

[0137] In this embodiment, a semantic similarity scoring model is used to achieve deduplication, which enhances the learning of difficult negative sample data. This enables the semantic similarity scoring model to improve the accuracy of identifying duplicate and non-duplicate corpora, thus effectively improving the deduplication efficiency of corpus data.

[0138] This embodiment also provides a model training apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0139] This embodiment provides a model training device, such as Figure 5 As shown, it includes:

[0140] The processing module 501 is used to set deduplication functions according to a first preset number of preset parameters, process the initial corpus data according to the set deduplication functions, and put the word data into the corresponding partition file according to the length of the word data in the processing result. The partition file is used to record word data of the corresponding length. The partition file is contained in a first preset number of partition file sets, and the partition file sets correspond to the preset parameters.

[0141] The aggregation module 502 is used to obtain a second preset number of target word data from the partition files contained in each partition file set according to the model training requirements, and aggregate the target word data to obtain a third preset number of training datasets.

[0142] The training module 503 is used to train the initial model using the training dataset to obtain a third preset number of trained models.

[0143] In some alternative implementations, after training the initial model using the training dataset to obtain a third preset number of trained models, the device is used to:

[0144] The third preset number of trained models are tested using a preset test dataset, and the test score corresponding to the trained model is determined based on the test results.

[0145] The model with the highest test score is selected as the target model, and the corresponding training dataset is saved.

[0146] In some optional implementations, the processing module 501 sets deduplication functions according to a first preset number of preset parameters, processes the initial corpus data according to the set deduplication functions, and places the word data into the corresponding partition file according to the length of the word data in the processing result, including:

[0147] The target setting parameter is determined from the first preset number of preset parameters;

[0148] By setting the target parameters, the deduplication function is set, and the deduplication function after setting is obtained;

[0149] The initial corpus data is processed according to the set deduplication function to obtain the processing result;

[0150] The length processing function is used to segment the data in the processing result to obtain a fourth preset number of words.

[0151] Use the partition file set corresponding to the target setting parameters as the target partition file set, and put the word data into the corresponding partition file in the target partition file set according to the length of the word data;

[0152] A new target setting parameter is determined from the first preset number of preset parameters. The new target setting parameter is used as the target setting parameter. The subsequent steps are executed starting from setting the deduplication function using the target setting parameter, until each preset parameter is used to set the target setting parameter.

[0153] In some optional implementations, the processing module 501 uses a length processing function to perform word segmentation on the data in the processing result to obtain a fourth preset number of word data, including:

[0154] Retrieve the rows of data contained in the processing result, and delete the rows of data whose length is less than a preset threshold to obtain the remaining rows of data;

[0155] The remaining rows of data are used as word data to obtain a fourth preset number of word data.

[0156] In some alternative implementations, the device is further configured to:

[0157] Create a first preset number of partition file sets, and generate a first identifier for each partition file set based on the first preset number of preset parameters;

[0158] Create a partition file corresponding to each preset output file partition in each partition file set, and generate a second identifier for each partition file based on the preset output file partition;

[0159] Create a file directory containing the first identifier and the second identifier.

[0160] In some optional implementations, the aggregation module 502, based on model training requirements, obtains a second preset number of target word data points from the partition files contained in each partition file set, and aggregates the target word data to obtain a third preset number of training datasets, including:

[0161] Determine the first target length of the word data based on the model training requirements;

[0162] In each set of partitioned files, the partition file corresponding to the first target length is taken as the first target partition file, and the word data contained in the first target partition file is taken as the target word data;

[0163] The target word data in each partition file set is summarized to obtain a third preset number of training datasets.

[0164] In some optional implementations, the aggregation module 502, based on model training requirements, obtains a second preset number of target word data points from the partition files contained in each partition file set, and aggregates the target word data to obtain a third preset number of training datasets, including:

[0165] Based on the model training requirements, determine the length and number of the second target in the word data;

[0166] Determine the text length threshold supported by the initial model;

[0167] If the text length threshold is greater than the second target length, the length between the second target length and the text length threshold shall be used as the third target length.

[0168] In each set of partition files, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file;

[0169] Determine the preset ratio, and based on the preset ratio and the target quantity, determine the first quantity and the second quantity;

[0170] Select a first number of words from the word data contained in the second target partition file as target word data, and select a second number of words from the word data contained in the third target partition file as target word data;

[0171] The target word data in each partition file set is summarized to obtain a third preset number of training datasets.

[0172] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0173] In this embodiment, the model training device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0174] This invention also provides a computer device having the above-described features. Figure 5 The model training device shown.

[0175] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 6As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0176] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPRS), or any combination thereof.

[0177] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0178] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0179] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0180] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0181] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0182] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0183] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined in this application.

Claims

1. A model training method, characterized in that, The method includes: Deduplication functions are set according to a first preset number of preset parameters. The initial corpus data is processed according to the set deduplication functions. The word data is placed into the corresponding partition file according to the length of the word data in the processing result. The partition file is used to record word data of the corresponding length. The partition file is contained in a first preset number of partition file sets. The partition file sets correspond to the preset parameters. The step of setting deduplication functions according to a first preset number of preset parameters, processing the initial corpus data according to the set deduplication functions, and placing the word data into the corresponding partition file according to the length of the word data in the processing result includes: A target setting parameter is determined from the first preset number of preset parameters; the deduplication function is set using the target setting parameter to obtain the set deduplication function; the initial corpus data is processed according to the set deduplication function to obtain the processing result; the data in the processing result is segmented using a length processing function to obtain a fourth preset number of word data; the partition file set corresponding to the target setting parameter is used as the target partition file set, and the word data is placed into the corresponding partition file in the target partition file set according to the length of the word data; a new target setting parameter is determined from the first preset number of preset parameters, the new target setting parameter is used as the target setting parameter, and subsequent steps are executed starting from setting the deduplication function using the target setting parameter until each preset parameter is used to set the target setting parameter; According to the model training requirements, a second preset number of target word data are obtained from the partition files contained in each partition file set, and the target word data are summarized to obtain a third preset number of training datasets; According to the model training requirements, the process involves obtaining a second preset number of target word data points from the partition files contained in each partition file set, and summarizing the target word data to obtain a third preset number of training datasets, including: Based on the model training requirements, a first target length for the word data is determined; in each partition file set, the partition file corresponding to the first target length is taken as the first target partition file, and the word data contained in the first target partition file is taken as the target word data; the target word data in each partition file set are summarized to obtain a third preset number of training datasets; The initial model is trained using the training dataset to obtain a third preset number of trained models.

2. The method according to claim 1, characterized in that, After training the initial model using the training dataset to obtain a third preset number of trained models, the method further includes: The third preset number of trained models are tested using a preset test dataset, and the test score corresponding to the trained model is determined based on the test results. The model with the highest test score is selected as the target model, and the training dataset corresponding to the target model is saved.

3. The method according to claim 1, characterized in that, The process of segmenting the data in the processing result using a length processing function to obtain a fourth preset number of word data includes: Obtain the row data contained in the processing result, and delete the row data whose data length is less than a preset threshold to obtain the remaining row data; The remaining row data is used as the word data to obtain a fourth preset number of word data.

4. The method according to claim 1, characterized in that, Before placing the word data into the corresponding partition file, the method further includes: Create a first preset number of partition file sets, and generate a first identifier for each partition file set according to the first preset number of preset parameters; In each of the partition file sets, a partition file corresponding to each preset output file partition is created, and a second identifier for each partition file is generated based on the preset output file partition; Create a file directory containing the first identifier and the second identifier.

5. The method according to claim 1, characterized in that, According to the model training requirements, the process involves obtaining a second preset number of target word data points from the partition files contained in each partition file set, and summarizing the target word data to obtain a third preset number of training datasets, including: Based on the model training requirements, determine the second target length and target number of the word data; Determine the text length threshold supported by the initial model; If the text length threshold is greater than the second target length, the length between the second target length and the text length threshold shall be used as the third target length; In each set of partition files, the partition file corresponding to the second target length is used as the second target partition file, and the partition file corresponding to the third target length is used as the third target partition file; A preset ratio is determined, and a first quantity and a second quantity are determined based on the preset ratio and the target quantity; A first number of words are selected from the word data contained in the second target partition file as target word data, and a second number of words are selected from the word data contained in the third target partition file as target word data; The target word data in each of the partition file sets are summarized to obtain a third preset number of training datasets.

6. A model training device, characterized in that, The device includes: The processing module is used to set deduplication functions according to a first preset number of preset parameters, process the initial corpus data according to the set deduplication functions, and put the word data into the corresponding partition file according to the length of the word data in the processing result. The partition file is used to record word data of the corresponding length. The partition file is contained in a first preset number of partition file sets, and the partition file sets correspond to the preset parameters. The processing module sets deduplication functions according to a first preset number of preset parameters, processes the initial corpus data according to the set deduplication functions, and places the word data into the corresponding partition file according to the length of the word data in the processing result. This includes: determining target setting parameters from the first preset number of preset parameters; setting the deduplication function using the target setting parameters to obtain the set deduplication function; processing the initial corpus data according to the set deduplication function to obtain the processing result; performing word segmentation on the data in the processing result using a length processing function to obtain a fourth preset number of word data; using the partition file set corresponding to the target setting parameters as the target partition file set, and placing the word data into the corresponding partition file in the target partition file set according to the length of the word data; determining a new target setting parameter from the first preset number of preset parameters, using the new target setting parameter as the target setting parameter, and executing subsequent steps starting from setting the deduplication function using the target setting parameter until each preset parameter has been used to set the target setting parameter. The aggregation module is used to obtain a second preset number of target word data from the partition file according to the model training requirements, and aggregate the target word data to obtain a third preset number of training datasets; The aggregation module, based on model training requirements, obtains a second preset number of target word data from the partition files contained in each partition file set, and aggregates the target word data to obtain a third preset number of training datasets. This includes: determining a first target length for the word data based on the model training requirements; in each partition file set, using the partition file corresponding to the first target length as a first target partition file, and using the word data contained in the first target partition file as target word data; and aggregating the target word data in each partition file set to obtain the third preset number of training datasets. The training module is used to train the initial model using the training dataset to obtain a third preset number of trained models.

7. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the model training method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the model training method according to any one of claims 1 to 5.

Images