Training data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a training data processing method and device, electronic equipment and a storage medium. The method comprises the steps that multiple pieces of text data are randomly extracted from a text data set; the data bucket corresponding to the text data with the maximum text length in the multiple pieces of text data is determined according to the text length range corresponding to each data bucket, the multiple data buckets have multiple text capacities, and the larger the text capacity is, the larger the corresponding minimum text length is; the data bucket is filled with the text data in the multiple pieces of text data in sequence from large to small according to the text length, and the data bucket is filled with the text data with the text length smaller than the remaining capacity of the data bucket preferentially; and after the data bucket is filled, outputting the data of the data bucket as training data, and returning to the step of randomly extracting multiple pieces of text data from the text data set until the bucket separation processing of the text data set is finished. The short text training effect, the modeling ability of the long context and the overall training speed can be considered.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training data processing method, device, electronic device and storage medium for a large language model. Background Art

[0002] Large language models (LLMs), represented by ChatGPT, have attracted extensive attention and research from academia and industry due to their outstanding performance in many tasks. Among them, how large models can achieve such outstanding performance has always been the focus of research. It is generally believed that the powerful semantic understanding and generation capabilities of large language models mainly come from pre-training on massive corpora. Considering the huge amount of data required to train large language models, the effective organization of input data during training is crucial.

[0003] In the related art, a pre-training data processing method uses each piece of data as independent training data. During batch training, the other data is padded with fillers according to the length of the longest text in the current batch to ensure that the length of the batch data is consistent before training. This method is more common in traditional pre-training tasks, but in the era of large language models, the amount of data has become huge (generally at the TB level). This method will add a large number of fillers and reduce the efficiency of training.

[0004] In the related art, another pre-training data processing method is to randomly concatenate all the data (using delimiters to split them), and then split them according to fixed lengths (such as 2048, 4096, 8192, etc.), so that the length of each data after segmentation is fixed, which is convenient for input into the model for batch training. This approach will concatenate multiple short texts into one training data, or a long text data will be split into multiple training data.

[0005] Although concatenating all data and splitting them into fixed lengths for training has achieved excellent performance on large language models, the defects of this approach are also obvious. First of all, there are many types of training data, including Internet web pages, social media data, books, codes, papers, etc. The length distribution of different data is quite different. For example, general web page data and social media data are relatively short, and book and paper data are very long. Therefore, the fixed length method cannot take into account the training effect of long and short texts. If the fixed length is set to be shorter, the training speed of short context length is faster, and at the same time, there is less text data spliced ​​in a training data, and there is less noise; but if the long text data is split too much, the model's long context modeling ability will be weakened. If the fixed length is set to be longer, the model can support longer context modeling, but because the model training complexity is proportional to the square of the sequence length, a longer context length will slow down the training speed. At the same time, considering that short text data accounts for a high proportion of the total data, if the fixed length is too long, there will be more short text data spliced ​​into a training data, which will generate noise and affect the model's language modeling ability.

[0006] Therefore, the fixed-length data segmentation method is not the best choice for large language model training, and there is room for improvement. Even if the phased training strategy is followed, when the second step of long text training is in progress, the short text training data will decrease, and the overall training data distribution will change, which may affect the model effect. Summary of the invention

[0007] According to one aspect of an embodiment of the present disclosure, a training data processing method is provided, comprising: randomly extracting multiple text data from a text data set, the text data set comprising text data of various text lengths; determining a data bucket corresponding to the text data with the largest text length among the multiple text data according to a text length range corresponding to each data bucket in the multiple data buckets, wherein each data bucket has a corresponding text capacity, multiple data buckets have various text capacities, and the larger the text capacity of the data bucket, the larger the corresponding minimum text length; filling the data bucket with text data from the multiple text data in order from large to small according to the text length, wherein, except for the text data with the largest text length, the data bucket is preferentially filled with text data with a text length less than the remaining capacity of the data bucket; after the data bucket is filled, outputting the data in the data bucket as a training data, and returning to the step of randomly extracting multiple text data from the text data set until the text data set bucketing processing is completed.

[0008] Optionally, the training data processing method further includes: after traversing the multiple randomly selected text data, if the ratio of the remaining capacity to the total capacity of the data bucket is less than a filler ratio threshold, filling the data bucket with fillers of a size corresponding to its remaining capacity.

[0009] Optionally, the above-mentioned training data processing method also includes: after traversing the multiple text data, if the ratio of the remaining capacity to the total capacity of the data bucket is greater than the filler ratio threshold, selecting text data with the shortest text length from the multiple text data, and cutting out a part corresponding to the remaining capacity from the selected text data to fill the data bucket.

[0010] Optionally, the above-mentioned training data processing method also includes: after the data bucket is filled, if there is text data in the multiple text data that has not been filled into the data bucket, returning to the step of determining the data bucket corresponding to the text data with the largest text length among the multiple text data according to the text length range corresponding to each data bucket in the multiple data buckets, so as to continue to bucket the text data that has not been filled into the data bucket.

[0011] Optionally, for text data with the longest text length, a portion having a size corresponding to the total capacity of the data bucket is cut out from the text data and filled into the data bucket.

[0012] Optionally, filling the data bucket with text data from a plurality of text data in order from large to small according to the text length comprises: selecting text data from a plurality of text data in order from large to small according to the text length, and for each selection of text data: if the text length of the text data is less than the remaining capacity of the data bucket, filling the data bucket with the text data; if the text length of the text data is greater than the remaining capacity of the data bucket, when the data bucket is empty, cutting out a portion of the text data corresponding to the remaining capacity and filling the data bucket; when the data bucket is not empty, returning to the step of selecting text data from the plurality of text data in order from large to small according to the text length.

[0013] Optionally, the above-mentioned training data processing method also includes: sorting multiple text data from large to small according to text length to obtain a text data sequence; wherein, filling the data bucket with text data from the multiple text data in sequence from large to small according to the text length includes: selecting text data from the text data sequence in sequence to fill the data bucket with text data from the multiple text data.

[0014] According to another aspect of an embodiment of the present disclosure, a training data processing device is provided, comprising: a selection module, for randomly extracting multiple text data from a text data set, the text data set comprising text data of various text lengths; a processing module, for: determining a data bucket corresponding to text data with the largest text length among multiple text data according to a text length range corresponding to each data bucket in multiple data buckets, each data bucket having a corresponding text capacity, multiple data buckets having various text capacities, and a data bucket having a larger text capacity has a larger corresponding minimum text length; filling the data bucket with text data from multiple text data in order from large to small according to text length, wherein, except for the text data with the largest text length, the data bucket is preferentially filled with text data with a text length less than the remaining capacity of the data bucket; after the data bucket is filled, outputting the data in the data bucket as a training data, and calling the selection module to continue the step of randomly extracting multiple text data from the text data set until all the text data in the text data set are divided into the data buckets.

[0015] Optionally, the processing module is further used to: after traversing multiple text data, if the ratio of the remaining capacity of the data bucket to the total capacity is less than a filler ratio threshold, fill the data bucket with fillers of a size corresponding to the remaining capacity.

[0016] Optionally, the processing module is also used to: after traversing multiple text data, if the ratio of the remaining capacity to the total capacity of the data bucket is greater than the filler ratio threshold, select text data with the shortest text length from the multiple text data, and cut out a part corresponding to the remaining capacity from the selected text data to fill the data bucket.

[0017] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method mentioned above in the embodiment of the present disclosure.

[0018] According to another aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method of the embodiment of the present disclosure.

[0019] One or more technical solutions provided in the embodiments of the present disclosure set multiple data buckets with different text lengths, randomly extract multiple text data from a text data set, determine the data bucket corresponding to the text data with the largest text length among the multiple text data according to the text length range corresponding to each data bucket in the multiple data buckets, and can be adaptively divided into data buckets of different sizes according to the length of the data, and the training data has different text lengths. The data bucket is filled with text data from multiple text data in order from large to small according to the text length, and the number of spliced ​​data in each training data can be constrained to avoid placing too much data to be spliced ​​into a training data to generate too much noise, and in addition to the text data with the largest text length, the data bucket is preferentially filled with text data with a text length less than the remaining capacity of the data bucket, which can reduce the proportion of data segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a training data processing method provided according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 2 A flowchart of another training data processing method provided according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 3 A schematic block diagram of a training data processing device according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] It should be noted that the execution subject of the training data processing method provided in the embodiment of the present disclosure may be one or more electronic devices, which is not limited in the present disclosure; wherein, the electronic device may be a terminal (i.e., a client) or a server, then when the execution subject includes multiple electronic devices, and the multiple electronic devices include at least one terminal and at least one server, the training data processing method provided in the embodiment of the present disclosure may be jointly executed by the terminal and the server. Accordingly, the terminal mentioned here may include but is not limited to: smart phones, tablet computers, laptops, desktop computers, smart watches, smart voice interaction devices, smart home appliances, vehicle terminals, and the like. The server mentioned here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing (cloud computing), cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, and the like.

[0031] Before introducing the specific solution, we first define the following three quantitative indicators to measure the effect of data partitioning:

[0032] (1) Filler ratio: It indicates the ratio of the end fillers in the training data after processing, that is, the ratio of the number of fillers to the number of all words. The lower the better, indicating a higher effective training rate. A filler ratio of 0 means there is no filler.

[0033] (2) Split ratio: The ratio of the original data to be split, that is, the ratio of the number of all original data to the original number. The lower the better, indicating that more data is fully trained. A split ratio of 0 means that all data is not split.

[0034] (3) Number of splicing: It indicates the average number of original data contained in a piece of processed training data. The lower the better, which means that the number of original data spliced ​​in a piece of training data is smaller and the noise is less.

[0035] In some implementations of the embodiments of the present disclosure, at least two of the above-mentioned quantitative indicators are taken into account. In some implementations of the embodiments of the present disclosure, at least three of the above-mentioned quantitative indicators are taken into account.

[0036] According to an embodiment of the present disclosure, a training data processing method for large language model pre-training is provided. The method can at least partially perform adaptive bucketing according to the text length of the text data, thereby taking into account the training effect of short texts, the modeling ability of long contexts, and the overall training speed.

[0037] Figure 1 A flowchart of a training data processing method according to an exemplary embodiment of the present disclosure is shown. Figure 1 As shown, the training data processing method includes steps S101 to S105.

[0038] Step S101 : randomly extracting a plurality of text data from a text data set, where the text data set includes text data of various text lengths.

[0039] In the embodiment of the present disclosure, the number K of text data to be processed each time is set. Considering the huge amount of data in the text data set, it is impossible to load it into the memory at one time for global bucket processing. Therefore, the text data needs to be processed in batches. The number can be selected according to the memory size. For example, it is set to 2000, which means that 2000 text data are selected for bucket processing each time.

[0040] Step S102, determining the data bucket corresponding to the text data with the largest text length among the multiple text data according to the text length range corresponding to each data bucket in the multiple data buckets.

[0041] In the embodiment of the present disclosure, each data bucket has a corresponding text capacity, multiple data buckets have multiple text capacities, and the larger the text capacity of a data bucket is, the larger the corresponding minimum text length is.

[0042] In this exemplary embodiment, the data buckets cover a range of context lengths from short to long: for example, data buckets of five sizes, such as 2048, 4096, 8192, 16384, and 32768. The text data is adaptively divided into different buckets according to the text length.

[0043] According to the text length of the text data with the largest text length, a data bucket that can accommodate the current length is selected. If the text length of the text data exceeds the text capacity of the largest bucket, the current data bucket is set to the largest bucket (such as 32768 in the aforementioned example). Further referring to the aforementioned example, if the text length of the text data is between 16384 and 32768, the current data bucket is set to a data bucket with a text capacity of 32768; if the text length of the text data is between 8192 and 16384, the current data bucket is set to a data bucket with a text capacity of 16384; if the text length of the text data is between 4096 and 8192, the current data bucket is set to a data bucket with a text capacity of 8192; and so on, according to the text length range corresponding to each data bucket in multiple data buckets, the data bucket corresponding to the text data with the largest text length among the multiple text data is determined.

[0044] Step S103, fill the data bucket with text data from the plurality of text data in descending order according to the text length. In the disclosed embodiment, in addition to the text data with the largest text length, the data bucket is preferentially filled with text data whose text length is smaller than the remaining capacity of the data bucket.

[0045] As an implementation mode, for the text data with the largest text length, a portion corresponding to the total capacity of the data bucket is cut out from the text data and filled into the data bucket. In this case, the data bucket is filled with the text data with the largest text length among the multiple text data randomly selected.

[0046] As an implementation mode, filling the data bucket with text data from a plurality of text data in order from large to small according to the text length includes:

[0047] Select text data from multiple text data in order from largest to smallest according to the length of the text, and for each selected text data:

[0048] If the text length of the text data is less than the remaining capacity of the data bucket, fill the text data into the data bucket;

[0049] If the length of the text data is greater than the remaining capacity of the data bucket, when the data bucket is empty, the text data is segmented into a portion corresponding to the remaining capacity and filled into the data bucket; when the data bucket is not empty, the process returns to the step of selecting text data from the multiple text data in descending order according to the text length. When the data bucket is empty, it indicates that no text data has been filled into the data bucket. If the text data is not segmented, it cannot be filled into any data bucket.

[0050] Step S104: after the data bucket is filled, the data in the data bucket is output as a training data. A training data is a concatenation result of the data filled in the data bucket.

[0051] In some embodiments, after the data bucket is filled, if there is text data in the multiple text data that has not been filled into the data bucket, return to step S102, and determine the data bucket corresponding to the text data with the largest text length in the multiple text data according to the text length range corresponding to each data bucket in the multiple data buckets, so as to continue to bucket the text data that has not been filled into the data bucket.

[0052] In the disclosed embodiment, the extracted multiple text data can be stored in the memory, and after one of the multiple text data is filled into the data bucket, the text data can be deleted from the corresponding memory area, so that only the text data of the extracted multiple text data that has not been filled into the data bucket is stored in the memory. Furthermore, after the data bucket is filled, if there is still text data in the memory area, the step of determining the data bucket corresponding to the text data with the largest text length among the text data stored in the memory (i.e., the remaining text data) is performed according to the text length range corresponding to each data bucket in the multiple data buckets, so as to continue to bucket the text data that has not been filled into the data bucket.

[0053] Step S105, determining whether the bucketing process is finished. If the bucketing process is not finished, returning to step S101 to randomly extract a plurality of text data from the text data set, and continuing to bucket the text data in the text data set.

[0054] In the disclosed embodiment, in step S101, after a plurality of text data are randomly extracted from the text data set, the extracted plurality of text data may be removed from the text data set, or the extracted plurality of text data may be marked as extracted, thereby extracting the remaining text data each time.

[0055] In the disclosed embodiment, the number of text data randomly extracted each time may be equal or unequal. As an example, K text data are extracted from the text data set each time, for example, 2000 text data are extracted each time. As another example, the number of text data extracted at least once is unequal to the number of text data extracted at other times.

[0056] In some embodiments, after traversing the above-mentioned multiple randomly extracted text data in step S103, if the ratio of the remaining capacity of the data bucket to the total capacity is less than the filler ratio threshold, fillers of a size corresponding to the remaining capacity of the data bucket are filled into the data bucket. Filler ratio: represents the ratio of the end fillers in the training data after processing, that is, the ratio of the number of fillers to the number of all words. The lower the better, indicating a higher effective training rate. A filler ratio of 0 means that there are no fillers. Through this embodiment, the filler ratio can be kept within the filler ratio threshold, thereby improving the effective training rate.

[0057] In some embodiments, after traversing the multiple randomly extracted text data in step S103, if the ratio of the remaining capacity of the data bucket to the total capacity is greater than the filler ratio threshold, the text data with the shortest text length is selected from the multiple text data, and a portion corresponding to the remaining capacity is cut out from the selected text data and filled into the data bucket. Optionally, the text data with the shortest text length is selected from the text data that is not filled into the data bucket from the multiple randomly extracted text data, and a portion corresponding to the remaining capacity is cut out from the selected text data and filled into the data bucket. Through this embodiment, the segmentation ratio can be reduced. The segmentation ratio is the ratio of the original data (the data in the text data set) to be segmented, that is, the ratio of the number of all the original data to the original number. The lower the better, indicating that more data is fully trained. The segmentation ratio is 0, indicating that all data have not been segmented. In this embodiment, each training data contains at most one segmented text data.

[0058] In some embodiments, after traversing the above-mentioned randomly selected multiple text data in step S103, if the ratio of the remaining capacity of the data bucket to the total capacity is less than the filler ratio threshold, fillers of a size corresponding to the remaining capacity are filled into the data bucket; if the ratio of the remaining capacity of the data bucket to the total capacity is greater than the filler ratio threshold, text data with the shortest text length is selected from multiple text data, and a portion of a size corresponding to the remaining capacity is cut out from the selected text data and filled into the data bucket. Through this embodiment, the text data can be adaptively divided into data buckets of different sizes according to the text length, taking into account the filler ratio and the segmentation ratio, so that the training data as a whole has fewer fillers and a lower segmentation ratio. At the same time, the number of spliced ​​data in each training data can be constrained to avoid placing too much data to be spliced ​​into one training data, thereby generating too much noise.

[0059] Figure 2 A flowchart of a training data processing method according to an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, the method includes steps S201 to S215.

[0060] In this exemplary embodiment, the data bucket covers a context length range from short to long: such as 5 data buckets of 2048, 4096, 8192, 16384, 32768, etc. The data is divided into different buckets according to the text length. The highest filler ratio allowed (i.e., filler ratio threshold) is set, which is generally lower than 1%, to constrain whether to set a filler at the end of a training data or to cut a part of the new data and splice it to the end. The number K of text data processed each time is set. Considering the huge amount of data in the text data set, it is impossible to load it into the memory at one time for global bucket processing. Therefore, it is necessary to process the text data in batches, which can be selected according to the memory size. For example, it is set to 2000, which means that 2000 text data are selected for bucket processing each time.

[0061] Step S201, randomly extract K text data from a text data set, where the text data set includes text data of various text lengths. For example, K is 2000.

[0062] Step S202, sorting the randomly selected K text data according to the text length to obtain a text data sequence. Exemplarily, the text length of the text data sequence decreases from the front to the back, that is, the first text data in the text data sequence is the text data with the largest text length among the K text data, the last text data in the text data is the text data with the smallest text length among the K text data, and the text length of the previous text data in the text data sequence is greater than the text length of any subsequent text data.

[0063] Step S203, determining that the first text data in the text data sequence is the data bucket corresponding to K text data according to the text length range corresponding to each data bucket in the multiple data buckets.

[0064] The first text data in the text data sequence is the text data with the largest text length among the K randomly selected text data. According to the text length of the text data, a data bucket that can accommodate the current length is selected. If the text length of the text data exceeds the text capacity of the maximum bucket, the current data bucket is set to the maximum bucket (such as 32768 in the above example). Further referring to the above example, if the text length of the text data is between 16384 and 32768, the current data bucket is set to a data bucket with a text capacity of 32768; if the text length of the text data is between 8192 and 16384, the current data bucket is set to a data bucket with a text capacity of 16384; if the text length of the text data is between 4096 and 8192, the current data bucket is set to a data bucket with a text capacity of 8192; and so on, according to the text length range corresponding to each data bucket in multiple data buckets, it is determined that the first text data in the text data sequence is the data bucket corresponding to K text data.

[0065] Step S204: select text data from the text data sequence in descending order according to the text length, and execute the subsequent steps for each selected text data.

[0066] Step S205, determining whether the text length of the currently selected text data is greater than the remaining capacity of the data bucket.

[0067] If the text length of the currently selected text data is less than the remaining capacity of the data bucket, execute step S206 to fill the text data into the data bucket. After the text data is filled into the data bucket, execute step S207 to determine whether the data bucket is full. If not, return to step S204 until all the text data in the text data sequence is traversed. If it is full, execute step S208 to output the text data in the data bucket as a training data. After executing step S208, execute step S209 to determine whether there is any text data in the text data sequence that has not been filled into the data bucket. If so, use the remaining text data sequence as a new text data sequence and return to step S203 to continue bucketing the text data that has not been filled into the data bucket; if not, return to step S201.

[0068] If the text length of the currently selected text data is greater than the remaining capacity of the data bucket, step S210 is executed to determine whether the data bucket is empty.

[0069] When the data bucket is empty, execute step S211, cut out a portion of the text data corresponding to the remaining capacity and fill it into the data bucket. After the filling, the data bucket is filled, execute step S208, and output the text data in the data bucket as training data.

[0070] Wherein, if the data bucket is not empty, step S212 is executed to determine whether all text data in the text data sequence are traversed. If all text data in the text data sequence are not traversed, return to step S204, and continue to select text data from the multiple text data in order from large to small according to the text length to select the next text data in the data sequence. If all text data in the text data sequence have been traversed, step S213 is executed to determine whether the ratio of the remaining capacity of the data bucket to the total capacity is less than the filler ratio threshold. If the ratio of the remaining capacity of the data bucket to the total capacity is less than the filler ratio threshold, step S214 is executed to fill the data bucket with a filler of a size corresponding to its remaining capacity, and then step S208 is executed to output the text data in the data bucket as a training data. If the ratio of the remaining capacity of the data bucket to the total capacity is greater than the filler ratio threshold, step S215 is executed to cut the last text data in the text data sequence into a part of a size corresponding to the remaining capacity and fill it into the data bucket, and then step S208 is executed to output the text data in the data bucket as a training data.

[0071] In some embodiments, after the text data is segmented in the above step S211 or step S215, the portion not filled into the data bucket is retained in the text data sequence, and the text data sequence is updated based on the text length of the remaining portion after segmentation, that is, the position of the text data in the text data sequence is adjusted.

[0072] like Figure 2 The method shown can adaptively divide the text data into buckets of different sizes according to the length of the text data, taking into account the filler ratio and the data segmentation ratio, and at the same time can constrain the number of spliced ​​data in each training data to avoid placing too much data in a training data and generating too much noise. Table 1 shows the performance of different data processing methods.

[0073] Table 1: Performance of different data processing methods

[0074] Training strategy Filler Ratio Split ratio Number of splicing Traditional filling method Very high none none Fixed length (short) none Very high Low Fixed length (long) none Low high Strategy of this embodiment Low Low Low

[0075] Referring to Table 1, which lists the performance of different data processing methods on three evaluation indicators, it can be seen that the bucketing strategy proposed in this embodiment can take into account various indicators and can greatly reduce the segmentation ratio and the number of splicing when a very low proportion of fillers are introduced.

[0076] According to another aspect of an embodiment of the present disclosure, a training data processing device is provided.

[0077] Figure 3 A structural block diagram of a training data processing device according to an embodiment of the present application is shown. Figure 3As shown, the device includes: a selection module 10 and a processing module 20.

[0078] The selection module 10 is used to randomly extract multiple text data from a text data set, and the text data set includes text data of multiple text lengths. The processing module 20 is used to: determine the data bucket corresponding to the text data with the largest text length among the multiple text data according to the text length range corresponding to each data bucket in the multiple data buckets, each data bucket has a corresponding text capacity, and the multiple data buckets have multiple text capacities, and the larger the text capacity of the data bucket, the larger the corresponding minimum text length; fill the text data of the multiple text data into the data bucket in order from large to small according to the text length, wherein, except for the text data with the largest text length, the text data with a text length less than the remaining capacity of the data bucket is preferentially filled into the data bucket; after the data bucket is filled, the data of the data bucket is output as a training data, and the selection module is called to continue the step of randomly extracting multiple text data from the text data set until all the text data in the text data set are divided into the data bucket.

[0079] In some implementations, the processing module 20 is further configured to: after traversing multiple text data, if the ratio of the remaining capacity of the data bucket to the total capacity is less than the filler ratio threshold, fill the data bucket with fillers of a size corresponding to the remaining capacity. Through this implementation, a low filling ratio and a low segmentation ratio are ensured, while ensuring that the number of original data spliced ​​in a training data is small.

[0080] In some embodiments, the processing module 20 is also used for: after traversing multiple text data, if the ratio of the remaining capacity to the total capacity of the data bucket is greater than the filler ratio threshold, selecting text data with the shortest text length from the multiple text data, and cutting out a part corresponding to the remaining capacity from the selected text data to fill the data bucket.

[0081] In some embodiments, the processing module 20 is also used for: after traversing the above-mentioned multiple randomly selected text data, if the ratio of the remaining capacity of the data bucket to the total capacity is less than the filler ratio threshold, fill the data bucket with fillers of a size corresponding to its remaining capacity; and if the ratio of the remaining capacity of the data bucket to the total capacity is greater than the filler ratio threshold, select the text data with the shortest text length from the multiple text data, and cut out a part of the size corresponding to the remaining capacity from the selected text data to fill the data bucket. Through this embodiment, the text data can be adaptively divided into data buckets of different sizes according to the text length, taking into account the filler ratio and the segmentation ratio, so that the training data as a whole has fewer fillers and a lower segmentation ratio. At the same time, the number of spliced ​​data in each training data can be constrained to avoid placing too much data to be spliced ​​into one training data, thereby generating too much noise.

[0082] In some embodiments, the processing module 20 is also used for: after the data bucket is filled, if there is text data in the multiple text data that has not been filled into the data bucket, determining the data bucket corresponding to the text data with the largest text length in the remaining (i.e., unfilled) multiple text data according to the text length range corresponding to each data bucket in the multiple data buckets, so as to continue to perform bucketing processing on the text data that has not been filled into the data bucket.

[0083] In some implementations, the processing module 20 is further configured to: for text data with the longest text length, cut out a portion corresponding to the total capacity of the data bucket from the text data and fill it into the data bucket.

[0084] In some implementations, the processing module 20 fills the data bucket with text data from the plurality of text data in order from largest to smallest text length, including:

[0085] Select text data from multiple text data in order from largest to smallest according to the length of the text, and for each selected text data:

[0086] If the text length of the currently selected text data is less than the remaining capacity of the data bucket, fill the currently selected text data into the data bucket;

[0087] If the text length of the currently selected text data is greater than the remaining capacity of the data bucket, when the data bucket is empty, a portion of the currently selected text data corresponding to the remaining capacity is cut out and filled into the data bucket; when the data bucket is not empty, text data is continued to be selected from multiple text data in order from large to small according to the text length.

[0088] According to one embodiment of the present disclosure, Figure 3 Each unit in the device shown can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present disclosure, the selection module 10 and the processing module 20 can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0089] According to another embodiment of the present disclosure, a computer program (including program code) running on a general electronic device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and a storage element can be used to construct the above-mentioned Figure 3 The selection module 10 and the processing module 20 shown in the device shown in the figure, etc. The computer program can be recorded on, for example, a computer storage medium, and loaded into the above-mentioned electronic device through the computer storage medium, and run therein.

[0090] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0091] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0092] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0093] refer to Figure 4 , a block diagram of an electronic device 400 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0094] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0095] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0096] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method training data processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the training data processing method in any other appropriate manner (e.g., by means of firmware).

[0097] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0098] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0102] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A training data processing method, characterized in that: include: Randomly extracting a plurality of text data from a text data set, wherein the text data set includes text data of various text lengths; Determine a data bucket corresponding to text data with the largest text length among the multiple text data according to a text length range corresponding to each data bucket in the multiple data buckets, each data bucket having a corresponding text capacity, the multiple data buckets having multiple text capacities, and the larger the text capacity of a data bucket, the larger the corresponding minimum text length; Filling the data bucket with text data from the plurality of text data in order from largest to smallest text length, wherein, except for the text data with the largest text length, the data bucket is preferentially filled with text data whose text length is smaller than the remaining capacity of the data bucket; After the data bucket is filled, the data in the data bucket is output as a training data, and the process returns to the step of randomly extracting a plurality of text data from the text data set until the text data set bucketing process is completed.

2. The method according to claim 1, characterized in that Also includes: After traversing the plurality of text data, if the ratio of the remaining capacity of the data bucket to the total capacity is less than a filler ratio threshold, fillers having a size corresponding to the remaining capacity are filled into the data bucket.

3. The method according to claim 1 or 2, characterized in that Also includes: After traversing the multiple text data, if the ratio of the remaining capacity to the total capacity of the data bucket is greater than the filler ratio threshold, text data with the shortest text length is selected from the multiple text data, and a part corresponding to the remaining capacity is cut out from the selected text data and filled into the data bucket.

4. The method according to claim 1 or 2, characterized in that: Also includes: After the data bucket is filled, if there is text data that has not been filled into the data bucket among the multiple text data, return to the step of determining the data bucket corresponding to the text data with the largest text length among the multiple text data according to the text length range corresponding to each data bucket in the multiple data buckets, so as to continue to bucket the text data that has not been filled into the data bucket.

5. The method according to claim 1 or 2, characterized in that: For the text data with the longest text length, a portion having a size corresponding to the total capacity of the data bucket is cut out from the text data and filled into the data bucket.

6. The method according to claim 1 or 2, characterized in that: Filling the data bucket with text data from the plurality of text data in order from largest to smallest according to the text lengths, including: Select text data from the plurality of text data in order from largest to smallest according to the length of the text, and for each selected text data: If the text length of the text data is less than the remaining capacity of the data bucket, fill the text data into the data bucket; If the text length of the text data is greater than the remaining capacity of the data bucket, when the data bucket is empty, a portion corresponding to the remaining capacity is cut from the text data and filled into the data bucket; when the data bucket is not empty, return to the step of selecting text data from the multiple text data in descending order according to the text length.

7. The method according to claim 6, characterized in that Also includes: The plurality of text data are sorted from large to small according to the text length to obtain a text data sequence; wherein, the text data among the plurality of text data are sequentially filled into the data bucket according to the text length from large to small, including: selecting text data in sequence from the text data sequence.

8. A training data processing device, characterized in that: include: A selection module, used for randomly extracting a plurality of text data from a text data set, wherein the text data set includes text data of various text lengths; Processing modules for: Determine a data bucket corresponding to text data with the largest text length among the multiple text data according to a text length range corresponding to each data bucket in the multiple data buckets, each data bucket having a corresponding text capacity, the multiple data buckets having multiple text capacities, and the larger the text capacity of a data bucket, the larger the corresponding minimum text length; Filling the data bucket with text data from the plurality of text data in order from largest to smallest text length, wherein, except for the text data with the largest text length, the data bucket is preferentially filled with text data whose text length is smaller than the remaining capacity of the data bucket; After the data bucket is filled, the data in the data bucket is output as a training data, and the selection module is called to continue the step of randomly extracting a plurality of text data from the text data set until the bucketing process of the text data set is completed.

9. The device according to claim 8, characterized in that The processing module is further used for: After traversing the plurality of text data, if the ratio of the remaining capacity of the data bucket to the total capacity is less than a filler ratio threshold, fillers having a size corresponding to the remaining capacity are filled into the data bucket.

10. The device according to claim 8 or 9, characterized in that The processing module is further used for: After traversing the multiple text data, if the ratio of the remaining capacity to the total capacity of the data bucket is greater than the filler ratio threshold, text data with the shortest text length is selected from the multiple text data, and a part corresponding to the remaining capacity is cut out from the selected text data and filled into the data bucket.

11. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images