Large language model training data packaging method and device and storage medium

Abstract

The invention discloses a large language model training data packaging method and device and a storage medium, and relates to the technical field of data processing.The method comprises the steps that initial sequences corresponding to training data are sorted according to the sequence from large to small, and an ordered list is obtained; according to the sequence in the ordered list, comparing the sequence length of the initial sequence with the barrel capacity of a preset barrel; determining the bucket with the bucket capacity greater than or equal to the sequence length and the minimum residual capacity as a target bucket, and putting the corresponding initial sequence into the target bucket; the initial sequences in one bucket are spliced to obtain a target sequence, and the target sequence is used for large language model training. According to the invention, filling of invalid data can be reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, device and storage medium for packaging training data of a large language model. Background Technology

[0002] The underlying network architecture of large language models has a fixed requirement for the length of input data. The computational dimensions of components such as attention mechanisms and fully connected layers are all designed based on a preset sequence length. It is necessary to process the input data to a uniform sequence length, such as through padding, truncation, and bucketing, to transform the original sequence into standardized data that conforms to the model input specifications.

[0003] Typically, a padding strategy involves uniformly padding sequences of different lengths to the maximum sequence length of the batch, thus meeting the model's requirement for fixed-length input. While this method is logically simple, in training tasks involving long sequences, it introduces massive amounts of invalid padding due to the significant difference between many sequences and the maximum batch length. This leads to large-scale invalid computations, reducing overall training efficiency and increasing computational costs.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a method, device and storage medium for packaging large language model training data, aiming to solve the technical problem of how to allocate the sequences corresponding to the training data, thereby reducing the waste of resources caused by invalid padding.

[0006] To address the aforementioned problems, this application provides a method for packaging large language model training data, the method comprising: The initial sequences corresponding to the training data are sorted in descending order to obtain an ordered list; According to the sorting in the ordered list, the sequence length of the initial sequence is compared with the capacity of the preset bucket; The bucket with the smallest remaining capacity that is greater than or equal to the sequence length is identified as the target bucket, and the corresponding initial sequence is placed into the target bucket. The initial sequence in one of the buckets is concatenated to obtain the target sequence, which is used for training a large language model.

[0007] In one embodiment, after the step of comparing the sequence length of the initial sequence with the bucket capacity of the preset bucket according to the sorting in the ordered list, the large language model training data packaging method further includes: If the sequence length is greater than the total capacity of all the buckets, then create a bucket based on the sequence length. The initial sequence is placed into the newly created bucket, and the bucket list is updated.

[0008] In one embodiment, the step of identifying the bucket with the smallest remaining capacity that is greater than or equal to the sequence length as the target bucket and placing the corresponding initial sequence into the target bucket includes: Determine the sequence length of the first sequence in the ordered list; Traverse the preset buckets and identify the preset buckets whose remaining capacity is greater than or equal to the sequence length as candidate buckets; Based on the binary search method, the bucket with the smallest remaining capacity is determined from the list of remaining bucket capacities of the candidate buckets, and the target bucket is obtained. The first and second sequences are placed into the target bucket, and the first and second sequences are removed from the ordered list until the ordered list is empty.

[0009] In one embodiment, before the step of traversing the preset buckets and identifying the preset buckets with remaining capacity greater than or equal to the sequence length as candidate buckets, the large language model training data packaging method further includes: Determine the remaining capacity of each preset bucket, and sort the remaining capacity in ascending order to obtain a list of remaining bucket capacities; When a sequence is added to the bucket, the remaining bucket capacity is updated according to the length of the sequence, and the ordered list is also updated.

[0010] In one embodiment, the step of concatenating the initial sequence in a bucket to obtain a target sequence, the target sequence being used for training a large language model, includes: Traverse the buckets and concatenate the initial sequence according to the order in which the sequence was placed to obtain the target sequence; The sum of the sequence lengths of each of the initial sequences is obtained to get the total length of the target sequence; If the total length is less than the preset maximum sequence length, the target sequence is padded according to the attention mask.

[0011] In one embodiment, after the step of traversing the buckets and concatenating the initial sequence according to the sequence insertion order to obtain the target sequence, the large language model training data packaging method further includes: Iterate through each token in the initial sequence and add an end character after the last token in the initial sequence; Assign position numbers to each of the tokens according to the order and the preset starting value; Once the initial sequence has been numbered, the number counter is reset to the initial value.

[0012] In one embodiment, after the step of comparing the sequence length of the initial sequence with the bucket capacity of the preset bucket according to the sorting in the ordered list, the large language model training data packaging method further includes: When the utilization rate of the first bucket is less than the preset first bucket utilization rate threshold, the initial sequence is divided into a long sequence and a normal sequence according to the preset long sequence threshold. Determine the fit between the long sequence and the preset bucket, and identify the bucket with the highest fit as the candidate bucket to obtain the long sequence allocation result; Based on the normal sequence and the maximum total filling amount of the bucket, a dynamic programming model is generated, and the normal sequence allocation result is obtained according to the dynamic programming model. Based on the ordinary sequence allocation result and the long sequence allocation result, the first allocation result is determined.

[0013] In one embodiment, before the step of concatenating the initial sequence in one of the buckets to obtain the target sequence, the large language model training data packaging method further includes: The utilization rate of the second bucket is determined based on the ratio of the filled length to the total capacity of the bucket. Buckets whose utilization rate is less than the preset second bucket utilization rate threshold are identified as low utilization rate buckets; The sequences in the low-utilization bucket are redistributed, and the migration benefits are determined based on the redistribution results; If the migration benefit is positive, then update the remaining bucket capacity and the first allocation result according to the allocation result to obtain the second allocation result.

[0014] In addition, to achieve the above objectives, this application also proposes a large language model training data packaging device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large language model training data packaging method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the large language model training data packaging method described above.

[0016] This application provides a method for packaging training data for large language models. The initial sequences are sorted by length from longest to shortest. Then, the sequences are compared with preset bucket capacities in this order, and the sequences are placed into buckets with capacities no less than their length. This prioritizes finding suitable buckets for long sequences, preventing them from being forced to split due to a lack of suitable buckets, and reducing semantic integrity loss and storage resource waste caused by sequence splitting. Selecting the bucket with a capacity no less than the sequence length and the smallest remaining capacity as the target bucket for the initial sequence maximizes bucket space utilization, avoiding the polarization of some buckets having large amounts of idle space while others quickly fill up. This ensures that the length of the concatenated target sequence is closer to the upper limit of the bucket capacity, making the model input dimension more regular, thereby improving data processing efficiency and model training stability. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of the first process for packaging training data for the large language model in this application; Figure 2 A second flowchart illustrating the method for packaging training data for the large language model in this application; Figure 3 A performance comparison diagram of training tasks for data packing strategies; Figure 4 This diagram illustrates the performance comparison of the Cmb-Packing method for packaging training data for large language models in this application. Figure 5 A performance comparison diagram of training tasks for data packaging using the Open-Trl-Packing strategy; Figure 6 A performance comparison diagram of training tasks for data packing using the Swift-Packing strategy; Figure 7 This is a schematic diagram of the hardware operating environment involved in the method for packaging large language model training data in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] To achieve the above objectives, this application proposes a method for packaging training data for a large language model. The initial sequences corresponding to the training data are sorted in descending order to obtain an ordered list. Based on the sorting in the ordered list, the sequence length of the initial sequence is compared with the capacity of a preset bucket. The bucket with a capacity greater than or equal to the sequence length and the smallest remaining capacity is identified as the target bucket, and the corresponding initial sequence is placed into the target bucket. The initial sequences in one bucket are concatenated to obtain the target sequence, which is used for training the large language model.

[0024] The underlying network architecture of large language models has a fixed requirement for the length of input data. The computational dimensions of components such as attention mechanisms and fully connected layers are all designed based on a preset sequence length. It is necessary to process the input data to a uniform sequence length, such as through padding, truncation, and bucketing, to transform the original sequence into standardized data that conforms to the model input specifications.

[0025] Typically, a padding strategy involves uniformly padding sequences of different lengths to the maximum sequence length of the batch, thus meeting the model's requirement for fixed-length input. While this method is logically simple, in training tasks involving long sequences, it introduces massive amounts of invalid padding due to the significant difference between many sequences and the maximum batch length. This leads to large-scale invalid computations, reducing overall training efficiency and increasing computational costs.

[0026] This application provides a method for packaging training data for large language models. The initial sequences are sorted by length from longest to shortest. Then, the sequences are compared with preset bucket capacities in this order, and the sequences are placed into buckets with capacities no less than their length. This prioritizes finding suitable buckets for long sequences, preventing them from being forced to split due to a lack of suitable buckets, and reducing semantic integrity loss and storage resource waste caused by sequence splitting. Selecting the bucket with a capacity no less than the sequence length and the smallest remaining capacity as the target bucket for the initial sequence maximizes bucket space utilization, avoiding the polarization of some buckets having large amounts of idle space while others quickly fill up. This ensures that the length of the concatenated target sequence is closer to the upper limit of the bucket capacity, making the model input dimension more regular, thereby improving data processing efficiency and model training stability.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with network communication and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or apparatus capable of performing the above functions. The following description uses a large language model training data packaging device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, embodiments of this application provide a method for packaging large language model training data, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for packaging training data for large language models in this application.

[0029] In this embodiment, the method for packaging training data for a large language model includes steps S10 to S40: Step S10: Sort the initial sequence corresponding to the training data in descending order to obtain an ordered list.

[0030] In this embodiment, the field mapping configuration is obtained, such as a JSON-formatted configuration file, database-stored mapping rules, etc. The field mapping configuration defines the correspondence between sequence length, sequence identifier and data source fields; data type conversion rules for each target field, such as the sequence identifier needing to be converted to a string type; and validity verification rules for each target field. A globally unique identifier set is initialized to store the parsed sequence identifiers.

[0031] The system reads a single record from a data source using a general data reading interface, encapsulates different data sources, and outputs a key-value pair structure in a unified format.

[0032] Based on the field mapping rules, extract the original field value corresponding to the sequence identifier from the key-value pair. If the field is missing, mark it as a missing field error, skip the record, and log it. If the original field value is a value type (integer / floating-point number), convert it to a string type using the prefix + numeric string conversion rule. For example, if the prefix is ​​"S", the numeric value 123 is converted to S123. If the original field value is a string type, retain the original string directly. If it contains invisible or special characters, perform character filtering. If the original field value is a binary type or other type, convert it to an encoded string to adapt to the identification requirements of binary data. Pass the converted sequence identifier to the globally unique identifier set for deduplication. If it already exists, mark it as a duplicate ID error, skip the record, and log it. If it does not exist, add the sequence identifier to the globally unique identifier set.

[0033] Based on the field mapping rules, extract the original field value corresponding to the sequence length from the key-value pairs of the records. If a field is missing, mark it as a missing field error, skip the record, and log it. If the original field value is a string, call the numeric parsing function to convert it; if the original field value is a floating-point number, first perform a rounding operation, then convert it to an integer. After successful conversion, perform a numeric range check. If the value is <0 or > the preset maximum sequence length, mark it as a numeric out-of-bounds error, skip the record, and log it.

[0034] For records where the sequence identifier and sequence length are successfully parsed, a generic sequence object is constructed, containing the sequence length and sequence identifier. The created sequences are then sorted in descending order of addition, resulting in an ordered list. If sequences have the same length, they are sorted in ascending order by sequence identifier. By sorting the initial sequences in descending order of length, longer sequences are processed first, allowing for priority matching of larger buckets and reducing redundant use of smaller buckets, thus optimizing bucket resource utilization. The descending order sorting also facilitates subsequent sequential bucket matching, preventing situations where long sequences cannot be matched to suitable buckets due to later processing.

[0035] Step S20: According to the sorting in the ordered list, compare the sequence length of the initial sequence with the bucket capacity of the preset bucket.

[0036] Step S30: The bucket with the capacity greater than or equal to the sequence length and the smallest remaining capacity is identified as the target bucket, and the corresponding initial sequence is placed into the target bucket.

[0037] In this embodiment, a predefined list of preset buckets is used. Each preset bucket includes attributes such as bucket ID, bucket capacity (i.e., the maximum sequence length that the bucket can hold), and the sequence list within the bucket. A preset bucket attribute dictionary is constructed, and the sequence list within the bucket is initialized to an empty list, resulting in a list of preset bucket objects, which includes the capacity attribute of each bucket and an empty sequence storage structure.

[0038] Traverse each initial sequence in the ordered list to determine its length. Then, traverse all buckets in the pre-defined bucket object list, extracting the capacity of each bucket and checking if the bucket capacity is greater than or equal to the sequence length. For each initial sequence, if a bucket meets the condition, select the first bucket or the one with the smallest capacity as the target bucket and add its sequence ID to the target bucket's sequence list. After completing the bucket allocation for the current sequence, continue traversing the next initial sequence in the ordered list. After traversal is complete, update the bucket lists of the pre-defined buckets and newly created buckets.

[0039] In one feasible implementation, if the sequence length is greater than the capacity of all the buckets, a bucket is created according to the sequence length; the initial sequence is placed into the newly created bucket, and the bucket list is updated.

[0040] In this embodiment, if no bucket meets the conditions, a new bucket is created, and its capacity is set to the sequence length, or rounded up to a fixed granularity according to a preset rule. The new bucket is added to the bucket list, and the sequence ID is added to the sequence list within the newly created bucket. After completing the bucket allocation for the current sequence, the next initial sequence in the ordered list is traversed. After the traversal is complete, the bucket lists of the preset buckets and the newly created buckets are updated, and the sequence list within each bucket contains the allocated sequence ID.

[0041] In this embodiment, traversing in descending order of sequence length can prioritize the processing of long sequences, avoiding the inability of long sequences to match the preset buckets due to post-processing; selecting the bucket with the smallest capacity that meets the conditions can maximize the space utilization of the buckets and reduce the waste of bucket resources; the capacity of the new bucket is set according to the sequence length or rounded up to ensure that the new bucket can just accommodate the corresponding sequence and avoid capacity redundancy.

[0042] Step S40: Concatenate the initial sequence in one of the buckets to obtain the target sequence, which is used for training the large language model.

[0043] In this embodiment, each bucket identifier in the sequence list dictionary of each bucket is traversed one by one, and the sequence identifier list in the bucket corresponding to the identifier is extracted; according to the order in which the sequence is put into the bucket, the complete data of the corresponding sequence is read from the complete sequence data dictionary, and these data are concatenated into a complete target sequence.

[0044] The stored bucket object contains: a bucket ID, a collection of initial sequence objects stored in the order they were added, each initial sequence object containing sequence data content and sequence length; and attributes such as the bucket's total capacity and remaining capacity. All bucket objects are traversed in bucket ID order, and the sequence list attributes of the current bucket are read to confirm that the bucket contains at least one initial sequence. If the sequence list is empty, the bucket is skipped.

[0045] For the current bucket's sequence list, read the initial sequence objects one by one in the order they were added to the bucket. Load the data pointer of the initial sequence from memory, pointing to the storage address of the sequence data content, such as the starting address of the Token ID array and the memory starting position of the text string. Read the data length information of the initial sequence, such as the number of elements in the Token ID array, the number of bytes in the text, and the size of the tensor. Accumulate the data lengths of all initial sequences in the bucket to obtain the total length of the target sequence. Call the memory allocation interface to allocate a contiguous memory space in the memory heap area, the size of which is the total amount of data in the target sequence, and record the starting address of this memory space.

[0046] Initialize the data write pointer to point to the first address of the target sequence memory space; traverse the initial sequence in the bucket in sequence and read the data content of the current initial sequence; call the memory copy function to copy the original data of the initial sequence to the memory location pointed to by the pointer in bytes / elements; update the pointer by moving the pointer forward by the number of bytes / elements offset corresponding to the current data length of the initial sequence.

[0047] After copying the data of all initial sequences, the memory space of the target sequence is encapsulated into a target sequence object, which contains the following attributes: target sequence ID, data content, total length, bucket ID, list of initial sequence IDs, etc. The target sequence object is stored in the target sequence list in memory, resulting in the target sequence list used for training large language models.

[0048] In this embodiment, the initial sequence is sorted by length from longest to shortest. Then, the sequences are compared with preset bucket capacities in this order, and the sequences are placed into buckets with capacities not less than their length. This prioritizes finding suitable buckets for long sequences, preventing them from being forced to split due to a lack of suitable buckets, and reducing semantic integrity loss and storage resource waste caused by sequence splitting. Selecting the bucket with a capacity not less than the sequence length and the smallest remaining capacity as the target bucket for the initial sequence maximizes bucket space utilization, avoiding the polarization of some buckets having large amounts of idle space while others quickly fill up. This ensures that the length of the concatenated target sequence is closer to the upper limit of the bucket capacity, making the model input dimension more regular, thereby improving data processing efficiency and model training stability.

[0049] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S30 may include steps S31 to S34: Step S31: Determine the sequence length of the first sequence in the ordered list.

[0050] Step S32: Traverse the preset buckets and identify the preset buckets whose remaining capacity is greater than or equal to the sequence length as candidate buckets.

[0051] In this embodiment, the initial sequence set is read, and the sequence identifier and sequence length of all sequences are extracted; Obtain the list of preset buckets, determine the bucket identifier, initial total capacity, and remaining bucket capacity of all preset buckets, construct a dictionary of preset bucket attributes, and initialize the dictionary of sequence lists within the buckets; sort the initial sequence set in descending order of sequence length to generate an ordered sequence list. Read the first initial sequence of the ordered sequence list as the identifier of the currently processed sequence; determine the sequence length corresponding to the first initial sequence.

[0052] Initialize the candidate bucket list to an empty list; iterate through each bucket identifier in the bucket attribute dictionary and extract the remaining bucket capacity corresponding to the bucket identifier; check if the remaining bucket capacity is greater than or equal to the sequence length corresponding to the first initial sequence. If it is true, add the bucket identifier to the candidate bucket list and record the corresponding remaining bucket capacity; if not, skip the bucket. By iterating through the remaining capacity of the preset buckets, select buckets that can accommodate the current sequence.

[0053] Step S33: Based on the binary search method, determine the bucket with the smallest remaining bucket capacity in the list of remaining bucket capacities of the candidate buckets, and obtain the target bucket.

[0054] In this embodiment, each bucket identifier in the candidate bucket list is traversed one by one, and the remaining capacity corresponding to the bucket is retrieved from the preset bucket attribute dictionary; an association list of bucket identifiers and remaining capacities is constructed, where each element in the list is a tuple in the format (bucket identifier, remaining capacity); all remaining capacity values ​​are extracted separately from the association list to form an independent list of remaining capacity values, and the order of the values ​​is consistent with that of the association list, thus obtaining the candidate bucket capacity association list and the candidate bucket remaining capacity value list.

[0055] Sort the list of remaining capacity values ​​in ascending order to obtain an ordered list of remaining capacity values. Iterate through each capacity value and its index in the ordered list. Find the index of the first occurrence of the capacity value in the original list of remaining capacity values. Extract the corresponding bucket identifier from the candidate bucket capacity association list based on this index, combine them into a triplet, and add it to the mapping list to construct a list of triplet pairs of ordered capacity-bucket identifier-original list index. If multiple buckets have the same remaining capacity and are all the minimum value, determine the unique bucket identifier according to a preset rule, such as the bucket identifier number being the smallest. This yields an ascending list of remaining capacity values ​​and a mapping list of capacity and bucket identifiers.

[0056] Initialize the boundary pointers for the binary search, setting the left pointer to 0 and the right pointer to the length of the ascending capacity list minus 1. Calculate the middle pointer, setting its value to half the integer value of the sum of the left and right pointer values. Compare the capacity value at the middle pointer position with the capacity value at the left pointer position: if the capacity value at the middle position is greater than the capacity value at the left pointer position, the minimum value is in the left half of the interval, and the right pointer is updated to the middle pointer minus 1; if they are equal, further check if there is a smaller value in the left half of the interval; when the left pointer value is greater than or equal to the right pointer value, terminate the search, confirming that the minimum value is at index 0 of the ascending sequence list. Extract the bucket identifier corresponding to index 0 from the capacity-bucket identifier mapping list; this identifier is the bucket with the smallest remaining capacity. First, sort the remaining capacities of the candidate buckets, then use binary search to find the bucket with the smallest remaining capacity, achieving optimal utilization of bucket resources and reducing capacity redundancy.

[0057] Optionally, the remaining capacity can be sorted in ascending order to generate a sorted list of candidate buckets with remaining capacity; the first element of the candidate bucket remaining capacity list, i.e., the element with the smallest remaining capacity, is extracted, and its corresponding bucket identifier is obtained to obtain the target bucket. If the candidate bucket remaining capacity list is empty, a new bucket identifier is generated, the initial total capacity of the new bucket is set, and the remaining bucket capacity is equal to the initial total capacity; the new bucket is then added to the bucket list.

[0058] Step S34: Place the first and second sequences into the target bucket and remove the first and second sequences from the ordered list until the ordered list is empty.

[0059] In this embodiment, the current sequence identifier is added to the sequence list corresponding to the target bucket identifier in the bucket sequence dictionary; the original remaining capacity of the target bucket is retrieved from the bucket attribute dictionary, and the new remaining capacity is calculated using the formula: new remaining capacity = original remaining capacity - current sequence length; the remaining capacity of the target bucket in the bucket attribute dictionary is updated to the calculated new remaining capacity; the first current sequence identifier is deleted from the ordered sequence list. This yields the bucket attribute dictionary with updated remaining capacity, the bucket sequence dictionary with added sequences, and the updated ordered sequence list. The ordered sequence list is checked to see if it is empty. If the list is not empty, the process returns to step S32 and continues processing the latest first and last sequence in the list; if the list is empty, the loop terminates.

[0060] In Experiment 1, the training efficiency of Cmb-Packing (the data packing method provided in this embodiment) and Padding was compared. The Cmb-Packing strategy was used to concatenate and pack samples, and a standard tokenizer was used to pad the samples to a uniform, fixed size, serving as a control group. Apart from the data organization method, the optimizer, learning rate, and training configurations were identical, and FlashAttention-2 was enabled in all cases. Training duration, padding rate, and throughput were used as test metrics. Training duration (s) represents the time required to complete one full training iteration; padding rate (%) represents the proportion of padding tokens to total tokens; and throughput (tokens / s) represents the number of valid tokens processed per second, reflecting the actual computing power utilization efficiency of the GPU. The results of Experiment 1 are shown in the table below:

[0061] Experimental data shows that Cmb-Packing exhibits significant efficiency advantages across different maximum sequence lengths. Training time is 1 / 2 to 1 / 4 that of Padding, shortening the overall training cycle. The fill rate is 2 to 4 times that of Padding. As `max_seq_length` (maximum sequence length) increases, the throughput of Padding decreases significantly, while Cmb-Packing's advantages become more pronounced in long sequence tasks. The fill rate of Cmb-Packing remains consistently below 5%, while the fill rate of Padding exceeds 30%, demonstrating that Cmb-Packing significantly converts inefficient computation into effective computing power.

[0062] In Experiment 2, the training task performance of Cmb-Packing was compared with that of mainstream packing strategies. The Padding group served as a standard control group. The Packing group compared the loss of model accuracy after training with Open-Trl-Packing, Cmb-Packing, and Swift-Packing compared to that of Padding. FlashAttention-2 was enabled in all groups. The test metrics included the loss convergence curve and task accuracy. The loss convergence curve was used to compare the convergence speed and stability of each group during training, and the task accuracy (%) was used to measure the accuracy on the validation set and to assess the impact of different packing strategies on model performance. The task accuracy test results of Experiment 2 are shown in the table below:

[0063] Please refer to Figures 3-6 Compared to the Open-Trl-Packing strategy, Cmb-Packing improves task accuracy by nearly 4%, and compared to the Swift-Packing strategy, it improves accuracy by nearly 10%. Compared to Open-Trl-Packing and Swift-Packing, Cmb-Packing exhibits faster convergence and smaller fluctuations in its loss curve, demonstrating stronger robustness and stability in long sequence tasks. Compared to the Padding strategy, although Padding achieves slightly higher accuracy (3.1%), its training time is approximately three times that of Cmb-Packing, making its cost-effectiveness less competitive. Overall, Cmb-Packing achieves optimal training cost-effectiveness while ensuring semantic isolation and accuracy fidelity.

[0064] Experiment 3 compares the packing performance of Cmb-Packing and Swift-Packing. Using the same dataset, the data packing time of the MS-Swift framework's packing strategy is compared with that of Cmb-Packing. Data sizes were set to 10,000, 20,000, 40,000, 80,000, 160,000, and 320,000 samples, with a fixed maximum sequence length of 8192. The test metric is packing time (s), representing the time required to complete data packing, used to evaluate the algorithm's efficiency and scalability. Experimental data are as follows:

[0065] Experimental results show that Cmb-Packing significantly outperforms Swift-Packing across all data sizes, with the advantage increasing exponentially with increasing data volume. Swift-Packing's packing time exhibits a non-linear, near-quadratic growth trend with data size. When the data volume increases from 10,000 to 320,000, the processing time increases from 0.85s to 843.19s, an increase of over 991 times. This performance degradation is consistent with its high time complexity of O(N*M). Cmb-Packing's time growth is almost linear. With the same data size of 320,000, Cmb-Packing completes packing in just 2.04s, making it 414 times more efficient than the packing strategy used in the open-source MS-Swift framework.

[0066] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, step S40 may also include steps S41 to S43: Step S41: Traverse the buckets and concatenate the initial sequence according to the order in which the sequence was placed to obtain the target sequence.

[0067] In this embodiment, all bucket objects are traversed in the order of bucket creation or in ascending order of bucket identifier. If the number of buckets is large, they can be traversed in batches. For the bucket currently being traversed, the total length of all initial sequences within the bucket is first calculated, and contiguous memory space is allocated as a buffer for the target sequence based on the total length. If special markers such as separators or end markers need to be added, additional space of the corresponding length must be reserved.

[0068] Following the order in which the sequences were placed into the buckets, each initial sequence is traversed. The actual data content of the current initial sequence is obtained, and the read sequence data is appended to the end of the target sequence buffer, byte by byte or element by element. For example, if the target sequence is an array of identifiers and the buffer is initially empty, after writing the first sequence [101, 102], the buffer becomes [101, 102]; after writing the second sequence [201, 202, 203], the buffer becomes [101, 102, 201, 202, 203]. After all initial sequences have been written, the buffer data is encapsulated into a target sequence object.

[0069] In one feasible implementation, step S41 may include the following steps: traversing each token of the initial sequence and adding an end character after the last token of the initial sequence; assigning position numbers to each token according to the order and a preset starting value; and resetting the number counter to the starting value after the numbering of a single initial sequence is completed.

[0070] In this implementation, each bucket in the bucket object list is read one by one, and the sequence list within each bucket is extracted. The token list corresponding to the initial sequence is read from the complete sequence data dictionary in sequence identifier order; a preset end-of-string character is added to the end of the token list, such as... <eos>(End-of-Sequence), forming a processed token list; initializing a number counter to a preset starting value, traversing the processed token list, and assigning a position number to each token, number = counter value, and incrementing the counter by 1 after assignment; after completing token numbering for a single initial sequence, resetting the number counter to the preset starting value, and recording the token list, length, and number mapping of the sequence. According to the sequence placement order, all processed initial sequence token lists in the bucket are sequentially spliced to form the target sequence.

[0071] In this embodiment, the tokens of each sample are independently numbered from the starting value, for example, from 0, [0..49, 0..29, 0..14], ensuring that the relative position encodings of different initial sequences are independent of each other. This can ensure that efficient operators such as FlashAttention-2 do not incorrectly associate tokens from different samples when performing attention calculations, thereby avoiding semantic interference between different samples and ensuring the accuracy and stability of model training.

[0072] Step S42, obtaining the sum of the sequence lengths of each initial sequence to obtain the total length of the target sequence.

[0073] Step S43, if the total length is less than the preset maximum sequence length, padding the target sequence according to the attention mask.

[0074] In this embodiment, the lengths of all processed initial sequences are added to obtain the total length of the target sequence. The total length of the temporary target sequence is compared with the preset maximum sequence length. If the total length of the temporary target sequence is greater than or equal to the preset maximum length, no padding is needed, and the target sequence is directly used as the final target sequence. At the same time, an attention mask array is generated, the array length is equal to the total length of the temporary target sequence, and the mask value of all positions is 1, indicating that all tokens are valid.

[0075] If the total length of the temporary target sequence is less than the preset maximum length, the length of padding needed is calculated, padding length = preset maximum length - total length of temporary target sequence; a corresponding number of padding symbols are added at the end of the temporary target sequence to form the final target sequence; then an attention mask array is generated, the array length is equal to the preset maximum length, and the mask value of the position corresponding to the target sequence before padding is 1, and the mask value of the position of the padding length after padding is 0. Padding according to the preset maximum length can make the target sequence meet the requirements of large language models for fixed length input; the attention mask can make the model ignore the calculation of the padding position during training, ensuring that the model only focuses on learning valid tokens and ensuring the training effect.

[0076] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and the subsequent will not be described. On this basis, before step S40, steps A10~A40 can also be included: Step A10, when the first bucket utilization is less than the preset first bucket utilization threshold, the initial sequence is divided into long sequence and normal sequence according to the preset long sequence threshold.

[0077] In this embodiment, the long sequence threshold is set, and the sequence with length greater than or equal to the threshold is determined as the long sequence, and the sequence with length less than the threshold is determined as the normal sequence. Optionally, the long sequence threshold can be set based on the business scenario, or can be determined according to the average capacity of the bucket; for example, 70% of the average capacity of the bucket is the long sequence threshold. The long sequence is stored separately as a long sequence list, and the normal sequence is stored as a normal sequence list.

[0078] Step A20, determine the degree of adaptation of the long sequence and the preset bucket, determine the bucket with the highest degree of adaptation as the candidate bucket, and obtain the long sequence allocation result.

[0079] In this embodiment, the first long sequence is taken out from the long sequence queue to be processed in order, and the length of the sequence is read; each bucket in the bucket list is traversed, and the remaining capacity and total capacity of the bucket are read; the degree of adaptation is calculated based on the length of the sequence, the remaining capacity and the total capacity. If the remaining capacity < sequence length, it is determined that the bucket cannot accommodate the sequence, the bucket is skipped, and the degree of adaptation is not calculated; if the remaining capacity ≥ sequence length, the degree of adaptation is calculated according to the degree of adaptation calculation formula, the degree of adaptation = 1-(bucket's remaining capacity-sequence length) / bucket total capacity, the calculation result is bound with the bucket identifier, and stored in the degree of adaptation candidate set. The maximum value of the degree of adaptation is selected from the degree of adaptation candidate set, and the corresponding candidate bucket is found; if the maximum value of the degree of adaptation ≥ preset degree of adaptation threshold, it is determined that the candidate bucket is the optimal allocation target, and the existing bucket allocation process is entered; if the maximum value of the degree of adaptation < preset degree of adaptation threshold or the degree of adaptation candidate set is empty, the new bucket creation process is triggered, and the total capacity of the new bucket is determined according to the sequence length and the rounding granularity rule, and the remaining capacity of the new bucket is initialized as the total capacity.

[0080] If the sequence is allocated to an existing bucket, the identifier of the sequence is written into the allocated sequence list of the bucket, the new remaining capacity of the bucket is calculated, and the remaining capacity field in the bucket capacity list is updated; if it is allocated to a newly created bucket, the identifier of the sequence is written into the allocated sequence list of the new bucket, the remaining capacity of the new bucket is calculated, and the total capacity, the remaining capacity, the allocated sequence list and other information of the new bucket are added to the bucket capacity list. The allocated bucket capacity list and the allocation result of the long sequence are obtained.

[0081] Step A30, generating a dynamic programming model based on the common sequence and the maximum total filling capacity of the barrels, and obtaining a common sequence allocation result according to the dynamic programming model.

[0082] Step A40, determining a first allocation result according to the common sequence allocation result and the long sequence allocation result.

[0083] In this embodiment, for the common sequence, the sequences in the common sequence are numbered in order, the length of each sequence is extracted and stored in an ordered length array; all remaining capacity values are extracted from the barrel remaining capacity list, and after removing duplicate values, they are sorted. A two-dimensional array dp[i][j] is created, which represents the maximum total length that the first i common sequences can fill under the constraint that the remaining capacity of all barrels does not exceed j, where i represents the number of common sequences to be allocated, j represents the remaining capacity threshold of the barrel, and the value range is 0 to the maximum remaining capacity of all barrels. In the initial state, dp[0][j]=0, when there is no common sequence to be allocated, no matter what the remaining capacity threshold j of the barrel is, the total length that can be filled is 0, and the remaining space of all barrels is not utilized. The number of rows of the two-dimensional array is the number of common sequences + 1, and the number of columns is the maximum remaining capacity of all barrels + 1, all elements of the first row (i=0) are assigned a value of 0.

[0084] For the i-th normal sequence, with length len_i, traverse all the bucket remaining capacity thresholds j, when the bucket remaining capacity threshold j is less than the current sequence length len_i, the sequence cannot be allocated to any bucket with remaining capacity ≤ j, so the maximum filling amount of the first i sequences is equal to the maximum filling amount of the first i-1 sequences, i.e. inherit the previous state, dp[i][j]=dp[i-1][j]. When the bucket remaining capacity threshold j is greater than or equal to the current sequence length len_i, the filling amount = the maximum filling amount of the first i-1 sequences at threshold j, dp[i][j]=dp[i-1][j] when the sequence is not allocated; the filling amount = the maximum filling amount of the first i-1 sequences at threshold j-len_i + the current sequence length, i.e. dp[i][j]=dp[i-1][j-len_i]+len_i, when the sequence is allocated, i.e. reserve len_i space for the current sequence, and the first i-1 sequences fill the remaining j-len_i space. Compare the size of dp[i-1][j] and dp[i-1][j-len_i]+len_i, dp[i][j]=max(dp[i-1][j], dp[i-1][j-len_i]+len_i). Exemplarily, if i=3 (the 3rd sequence, length 20), j=50, then dp[3]

[50] =max(dp[2]

[50] , dp[2]

[30] +20), if dp[2]

[50] =40, dp[2]

[30] =25, then dp[3]

[50] =max(40, 25+20)=45, indicating that allocating the 3rd sequence can improve the total filling amount.

[0085] After the calculation of each row is completed, invalid states are deleted. For the bucket remaining capacity j, only the state with j≥ the minimum normal sequence length is retained, and invalid states are removed; for dp[i][j], if its value is less than dp[i][j-1]×α, where α is a pruning coefficient, it is considered that the state is worthless, and is directly removed, reducing the calculation amount. Output result: dynamic programming optimal allocation scheme of normal sequences, and updated bucket remaining capacity list.

[0086] After the dynamic programming calculation is completed, dp[len(L_normal)][max_cap] is the maximum total filling amount of the normal sequence, and the specific allocation barrel of each sequence is derived by backtracking the dp array. Starting from i=len(the number of normal sequence sequences) and j=max_cap, compare dp[i][j] and dp[i-1][j]. If dp[i][j]>dp[i-1][j], the i-th sequence is allocated, and the barrel with a remaining capacity greater than or equal to the sequence length and a total capacity closest to j after filling is found. If dp[i][j]=dp[i-1][j], the i-th sequence is not allocated or the allocation has no gain, and the sequence is skipped. For the i-th sequence determined to be allocated, the remaining capacity list of the barrel is traversed, and the barrel with the smallest capacity and a remaining capacity greater than or equal to len_i is selected. The sequence is allocated to the barrel, and the remaining capacity of the barrel is updated. When i=0, the backtracking is completed, and the barrel allocation mapping table of all normal sequences is obtained. The global optimal allocation of the normal sequence is modeled by dynamic programming, the state space is reduced by heuristic pruning, and the calculation efficiency is improved under the premise of ensuring the approximate optimal.

[0087] After the allocation of the long sequence and the normal sequence is completed, the preliminary allocation result of the sequence is determined, including an allocation mapping dictionary, wherein the key is the sequence identifier, and the value is the corresponding barrel identifier; and the filled sequence list of each barrel, which stores the sequence identifier and length in the allocation order. After the long sequence heuristic pre-allocation and the normal sequence dynamic programming allocation, the remaining capacity set of all barrels is determined, and the total capacity of each barrel is associated.

[0088] For each bucket B, identified as b_id, calculate the bucket utilization = total filled length of the bucket / total capacity of the bucket, set a utilization threshold; traverse all buckets, and put the buckets with utilization < threshold into a low utilization bucket list. For each bucket in the low utilization bucket list, extract all sequence sets in the bucket; traverse each sequence in the sequence set, exclude the original bucket, and traverse all other valid buckets as candidate target buckets B_target; calculate the migration benefit, migration benefit = promotion value - loss value; original bucket utilization loss value = utilization of bucket B before migration - utilization of bucket B after migration, filled length of bucket B after migration = original filled length - len_S, where len_S is the sequence length; loss value = (L_B / T_B) - ((L_B - len_S) / T_B) = len_S / T_B, L_B is the original filled length of bucket B, and T_B is the total capacity of bucket B; target bucket utilization promotion value = utilization of bucket B_target after migration - utilization of bucket B_target before migration, filled length of bucket B_target after migration = original filled length + len_S; promotion value = ((L_target + len_S) / T_target) - (L_target / T_target), L_target is the original filled length of bucket B_target, and T_target is the total capacity of bucket B_target.

[0089] If the migration benefit > 0, perform sequence migration, remove the sequence S from the sequence list of the bucket B and add it to the sequence list of the bucket B_target; at the same time, update the remaining capacity of the bucket B and the bucket B_target; if the migration benefit ≤ 0, give up the migration of the sequence and keep it in the original bucket B. By migrating the sequences in the inefficient buckets to other buckets, the global utilization is improved, and the number of inefficient buckets is reduced.

[0090] Traverse all buckets, if the filled length of a certain bucket = 0, determine that it is an empty bucket; if the empty bucket is a preset bucket, mark it as idle, keep the bucket identifier but do not participate in sequence splicing in the future, and reuse it when a new sequence is allocated; if it is a dynamically created new bucket, delete the bucket and remove it from the bucket list.

[0091] Optionally, if there are multiple low utilization buckets, and the total length of the filled sequences is ≤ the total capacity of a certain preset bucket, the sequences can be merged into a single bucket, and the original low utilization buckets are deleted.

[0092] Based on the final allocation result, the filled sequence list of each bucket is traversed, and each sequence is processed one by one in the arrangement order of the sequence in the bucket; for the token of a single sequence, the token list of the sequence is read, the token list corresponding to the current sequence identifier is called, and a preset end symbol is appended at the end of the token list; a position number is allocated, an initial number counter is initialized to a preset starting value, such as 0, the processed token list is traversed, and a unique position number is allocated to each token, and the counter is incremented by 1 each time; after the processing of a single sequence is completed, the counter is reset; the sequence identifier, the processed token list, and the associated data of the position number mapping are obtained.

[0093] For each bucket, the processed token list of the filled sequence of the bucket is spliced in order to form the original target token sequence of the bucket; the length Total_Len of the original target token sequence is counted, and if Total_Len≥the preset maximum sequence length, the original target token sequence is the final target sequence of the bucket; if Total_Len<the preset maximum sequence length, the padding length Pad_Len =the preset maximum length-Total_Len is calculated, and Pad_Len padding symbols are appended at the end of the original target token sequence to form the final target sequence. An array Attention_Mask with the same length as the final target sequence is constructed; for the first Total_Len positions, that is, the effective token + end symbol, the mask value is set to 1; for the last Pad_Len positions, that is, the padding symbol, the mask value is set to 0; for all valid buckets, the above steps are repeated to generate the final target sequence and the attention mask corresponding to each bucket, and the target sequence set used for training is obtained.

[0094] In this embodiment, all possible allocation combinations are traversed through state transition, ensuring that the allocation scheme with the maximum total padding amount is found within the state space, solving the problem of local optimization but global suboptimality; the heuristic pruning reduces the time complexity of dynamic programming from O(n×max_cap) to O(n×max_cap×β) by filtering invalid states and low-value states, where n is the number of ordinary sequences, and β is the pruning coefficient, which improves the calculation efficiency under the premise of ensuring approximate optimality; the dynamic programming allocation of the ordinary sequence is based on the remaining space of the bucket after the pre-allocation of the long sequence, which not only avoids the resource competition between the long sequence and the ordinary sequence, but also fills the fragmented space left after the allocation of the long sequence, improving the overall bucket utilization.

[0095] The application provides a large language model training data packaging device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the large language model training data packaging method in the above-mentioned embodiment one.

[0096] Reference will be made to the following Figure 7 which shows a structural schematic diagram of a large language model training data packaging device suitable for being used to implement the embodiments of the application. The large language model training data packaging device in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, personal digital assistants (PDA), tablet computers (PAD), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 7 The large language model training data packaging device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the application.

[0097] As Figure 7 As shown, the large language model training data packaging device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the large language model training data packaging device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the large language model training data packaging device to communicate wirelessly or wiredly with other devices to exchange data. Although a large language model training data packaging device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0098] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0099] The large language model training data packaging device provided in this application, employing the large language model training data packaging method described in the above embodiments, can solve the technical problem of how to allocate sequences corresponding to training data, thereby reducing resource waste caused by invalid padding. Compared with the prior art, the beneficial effects of the large language model training data packaging device provided in this application are the same as those of the large language model training data packaging method provided in the above embodiments, and other technical features in this large language model training data packaging device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0100] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0102] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the large language model training data packaging method described in the above embodiments.

[0103] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0104] The aforementioned computer-readable storage medium may be included in a large language model training data packaging device; or it may exist independently and not be assembled into the large language model training data packaging device. The aforementioned computer-readable storage medium carries one or more programs that, when executed by the large language model training data packaging device, cause the large language model training data packaging device to: sort the initial sequences corresponding to the training data in descending order to obtain an ordered list; compare the sequence length of the initial sequence with the capacity of a preset bucket according to the sorting in the ordered list; identify the bucket with a capacity greater than or equal to the sequence length and the smallest remaining capacity as the target bucket, and place the corresponding initial sequence into the target bucket; and concatenate the initial sequences in one of the buckets to obtain a target sequence, which is used for large language model training.

[0105] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the client computer, partially on the client computer, as a standalone software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the client computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0108] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described method for packaging training data for large language models. This method solves the technical problem of how to allocate sequences corresponding to training data, thereby reducing resource waste caused by invalid padding. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the large language model training data packaging method provided in the above embodiments, and will not be elaborated upon here.

[0109] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.< / eos>

Claims

1. A method for packing training data of a large language model, characterized in that, The large language model training data packaging method comprises: sorting initial sequences corresponding to training data in descending order to obtain an ordered list; comparing the sequence length of the initial sequence with the bucket capacity of a preset bucket according to the order in the ordered list; confirming a bucket with the largest remaining capacity as a target bucket when the bucket capacity is greater than or equal to the sequence length, and placing the corresponding initial sequence into the target bucket; splicing the initial sequences in one bucket to obtain a target sequence, wherein the target sequence is used for large language model training. 2.The large language model training data packing method of claim 1, wherein, After the step of comparing the sequence length of the initial sequence with the bucket capacity of a preset bucket according to the order in the ordered list, the large language model training data packaging method further comprises: if the sequence length is greater than all the bucket capacities, creating a bucket according to the size of the sequence length; placing the initial sequence into the newly created bucket and updating the bucket list. 3.The large language model training data packing method of claim 1, wherein, The step of confirming a bucket with the largest remaining capacity as a target bucket when the bucket capacity is greater than or equal to the sequence length, and placing the corresponding initial sequence into the target bucket, comprises: determining the sequence length of the first sequence in the ordered list; traversing the preset bucket to confirm a candidate bucket when the remaining bucket capacity of the preset bucket is greater than or equal to the sequence length; determining the bucket with the smallest remaining bucket capacity in the remaining bucket capacity list of the candidate bucket based on a binary search method to obtain the target bucket; placing the first sequence into the target bucket and removing the first sequence from the ordered list until the ordered list is empty. 4.The large language model training data packing method of claim 3, wherein, Before the step of traversing the preset bucket to confirm a candidate bucket when the remaining bucket capacity of the preset bucket is greater than or equal to the sequence length, the large language model training data packaging method further comprises: determining the remaining bucket capacity of each preset bucket, arranging the remaining bucket capacities in ascending order to obtain a remaining bucket capacity list; updating the remaining bucket capacity according to the length of the sequence when the sequence is placed into the bucket, and updating the ordered list. 5.The large language model training data packing method of claim 1, wherein, The step of splicing the initial sequences in one bucket to obtain a target sequence, wherein the target sequence is used for large language model training, comprises: traversing the bucket to splice the initial sequences according to the sequence placement order to obtain the target sequence; obtaining the sum of the sequence lengths of each initial sequence to obtain the total length of the target sequence; if the total length is less than a preset maximum sequence length, filling the target sequence according to an attention mask. 6.The large language model training data packing method of claim 5, wherein, After the step of traversing the bucket to splice the initial sequences according to the sequence placement order to obtain the target sequence, the large language model training data packaging method further comprises: traversing each token of the initial sequence to add an end symbol after the last token of the initial sequence; assigning a position number to each token according to the order and a preset starting value; resetting the number counter to the starting value when the numbering of a single initial sequence is completed. 7.The large language model training data packing method of claim 1, wherein, The large language model training data packing method further comprises the following steps after the step of comparing the sequence length of the initial sequence with the bucket capacity of the preset bucket according to the sorting in the ordered list: When the first bucket utilization is less than a preset first bucket utilization threshold, the initial sequence is divided into a long sequence and a normal sequence according to a preset long sequence threshold; An adaptation degree of the long sequence and the preset bucket is determined, the bucket with the highest adaptation degree is determined as a candidate bucket, and a long sequence allocation result is obtained; A dynamic programming model is generated based on the normal sequence and the maximum total filling capacity of the bucket, and a normal sequence allocation result is obtained according to the dynamic programming model; A first allocation result is determined according to the normal sequence allocation result and the long sequence allocation result. 8.The large language model training data packing method of claim 7, wherein, The large language model training data packing method further comprises the following steps before the step of splicing the initial sequence in one of the buckets to obtain a target sequence: A second bucket utilization is determined based on the ratio of the filled length to the total capacity of the bucket; The bucket with the second bucket utilization less than a preset second bucket utilization threshold is determined as a low utilization bucket; The sequences in the low utilization bucket are re-allocated, and a migration benefit is determined according to the allocation result; If the migration benefit is positive, the first allocation result and the bucket remaining capacity are updated according to the allocation result to obtain a second allocation result. 9.A large language model training data packing device, characterized in that, The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the large language model training data packing method according to any one of claims 1 to 8.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the large language model training data packing method according to any one of claims 1 to 8.

Images