Large model data loading and long text processing method and system

Abstract

The invention provides a large model data loading and long text processing method and system. The large model data loading and long text processing method comprises the steps that S1, data blocks are loaded from a storage medium batch by batch in a streaming reading mode; s2, whether the text length of each document in the data blocks is larger than a preset value or not is judged, if yes, the documents are divided, and texts meeting preset requirements are obtained; s3, whether the length of the text meeting the preset requirement is larger than a preset value or not is judged, and if yes, oversampling processing is conducted on the text; s4, splicing a plurality of texts in each document to obtain a new long text, and performing word segmentation processing on the obtained new long text to obtain a text sample; and S5, storing the obtained text samples into a cache Ccache in a memory, and when the number of the text samples in the cache Ccache reaches the batch size Bsize, outputting the content in the cache to a large model for training.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and more specifically, to a method and system for loading large model data and processing long texts. More specifically, it relates to a method and system for preprocessing and loading data for large-scale language models in long text training. Background Art

[0002] Processing long texts is a challenge in natural language processing. Traditional text processing methods often truncate or segment long texts, resulting in a loss of contextual information, which in turn affects model learning. Especially in large-scale pre-trained models, efficiently processing long texts while preserving their contextual information is key to improving model performance.

[0003] Furthermore, in real-world applications, data may be stored in a variety of formats, such as JSON and Parquet. Processing these formats often requires different parsing methods, increasing processing complexity. Existing text preprocessing methods are typically optimized for a single data format and lack unified processing for multiple data formats. This results in low data loading efficiency and inflexible handling of diverse data sources.

[0004] Patent document CN118428440A (application number: 202410510886.8) discloses a long text large model training method and device, which belongs to the field of model training technology. The method includes: obtaining the original input matrix corresponding to the long text data; performing feature enhancement and multi-semantic transfer processing on the original input matrix to obtain a first matrix; performing feature extraction and data enhancement processing on the first matrix to obtain a second matrix; calculating the query matrix, key matrix and value matrix based on the second matrix; calculating the self-attention weight based on the query matrix, key matrix and value matrix; shifting the query matrix, key matrix and value matrix; calculating the gradient based on the self-attention weight, and updating the model parameters using the gradient descent method according to the loss function of the model; repeating the above steps until the model converges to obtain a long text large model. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for loading large model data and processing long text.

[0006] A method for loading large model data and processing long text provided by the present invention includes:

[0007] Step S1: Loading data blocks from the storage medium in batches using a streaming reading method;

[0008] Step S2: determining whether the text length of each document in the data block is greater than a preset value; if so, dividing the document to obtain text that meets the preset requirements;

[0009] Step S3: determining whether the length of the text meeting the preset requirements is greater than a preset value; if so, oversampling the text;

[0010] Step S4: concatenate multiple texts in each document to obtain a new long text, and perform word segmentation on the obtained new long text to obtain a text sample;

[0011] Step S5: Store the obtained text sample into the cache C in the memory cache In the cache C cache The number of text samples in the batch size reaches B size When the data is processed, the content in the cache is output to the large model for training.

[0012] Preferably, the step S1 includes:

[0013] For each data source, read the data block Represents the data of batch i, including B size A sample of documents; the streamed data is processed as follows:

[0014]

[0015] Among them, T i Represents a single document sample in a data block.

[0016] Preferably, step S2 includes:

[0017] Assume that the length of document D is |D|, and the document segmentation process is performed according to the following formula: If |D|>L max , then divide the document D into multiple segments T'1, T'2, ..., T' k ,in:

[0018] T′ i =D[(i-1)·L max , i·L max ]

[0019] Among them, T′ i represents the i-th text segment, k is the total number of segments after segmentation, and satisfies:

[0020]

[0021] Preferably, step S3 includes:

[0022] The oversampling operation will be based on the set oversampling ratio ρ oversample Repeat the text multiple times to generate multiple copies; the oversampling process is expressed as:

[0023]

[0024] Preferably, step S4 includes:

[0025] Concatenate multiple short texts in each document. Assume that the input text set is T 11 ,T 22 ,…,T kk , where T ii is the oversampled text; the concatenation operation combines these text fragments into a new long text and adds a separator sep_token between each text:

[0026] T concat =T 11 concat[sep_token]concatT 22 …T kk

[0027] The long text will be fed into the word segmenter for word segmentation, and the resulting token sequence is:

[0028] tokens(T concat )=tokenize(T concat ).

[0029] Preferably, step S5 includes:

[0030] After each reading and word segmentation operation, the processed text sample is stored in the cache C in the memory cache In; whenever cache C cache The number of text samples in the batch size reaches B size , the contents of the cache will be output to the model for training:

[0031] if|C cache |≥B size , output·block

[0032] After each output, a portion of the cache is cleared and the remaining portion is retained for further processing;

[0033] C cache ←C cache \{d1,d2,...,d output}

[0034] Among them, {d1,d2,…,d output} represents the output data block.

[0035] A large model data loading and long text processing system provided by the present invention includes:

[0036] Module M1: uses streaming reading to load data blocks from the storage medium in batches;

[0037] Module M2: Determine whether the text length of each document in the data block is greater than a preset value. If so, divide the document to obtain text that meets the preset requirements.

[0038] Module M3: Determine whether the length of the text that meets the preset requirements is greater than a preset value. If it is greater than the preset value, oversampling the text is performed;

[0039] Module M4: concatenates multiple texts in each document to obtain a new long text, and performs word segmentation on the new long text to obtain a text sample;

[0040] Module M5: Store the obtained text sample into the cache C in the memory cache In the cache C cache The number of text samples in the batch size reaches B size When the data is processed, the content in the cache is output to the large model for training.

[0041] Preferably, the module M1 includes:

[0042] For each data source, read the data block Represents the data of batch i, including B size A sample of documents; the streamed data is processed as follows:

[0043]

[0044] Among them, T i Represents a single document sample in a data block;

[0045] The module M2 includes:

[0046] Assume that the length of document D is |D|, and the document segmentation process is performed according to the following formula: If |D|>L max , then divide the document D into multiple segments T′1, T′2, ..., T′ k ,in:

[0047] T′ i =D[(i-1)·L max , i·L max ]

[0048] Among them, T′ i represents the i-th text segment, k is the total number of segments after segmentation, and satisfies:

[0049]

[0050] Preferably, the module M3 includes:

[0051] The oversampling operation will be based on the set oversampling ratio ρ oversample Repeat the text multiple times to generate multiple copies; the oversampling process is expressed as:

[0052]

[0053] The module M4 includes:

[0054] Concatenate multiple short texts in each document. Assume that the input text set is T 11 ,T 22 ,…,T kk , where T ii is the oversampled text; the concatenation operation combines these text fragments into a new long text and adds a separator sep_token between each text:

[0055] T concat =T 11 concat[sep_token]concatT 22 …T kk

[0056] The long text will be fed into the word segmenter for word segmentation, and the resulting token sequence is:

[0057] tokens(T concat )=tokenize(T concat ).

[0058] Preferably, the module M5 includes:

[0059] After each reading and word segmentation operation, the processed text sample is stored in the cache C in the memory cache In; whenever cache C cache The number of text samples in the batch size reaches B size , the contents of the cache will be output to the model for training:

[0060] if|C cache |≥B size , output·block

[0061] After each output, a portion of the cache is cleared and the remaining portion is retained for further processing;

[0062] C cache ←C cache \{d1,d2,...,d output}

[0063] Among them, {d1,d2,…,doutput} represents the output data block.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. High data loading efficiency: Through streaming reading and block processing, we can avoid memory overflow caused by loading large amounts of data at once, while ensuring efficient data processing.

[0066] 2. Long text processing capabilities: Through operations such as document segmentation, text concatenation and word segmentation, and oversampling, the learning ability of large models for long texts is improved, especially in terms of diverse text samples and contextual understanding;

[0067] 3. Memory optimization: By monitoring memory usage in real time, dynamically adjusting data batch sizes or clearing caches, we ensure that the system can run stably and efficiently when processing large amounts of data.

[0068] 4. Flexibility and scalability: unified processing of data files in different formats, such as Parquet and JSON, improves the flexibility and scalability of the system and meets the processing requirements of various data sources;

[0069] 5. The present invention adopts an oversampling strategy to enhance the scarcity of long text samples, ensuring that the model is exposed to more long text instances during training, increasing the diversity of training data, and improving the robustness and accuracy of the model for long texts;

[0070] 6. The memory optimization mechanism ensures efficient memory usage when processing large-scale data by monitoring memory usage in real time. By dynamically adjusting the batch size or clearing the cache, the present invention can avoid memory overflow and ensure the stability and efficiency of the model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0072] Figure 1 Flowchart of the improved large model data loading and long text processing method. DETAILED DESCRIPTION

[0073] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0074] Example 1

[0075] The present invention provides an improved large-model data loading and text processing method and system, which can uniformly process multiple data formats, and enhance the contextual understanding ability of the large model in long text training by splicing multiple short texts into a long text for word segmentation, and then combining oversampling and streaming reading, thereby improving the model's learning effect and reasoning ability for long content. In addition, in order to cope with the memory overflow problem that may occur during large-scale data processing, the present invention also introduces a memory optimization mechanism. This mechanism dynamically monitors memory usage, and when the memory usage exceeds the set threshold, it automatically clears the cache to ensure that memory resources are efficiently utilized, avoiding system crashes or reduced processing efficiency due to excessive memory usage. Through these measures, the present invention not only improves the learning ability of the large model for long texts, but also effectively optimizes memory usage, improves data processing efficiency and the stability of model training.

[0076] The improved large model data loading and text processing method, such as Figure 1 As shown, including:

[0077] Step 1: Data loading and file parsing;

[0078] Use streaming reading to load data from the storage medium in batches;

[0079] The data is read by dividing the file into blocks, and the size of each data block is B. size , where B size The batch size is set in the configuration. Each data block read contains several samples for subsequent text splicing, word segmentation and oversampling processing. For each data source, such as Parquet, JSON, etc., the data block read Represents the data of batch i, including B size The data read by streaming is processed as follows:

[0080]

[0081] Among them, T i Represents a single text sample in a data block.

[0082] This paper effectively improves data loading efficiency through streaming reading and batch processing, can process large-scale data sets, and avoids memory overflow problems caused by loading all data at once; it is particularly suitable for large-scale language model training, supports data sources in different formats (such as Parquet, JSON, etc.), and greatly improves the flexibility and scalability of data processing.

[0083] Step 2: Document segmentation;

[0084] Since large-scale models have a limit on the length of input text, long documents need to be split into smaller segments, and the length of each segment cannot exceed the maximum length L supported by the word segmenter. max Assume that the length of document D is |D|, and the document segmentation process is performed according to the following formula: If |D|>L max , then divide the document D into multiple segments T′ i , T′2,...,T′ k ,in:

[0085] T′ i =D[(i-1)·L max , i·L max ]

[0086] Here, T′ i represents the i-th text segment, k is the total number of segments after segmentation, and satisfies:

[0087]

[0088] Step 3: Oversampling of long texts;

[0089] For lengths exceeding the set threshold L threshold The text is oversampled to enhance the diversity of the data. The oversampling operation will be based on the set oversampling ratio ρ oversample The text is repeated multiple times to generate multiple copies, which are then fed into the subsequent processing pipeline.

[0090] The oversampling process can be expressed as:

[0091]

[0092] Step 4: Text splicing and word segmentation;

[0093] Concatenate multiple short texts in each document. Assume that the input text set is T 11 ,T 22 ,…,T kk , where T ii is the oversampled text. The concatenation operation combines these text fragments into a new long text and adds a separator sep_token between each text:

[0094] T concat =T 11 concat[sep_token]concatT 22 …T kk

[0095] The long text will be fed into the word segmenter for word segmentation, and the resulting token sequence is:

[0096] tokens(T concat )=tokenize(T concat )

[0097] By combining long text segmentation and concatenation, this paper fully leverages the contextual information of long texts, improving the learning ability of large models when processing long texts. By concatenating multiple short texts into a single long text and processing it through a word segmenter, the model's ability to understand and generate long texts is effectively enhanced, particularly in tasks with long context dependencies, significantly improving model performance.

[0098] Step 5: Cache management and data block generation;

[0099] After each reading and word segmentation operation, the processed text sample is stored in the cache C in the memory cache The contents stored in the cache include input_ids, attention_mask and position_ids. Whenever cache C cache The number of text samples in the batch size reaches B size When the cache content is output to the model for training, that is:

[0100] if|C cache ||≥B size , output·block

[0101] The output includes:

[0102] {input_ids,attention_mask,position_ids}

[0103] After each output, a portion of the cache is cleared and the remaining portion is retained for continued processing.

[0104] C cache ←C cache \{d1,d2,...,d output}

[0105] Step 5: Memory optimization;

[0106] In order to prevent memory overflow, each time data is loaded, the cache will be updated to: cache ←C cache \{d1, d2, ..., d output}

[0107] Among them, the output data block {d1,d2,…,d output} will be fed into the model training, and the remaining part will remain in the cache waiting for the next processing. This process ensures that the system can maintain efficient memory usage when processing large data. The memory management strategy is:

[0108] ifM current ≥M max ,clean cache

[0109] This can effectively control memory usage and avoid memory overflow.

[0110] The present invention also provides an improved large model data loading and text processing system, which can be implemented by executing the process steps of the improved large model data loading and text processing method. That is, those skilled in the art can understand the improved large model data loading and text processing method as a preferred implementation of the improved large model data loading and text processing system.

[0111] Through efficient data loading, long text processing, oversampling enhancement and memory optimization, this invention not only improves the efficiency of large model training, but also enhances the model's learning ability for long texts, ensures memory stability in large-scale data set processing, and thus improves the model's performance in complex tasks.

[0112] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0113] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for loading large model data and processing long text, characterized in that: include: Step S1: Loading data blocks in batches from the storage medium using a streaming reading method; Step S2: determining whether the text length of each document in the data block is greater than a preset value; if so, dividing the document to obtain text that meets the preset requirements; Step S3: determining whether the length of the text meeting the preset requirements is greater than a preset value; if so, oversampling the text; Step S4: concatenate multiple texts in each document to obtain a new long text, and perform word segmentation on the obtained new long text to obtain a text sample; Step S5: Store the obtained text sample into the cache C in the memory cache In the cache C cache The number of text samples in the batch size reaches B size When the data is processed, the content in the cache is output to the large model for training.

2. The large model data loading and long text processing method according to claim 1 is characterized in that: The step S1 comprises: For each data source, read the data block Represents the data of batch i, including B size A sample of documents; the streamed data is processed as follows: Among them, T i Represents a single document sample in a data block.

3. The large model data loading and long text processing method according to claim 1 is characterized in that: The step S2 comprises: Assume that the length of document D is |D|, and the document segmentation process is performed according to the following formula: If |D|>L max , then divide the document D into multiple segments T'1, T'2, ..., T' k ,in: T’ i =D[(i-1)·L max ,i·L max ] Among them, T' i represents the i-th text segment, k is the total number of segments after segmentation, and satisfies:

4. The large model data loading and long text processing method according to claim 1 is characterized in that: The step S3 comprises: The oversampling operation will be based on the set oversampling ratio ρ oversample Repeat the text multiple times to generate multiple copies; the oversampling process is expressed as:

5. The large model data loading and long text processing method according to claim 1 is characterized in that: The step S4 comprises: Concatenate multiple short texts in each document. Assume that the input text set is T 11 ,T 22 ,…,T kk , where T ii is the oversampled text; the concatenation operation combines these text fragments into a new long text and adds a separator sep_token between each text: T concat =T 11 concat[sep_token]concatT 22 ...T kk The long text will be fed into the word segmenter for word segmentation, and the resulting token sequence is: tokens(T concat )=tokenize(T concat )。 6. The large model data loading and long text processing method according to claim 1 is characterized in that: The step S5 comprises: After each reading and word segmentation operation, the processed text sample is stored in the cache C in the memory cache In; whenever cache C cache The number of text samples in the batch size reaches B size , the contents of the cache will be output to the model for training: if|C cache |≥B size ,output·block After each output, a portion of the cache is cleared and the remaining portion is retained for further processing; C cache ←C cache \{d1,d2,...,d output } Among them, {d1,d2,…,d output } represents the output data block.

7. A large model data loading and long text processing system, characterized by: include: Module M1: uses streaming reading to load data blocks from the storage medium in batches; Module M2: Determine whether the text length of each document in the data block is greater than a preset value. If so, divide the document to obtain text that meets the preset requirements. Module M3: Determine whether the length of the text that meets the preset requirements is greater than a preset value. If it is greater than the preset value, oversampling the text is performed; Module M4: concatenates multiple texts in each document to obtain a new long text, and performs word segmentation on the new long text to obtain a text sample; Module M5: Store the obtained text sample into the cache C in the memory cache In the cache C cache The number of text samples in the batch size reaches B size When the data is processed, the content in the cache is output to the large model for training.

8. The large model data loading and long text processing system according to claim 7 is characterized in that: The module M1 includes: For each data source, read the data block Represents the data of batch i, including B size A sample of documents; the streamed data is processed as follows: Among them, T i Represents a single document sample in a data block; The module M2 includes: Assume that the length of document D is |D|, and the document segmentation process is performed according to the following formula: If |D|>L max , then divide the document D into multiple segments T'1, T'2, ..., T' k ,in: T′ i =D[(i-1)·L max ,i·L max ] Among them, T′ i represents the i-th text segment, k is the total number of segments after segmentation, and satisfies:

9. The large model data loading and long text processing system according to claim 7, characterized in that: The module M3 includes: The oversampling operation will be based on the set oversampling ratio ρ iversample Repeat the text multiple times to generate multiple copies; the oversampling process is expressed as: The module M4 includes: Concatenate multiple short texts in each document. Assume that the input text set is T 11 ,T 22 ,…,T kk , where T ii is the oversampled text; the concatenation operation combines these text fragments into a new long text and adds a separator sep_token between each text: T concat =T 11 concat[sep_token]concatT 22 ...T kk The long text will be fed into the word segmenter for word segmentation, and the resulting token sequence is: tokens(T concat )=tokenize(T concat ) 10. The large model data loading and long text processing system according to claim 7, characterized in that: The module M5 includes: After each reading and word segmentation operation, the processed text sample is stored in the cache C in the memory cache In; whenever cache C cache The number of text samples in the batch size reaches B size , the contents of the cache will be output to the model for training: if|C cache |≥B size ,output·block After each output, a portion of the cache is cleared and the remaining portion is retained for further processing; C cache ←C cache \{d1,d2,...,d output } Among them, {d1,d2,…,d output } represents the output data block.

Images