Large language model data efficient preprocessing method and system based on pipeline parallelism

By dividing the preprocessing task of large language model data into multiple undecoupled links and adopting parallel processing methods, each batch of data is divided into micro-batch data, which solves the problem of low preprocessing efficiency of large language model data, and realizes the implementation of the Efficient data processing and model training preparation.

CN118963992APending Publication Date: 2024-11-15NO 15 INST OF CHINA ELECTRONICS TECH GRP
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410933105.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Large language model data preprocessing is inefficient, especially when processing industry private data from different formats and sources, it requires a lot of time and manpower.

Method used

Using the data preprocessing method based on pipeline parallelism, the data preprocessing task is divided into multiple undecoupled processing links, and each batch of data is divided into multiple micro-batch data through parallel processing method to realize the same time period. Parallel operation of different processing links.

Benefits of technology

It significantly improves the efficiency of preprocessing data of large language models, shortens model training waiting time, and ensures unified data format and complete information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963992A_ABST
    Figure CN118963992A_ABST
Patent Text Reader

Abstract

The invention discloses an efficient preprocessing method and system for large language model data based on pipeline parallelism, and belongs to the technical field of natural language process.The method comprises the steps that a pipeline processing framework is constructed, a data preprocessing task is divided into a plurality of processing links, and the processing links cannot be decoupled and are executed in sequence; the method comprises the following steps: dividing each batch of data in data processing into a plurality of micro-batch data by adopting a parallel processing method on the basis of a pipeline processing framework, taking the micro-batch data as minimum units of data processing, and starting data processing in sequence; each time the micro-batch data completes one processing link, the next processing link is automatically entered, and parallel operation of different data processing links in the same time period is achieved. According to the method, a data preprocessing process is summarized into eight processing links which are executed in sequence; a parallel processing method is further designed, parallel operation of different data processing links in the same time period is achieved, and the data preprocessing efficiency can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically to a pipeline-parallel-based large language model data efficient preprocessing method and system. Background Art

[0002] The big language model is a model in natural language processing technology, such as ChatGPT, etc. It belongs to the text generation mode and can achieve high-quality natural language understanding and generation. With the support of "big model + big data + big computing power", the big language model can complete a variety of tasks through natural language interaction, and has multi-scenario, multi-purpose, and interdisciplinary task processing capabilities. It can play an important role in many fields such as economy, law, and society. Through the process of pre-training and fine-tuning, the big language model can understand and generate natural language text, and play a role in various application scenarios, such as automatic question and answer, intelligent customer service, speech recognition, machine translation, etc., pushing artificial intelligence technology to a new height.

[0003] Before training a large model, it is necessary to pre-process a large amount of data with different capabilities and task forms. At the same time, the data should be analyzed and labeled in combination with the training objectives to facilitate flexible selection of data during training. In model training, data is king. Pre-training of large models requires learning sufficient knowledge from massive text data and storing it in its model parameters. From the source point of view, text data is mainly divided into two types: open source general data and industry private data. The characteristic of general data is that the volume is very large. Usually, an open source general data set is a massive, unstructured, multilingual web page data set. Different open source data sets have different data format definitions and extraction methods, but the format of data sets from the same source is fixed. Industry private data is unique to a certain field, such as conversations, books, codes, technical reports, and instruction documents in the military field. Unlike the fixed format of general data sets, industry private data is very troublesome to process. Due to the various data sources and inconsistent formats, it is even time-consuming to process one piece of data in one way.

[0004] Therefore, how to improve the efficiency of large language model data preprocessing is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a method and system for efficient preprocessing of large language model data based on pipeline parallelism, which at least solves some of the above-mentioned technical problems. The data preprocessing task is divided into multiple processing links, and the processing links cannot be decoupled and are executed in sequence. A parallel processing method is also designed to realize parallel operation of different data processing links in the same time period, which can effectively improve the efficiency of large language model data preprocessing.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a method for efficient preprocessing of large language model data based on pipeline parallelism, the method comprising:

[0008] Build a pipeline processing framework to divide the data preprocessing task into multiple processing links. Each processing link cannot be decoupled and is executed in sequence.

[0009] Based on the pipeline processing framework, a parallel processing method is adopted to divide each batch of data in data processing into multiple micro-batch data. As the smallest unit of data processing, data processing is started in sequence; each micro-batch data automatically enters the next processing link after completing each processing link, thereby realizing parallel operation of different data processing links in the same time period.

[0010] Preferably, the data preprocessing task is divided into eight processing steps to be executed sequentially, namely: deduplication, detoxification, cleaning, classification, indicator statistics, intra-class performance analysis, inter-class conflict analysis and multi-dimensional data labeling.

[0011] Preferably, the specific processing process of the eight processing links is:

[0012] Deduplication: Use the similarity of feature values ​​between sample data to define samples with high similarity as duplicate samples and remove them;

[0013] Detoxification: Remove sensitive, discriminatory, illegal, and confidential content;

[0014] Cleaning: Use regular expressions to delete special symbols, extra spaces, consecutive punctuation marks, empty data lines, and short useless text;

[0015] Classification: Classify the text content into the fields of general knowledge, mathematics, and code;

[0016] Indicator statistics: Statistics on the length of text data in each category after vocabulary conversion, vocabulary coverage, and whether there is a chain of thought;

[0017] Intra-category performance analysis: Analyze the differences in content length, breadth, knowledge depth, task type, and data magnitude for each category under statistical indicators;

[0018] Inter-class conflict analysis: Analyze the conflict when data of different capabilities are fed into the model at the same stage, and the relationship between positive and negative impacts and changes in data magnitude;

[0019] Multi-dimensional data labeling: Based on the analysis results, multi-dimensional labels are added to the data based on the original categories.

[0020] Preferably, each batch of data in data processing is divided into 4 micro batches of data.

[0021] In a second aspect, the present invention further provides a system for efficiently preprocessing large language model data based on pipeline parallelism, which is applied to the above-mentioned method for efficiently preprocessing large language model data based on pipeline parallelism to perform data processing, and the system includes:

[0022] The pipeline processing module is used to build a pipeline processing framework, which divides the data preprocessing task into multiple processing links. Each processing link cannot be decoupled and is executed in sequence.

[0023] The parallel processing module is used to divide each batch of data in data processing into multiple micro-batch data based on the pipeline processing framework and adopt a parallel processing method. As the smallest unit of data processing, the data processing is started in sequence; each micro-batch data automatically enters the next processing link after completing each processing link, thereby realizing the parallel operation of different data processing links in the same time period.

[0024] Preferably, in the pipeline processing module, the data preprocessing task is divided into eight processing steps to be executed sequentially, namely: deduplication, detoxification, cleaning, classification, indicator statistics, intra-class performance analysis, inter-class conflict analysis and multi-dimensional data labeling.

[0025] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned efficient preprocessing method for large language model data based on pipeline parallelism.

[0026] In a fourth aspect, the present invention also provides a storage medium having stored thereon one or more programs readable by a computing device, wherein the one or more programs include instructions which, when executed by the computing device, enable the computing device to perform an efficient preprocessing method for large language model data based on pipeline parallelism as described above.

[0027] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0028] The present invention provides a method and system for efficient preprocessing of large language model data based on pipeline parallelism. The present invention constructs a pipeline processing framework, divides the data preprocessing task into multiple processing links, and each processing link cannot be decoupled and is executed in sequence. A parallel processing method is also designed to realize the parallel operation of different data processing links in the same time period. The use of the present invention is convenient for effectively improving the efficiency of large language model data preprocessing.

[0029] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0030] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0033] Figure 1 A schematic diagram of a pipeline-based preprocessing process provided by an embodiment of the present invention.

[0034] Figure 2 A schematic diagram of the principles of a method for efficient preprocessing of large language model data based on pipeline parallelism provided in an embodiment of the present invention.

[0035] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0037] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this document or may be performed in parallel. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0038] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Embodiment 1:

[0040] The embodiment of the present invention provides a method for efficient preprocessing of large language model data based on pipeline parallelism, which mainly includes:

[0041] Build a pipeline processing framework to divide the data preprocessing task into multiple processing links. Each processing link cannot be decoupled and is executed in sequence.

[0042] Based on the pipeline processing framework, a parallel processing method is adopted to divide each batch of data in data processing into multiple micro-batch data. As the smallest unit of data processing, data processing is started in sequence; each micro-batch data automatically enters the next processing link after completing each processing link, thereby realizing parallel operation of different data processing links in the same time period.

[0043] 1. The relevant principles and specific implementation methods involved in the method of the present invention are introduced below:

[0044] In this embodiment, see Figure 1 As shown, the present invention proposes a pipeline data preprocessing framework. Based on a large amount of practice, the data preprocessing process is standardized into eight processing links: each link cannot be decoupled and must be executed strictly in sequence. From the perspective of data preprocessing, the eight processing links can be divided into two parts: data engineering and data science. Data engineering mainly includes engineering processing methods such as data deduplication, detoxification, cleaning, and classification. Data science mainly includes data indicator statistics, single-category data performance analysis (intra-class performance analysis), conflict analysis during training of multiple types of data (inter-class conflict analysis), and multi-dimensional data labeling. The details are as follows:

[0045] Deduplication: Use the similarity of feature values ​​between samples to define samples with high similarity as duplicate samples and remove them.

[0046] Detoxification: Remove sensitive, discriminatory, illegal, confidential and other non-compliant content.

[0047] Cleaning: Use regular expressions to delete special symbols, extra spaces, consecutive punctuation marks, empty data lines, short useless text, etc. to achieve the purpose of preliminary cleaning.

[0048] Classification: Classify the text content into fields such as general knowledge, mathematics, and code.

[0049] Indicator statistics: Statistics on the text length, vocabulary coverage, and whether there is a chain of thought (CoT) in each category of text data after vocabulary conversion;

[0050] Intra-category performance analysis: Analyze the differences in content length, breadth, knowledge depth, task type, data volume, etc. for each category under the above statistical indicators;

[0051] Inter-class conflict analysis: When training and fine-tuning large models, it is necessary to comprehensively consider the relationship between the target model capabilities, the data capabilities, data volume, data ratio, and training strategy during the acquisition process. Before that, it is necessary to analyze the conflicts when data of different capabilities are injected into the model at the same stage, as well as the relationship between positive and negative impacts and changes in data magnitude.

[0052] Multi-dimensional data labeling: Based on the above analysis results, multi-dimensional labels are added to the data on the basis of the original major categories to facilitate accurate data selection in terms of capabilities and magnitude during model training.

[0053] In this embodiment, the above data preprocessing steps are a serial process, that is, the original data needs to complete eight steps in sequence during the data preprocessing process, including data deduplication, detoxification, cleaning, classification, indicator statistics, intra-class performance analysis, inter-class conflict analysis and data multi-dimensional labeling.

[0054] In another embodiment, the above data preprocessing process may follow a two-layer serial process. First, the original data needs to complete the two links of data engineering and data science in the outer process in sequence, that is, it is necessary to wait for all data to complete the data engineering link and obtain a clean, non-toxic data set before entering the data science link; secondly, the data is arranged in descending order according to the demand for storage space during the processing process. In the inner process data engineering, the four small links of deduplication, detoxification, cleaning, and classification are completed in sequence. In the inner process data science, the four small links of indicator statistics, intra-class performance analysis, inter-class conflict analysis, and multi-dimensional data labeling are completed in sequence.

[0055] However, in serial processing, due to the huge amount of data for large model training (starting from 2TB), the data needs to be divided into batches according to the device storage space, and then each batch of data is fed into the preprocessing stage to obtain samples that can be used for model training in a standard format. In addition to storing the data of the current batch, each stage of preprocessing requires an average of 8 times the cache space to store intermediate results according to actual engineering statistics. In particular, in the deduplication stage, the current sample and all samples in the database need to be calculated for duplication. At this time, the larger the cache space, the more samples are stored and the higher the calculation efficiency. However, the size of the additional cache required to store intermediate results in each stage is different, resulting in different amounts of data that can be processed in each stage. Due to the limited cache space of the device, in order to ensure that the memory does not overflow, when processing eight stages in serial, it is necessary to take the minimum value of the amount of data contained in each batch in the eight stages, which greatly reduces the processing efficiency.

[0056] See also Figure 1 As shown, if the processing is carried out in a pipeline serial manner, assuming that the processing time of each link is t i , i represents the number of links, i∈{1,2,…,8}. To ensure the cache of intermediate data, the maximum number of samples processed in the i-th link is d under the effective use of memory. i , a total of N batches can complete the processing of all data. Then the waiting time required for the model to start training is:

[0057]

[0058] The total amount of data processed is:

[0059]

[0060] And satisfy:

[0061] d i <d k , when i <k (1)

[0062] Where, d k It represents the maximum number of samples processed in the kth stage, k represents the number of stages, k∈{1,2,…,8}.

[0063] The goal of data preprocessing is to selectively feed clean, content-rich samples into the model for parameter training, optimization, and knowledge learning. However, during the model training process, the number of model parameters is huge (in the tens of billions), and the amount of training data is huge (in TB). Due to limited computing power, during the training of large models, data also needs to be fed into the model in batches for distributed training. Therefore, it is not necessary to wait until all data has been processed before starting the model training phase. Figure 2As shown, in order to shorten the time spent on data preprocessing and the waiting time for model training, this embodiment proposes an efficient preprocessing method for large language model data based on pipeline parallelism. The horizontal axis is the processing time, the vertical axis is the processing link, and the entire process is executed in the order of data engineering (green) and data science (orange).

[0064] In this embodiment, the data of each batch d i Designed for smaller micro-batches Processing time is j represents the jth micro-batch, j∈{1,2,…,n}; for the sake of convenience, n=4. Figure 2 As shown, from the perspective of data processing flow, a micro batch of data goes through Complete the data engineering step and then wait After completion, enter the data science phase. Then wait After completion, the batch of data has completed all preprocessing work. From the timeline, the first time period, Only micro-batches with less data need to be processed The second time period The amount of data that needs to be processed is and The sum of , and so on.

[0065] First, assume that d i Under the condition of unchanged

[0066] 1) The amount of data processed is The deduplication process;

[0067] 2) The amount of data processed is The deduplication process and data volume are Detoxification process;

[0068] 3) The amount of data processed is The deduplication process has a data volume of The detoxification process and data volume are The cleaning process;

[0069] 4) The amount of data processed is The deduplication process has a data volume of The detoxification process, the data volume is The data volume is The classification stage;

[0070] 5) And so on, finally The amount of data processed is The multi-dimensional labeling stage of data.

[0071] Then, the total processing time from stage 1 to stage 4 is:

[0072]

[0073] Since the amount of data in each batch is divided into the original but

[0074] Formula (2) can be simplified as:

[0075]

[0076] In the actual data processing stage, t 2 >t 1 >t 3 >t 4 ;

[0077] Formula (3) can be further simplified as

[0078]

[0079] Compared with traditional serial processing, Length of time.

[0080] Similarly, the 5th to 8th stages are similar to the first 4 stages. 5 >t 6 >t 7 >t 8 , it can be seen that the last four steps save time compared with the traditional serial processing method. Length of time.

[0081] In summary, the data preprocessing method based on pipeline parallelism saves overall Length of time.

[0082] However, compared with the serial method, d i It can be expanded appropriately. Figure 2 It can be seen that the amount of data to be processed in step 4) is the largest.

[0083] From formula 1), we can know that:

[0084] D <d 1

[0085] That is, the amount of data that needs to be processed in step 4) is still less than the first link in the serial method, which is the link that occupies the most memory. So step 4) still has additional cache to process more data, improve efficiency, and shorten time. Therefore, the amount of data that can be processed in each link is d i can be appropriately expanded, so the overall time saved is greater than It has been confirmed through multiple experiments that in actual operating environments, compared with the original serial processing method, the waiting time for subsequent model training can be significantly shortened, effectively improving the efficiency of data preprocessing.

[0086] From the description of the above embodiments, those skilled in the art can know that: for the problem of preprocessing large language model data, the embodiments of the present invention provide an efficient preprocessing method for large language model data based on pipeline parallelism, which has the following innovative advantages:

[0087] 1. A pipeline preprocessing framework was designed, and data preprocessing was standardized into eight steps: data deduplication, detoxification, cleaning, classification, indicator statistics, intra-class performance analysis, inter-class conflict analysis, and data multi-dimensional labeling. Each step runs in sequence, realizing data engineering and data science functions for large-scale open source general data and industry private data in different formats, sources, task forms, and capabilities, ensuring the realization of data preprocessing functions and providing high-quality data with unified format, complete information, and easy extraction and combination for model training.

[0088] 2. Based on the above processing framework, a parallel processing method is designed to further divide each batch into n micro-batches with smaller data volumes as the smallest unit of data processing. After each minimum unit of data is completed, it automatically enters the preprocessing stage of the next link, thereby realizing the simultaneous operation of different data processing links in each time period to improve the efficiency of data preprocessing.

[0089] Through the above design, the present invention ensures the realization of the preprocessing function of data from different sources under the pipeline framework, and at the same time achieves the effect of greatly shortening the waiting time for subsequent model training under the same computing power conditions.

[0090] Embodiment 2:

[0091] The present invention also provides a system for efficiently preprocessing large language model data based on pipeline parallelism, which is applied to the method for efficiently preprocessing large language model data based on pipeline parallelism described in the above embodiment 1 to perform efficient data preprocessing. The system includes:

[0092] The pipeline processing module is used to build a pipeline processing framework, which divides the data preprocessing task into multiple processing links. Each processing link cannot be decoupled and is executed in sequence.

[0093] The parallel processing module is used to divide each batch of data in data processing into multiple micro-batch data based on the pipeline processing framework and adopt a parallel processing method. As the smallest unit of data processing, the data processing is started in sequence; each micro-batch data automatically enters the next processing link after completing each processing link, thereby realizing the parallel operation of different data processing links in the same time period.

[0094] An embodiment of the present invention provides an efficient preprocessing system for large language model data based on pipeline parallelism, and its implementation principle and technical effects are the same as those of the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment, which will not be repeated here.

[0095] Example 3

[0096] Reference Figure 3 As shown, an embodiment of the present invention further provides an electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, the processor executes the computer program to implement an efficient preprocessing method for large language model data based on pipeline parallelism in the above-mentioned embodiment 1.

[0097] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 11, and calling data stored in the memory 11.

[0098] Embodiment 4:

[0099] An embodiment of the present invention also provides a storage medium on which one or more programs readable by a computing device are stored. The one or more programs include instructions. When the instructions are executed by the computing device, the computing device executes an efficient preprocessing method for large language model data based on pipeline parallelism in Example 1.

[0100] Examples of computer-readable storage media include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card-type memory (such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage system, optical data storage system, hard disk, solid-state disk, and any other system configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent system, a server, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0101] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0103] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0104] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An efficient preprocessing method for large language model data based on pipeline parallelism, characterized in that: The method includes: Build a pipeline processing framework to divide the data preprocessing task into multiple processing links. Each processing link cannot be decoupled and is executed in sequence. Based on the pipeline processing framework, a parallel processing method is adopted to divide each batch of data in data processing into multiple micro-batch data. As the smallest unit of data processing, data processing is started in sequence; each micro-batch data automatically enters the next processing link after completing each processing link, thereby realizing parallel operation of different data processing links in the same time period.

2. According to claim 1, a pipeline-parallel-based large language model data efficient preprocessing method is characterized in that: The data preprocessing task is divided into eight processing steps that are executed sequentially, namely: deduplication, detoxification, cleaning, classification, indicator statistics, intra-class performance analysis, inter-class conflict analysis and multi-dimensional data labeling.

3. According to claim 2, a method for efficient preprocessing of large language model data based on pipeline parallelism is characterized in that: The specific processing process of the eight processing links is as follows: Deduplication: Use the similarity of feature values ​​between sample data to define samples with high similarity as duplicate samples and remove them; Detoxification: Remove sensitive, discriminatory, illegal, and confidential content; Cleaning: Use regular expressions to delete special symbols, extra spaces, consecutive punctuation marks, empty data lines, and short useless text; Classification: Classify the text content into the fields of general knowledge, mathematics, and code; Indicator statistics: Statistics on the length of text data in each category after vocabulary conversion, vocabulary coverage, and whether there is a chain of thought; Intra-category performance analysis: Analyze the differences in content length, breadth, knowledge depth, task type, and data magnitude for each category under statistical indicators; Inter-class conflict analysis: Analyze the conflict when data of different capabilities are fed into the model at the same stage, and the relationship between positive and negative impacts and changes in data magnitude; Multi-dimensional data labeling: Based on the analysis results, multi-dimensional labels are added to the data based on the original categories.

4. The method for efficient preprocessing of large language model data based on pipeline parallelism according to claim 1, characterized in that: Divide each batch of data in data processing into 4 micro batches.

5. An efficient preprocessing system for large language model data based on pipeline parallelism, characterized in that: When applied, the method for efficiently preprocessing large language model data based on pipeline parallelism according to any one of claims 1 to 4 is executed to process data, and the system includes: The pipeline processing module is used to build a pipeline processing framework, which divides the data preprocessing task into multiple processing links. Each processing link cannot be decoupled and is executed in sequence. The parallel processing module is used to divide each batch of data in data processing into multiple micro-batch data based on the pipeline processing framework and adopt a parallel processing method. As the smallest unit of data processing, the data processing is started in sequence; each micro-batch data automatically enters the next processing link after completing each processing link, thereby realizing the parallel operation of different data processing links in the same time period.

6. The system for efficient preprocessing of large language model data based on pipeline parallelism according to claim 5, characterized in that: In the pipeline processing module, the data preprocessing task is divided into eight processing links that are executed sequentially, namely: deduplication, detoxification, cleaning, classification, indicator statistics, intra-class performance analysis, inter-class conflict analysis and multi-dimensional data labeling.

7. An electronic device, characterized in that: It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement an efficient preprocessing method for large language model data based on pipeline parallelism as described in any one of claims 1 to 4.

8. A storage medium having stored thereon one or more programs readable by a computing device, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform an efficient preprocessing method for large language model data based on pipeline parallelism as described in any one of claims 1 to 4.

Citation Information

Cited By

  • Intelligent question answering system construction method and system based on LLM large language model

    CN119691140A