Large language model optimization method

Abstract

The invention provides a large language model optimization method. The optimization method comprises the steps that an original corpus is acquired and preprocessed; constructing a deep learning model based on a graph neural network; training the model according to the preprocessed corpus to obtain a trained model; the model is split into a distributed training system for training in a hybrid parallel mode, wherein the hybrid parallel mode comprises a data parallel mode, an assembly line parallel mode and a tensor parallel mode; the distributed training system comprises at least one computing node, and each computing node in the at least one computing node comprises at least one host and at least one device; performing performance prediction on the model; and performing model optimization according to a prediction result. According to the method, optimization is systematically carried out from three aspects of data screening, architecture clipping and model optimization from three aspects of influencing model space-time consumption, namely data, model architecture and a model operation mechanism, so that effective operation of the large language model under the condition of limited space-time is realized.

Description

Technical Field

[0001] The present invention relates to the field of model optimization, and in particular to a large language model optimization method. Background Art

[0002] Large language models, also known as "base models" or "large models" (LLM), are currently considered to be the core engine of general artificial intelligence technology and have become the focus of global scientific and technological competition. Long sequence input is very critical when large language models handle complex tasks (such as video stream processing and protein property prediction). However, when calculating long sequence tasks, a large amount of computing and memory resources are required. In order to adapt to the memory limitations of GPUs, existing research mainly includes: directly applying 3D parallelism or existing automatic parallelization frameworks, but these frameworks are inefficient in training because these systems require us to keep the entire sequence in one GPU, which limits the length of the input sequence. For long sequence tasks, the existing large language models process too slowly, resulting in insufficient execution performance. For example, Transformer has revolutionized almost all natural language processing (NLP) tasks, but suffers from memory and computational complexity, which is quadratic with sequence length. In contrast, recurrent neural networks (RNNs) show linear growth in memory, but it is difficult to match Transformer performance due to limitations in parallelization and scalability. A study proposed a novel model architecture, RWKV, combined with efficient parallel training Transformer. Deep learning has driven the rapid development of artificial intelligence. The dominant technologies include RNN and Transformer, each of which has its own advantages and disadvantages. RNN requires less memory, especially for processing long sequences. However, they are affected by the time dimension and parallelization difficulties during training due to gradient disappearance, which limits their scalability; Transformer has become a powerful alternative to RNN, good at managing local and remote dependencies and supporting parallel training. Like ChatGPT, GPT3, LLaMA and Chinchilla demonstrate the potential of Transformers in NLP. However, the quadratic complexity of the self-attention mechanism of Transformers makes it computationally and memory intensive in processing long sequences and limited resource tasks. This advantage has led many researchers to focus on the scalability of Transformers. For long sequence tasks, researchers have proposed several methods to train LLMs for long sequences without significantly sacrificing training speed. However, they are subject to three limitations: first, existing large models consume a lot of resources and have high requirements for computing power and storage; second, solutions with good usability are still not effective enough in terms of training results; third, the application scenarios are limited because many application scenarios cannot be supported. In response to these pain points, the design and optimization of lightweight architectures for large language models is driven by the demand for using large models under computing and memory intensive conditions, and is urgently needed. Summary of the invention

[0003] In order to address the deficiencies in the prior art, the present invention provides a large language model optimization method, which starts from three aspects that affect the model's time and space consumption, namely, data, model architecture, and model operation mechanism, and systematically optimizes the three aspects of data screening, architecture tailoring, and model optimization, aiming to achieve effective operation of large language models under limited time and space.

[0004] In order to achieve the above object, the specific scheme adopted by the present invention is: A large language model optimization method mainly includes the following steps: S1. Obtain real-time user behavior data and real-time load data of each sub-model of the large language model, that is, obtain the original corpus, preprocess the data in the original corpus, and obtain the preprocessed corpus; S2. Build a deep learning model based on graph neural network according to the preprocessed corpus, i.e. the original model; S3. The original model is trained according to the preprocessed corpus to obtain a trained model; the original model is split into a distributed training system for training in a hybrid parallel mode, wherein the hybrid parallel mode includes a data parallel mode, a pipeline parallel mode, and a tensor parallel mode; the distributed training system includes at least one computing node, and each computing node in the at least one computing node includes at least one host and at least one device; S4. Predict the performance of the trained model; S5. Optimize the trained model according to the prediction results.

[0005] Furthermore, in step S1, the specific method for preprocessing the data in the original corpus is: S11, using a method based on the fusion of classifier and heuristic strategy to delete low-quality data and perform primary preprocessing; S12, deleting duplicate data at the sentence level, document level, and corpus level, respectively, and then performing secondary preprocessing; S13, detect and delete identifiable personal information based on keyword recognition, and perform three-level pre-processing; S14, performing word segmentation processing on the data after the three-level preprocessing to obtain multiple word segments; determining the word segmentation codes corresponding to the multiple word segments respectively; and generating a preprocessed corpus according to the word segmentation codes corresponding to the multiple word segments respectively.

[0006] Furthermore, in step S3, during the pipeline parallel design, distributed resources are used to train different consecutive layers of the graph neural network, that is, block-based pipeline training is used.

[0007] Furthermore, in step S3, a Fat-Tree network is used to design the graph neural network topology structure.

[0008] Furthermore, in step S3, RDMA technology is used to transmit data of the graph neural network.

[0009] Furthermore, in step S3, when performing original model training, the optimizer used includes but is not limited to SGD, Adam, and RMSprop.

[0010] Furthermore, in step S3, a regularization technique is used to improve the generalization ability of the model, and the regularization technique includes but is not limited to L1 regularization and L2 regularization.

[0011] Beneficial effects: Based on artificial intelligence theory and big model theory, this paper explores the design method of lightweight big model architecture on multi-core platform, aiming to make the big language model architecture as light as possible under the condition of limited space-time resources, so as to achieve the best matching of software and hardware resources, and provide the necessary and urgently needed big model architecture design method and related basic theory for the limited computing and storage resources in multi-core environment. It has the following two advantages: (1) Effective construction of lightweight large models In view of the problem that traditional large models have large number of parameters, complex network structure and long training time, which cause them to occupy a lot of time, space and computing resources, we analyze the resource consumption points of large models, start from the input end of the model, that is, data analysis, delete redundant and erroneous data to streamline data; start from network structure analysis, and achieve reasonable and effective cutting at the network structure level; by analyzing the operation mechanism of the model, we study the optimization method of the model; thus, we can achieve "slimming" of large models in three aspects: data end, model structure and model operation mechanism; (2) Expansion of large model application scenarios By designing multimodal large models for hardware constraints and using deep model compression methods such as model pruning, model quantization, and knowledge distillation, we can reduce the network size (number of network layers, number of nodes, etc.) and computing and storage requirements of large models, thereby meeting different hardware environments and expanding the application scenarios of large models. The present invention provides a parallel method to support the lightweight design and optimization of large models in response to the increasing demand for large language models, especially in scenarios with limited spatiotemporal resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a data preprocessing flow chart in the present invention.

[0013] Figure 2 It is a schematic diagram of data parallel mode, pipeline parallel mode and tensor parallel mode.

[0014] Figure 3 This is a schematic diagram of the model’s pipeline design and training.

[0015] Figure 4 This is the Fat-Tree network architecture diagram.

[0016] Figure 5 It is a remote direct memory access protocol. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0018] The present invention provides a large language model optimization method, which describes the modules designed and optimized in four dimensions (data dimension, method dimension, network architecture dimension and system structure dimension) of the large language model from the four levels of data layer, method design layer, network architecture layer and system structure layer. The large language model optimization method in the present invention mainly includes the following steps: S1. Obtain real-time user behavior data and real-time load data of each sub-model of the large language model, that is, obtain the original corpus, preprocess the data in the original corpus, and obtain the preprocessed corpus; S2. Build a deep learning model based on graph neural network according to the preprocessed corpus, i.e. the original model; S3. The original model is trained according to the preprocessed corpus to obtain a trained model; the original model is split into a distributed training system for training in a hybrid parallel mode, wherein the hybrid parallel mode includes a data parallel mode, a pipeline parallel mode, and a tensor parallel mode; the distributed training system includes at least one computing node, and each computing node in the at least one computing node includes at least one host and at least one device; S4. Predict the performance of the trained model; S5. Optimize the trained model according to the prediction results.

[0019] The present invention adopts a hierarchical approach and summarizes the design and optimization methods of large language models into a hierarchical framework with a feedback loop, namely "input data optimization layer -> network structure tailoring layer -> model optimization layer -> evaluation method layer". Starting from classification analysis, collection, and optimization of the input data of the large model, the research begins layer by layer. The following is a detailed description of each optimization part.

[0020] (1) Data preprocessing framework After collecting a large amount of text data, preprocessing the data, especially eliminating noisy data, redundant, irrelevant and potentially harmful data, is essential for building a pre-training corpus, because such data may greatly affect the capabilities and performance of large language models. Please refer to Figure 1 .

[0021] Firstly, in order to delete the low-quality data in the collected corpus, a method based on classifier and heuristic fusion is adopted, that is, a classifier is selected based on high-quality text training, high-quality data is used as positive samples, and candidate data is sampled as negative samples to train a binary classifier, and a score measuring the quality of each data is predicted, and it is used to identify and filter low-quality data; and low-quality text is eliminated by designing a set of carefully designed rules.

[0022] Secondly, based on existing findings: duplicate data in the corpus will reduce the diversity of the language model, which may lead to instability in the training process and thus affect model performance. i) At the sentence level, remove low-quality sentences containing repeated words and phrases; ii) At the document level, rely on the repetition ratio of surface features between documents to detect and remove duplicate documents containing similar content; iii) At the dataset level, in order to avoid dataset contamination problems, prevent overlap between the training set and the evaluation set by removing duplicate text that may appear in the test set from the training set.

[0023] Thirdly, most pre-trained text data comes from online sources, including user-generated content involving sensitive or personal information, which may increase the risk of privacy leakage. Therefore, it is necessary to remove personally identifiable information (PII) from the pre-training corpus. A rule-based method is constructed to implement keyword spotting to detect and remove personally identifiable information.

[0024] Finally, the data is segmented to obtain a plurality of segmented words; the segmented word codes respectively corresponding to the plurality of segmented words are determined; and a preprocessed corpus is generated according to the segmented word codes respectively corresponding to the plurality of segmented words.

[0025] (2) Lightweight architecture design method for large AI models The formal expression of the AI ​​big model based on the Graph Neural Network (GNN) is to use G=(V, E) to express the big model, where V represents the set of big model nodes and E represents the set of big model variables.

[0026] Secondly, lightweight architecture design. When training large models, data parallelism, pipeline parallelism, and tensor parallelism exist simultaneously, such as Figure 2As shown in the figure. The "parameter-plane large network" required for data parallelism and pipeline parallelism requires cross-server communication, and can reach a scale of hundreds of thousands or even millions of cards, with the characteristics of ultra-large scale, high network capacity, and high access bandwidth. The "parameter-plane small network" for achieving tensor parallelism is usually limited to a single server, with the characteristics of small scale, ultra-large capacity, and high access bandwidth.

[0027] Secondly, in terms of pipeline parallel design, distributed resources are used to train different consecutive layers of GNN, that is, block-based pipeline training is used. In terms of model scheduling strategy optimization, the graph neural network is divided into many blocks, and all GPUs use a pipeline method to process the divided blocks layer by layer; and an optimized pipeline scheduling strategy is used to complete the allocation of task blocks (such as Figure 3 as shown).

[0028] Again, in terms of network topology design, use Fat-Tree network, such as Figure 4 As shown in the figure. A single server is equipped with a high-performance 400 Gbit / s NIC. K servers form a group and are interconnected through a top-of-rack (ToR) switch. The ToR switch is connected to the aggregation switch to form a Pod to achieve cross-rack connectivity. The Pod is connected to the backbone switch to ensure that the servers in the central processing unit (CPU) cluster can achieve any-to-any communication.

[0029] Finally, RDMA technology is used in data transmission technology. RDMA allows applications to directly access the memory of remote nodes without going through the kernel. It has the advantages of high throughput, low latency, and no CPU usage. It can improve model training efficiency and is more suitable for large model training. RDMA mainly includes three types of protocols: InfiniBand (IB for short), RDMA based on Ethernet (RoCE), and RDMA based on TCP / IP protocol stack (iWARP). All three protocols comply with the RDMA standard and use the same upper layer interface. Figure 5As shown in the figure. IB defines a new set of hierarchical architectures from the link layer to the transport layer, and is a dedicated technology designed for high-performance computing. IB requires dedicated equipment when deployed, such as IB dedicated switches, IB dedicated network cards, IB dedicated cables, etc., and is not compatible with existing Ethernet devices. RoCE extends the Ethernet protocol stack to enable standard Ethernet devices to support RDMA operations, achieving a combination of high-performance remote memory access with Ethernet ease of use and wide deployment. iWARP is a TCP-based RDMA protocol proposed by the Internet Engineering Task Force (IETF). Since TCP is a connection-oriented reliable protocol, iWARP has better reliability than RoCEv2 and IB in lossy network scenarios. However, a large number of TCP connections will consume a lot of memory resources. In addition, TCP's complex flow control and other mechanisms will cause performance problems, limiting its scope of application. At this stage, it is not used on a large scale.

[0030] (3) System structure design for large AI models First, the architecture design of the AI ​​big model includes the overall structure of the model and the relationship between each component. Since the AI ​​big model consists of an input layer, a hidden layer, and an output layer, and the hidden layer can be subdivided into multiple sublayers, the connection method between the layers, the selection of activation functions, the setting of loss functions, etc. are comprehensively considered during the design. For example, in deep learning, commonly used activation functions include ReLU, Sigmoid, and Tanh, while the loss function is selected according to the specific task, such as cross entropy for classification tasks and mean square error for regression tasks.

[0031] Secondly, the design of AI large model architecture involves aspects such as parameter initialization, optimizer selection and regularization. Parameter initialization has an important impact on the training and convergence speed of the model, and is usually initialized using Gaussian distribution or uniform distribution. The choice of optimizer is also crucial. Common optimizers include SGD, Adam, RMSprop, etc. They each have their own advantages and disadvantages and need to be selected according to the specific situation. In addition, regularization methods such as L1 / L2 regularization, Dropout, etc. can help prevent overfitting and improve the generalization ability of the model.

[0032] Secondly, the design of model interpretability and scalability. Interpretability means that the results of the model output can be understood and explained, which helps to increase user trust and improve the credibility of the model. Scalability means that the model can adapt to data and tasks of different scales and complexities while maintaining good performance and results. In order to improve the interpretability and scalability of the model, technologies such as attention mechanism and orthogonal initialization can be used.

[0033] In summary, a deep understanding of the architecture and design of large AI models is crucial to improving model performance and effectiveness. By rationally designing the model’s structure, parameter initialization, optimizer selection, and regularization methods, the model’s convergence speed and generalization ability can be improved while maintaining the model’s interpretability and scalability.

[0034] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Any equivalent changes or modifications made according to the essence of the present invention should be included in the protection scope of the present invention.

Claims

1. A large language model optimization method, characterized in that: The main steps are as follows: S1. Obtain real-time user behavior data and real-time load data of each sub-model of the large language model, that is, obtain the original corpus, preprocess the data in the original corpus, and obtain the preprocessed corpus; S2. Build a deep learning model based on graph neural network according to the preprocessed corpus, i.e. the original model; S3. The original model is trained according to the preprocessed corpus to obtain a trained model; the original model is split into a distributed training system for training in a hybrid parallel mode, wherein the hybrid parallel mode includes a data parallel mode, a pipeline parallel mode, and a tensor parallel mode; the distributed training system includes at least one computing node, and each computing node in the at least one computing node includes at least one host and at least one device; S4. Predict the performance of the trained model; S5. Optimize the trained model according to the prediction results.

2. A large language model optimization method according to claim 1, characterized in that: In step S1, the specific method for preprocessing the data in the original corpus is: S11, using a method based on the fusion of classifier and heuristic strategy to delete low-quality data and perform primary preprocessing; S12, deleting duplicate data at the sentence level, document level, and corpus level, respectively, and then performing secondary preprocessing; S13, detect and delete identifiable personal information based on keyword recognition, and perform three-level pre-processing; S14, performing word segmentation processing on the data after the three-level preprocessing to obtain multiple word segments; determining the word segmentation codes corresponding to the multiple word segments respectively; A preprocessed corpus is generated according to the word segmentation codes respectively corresponding to the multiple word segmentations.

3. A large language model optimization method according to claim 1, characterized in that: In step S3, when designing the pipeline in parallel, distributed resources are used to train different consecutive layers of the graph neural network, that is, block-based pipeline training is used.

4. A large language model optimization method according to claim 1, characterized in that: In step S3, the Fat-Tree network is used to design the graph neural network topology structure.

5. The large language model optimization method according to claim 1, characterized in that: In step S3, RDMA technology is used to transmit data of the graph neural network.

6. A large language model optimization method according to claim 1, characterized in that: In step S3, when performing original model training, the optimizers used include but are not limited to SGD, Adam, and RMSprop.

7. A large language model optimization method according to claim 1, characterized in that: In step S3, a regularization technique is used to improve the generalization ability of the model, and the regularization technique includes but is not limited to L1 regularization and L2 regularization.

Images