Method and software for training large language model on computing power cloud

Abstract

The application relates to a computing power cloud large language model training optimization method, which comprises the following steps of: in an offline tool software, Tokenizer internal order is disturbed to obtain a new Token sequence; an Embedding matrix is adjusted according to corresponding positions according to the new Token sequence to obtain a new Embedding matrix corresponding to the new Token sequence; the Tokenizer is retrained according to the new Token sequence and the new Embedding matrix; a corresponding Tokenized sample vector is generated according to the retrained Tokenizer; a large language model after modification is tested, sample vectors and corresponding Embedding matrices are uploaded, and a supervised fine-tuning process is performed. The application can reduce the risk of data leakage, and it is beneficial for computing power service providers and users to reduce the risk of data leakage. The application can reduce the possibility of data leakage for service providers and make users more confident in using cloud services. No additional calculation and storage are introduced. Generally, users renting computing power cloud need to use GPU or TPU acceleration functions, and the Tokenize process generally uses the calculation capacity of CPU, so that Tokenize on the user side does not introduce additional calculation and storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a computing power cloud large language model training optimization method and software. BACKGROUND

[0002] An important application scenario of the current computing power cloud is the training of large language models, including pre-training, supervised fine-tuning, and reinforcement learning fine-tuning. The computing power cloud integrates computing power into various forms of cloud services. For example, it can be a GPU card on a virtual machine, or it can provide a customized training interface. For the customized training interface scenario, the user can specify various configuration parameters of the framework after uploading the data for training. Generally, existing large models are basically based on the Transformer architecture, and the framework used is also a general framework such as transformers and deepspeed. Generally, the form of directly uploading text training data is adopted, which leads to the risk of data leakage for the user. If the cloud platform does not have adequate permission control or is subject to a hacker attack, it may be leaked. Since much of the data of large language models is text data or text data in a general compressed format, the loss caused by leakage is greater than that of files in a proprietary format. SUMMARY

[0003] Therefore, the present application proposes a computing power cloud large language model training optimization method to solve the problems embodied in the background art.

[0004] The computing power cloud large language model training optimization method provided by the present application comprises the following steps:

[0005] In the offline tool software, the internal order of the Tokenizer is disturbed to obtain a new Token sequence;

[0006] According to the new Token sequence, the Embedding matrix is adjusted according to the corresponding position to obtain a new Embedding matrix corresponding to the new Token sequence;

[0007] According to the new Token sequence and the new Embedding matrix, the Tokenizer is retrained;

[0008] According to the retrained Tokenizer, a corresponding Tokenized sample vector is generated;

[0009] The modified large language model is tested, the sample vector and the corresponding Embedding matrix are uploaded, and the supervised fine-tuning process is performed.

[0010] Optionally, the computing power cloud large language model training optimization method comprises the following steps:

[0011] The Tokenizer internal order is disturbed by randomly arranging the Token sequence in the Tokenizer to obtain a new Token sequence.

[0012] Optionally, the retraining of the Tokenizer comprises:

[0013] The Tokenizer is retrained according to the self-owned training data, and the newly found Token is added to the Tokenizer to establish a correspondence between the new and old Tokenizers.

[0014] Optionally, the establishing of the correspondence between the new and old Tokenizers further comprises:

[0015] The correspondence between the original Tokenizer and the new Tokenizer is established by Tokenizing the same text according to the new and old Tokenizers;

[0016] The vectors at the corresponding positions of the original Embedding matrix are copied to the corresponding positions of the new Embedding matrix; and

[0017] When there is no corresponding vector in the original Embedding matrix, the vector is filled with random initialization or the mean of other vectors.

[0018] Optionally, the large language model training optimization method on the computing power cloud comprises:

[0019] The weight matrix is disturbed, a small amount of disturbance is applied to the weight matrix to avoid data leakage from being backtracked to the correspondence of each Token; and the user can customize the distribution and amplitude of the random amount of disturbance of the disturbance matrix.

[0020] Optionally, the generating of the corresponding Embedding matrix and Tokenized sample vector according to the retrained Tokenizer further comprises:

[0021] According to the retrained Tokenizer, the input text data is segmented to generate Tokens, and the Embedding matrix corresponding to the modified Tokenizer is generated.

[0022] Optionally, the testing of the modified large language model comprises:

[0023] The modified large language model is tested to see if it can restore the original model to avoid compatibility problems caused by modification.

[0024] The application also provides a device for large language model training on a computing power cloud, characterized in that the device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the large language model training optimization method on the computing power cloud according to any one of claims 1 to 7.

[0025] Optionally, the storage medium is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the large language model training optimization method on the computing power cloud according to any one of claims 1 to 7.

[0026] The application also provides an electronic device, characterized in that it comprises:

[0027] A computer device, characterized in that it comprises a memory and a processor, and the memory stores a computer program, wherein the processor executes the computer program to implement the large language model training optimization method on the computing power cloud according to any one of claims 1 to 7.

[0028] The application has the beneficial effects of reducing the risk of data leakage, which is beneficial to both the computing power service provider and the user. The application can reduce the possibility of data leakage for the service provider and make the user more confident in using the cloud service. No additional computation and storage are introduced. Generally, users renting computing power clouds need to use GPU or TPU acceleration functions, and the Tokenize process generally uses the computing power of the CPU, so the user side performing Tokenize will not introduce additional computation and storage. BRIEF DESCRIPTION OF DRAWINGS

[0029] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings needed in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0030] Figure 1 A flowchart of a large language model training optimization method on a computing power cloud disclosed by the present application is shown.

[0031] Figure 2 A flowchart of a supervised fine-tuning process disclosed by the present application is shown. DETAILED DESCRIPTION

[0032] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings represent the same or similar elements. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0033] The terms "first", "second", etc. are used only for the purpose of description and should not be interpreted in a relative way unless specifically stated. Thus, features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless specifically limited otherwise.

[0034] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0035] In addition, for the purpose of better illustrating the present application, numerous specific details are set forth in the following detailed description. One skilled in the art will understand, however, that the application can be practiced without certain of the specific details herein. In some instances, well-known methods, structures, elements, and circuits have not been described in detail in order to avoid obscuring the application.

[0036] The present application proposes an algorithm for training large language models on computing power clouds. The main feature of this algorithm is that it does not leak user training data during training. When training on a cloud platform, some users may have the need to encrypt training data. However, most existing large language model training platforms do not provide this function. The present application proposes a method of processing data during Tokenize. When submitting training data on the platform, only processed data is uploaded. Through this method, the risk of data leakage can be reduced.

[0037] The computing power cloud is a cloud service that provides GPU (Graphics Processing Unit), TPU (Tensor Processing Unit) and other acceleration cards; Tokenize is the process of dividing text into Token, which involves text segmentation, punctuation removal, lowercasing and other processing to generate a Token sequence. The purpose of Tokenize is to convert the original text data into a form that can be understood and processed by machines. In practical applications, Tokenize is usually used in the text preprocessing stage to provide basic data for subsequent natural language understanding, text classification, sentiment analysis and other tasks. Token is the basic unit of natural language processing, usually referring to a word or symbol. In text, Token can be a word, punctuation, number or other special character. The concept of Token is similar to the word or symbol we understand in daily life. In NLP (Natural Language Processing), by Tokenizing text, we can convert continuous character sequences into discrete Token sequences to lay the foundation for subsequent text processing and analysis. Supervised fine-tuning, also known as instruction fine-tuning, uses supervised instruction data to train large language models to respond to user instructions. Reinforcement learning fine-tuning, also known as preference alignment, provides positive and negative examples based on instruction fine-tuning to train large language models to produce results that conform to human values.

[0038] As shown in Figure 2 The supervised fine-tuning flowchart includes the following contents:

[0039] First, the text data is converted into Token through Tokenize, which involves text segmentation, punctuation removal, lowercasing and other processing to convert the original text data into a form that can be understood and processed by machines to generate the corresponding Token sequence.

[0040] Second, the generated multiple data samples are batched to form a small batch of data, which is used to process the data samples in batches, effectively utilizing the parallel computing power of GPU to improve computing efficiency, and small batch training can greatly reduce memory consumption compared to processing samples one by one.

[0041] Then, for data samples of different lengths in the batch, collate is used to pad the data to ensure that all samples have the same length before entering the Transformer model.

[0042] Finally, the Trainer in the Transformer model is called for training, which encapsulates the main logic of the training loop, including data loading, forward propagation, loss calculation, backpropagation, and parameter updating, etc.

[0043] Take the supervised fine-tuning as an example to illustrate the specific steps and improvements of the method. As shown in the figure, after the text data is Tokenized, it becomes a series of numbers, and then these numbers are converted into vector inputs through the Embedding matrix. The Embedding matrix determines the vector value corresponding to each Token. Figure 2

[0044] The Embedding matrix is a technique for converting discrete data (such as words, objects, categories, etc.) into continuous vectors. In machine learning and deep learning, Embedding usually refers to the conversion of high-dimensional or discrete data representation into a low-dimensional real vector. This process helps data representation and calculation in vector space. In natural language processing, word embedding is the most classic application of Embedding, which maps words from a discrete index to a low-dimensional continuous vector. These vectors help capture semantic relationships and contextual information in language.

[0045] Then, the sample is generated using the Transformer model. For each Token position of the sample, an output Logit vector of the Transformer model is obtained, with a dimension of the total number of Tokens. The Logit vector is the prediction of the next token at this position. Taking the maximum or sampling can obtain the output Token value.

[0046] The Transformer model is a deep learning model architecture for natural language processing and sequence-to-sequence tasks, mainly composed of an encoder and a decoder. The encoder is responsible for converting input sequences into a series of vector representations, while the decoder is responsible for generating output sequences. The self-attention mechanism is the core of the Transformer model, which calculates the similarity between each position in the input sequence and all other positions. Between the encoder and the decoder, there are multiple self-attention layers and feedforward neural network layers.

[0047] If text is needed, Detokenize is performed to convert it into text. In supervised training, if the intermediate text output result does not need to be viewed, Detokenize is not needed to obtain the text. Instead, the loss value of the output Token true value and the Logit vector prediction value is used for backpropagation to train.

[0048] ​Through the above analysis, the application proposes that when training on a cloud platform, only the Embedding matrix and the sample vector (including the input vector and the output target vector) after Tokenizer need to be uploaded.

[0049] In this way, it is not necessary to upload the original text data. It is equivalent to uploading Figure 2 In the prior art, the general cloud service is operated at the first arrow, and the application is operated at the second arrow.

[0050] The Tokenizer is a tool or library for performing Tokenize operations. In the field of NLP, Tokenizer plays an important role in converting text into Token sequences. Different Tokenizers may use different algorithms and strategies for Tokenization to adapt to different language characteristics and application requirements. For example, for Chinese text, the Tokenizer is a commonly used Tokenizer that can cut Chinese sentences into individual words or word groups.

[0051] As shown in Figure 1 Fig. 1 is a method for training and optimizing a large language model on a computing power cloud according to an embodiment of the application. The method includes the following contents:

[0052] S100, the application changes the Tokenizer configuration in the software and disturbs the order in the Tokenizer to obtain a new Token sequence.

[0053] Specifically, the order in the Tokenizer refers to the order in which the Tokenizer processes text and breaks it down into smaller units (such as words, characters, or phrases). This process includes: tokenization, breaking down text into smaller units such as words or characters; building a dictionary, building a dictionary mapping based on the tokenization results, and if using pre-trained word vectors, the dictionary mapping needs to be processed according to the word vector file; data conversion, converting the tokenized data into a digital sequence, which usually involves mapping each unit to a unique digital ID; padding and truncation, padding short data and truncating long data when inputting the model in batches to ensure that the data length meets the model requirements and ensures that the data dimensions within the batch are consistent.

[0054] S200, the application adjusts the Embedding matrix according to the new Token sequence in the software according to the corresponding position to obtain a new Embedding matrix corresponding to the new Token sequence.

[0055] Specifically, the Embedding matrix, as a continuous step closely related and complementary to the Tokenizer, should also be changed according to the corresponding position when the internal order of the Tokenizer is randomly shuffled.

[0056] S300, the application re-trains the Tokenizer in software according to the new Token sequence and the new Embedding matrix.

[0057] The re-training of the Tokenizer refers to re-training the Tokenizer according to the self-owned training data, adding the newly found Token to the Tokenizer, and then establishing the correspondence between the original Tokenizer and the new Tokenizer by Tokenizing the same text according to the old and new Tokenizers. The vectors in the corresponding position of the original Embedding matrix are copied, and the vectors not in the original Embedding matrix are filled with random initialization or the mean of other vectors.

[0058] Among them, the application disturbs the weight matrix in software to avoid the corresponding relationship of each token being deduced from the Embedding matrix when data leakage occurs. Because part of the pre-trained model weights used for supervised fine-tuning is universal, if data leakage occurs, the corresponding relationship of each Token can be deduced from the Embedding matrix. Therefore, the application supports a small amount of disturbance to the weight matrix, and the user can customize the distribution and amplitude of the random amount of disturbance.

[0059] The weight matrix can be obtained by pre-training and fine-tuning, in which case the pre-trained weight is used, and then fine-tuned on a specific task. Pre-training refers to training the model on a large-scale, general task, for example, the BERT model is pre-trained on a large-scale unsupervised text corpus, the tasks include Masked Language Modeling and Next Sentence Prediction. During the pre-training process, the model learns general language features and patterns. Fine-tuning refers to further training the model on the training data of a specific task based on the pre-trained model, for example, the pre-trained BERT model can be fine-tuned using a small-scale sentiment analysis dataset to adapt the model to the sentiment analysis task.

[0060] S400, the application generates the corresponding matrix and vector after the Tokenizer configuration change is completed.

[0061] Specifically, according to the retrained tokenizer configuration, the corresponding embedding matrix is generated according to the new tokenizer configuration, and the sample vector converted by the newly generated embedding matrix. Finally, it is also necessary to test whether the modified model can restore the original model to avoid compatibility problems caused by modification.

[0062] S500, after verifying the modified large language model, upload the newly generated matrix and sample vector to the computing power cloud to continue the supervised fine-tuning process.

[0063] Specifically, by testing whether the modified large language model can be restored to the original model, it is verified whether there is a compatibility problem. Among them, the newly generated matrix and sample vector are sent to the computing power cloud, which not only ensures the normal operation of the supervised fine-tuning process, but also avoids information leakage caused by directly uploading text data.

[0064] The specific implementation process of the present application is as follows:

[0065] When supervised fine-tuning, the general data flow is as shown in Figure 2 .

[0066] Among them, after tokenizing the text data, it becomes a digital one by one, and then it is converted into a vector input through the Embedding matrix. The Embedding matrix determines the vector value corresponding to each token.

[0067] Then generate the sample using the Transformer model. For each token position of the sample, an output logit vector of the Transformer model is obtained, and the vector dimension is the total number of all tokens.

[0068] The logit vector is the prediction of the next token at this position, and the output token value can be obtained by taking the maximum or sampling.

[0069] If you need to get the text, you need to detokenize to convert it into text. During supervised training, if you don't need to view the intermediate text output result, you don't need to detokenize to get the text. Instead, the loss value of the output token true value and the logit vector prediction value is directly used for back propagation to train.

[0070] Through the above analysis, the present application proposes that when training on a cloud platform, only the following information needs to be uploaded:

[0071] 1. Embedding matrix

[0072] 2. The sample vector after tokenizer, including input vector and output target vector.

[0073] This way, the original text data does not need to be uploaded. In the above data flow diagram, the general cloud service operates at the first arrow, and the present application operates at the second arrow.

[0074] The present application is in the form of a tool software that is used offline by users to generate tokenizer configuration files, sample vectors after tokenization, and corresponding embedding matrices.

[0075] The tool software provides the following functions:

[0076] 1. Randomly shuffle the order in tokenizer, and change the embedding matrix according to the corresponding positions.

[0077] 2. The tokenizer can be retrained on the own training data, and new tokens found can be added to the tokenizer. Then, by tokenizing the same text according to the new and old tokenizers, the correspondence between the original tokenizer and the new tokenizer is established. The vectors in the corresponding positions of the original matrix are copied over, and the vectors not in the original embedding matrix are filled with random initialization or other vector means.

[0078] 3. Since part of the pre-trained model weights used for supervised fine-tuning are universal, if data is leaked, it may be possible to infer the corresponding relationship of each token from the embedding matrix. The software supports a small amount of perturbation of the weight matrix, and the user can customize the distribution and amplitude of the random amount of perturbation.

[0079] 4. Generate the corresponding embedding matrix and sample vector after tokenizer according to the modified tokenizer. Note that the labels for supervised training are also included in the sample vector after tokenizer.

[0080] 5. Test the ability to recover the original model from the modified model to avoid compatibility problems caused by modification.

[0081] Note that if new tokens are added above, since their embeddings are randomly generated or filled with mean vectors, the parameters for them need to be specified for training during training.

[0082] The application can greatly reduce the risk of data leakage, and it is beneficial for both the computing power service provider and the user to reduce the risk of data leakage. The application can enable the service provider to reduce the possibility of data leakage, and enable the user to use the cloud service more confidently. No additional calculation and storage are introduced. Generally, users renting computing power cloud need to use GPU or TPU acceleration functions. The Tokenize process generally also uses the computing power of CPU, so the user side performing Tokenize will not introduce additional calculation and storage.

[0083] The embodiments of the application have been described above, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles, practical applications, or improvements to the technology in the market of the embodiments, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A method for optimizing training of large language models on a computing power cloud, characterized in that, Comprising: In the offline tool software, the internal order of the Tokenizer is disturbed to obtain a new Token sequence; According to the new Token sequence, the Embedding matrix is adjusted according to the corresponding position to obtain a new Embedding matrix corresponding to the new Token sequence; According to the new Token sequence and the new Embedding matrix, the Tokenizer is retrained; According to the retrained Tokenizer, a corresponding Tokenized sample vector is generated; Test the modified large language model, upload the sample vector and the corresponding Embedding matrix, and perform a supervised fine-tuning process.

2. The large language model training optimization method on the computing power cloud according to claim 1, characterized in that: The Token sequence in the Tokenizer is randomly arranged to disturb the internal order of the Tokenizer and obtain a new Token sequence.

3. The method of claim 1, wherein, The retraining of the Tokenizer includes: Retrain the Tokenizer according to the self-owned training data, add the newly found Token to the Tokenizer, and establish the correspondence between the new and old Tokenizers.

4. The computing power cloud-based large language model training optimization method of claim 3, wherein, The correspondence between the new and old Tokenizers includes: Tokenize the same text according to the new and old Tokenizers to establish the correspondence between the original Tokenizer and the new Tokenizer; Copy the vectors of the corresponding positions of the original Embedding matrix to the corresponding positions of the new Embedding matrix; and When there is no corresponding vector in the original Embedding matrix, use random initialization or the mean of other vectors to fill it.

5. The computing power cloud-based large language model training optimization method of claim 1, wherein, Further comprising: Disturb the weight matrix to slightly disturb the weight matrix to avoid data leakage from being backtracked to the correspondence of each Token; And the user can customize the distribution and amplitude of the random amount of disturbance of the disturbance weight matrix.

6. The computing power cloud-based large language model training optimization method of claim 1, wherein, The retrained Tokenizer generates a corresponding Embedding matrix and Tokenized sample vector, including: According to the retrained Tokenizer, the input text data is segmented to generate Token, and the Embedding matrix corresponding to the modified Tokenizer is generated.

7. The computing power cloud-based large language model training optimization method of claim 1, wherein, The modified large language model includes: Test whether the modified large language model can restore the original model to avoid compatibility problems caused by modification.

8. An apparatus for training a large language model on a computing power cloud, characterized in that, Comprising: Memory, processor and computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the large language model training optimization method on the computing power cloud according to any one of claims 1 to 7.

9. An electronic device, comprising: Comprising: A computer device, characterized in that, Comprising memory and processor, the memory storing a computer program, wherein the processor executes the computer program to implement the large language model training optimization method on the computing power cloud according to any one of claims 1 to 7.

Images