Method and software for optimizing big language model training on computing power cloud

Abstract

The invention relates to a big language model training optimization method on computing power cloud, which comprises the following steps of: in offline tool software, disorganizing an internal sequence of Tokenizer to obtain a new Token sequence; according to the new Token sequence, an Embedding matrix is adjusted according to the corresponding position, and a new Embedding matrix corresponding to the new Token sequence is obtained; according to the new Token sequence and the new Embedding matrix, a Tokenizer is retrained, and according to the new Token sequence and the new Embedding matrix, the Tokenizer is retrained; according to the retrained Tokenizer, generating a sample vector after the corresponding Tokenize, and according to the retrained Tokenizer, generating a sample vector after the Tokenize; and testing the modified large language model, uploading a sample vector and a corresponding Embedding matrix, and executing a supervision fine tuning process. According to the method and the device, the risk of data leakage can be reduced, reduction of the risk of data leakage is beneficial for computing power service providers and users, the possibility of data leakage of the service providers can be reduced, and the users can use cloud services more trustingly. Compared with the prior art, the method has the advantages that extra calculation and storage space are not introduced, a GPU or TPU acceleration function needs to be used for renting the computing power cloud by a general user, and the computing power of a CPU is generally utilized in the process of Tokenization, so that extra calculation and storage space are not introduced when the user side performs Tokenization.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a large language model training optimization method and software on a computing cloud. Background Art

[0002] An important application scenario of computing cloud at present is the training of large language models, including pre-training, supervised fine-tuning and reinforcement learning fine-tuning. Computing cloud will integrate computing power into various forms of cloud services. For example, it can be a GPU card on a virtual machine, or provide a customized training interface. For customized training interface scenarios, after the user uploads the data, he can specify various configuration parameters of the framework for training. Generally, the existing large models are basically based on the Transformer architecture, and the frameworks used are also general frameworks such as transformers and deepspeed. Generally, the form of directly uploading text training data is adopted, which leads to the risk of leakage of user data. If the cloud platform does not have sufficient permission control or is attacked by hackers, it may be leaked. Since much of the data of the large language model is text data or text data in a common compression format, the loss caused by leakage is greater than that of files in proprietary formats. Summary of the invention

[0003] In view of this, the present application proposes a large language model training optimization method on a computing cloud to solve the problems reflected in the above-mentioned background technology.

[0004] This application provides a large language model training optimization method on a computing cloud, including:

[0005] In the offline tool software, disrupt the internal order of the Tokenizer to obtain a new Token sequence;

[0006] According to the new Token sequence, the Embedding matrix is ​​adjusted according to the corresponding position to obtain a new Embedding matrix corresponding to the new Token sequence;

[0007] Retrain the Tokenizer according to the new Token sequence and the new Embedding matrix;

[0008] Generate the corresponding Tokenized sample vector based on the retrained Tokenizer;

[0009] Test the modified large language model, upload sample vectors and corresponding embedding matrices, and perform the supervised fine-tuning process.

[0010] Optionally, the large language model training optimization method on the computing cloud is characterized by:

[0011] The Token sequence in the Tokenizer is randomly arranged to disrupt the internal order of the Tokenizer and obtain a new Token sequence;

[0012] Optionally, the retraining of Tokenizer includes:

[0013] Retrain the Tokenizer based on your own training data, add the newly found Token to the Tokenizer, and establish the correspondence between the new and old Tokenizers.

[0014] Optionally, establishing the correspondence between the new and old Tokenizers further includes:

[0015] By tokenizing the same text with the old and new Tokenizers, the corresponding relationship between the original Tokenizer and the new Tokenizer is established;

[0016] Copy the vectors at the corresponding positions of the original Embedding matrix to the corresponding positions of the new Embedding matrix; and

[0017] When there is no corresponding vector in the original Embedding matrix, it is filled with random initialization or the mean of other vectors.

[0018] Optionally, the large language model training optimization method on the computing power cloud is characterized by comprising:

[0019] The perturbation weight matrix is ​​used to slightly perturb the weight matrix to avoid data leakage that may lead to the inverse deduction of the corresponding relationship between the various tokens; and the user can customize the distribution and amplitude of the random amount of the perturbation of the perturbation matrix.

[0020] Optionally, the generating a corresponding Embedding matrix and a Tokenized sample vector according to the retrained Tokenizer further includes:

[0021] According to the retrained Tokenizer, the input text data is segmented to generate Tokens, and an Embedding matrix corresponding to the modified Tokenizer is generated.

[0022] Optionally, the testing of the modified large language model includes:

[0023] Test whether the modified large language model can be restored to the original model to avoid compatibility issues caused by the modification.

[0024] The present application also provides a device for training a large language model on a computing cloud, characterized in that the device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the large language model training optimization method on a computing cloud as described in any one of claims 1 to 7.

[0025] Optionally, the storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large language model training optimization method on the computing cloud are implemented as described in any one of claims 1 to 7.

[0026] The present application also provides an electronic device, characterized in that it includes:

[0027] A computer device, characterized in that it includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the large language model training optimization method on the computing power cloud described in any one of claims 1 to 7 is implemented.

[0028] The beneficial effects of the present application are: reducing the risk of data leakage, which is beneficial to both computing power service providers and users. The present invention can reduce the possibility of data leakage for service providers and can make users feel more assured to use cloud services. No additional computing and storage capacity is introduced. Generally, users who rent computing power clouds need to use GPU or TPU acceleration functions. The Tokenize process generally also utilizes the computing power of the CPU. Therefore, Tokenizing on the user side does not introduce additional computing and storage capacity. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0030] Figure 1 A flow chart of a large language model training optimization method on a computing cloud disclosed in the present application is shown;

[0031] Figure 2 A supervisory fine-tuning flowchart disclosed in the present application is shown. DETAILED DESCRIPTION

[0032] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0033] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0034] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0036] This application proposes an algorithm for training a large language model on a computing cloud. The main feature of this algorithm is that the user's training data will not be leaked during training. When training on a cloud platform, some users may need to encrypt their training data. However, most existing large language model training platforms often do not provide this function. The present invention proposes a method for processing data during the Tokenize process. When submitting training data on the platform, only the processed data is uploaded. This method can reduce the risk of data leakage.

[0037] Among them, the computing cloud is a cloud service that provides acceleration cards such as GPU (Graphics Processing Unit) and TPU (Tensor Processing Unit); the Tokenize is the process of dividing text into Tokens, which involves word segmentation, punctuation removal, lowercase conversion and other processing of the text to generate a Token sequence. The purpose of Tokenize is to convert the original text data into a form that can be understood and processed by the machine. In practical applications, Tokenize is usually used in the text preprocessing stage to provide basic data for subsequent natural language understanding, text classification, sentiment analysis and other tasks; the Token is the basic unit in natural language processing, usually referring to a word or symbol. In the text, a Token can be a word, punctuation, number or other special character. The concept of a Token is similar to a "word" or "symbol" that we understand in daily life. In NLP (Natural Language Processing), by tokenizing the text, we can convert a continuous character sequence into a discrete token sequence, laying the foundation for subsequent text processing and analysis; the supervised fine-tuning is also called instruction fine-tuning, which uses supervised instruction data and mainly trains the large language model to have the ability to respond to user instructions; the reinforcement learning fine-tuning is also called preference alignment, which gives positive and negative examples on the basis of instruction fine-tuning, and trains the large language model so that the results it gives are consistent with human value orientation.

[0038] like Figure 2 As shown, the supervised fine-tuning flowchart includes the following:

[0039] First, the text data is converted into Tokens through Tokenize. During this process, the text is segmented, punctuation is removed, and lowercase is performed. The original text data is converted into a form that can be understood and processed by the machine in order to generate the corresponding Token sequence.

[0040] Secondly, the generated multiple data samples are batch-integrated to form a small batch of data, so as to perform batch processing on the data samples, effectively utilizing the parallel computing capabilities of the GPU to improve computing efficiency. In addition, small batch training will greatly reduce memory consumption compared to processing samples one by one.

[0041] Then, for data samples of different lengths in the batch, collate is used to pad the data to ensure that all samples have the same length before entering the Transformer model.

[0042] Finally, the Trainer in the Transformer model is called for training. The Trainer encapsulates the main logic of the training loop, including data loading, forward propagation, loss calculation, back propagation, and parameter update steps.

[0043] Take supervised fine-tuning as an example to illustrate the specific steps and improvements of this method. Figure 2 As shown in the figure, after the text data is Tokenized, it becomes numbers, and then these numbers are converted into vector inputs through the Embedding matrix. The Embedding matrix determines the vector value corresponding to each Token.

[0044] The embedding matrix is ​​a technique for converting discrete data (such as words, objects, categories, etc.) into continuous vectors. In machine learning and deep learning, embedding usually refers to converting high-dimensional or discrete data representations into a low-dimensional real vector. This process helps to represent and calculate data in vector space. In natural language processing, word embedding is the most classic application of embedding, which maps words from a discrete index to a low-dimensional continuous vector. These vectors help capture semantic relationships and contextual information in language.

[0045] Then, the sample is generated using the Transformer model. For each Token position of the sample, a Transformer model output Logit vector is obtained, and the vector dimension is the total number of all Tokens. The Logit vector is the prediction of the next token at this position. The output Token value can be obtained by taking the maximum or sampling.

[0046] The Transformer model is a deep learning model architecture for natural language processing and sequence-to-sequence tasks. It mainly consists of two parts: an encoder and a decoder. The encoder is responsible for converting the input sequence into a series of vector representations, while the decoder is responsible for generating the output sequence. The self-attention mechanism is the core of the Transformer model. It is implemented by calculating the similarity between each position in the input sequence and all other positions. There are multiple self-attention layers and feedforward neural network layers between the encoder and the decoder.

[0047] If you need to get the text, then detokenize it to convert it into text. During supervised training, if you do not need to view the intermediate text output results, you do not need to detokenize to get the text. Instead, you can directly use the true value of the output token and the loss value of the Logit vector prediction value to perform back propagation for training.

[0048] Through the above analysis, this application proposes that when training on the cloud platform, you only need to upload the Embedding matrix and the sample vector after Tokenizer (including the input vector and the output target vector).

[0049] This way, you don't have to upload the original text data. Figure 2 In the figure, the general cloud service is operated at the first arrow, and this application is operated at the second arrow.

[0050] The Tokenizer is a tool or library for performing the Tokenize operation. In the field of NLP, Tokenizer plays an important role in converting text into a sequence of Tokens. Different Tokenizers may use different algorithms and strategies for Tokenization to adapt to different language characteristics and application requirements. For example, for Chinese text, the segmenter is a commonly used Tokenizer that can segment Chinese sentences into independent words or phrases.

[0051] like Figure 1 As shown, a large language model training optimization method on a computing cloud in one embodiment of the present application is shown. The method includes the following contents:

[0052] S100, the present application is to change the Tokenizer configuration in the software, disrupt the order in the Tokenizer, and obtain a new Token sequence.

[0053] Specifically, the order in Tokenizer refers to the order in which the tokenizer breaks down the text into smaller units (such as words, characters, or words) when processing text. This process includes: word segmentation, breaking down the text into smaller units, such as words or characters; building a dictionary, building a dictionary mapping based on the results of word segmentation. If pre-trained word vectors are used, the dictionary mapping needs to be processed according to the word vector file; data conversion, converting the segmented data into a digital sequence, which usually involves mapping each unit to a unique digital ID; padding and truncation, when batch input into the model, padding is performed on data that is too short and truncating is performed on data that is too long to ensure that the data length meets the requirements of the model and that the data dimension within the batch is consistent.

[0054] S200, the present application adjusts the Embedding matrix according to the corresponding position according to the new Token sequence in the software to obtain a new Embedding matrix corresponding to the new Token sequence.

[0055] Specifically, the Embedding matrix, as a continuous step that is closely related and complementary to the Tokenizer, should also be changed according to the corresponding position as the internal order of the Tokenizer is randomly disrupted.

[0056] S300, the present application is to retrain the Tokenizer in software according to the new Token sequence and the new Embedding matrix.

[0057] The retraining of the Tokenizer refers to retraining the Tokenizer based on the own training data, adding the newly found Token to the Tokenizer, and then tokenizing the same text according to the old and new Tokenizers to establish the correspondence between the original Tokenizer and the new Tokenizer. The vectors at the corresponding positions of the original Embedding matrix are copied, and the vectors that are not in the original Embedding matrix are filled with random initialization or the mean of other vectors.

[0058] Among them, this application perturbs the weight matrix in the software to avoid inferring the correspondence between each token from the Embedding matrix when data is leaked. Since some pre-trained model weights used for supervised fine-tuning are universal, if data is leaked, it may be possible to infer the correspondence between each token from the Embedding matrix. Therefore, this application supports a small amount of perturbation of the weight matrix, and users can customize the distribution and amplitude of the random amount of perturbation.

[0059] The weight matrix can be obtained through pre-training and fine-tuning, in which case the pre-trained weights are used and then fine-tuned on a specific task. Pre-training refers to training a model on a large-scale, general task. For example, the BERT model is pre-trained on a large-scale unsupervised text corpus. The tasks include masked language modeling and next sentence prediction. During the pre-training process, the model learns common language features and patterns. Fine-tuning refers to further training the model on the training data of a specific task based on the pre-trained model. For example, a small-scale sentiment analysis dataset can be used to fine-tune the model based on the pre-trained BERT model to adapt it to sentiment analysis tasks.

[0060] S400, this application generates corresponding matrices and vectors after the Tokenizer configuration change is completed.

[0061] Specifically, according to the retrained Tokenizer configuration, the corresponding Embedding matrix is ​​generated according to the new Tokenizer configuration, as well as the sample vector converted by the newly generated Embedding matrix. Finally, it is necessary to test whether the modified model can be restored to the original model to avoid compatibility issues caused by the modification.

[0062] S500, this application uploads the newly generated matrix and sample vector to the computing cloud after checking the modified large language model, and continues to execute the supervised fine-tuning process.

[0063] Specifically, we test whether the modified large language model can be restored to the original model to verify whether there are compatibility issues. Sending the newly generated matrix and sample vector to the computing cloud ensures the normal operation of the supervised fine-tuning process and avoids information leakage caused by directly uploading text data.

[0064] The specific implementation process of this application is as follows:

[0065] In supervised fine-tuning, the general data flow is as follows Figure 2 shown.

[0066] After the text data is tokenized, it becomes numbers, which are then converted into vector inputs through the Embedding matrix. The Embedding matrix determines the vector value corresponding to each token.

[0067] Then the sample is generated using the Transformer model. For each token position of the sample, a Transformer model output logit vector is obtained, and the vector dimension is the total number of all tokens.

[0068] The logit vector is the prediction of the next token at that position. By taking the maximum or sampling, we can get the output token value.

[0069] If you need to get text, detokenize it and convert it into text. During supervised training, if you do not need to view the intermediate text output results, you do not need to detokenize to get text. Instead, you can directly use the true value of the output token and the loss value of the logit vector prediction value to perform back propagation for training.

[0070] Through the above analysis, the present invention proposes that when training on the cloud platform, only the following information needs to be uploaded:

[0071] 1. Embedding Matrix

[0072] 2. The sample vector after tokenizer, including the input vector and the output target vector.

[0073] In this way, it is not necessary to upload the original text data. Equivalent to the above data flow diagram, the general cloud service operates at the first arrow, and the present invention operates at the second arrow.

[0074] The embodiment of the present invention is a tool software, which is used offline and allows users to generate tokenizer configuration files, tokenized sample vectors and corresponding embedding matrices.

[0075] The tool software provides the following functions:

[0076] 1. Randomly shuffle the order in the tokenizer and change the embedding matrix according to the corresponding position.

[0077] 2. You can retrain the tokenizer with your own training data and add the newly found tokens to the tokenizer. Then, by tokenizing the same text with the old and new tokenizers, establish the correspondence between the original tokenizer and the new tokenizer. Copy the vectors at the corresponding positions of the original matrix, and fill in the vectors that are not in the original embedding matrix with random initialization or the mean of other vectors.

[0078] 3. Since some pre-trained model weights used for supervised fine-tuning are universal, if the data is leaked, it may be possible to infer the correspondence between each token from the embedding matrix. The software supports a small amount of perturbation of the weight matrix, and users can customize the distribution and amplitude of the random amount of perturbation.

[0079] 4. Generate the corresponding embedding matrix and tokenized sample vector according to the modified tokenizer. Note that the label of supervised training is also included in the tokenized sample vector.

[0080] 5. Test the ability to restore the modified model back to the original model to avoid compatibility issues caused by modifications.

[0081] It should be noted that if a new token is added, its embedding is randomly generated or filled with a mean vector, so its parameters need to be specified during the training process.

[0082] This application can greatly reduce the risk of data leakage, which is beneficial to both computing service providers and users. The present invention can reduce the possibility of data leakage for service providers and can make users feel more at ease using cloud services. No additional computing and storage capacity is introduced. Generally, users who rent computing cloud need to use GPU or TPU acceleration functions. The Tokenize process generally also utilizes the computing power of the CPU. Therefore, Tokenizing on the user side does not introduce additional computing and storage capacity.

[0083] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A large language model training optimization method on a computing cloud, characterized in that: include: In the offline tool software, disrupt the internal order of the Tokenizer to obtain a new Token sequence; According to the new Token sequence, the Embedding matrix is ​​adjusted according to the corresponding position to obtain a new Embedding matrix corresponding to the new Token sequence; Retrain the Tokenizer according to the new Token sequence and the new Embedding matrix; Generate the corresponding Tokenized sample vector based on the retrained Tokenizer; Test the modified large language model, upload sample vectors and corresponding embedding matrices, and perform the supervised fine-tuning process.

2. The method for optimizing large language model training on a computing cloud as claimed in claim 1, characterized in that: The Token sequence in the Tokenizer is randomly arranged to disrupt the internal order of the Tokenizer and obtain a new Token sequence.

3. The method for optimizing large language model training on a computing cloud as claimed in claim 1, characterized in that: The retraining of Tokenizer includes: Retrain the Tokenizer based on your own training data, add the newly found Token to the Tokenizer, and establish the correspondence between the new and old Tokenizers.

4. The method for optimizing large language model training on a computing cloud as claimed in claim 3, characterized in that: The establishing of the correspondence between the new and old Tokenizers includes: By tokenizing the same text with the old and new Tokenizers, the corresponding relationship between the original Tokenizer and the new Tokenizer is established; Copy the vectors at the corresponding positions of the original Embedding matrix to the corresponding positions of the new Embedding matrix; and When there is no corresponding vector in the original Embedding matrix, it is filled with random initialization or the mean of other vectors.

5. The method for optimizing large language model training on a computing cloud as claimed in claim 1, characterized in that: Also includes: Perturb the weight matrix, and perform a small amount of perturbation on the weight matrix through the perturbation weight matrix to avoid data leakage and the corresponding relationship of each token being inferred; And the user can customize the distribution and amplitude of the random amount of the perturbation of the perturbation matrix.

6. The method for optimizing large language model training on a computing cloud as claimed in claim 1, characterized in that: The generating of the corresponding Embedding matrix and the Tokenized sample vector according to the retrained Tokenizer includes: According to the retrained Tokenizer, the input text data is segmented to generate Tokens, and an Embedding matrix corresponding to the modified Tokenizer is generated.

7. The method for optimizing large language model training on a computing cloud as claimed in claim 1, characterized in that: The modified large language model of the test includes: Test whether the modified large language model can be restored to the original model to avoid compatibility issues caused by the modification.

8. A device for training a large language model on a computing cloud, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large language model training optimization method on a computing cloud as described in any one of claims 1 to 7.

9. The device for training a large language model on a computing cloud according to claim 8, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large language model training optimization method on the computing cloud are implemented as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A computer device, characterized in that It includes a memory and a processor, the memory stores a computer program, and the processor implements the large language model training optimization method on the computing power cloud as described in any one of claims 1 to 7 when executing the computer program.

Images