A long sequence processing method, system and medium based on large language model

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a long sequence processing method, system, and medium based on a large language model, comprising: using a long sequence as a training sample input parameter to a large language model with frozen parameters; after a single forward propagation and backpropagation, using the gradient of the token embedding output by the embedding layer as an indicator to identify key tokens, and using the gradient of the irrelevant token embedding to adjust the irrelevant token embedding, while keeping the key token embedding unchanged, to obtain a denoised token embedding; unfreezing the model parameters, subjecting the denoised token embedding to a single forward propagation and backpropagation, and updating the model parameters until the network converges; and inputting the long sequence to be processed into the trained large language model to obtain a response result. The present invention helps the large language model to efficiently and accurately learn to extract key tokens during the training process, effectively suppresses background noise in the long sequence, and enables the trained large language model to output accurate response results.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a long sequence processing method, system and medium based on a large language model. Background Art

[0002] The ability to process long input sequences has become a fundamental requirement for large language models (LLMs). Currently, a handful of cutting-edge long-context models (LCMs) are capable of handling input sequences exceeding millions of tokens, meeting the demands of complex toolchains and workflows. For example, in the legal field, long-context models, leveraging their ability to process massive amounts of legal documents (such as thousands of pages of contracts and casebooks), can rapidly correlate regulatory clauses, identify points of dispute, and match precedents through deep semantic analysis and key information extraction, providing efficient support for legal research and case analysis. In the field of project code analysis, long-context models can comprehensively analyze code projects containing multiple modules and files, extracting key information from massive lines of code and building relationships between code elements. This improves the quality and efficiency of software development, ensuring stable and reliable operation over long-term iterations.

[0003] Existing long-context models process long input sequences using a retrieval-generation approach. This involves first implicitly identifying key tokens within the context and then aggregating the context based on these key tokens to generate output. However, key tokens can be overwhelmed by excessive irrelevant information in the context during identification. Failure to accurately identify key tokens can lead to a distorted understanding of the semantics of long input sequences, resulting in reduced accuracy in the model's output. Summary of the Invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art that the key tokens cannot be accurately identified, resulting in a decrease in the accuracy of the model output results.

[0005] To solve the above technical problems, the present invention provides a long sequence processing method based on a large language model, comprising:

[0006] S1: The token sequence corresponding to the long sequence is used as the training sample input parameter of the frozen large language model, and the output sequence is obtained after one forward propagation. The loss function value between the output sequence and the true label is calculated. The loss function value is differentiated in the backpropagation to obtain the gradient of each token embedding in the token embedding sequence output by the embedding layer.

[0007] S2: The average value of the gradients embedded in each token is used as a threshold; if the gradient embedded in a token is greater than or equal to the threshold, the token is a key token; if the gradient embedded in a token is less than the threshold, the token is an irrelevant token;

[0008] S3: Use the gradient of irrelevant token embedding to adjust irrelevant token embedding, while the key token embedding remains unchanged, to obtain the denoised token embedding sequence;

[0009] S4: Unfreeze the model parameters, pass the denoised token embedding sequence through forward propagation and backpropagation, and update the model parameters. Return to S1 until the model parameters converge, and the trained large language model is obtained.

[0010] S5: Input the long sequence to be processed into the trained large language model to obtain the response result.

[0011] Preferably, the large language model includes an embedding layer, a position encoding layer, a multi-layer stacked self-attention-feedforward layer and an output layer connected in sequence.

[0012] Preferably, converting the long sequence into a token sequence includes: using a word segmenter to split the long sequence into a concatenation of several words, and assigning an integer index to each word as a token to obtain a token sequence corresponding to the long sequence.

[0013] Preferably, the average value of the gradients embedded in each token is used as the threshold, and the formula is expressed as:

[0014] ;

[0015] in, is the threshold, is the i-th token, is a token sequence, is the token embedding sequence output by the embedding layer, is the number of tokens in the token sequence, Embed for the i-th token; is the true label, is the output sequence, is the loss function, are model parameters; is the gradient of the i-th token embedding, are the embedding layer parameters, According to the loss function value The gradient of the obtained embedding of the i-th token.

[0016] Preferably, if the gradient of token embedding is greater than or equal to the threshold, the token is a key token; if the gradient of token embedding is less than the threshold, the token is an irrelevant token, and the formula is expressed as:

[0017] ;

[0018] in, is the i-th token, for Token ID, hour, is the key token, hour, For irrelevant tokens; is a token sequence, is the token embedding sequence output by the embedding layer, is the number of tokens in the token sequence, Embed for the i-th token; is the true label, is the output sequence, is the loss function, are model parameters; is the gradient of the i-th token embedding, are the embedding layer parameters, According to the loss function value The gradient of the i-th token embedding is obtained, is the threshold.

[0019] Preferably, the irrelevant token embedding is adjusted using the gradient of the irrelevant token embedding, and the key token embedding remains unchanged, to obtain the denoised token embedding sequence, which is expressed as:

[0020] ;

[0021] in, is the i-th token, For the i-th token embedding, is the denoised token embedding of the i-th token, for Token ID, is the gradient of the i-th token embedding, is the learning rate, To control the hyperparameters of denoising;

[0022] The denoised token embedding sequence is represented as .

[0023] Preferably, The value is on the order of 1e-5.

[0024] Preferably, the loss function of the large language model is cross entropy loss.

[0025] The present invention also provides a long sequence processing system based on a large language model, comprising:

[0026] The gradient calculation module is used to use the token sequence corresponding to the long sequence as the training sample input parameter of the frozen large language model, and obtain the output sequence after a forward propagation. The loss function value between the output sequence and the true label is calculated. The loss function value is differentiated in the back-propagation to obtain the gradient of each token embedding in the token embedding sequence output by the embedding layer.

[0027] A key token identification module is configured to use the average value of the gradients embedded in each token as a threshold; if the gradient embedded in a token is greater than or equal to the threshold, the token is a key token; if the gradient embedded in a token is less than the threshold, the token is an irrelevant token;

[0028] The denoising training module is used to adjust the irrelevant token embeddings using the gradient of the irrelevant token embeddings, while the key token embeddings remain unchanged, to obtain the denoised token embedding sequence;

[0029] The parameter update module is used to unfreeze the model parameters and update the model parameters by embedding the denoised tokens into the sequence through forward propagation and backpropagation. The module returns to the gradient calculation module until the model parameters converge, thus obtaining a fully trained large language model.

[0030] The inference module is used to input the long sequence to be processed into the trained large language model to obtain the response result.

[0031] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned long sequence processing method based on a large language model are implemented.

[0032] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0033] The long sequence processing method based on a large language model described in the present invention uses the gradient of the token embedding output by the embedding layer as the basis for screening key tokens during model training. Compared with traditional key token screening methods, it has higher computational efficiency and good versatility. The irrelevant token embeddings are further denoised using the gradient of the irrelevant token embeddings, while the key token embeddings are kept unchanged. This can suppress the model's attention to irrelevant tokens, thereby enhancing the model's attention to key tokens, effectively suppressing background noise in long sequences, and helping the large language model to efficiently and accurately learn to extract key tokens from long sequences during training. This enables the trained large language model to effectively identify key information in long sequences during inference and output accurate response results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0035] Figure 1 It is a flow chart of a long sequence processing method based on a large language model of the present invention;

[0036] Figure 2 This is a flow chart of the present invention for training a large language model;

[0037] Figure 3 This is a comparison chart of the CDT key token detection method proposed in the present invention and the IG score;

[0038] Figure 4 This is a comparison chart of the CDT key token detection method proposed in the present invention, LongPPL, and attention weight;

[0039] Figure 5 This is a comparison chart of the three training methods of SFT, DPO and CDT proposed by the present invention;

[0040] Figure 6 This is a comparison of the attention matrix before and after the model uses CDT, where Figure 6 (a) in the figure is the attention matrix diagram of the 24th layer using the pre-CDT model. Figure 6 (b) in the figure is the attention matrix diagram of the model at the 24th layer after using CDT. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0042] Reference Figure 1 As shown, the present invention provides a long sequence processing method based on a large language model, comprising:

[0043] S1: Reference Figure 2 As shown, the token sequence corresponding to the long sequence is used as a training sample input parameter frozen large language model, and the output sequence is obtained after one forward propagation; the loss function value between the output sequence and the true label is calculated, and the loss function value is derived in the back propagation to obtain the gradient of each token embedding in the token embedding sequence output by the embedding layer; the long sequence includes a long text sequence or a long code sequence;

[0044] S2: The gradient of the token embedding output by the embedding layer is positively correlated with the attention value, and the average of the gradients of each token embedding is used as the threshold; if the gradient of the token embedding is greater than or equal to the threshold, the token is a key token; if the gradient of the token embedding is less than the threshold, the token is an irrelevant token;

[0045] S3: Use the gradient of irrelevant token embedding to adjust irrelevant token embedding, while the key token embedding remains unchanged, to obtain the denoised token embedding sequence;

[0046] S4: Unfreeze the model parameters, pass the denoised token embedding sequence through forward propagation and backpropagation, and update the model parameters. Return to S1 until the model parameters converge, and the trained large language model is obtained.

[0047] S5: Input the long sequence to be processed into the trained large language model to obtain the response result.

[0048] Preferably, converting a long sequence into a token sequence includes using a tokenizer to split the long sequence into a concatenation of several words, assigning each word an integer index as a token, and obtaining a token sequence corresponding to the long sequence. For example, "The weather is very good today" is split into four words: "Today, the weather is very good." Each word is then assigned an integer index, such as "0, 1, 2, 3," for subsequent calculations. These indexed words are called "tokens." For any long sequence, its corresponding token sequence, i.e., the corresponding integer index sequence, can be obtained.

[0049] Preferably, the large language model includes an embedding layer, a position encoding layer, a multi-layer stacked self-attention-feedforward layer and an output layer connected in sequence. The embedding layer is used to generate word vectors for tokens to obtain token embeddings. Taking into account that the same word vector may be assigned to the same token in different positions multiple times, in order to distinguish the same tokens in different positions, the token embedding output by the embedding layer will pass through the position encoding layer to add position encoding to the word vector. The purpose of the self-attention-feedforward layer is to improve the fitting ability of the model, and the multi-layer stacking further improves the representation ability of the model. Finally, the output features of the self-attention-feedforward layer are passed into the output layer to obtain the probability of the next word for each word, which can be autoregressive during decoding.

[0050] In order to discover the attention interactions between tokens, existing technologies usually use the integrated gradient (IG) score for key token detection. The IG score of the layer attention module is defined as:

[0051] ;

[0052] in, For the IG score of the layer attention module, is the loss function, are model parameters, is the input token sequence, is the true label, For the The attention matrix of the h-th attention head of the layer attention module.

[0053] By calculating each token Its real label IG scores between, e.g. , i and j are token position indices. The IG score of each token is obtained by summing up from all attention heads. The higher the average value of the IG score, the greater the contribution of the token to the true label.

[0054] Considering the multi-layer stacked self-attention-feedforward layer structure of a large language model, consider the case with only one embedding layer and one self-attention-feedforward layer. The attention mechanism can be expressed as:

[0055] ;

[0056] ;

[0057] in, are query, key and value respectively, is the parameter matrix, is the attention value, For attention output, is the token embedding sequence output by the embedding layer, and represent the sequence length and model dimension respectively.

[0058] Assume the loss function is , according to the chain rule, the loss function is The derivative of is:

[0059] ;

[0060] Because there is and ,but and The gradient relationship between them is:

[0061] ;

[0062] To further eliminate The influence of the function can be further decomposed into:

[0063] ;

[0064] in, Score for attention, for The Jacobian of the function, whose elements are , Kronecker function, k is the token position index.

[0065] For attention score Each element in , about The gradient of can be written as:

[0066] ;

[0067] Therefore, for a large language model containing a self-attention-feedforward layer, which includes an attention mechanism, we can deduce the relationship between the gradient of the token embedding output by the embedding layer and the attention value:

[0068] ;

[0069] in, is the loss function value, is the token embedding of the i-th position in the token embedding sequence, is the attention value of the token embedding at the i-th position to the token embedding at the j-th position, The attention score of the token embedding at position i on the token embedding at position j.

[0070] The above derivation process proves that the gradient of the token embedding output by the embedding layer is positively correlated with the attention value. That is, the larger the gradient of the token embedding output by the embedding layer, the larger the attention weight value, indicating that the corresponding token is more critical. The gradient of the token embedding output by the embedding layer can be used as a criterion for identifying key tokens, which is more computationally efficient than the IG score.

[0071] Specifically, the average value of the gradient of each token embedding in a long sequence is used as the threshold, and the formula is expressed as:

[0072] ;

[0073] in, is the threshold, is the i-th token, is a token sequence, is the token embedding sequence output by the embedding layer, is the number of tokens in the token sequence, Embed for the i-th token; is the true label, is the output sequence, is the loss function, are model parameters; is the gradient of the i-th token embedding, are the embedding layer parameters, According to the loss function value The gradient of the obtained embedding of the i-th token.

[0074] The average value of the gradient embedded in each token is used as the threshold; if the gradient embedded in the token is greater than or equal to the threshold, the token is a key token; if the gradient embedded in the token is less than the threshold, the token is an irrelevant token. The formula is expressed as:

[0075] ;

[0076] in, for Token ID, hour, is the key token, hour, Unrelated token.

[0077] In order to suppress background noise, the present invention uses the gradient of token embedding to perform context denoising training (CDT). That is, the irrelevant token embedding is adjusted with the gradient of irrelevant token embedding, while the key token embedding remains unchanged. The denoised token embedding sequence is obtained, which is expressed as:

[0078] ;

[0079] in, is the i-th token, For the i-th token embedding, is the denoised token embedding of the i-th token, for Token ID, is the gradient of the i-th token embedding, is the learning rate, To control the hyperparameters of denoising;

[0080] The denoised token embedding sequence is represented as .

[0081] Preferably, in order not to interfere too much with the embedding layer parameters, The value is usually on the order of 1e-5.

[0082] Taking the gradient of the token embedding output by the embedding layer can comprehensively regulate the influence of the same token at different positions on the token during training to improve model performance.

[0083] Loss function used to update model parameters Expressed as:

[0084] .

[0085] Preferably, the loss function is cross entropy loss.

[0086] In summary, the long sequence processing method based on a large language model described in the present invention uses the gradient of the token embedding output by the embedding layer as the basis for screening key tokens when training the model. Compared with traditional key token screening methods, it has higher computational efficiency and good versatility. It further denoises the irrelevant token embeddings using the gradient of the irrelevant token embeddings, keeping the key token embeddings unchanged, which can suppress the model's attention to irrelevant tokens, thereby enhancing the model's attention to key tokens, effectively suppressing background noise in long sequences, and helping the large language model to efficiently and accurately learn to extract key tokens of long sequences during the training process, so that the trained large language model can effectively identify the key information of long sequences during inference and output accurate response results.

[0087] In order to experimentally prove that it is reasonable to use the gradient of the token embedding output by the embedding layer to approximate the IG score, this embodiment uses these two methods to detect key tokens on synthetic multi-hop data, and respectively counts the proportion of the gradient or score of these key tokens in the global context, as well as the overlap of the tokens detected by the two methods. The results are shown in Figure 2. Figure 3 As shown. It can be seen that the detection effects of the two are similar on key clues, and Figure 3 The proportion of tokens detected by the gradient of the token embedding output by the embedding layer is higher than that detected by the IG score, which indicates that the key tokens detected by the gradient of the embedding layer output are more important.

[0088] In order to further illustrate the reliability of the CDT key token detection method proposed in this invention, this example compares it with LongPPL and attention weight. The results are as follows: Figure 4 As shown in the figure, the CDT key token detection method is significantly better than LongPPL. As for the attention weight, since it is an indicator that directly reflects whether the token is important, the CDT key token detection method has a higher detection rate on supporting answer tokens and a lower detection rate on interference tokens and irrelevant tokens. This shows that the CDT key token detection method of the present invention can not only capture key information more accurately, but also block interference information to a certain extent.

[0089] Based on the CDT key token detection method, to evaluate the effectiveness of the CDT training strategy, this example post-trained the Llama-3-8B-Base, Llama-3.1-8B-Base, and Llama-3.1-8B-Instruct models. These models were evaluated on multiple long-context benchmarks and compared with several methods post-trained using the same datasets and models. To test CDT's performance on natural language tasks, LongBench-E was used as the benchmark for evaluation. The results are shown in Table 1, where the bolded data represents the optimal results.

[0090] Table 1. Evaluation results of various models using LongBench-E

[0091]

[0092] As can be seen from Table 1, the CDT strategy proposed in this paper achieves the best comprehensive performance on all three models compared with similar models, especially in the multi-hop task scenario, comprehensively surpassing other methods.

[0093] To further test CDT's noise suppression capabilities, this example further evaluates it on a synthesis task. This task uses only cues as key information and extends the context length through noise context padding. Specifically, this example uses RULER and Babilong for evaluation, focusing on long contexts of 64k and 128k by controlling padding. Table 2 shows the evaluation results of each model on the RULER (left) and Babilong (right) benchmarks, with the best results indicated in bold.

[0094] Table 2. Evaluation results of each model on the RULER and Babilong benchmarks

[0095]

[0096] Since the information of key tokens is relatively sparse, the CDT strategy proposed in this paper can significantly suppress noise and capture key information more accurately than other methods. As can be seen from Table 2, CDT has achieved the best performance in long context scenarios (>64k).

[0097] Since CDT explicitly intervenes in the training process, in order to prove that the CDT strategy can gradually converge during training, under the same other conditions, this example compares the performance improvement of the model on LongBench-E during training using three methods: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and CDT. Figure 5 As shown in the figure, the CDT method of the present invention consumes only a small amount of time during training compared to DPO (the time for one forward propagation - backpropagation to obtain the embedding layer gradient), but it can significantly improve the performance of LCM. This improvement increases steadily with training, and like other methods, the convergence boundary appears at about 250 steps.

[0098] Since the prerequisite for using the CDT strategy is only to ensure that the model contains an embedding layer, and the embedding layer is a component common to all types of large language models, this embodiment also uses the CDT strategy for post-training on Qwen2.5-7B-Instruct and Mistral-V0.3-Instruct, as shown in Table 3.

[0099] Table 3. Training results after using CDT strategy on Qwen2.5-7B-Instruct and Mistral-V0.3-Instruct

[0100]

[0101] It can be seen that the CDT strategy still achieved better overall performance on both the LongBench-E and Babilong benchmarks.

[0102] This example also randomly selects the same 128k context samples and performs inference on the base model and the model trained using the CDT strategy, and plots the average of the attention matrix of the last 24 layers. For ease of display, the attention matrix is ​​averaged for each 3k*3k block. The results are shown in Figure 6, where Figure 6 (a) in the figure is the attention matrix diagram of the 24th layer using the pre-CDT model. Figure 6 (b) shows the attention matrix of the model at layer 24 after using CDT. Colors closer to red indicate greater attention paid by the model to the corresponding context fragment. As shown in the red box, the model trained with CDT pays significantly more attention to key information.

[0103] Based on the above-mentioned long sequence processing method based on a large language model, the present invention also provides a long sequence processing system based on a large language model, comprising:

[0104] The gradient calculation module is used to use the token sequence corresponding to the long sequence as the training sample input parameter of the frozen large language model, and obtain the output sequence after a forward propagation. The loss function value between the output sequence and the true label is calculated. The loss function value is differentiated in the back-propagation to obtain the gradient of each token embedding in the token embedding sequence output by the embedding layer.

[0105] A key token identification module is configured to use the average value of the gradients embedded in each token as a threshold; if the gradient embedded in a token is greater than or equal to the threshold, the token is a key token; if the gradient embedded in a token is less than the threshold, the token is an irrelevant token;

[0106] The denoising training module is used to adjust the irrelevant token embeddings using the gradient of the irrelevant token embeddings, while the key token embeddings remain unchanged, to obtain the denoised token embedding sequence;

[0107] The parameter update module is used to unfreeze the model parameters and update the model parameters by embedding the denoised tokens into the sequence through forward propagation and backpropagation. The module returns to the gradient calculation module until the model parameters converge, thus obtaining a fully trained large language model.

[0108] The inference module is used to input the long sequence to be processed into the trained large language model to obtain the response result.

[0109] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned long sequence processing method based on a large language model are implemented.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A long sequence processing method based on a large language model, characterized in that: include: S1: The token sequence corresponding to the long sequence is used as the training sample input parameter of the large language model, and the output sequence is obtained after one forward propagation; Calculate the loss function value between the output sequence and the true label, and derive the loss function value in the back propagation to obtain the gradient of each token embedding in the token embedding sequence output by the embedding layer; S2: The average value of the gradients embedded in each token is used as a threshold; if the gradient of a token embedded is greater than or equal to the threshold, the token is a key token; If the gradient of the token embedding is less than the threshold, the token is an irrelevant token; S3: Use the gradient of irrelevant token embedding to adjust irrelevant token embedding, while the key token embedding remains unchanged, to obtain the denoised token embedding sequence; S4: Unfreeze the model parameters, pass the denoised token embedding sequence through forward propagation and backpropagation, and update the model parameters. Return to S1 until the model parameters converge, and the trained large language model is obtained. S5: Input the long sequence to be processed into the trained large language model to obtain the response result.

2. A long sequence processing method based on a large language model according to claim 1, characterized in that: The large language model includes an embedding layer, a position encoding layer, a multi-layer stacked self-attention-feedforward layer and an output layer connected in sequence.

3. The long sequence processing method based on a large language model according to claim 1, characterized in that: Converting a long sequence into a token sequence includes: using a word segmenter to split the long sequence into a concatenation of several words, and assigning an integer index to each word as a token to obtain a token sequence corresponding to the long sequence.

4. The long sequence processing method based on a large language model according to claim 1, characterized in that: The average value of the gradient of each token embedding is used as the threshold, and the formula is expressed as: ； in, is the threshold, is the i-th token, is a token sequence, is the token embedding sequence output by the embedding layer, is the number of tokens in the token sequence, Embed for the i-th token; is the true label, is the output sequence, is the loss function, are model parameters; is the gradient of the i-th token embedding, are the embedding layer parameters, According to the loss function value The gradient of the obtained embedding of the i-th token.

5. The long sequence processing method based on a large language model according to claim 4, characterized in that: If the gradient of token embedding is greater than or equal to the threshold, the token is a key token; if the gradient of token embedding is less than the threshold, the token is an irrelevant token. The formula is expressed as: ； in, is the i-th token, for Token ID, hour, is the key token, hour, For irrelevant tokens; is a token sequence, is the token embedding sequence output by the embedding layer, is the number of tokens in the token sequence, Embed for the i-th token; is the true label, is the output sequence, is the loss function, are model parameters; is the gradient of the i-th token embedding, are the embedding layer parameters, According to the loss function value The gradient of the i-th token embedding is obtained, is the threshold.

6. The long sequence processing method based on a large language model according to claim 5, characterized in that: The irrelevant token embedding is adjusted using the gradient of the irrelevant token embedding, while the key token embedding remains unchanged, and the denoised token embedding sequence is obtained. The formula is expressed as: ； in, is the i-th token, For the i-th token embedding, is the denoised token embedding of the i-th token, for Token ID, is the gradient of the i-th token embedding, is the learning rate, To control the hyperparameters of denoising; The denoised token embedding sequence is represented as .

7. The long sequence processing method based on a large language model according to claim 6, characterized in that: The value is on the order of 1e-5.

8. The long sequence processing method based on a large language model according to claim 1, characterized in that: The loss function of the large language model is cross entropy loss.

9. A long sequence processing system based on a large language model, characterized in that: include: The gradient calculation module is used to take the token sequence corresponding to the long sequence as the training sample input parameter of the large language model, and obtain the output sequence after one forward propagation; Calculate the loss function value between the output sequence and the true label, and derive the loss function value in the back propagation to obtain the gradient of each token embedding in the token embedding sequence output by the embedding layer; A key token identification module is used to take the average value of the gradient of each token embedding as the threshold; If the gradient of the token embedding is greater than or equal to the threshold, then the token is a key token; If the gradient of the token embedding is less than the threshold, the token is an irrelevant token; The denoising training module is used to adjust the irrelevant token embeddings using the gradient of the irrelevant token embeddings, while the key token embeddings remain unchanged, to obtain the denoised token embedding sequence; The parameter update module is used to unfreeze the model parameters, embed the denoised token sequence through a forward propagation and a backward propagation, and update the model parameters; Return to the gradient calculation module until the model parameters converge and obtain the trained large language model; The inference module is used to input the long sequence to be processed into the trained large language model to obtain the response result.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a long sequence processing method based on a large language model as described in any one of claims 1 to 8.

Images