Infinite context large model processing method and device based on lexical element memory

Abstract

The invention discloses an infinite context large model processing method and device based on lexical element memory. The method comprises the following steps: firstly, segmenting an ultra-long text into text paragraphs with fixed lengths; secondly, constructing a local attention module based on a decoder type Transform, calculating text paragraphs, and outputting a local attention result and a related vector; and thirdly, constructing a token lexical element memory pool to maintain each lexical element memory unit, and calculating cross-paragraph global attention and outputting a result in combination with the query vector. And fourthly, fusing local and global attention results to realize an infinite context. And 5, based on the correlation vector, incrementally updating the lexical token memory pool and the normalization item, and providing support for subsequent paragraph processing. And 6, constructing a paragraph memory module, and storing the hidden state of the last lexical element of the paragraph for generating the first lexical element of the next text paragraph. The problems that in the prior art, when super-long texts are processed, calculation and storage expenses are large, and semantic information is prone to being lost are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a method and device for processing an infinite context large model based on word unit memory. BACKGROUND

[0002] With the rapid development of natural language processing (NLP) technology, the Transformer architecture has become the basic model of core applications such as machine translation, text generation, and question and answer systems. The self-attention mechanism of the Transformer architecture has advantages in capturing long-distance dependencies of text, and has promoted the performance breakthrough of various NLP tasks. In recent years, infinite context modeling has become a research hotspot, and researchers have proposed various improvement schemes: Infini-Transformer integrates local mask attention and long linear attention to achieve efficient processing of millions of tokens; EdgeInfinite introduces KV cache compression and dynamic memory reservation mechanism to adapt to the resource constraints of edge devices; Transformer-XL and Compressive Transformer respectively try to extend the context window of the model through segment-level memory and history state compression, which provides diversified solutions for long text processing.

[0003] However, the existing technology still has significant limitations: the attention mechanism of the traditional Transformer has a quadratic growth in computational complexity and storage overhead. When faced with tens of thousands or even millions of tokens of ultra-long text, the memory consumption and computation time increase dramatically, making it difficult to meet the actual application requirements. Although the linear attention method of Infini-Transformer and other methods reduces the complexity, it has the problem of loss of semantic information. The history state compression strategy of Compressive Transformer easily leads to distortion of key information. The external memory retrieval efficiency of Memorizing Transformer is low, and the training complexity is high. At the same time, the memory granularity of most methods stays at the paragraph level, and cannot realize fine-grained cross-paragraph semantic association. In the scenes of long document summarization, cross-chapter reasoning, and ultra-long dialogue modeling, it is difficult to maintain context consistency, which limits the application potential of large models in ultra-long text processing scenes.

[0004] Therefore, there is an urgent need for a method for processing an infinite context large model based on word unit memory to solve the problems of large computational and storage overhead, loss of semantic information, and poor cross-paragraph context consistency when processing ultra-long text in the existing technology. SUMMARY

[0005] To this end, the application provides a word token memory-based infinite context large model processing method and device, which solves the problems of large calculation and storage overhead, easy loss of semantic information, poor cross-paragraph context consistency and the like when a traditional Transformer processes an ultra-long text, and realizes efficient processing of a million tokens, and balances fine-grained semantic association and resource utilization efficiency.

[0006] In order to achieve the above-mentioned purpose, the application provides the following technical scheme: a word token memory-based infinite context large model processing method, characterized by comprising:

[0007] segmenting an ultra-long text to be processed, and dividing the ultra-long text into a plurality of text paragraphs of a fixed length;

[0008] constructing a local attention module based on a decoder type Transformer; performing local attention calculation on the text paragraphs through the local attention module, and outputting a local attention calculation result, a query vector, a key vector and a value vector;

[0009] constructing a token memory pool; maintaining a memory unit for each token in the token memory pool; performing cross-paragraph global attention calculation on the query vector, combining a memory matrix and a normalization term of a corresponding word token in the token memory pool, and outputting a global attention calculation result through an activation function;

[0010] based on the local attention calculation result and the global attention calculation result, performing fusion through a linear layer, and outputting a fusion result to realize infinite context; in a model training process, converting a task into a language modeling task, taking cross entropy as a loss function, calculating an error between a text generated by the fusion result and a real text, updating a training parameter of the model according to the error through back propagation;

[0011] after completing fusion output of a previous text paragraph, updating the memory unit and the normalization term in the token memory pool based on the query vector, the key vector and the value vector through an incremental update strategy, and providing memory support for processing of a subsequent text paragraph;

[0012] constructing a paragraph memory module, saving a last token hidden state of a current paragraph which has completed local attention calculation through the paragraph memory module; and injecting the last token hidden state of the previous text paragraph into a subsequent text paragraph for generation of a first token when performing local attention calculation on the subsequent text paragraph.

[0013] As a preferred scheme of the word token memory-based infinite context large model processing method, a local attention calculation formula is:

[0014]

[0015] wherein, is the local attention calculation result; Q is a query vector; K is a key vector; V is a value vector; and d is a vector dimension.

[0016] As a preferred solution of the unlimited context large model processing method based on the word memory, the global attention calculation formula is:

[0017]

[0018] wherein, is the global attention calculation result; is the memory matrix of the corresponding word; is an ELU+1 activation function; is a normalization term.

[0019] As a preferred solution of the unlimited context large model processing method based on the word memory, in the process of fusing through a linear layer to obtain the fusion result, the calculation formula of the fusion result is:

[0020]

[0021] wherein, O is the fusion result; and are trainable weight matrices;

[0022] In the process of calculating the error of the fusion result and the real token of the text by taking cross entropy as the loss function, the cross entropy loss function expression is:

[0023]

[0024] wherein, is the probability of the model predicting that the nth token is ; is the real probability of the token.

[0025] As a preferred solution of the unlimited context large model processing method based on the word memory, in the process of updating the memory unit and the normalization term in the token memory pool through the incremental update strategy, the update formula of the incremental update strategy is:

[0026]

[0027]

[0028] wherein, is the updated memory unit; is an updated normalization term; N is the number of tokens of the current text paragraph.

[0029] The application also provides a token memory-based unlimited context large model processing device, which adopts the token memory-based unlimited context large model processing method and comprises:

[0030] a text segmentation processing unit configured to perform segmentation processing on the to-be-processed long text and divide the long text into a plurality of text paragraphs with a fixed length;

[0031] a local attention module construction and calculation unit configured to construct a local attention module based on a decoder-type Transformer, perform local attention calculation on the text paragraphs through the local attention module, and output a local attention calculation result, a query vector, a key vector, and a value vector;

[0032] a global attention calculation unit configured to construct a token memory pool, maintain a memory unit for each token in the token memory pool, perform cross-paragraph global attention calculation on the memory matrix and the normalization term of the corresponding token in the token memory pool based on the query vector through an activation function, and output a global attention calculation result;

[0033] an attention result fusion and model training unit configured to fuse the local attention calculation result and the global attention calculation result through a linear layer, output a fusion result, and realize unlimited context; in a model training process, a task is converted into a language modeling task, a cross entropy is used as a loss function, an error between a text generated by the fusion result and a real text is calculated, and a training parameter of a model is updated according to the error through back propagation;

[0034] a token memory pool updating unit configured to, after the fusion output of the previous text paragraph is completed, update the memory unit and the normalization term in the token memory pool based on the query vector, the key vector, and the value vector through an incremental updating strategy, and provide memory support for processing of a subsequent text paragraph;

[0035] a paragraph memory module construction and processing unit configured to construct a paragraph memory module, save the last token hidden state of the current text paragraph after the local attention calculation is completed through the paragraph memory module, and inject the last token hidden state of the previous text paragraph into a subsequent text paragraph for generation of a first token when local attention calculation of the subsequent text paragraph is performed.

[0036] As a preferred solution of the token memory-based unlimited context large model processing device, in the local attention module construction and calculation unit, the local attention calculation formula is:

[0037]

[0038] wherein, is the local attention calculation result; Q is the query vector; K is the key vector; V is the value vector; and d is the vector dimension.

[0039] As a preferred solution of the word piece memory-based infinite context large model processing device, in the global attention calculation unit, the global attention calculation formula is:

[0040]

[0041] wherein, is the global attention calculation result; is the memory matrix corresponding to the word piece; is the ELU+1 activation function; is the normalization term.

[0042] As a preferred solution of the word piece memory-based infinite context large model processing device, in the attention result fusion and model training unit, in the process of fusion through a linear layer to obtain the fusion result, the calculation formula of the fusion result is:

[0043]

[0044] wherein, O is the fusion result; and are trainable weight matrices;

[0045] In the process of calculating the error between the fusion result and the true token of the text by taking cross entropy as the loss function, the cross entropy loss function expression is:

[0046]

[0047] wherein, is the probability that the model predicts the nth token to be ; is the true probability of the token.

[0048] As a preferred solution of the word piece memory-based infinite context large model processing device, in the Token memory pool updating unit, in the process of updating the memory unit and the normalization term in the Token memory pool through the incremental update strategy, the update formula of the incremental update strategy is:

[0049]

[0050]

[0051] wherein, is an updated memory unit; is an updated normalization term; N is the number of tokens of the current text passage.

[0052] The present application has the following advantages:

[0053] First, breaking the context length limit: implementing unlimited context modeling with token-level independent memory units, supporting processing of millions of text or even longer, far exceeding the upper limit of traditional models.

[0054] Second, considering semantic integrity and efficiency: avoiding information loss caused by paragraph-level compression, while reducing memory consumption by more than an order of magnitude through linear incremental update and normalization compression.

[0055] Third, strong cross-passage consistency: combining token memory pool and passage memory module, double guaranteeing cross-passage semantic association, effectively avoiding context breakage.

[0056] Fourth, high deployment flexibility: compatible with dynamic memory compression, quantization optimization and gating mechanism, can stably run on consumer-grade graphics cards and edge devices, with outstanding practical value.

[0057] Fifth, continuous performance optimization: as the input length increases, the token memory accumulation effect is significant, the model perplexity gradually decreases, and the long-distance dependency capture ability continuously improves. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can be derived from the provided drawings without creative labor.

[0059] The structures, proportions, sizes, etc. shown in the specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and do not define the limiting conditions for the implementation of the present application, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that the present application can produce, should still fall within the scope of the technical content disclosed by the present application.

[0060] Figure 1 is a flowchart of the token memory-based unlimited context large model processing method provided in Embodiment 1 of the present application;

[0061] Figure 2A Token memory pool structure schematic diagram in the token memory based unlimited context large model processing method provided in Embodiment 1 of the present application;

[0062] Figure 3 An architecture schematic diagram of the token memory based unlimited context large model processing device provided in Embodiment 2 of the present application. DETAILED DESCRIPTION

[0063] The embodiments of the present application will be described in detail by specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0064] Embodiment 1

[0065] Reference Figure 1 , Embodiment 1 of the present application provides a token memory based unlimited context large model processing method, comprising the following steps:

[0066] S1, segmenting the to-be-processed super-long text to divide it into several text paragraphs of fixed length;

[0067] S2, constructing a local attention module based on a decoder type Transformer; performing local attention calculation on the text paragraphs through the local attention module, and outputting the local attention calculation result, query vector, key vector and value vector;

[0068] S3, constructing a Token memory pool; maintaining each token memory unit through the Token memory pool; based on the query vector, combining the memory matrix and the normalization term of the corresponding token in the Token memory pool, and performing cross-paragraph global attention calculation through the activation function, and outputting the global attention calculation result;

[0069] S4, based on the local attention calculation result and the global attention calculation result, performing fusion through a linear layer to output the fusion result, so as to realize unlimited context; in the model training process, the task is converted into a language modeling task, the cross entropy is taken as the loss function, the error between the text generated by the fusion result and the real text is calculated, the training parameters of the model are updated through back propagation according to the error;

[0070] S5, after completing the fusion output of the previous text paragraph, based on the query vector, the key vector and the value vector, the memory units and the normalization terms in the token word memory pool are updated through an incremental update strategy, providing memory support for the processing of the subsequent text paragraph;

[0071] S6, a paragraph memory module is constructed, the last token hidden state of the current paragraph after completing local attention calculation is saved through the paragraph memory module; when performing local attention calculation of the subsequent text paragraph, the last token hidden state of the previous text paragraph is injected into the subsequent text paragraph for the generation of the first token.

[0072] In this embodiment, in step S1, the to-be-processed super-long text is segmented and processed to divide it into several text paragraphs of fixed length.

[0073] Specifically, first, the fixed length standard of text segmentation is determined, preferably 4096 word units, which balances the local semantic integrity and computational efficiency, and adapts to the basic calculation dimension of the decoder type Transformer; then, the paragraphs are divided in the way of "continuous cutting from beginning to end", ensuring that there is no omission or repetition of text information, and each paragraph forms an independent calculation unit, providing structured input for subsequent local attention calculation and cross-paragraph memory association in stages, avoiding the problems of GPU overflow or calculation stagnation caused by text length exceeding the limit.

[0074] In this embodiment, in step S2, a local attention module is constructed based on a decoder type Transformer; the local attention module is used to perform local attention calculation on the text paragraph, and outputs the local attention calculation result, the query vector, the key vector and the value vector.

[0075] Specifically, the local attention module is built based on the classical structure of the decoder type Transformer, which reuses its mature semantic modeling capability to ensure the accuracy of short-distance dependency capture within the paragraph; then, the single text paragraph divided in step S1 is converted into a vector representation through a word embedding layer and input into the local attention module; the module first generates the query vector, the key vector and the value vector through linear transformation, and then performs calculation according to the attention calculation formula, finally outputs the local attention calculation result reflecting the token association relationship within the paragraph, while retaining the query vector, the key vector and the value vector for subsequent global attention calculation and memory update, forming the logical connection of "current intra-paragraph modeling-subsidiary cross-paragraph support".

[0076] The local attention calculation formula is:

[0077]

[0078] In the formula, is the local attention calculation result; Q is the query vector; K is the key vector; V is the value vector; and d is the vector dimension.

[0079] In this embodiment, in step S3, a Token memory pool is constructed; each token memory unit is maintained through the Token memory pool; based on the query vector, the memory matrix of the corresponding token in the Token memory pool and the normalization term are combined to perform cross-paragraph global attention calculation through an ELU+1 activation function, and a global attention calculation result is output.

[0080] Specifically, the Token memory pool is constructed as shown in the following table: Figure 2 The core is to allocate a dedicated memory unit for each independent token in the vocabulary, and the memory unit is initialized as an empty matrix and dynamically updated with the text processing process. In the current paragraph processing stage, for the query vector of each token, the Token memory pool is first searched: if the token has appeared before, i.e., there is a corresponding memory matrix in the memory pool, the memory matrix is extracted; if it is the first appearance, a zero vector is temporarily used instead, and the complete memory matrix is supplemented through the memory update step later. Then, the normalization term and the ELU+1 activation function are combined, and the global attention calculation formula is used for global attention calculation, and finally the global attention calculation result integrating the cross-paragraph historical semantics is output, realizing the semantic association of “current token-historical appearing scene”.

[0081] The global attention calculation formula is as follows:

[0082]

[0083] In the formula, is the global attention calculation result; is the memory matrix of the corresponding token; is the ELU+1 activation function; is the normalization term.

[0084] In this embodiment, in step S4, based on the local attention calculation result and the global attention calculation result, a linear layer is used for fusion, and a fusion result is output; in the model training process, the task is converted into a language modeling task, cross-entropy is used as the loss function, the error between the fusion result and the real token of the text is calculated; through back propagation, the training parameters of the model are updated according to the error.

[0085] Specifically, the fusion logic is first clarified: local attention calculation results focus on the semantics within the current segment, while global attention calculation results relate to historical semantics across segments. A trainable weight matrix is ​​set through linear layers, and weighted fusion is performed according to the fusion formula, so that the output result simultaneously considers local coherence and global relevance. The final output is the predicted probability distribution of each token, i.e., the fusion result. During the training phase, various tasks such as long document summarization and cross-chapter reasoning are uniformly transformed into language modeling tasks. The cross-entropy loss function is used to calculate the error between the fusion result and the real tokens in the text. The smaller the error, the more accurate the prediction. Then, the error is backpropagated to each layer of the model through the backpropagation algorithm to update all trainable parameters such as the weights of the local attention module, the token memory pool update coefficient, and the weights of the linear layers, continuously optimizing the model's semantic capture and memory association capabilities.

[0086] In the process of fusing through a linear layer to obtain the fusion result, the calculation formula for the fusion result is as follows:

[0087]

[0088] In the formula, O represents the fusion result; and All are trainable weight matrices;

[0089] In calculating the error between the fusion result and the real text token using cross-entropy as the loss function, the expression for the cross-entropy loss function is:

[0090]

[0091] In the formula, The model predicts the nth token as The probability of; This represents the true probability of the token.

[0092] In this embodiment, in step S5, after the fusion output of the previous text segment is completed, based on the query vector, the key vector and the value vector, the memory unit and the normalization term in the Token memory pool are updated through an incremental update strategy to provide updated memory support for the processing of the next text segment.

[0093] Specifically, the query vector, the key vector and the value vector generated in step S2 are used to update the memory unit corresponding to the token in the Token memory pool according to the incremental update formula. This method does not need to reconstruct the entire memory matrix, but only accumulates the semantic information of the current paragraph on the basis of the historical memory, which significantly reduces the computational complexity. At the same time, the normalization term is updated to ensure the numerical stability of the subsequent global attention calculation. The updated Token memory pool and normalization term will directly support the global attention calculation of the next text paragraph, realize the continuous association of “previous paragraph memory-next paragraph semantics”, and ensure the coherence of the infinite context processing.

[0094] In the process of updating the memory unit in the Token memory pool and the normalization term by the incremental update strategy, the update formula of the incremental update strategy is:

[0095]

[0096]

[0097] In the formula, is the updated memory unit; is the updated normalization term; N is the number of tokens of the current text paragraph.

[0098] In step S6 of the embodiment, a paragraph memory module is constructed, and the last token hidden state and the paragraph compression summary vector of the previous text paragraph which have completed the local attention calculation are saved through the paragraph memory module. When the local attention calculation of the next text paragraph is performed, the last token hidden state and the paragraph compression summary vector of the previous text paragraph are injected into the generation process of the first token of the next text paragraph.

[0099] Specifically, the paragraph memory module is constructed, and the storage objects contain two types of key information: one is the hidden state of the last token of the previous text paragraph (reflecting the semantic trend at the end of the paragraph); the other is the paragraph compression summary vector obtained by mean pooling or attention compression (reflecting the core theme of the paragraph). The double information avoids the semantic deviation caused by single memory source. When processing the next text paragraph, the generation stage of the first token is selected as the injection node because it is the starting point of the semantic development of the next paragraph. Injecting the key memory information of the previous paragraph into it can make the next paragraph carry forward the previous semantic information from the initial stage, effectively avoiding the logical break and theme drift problems when crossing paragraphs, and forming the support of “smooth transition of semantic between paragraphs”.

[0100] In a possible embodiment, a verification example is provided as follows:

[0101] In order to verify the effectiveness of the infinite context Transformer model based on Token memory proposed in the application, the existing LLaMA-1.3B model is modified. The original maximum input length of the model is 4096 tokens, which cannot meet the demand of long text processing. Therefore, by introducing the Token memory mechanism, the input length is extended to 32K, 512K or even 1M.

[0102] In this embodiment, a suitable data set is first selected: the PG-19 data set contains books published before 1919, and the text length is long, which is suitable for testing the long context modeling ability of the model and can evaluate the performance of the model in language modeling.

[0103] In terms of experimental settings, a 24-layer Transformer structure is used, each layer contains 16 attention heads, the model dimension is 2048, the paragraph length is set to 4096 tokens, and the input length is 32K, 512K and 1M respectively to verify the performance of the model under different context lengths. The optimizer is Adafactor, the learning rate is tested at 1e-2, 1e-3 and 1e-4, and the cosine decay scheduler is used for dynamic adjustment. The hardware environment is three NVIDIA RTX A6000 graphics cards, which ensures the stability and efficiency of large-scale training.

[0104] During training, cross-entropy is used as the loss function, and all tasks are converted into language modeling tasks, i.e. predicting the probability distribution of the next token. The training algorithm includes text segmentation, local attention calculation, Token memory and paragraph memory retrieval, global attention calculation and fusion, memory update and normalization, and parameter update through backpropagation. The process ensures that the model can perform local calculation at the paragraph level, while achieving global information transmission across paragraphs through Token memory, thereby realizing infinite context modeling.

[0105] All test results are shown in Table 1:

[0106] Table 1 Training and testing results under all conditions

[0107] In the language modeling task, training and testing are performed on the PG-19 data set, and the test results are shown in Table 2:

[0108] Table 2 Comparison of the application with the current most advanced model

[0109] As shown in Table 2, the model of this invention achieves optimal perplexity for different input lengths, specifically 9.36 for 32K input, 8.84 for 512K input, and 8.03 for 1M input, significantly outperforming existing Infini-Transformer models. This indicates that as the input length increases, the token memory gradually fills, and the model performance continuously improves, better capturing long-distance dependencies. These results fully demonstrate the effectiveness and advantages of the token memory mechanism in long text language modeling tasks.

[0110] To further enhance the model's innovation and practicality, three improvements were introduced during implementation. First, a dynamic memory compression mechanism, drawing inspiration from the Infini-attention concept of Infini-Transformer, automatically performs linear compression of the token memory after each segment calculation, ensuring linear growth in memory overhead under millions of inputs and avoiding the quadratic complexity problem of traditional KV caching. Second, a trainable memory gating module, borrowing from the design of EdgeInfinite, adds a gating function G to the token memory pool to determine whether to retain, compress, or merge the memory of a particular token, thus solving the irreversible replacement problem of traditional KV caching. These improvements are all implemented within the existing training framework, demonstrating innovation without additional experimental verification and significantly improving the model's practicality and creativity.

[0111] In terms of inference optimization, by combining LMDeploy's W4A16 quantization technology, the memory storage precision is reduced from FP16 to INT4, reducing memory storage overhead by more than 60% with almost no performance loss. These optimizations enable the model of this invention to handle millions of contexts on consumer-grade graphics cards (such as RTX 4090), greatly enhancing the practical deployment value of the model.

[0112] In summary, the specific embodiment of the present application realizes a truly unlimited context Transformer model by introducing the Token memory mechanism in the Sheared-LLaMA-1.3B model and combining dynamic memory compression, trainable gating, and hybrid paragraph memory. This model has achieved excellent performance in language modeling and summarization tasks, significantly outperforming existing models, fully demonstrating the effectiveness and innovation of the Token memory mechanism. In addition, the present application also demonstrates the potential in reasoning optimization and multi-modal expansion, providing a complete solution for industrial applications. Through these structural improvements, the present application not only breaks through the context limitations of traditional Transformers, but also provides new possibilities for large-scale language models in long text processing, cross-chapter reasoning, and continuous dialogue scenarios, with important theoretical significance and application value.

[0113] The application scenarios of the present application are as follows:

[0114] In the long document summarization scenario, the present application captures the core logic of a million token text by word-level memory and cross-paragraph semantic association, generating a complete and topic-oriented summary.

[0115] In the cross-chapter reasoning scenario, the present application realizes the logical association and reasoning of cross-chapter content such as books and papers through fine-grained memory tracing of the Token memory pool, efficiently solving cross-paragraph related problems.

[0116] In the ultra-long dialogue modeling scenario, the present application retains the historical semantic of the dialogue by continuously updating the word memory and paragraph memory, avoiding context break and information forgetting in long dialogue.

[0117] In the edge device long text processing scenario, the present application reduces the memory overhead by linear incremental update and quantization optimization, enabling mobile devices and embedded devices to efficiently process ultra-long text tasks.

[0118] In the legal / medical document analysis scenario, the present application accurately extracts key information from ultra-long legal texts and medical records with high semantic integrity and cross-paragraph consistency, supporting in-depth analysis in professional scenarios.

[0119] In the code generation and audit scenario, the present application can handle a million lines of code text, capture code syntax and logical dependencies through word memory, and assist in realizing cross-file code generation, vulnerability tracing, and logic verification.

[0120] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of the embodiments can also be applied to a distributed scenario, and be completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present disclosure, and the multiple devices can interact with each other to complete the method.

[0121] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order described above and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0122] Embodiment 2

[0123] Referring to Figure 3 Embodiment 2 of the present disclosure also provides a token memory-based unlimited context large model processing device, comprising:

[0124] A text segmentation processing unit 100 is configured to perform segmentation processing on a to-be-processed super-long text, and divide the super-long text into a plurality of text paragraphs with a fixed length.

[0125] A local attention module construction and calculation unit 200 is configured to construct a local attention module based on a decoder-type Transformer; perform local attention calculation on the text paragraphs through the local attention module, and output a local attention calculation result, a query vector, a key vector, and a value vector.

[0126] A global attention calculation unit 300 is configured to construct a Token memory pool; maintain a memory unit for each token in the Token memory pool; perform cross-paragraph global attention calculation through an activation function based on the query vector, in combination with a memory matrix and a normalization term of a corresponding token in the Token memory pool, and output a global attention calculation result.

[0127] An attention result fusion and model training unit 400 is configured to fuse the local attention calculation result and the global attention calculation result through a linear layer, output a fusion result, and realize unlimited context; in a model training process, convert a task into a language modeling task, take cross entropy as a loss function, calculate an error between a text generated by the fusion result and a real text, update training parameters of a model through back propagation according to the error, and the like.

[0128] The token memory pool updating unit 500 is configured to, after completing fusion output of a previous text paragraph, update the memory unit and the normalization term in the token memory pool based on the query vector, the key vector, and the value vector through an incremental updating strategy, so as to provide memory support for processing of a subsequent text paragraph.

[0129] The paragraph memory module construction and processing unit 600 is configured to construct a paragraph memory module, save the last token hidden state of the current paragraph after completing local attention calculation through the paragraph memory module, and inject the last token hidden state of the previous text paragraph into the subsequent text paragraph for generation of a first token when performing local attention calculation of the subsequent text paragraph.

[0130] In this embodiment, the local attention calculation formula in the local attention module construction and calculation unit 200 is as follows:

[0131]

[0132] In the formula, Attention is a local attention calculation result, Q is a query vector, K is a key vector, V is a value vector, and d is a vector dimension.

[0133] In this embodiment, the global attention calculation formula in the global attention calculation unit 003 is as follows:

[0134]

[0135] In the formula, Attention is a global attention calculation result. is a memory matrix of a corresponding token, is an ELU+1 activation function, is a normalization term.

[0136] In this embodiment, in the process of fusion through a linear layer to obtain the fusion result in the attention result fusion and model training unit 400, the calculation formula of the fusion result is as follows:

[0137]

[0138] In the formula, O is a fusion result. and are trainable weight matrices.

[0139] In the process of calculating the error between the fusion result and a real token of the text by taking cross entropy as a loss function, the expression of the cross entropy loss function is as follows:

[0140] ​​

[0141] wherein, is the probability that the n-th token is predicted by the model; is the true probability of the token.

[0142] In the Token memory pool updating unit 500, in the process of updating the memory unit and the normalization term in the Token memory pool by the incremental updating strategy, the updating formula of the incremental updating strategy is:

[0143]

[0144]

[0145] wherein, is the updated memory unit; is the updated normalization term; N is the number of tokens of the current text passage.

[0146] It should be noted that the information interaction and execution process between the modules of the system described above are based on the same concept as the method embodiment in Embodiment 1 of the present application, and the technical effects brought by them are the same as those of the method embodiment of the present application. For specific content, please refer to the description of the method embodiment in the foregoing method embodiment of the present application. Here, it will not be repeated.

[0147] Embodiment 3

[0148] Embodiment 3 of the present application provides a non-transitory computer readable storage medium, the computer readable storage medium stores a program code of a token memory based infinite context large model processing method, the program code includes instructions for executing the token memory based infinite context large model processing method of embodiment 1 or any possible implementation manner thereof.

[0149] The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (Solid State Disk, SSD)) and the like.

[0150] Embodiment 4

[0151] Embodiment 4 of the present application provides an electronic device, comprising a memory and a processor.

[0152] ​The processor and the memory complete mutual communication through a bus; the memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the token memory-based infinite context large model processing method of embodiment 1 or any possible implementation manner thereof.

[0153] Specifically, the processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which realizes by reading software codes stored in a memory. The memory can be integrated in the processor or exist independently outside the processor.

[0154] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable systems. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode.

[0155] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computing system, which can be concentrated on a single computing system or distributed on a network composed of multiple computing systems, and optionally, they can be realized by program codes executable by a computing system, so that they can be stored in a storage system and executed by a computing system, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0156] Although the present application has been described in detail with general description and specific embodiments above, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present application. Therefore, these modifications or improvements made on the basis of not deviating from the spirit of the present application, all belong to the scope of protection claimed by the present application.

Claims

1. A method for processing large-scale infinite context models based on lexical memory, characterized in that, include: The excessively long text to be processed is segmented into several text paragraphs of fixed length; A local attention module is constructed based on a decoder-type Transformer; the local attention module is used to perform local attention calculations on the text paragraph, and outputs the local attention calculation results, query vector, key vector, and value vector; Build a token memory pool; The memory pool maintains the memory unit for each token. Based on the query vector, combined with the memory matrix and normalization term of the corresponding word in the Token memory pool, cross-segment global attention calculation is performed through the activation function, and the global attention calculation result is output. Based on the local attention calculation results and the global attention calculation results, a linear layer is used to fuse the data and output the fused result to achieve infinite context. During model training, the task is transformed into a language modeling task. Cross-entropy is used as the loss function to calculate the error between the text generated by the fused result and the real text. Through backpropagation, the training parameters of the model are updated according to the error. After the previous text segment is fused and output, based on the query vector, the key vector, and the value vector, the memory units and the normalization terms in the token memory pool are updated using an incremental update strategy to provide memory support for the processing of subsequent text segments. A paragraph memory module is constructed to save the hidden state of the last token of the current paragraph that has completed local attention calculation. When performing local attention calculation for the next text paragraph, the hidden state of the last token of the previous text paragraph is injected into the next text paragraph for the generation of the first token.

2. The method for processing an infinite contextual large model based on lexical memory according to claim 1, characterized in that, The formula for calculating local attention is: ； In the formula, The result of local attention calculation; Q is the query vector; K is the key vector; V is the value vector; d is the vector dimension.

3. The method for processing an infinite contextual large model based on lexical memory according to claim 2, characterized in that, The formula for calculating global attention is: ； In the formula, This is the result of global attention calculation; This is the memory matrix for the corresponding word elements; It is the ELU+1 activation function; This is the normalization term.

4. The method for processing an infinite contextual large model based on lexical memory according to claim 3, characterized in that, In the process of fusing through a linear layer to obtain the fusion result, the calculation formula for the fusion result is as follows: ； In the formula, O represents the fusion result; and All are trainable weight matrices; In calculating the error between the fusion result and the real text token using cross-entropy as the loss function, the expression for the cross-entropy loss function is: ； In the formula, The model predicts the nth token as The probability of; This represents the true probability of the token.

5. The method for processing an infinite contextual large model based on lexical memory according to claim 4, characterized in that, During the process of updating the memory units and the normalization term in the Token memory pool using the incremental update strategy, the update formula of the incremental update strategy is: ； ； In the formula, For the updated memory units; The updated normalization term; N is the number of tokens in the current text paragraph.

6. An infinite contextual large model processing device based on lexical memory, employing the infinite contextual large model processing method based on lexical memory as described in any one of claims 1-5, characterized in that, include: The text segmentation processing unit is used to segment the extremely long text to be processed into several text paragraphs of fixed length. The local attention module construction and computation unit is used to construct a local attention module based on a decoder-type Transformer; perform local attention computation on the text paragraph through the local attention module, and output the local attention computation result, query vector, key vector and value vector; A global attention computation unit is used to construct the token memory pool; The memory pool maintains the memory unit for each token. Based on the query vector, combined with the memory matrix and normalization term of the corresponding word in the Token memory pool, cross-segment global attention calculation is performed through the activation function, and the global attention calculation result is output. The attention result fusion and model training unit is used to fuse the local attention calculation results and the global attention calculation results through a linear layer, and output the fusion result to achieve infinite context. During model training, the task is transformed into a language modeling task. Using cross-entropy as the loss function, the error between the text generated by the fusion result and the real text is calculated. Through backpropagation, the training parameters of the model are updated according to the error. The Token memory pool update unit is used to update the memory units and normalization terms in the token lexical memory pool based on the query vector, the key vector and the value vector after the previous text segment is fused and output, through an incremental update strategy, so as to provide memory support for the processing of subsequent text segments. The paragraph memory module construction and processing unit is used to construct the paragraph memory module, which saves the hidden state of the last token of the current paragraph after completing the local attention calculation; when performing local attention calculation for the next text paragraph, the hidden state of the last token of the previous text paragraph is injected into the next text paragraph for the generation of the first token.

7. The device for processing an infinite contextual large model based on lexical memory according to claim 6, characterized in that, In the local attention module construction and calculation unit, the local attention calculation formula is as follows: ； In the formula, The result of local attention calculation; Q is the query vector; K is the key vector; V is the value vector; d is the vector dimension.

8. The device for processing an infinite contextual large model based on lexical memory according to claim 7, characterized in that, In the global attention calculation unit, the global attention calculation formula is: ； In the formula, This is the result of global attention calculation; This is the memory matrix for the corresponding word elements; It is the ELU+1 activation function; This is the normalization term.

9. The device for processing an infinite contextual large model based on lexical memory according to claim 8, characterized in that, In the attention result fusion and model training unit, during the process of fusing through a linear layer to obtain the fusion result, the calculation formula for the fusion result is as follows: ； In the formula, O represents the fusion result; and All are trainable weight matrices; In calculating the error between the fusion result and the real text token using cross-entropy as the loss function, the expression for the cross-entropy loss function is: ； In the formula, The model predicts the nth token as The probability of; This represents the true probability of the token.

10. The device for processing an infinite contextual large model based on lexical memory according to claim 9, characterized in that, In the Token memory pool update unit, during the process of updating the memory units and the normalization terms in the Token memory pool using the incremental update strategy, the update formula of the incremental update strategy is: ； ； In the formula, For the updated memory units; The updated normalization term; N is the number of tokens in the current text paragraph.

Images