Streaming processing system and method for large language model

Abstract

The invention relates to a streaming processing system and method of a large language model, in an off-line environment, a large language model framework is called through a large language model module so as to operate according to a batch processing mode, and in an on-line environment, streaming model parameters are used as model parameters of the large language model framework, so that the large language model framework operates according to the batch processing mode. The big language model architecture is called to realize streaming reasoning through a token decoupling cache mechanism, and parameter optimization is carried out on the big language model architecture by adopting a streaming attention mask mechanism through a set model parameter optimization module, so that the attention range of the big language model architecture is dynamically adjusted; a large language model architecture only pays attention to current available input and context to obtain streaming model parameters, it is ensured that a large language model module meets the expandability requirement between a batch processing mode and a streaming mode, and the calculation cost is low.

Description

Technical Field

[0001] The present invention relates to the technical field of computer models, and in particular, to a streaming processing system and method for large language models. Background Art

[0002] Large Language Models (LLMs) mainly adopt a batch-processing mode, that is, the complete input data is processed at once in an offline environment to generate corresponding outputs. However, with the development of application scenarios such as real-time interaction, continuous speech recognition, and intelligent customer service, the demand for the streaming-processing ability of large language models is increasing day by day. In the streaming-processing mode, data is input in the form of a continuous stream, and the large language model needs to process and generate outputs in real time, which puts forward new requirements for existing large language models.

[0003] In the prior art, for the streaming processing of large language models, the main methods adopted are batch-streaming methods, interleaved-streaming methods, and constructing a new architecture optimized for streaming tasks.

[0004] In the batch-streaming method, every time new input data arrives, all the input data and the previously generated outputs are re-encoded to ensure the large language model's understanding of the complete context. This method can maintain the compatibility between streaming processing and the existing batch-processing architecture, but its computational cost is extremely high. Especially in long-text or long-time interaction tasks, the computational overhead increases exponentially with the increase in the generated content, making it difficult to meet the requirements of efficient streaming processing.

[0005] The interleaved-streaming method directly encodes the streaming data in the order of arrival without reprocessing historical information, thereby reducing the computational cost and improving the inference efficiency. However, since the large language model adopts a batch-processing mode in the pre-training stage, this mismatch in the attention mechanism may lead to incomplete context encoding, affecting the generation quality of the model. Especially in tasks involving long-distance dependencies, the problem of information loss is particularly serious.

[0006] Some prior arts have also explored new architectures specifically optimized for streaming tasks, that is, constructing a new architecture optimized for streaming tasks. These methods can improve the adaptability of streaming processing to a certain extent, but often require additional training costs. Some new architectures are difficult to be compatible with pre-trained large language models, lack scalability, and are difficult to make full use of existing pre-trained large language models, resulting in high migration and deployment costs and being difficult to be popularized in practical applications. Summary of the Invention

[0007] The technical problem to be solved by the present invention is how to overcome the technical defects of high computational cost, mismatch with pre-trained large language models, and difficulty in expansion existing in the streaming processing of large language models. To overcome the above defects of the prior art, the present invention provides a streaming processing system and method for large language models, specifically including a streaming processing system for large language models and a streaming processing method for large language models.

[0008] A streaming processing system for large language models provided by the present invention includes: A large language model module, which has a built-in large language model architecture and is set to call the large language model architecture in an offline environment to run in a batch processing mode, while in an online environment, it uses streaming model parameters as the model parameters of the large language model architecture and calls the large language model architecture to achieve streaming inference through a token decoupling cache mechanism; A model parameter optimization module, which is electrically connected to the large language model module and is set to optimize the parameters of the large language model architecture by using a streaming attention mask mechanism to obtain the streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process, so that the large language model architecture only focuses on the currently available input and context.

[0009] In the streaming processing system for large language models disclosed by the present invention, by setting a large language model module and a model parameter optimization module, the large language model architecture can be called by the large language model module in an offline environment to run in a batch processing mode, while in an online environment, it uses streaming model parameters as the model parameters of the large language model architecture and calls the large language model architecture to achieve streaming inference through a token decoupling cache mechanism. Moreover, the model parameter optimization module can optimize the parameters of the large language model architecture by using a streaming attention mask mechanism to dynamically adjust the attention range of the large language model architecture, so that the large language model architecture only focuses on the currently available input and context and obtains streaming model parameters. Therefore, the present invention uses the large language model architecture as the underlying architecture, overcomes the defect of mismatch between streaming processing and pre-trained large language models, and the large language model module is seamlessly compatible with the batch processing mode according to different operating modes in the online and offline states, and can thus be universal in different tasks (such as machine translation, speech recognition, real-time dialogue, etc.), ensuring that the large language model module can meet the scalability requirements between the batch processing mode and the streaming mode, and has a low computational cost.

[0010] In a possible implementation manner, the way for the large language model module to achieve streaming inference through a token decoupling cache mechanism is: Set an initial input token cache pool and an output token cache pool, and then alternately execute reading tokens and generating tokens; When reading a token, perform attention calculation on the token to be read and the input token cache pool to obtain a first attention calculation result, and store the first attention calculation result in the input token cache pool; When generating a token, first take out the latest token from the output token cache pool, and then perform attention calculation on this token, the input token cache pool, and the output token cache pool to obtain a second attention calculation result, and store the second attention calculation result in the output token cache pool; This solution designs a dual-channel token cache pool to store input and output tokens respectively. Through one-way cache pool information transfer, it avoids the coupling of output tokens with output tokens and realizes the streaming inference of the large language model.

[0011] In a possible implementation manner, the model parameter optimization module is set to perform the following steps: A1: Construct an attention mask matrix; A2: Retrieve several frames of streaming input data and output data of the large language model framework, and arrange each frame of the streaming input data and output data in the data format of batch processing to form corresponding streaming input-output data; A3: Perform grouped position encoding on each piece of streaming input-output data to obtain respective encoded data, integrate them into an encoded data matrix, and obtain the query matrix, key matrix, and value matrix of the encoded data matrix by means of mapping through the linear layer of the transformer architecture; A4: Substitute the encoded data matrix, the attention mask matrix, the query matrix, the key matrix, and the value matrix into the attention functional for evaluating the attention degree of the large language model architecture to the current available input and context to obtain the attention score matrix of the large language model architecture; A5: Use the attention score matrix as the attention score matrix for the forward propagation of the large language model architecture, so that the large language model architecture performs forward propagation, and update the parameters of the large language model architecture through the gradient backpropagation method to obtain the streaming model parameters; This solution can solve the problem of position encoding mismatch in streaming inference by performing grouped position encoding. By performing independent encoding of the position information of the input and output, unnecessary recoding calculations in the streaming inference process can be reduced, and finally the computational overhead can be reduced.

[0012] In a possible implementation manner, the attention mask matrix is a square matrix in the real number field with the following structure: The order of the square matrix is equal to the length of the encoded data; For the value of the element at the p-th row and q-th column of the square matrix, If p < q, the value is 1; If p ≥ q, there are the following cases: When p < m and q < m or when p > m and q > m, the value is 0; When p > m and q < m, if p - m < q is also satisfied, the value is 0; When p > m and q < m, if p - m ≥ q is also satisfied, the value is 1; where m represents the length of a frame of streaming input data of the large language model framework; This solution can ensure that the large language model only focuses on the currently available input and context by constructing an attention mask matrix, and also helps to train the large language model in combination with the large language model framework and grouped positional encoding.

[0013] In a possible implementation manner, in step A3, the encoded data obtained by performing grouped positional encoding on a certain streaming input-output data is expressed as follows: , In the formula, represents the encoded data obtained by performing grouped positional encoding on the streaming input data of the streaming input-output data; represents the encoded data obtained by performing grouped positional encoding on the output data of the streaming input-output data; This solution can ensure the efficient operation of grouped positional encoding for streaming input-output data, so as to improve the computational efficiency of streaming inference through grouped positional encoding.

[0014] In a possible implementation manner, step A4 includes the following steps: A41: Right-multiply the query matrix by the diagonal rotation matrix to obtain a first encoded matrix, and right-multiply the key matrix by the diagonal rotation matrix to obtain a second encoded matrix; A42: Substitute the attention mask matrix, the first encoded matrix, the second encoded matrix, and the value matrix into the attention function formula to obtain the attention score matrix of the large language model architecture; This solution helps to achieve dynamic adjustment of the attention range, enabling the model to only focus on the currently available input and context, and solving the problem of mismatch between streaming processing and the attention of pre-trained large language models.

[0015] In a possible implementation manner, the diagonal rotation matrix is: , , In the formula, represents the diagonal rotation matrix; represents a diagonal block matrix; represents the rotation angle; represents the algorithm dimension of the large language model architecture.

[0016] The above calculation formula is based on a second-order rotation matrix to form a block rotation matrix, achieving the efficient acquisition of the first coding matrix and the second coding matrix.

[0017] In a possible implementation manner, the calculation formula of the attention function is as follows: , In the formula, represents the attention score matrix; represents the query matrix; represents the key matrix; represents the attention mask matrix; represents the value matrix; represents the transpose of the matrix; For the matrix , the definition formula of the function is: .

[0018] Through the above calculation formula, not only can the attention score matrix be calculated to ensure the orderly and efficient progress of the large language model training, but also the calculation complexity is small and it is easy to implement.

[0019] Another technical solution of the present invention is to provide a streaming processing method for a large language model, including the following steps: S1: Optimize the parameters of the large language model architecture by using a streaming attention mask mechanism through a model parameter optimization module to obtain streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process, so that the large language model architecture only focuses on the current available input and context; S2: Use the streaming model parameters as the model parameters of the large language model architecture through the large language model module, and call the large language model architecture to achieve streaming inference through a token decoupling cache mechanism.

[0020] The method disclosed in this application adopts a large language model architecture as the underlying architecture. First, through the model parameter optimization module, a streaming attention mask mechanism is used to optimize the parameters of the large language model architecture to obtain streaming model parameters. Subsequently, through the large language model module, the streaming model parameters are used as the model parameters of the large language model architecture, and the large language model architecture is called to achieve streaming inference through the token decoupling cache mechanism, overcoming the defect of the mismatch between streaming processing and pre-trained large language models, achieving seamless compatibility with the batch processing mode, and thus being applicable to different tasks (such as machine translation, speech recognition, real-time dialogue, etc.), ensuring the scalability of the large language model module between the batch processing mode and the streaming mode, and having a relatively low computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 FIG. is a schematic structural diagram of a streaming processing system for a large language model disclosed in an embodiment of the present application; Figure 2 FIG. is a flowchart of the operation of the model parameter optimization module disclosed in an embodiment of the present application; Figure 3 FIG. is a schematic diagram of the operation of the token decoupling cache mechanism disclosed in an embodiment of the present application; Figure 4 FIG. is a translation performance image of English-French execution disclosed in an embodiment of the present application; Figure 5 FIG. is a translation performance image of English-German disclosed in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] First of all, those skilled in the art should understand that these embodiments are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments according to needs to adapt to specific application scenarios.

[0023] In the embodiments of the present application, unless otherwise clearly specified and limited, the electrical connection between the first feature and the second feature means that there is an electrical signal transmission between the first feature and the second feature, that is, there is an electrical relationship, and the way to achieve the electrical signal transmission can be wire electrical connection, radio connection, electrical connection of an electromagnetic medium (such as a semiconductor), communication implemented by a channel, etc.

[0024] In the embodiments of the present application, unless otherwise clearly specified and limited, the first feature being "on" or "under" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the horizontal height of the first feature is higher than that of the second feature. The first feature being "under", "beneath" and "underneath" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the horizontal height of the first feature is less than that of the second feature.

[0025] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] See Figures 1 to 3 , the embodiments of the present application disclose a streaming processing system for a large language model. The structural schematic diagram of the streaming processing system is as shown in Figure 1 . The streaming processing system includes a large language model module and a model parameter optimization module. Among them, the large language model module has a built-in large language model architecture, and the model parameter optimization module is electrically connected to the large language model module.

[0027] In this streaming processing system, the large language model module is configured to call the large language model architecture in an offline environment to run in a batch processing mode, and in an online environment, use streaming model parameters as the model parameters of the large language model architecture, and call the large language model architecture to achieve streaming inference through a token decoupling cache mechanism. In this embodiment, being in an online environment means that the large language model module has received tasks that need to be processed in real time or near real time, and is in a working environment that requires fast response and processes a large amount of data. While being in an offline environment means that the large language model module has received tasks that can be batch processed without real-time data input, such as data mining and machine learning model training, and is in a working environment where the processing time is not restricted and can run in the background.

[0028] Figure 3 For the schematic diagram of the operation of the token decoupling cache mechanism, see Figure 3 . In this embodiment, the way for the large language model module to achieve streaming inference through the token decoupling cache mechanism is as follows: set an initial input token cache pool and an output token cache pool, and then alternately execute token reading and token generation. When reading tokens, perform attention calculation on the tokens to be read and the input token cache pool to obtain a first attention calculation result, and store the first attention calculation result in the input token cache pool. As shown in Figure 3As shown, the output token cache pool cannot participate in the attention calculation of the tokens to be read. When generating tokens, first take out the latest token from the output token cache pool, and then perform attention calculation on this token with the input token cache pool and the output token cache pool to obtain the second attention calculation result, and store the second attention calculation result in the output token cache pool.

[0029] See Figure 2 , in this streaming processing system, the model parameter optimization module is set to optimize the parameters of the large language model architecture using a streaming attention mask mechanism to obtain streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process, so that the large language model architecture only focuses on the currently available input and context.

[0030] See Figure 2 , in this embodiment, the model parameter optimization module is set to perform the following steps: A1: Construct an attention mask matrix; specifically in this embodiment, the constructed attention mask matrix is a square matrix in the real number field with the following structure: The order of the square matrix is equal to the length of the encoded data; For the value of the element at the p-th row and q-th column of the square matrix, if p < q, the value is 1; if p ≥ q, there are the following cases: when p < m and q < m or when p > m and q > m, the value is 0; when p > m and q < m, if p - m < q is also satisfied, the value is 0; when p > m and q < m, if p - m ≥ q is also satisfied, the value is 1; where m represents the length of a frame of streaming input data of the large language model framework, that is, the length of the streaming input data in a frame of streaming input and output data.

[0031] A2: Retrieve several frames of streaming input data and output data of the large language model framework, arrange each frame of streaming input data and output data in the data format of batch processing to form corresponding streaming input and output data.

[0032] A3: Perform grouped position encoding on each streaming input and output data to obtain their respective encoded data, integrate them into an encoded data matrix, and obtain the query matrix (Query), key matrix (Key), and value matrix (Value) of the encoded data matrix by mapping the encoded data matrix through the linear layer mapping of the Transformer architecture.

[0033] Specifically in step A3, the encoded data obtained by performing grouped position encoding on a certain streaming input and output data is expressed as follows: , wherein, represents the encoded data of the streaming input data of the large language model framework; represents the encoded data of the output data of the large language model framework.

[0034] Taking the encoded data of each streaming input-output data as the rows of a matrix, and integrating them into a matrix in chronological order to obtain an encoded data matrix. Subsequently, a query matrix, a key matrix, and a value matrix of the encoded data matrix are obtained by mapping the encoded data matrix through the linear layer mapping of the transformer architecture.

[0035] A4: Substitute the encoded data, the attention mask matrix, the query matrix, the key matrix, and the value matrix into the attention function formula for evaluating the attention degree of the large language model architecture to the current available input and context, to obtain the attention score matrix of the large language model architecture.

[0036] Specifically, in this embodiment, step A4 includes the following steps: A41: Right-multiply the query matrix by the diagonal rotation matrix to obtain a first encoded matrix, and right-multiply the key matrix by the diagonal rotation matrix to obtain a second encoded matrix. The diagonal rotation matrix is: , , wherein, represents the diagonal rotation matrix; represents the diagonal block matrix; represents the rotation angle; represents the algorithm dimension of the large language model architecture.

[0037] A42: Substitute the attention mask matrix, the first encoded matrix, the second encoded matrix, and the value matrix into the attention function formula to obtain the attention score matrix of the large language model architecture. The calculation formula of the attention function formula is as follows: , wherein, represents the attention score matrix; represents the query matrix; represents the key matrix; represents the attention mask matrix; represents the value matrix; Represents the transpose of the matrix.

[0038] For the matrix , The definition formula of the function is: , In this embodiment, the attention mask matrix is a square matrix of order is also a square matrix of order. Without loss of generality, let , Furthermore, , Finally, multiply the above formula on the right by the value matrix to obtain the attention score matrix.

[0039] A5: Use the attention score matrix as the attention score matrix for the forward propagation of the large language model architecture, so that the large language model architecture performs forward propagation, and update the parameters of the large language model architecture through the gradient backpropagation method to obtain the streaming model parameters.

[0040] In this embodiment, the method for updating the parameters of the large language model architecture through the gradient backpropagation method is: use the model algorithm of the Adam optimizer to perform gradient backpropagation on the large language model architecture to achieve parameter update, obtain the streaming model parameters, and transmit the streaming model parameters to the large language model module.

[0041] The following will further disclose the usage method of the streaming processing system of the large language model in this embodiment. The method includes the following steps: S1: Optimize the parameters of the large language model architecture through the model parameter optimization module by adopting the streaming attention mask mechanism and obtain the streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process, so that the large language model architecture only focuses on the current available input and context.

[0042] S2: Use the streaming model parameters as the model parameters of the large language model architecture through the large language model module, and call the large language model architecture to implement streaming inference through the token decoupling cache mechanism.

[0043] The following will detail the technical effects of the streaming processing system of the large language model in this embodiment. Refer to Figure 4 and Figure 5, in this embodiment, the streaming processing system is arranged on a device for performing streaming text translation tasks for testing, and four different large language models are selected, namely Gemma2-2B, Phi3-mini, Llama3.1-8B, and Falcon-1B. These large language models are respectively arranged in the large language model module to perform English-French translation and English-German translation. The obtained performance detection result diagrams are as shown in Figure 4 and Figure 5 . In the performance images, the abscissa is the number of words, and the ordinate is the similarity index (BLEU) for evaluating translation quality. Gemma2 (Ours) represents the performance image of arranging the Gemma2-2B model in the large language model module, Phi3 (Ours) represents the performance image of arranging the Phi3-mini model in the large language model module, Llama (Ours) represents the performance image of arranging the Llama3.1-8B model in the large language model module, and Falcon (Ours) represents the performance image of arranging the Falcon-1B model in the large language model module. It can be seen from Figure 4 and Figure 5 that as the number of words increases, the similarity also increases, and the translation quality is significantly improved, further indicating that the performance of the streaming processing system is good.

[0044] Meanwhile, the streaming processing system of the large language model disclosed in this embodiment, by setting the large language model module and the model parameter optimization module, in the offline environment, the large language model architecture is called by the large language model module to run in batch processing mode, while in the online environment, the streaming model parameters are used as the model parameters of the large language model architecture, and the large language model architecture is called to achieve streaming inference through the token decoupling cache mechanism. Moreover, the streaming attention mask mechanism can be adopted by the set model parameter optimization module to optimize the parameters of the large language model architecture, dynamically adjust the attention range of the large language model architecture, so that the large language model architecture only focuses on the current available input and context, and obtains the streaming model parameters. Therefore, the present invention adopts the large language model architecture as the underlying architecture, overcomes the defect of incompatibility with the pre-trained large language model. The large language model module is seamlessly compatible with the batch processing mode by setting two states of online and offline, and thus can be universal in different tasks (such as machine translation, speech recognition, real-time dialogue, etc.), ensuring that the large language model module can meet the scalability requirements between the batch processing mode and the streaming mode, and the computing cost is relatively low.

[0045] In the description of the embodiments of the present application, it should be noted that in the description of the present application, the terms "inner", "outer", and other terms indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for convenience of description and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present application.

[0046] In the description of the present application, the description referring to terms such as "one embodiment", "some embodiments", "in this embodiment", "specific examples", or "some examples" means that the specific features, mechanisms, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0047] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A streaming processing system for a large language model, characterized in that: include: A large language model module, having a built-in large language model architecture, configured to call the large language model architecture in an offline environment to run in batch mode, and to use streaming model parameters as model parameters of the large language model architecture in an online environment, and to call the large language model architecture to implement streaming reasoning through a token decoupled cache mechanism; The model parameter optimization module is electrically connected to the large language model module and is configured to use a streaming attention mask mechanism to perform parameter optimization on the large language model architecture to obtain the streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process so that the large language model architecture only focuses on currently available inputs and contexts.

2. The streaming processing system for a large language model according to claim 1, characterized in that: The large language model module implements streaming reasoning through a token decoupling cache mechanism as follows: Set up the initial input token buffer pool and output token buffer pool, and then perform token reading and token generation alternately; When reading a token, performing attention calculation on the token to be read and the input token buffer pool to obtain a first attention calculation result, and storing the first attention calculation result in the input token buffer pool; When generating a token, first take out the latest token from the output token cache pool, then perform attention calculation on the token with the input token cache pool and the output token cache pool to obtain a second attention calculation result, and store the second attention calculation result in the output token cache pool.

3. The streaming processing system for a large language model according to claim 1 or 2, characterized in that: The model parameter optimization module is configured to perform the following steps: A1: Construct the attention mask matrix; A2: Retrieving several frames of streaming input data and output data of the large language model framework, arranging each frame of the streaming input data and output data in a batch processing data format to form corresponding streaming input and output data; A3: performing group position encoding on each stream input and output data respectively to obtain respective encoded data, and integrating them into an encoded data matrix, and mapping the encoded data matrix through a linear layer mapping of a converter architecture to obtain a query matrix, a key matrix and a value matrix of the encoded data matrix; A4: Substituting the encoded data matrix, the attention mask matrix, the query matrix, the key matrix, and the value matrix into an attention function for evaluating the degree of attention of the large language model architecture to the currently available input and context, to obtain an attention score matrix of the large language model architecture; A5: Use the attention score matrix as the attention score matrix for the forward propagation of the large language model architecture, so that the large language model architecture performs forward propagation, and update the parameters of the large language model architecture through gradient back propagation to obtain the streaming model parameters.

4. The streaming processing system for a large language model according to claim 3, characterized in that: The attention mask matrix is ​​a real field square matrix with the following structure: The order of the square matrix is ​​equal to the length of the encoded data; For the value of the element at the pth row and qth column of the square matrix, If p<q, the value is 1; If p≥q, then there are the following situations: When p<m and q<m or when p>m and q>m, the value is 0; When p>m and q<m, if pm<q is also satisfied, the value is 0; When p>m and q<m, if pm≥q is also satisfied, the value is 1; Wherein, m represents the length of one frame of streaming input data of the large language model framework.

5. The streaming processing system for a large language model according to claim 4, characterized in that: In step A3, the encoded data obtained by performing group position encoding on a certain stream input and output data is expressed as follows: ， In the formula, represents encoded data obtained by grouping position encoding of streaming input data of streaming input output data; Represents encoded data obtained by performing group position encoding on output data of streaming input and output data.

6. The streaming processing system for a large language model according to claim 5, characterized in that: The step A4 comprises the following steps: A41: right-multiply the query matrix by a diagonal rotation matrix to obtain a first encoding matrix, and right-multiply the key matrix by the diagonal rotation matrix to obtain a second encoding matrix; A42: Substitute the attention mask matrix, the first encoding matrix, the second encoding matrix and the value matrix into the attention function to obtain the attention score matrix of the large language model architecture.

7. The streaming processing system for a large language model according to claim 6, characterized in that: The diagonal rotation matrix is: ， ， In the formula, represents the diagonal rotation matrix; represents a diagonal block matrix; Represents the rotation angle; Represents the algorithmic dimension of the large language model architecture.

8. The streaming processing system for a large language model according to claim 7, characterized in that: The calculation formula of the attention function is as follows: ， In the formula, represents the attention score matrix; represents the query matrix; represents the bond matrix; represents the attention mask matrix; represents the value matrix; Represents the transpose of a matrix; For the matrix , The function definition is: 。 9. A streaming processing method for a large language model, characterized in that: A streaming processing system for a large language model according to any one of claims 1 to 8, comprising the following steps: S1: Optimize the parameters of the large language model architecture by using a streaming attention mask mechanism through a model parameter optimization module and obtain streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process, so that the large language model architecture only pays attention to the currently available input and context; S2: Using the streaming model parameters as model parameters of the large language model architecture through the large language model module, and calling the large language model architecture to implement streaming reasoning through a token decoupling cache mechanism.

Images