A streaming processing system and method for large language models

Abstract

The present invention relates to a streaming processing system and method for a large language model. In an offline environment, a large language model architecture is called by a large language model module to operate in batch mode. In an online environment, streaming model parameters are used as model parameters of the large language model architecture, and the large language model architecture is called to implement streaming reasoning through a token decoupling cache mechanism. A model parameter optimization module is also provided to optimize the parameters of the large language model architecture using a streaming attention mask mechanism to dynamically adjust the attention range of the large language model architecture, so that the large language model architecture only focuses on the currently available input and context to obtain streaming model parameters. This ensures that the large language model module meets the scalability requirements between batch mode and streaming mode, and has low computational cost.

Description

Technical Field

[0001] The present invention relates to the field of computer model technology, and in particular to a streaming processing system and method for a large language model. Background Art

[0002] Large language models (LLMs) primarily utilize a batch processing model, processing the entire input data at once in an offline environment before generating the corresponding output. However, with the development of application scenarios such as real-time interaction, continuous speech recognition, and intelligent customer service, the demand for streaming processing capabilities for large language models is growing. In streaming processing, data is input in a continuous stream, and large language models must process and generate output in real time, placing new demands on existing large language models.

[0003] In existing technologies, the main methods for streaming processing of large language models include batch streaming, interleaved streaming, and building a new architecture for stream task optimization.

[0004] Batch streaming methods re-encode all input data and previously generated outputs each time new input data arrives, ensuring that the large language model understands the complete context. This approach maintains compatibility between streaming and existing batch processing architectures, but it is computationally expensive, especially for long text or long-term interactive tasks. The computational overhead increases quadratically with the amount of generated content, making it difficult to meet the requirements of efficient streaming processing.

[0005] Interleaved streaming methods directly encode streaming data in the order it arrives, without reprocessing historical information, thereby reducing computational costs and improving inference efficiency. However, because large language models use batch processing during pre-training, this mismatch in attention mechanisms can lead to incomplete context encoding, affecting model generation quality. Information loss is particularly severe in tasks involving long-range dependencies.

[0006] Some existing technologies have also explored novel architectures specifically optimized for streaming tasks, specifically building novel architectures optimized for streaming tasks. These approaches can improve the adaptability of streaming processing to a certain extent, but they often require additional training costs. Some of these new architectures are incompatible with large pre-trained language models, lack scalability, and have difficulty fully utilizing existing pre-trained large language models. This leads to high migration and deployment costs, making them difficult to promote in practical applications. Summary of the Invention

[0007] The technical problem to be solved by the present invention is how to overcome the technical shortcomings of existing streaming processing of large language models, such as high computational cost, incompatibility with pre-trained large language models, and difficulty in scalability. To overcome these shortcomings of the existing technology, the present invention provides a streaming processing system and method for large language models, specifically comprising a streaming processing system and a streaming processing method for large language models.

[0008] The present invention provides a streaming processing system for a large language model, comprising:

[0009] A large language model module, having a built-in large language model architecture, configured to call the large language model architecture to run in batch mode in an offline environment, and to use streaming model parameters as model parameters of the large language model architecture in an online environment, and to call the large language model architecture to implement streaming reasoning through a token decoupled caching mechanism;

[0010] A model parameter optimization module is electrically connected to the large language model module and is configured to use a streaming attention mask mechanism to optimize the parameters of the large language model architecture to obtain the streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process so that the large language model architecture only focuses on the currently available input and context.

[0011] The disclosed large language model streaming processing system, by providing a large language model module and a model parameter optimization module, can call the large language model architecture in batch mode in an offline environment through the large language model module. In an online environment, the streaming model parameters are used as the model parameters of the large language model architecture, and the large language model architecture is called to implement streaming inference through a token decoupling caching mechanism. Furthermore, the model parameter optimization module can also optimize the parameters of the large language model architecture using a streaming attention masking mechanism to dynamically adjust the attention range of the large language model architecture, so that the large language model architecture focuses only on the currently available input and context, and obtains the streaming model parameters. Therefore, the present invention utilizes the large language model architecture as the underlying architecture, overcoming the drawback of streaming processing that is incompatible with pre-trained large language models. The large language model module is seamlessly compatible with batch processing mode based on its different operating modes in both online and offline states, and can be used universally across different tasks (such as machine translation, speech recognition, real-time conversation, etc.), ensuring that the large language model module can meet the scalability requirements between batch and streaming modes with low computational cost.

[0012] In one possible implementation, the large language model module implements streaming reasoning through a token decoupling cache mechanism as follows:

[0013] Set up the initial input token buffer pool and output token buffer pool, and then alternately execute token reading and token generation;

[0014] When reading a token, performing attention calculation on the token to be read and the input token buffer pool to obtain a first attention calculation result, and storing the first attention calculation result in the input token buffer pool;

[0015] When generating a token, first take out the latest token from the output token cache pool, then perform attention calculation on the token with the input token cache pool and the output token cache pool to obtain a second attention calculation result, and store the second attention calculation result in the output token cache pool;

[0016] This solution designs a dual-channel token buffer pool to store input and output tokens respectively. Through one-way buffer pool information transmission, it avoids the coupling between input tokens and output tokens, and realizes streaming reasoning of large language models.

[0017] In one possible implementation, the model parameter optimization module is configured to perform the following steps:

[0018] A1: Construct attention mask matrix;

[0019] A2: Retrieving multiple frames of streaming input data and output data of the large language model framework, and arranging each frame of the streaming input data and output data in a batch data format to form corresponding streaming input and output data;

[0020] A3: performing group position encoding on each stream input and output data to obtain respective encoded data, and integrating the encoded data into an encoded data matrix, and mapping the encoded data matrix through a linear layer mapping of a transformer architecture to obtain a query matrix, a key matrix, and a value matrix of the encoded data matrix;

[0021] A4: Substituting the encoded data matrix, the attention mask matrix, the query matrix, the key matrix, and the value matrix into an attention function for evaluating the degree of attention of the large language model architecture to the currently available input and context, to obtain an attention score matrix of the large language model architecture;

[0022] A5: Using the attention score matrix as the attention score matrix for the forward propagation of the large language model architecture, so that the large language model architecture performs forward propagation, and updating the parameters of the large language model architecture through gradient backpropagation to obtain the streaming model parameters;

[0023] This solution can solve the problem of positional encoding mismatch in streaming reasoning by performing grouped position encoding. By independently encoding the input and output position information, it can reduce unnecessary re-encoding calculations during the streaming reasoning process, ultimately reducing computational overhead.

[0024] In one possible implementation, the attention mask matrix is ​​a real field square matrix with the following structure:

[0025] The order of the square matrix is ​​equal to the length of the encoded data;

[0026] For the value of the element at the pth row and qth column of the square matrix,

[0027] If p < q, the value is 1;

[0028] If p≥q, then there are the following situations:

[0029] When p<m and q<m or when p>m and q>m, the value is 0;

[0030] When p>m and q<m, if pm<q is also satisfied, the value is 0;

[0031] When p>m and q<m, if pm≥q is also satisfied, the value is 1;

[0032] Wherein, m represents the length of one frame of streaming input data of the large language model framework;

[0033] This scheme ensures that the large language model only focuses on the currently available input and context by constructing an attention mask matrix. It also helps to train the large language model by combining the large language model framework and group position encoding.

[0034] In a possible implementation, in step A3, the encoded data obtained by performing group position encoding on a certain stream input / output data is expressed as follows:

[0035] ,

[0036] Where,

[0037] represents encoded data obtained by performing group position encoding on streaming input data of streaming input and output data;

[0038] represents encoded data obtained by performing group position encoding on output data of the streaming input and output data;

[0039] This solution can ensure the high efficiency of group position encoding of streaming input and output data, thereby improving the computational efficiency of streaming reasoning through group position encoding.

[0040] In a possible implementation, step A4 includes the following steps:

[0041] A41: right-multiply the query matrix by a diagonal rotation matrix to obtain a first encoding matrix, and right-multiply the key matrix by the diagonal rotation matrix to obtain a second encoding matrix;

[0042] A42: Substitute the attention mask matrix, the first encoding matrix, the second encoding matrix, and the value matrix into the attention function to obtain an attention score matrix of the large language model architecture;

[0043] This solution helps to dynamically adjust the attention range so that the model only focuses on the currently available input and context, solving the problem of attention mismatch between streaming processing and pre-trained large language models.

[0044] In one possible implementation, the diagonal rotation matrix is:

[0045] ,

[0046] ,

[0047] Where,

[0048] represents the diagonal rotation matrix;

[0049] represents a diagonal block matrix;

[0050] Represents the rotation angle;

[0051] Represents the algorithmic dimension of the large language model architecture.

[0052] The above calculation formula is based on the second-order rotation matrix to form a block rotation matrix, thereby achieving high-efficiency acquisition of the first encoding matrix and the second encoding matrix.

[0053] In one possible implementation, the attention function is calculated as follows:

[0054] ,

[0055] Where,

[0056] represents the attention score matrix;

[0057] represents the query matrix;

[0058] represents the bond matrix;

[0059] represents the attention mask matrix;

[0060] represents the value matrix;

[0061] Represents the transpose of a matrix;

[0062] For the matrix , The function definition is:

[0063] .

[0064] The above formula can not only calculate the attention score matrix to ensure the orderly and efficient training of large language models, but also has low computational complexity and is easy to implement.

[0065] Another technical solution of the present invention is to provide a streaming processing method for a large language model, comprising the following steps:

[0066] S1: Optimize the parameters of the large language model architecture using a streaming attention mask mechanism through the model parameter optimization module to obtain streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process so that the large language model architecture only focuses on the currently available input and context;

[0067] S2: Using the streaming model parameters as model parameters of the large language model architecture through the large language model module, and calling the large language model architecture to implement streaming reasoning through a token decoupling cache mechanism.

[0068] The method disclosed in this application uses a large language model architecture as the underlying architecture. First, the model parameter optimization module uses a streaming attention mask mechanism to optimize the parameters of the large language model architecture to obtain streaming model parameters. The streaming model parameters are then used as the model parameters of the large language model architecture by the large language model module, and the large language model architecture is called to implement streaming reasoning through a token decoupling cache mechanism. This overcomes the defect that streaming processing does not match the pre-trained large language model, achieving seamless compatibility with batch processing mode. It can then be used in different tasks (such as machine translation, speech recognition, real-time conversation, etc.), ensuring the scalability of the large language model module between batch mode and streaming mode, and at a low computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of the structure of a streaming processing system for a large language model disclosed in an embodiment of the present application;

[0070] Figure 2 This is a flow chart of the operation of the model parameter optimization module disclosed in the embodiments of this application;

[0071] Figure 3 This is a schematic diagram of the operation of the token decoupling cache mechanism disclosed in the embodiments of this application;

[0072] Figure 4 This is an image of the English-French translation performance disclosed in the embodiments of this application;

[0073] Figure 5 This is an image of the English-German translation performance disclosed in the examples of this application. DETAILED DESCRIPTION

[0074] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of the present application and are not intended to limit the scope of protection of the embodiments of the present application. Those skilled in the art may adjust them as needed to suit specific application scenarios.

[0075] In the embodiments of the present application, unless otherwise clearly specified and limited, the electrical connection between the first feature and the second feature means that there is transmission of electrical signals between the first feature and the second feature, that is, there is an electrical relationship, and the way to achieve the transmission of electrical signals may be electrical connection of wires, radio connection, electrical connection of electromagnetic media (such as semiconductors), communication achieved by channels, etc.

[0076] In the embodiments of the present application, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Furthermore, a first feature being "above," "above," and "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0077] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] See also Figures 1 to 3 , the embodiment of the present application discloses a streaming processing system for a large language model, the structural diagram of the streaming processing system is as follows Figure 1 As shown, the streaming processing system includes a large language model module and a model parameter optimization module, wherein the large language model module has a built-in large language model architecture, and the model parameter optimization module is electrically connected to the large language model module.

[0079] In this streaming processing system, the large language model module is configured to call the large language model architecture in an offline environment to run in batch mode, while in an online environment, the streaming model parameters are used as the model parameters of the large language model architecture, and the large language model architecture is called to implement streaming reasoning through a token decoupling caching mechanism. In this embodiment, the online environment refers to the large language model module receiving tasks that require real-time or near real-time processing and is in a working environment that requires fast response and large amounts of data processing, while the offline environment refers to the large language model module receiving tasks that can be processed in batches without real-time data input, such as data mining and machine learning model training, and is in a working environment where processing time is not restricted and can run in the background.

[0080] Figure 3 For a schematic diagram of the token decoupling cache mechanism, see Figure 3 In this embodiment, the large language model module implements streaming reasoning through the token decoupling cache mechanism in the following way: set the initial input token cache pool and output token cache pool, and then alternately execute token reading and token generation. When reading tokens, the token to be read is subjected to attention calculation with the input token cache pool to obtain a first attention calculation result, and the first attention calculation result is stored in the input token cache pool. Figure 3 As shown, the output token pool cannot participate in the attention calculation of the token to be read. When generating a token, the latest token from the output token pool is first taken out. Then, the attention calculation is performed on this token with the input token pool and the output token pool to obtain the second attention calculation result, which is then stored in the output token pool.

[0081] See also Figure 2 In this streaming processing system, the model parameter optimization module is set to use a streaming attention mask mechanism to optimize the parameters of the large language model architecture to obtain the streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process, so that the large language model architecture only focuses on the currently available input and context.

[0082] See also Figure 2 In this embodiment, the model parameter optimization module is configured to perform the following steps:

[0083] A1: Construct an attention mask matrix. Specifically, in this embodiment, the constructed attention mask matrix is ​​a real field square matrix with the following structure:

[0084] The order of the square matrix is ​​equal to the length of the encoded data;

[0085] For the value of the element at the pth row and qth column of the square matrix,

[0086] If p < q, the value is 1; if p ≥ q, the following situations occur:

[0087] When p<m and q<m or when p>m and q>m, the value is 0;

[0088] When p>m and q<m, if pm<q is also satisfied, the value is 0;

[0089] When p>m and q<m, if pm≥q is also satisfied, the value is 1;

[0090] Here, m represents the length of one frame of streaming input data of the large language model framework, that is, the length of the streaming input data in one frame of streaming input and output data.

[0091] A2: Retrieve several frames of streaming input and output data from the large language model framework, and arrange each frame of streaming input and output data in a batch data format to form corresponding streaming input and output data.

[0092] A3: Group and position encode each streaming input and output data to obtain its own encoded data, and integrate them into an encoded data matrix. The encoded data matrix is ​​mapped through the linear layer mapping of the transformer architecture (Transformer) to obtain the query matrix (Query), key matrix (Key), and value matrix (Value) of the encoded data matrix.

[0093] Specifically, in step A3, the encoded data obtained by performing group position encoding on a certain stream input and output data is expressed as follows:

[0094] ,

[0095] Where,

[0096] Encoded data representing the streaming input data of the large language model framework;

[0097] Encoded data representing the output data of a large language model framework.

[0098] The encoded data of each streaming input and output data is used as a matrix row and is integrated into a matrix in time sequence to obtain the encoded data matrix. The encoded data matrix is ​​then mapped through the linear layer mapping of the transformer architecture to obtain the query matrix, key matrix, and value matrix of the encoded data matrix.

[0099] A4: Substitute the encoded data, attention mask matrix, query matrix, key matrix, and value matrix into the attention function used to evaluate the degree of attention of the large language model architecture to the currently available input and context to obtain the attention score matrix of the large language model architecture.

[0100] Specifically, in this embodiment, step A4 includes the following steps: A41: right-multiply the query matrix with the diagonal rotation matrix to obtain the first encoding matrix, and right-multiply the key matrix with the diagonal rotation matrix to obtain the second encoding matrix. The diagonal rotation matrix is:

[0101] ,

[0102] ,

[0103] Where,

[0104] represents the diagonal rotation matrix;

[0105] represents a diagonal block matrix;

[0106] Represents the rotation angle;

[0107] Represents the algorithmic dimension of the large language model architecture.

[0108] A42: Substitute the attention mask matrix, the first encoding matrix, the second encoding matrix, and the value matrix into the attention function to obtain the attention score matrix of the large language model architecture. The calculation formula of the attention function is as follows:

[0109] ,

[0110] Where,

[0111] represents the attention score matrix;

[0112] represents the query matrix;

[0113] represents the bond matrix;

[0114] represents the attention mask matrix;

[0115] Representative value matrix;

[0116] Represents the transpose of a matrix.

[0117] For the matrix , The function definition is:

[0118] ,

[0119] In this embodiment, the attention mask matrix is square matrix, matrix Also Order square matrix, let

[0120] ,

[0121] and then,

[0122] ,

[0123] Finally, multiply the value matrix by the above formula to get the attention score matrix.

[0124] A5: Use the attention score matrix as the forward propagation attention score matrix of the large language model architecture to enable the large language model architecture to perform forward propagation, and update the parameters of the large language model architecture through gradient backpropagation to obtain the streaming model parameters.

[0125] In this embodiment, the method of updating the parameters of the large language model architecture through gradient back propagation is: using the model algorithm of the Adam optimizer to perform gradient back propagation on the large language model architecture to achieve parameter update, obtain streaming model parameters, and transmit the streaming model parameters to the large language model module.

[0126] The following further discloses a method for using the streaming processing system of a large language model in this embodiment. The method includes the following steps:

[0127] S1: Through the model parameter optimization module, the streaming attention mask mechanism is used to optimize the parameters of the large language model architecture and obtain the streaming model parameters. During the parameter optimization process, the attention range of the large language model architecture is dynamically adjusted so that the large language model architecture only focuses on the currently available input and context.

[0128] S2: Use the streaming model parameters as the model parameters of the large language model architecture through the large language model module, and call the large language model architecture to implement streaming reasoning through the token decoupling caching mechanism.

[0129] The following will describe in detail the technical effects of the streaming processing system for the large language model in this embodiment. Figure 4 and Figure 5 In this embodiment, the streaming processing system is deployed on a device that performs streaming text translation tasks for testing. Four different large language models are selected, namely Gemma2-2B, Phi3-mini, LLama3.1-8B and Falcon-1B. These large language models are respectively deployed in the large language model module to perform English-French translation and English-German translation. The performance test results are shown in the figure. Figure 4and Figure 5 As shown in the performance graph, the horizontal axis is the word count, and the vertical axis is the similarity index (BLEU) for evaluating translation quality. Gemma2 (Ours) represents the performance image of placing the Gemma2-2B model in the large language model module, Phi3 (Ours) represents the performance image of placing the Phi3-mini model in the large language model module, LLama (Ours) represents the performance image of placing the LLama3.1-8B model in the large language model module, and Falcon (Ours) represents the performance image of placing the Falcon-1B model in the large language model module. Figure 4 and Figure 5 It can be seen that as the number of words increases, the similarity also increases, and the translation quality is significantly improved, which shows that the performance of the streaming processing system is good.

[0130] At the same time, the streaming processing system for a large language model disclosed in this embodiment, by providing a large language model module and a model parameter optimization module, allows the large language model architecture to be called in batch mode in an offline environment through the large language model module. In an online environment, the streaming model parameters are used as the model parameters of the large language model architecture, and the large language model architecture is called to implement streaming inference through a token decoupling caching mechanism. Furthermore, the model parameter optimization module can also optimize the parameters of the large language model architecture using a streaming attention masking mechanism to dynamically adjust the attention range of the large language model architecture, so that the large language model architecture focuses only on the currently available input and context, and obtains the streaming model parameters. Therefore, the present invention utilizes the large language model architecture as the underlying architecture, overcoming the drawback of incompatibility with pre-trained large language models. By setting two states, online and offline, the large language model module is seamlessly compatible with batch mode, and can be used in different tasks (such as machine translation, speech recognition, real-time conversation, etc.), ensuring that the large language model module can meet scalability requirements between batch mode and streaming mode, and at a low computational cost.

[0131] In the description of the embodiments of the present application, it should be noted that in the description of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present application.

[0132] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are mutually inconsistent.

[0133] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A streaming processing system for a large language model, characterized in that: include: A large language model module, having a built-in large language model architecture, configured to call the large language model architecture to run in batch mode in an offline environment, and to use streaming model parameters as model parameters of the large language model architecture in an online environment, and to call the large language model architecture to implement streaming reasoning through a token decoupled caching mechanism; a model parameter optimization module, electrically connected to the large language model module, configured to optimize the parameters of the large language model architecture using a streaming attention masking mechanism to obtain the streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process so that the large language model architecture only focuses on currently available input and context; The large language model module implements streaming reasoning through the token decoupling cache mechanism as follows: Set up the initial input token buffer pool and output token buffer pool, and then alternately execute token reading and token generation; When reading a token, performing attention calculation on the token to be read and the input token buffer pool to obtain a first attention calculation result, and storing the first attention calculation result in the input token buffer pool; When generating a token, first take out the latest token from the output token cache pool, then perform attention calculation on the token with the input token cache pool and the output token cache pool to obtain a second attention calculation result, and store the second attention calculation result in the output token cache pool.

2. The streaming processing system for a large language model according to claim 1, characterized in that The model parameter optimization module is configured to perform the following steps: A1: Construct attention mask matrix; A2: Retrieving multiple frames of streaming input data and output data of the large language model framework, and arranging each frame of the streaming input data and output data in a batch data format to form corresponding streaming input and output data; A3: performing group position encoding on each stream input and output data to obtain respective encoded data, and integrating the encoded data into an encoded data matrix, and mapping the encoded data matrix through a linear layer mapping of a transformer architecture to obtain a query matrix, a key matrix, and a value matrix of the encoded data matrix; A4: Substituting the encoded data matrix, the attention mask matrix, the query matrix, the key matrix, and the value matrix into an attention function for evaluating the degree of attention of the large language model architecture to the currently available input and context, to obtain an attention score matrix of the large language model architecture; A5: Use the attention score matrix as the attention score matrix of the forward propagation of the large language model architecture to enable the large language model architecture to perform forward propagation, and update the parameters of the large language model architecture through gradient backpropagation to obtain the streaming model parameters.

3. The streaming processing system for a large language model according to claim 2, characterized in that The attention mask matrix is ​​a real field square matrix with the following structure: The order of the square matrix is ​​equal to the length of the encoded data; For the value of the element at the pth row and qth column of the square matrix, If p < q, the value is 1; If p≥q, then the following situations occur: When p<m and q<m or when p>m and q>m, the value is 0; When p>m and q<m, if pm<q is also satisfied, the value is 0; When p>m and q<m, if pm≥q is also satisfied, the value is 1; Here, m represents the length of the streaming input data of the large language model framework.

4. The streaming processing system for a large language model according to claim 3, characterized in that In step A3, the encoded data obtained by performing group position encoding on a certain stream input and output data is expressed as follows: ， Where, represents encoded data obtained by performing group position encoding on streaming input data of streaming input and output data; Represents encoded data obtained by performing group position encoding on output data of streaming input and output data.

5. The streaming processing system for a large language model according to claim 4, characterized in that Step A4 includes the following steps: A41: right-multiply the query matrix by a diagonal rotation matrix to obtain a first encoding matrix, and right-multiply the key matrix by the diagonal rotation matrix to obtain a second encoding matrix; A42: Substitute the attention mask matrix, the first encoding matrix, the second encoding matrix and the value matrix into the attention function to obtain the attention score matrix of the large language model architecture.

6. The streaming processing system for a large language model according to claim 5, characterized in that The diagonal rotation matrix is: ， ， Where, represents the diagonal rotation matrix; represents a diagonal block matrix; Represents the rotation angle; Represents the algorithmic dimension of the large language model architecture.

7. The streaming processing system for a large language model according to claim 6, characterized in that The calculation formula of the attention function is as follows: ， Where, represents the attention score matrix; represents the query matrix; represents the bond matrix; represents the attention mask matrix; represents the value matrix; Represents the transpose of a matrix; For the matrix , The function definition is: 。 8. A streaming processing method for a large language model, characterized in that: A streaming processing system for a large language model according to any one of claims 1 to 7, comprising the following steps: S1: Optimize the parameters of the large language model architecture using a streaming attention mask mechanism through the model parameter optimization module to obtain streaming model parameters, and dynamically adjust the attention range of the large language model architecture during the parameter optimization process so that the large language model architecture only focuses on the currently available input and context; S2: Using the streaming model parameters as model parameters of the large language model architecture through the large language model module, and calling the large language model architecture to implement streaming reasoning through a token decoupling cache mechanism.

Images