Streaming thinking and reasoning system and method for large language model

Abstract

The invention relates to the technical field of large language models, in particular to a streaming thinking and reasoning system and method for a large language model, and the system comprises a streaming thinking data generation unit, a streaming training unit and a streaming parallel reasoning unit. The streaming thinking data generation unit is used for splitting the continuous reasoning data into a plurality of reasoning thinking units to obtain a plurality of thinking fragments; the streaming training unit is used for pre-training the large language model by adopting an independent position coding mechanism and an independent streaming mask mechanism, so that the large language model adapts to a streaming thinking normal form; and the streaming parallel reasoning unit is used for performing cross-GPU streaming parallel reasoning by using a trained large language model for a plurality of thinking fragments, performing cross-GPU decoupling on an input token cache pool and a reasoning thinking token cache pool, and realizing cross-GPU concurrent execution of reading and thinking during reasoning. Compared with the prior art, the method has the advantages of remarkably reducing delay, improving user interaction experience and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, and in particular to a large language model streaming thinking and reasoning system and method. BACKGROUND

[0002] In recent years, the thinking chain reasoning of large language models (LLMs) has demonstrated strong reasoning and thinking capabilities. However, the existing LLMs' reasoning and thinking paradigm is based on a "batch processing" mode, i.e., the model first needs to wait for the user to input the complete context or question, and then starts reasoning and thinking. This traditional "think after complete input" mode is usually divided into two rigid stages in technology: Prefill Phase: the model receives and processes all input tokens of the user, and computes the initial key-value cache (KV Cache) at once.

[0003] Decode Phase: the model starts to generate tokens autoregressively based on the KV Cache generated in the Prefill Phase.

[0004] This mode has the following significant shortcomings: 1) High TTFT (Time To First Token): the user has to wait for the model to complete the prefill computation of all input content before seeing the first output token. For long input reasoning and thinking tasks (such as long document summarization, complex multi-round dialogue history), the prefill phase can be very time-consuming, resulting in a very high TTFT.

[0005] 2) Separation of reading and thinking: in the traditional mode, reading (prefill) and thinking (decode) are completely separated. This means that the model cannot start generating intermediate inferences or thinking steps immediately when it receives early key information (e.g., the first few conditions of a complex mathematical problem), but has to wait until all information is input. This does not conform to human cognitive habits and can lead to insufficient attention to early information when processing long-distance dependencies and complex reasoning.

[0006] The existing technologies generally have high reasoning delay, poor interactive experience, and separation of reading and thinking, and there is an urgent need for a new streaming thinking framework that can achieve low-latency, high-concurrency streaming thinking without sacrificing model performance.

[0007] A search revealed Chinese invention patent application publication number CN120181070A, which discloses a streaming processing system and method for a large language model. In an offline environment, the large language model module calls the large language model architecture to run in batch processing mode. In an online environment, streaming model parameters are used as model parameters for the large language model architecture, which then uses a token-decoupled caching mechanism to achieve streaming inference. Furthermore, a model parameter optimization module employs a streaming attention mask mechanism to optimize the parameters of the large language model architecture, dynamically adjusting its attention scope so that it only focuses on currently available input and context to obtain streaming model parameters. This ensures the large language model module meets the scalability requirements between batch and streaming modes while maintaining low computational cost. However, this existing patent application suffers from issues such as word-level attention masking and a token-decoupled caching mechanism using two sets of serial cache pools, resulting in alternating read and write operations and thus high latency and low concurrency.

[0008] How to achieve low-latency, high-concurrency streaming thinking and reasoning for large language models has become a technical problem that needs to be solved. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a streaming thinking and reasoning system and method for a large language model.

[0010] The objective of this invention can be achieved through the following technical solutions: According to a first aspect of the present invention, a streaming thinking and reasoning system for a large language model is provided, the system comprising a streaming thinking data generation unit, a streaming training unit, and a streaming parallel reasoning unit; The streaming thinking data generation unit is used to split continuous reasoning data into several reasoning thinking units. After each input unit arrives, the large language model is immediately triggered to generate corresponding thinking segments, and finally multiple thinking segments are obtained. The streaming training unit uses an independent positional encoding mechanism and a streaming masking mechanism to pre-train the large language model, enabling the large language model to adapt to the streaming thinking paradigm. The streaming parallel inference unit performs cross-GPU streaming parallel inference on multiple thought fragments output by the streaming thinking data generation unit using a trained large language model. Specifically, it sets up an input token cache pool, a reasoning thinking token cache pool, and an answer token cache pool, and decouples the input token cache pool and the reasoning thinking token cache pool across GPUs to achieve cross-GPU concurrent execution of reading and thinking during inference.

[0011] As a preferred technical solution, the process of splitting continuous reasoning data into several reasoning units includes: Step 1.1, to stream the input sentence , Inserting sentence-level boundary markers <eos>to define the smallest unit of reasoning thought; Step 1.2, guiding the large language model to generate sequential reasoning for the streaming input sentence, when encountering <eos>The step is terminated at this time and the addition <eot>Marking; Step 1.3, introducing a teacher large language model to reconstruct the reasoning step of step 1.2 to obtain reasoning and thinking content corresponding to each reasoning thinking unit; Step 1.4, guiding the large language model to summarize and reflect on the thinking process, adjusting the depth of thinking according to the difficulty of the task, and generating global reasoning and thinking content.

[0012] As a preferred technical solution, the process of cross-GPU streaming parallel reasoning includes: Initialize the input token cache pool, the reasoning and thinking token cache pool, and the answer token cache pool; The step of updating the input token cache pool: on the first GPU device, perform attention calculation on the tokens of the input reasoning unit and the input cache pool, and update the obtained first attention result to the input token cache pool; The step of updating the reasoning and thinking token cache pool: on the second GPU device, if the current input reasoning unit has corresponding reasoning and thinking content, wait for new input, otherwise perform attention calculation on the tokens of the reasoning and thinking token cache pool and the updated input token cache pool, and update the obtained second attention result to the reasoning and thinking token cache pool; Repeat the steps of updating the input token cache pool and updating the reasoning and thinking token cache pool in parallel until the input content and the reasoning and thinking content are no longer updated; Merge the input token cache pool and the reasoning and thinking token cache pool to obtain a merged cache pool, and perform attention calculation on the answer token cache pool and the merged cache pool, and update the obtained third attention result to the answer token cache pool.

[0013] As a preferred technical solution, the streaming mask mechanism is based on the input sentence, and through dynamic time constraints, it ensures that each step in the streaming reasoning only depends on the historical tokens and the generated thinking content.

[0014] As a preferred technical solution, the streaming mask mechanism specifically includes: Initialize the streaming attention mask matrix to , where and are the token lengths of the input content and the reasoning and thinking content, respectively; For any position in the attention mask matrix , if , mask the corresponding position ; Block the attention from the thinking step t to the input sentence sequence number greater than t on the attention mask matrix to obtain the streaming mask matrix.

[0015] As a preferred technical solution, the independent location coding mechanism is specifically as follows: The streaming training data is mapped through a linear layer to obtain the query matrix Q, the key matrix K, and the value matrix V; For length of n Input sentence ， Its token sequence has a length of The data is assigned location numbers. The corresponding reasoning sequence is: Its token sequence is of length The data is assigned location numbers as follows: ; Define rotation matrix ,in It is the dimension of the model. It represents the position number, and diag is the diagonal block matrix. ,in i The model represents the first i One dimension, Indicates the corresponding rotation angle; For any element of the streaming training data Its location information is encoded as QR(m) and KR(m).

[0016] As a preferred technical solution, the system further includes a quality control unit, which is used to evaluate, filter or correct the quality of the thought fragments generated by the streaming thought data generation unit, and output the results to the streaming parallel inference unit for parallel inference.

[0017] As a preferred technical solution, the quality control unit uses granular score to evaluate whether the number of input units and thought segments match, and uses sentence-level similarity to evaluate whether the reasoning process is performed in a streaming, sequential manner. If a data fails either the granularity assessment or the sequence consistency assessment twice consecutively, it is discarded.

[0018] According to a second aspect of the present invention, a streaming thinking and reasoning method for a large language model is provided, the method being applicable to the streaming thinking and reasoning system for a large language model described in the first aspect and various possible technical solutions of the first aspect, the method comprising: Step 1: The streaming thinking data generation unit breaks down continuous reasoning data into several reasoning thinking units. After each input unit arrives, the large language model is immediately triggered to generate the corresponding thinking fragments, ultimately resulting in multiple thinking fragments. Step 2: The streaming training unit uses an independent positional encoding mechanism and a streaming masking mechanism to pre-train the large language model, enabling the large language model to adapt to the streaming thinking paradigm. Step 3, for the multiple thinking segments output by the streaming thinking data generation unit, the streaming parallel reasoning unit uses the trained large language model to perform cross-GPU streaming parallel reasoning, specifically: input token cache pool, reasoning thinking token cache pool and answer token cache pool are set respectively, the input token cache pool and the reasoning thinking token cache pool are decoupled across GPUs, and cross-GPU concurrent execution of reading and thinking is realized during reasoning.

[0019] As a preferred technical solution, the method further comprises quality evaluation, screening or error correction on the thinking segments output by the streaming thinking data generation unit, specifically: the granularity score is used to evaluate whether the number of input units and reasoning thinking units matches, the sentence-level similarity is used to evaluate whether the reasoning process is performed in a streaming, sequential manner, and if the granularity evaluation and sequence consistency evaluation fail consecutively, the data is discarded.

[0020] Compared with the prior art, the present application has the following beneficial effects: 1) The present application produces intermediate thinking segments corresponding to the input units in parallel while receiving the input units. This streaming thinking paradigm of large language models enables reading and thinking to be performed in parallel, without waiting for the complete input before starting to think, significantly reducing the delay and improving the user interaction experience.

[0021] 2) The present application uses independent position encoding and streaming model specific mask mechanism to train the large language model in streaming scenario, solving the position information confusion problem and streaming constraint problem caused by concurrent input stream and thinking stream, making the large language model adapt to the streaming thinking paradigm, and keeping the reasoning performance basically unchanged in reasoning tasks.

[0022] 3) The streaming parallel reasoning mechanism of the present application improves early attention, forcing the large language model to generate thinking segments almost immediately after receiving early input information, reducing the number of input tokens waiting for reasoning by about 80%, and reducing the overall answer delay by more than 60%, thus significantly reducing the reasoning delay, improving the parallel efficiency, cache resource utilization and user interaction experience.

[0023] 4) The quality control unit of the present application ensures the quality of streaming data through fine-grained evaluation and sequence consistency evaluation, prevents the accumulation of low-quality or speculative thinking segments generated by the large language model when receiving incomplete input, and ensures the accuracy of the final output. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a structural schematic diagram of the streaming thinking and reasoning system in the present application; Figure 2 is a flowchart of the streaming thinking and reasoning method in the present application; Figure 3 A schematic diagram contrasting the principles of streaming thinking and reasoning and batch processing of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0026] The present application aims to solve the problems of waiting for complete input mode before starting reasoning or thinking in the existing large language model thinking chain reasoning, resulting in high first word latency (TTFT), poor interactive experience, separation of reading and thinking, and the like.

[0027] The present embodiment relates to a large language model streaming thinking and reasoning system, and proposes a streaming thinking (Streaming-Thinking) paradigm, which changes the thinking process of a large language model from the traditional mode of thinking after waiting for complete input to a parallel mode of reading and thinking.

[0028] As Figure 1 The system includes a streaming thinking data generation unit, a quality control unit, a streaming training unit, and a streaming parallel reasoning unit. Through the cooperative work of these modules, the present application significantly reduces the delay of starting thinking while maintaining the basic performance of thinking, and improves the attention of a large language model to early input information.

[0029] The streaming thinking data generation unit is used to split continuous reasoning data into several reasoning thinking units (Reasoning Chunks). After each input unit arrives, the large language model generates the corresponding thinking fragment (Thought Fragments) immediately; and after all the reasoning thinking units end, in-depth thinking is performed.

[0030] The quality control unit is used to evaluate, screen, or correct the quality of the thinking fragments generated by the streaming thinking data generation unit. This module is used to prevent the accumulation of low-quality or speculative thinking fragments generated by the model when receiving incomplete input, and to ensure the accuracy of the final output.

[0031] The streaming training unit is used to adapt the pre-trained LLM to the above-mentioned streaming thinking paradigm, which includes two key technical points: an independent position encoding mechanism and a streaming mask mechanism. The independent position encoding mechanism is used to solve the position information confusion problem caused by the concurrency of the input stream and the thinking stream, and the streaming mask mechanism is used to solve the streaming constraint problem caused by the concurrency of the input stream and the thinking stream.

[0032] The streaming parallel inference unit adopts a parallel inference mechanism, and realizes concurrent execution of reading and thinking during reasoning. The core is to decouple the KV cache generated by input tokens and inference thinking tokens.

[0033] Further, the process of generating streaming reasoning thinking content by the streaming thinking data generation unit includes: (1.1) First, for the streaming input sentence , Insert sentence-level boundary markers <eos>to define the smallest unit of reasoning thought; (1.2) Direct the LLM to generate sequential reasoning for the preceding sentence, upon encountering <eos>The step is terminated at this time and the addition <eot>mark; (1.3) Introduce a more powerful teacher LLM reconstruction (1.2) reasoning steps to obtain the reasoning and thinking content corresponding to each reasoning and thinking unit. .

[0034] (1.4) After obtaining the content of sequential reasoning, guide LLM to summarize and reflect on the thinking process, adjust the depth of thinking according to the different difficulty of the task, and generate global reasoning thinking content.

[0035] Furthermore, the evaluation of the quality control unit is achieved through the following sub-steps: (2.1) Granularity evaluation: A granularity score is defined to evaluate whether the number of input units and thought fragments match. It is defined as the number of boundary markers in the input. The number of boundary markers in the output The ratio: ; (2.2) Sequence Consistency Assessment: A sequence consistency score is defined using sentence-level similarity to assess whether the reasoning process proceeds in a streaming, order-preserving manner. The similarity is calculated by the SentenceBERT model; As a reference sequence, Candidate sequences are the inference segments output by the model in real time. (2.3) Repeat the generation process of step one. If the granularity evaluation or sequence consistency evaluation fails twice in a row, the data is discarded.

[0036] Furthermore, the streaming training mechanism unit processes the streaming training data through the following sub-steps: (3.1) The streaming training data is mapped through a linear layer to obtain Q, K and V (corresponding to Query, Key and Value); (3.2) Introduce an independent position encoding mechanism: for lengths of n Input sentence , ， Its token sequence has a length of Data Assign position numbers as The corresponding reasoning sequence is: Its token sequence is of length Data The location numbers are assigned as follows: Define the rotation matrix. ,in It is the dimension of the model. It is the position number, and diag is the diagonal block matrix; ,in i The model represents the first i There are three dimensions, i = 1, 2, ... ; This represents the corresponding rotation angle; for training data any element Its location information can be encoded as QR(m) and KR(m); (3.3) Introducing a sentence-level streaming masking mechanism: Initializing the streaming attention mask matrix as follows ,in and These are the token lengths for the input content and the reasoning content, respectively; for the attention mask matrix... any position in ,if Then the corresponding position will be covered. In the matrix Blocking from the thinking steps t The input sentence is greater than t Attention is used to obtain the streaming mask matrix: Where T is the length of the input data token. Represents a number that is infinitely large. It's an indicator function; values ​​that don't meet the conditions are assigned 0. When the condition in the indicator function is satisfied... For an infinite value, the streaming mask matrix Cover the original M ( i , j This prevents the location from being considered; when the condition in the indicator function is not met, 0, streaming mask matrix equal to the original M ( i , j This means that only historical tokens are allowed to be considered. Dynamic time constraints ensure that each step in streaming reasoning relies solely on historical tokens, preventing premature disclosure of future information from disrupting the reasoning logic.

[0037] (3.4) For streaming training data, design an attention score calculation formula to achieve attention forward propagation: , where d is the algorithm dimension of the LLM model architecture.

[0038] The application overcomes the quality decline of stream processing through a stream training mechanism: independent position coding ensures that the input stream and the thinking stream still have conflict-free relative and absolute position awareness when parallel, solving the position confusion problem; sentence-level stream attention mask forces large language models to follow logical information flow, ensuring causality, preventing large language models from "looking ahead" cheating, and solving the mismatch with pre-training.

[0039] Further, the stream parallel reasoning of the stream parallel reasoning unit is implemented through the following sub-steps: (4.1) Initialize the input token cache pool , the reasoning thinking token cache pool , and the answer token cache pool ; (4.2) On the first GPU device, perform attention calculation on the tokens of the input reasoning unit and the input cache pool , and update the obtained first attention result to ; (4.3) On the second GPU device, if the current input reasoning unit has corresponding reasoning thinking content, wait for new input, otherwise perform attention calculation on the tokens of the reasoning thinking token cache pool and the updated , and update the obtained second attention result to ; (4.4) Repeat steps (4.2) and (4.3) in parallel until the input content and the reasoning thinking content are no longer updated; continue to reuse the existing KV cache to avoid repeated calculation; (4.5) Merge and to obtain a merged cache pool, and perform attention calculation on the answer token cache pool and the merged cache pool, and update the obtained third attention result to .

[0040] The stream parallel reasoning unit adopts a three-cache pool design across GPUs in parallel. The large language model can read and think in parallel, and write the answer after thinking, which is a read-write synchronous stream of multi-GPU parallel. Compared with the existing read-write alternating stream of single-GPU serial, the multi-cache pool and multi-GPU parallel read-write simultaneously process input calculation, reasoning calculation, and answer generation, thereby improving parallel efficiency and cache resource utilization, enabling faster output of intermediate results and final answers, and improving user interaction experience.

[0041] Figure 3 The stream parallel reasoning mechanism of the application is shown in the upper part of Figure 3 , which replaces the rigid pre-padding-decoding two-stage process described in the background art Figure 3 Decoupling the left part of the language model allows large language models to begin generating thought fragments almost immediately after receiving the first input unit, without waiting for complete input. This reduces the number of input tokens (TTFT) that inference begins to wait for by approximately 80%, resulting in an overall response latency reduction of over 60%. Consequently, inference latency is significantly reduced, enhancing the user's interactive experience.

[0042] The streaming thinking paradigm of this invention forces large language models to immediately produce intermediate thinking fragments corresponding to the input units in parallel upon receiving early inputs. This enhances the model's attention to early input information, and the generated thinking fragments provide intermediate steps for the large language model's final decision, thereby enhancing the interpretability of the reasoning process.

[0043] This embodiment also relates to a streaming thinking and reasoning method for a large language model, such as... Figure 2 The method includes the following steps: Step 1: The streaming thinking data generation unit adopts the streaming thinking paradigm of the large language model. By using marker symbols, continuous reasoning data is divided into several reasoning thinking units, and global reasoning thinking content is generated. Specifically, the smallest reasoning thinking unit is defined. Upon receiving each input unit, the large language model is immediately triggered to generate the corresponding thinking fragment, reconstructing multiple reasoning thinking units and generating global reasoning thinking content.

[0044] Step 2 involves quality assessment, screening, or error correction of the thinking fragments output by the streaming thinking data generation unit. Specifically, granularity scores are used to assess whether the number of input units and reasoning thinking units matches, and sentence-level similarity is used to assess whether the reasoning process is conducted in a streaming, sequential manner. If at least one of the granularity assessment or sequence consistency assessment fails twice consecutively, the data is discarded.

[0045] Step 3: The streaming training data is mapped through a linear layer to obtain Q, K, and V. The streaming input sentences and reasoning content are independently encoded at their positions. A streaming masking mechanism is used to train the large language model in a streaming scenario. Through dynamic time constraints, it is ensured that each step in streaming reasoning relies only on historical tokens to avoid premature leakage of future information that could disrupt the reasoning logic.

[0046] Step 4, Streaming Parallel Inference: The streaming parallel inference unit separates the token cache pool and the inference think token cache pool, and enables parallel reading and inference across GPUs through streaming parallel inference. Specifically: Step 4.1, Initialize the input token cache pool Reasoning token cache pool and answer token cache pool ; Step 4.2, on the first GPU device, perform attention calculation on the tokens of the input reasoning unit and the input cache pool , and update the obtained first attention result to ; Step 4.3, on the second GPU device, if the current input reasoning unit has corresponding reasoning thinking content, wait for new input, otherwise perform attention calculation on the tokens of the reasoning thinking token cache pool and the updated , and update the obtained second attention result to ; Step 4.4, repeat step 4.2 and step 4.3 in parallel until the input content and the reasoning thinking content are no longer updated; Step 4.5, merge and to obtain a merged cache pool, and perform attention calculation on the answer token cache pool and the merged cache pool, and update the obtained third attention result to .

[0047] To verify the beneficial effects of the present application, verification is carried out on a mathematical reasoning task. Compared with the Batch-thinking and Interleaved-Thinking models, the Streaming-Thinking of the present application achieves the best performance in accuracy (Acc) and delay (TTFT and Delay), as shown in Table 1.

[0048] Table 1

[0049] wherein GSM - Symbolic and MetaMathQA are two different mathematical reasoning task datasets, and D1~D3 represent different global reasoning depths after sequential reasoning ends.

[0050] The electronic device of the present application includes a central processing unit (CPU) which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0051] A number of components in the device are connected to the I / O interface, including: input units, such as a keyboard, mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as magnetic disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication units allow the device to exchange information / data with other devices over computer networks, such as the Internet, and / or various telecommunication networks.

[0052] The processing units perform the various methods and processes described above. For example, in some embodiments, the methods can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage units. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the device via the ROM and / or the communication units. When the computer program is loaded onto the RAM and executed by the CPU, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the methods by way of other means, such as by way of firmware.

[0053] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0054] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow charts and / or block diagrams to be implemented. The program code can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0055] In the context of the present application, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0056] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / eot> < / eos> < / eos> < / eot> < / eos> < / eos>

Claims

1. A streaming thinking and reasoning system for large language models, characterized in that, The system comprises a streaming thinking data generation unit, a streaming training unit and a streaming parallel inference unit; The streaming thinking data generation unit is configured to split continuous inference data into a plurality of inference thinking units, and trigger a large language model to generate corresponding thinking segments immediately after each input unit arrives, so as to finally obtain a plurality of thinking segments; The streaming training unit is configured to pre-train the large language model by using an independent position encoding mechanism and a streaming mask mechanism, so that the large language model is adapted to the streaming thinking paradigm; The streaming parallel inference unit is configured to perform cross-GPU streaming parallel inference on the plurality of thinking segments output by the streaming thinking data generation unit by using the trained large language model, specifically by setting an input token cache pool, an inference thinking token cache pool and an answer token cache pool, decoupling the input token cache pool and the inference thinking token cache pool across GPUs, and performing cross-GPU concurrent execution of reading and thinking during inference.

2. The stream-of-consciousness thinking and reasoning system of a large language model according to claim 1, wherein, The process of splitting the continuous inference data into a plurality of inference thinking units comprises: Step 1.1, to stream the input sentence , Inserting sentence-level boundary markers <eos>defining the smallest inference thinking unit;< / eos> Step 1.2, guiding a large language model to generate sequential reasoning for a streaming input sentence, upon encountering <eos>The step is terminated at this time and the addition <eot>marking;< / eot> < / eos> Step 1.3, introducing a teacher large language model to reconstruct the inference step of step 1.2, to obtain inference and thinking content corresponding to each inference thinking unit; Step 1.4, guiding the large language model to summarize and reflect on the thinking process, adjusting the depth of thinking according to the difficulty of the task, and generating global inference thinking content.

3. The stream-of-consciousness thinking and reasoning system of a large language model of claim 1, wherein, The process of cross-GPU streaming parallel inference comprises: initializing the input token cache pool, the inference thinking token cache pool and the answer token cache pool; updating the input token cache pool: on the first GPU device, performing attention calculation on the tokens of the input inference unit and the input cache pool, and updating the obtained first attention result to the input token cache pool; updating the inference thinking token cache pool: on the second GPU device, if the current input inference unit has corresponding inference thinking content, waiting for new input, otherwise performing attention calculation on the tokens of the inference thinking token cache pool and the updated input token cache pool, and updating the obtained second attention result to the inference thinking token cache pool; repeating the steps of updating the input token cache pool and updating the inference thinking token cache pool in parallel until the input content and the inference thinking content are no longer updated; merging the input token cache pool and the inference thinking token cache pool to obtain a merged cache pool, and performing attention calculation on the answer token cache pool and the merged cache pool, and updating the obtained third attention result to the answer token cache pool.

4. The stream of consciousness and reasoning system of a large language model of claim 1, wherein, The streaming mask mechanism is based on an input sentence, and ensures that each step in the streaming inference only depends on historical tokens and generated thinking content by dynamic time constraints.

5. The stream of consciousness and reasoning system of a large language model according to claim 4, wherein, The streaming mask mechanism specifically comprises: Initialize the flow attention mask matrix as wherein and are the token lengths of the input content and the inferential thinking content, respectively. For attention mask matrix is Any position in If Mask the corresponding position ; In the attention mask matrix The upper block breaks from the thinking step t to the input sentence sequence number greater t attention, get streaming mask matrix.

6. The stream of consciousness and reasoning system of a large language model of claim 1, wherein, The independent position encoding mechanism specifically comprises: The streaming training data is mapped to a query matrix Q, a key matrix K and a value matrix V through a linear layer; For length of n Input sentence ， Its token sequence has a length of The data is assigned location numbers. The corresponding reasoning sequence is: Its token sequence is of length The data is assigned location numbers as follows: ; Definition of rotation matrix where is the dimension of the model, is the position number, diag is a diagonal block matrix, where i denotes the i dimension of the model, denotes the corresponding rotation angle; For any one element of the streaming training data whose position information is encoded as QR(m) and KR(m).

7. The stream of consciousness and reasoning system of a large language model according to claim 1, wherein, The system further comprises a quality control unit configured to perform quality evaluation, screening or error correction on the thinking segments generated by the streaming thinking data generation unit, and output the results to the streaming parallel inference unit for parallel inference.

8. The stream of consciousness and reasoning system of a large language model according to claim 7, wherein, The quality control unit uses granularity score to evaluate whether the number of input units and thinking segments matches, and uses sentence-level similarity to evaluate whether the reasoning process is in a streaming, sequential manner; If the granularity evaluation or sequence consistency evaluation fails twice in a row, the data is discarded.

9. A method for streaming thinking and reasoning of a large language model, characterized in that, The method is suitable for the streaming thinking and reasoning system of the large language model according to any one of claims 1-8, and the method comprises: Step 1, the streaming thinking data generation unit splits the continuous reasoning data into several reasoning thinking units, and triggers the large language model to generate the corresponding thinking segments after each input unit arrives, and finally obtains multiple thinking segments; Step 2, the streaming training unit pre-trains the large language model using an independent position encoding mechanism and a streaming mask mechanism, so that the large language model adapts to the streaming thinking paradigm; Step 3, for the multiple thinking segments output by the streaming thinking data generation unit, the streaming parallel reasoning unit uses the trained large language model for cross-GPU streaming parallel reasoning, specifically: setting input token cache pool, reasoning thinking token cache pool and answer token cache pool respectively, decoupling the input token cache pool and the reasoning thinking token cache pool across GPUs, and realizing cross-GPU concurrent execution of reading and thinking during reasoning.

10. The stream-of-consciousness thinking and reasoning method of a large language model according to claim 9, wherein, The method further comprises quality evaluation, screening or error correction of the thinking segments output by the streaming thinking data generation unit, specifically: using granularity score to evaluate whether the number of input units and reasoning thinking units matches, using sentence-level similarity to evaluate whether the reasoning process is in a streaming, sequential manner, and discarding the data if the granularity evaluation and sequence consistency evaluation fail twice in a row.

Images