A large language model's streaming thinking and reasoning system and method

Abstract

The application relates to the technical field of large language models, in particular to a large language model streaming thinking and reasoning system and method, which comprises a streaming thinking data generation unit, a streaming training unit and a streaming parallel reasoning unit; the streaming thinking data generation unit is used for splitting continuous reasoning data into a plurality of reasoning thinking units to obtain a plurality of thinking segments; the streaming training unit adopts an independent position coding mechanism and a streaming mask mechanism to pre-train a large language model, so that the large language model is adapted to a streaming thinking paradigm; the streaming parallel reasoning unit uses the trained large language model to perform cross-GPU streaming parallel reasoning on the plurality of thinking segments, decouples an input token cache pool and a reasoning thinking token cache pool, and realizes cross-GPU concurrent execution of reading and thinking during reasoning. Compared with the prior art, the application has the advantages of significantly reducing delay and improving user interaction experience.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a streaming thinking and reasoning system and method for large language models. Background Technology

[0002] In recent years, Large Language Models (LLMs) have demonstrated powerful reasoning capabilities in their thought chain reasoning. However, the reasoning paradigm of existing LLMs is based on a "batch processing" model, meaning the model first needs to wait for the user to input complete context or questions before it begins reasoning and thinking. This traditional "waiting for complete input before thinking" model is technically typically divided into two rigid phases:

[0003] Prefill Phase: The model receives and processes all user input tokens and calculates and generates the initial key-value cache in one go.

[0004] And the Decode Phase: Based on the KV Cache generated in the pre-filling phase, the model begins to generate tokens autoregressively.

[0005] This model has the following significant drawbacks:

[0006] 1) High Time To First Token (TTFT): Users must wait for the model to complete the pre-filling computation of all input content before they can see the first output token. For reasoning and thinking tasks with long inputs (such as long document summarization, complex multi-turn dialogue history), the pre-filling stage can be very time-consuming, resulting in extremely high Time To First Token (TTFT).

[0007] 2) Separation of Reading and Thinking: In the traditional model, reading (pre-filling) and thinking (decoding) are completely separated. This means that when the model receives early key information (such as the first few conditions of a complex mathematical problem), it cannot immediately begin generating intermediate inferences or thinking steps, but must wait for all information input to be completed. This does not conform to human cognitive habits and may also lead to insufficient attention to early information by the model when dealing with long-distance dependencies and complex reasoning.

[0008] Existing technologies generally suffer from high inference latency, poor interactive experience, and separation of reading and thinking. There is an urgent need for a new streaming thinking framework that can achieve low latency and high concurrency streaming thinking without sacrificing model performance.

[0009] A search revealed Chinese invention patent application publication number CN120181070A, which discloses a streaming processing system and method for a large language model. In an offline environment, the large language model module calls the large language model architecture to run in batch processing mode. In an online environment, streaming model parameters are used as model parameters for the large language model architecture, which then uses a token-decoupled caching mechanism to achieve streaming inference. Furthermore, a model parameter optimization module employs a streaming attention mask mechanism to optimize the parameters of the large language model architecture, dynamically adjusting its attention scope so that it only focuses on currently available input and context to obtain streaming model parameters. This ensures the large language model module meets the scalability requirements between batch and streaming modes while maintaining low computational cost. However, this existing patent application suffers from issues such as word-level attention masking and a token-decoupled caching mechanism using two sets of serial cache pools, resulting in alternating read and write operations and thus high latency and low concurrency.

[0010] How to achieve low-latency, high-concurrency streaming thinking and reasoning for large language models has become a technical problem that needs to be solved. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a streaming thinking and reasoning system and method for a large language model.

[0012] The objective of this invention can be achieved through the following technical solutions:

[0013] According to a first aspect of the present invention, a streaming thinking and reasoning system for a large language model is provided, the system comprising a streaming thinking data generation unit, a streaming training unit, and a streaming parallel reasoning unit;

[0014] The streaming thinking data generation unit is used to split continuous reasoning data into several reasoning thinking units. After each input unit arrives, the large language model is immediately triggered to generate corresponding thinking segments, and finally multiple thinking segments are obtained.

[0015] The streaming training unit uses an independent positional encoding mechanism and a streaming masking mechanism to pre-train the large language model, enabling the large language model to adapt to the streaming thinking paradigm.

[0016] The streaming parallel inference unit performs cross-GPU streaming parallel inference on multiple thought fragments output by the streaming thinking data generation unit using a trained large language model. Specifically, it sets up an input token cache pool, a reasoning thinking token cache pool, and an answer token cache pool, and decouples the input token cache pool and the reasoning thinking token cache pool across GPUs to achieve cross-GPU concurrent execution of reading and thinking during inference.

[0017] As a preferred technical solution, the process of splitting continuous reasoning data into several reasoning units includes:

[0018] Step 1.1: Input sentences for streaming. , Insert sentence-level boundary markers <eos>To define the smallest unit of reasoning and thinking;

[0019] Step 1.2 guides the large language model to generate inferences about the order of sentences generated from the streaming input, when encountering... <eos>Terminate this step and add... <eot>mark;

[0020] Step 1.3 introduces the teacher's large language model to reconstruct the reasoning steps in Step 1.2, and obtains the reasoning and thinking content corresponding to each reasoning and thinking unit;

[0021] Step 1.4 guides the large language model to summarize and reflect on the thinking process, adjusts the depth of thinking according to the different difficulty of the task, and generates global reasoning content.

[0022] As a preferred technical solution, the cross-GPU streaming parallel inference process includes:

[0023] Initialize the input token cache pool, the reasoning token cache pool, and the answer token cache pool;

[0024] The steps for updating the input token cache pool are as follows: On the first GPU device, the tokens of the input inference unit are used to perform attention calculation with the input cache pool, and the first attention result is updated to the input token cache pool.

[0025] The steps for updating the inference token cache pool are as follows: On the second GPU device, if the current input inference unit has corresponding inference content, wait for new input; otherwise, perform attention calculation between the token in the inference token cache pool and the updated input token cache pool, and update the obtained second attention result to the inference token cache pool.

[0026] The steps of updating the input token cache pool and updating the reasoning token cache pool are repeated in parallel until the input content and reasoning content are no longer updated.

[0027] The input token cache pool and the reasoning token cache pool are merged to obtain a merged cache pool. Attention is calculated between the answer token cache pool and the merged cache pool, and the obtained third attention result is updated to the answer token cache pool.

[0028] As a preferred technical solution, the streaming masking mechanism is based on the input sentence and ensures, through dynamic time constraints, that each step in streaming reasoning relies only on historical tokens and generated thought content.

[0029] As a preferred technical solution, the streaming masking mechanism specifically includes:

[0030] Initialize the streaming attention mask matrix as follows: ,in and These are the token lengths for the input content and the reasoning / thinking content, respectively;

[0031] For the attention mask matrix is any position in ,if Then the corresponding position will be covered. ;

[0032] In attention mask matrix Blocking from the thinking steps t When the input sentence number is greater than t Attention is drawn to obtain the streaming mask matrix.

[0033] As a preferred technical solution, the independent location coding mechanism is specifically as follows:

[0034] The streaming training data is mapped through a linear layer to obtain the query matrix Q, the key matrix K, and the value matrix V;

[0035] For length of n Input sentence ， Its token sequence has a length of The data is assigned location numbers. The corresponding reasoning sequence is: Its token sequence is of length The data is assigned location numbers as follows: ;

[0036] Define rotation matrix ,in It is the dimension of the model. It represents the position number, and diag is the diagonal block matrix. ,in i The model represents the first i One dimension, Indicates the corresponding rotation angle;

[0037] For any element of the streaming training data Its location information is encoded as QR(m) and KR(m).

[0038] As a preferred technical solution, the system further includes a quality control unit, which is used to evaluate, filter or correct the quality of the thought fragments generated by the streaming thought data generation unit, and output the results to the streaming parallel inference unit for parallel inference.

[0039] As a preferred technical solution, the quality control unit uses granular score to evaluate whether the number of input units and thought segments match, and uses sentence-level similarity to evaluate whether the reasoning process is performed in a streaming, sequential manner.

[0040] If a data fails either the granularity assessment or the sequence consistency assessment twice consecutively, it is discarded.

[0041] According to a second aspect of the present invention, a streaming thinking and reasoning method for a large language model is provided, the method being applicable to the streaming thinking and reasoning system for a large language model described in the first aspect and various possible technical solutions of the first aspect, the method comprising:

[0042] Step 1: The streaming thinking data generation unit breaks down continuous reasoning data into several reasoning thinking units. After each input unit arrives, the large language model is immediately triggered to generate the corresponding thinking fragments, ultimately resulting in multiple thinking fragments.

[0043] Step 2: The streaming training unit uses an independent positional encoding mechanism and a streaming masking mechanism to pre-train the large language model, enabling the large language model to adapt to the streaming thinking paradigm.

[0044] Step 3: For the multiple thought fragments output by the streaming thinking data generation unit, the streaming parallel inference unit uses the trained large language model to perform cross-GPU streaming parallel inference. Specifically, it sets up an input token cache pool, an inference thinking token cache pool, and an answer token cache pool respectively, decouples the input token cache pool and the inference thinking token cache pool across GPUs, and realizes cross-GPU concurrent execution of reading and thinking during inference.

[0045] As a preferred technical solution, the method further includes quality assessment, screening, or error correction of the thinking segments output by the streaming thinking data generation unit. Specifically, it uses granularity score to assess whether the number of input units and reasoning thinking units matches, and uses sentence-level similarity to assess whether the reasoning process is performed in a streaming, sequential manner. If the granularity assessment and sequence consistency assessment fail twice consecutively, the data is discarded.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) This invention generates intermediate thought segments corresponding to the input unit in parallel while accepting the input unit. This streaming thinking paradigm of the large language model enables reading and thinking to proceed in parallel, without having to wait for complete input before starting to think, significantly reducing latency and improving the user interaction experience.

[0048] 2) This invention uses independent positional encoding and a streaming model-specific masking mechanism to train large language models in streaming scenarios, solving the problems of positional information confusion and streaming constraints caused by the concurrency of input stream and thought stream, enabling large language models to adapt to streaming thinking paradigms and maintain reasoning performance with minimal degradation in tasks such as reasoning.

[0049] 3) The streaming parallel inference mechanism of this invention improves early attention, forcing large language models to generate thought fragments almost immediately upon receiving early input information, reducing the number of input tokens waiting to start inference by about 80%, and reducing the overall response latency by more than 60%. Therefore, it significantly reduces inference latency, improves parallel efficiency, cache resource utilization, and user interaction experience.

[0050] 4) The quality control unit of the present invention ensures the quality of streaming data through fine-grained evaluation and sequence consistency evaluation, prevents the accumulation of low-quality or speculative thought fragments generated by the large language model when receiving incomplete input, and ensures the accuracy of the final output. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the flow-based thinking and reasoning system in this invention;

[0052] Figure 2 This is a flowchart illustrating the flow-based thinking and reasoning method in this invention;

[0053] Figure 3 This is a schematic diagram comparing the principles of streaming thinking and reasoning with batch processing in this invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] This invention aims to solve the problems in existing large language model reasoning, such as waiting for complete input before starting reasoning or thinking, which leads to high first-word delay (TTFT), poor interactive experience, and separation of reading and thinking.

[0056] This embodiment relates to a streaming thinking and reasoning system for a large language model, proposing a streaming-thinking paradigm that transforms the thinking process of a large language model from the traditional mode of waiting for complete input before thinking to a mode of reading and thinking in parallel.

[0057] like Figure 1 The system includes: a streaming thinking data generation unit, a quality control unit, a streaming training unit, and a streaming parallel inference unit. Through the collaborative work of these modules, this invention significantly reduces the latency of thinking initiation and enhances the attention of large language models to early input information while maintaining essentially no reduction in thinking performance.

[0058] The streaming thinking data generation unit is used to break down continuous reasoning data into several reasoning chunks. Upon arrival of each input chunk, the large language model is immediately triggered to generate corresponding thought fragments; and after all reasoning chunks have been completed, deep thinking is performed.

[0059] The quality control unit is used to assess, filter, or correct the quality of thought fragments generated by the streaming thinking data generation unit. This module prevents the accumulation of low-quality or speculative thought fragments generated by the model when receiving incomplete input, ensuring the accuracy of the final output.

[0060] The streaming training unit is used to adapt the pre-trained LLM to the aforementioned streaming thinking paradigm, and it includes two key technical points: an independent positional encoding mechanism and a streaming masking mechanism. The independent positional encoding mechanism is used to solve the problem of positional information confusion caused by the concurrency of input stream and thought stream, while the streaming masking mechanism is used to solve the streaming constraint problem caused by the concurrency of input stream and thought stream.

[0061] The streaming parallel reasoning unit adopts a parallel inference mechanism to achieve concurrent execution of reading and thinking during reasoning. Its core lies in decoupling the key-value cache generated by the input token and the reasoning token.

[0062] Furthermore, the process by which the streaming thinking data generation unit generates streaming reasoning content includes:

[0063] (1.1) First, input the sentence for streaming. , Insert sentence-level boundary markers <eos>To define the smallest unit of reasoning and thinking;

[0064] (1.2) Guide the LLM to generate sequential reasoning for the preceding sentences, when encountering... <eos>Terminate this step and add... <eot>mark;

[0065] (1.3) Introduce a more powerful teacher LLM reconstruction (1.2) reasoning steps to obtain the reasoning and thinking content corresponding to each reasoning and thinking unit. .

[0066] (1.4) After obtaining the content of sequential reasoning, guide LLM to summarize and reflect on the thinking process, adjust the depth of thinking according to the different difficulty of the task, and generate global reasoning thinking content.

[0067] Furthermore, the evaluation of the quality control unit is achieved through the following sub-steps:

[0068] (2.1) Granularity evaluation: A granularity score is defined to evaluate whether the number of input units and thought fragments match. It is defined as the number of boundary markers in the input. The number of boundary markers in the output The ratio: ;

[0069] (2.2) Sequence Consistency Assessment: A sequence consistency score is defined using sentence-level similarity to assess whether the reasoning process proceeds in a streaming, order-preserving manner. The similarity is calculated by the SentenceBERT model; As a reference sequence, Candidate sequences are the inference segments output by the model in real time.

[0070] (2.3) Repeat the generation process of step one. If the granularity evaluation or sequence consistency evaluation fails twice in a row, the data is discarded.

[0071] Furthermore, the streaming training mechanism unit processes the streaming training data through the following sub-steps:

[0072] (3.1) The streaming training data is mapped through a linear layer to obtain Q, K and V (corresponding to Query, Key and Value);

[0073] (3.2) Introduce an independent position encoding mechanism: for lengths of n Input sentence , ， Its token sequence has a length of Data Assign position numbers as The corresponding reasoning sequence is: Its token sequence is of length Data The location numbers are assigned as follows: Define the rotation matrix. ,in It is the dimension of the model. It is the position number, and diag is the diagonal block matrix; ,in i The model represents the first i There are three dimensions, i = 1, 2, ... ; This represents the corresponding rotation angle; for training data any element Its location information can be encoded as QR(m) and KR(m);

[0074] (3.3) Introducing a sentence-level streaming masking mechanism: Initializing the streaming attention mask matrix as follows ,in and These are the token lengths for the input content and the reasoning content, respectively; for the attention mask matrix... any position in ,if Then the corresponding position will be covered. In the matrix Blocking from the thinking steps t The input sentence is greater than t Attention is used to obtain the streaming mask matrix: Where T is the length of the input data token. Represents a number that is infinitely large. It's an indicator function; values ​​that don't meet the conditions are assigned 0. When the condition in the indicator function is satisfied... For an infinite value, the streaming mask matrix Cover the original M ( i , j This prevents the location from being considered; when the condition in the indicator function is not met, 0, streaming mask matrix equal to the original M ( i , j This means that only historical tokens are allowed to be considered. Dynamic time constraints ensure that each step in streaming reasoning relies solely on historical tokens, preventing premature disclosure of future information from disrupting the reasoning logic.

[0075] (3.4) For streaming training data, design an attention score calculation formula to achieve attention forward propagation: , where d is the algorithm dimension of the LLM model architecture.

[0076] This invention overcomes the quality degradation of streaming processing through a streaming training mechanism: independent positional encoding ensures that the input stream and the thought stream still have conflict-free relative and absolute positional awareness when running in parallel, solving the positional confusion problem; sentence-level streaming attention masking forces large language models to follow logical information flow, ensuring causal relationships, preventing large language models from "look-ahead" cheating, and solving the problem of mismatch with pre-training.

[0077] Furthermore, the streaming parallel inference of the streaming parallel inference unit is implemented through the following sub-steps:

[0078] (4.1) Initialize the input token cache pool Reasoning token cache pool and answer token cache pool ;

[0079] (4.2) On the first GPU device, the token of the input inference unit is compared with the input cache pool. Perform attention calculations and update the initial attention result to... ;

[0080] (4.3) On the second GPU device, if the current input inference unit has corresponding inference content, it waits for new input; otherwise, it retrieves the inference token cache. The token and the updated Perform attention calculations and update the obtained second attention result to ;

[0081] (4.4) Repeat steps (4.2) and (4.3) in parallel until the input content and reasoning content are no longer updated; continuously reuse the existing KV cache to avoid redundant calculations;

[0082] (4.5) Merging and Get the merged cache pool and the answer token cache pool. Attention calculations are performed with the merged cache pool, and the obtained third attention result is updated to... .

[0083] The streaming parallel inference unit adopts a three-group cache pool design across GPUs. The large language model can think in parallel while being read in, and write the answer after thinking is completed. This is a multi-GPU parallel read-write synchronous streaming, which is different from the existing single-GPU serial read-write alternating streaming. By using multiple cache pools and multi-GPU parallel read-write, it can process input calculation, inference calculation and answer generation at the same time, thus improving parallel efficiency and cache resource utilization. It can output intermediate results and final answers faster and improve the user interaction experience.

[0084] Figure 3 This invention demonstrates the streaming parallel inference mechanism ( Figure 3 The upper part describes the rigid pre-filling-decoding two-stage process in the background art. Figure 3 Decoupling the left part of the language model allows large language models to begin generating thought fragments almost immediately after receiving the first input unit, without waiting for complete input. This reduces the number of input tokens (TTFT) that inference begins to wait for by approximately 80%, resulting in an overall response latency reduction of over 60%. Consequently, inference latency is significantly reduced, enhancing the user's interactive experience.

[0085] The streaming thinking paradigm of this invention forces large language models to immediately produce intermediate thinking fragments corresponding to the input units in parallel upon receiving early inputs. This enhances the model's attention to early input information, and the generated thinking fragments provide intermediate steps for the large language model's final decision, thereby enhancing the interpretability of the reasoning process.

[0086] This embodiment also relates to a streaming thinking and reasoning method for large language models, such as Figure 2 The method includes the following steps:

[0087] Step 1: The streaming thinking data generation unit adopts the streaming thinking paradigm of the large language model. By using marker symbols, continuous reasoning data is divided into several reasoning thinking units, and global reasoning thinking content is generated. Specifically, the smallest reasoning thinking unit is defined. Upon receiving each input unit, the large language model is immediately triggered to generate the corresponding thinking fragment, reconstructing multiple reasoning thinking units and generating global reasoning thinking content.

[0088] Step 2 involves quality assessment, screening, or error correction of the thinking fragments output by the streaming thinking data generation unit. Specifically, granularity scores are used to assess whether the number of input units and reasoning thinking units matches, and sentence-level similarity is used to assess whether the reasoning process is conducted in a streaming, sequential manner. If at least one of the granularity assessment or sequence consistency assessment fails twice consecutively, the data is discarded.

[0089] Step 3: The streaming training data is mapped through a linear layer to obtain Q, K, and V. The streaming input sentences and reasoning content are independently encoded at their positions. A streaming masking mechanism is used to train the large language model in a streaming scenario. Through dynamic time constraints, it is ensured that each step in streaming reasoning relies only on historical tokens to avoid premature leakage of future information that could disrupt the reasoning logic.

[0090] Step 4, Streaming Parallel Inference: The streaming parallel inference unit separates the token cache pool and the inference think token cache pool, and enables parallel reading and inference across GPUs through streaming parallel inference. Specifically:

[0091] Step 4.1, Initialize the input token cache pool Reasoning token cache pool and answer token cache pool ;

[0092] Step 4.2, on the first GPU device, the token of the input inference unit is compared with the input cache pool. Perform attention calculations and update the initial attention result to... ;

[0093] Step 4.3: On the second GPU device, if the currently input inference unit has corresponding inference content, wait for new input; otherwise, retrieve the inference token cache pool. The token and the updated Perform attention calculations and update the obtained second attention result to ;

[0094] Step 4.4, repeat steps 4.2 and 4.3 in parallel until the input content and reasoning content are no longer updated;

[0095] Step 4.5, Merge and Get the merged cache pool and the answer token cache pool. Attention calculations are performed with the merged cache pool, and the obtained third attention result is updated to... .

[0096] To verify the beneficial effects of this invention, it was validated on a mathematical reasoning task. Compared with batch-thinking and serial interleaved-thinking models, the parallel streaming-thinking of this invention achieves the best performance in both accuracy (Acc) and latency (TTFT and Delay), as shown in Table 1.

[0097] Table 1

[0098]

[0099] Among them, GSM-Symbolic and MetaMathQA are two different datasets for mathematical reasoning tasks, and D1~D3 represent different global reasoning depths after sequential reasoning.

[0100] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0101] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] The processing unit performs the various methods and processes described above. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the methods by any other suitable means (e.g., by means of firmware).

[0103] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0104] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / eot> < / eos> < / eos> < / eot> < / eos> < / eos>

Claims

1. A streaming thinking and reasoning system based on a large language model, characterized in that, The system includes a streaming thinking data generation unit, a streaming training unit, and a streaming parallel inference unit; The streaming thinking data generation unit is used to split continuous reasoning data into several reasoning thinking units. After each input unit arrives, the large language model is immediately triggered to generate corresponding thinking segments, and finally multiple thinking segments are obtained. The streaming training unit uses an independent positional encoding mechanism and a streaming masking mechanism to pre-train the large language model, enabling the large language model to adapt to the streaming thinking paradigm. The streaming parallel inference unit performs cross-GPU streaming parallel inference on multiple thought fragments output by the streaming thinking data generation unit using a trained large language model. Specifically, it sets up an input token cache pool, a reasoning thinking token cache pool, and an answer token cache pool, and decouples the input token cache pool and the reasoning thinking token cache pool across GPUs to achieve cross-GPU concurrent execution of reading and thinking during inference. The cross-GPU streaming parallel inference process includes: Initialize the input token cache pool, the reasoning token cache pool, and the answer token cache pool; The steps for updating the input token cache pool are as follows: On the first GPU device, the tokens of the input inference unit are used to perform attention calculation with the input cache pool, and the first attention result is updated to the input token cache pool. The steps for updating the inference token cache pool are as follows: On the second GPU device, if the current input inference unit has corresponding inference content, wait for new input; otherwise, perform attention calculation between the token in the inference token cache pool and the updated input token cache pool, and update the obtained second attention result to the inference token cache pool. The steps of updating the input token cache pool and updating the reasoning token cache pool are repeated in parallel until the input content and reasoning content are no longer updated. The input token cache pool and the reasoning token cache pool are merged to obtain a merged cache pool. Attention is calculated between the answer token cache pool and the merged cache pool, and the obtained third attention result is updated to the answer token cache pool.

2. The streaming thinking and reasoning system based on a large language model according to claim 1, characterized in that, The process of splitting continuous reasoning data into several reasoning units includes: Step 1.1: Input sentences for streaming. , Insert sentence-level boundary markers <eos> To define the smallest unit of reasoning and thinking;< / eos> Step 1.2 guides the large language model to generate inferences about the order of sentences generated from the streaming input, when encountering... <eos>Terminate this step and add... <eot> mark;< / eot> < / eos> Step 1.3 introduces the teacher's large language model to reconstruct the reasoning steps in Step 1.2, and obtains the reasoning and thinking content corresponding to each reasoning and thinking unit; Step 1.4 guides the large language model to summarize and reflect on the thinking process, adjusts the depth of thinking according to the different difficulty of the task, and generates global reasoning content.

3. The streaming thinking and reasoning system based on a large language model according to claim 1, characterized in that, The streaming masking mechanism is based on the input sentence and ensures, through dynamic time constraints, that each step in streaming reasoning relies only on historical tokens and generated thought content.

4. The streaming thinking and reasoning system based on a large language model according to claim 1, characterized in that, The independent location encoding mechanism is specifically as follows: The streaming training data is mapped through a linear layer to obtain the query matrix Q, the key matrix K, and the value matrix V; For length of n Input sentence ， Its token sequence has a length of The data is assigned location numbers. The corresponding reasoning sequence is: Its token sequence is of length The data is assigned location numbers as follows: ; Define rotation matrix ,in It is the dimension of the model. It represents the position number, and diag is the diagonal block matrix. ,in i The model represents the first i One dimension, Indicates the corresponding rotation angle; For any element of the streaming training data Its location information is encoded as QR(m) and KR(m).

5. The streaming thinking and reasoning system based on a large language model according to claim 1, characterized in that, The system also includes a quality control unit, which is used to evaluate, filter or correct the quality of the thought fragments generated by the streaming thought data generation unit, and output the results to the streaming parallel inference unit for parallel inference.

6. The streaming thinking and reasoning system based on a large language model according to claim 5, characterized in that, The quality control unit uses granular score to evaluate whether the number of input units and thought fragments match, and uses sentence-level similarity to evaluate whether the reasoning process is performed in a streaming, sequential manner. If a data fails either the granularity assessment or the sequence consistency assessment twice consecutively, it is discarded.

7. A streaming thinking and reasoning method based on a large language model, characterized in that, The method is applicable to a streaming thinking and reasoning system based on a large language model as described in any one of claims 1 to 6, and the method includes: Step 1: The streaming thinking data generation unit breaks down continuous reasoning data into several reasoning thinking units. After each input unit arrives, the large language model is immediately triggered to generate the corresponding thinking fragments, ultimately resulting in multiple thinking fragments. Step 2: The streaming training unit uses an independent positional encoding mechanism and a streaming masking mechanism to pre-train the large language model, enabling the large language model to adapt to the streaming thinking paradigm. Step 3: For the multiple thought fragments output by the streaming thinking data generation unit, the streaming parallel inference unit uses the trained large language model to perform cross-GPU streaming parallel inference. Specifically, it sets up an input token cache pool, an inference thinking token cache pool, and an answer token cache pool respectively, decouples the input token cache pool and the inference thinking token cache pool across GPUs, and realizes cross-GPU concurrent execution of reading and thinking during inference.

8. The streaming thinking and reasoning method for a large language model according to claim 7, characterized in that, The method also includes quality assessment, screening, or error correction of the thinking segments output by the streaming thinking data generation unit. Specifically, it uses granularity score to assess whether the number of input units and reasoning thinking units matches, and uses sentence-level similarity to assess whether the reasoning process is performed in a streaming, sequential manner. If the granularity assessment and sequence consistency assessment fail twice consecutively, the data is discarded.

Images