Model reasoning method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a model reasoning method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: performing attention calculation on inferred tokens by adopting first numerical precision through a large language model to obtain a first attention score corresponding to each inferred token; screening a target tokens from the inferred tokens based on the first attention score; reasoning the input sequence by adopting second numerical precision through the large language model to obtain a reasoning result output by the large language model; the input sequence comprises a target token and tokens to be input corresponding to the token to be inferred; the tokens to be input are tokens pre-selected from the inferred tokens according to a preset rule; the first numerical precision is lower than the second numerical precision. According to the method, the sparse processing of the token is realized by adopting mixed precision calculation, so that the reasoning efficiency of the large language model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model reasoning, in particular to a model reasoning method and device, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the enhancement of the long text understanding ability of the large language model, its application in long text input is also increasing, such as question and answer for single or multiple document contents, summary for long articles, etc. However, the large language model reasoning in the long text input scenario often requires a large amount of calculation time. For a Qwen2 model (thousand question model) containing 7 billion parameters, if a user inputs a document with a length of 18000 tokens and asks a question at a time, the large language model needs to calculate 3000 milliseconds to generate a complete reply. In the prior art, in order to improve the processing speed of long text, the text processing process is usually optimized in three aspects of reducing memory occupation, accelerating decoding process, and enhancing the generalization ability of the model to long text.

[0003] Then, the reasoning speed of the text reasoning method in the prior art still needs to be improved. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a model reasoning method, device, electronic device, storage medium and program product to improve the reasoning efficiency of the large language model.

[0005] In a first aspect, the embodiments of the present application provide a model reasoning method, comprising: Through the large language model, the attention calculation of the already-reasoned tokens is performed using a first numerical precision to obtain a first attention score corresponding to each already-reasoned token; Target tokens are selected from the already-reasoned tokens based on the first attention score; Through the large language model, the input sequence is reasoned using a second numerical precision to obtain a reasoning result output by the large language model; the input sequence includes the target tokens and the to-be-input tokens corresponding to the to-be-reasoned tokens; the to-be-input tokens are tokens preselected from the already-reasoned tokens according to a preset rule; the first numerical precision is lower than the second numerical precision.

[0006] The embodiment of the application adopts a first numerical precision to perform attention calculation on the reasoned tokens, filters target tokens based on the first attention scores of each reasoned token obtained through calculation, and inputs the target tokens and the to-be-input tokens in the sliding window corresponding to the to-be-reasoned tokens into the large language model to perform reasoning in a second numerical precision. This hybrid precision calculation realizes the sparse processing of tokens, thereby improving the reasoning efficiency of the large language model.

[0007] In a possible implementation of the first aspect, the target tokens are filtered from the reasoned tokens based on the first attention scores, including: reasoned tokens corresponding to the first attention scores greater than a preset threshold are filtered as the target tokens; or, a preset number of reasoned tokens are selected as the target tokens from large to small based on the first attention scores; or, the target tokens are filtered from the reasoned tokens based on the first attention scores by using a reinforcement learning algorithm.

[0008] In the embodiment of the application, since the first attention scores reflect the relevance and importance of the reasoned tokens to the current reasoning context, the valuable tokens can be selected to participate in the subsequent reasoning by selecting the reasoned tokens with a suitable preset threshold, a preset number of larger first attention scores, and using a reinforcement learning algorithm to select the target tokens, thereby improving the accuracy of the reasoning of the large language model.

[0009] In a possible implementation of the first aspect, the preset threshold is determined by the following method: context information entropy is calculated based on the reasoned tokens; the preset threshold is determined based on the context information entropy and the sequence length of the reasoned tokens.

[0010] In the embodiment of the application, the context information entropy is calculated based on the reasoned tokens, and the preset threshold is determined based on the context information entropy and the sequence length of the reasoned tokens. This dynamic threshold mode avoids the problem of too strict / lenient filtering caused by fixation.

[0011] In a possible implementation of the first aspect, the attention calculation on the reasoned tokens is performed by the large language model in a first numerical precision, including: the large language model is run by the first computing unit in the GPU to perform the attention calculation on the reasoned tokens in the first numerical precision; the first numerical precision includes FP4 and INT4; The input sequence is inferred by the large language model using the second numerical precision, including: The large language model is run by the second computing unit in the GPU to infer the input sequence using the second numerical precision; the second numerical precision includes FP16 and FP32.

[0012] The embodiments of the present application cooperate with heterogeneous hardware through mixed precision calculation, realize efficient token sparsification processing without introducing additional parameterized compression modules, and improve the calculation efficiency and resource utilization of the large language model in the long context sequence inference scenario.

[0013] In a possible implementation manner of the first aspect, the attention calculation process performed by the first computing unit using the first numerical precision and the attention calculation process performed by the second computing unit using the second numerical precision are executed in parallel.

[0014] The embodiments of the present application separate the calculation tasks of the first computing unit and the second computing unit through asynchronous task scheduling, so that the two computing units work in parallel, and the overall throughput efficiency is improved.

[0015] In a possible implementation manner of the first aspect, the target tokens are filtered from the inferred tokens based on the first attention scores, including: The intermediate tokens are filtered from the inferred tokens based on the first attention scores. The attention calculation is performed on the intermediate tokens using the third numerical precision to obtain the second attention scores corresponding to the intermediate tokens, respectively. The target tokens are filtered from the intermediate tokens based on the second attention scores.

[0016] The embodiments of the present application maximize the calculation efficiency while ensuring the inference accuracy of the large language model by establishing a multi-level precision cooperative pipeline.

[0017] In a possible implementation manner of the first aspect, the to-be-input tokens include the first preset number of inferred tokens closest to the to-be-inferred tokens and the first second preset number of tokens in the inferred tokens.

[0018] In the embodiments of the present application, the inferred tokens closest to the to-be-inferred tokens have the greatest influence on the to-be-inferred tokens, and the first tokens in the sequence of inferred tokens also have a greater influence on the to-be-inferred tokens. Therefore, these two parts of tokens are used as to-be-input tokens for inference on the to-be-inferred tokens, which improves the inference accuracy of the large language model.

[0019] In a second aspect, an embodiment of the present application provides a model inference device, comprising: The attention calculation module is configured to perform attention calculation on the inferred tokens by the large language model using a first numerical precision to obtain a first attention score corresponding to each inferred token. The screening module is configured to screen target tokens from the inferred tokens based on the first attention scores. The inference module is configured to perform inference on the input sequence by the large language model using a second numerical precision to obtain an inference result output by the large language model; the input sequence comprises the target tokens and to-be-input tokens corresponding to the to-be-inferred tokens; the to-be-input tokens are tokens preselected from the inferred tokens according to a preset rule; the first numerical precision is lower than the second numerical precision.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory and a bus, wherein: The processor and the memory complete mutual communication through the bus. The memory stores program instructions executable by the processor, and the processor invoking the program instructions can execute the method of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a non-transitory computer readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions make the computer execute the method in each possible implementation manner of the first aspect.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which are read and run by a processor to execute the method in each possible implementation manner of the first aspect.

[0023] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be understood through implementation of the embodiments of the present application. The purpose and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0025] Figure 1 A model inference method flowchart provided by the embodiments of the present application; Figure 2 A model inference device structure schematic diagram provided by the embodiments of the present application; Figure 3 An electronic device physical structure schematic diagram provided by the embodiments of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the technical solutions of the present application will be described in detail below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion.

[0028] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is more than two, unless otherwise explicitly and specifically limited.

[0029] In this paper, the reference to "embodiments" means that the specific features, structures or properties described in conjunction with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] In the description of the embodiments of the present application, the term "and / or" is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0031] In the description of the embodiments of the present application, the term "a plurality of" refers to two or more (including two), and similarly, "a plurality of groups" refers to two or more groups (including two groups), and "a plurality of pieces" refers to two or more pieces (including two pieces).

[0032] In the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the technical terms "mounting", "connecting", "connecting", "fixing" and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanical connection, or it can be electrical connection; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0033] Large language models are complex deep learning models in the field of artificial intelligence, mainly used for processing natural language related tasks. Large language models are usually based on neural network architectures such as the Transformer architecture. The Transformer architecture can process sequence data in parallel through the self-attention (self-attention) mechanism, and can effectively capture long-distance dependencies between different positions in the sequence. This architecture enables the model to understand the long-distance associations between words or phrases in the text, such as in understanding the semantic connection between the beginning and end of a long sentence. Large language models are trained on a large amount of text data, which comes from a wide range of sources, including books, web pages, articles, etc. During the training process, the model learns the statistical laws of language, including the co-occurrence relationship of words, grammatical structure, and semantic information, etc. For example, by analyzing a large amount of text, the model can understand that "dog" and "tail" are often a belonging relationship, so it can correctly handle such relationships when generating or understanding text.

[0034] With the increasing demand for long context processing of large language models (LLMs), traditional full attention mechanisms face significant challenges in terms of computational time and memory consumption. To address this issue, sparse attention mechanisms have become a research hotspot. Sparse attention reduces computational redundancy, improves model efficiency, and maintains stable performance. DeepSeek and Kimi have introduced NSA and MoBA, respectively, aiming to further optimize sparse attention mechanisms to address the efficiency bottleneck in long text processing.

[0035] Among them, DeepSeek proposes NSA (Novel Sparse Attention), which optimizes end-to-end training and inference through hierarchical sparse strategies and hardware-friendly design; Kimi introduces MoBA (Mixture of Block Attention), which combines the advantages of existing tokens eviction strategies and sparse attention, providing important support for the practical application of large language models in long context tasks.

[0036] DeepSeek's NSA introduces a small neural network for token compression, which to some extent alleviates the computational pressure brought by long sequences. However, this reliance on additional parameterized modules increases model complexity and deployment costs.

[0037] The embodiments of the present application propose a model inference method that fully utilizes existing hardware resources without the need for additional small models to improve the inference efficiency of large language models. The specific method is as follows: for a certain token to be inferred by the large language model, the large language model uses a first numerical precision to perform attention calculation on the already inferred tokens, obtaining a first attention score corresponding to each already inferred token, and selecting target tokens from the already inferred tokens based on the first attention score; then, the target tokens and the to-be-input tokens corresponding to the to-be-inferred token are merged as the input of the large language model, and the large language model performs inference based on a second numerical precision to obtain an inference result. On the one hand, the inference efficiency of the large language model is improved, and on the other hand, the inference accuracy of the large language model is guaranteed.

[0038] The model inference method provided by the embodiments of the present application will be described in detail below.

[0039] It can be understood that the model inference method provided in the embodiments of the present application can be applied to an electronic device, which includes a terminal and a server; the terminal can be specifically a smartphone, a tablet computer, a computer, a personal digital assistant (PDA), etc.; and the server can be specifically an application server or a Web server.

[0040] Figure 1 A flowchart of a model inference method provided in the embodiments of the present application is shown in FIG. 1, which includes the following steps. Figure 1 Step 101: performing attention calculation on the already-inferred tokens by using a first numerical precision through a large language model to obtain a first attention score corresponding to each already-inferred token; Step 102: filtering target tokens from the already-inferred tokens based on the first attention score; Step 103: performing inference on an input sequence by using a second numerical precision through the large language model to obtain an inference result output by the large language model; the input sequence includes the target tokens and to-be-inferred tokens; the to-be-inferred tokens are tokens preselected from the already-inferred tokens according to a preset rule; and the first numerical precision is lower than the second numerical precision.

[0041] In the specific implementation process, the numerical precision refers to how many binary bits are used to represent numerical values in a computer, which can reflect the representation range and precision of data. It mainly includes integer precision (for example: int, etc.) and floating-point precision (for example: FP32, FP64, etc.). In the embodiments of the present application, the first numerical precision is lower than the second numerical precision. It should be noted that the specific values of the first numerical precision and the second numerical precision used by the large language model can be preconfigured or can be adaptively adjusted according to the computer hardware environment. For example, when the current load of the computer hardware is large, the first numerical precision and / or the second numerical precision can be appropriately reduced; when the current load of the computer hardware is small, the first numerical precision and / or the second numerical precision can be appropriately increased.

[0042] In the large language model, the input text will be converted into a vector form. Usually, each word or phrase will be mapped to a fixed-dimensional vector, called word embedding. These word embedding vectors contain semantic information of words and can capture the similarity between words. In attention calculation, the input word embedding vector will be converted into a query vector, a key vector and a value vector. The query vector can be regarded as the information to be focused on at present, the key vector is the reference information used to match with other vectors, and the value vector is the information content to be finally obtained.​

[0043] The attention calculation in the large language model mainly includes dot product calculation of attention score, scaling operation, softmax normalization, weighted summation, etc., and finally obtains the attention score. In the Transformer architecture, the self-attention mechanism is a core component. The output of each word depends not only on its own input but also on the input of all other words. In the encoder, the self-attention mechanism enables each word to establish a connection with other words, thereby capturing long-distance dependencies between words in a sentence.

[0044] The reasoned tokens include an original sequence and tokens that have been reasoned by the large language model in the reasoning process. The original sequence refers to the sequence of the content input by the user to the large language model after being serialized. Therefore, the reasoned tokens include a plurality of reasoned tokens.

[0045] The embodiments of the present application use the large language model to perform attention calculation on the reasoned tokens, and use a first numerical precision for calculation. The first numerical precision usually refers to relatively low numerical precision, such as 16-bit floating point (FP16) or lower.

[0046] When the large language model performs attention calculation, it takes the reasoned tokens as input, and through the attention mechanism inside the model, calculates the correlation weight between each reasoned token and other tokens, that is, the first attention score.

[0047] This calculation process is a key step for the model to understand the semantic relationship between the reasoned tokens. By calculating with lower numerical precision, the calculation amount and resource consumption can be greatly reduced under the premise of ensuring a certain calculation precision, and the calculation speed can be improved, providing a basis for the subsequent screening step.

[0048] After obtaining the first attention score, the target tokens are screened based on the first attention score. There can be various screening methods, for example, the reasoned tokens corresponding to the first attention score greater than a preset threshold can be taken as the target tokens; or a preset number of reasoned tokens can be selected from large to small as the target tokens; or a reinforcement learning algorithm can be used to screen the target tokens, etc.

[0049] The purpose of screening the target tokens is to extract the part of tokens that are most critical to the current reasoning task from the reasoned tokens, so that in the subsequent reasoning process, more focus can be placed on these important information, thereby improving the reasoning efficiency and the accuracy of the results.

[0050] The input sequence is inferred by the large language model using a second numerical precision. The second numerical precision is higher than the first numerical precision, and is usually 32-bit floating point (FP32) or higher. During inference, a block matrix multiplication strategy can be used.

[0051] The input sequence includes two parts: the selected target tokens and the to-be-input tokens corresponding to the to-be-inferred tokens. The to-be-input tokens are preselected from the already-inferred tokens according to a preset rule. These preselected tokens can be selected based on some simple rules, such as selecting in order or selecting at fixed intervals.

[0052] In this stage, the input sequence is inferred using high-precision numbers, which ensures accurate processing of key information and thus obtains more accurate and reliable large language model output inference results.

[0053] The embodiments of the present application use low-precision numbers in the attention calculation stage to quickly select target tokens, reducing the computational load and resource consumption in the subsequent high-precision inference stage and improving the inference speed. At the same time, the combination of the target tokens and the to-be-input tokens is inferred with high precision, which ensures accurate processing of key information and ensures the accuracy and reliability of the inference results. This combination of low-precision and high-precision calculations ensures the performance of model inference while optimizing resource utilization and inference efficiency, and is particularly suitable for long-context inference scenarios.

[0054] On the basis of the above embodiments, the target tokens are selected from the already-inferred tokens based on the first attention scores, including: The already-inferred tokens corresponding to the first attention scores greater than a preset threshold are selected as the target tokens; or, The target tokens are selected as the target tokens based on the first attention scores from large to small; or, The target tokens are selected from the already-inferred tokens based on the first attention scores using a reinforcement learning algorithm.

[0055] In the specific implementation process, the first attention score reflects the relevance and importance of the token to the current inference context, and a higher first attention score means that the token is more likely to have a key impact on subsequent inference. Therefore, the embodiments of the present application provide a variety of methods for selecting target tokens, which will be introduced below for each method: (1) The preset threshold value can be a fixed value determined based on experience or a large number of experiments, or can be dynamically determined based on context information entropy and sequence length and other factors. After the first attention score is calculated, the reasoned tokens corresponding to the first attention score greater than the preset threshold value are taken as the target tokens.

[0056] (2) The first attention scores can be sorted according to their sizes, and the top preset number of reasoned tokens are selected as the target tokens starting from the maximum first attention score. The preset number can be a fixed value (percentage or specific value), for example, the top 10% of tokens are selected as the target tokens, or the top 1000 tokens are selected. The preset number can also be dynamic, which can be determined according to the hardware environment, for example, if the current load is high, the number of selected tokens can be appropriately reduced; if the current load is low, the number of selected tokens can be appropriately increased.

[0057] (3) The reinforcement learning algorithm learns how to select target tokens based on the first attention score to maximize a certain long-term objective function, such as the reasoning performance of the model, the quality of the generated text, etc. through interaction with the environment. The reinforcement learning agent takes action (selects target tokens) according to the current state (reasoned tokens and their attention scores, etc.), and then adjusts its strategy according to the reward signal feedback from the environment.

[0058] In the process of selecting target tokens using the reinforcement learning algorithm, the initial reinforcement learning agent defines its state space (including information of reasoned tokens, attention scores, etc.), action space (different tokens selection strategies), and reward function (such as accuracy and coherence of reasoning results, etc.). The agent interacts with the environment and selects target tokens according to the current state. The reward signal is calculated based on the actual reasoning results of the large language model and fed back to the agent. The agent updates its strategy according to the reward signal and continuously learns and optimizes the method of selecting target tokens.

[0059] In addition, the importance weights of tokens within the sliding window can also be used for weighted selection, such as a fixed-length forward sliding window with the current token as the right side of the window, and the top window size number of tokens are selected. The tokens within the sliding window can also be weighted and calculated based on similarity, and the tokens with a similarity greater than a preset value in the sliding window are extracted as model inputs, so as to better preserve local key information. The size of the sliding window is set according to experience.

[0060] In the embodiments of the present application, since the first attention score reflects the relevance and importance of the reasoned tokens to the current reasoning context, by selecting a preset number of reasoned tokens with larger first attention scores and using a reinforcement learning algorithm to select target tokens, valuable tokens can be selected to participate in subsequent reasoning, thereby improving the accuracy of large language model reasoning.

[0061] On the basis of the above-mentioned embodiments, the preset threshold is determined by the following method: The context information entropy is calculated based on the reasoned tokens; The preset threshold is determined based on the context information entropy and the sequence length of the reasoned tokens.

[0062] In the specific implementation process, if the preset threshold is set too high, the number of target tokens screened out will be small, and the number of tokens participating in subsequent large language model reasoning will be small, which will affect the reasoning accuracy. If the preset threshold is set too low, the number of target tokens screened out will be large, and accordingly, the number of tokens participating in subsequent large language model reasoning will be large, which will increase the computational load. Therefore, setting a reasonable preset threshold is the key.

[0063] The embodiments of the present application provide a dynamic determination method of the preset threshold, that is: The context information entropy is calculated according to the following formula: ; Wherein, is the context information entropy; is the reasoned tokens; is the i-th reasoned token, is the frequency of the i-th reasoned token in the to-be-input tokens.

[0064] The preset threshold is calculated according to the following formula: ; Wherein, is the preset threshold; is the sequence length of the reasoned tokens; is the Sigmoid function, which maps the entropy value to (0, 1); , , is a learnable parameter, and its default value can be 0.8, 0.2, 0.1 respectively.

[0065] The embodiment of the application calculates the context information entropy through the reasoned tokens, and determines the preset threshold based on the context information entropy and the sequence length of the reasoned tokens. This dynamic threshold mode avoids the problem of too strict / lenient screening caused by fixed stubbornness.

[0066] On the basis of the above-mentioned embodiment, the first numerical precision is used to perform attention calculation on the reasoned tokens by the large language model, including: The first calculation unit in the GPU is used to run the large language model to perform attention calculation on the reasoned tokens by using the first numerical precision; the first numerical precision includes FP4 and INT4; The second numerical precision is used to perform reasoning on the input sequence by the large language model, including: The second calculation unit in the GPU is used to run the large language model to perform reasoning on the input sequence by using the second numerical precision; the second numerical precision includes FP16 and FP32.

[0067] In the specific implementation process, the GPU includes two calculation units, which are referred to as the first calculation unit and the second calculation unit in the embodiment of the application, wherein the calculation numerical precision of the first calculation unit is lower than that of the second calculation unit. The first calculation unit can be a high-throughput matrix operation unit (Tensor Core), and the second calculation unit can be a flexible control flow unit (CUDA Core). It should be noted that the names of the corresponding first calculation unit and second calculation unit of different GPU manufacturers will be different. For ease of introduction, the first calculation unit is referred to as Tensor Core and the second calculation unit is referred to as CUDA Core in the embodiment of the application.

[0068] It should be noted that the GPU can also include a memory management module for coordinating the transmission of intermediate results between the first calculation unit and the second calculation unit, supporting non-blocking data copy technology to reduce the data transmission delay between the host and the device.

[0069] The Tensor Core in the GPU is used to perform attention calculation in the embodiment of the application, which is to efficiently process large-scale parallel computing requirements. Attention calculation involves a large number of matrix operations, and the GPU has a highly parallel architecture, which can handle multiple computing tasks at the same time, greatly speeding up the calculation. For example, when processing a text sequence containing thousands of tokens, the GPU can simultaneously calculate the attention relationship between multiple tokens, which significantly improves the speed compared with traditional CPU calculation.

[0070] It can be understood that the model inference method provided by the embodiments of the present application is applicable to long context, for example, a scenario with a length exceeding 32K tokens, and does not need to rely on an additional neural network structure for token compression, but can realize efficient context sparsification processing only through hardware-level mixed precision calculation. Thus, parameter redundancy and deployment complexity caused by introducing a small compression network in the NSA method are avoided.

[0071] The first numerical precision can be FP4 or INT4. FP4 is a 4-bit floating-point number format, which, although its representation range is much smaller than that of FP16 and FP32, can provide sufficient calculation accuracy in some specific calculation scenarios, while greatly reducing the calculation amount and storage requirement. INT4 is a 4-bit integer format, which is suitable for some operations with low requirements on numerical precision, and can further reduce the calculation complexity and resource consumption. Using FP4 and INT4 for attention calculation can significantly improve the calculation efficiency on the premise of ensuring a certain calculation accuracy, so that the model can process the already-inferred tokens faster to obtain the first attention score.

[0072] Similarly, by utilizing the powerful computing capability of the GPU, the second computing unit in the GPU is used to run the large language model to perform high-precision inference tasks. By distributing attention calculation and inference tasks to different computing units, parallel processing of tasks can be realized, the utilization rate of the GPU is improved, and the efficiency of the entire model inference process is further optimized.

[0073] The second numerical precision can be FP16 and FP32. FP16 is a 16-bit floating-point number format, which has a faster calculation speed than FP32 (32-bit floating-point number format), but slightly lower precision. However, for many deep learning tasks, FP16 precision is sufficient, and can reduce memory occupancy and calculation amount. FP32 provides higher calculation precision, which is suitable for the inference stage with higher precision requirements. When inferring the input sequence, using FP16 and FP32 can ensure that the model can accurately process the combined information of the target tokens and the to-be-input tokens to generate high-quality inference results. This is because the inference stage needs to accurately calculate the probability distribution of each token, the context representation of the language model, and other key information, and higher numerical precision helps to maintain the accuracy of these calculations.

[0074] It should be noted that the first numerical precision and the second numerical precision can be pre-configured, or the first numerical precision and the second numerical precision can be automatically adjusted according to the hardware environment.

[0075] The embodiments of the present application realize efficient token sparsification processing and improve the calculation efficiency and resource utilization of a large language model in a long context sequence reasoning scenario by mixed precision calculation and heterogeneous hardware cooperation without introducing additional parameterized compression modules.

[0076] On the basis of the above-mentioned embodiments, the attention calculation process performed by the first calculation unit with the first numerical precision and the attention calculation process performed by the second calculation unit with the second numerical precision are executed in parallel.

[0077] In the specific implementation process, the first calculation unit and the second calculation unit simultaneously process different tasks in the same time period, thereby fully utilizing the parallel computing capability of the GPU. For example, when the first calculation unit completes the attention calculation of the already-reasoned tokens corresponding to the current token to be reasoned and screens out the target tokens, the second calculation unit performs reasoning on the input sequence containing the target tokens. At this time, the first calculation unit can calculate the attention of the already-reasoned tokens corresponding to the next token to be reasoned, and so on, thereby realizing independent parallel processing of tasks by the first calculation unit and the second calculation unit.

[0078] The embodiments of the present application separate the calculation tasks of the first calculation unit and the second calculation unit by asynchronous task scheduling, so that the two calculation units work in parallel, thereby improving the overall throughput efficiency.

[0079] On the basis of the above-mentioned embodiments, the target tokens are screened from the already-reasoned tokens based on the first attention scores, including: intermediate tokens are screened from the already-reasoned tokens based on the first attention scores; The attention of the intermediate tokens is calculated with the third numerical precision to obtain second attention scores corresponding to the intermediate tokens, respectively; The target tokens are screened from the intermediate tokens based on the second attention scores.

[0080] In the specific implementation process, the GPU can further include a third processing unit, and the numerical precision handled by the third processing unit is between the numerical precision handled by the first processing unit and the numerical precision handled by the second processing unit. In the screening of the target tokens, the attention values of the reasoned tokens are calculated by the large language model using the first numerical precision, and the first attention scores of each reasoned token are obtained; the intermediate tokens are screened from the reasoned tokens based on the first attention scores, and the method for screening the intermediate tokens includes: starting from the maximum value of the first attention scores, a preset number of reasoned tokens are selected as the intermediate tokens; or, the reasoned tokens with the first attention scores greater than a preset value are selected as the intermediate tokens.

[0081] After the intermediate tokens are obtained, the intermediate tokens are input into the large language model, and the attention calculation is performed on the intermediate tokens by the large language model using the third numerical precision, and the second attention scores corresponding to each intermediate token are obtained. The target tokens are screened from the intermediate tokens based on the second attention scores. The conditions for screening the target tokens can refer to the conditions for screening the target tokens in the above-mentioned embodiments, and the embodiments of the present application will not be repeated.

[0082] In some embodiments, in addition to setting one layer of screening or two layers of screening, more layers of screening can also be set according to actual needs, and the embodiments of the present application do not make specific limitations.

[0083] The embodiments of the present application maximize the calculation efficiency while ensuring the reasoning accuracy of the large language model by establishing a multi-level precision cooperative pipeline.

[0084] On the basis of the above-mentioned embodiments, the to-be-input tokens include the first preset number of reasoned tokens closest to the to-be-reasoned tokens and the first second preset number of tokens in the reasoned tokens.

[0085] In the specific implementation process, when reasoning a token by using the large language model, it is hoped that the input for reasoning the token is the context sequence with high correlation with the token, so as to improve the accuracy of the large language model reasoning on one hand, and reduce the calculation amount of the large language model on the other hand. Therefore, when reasoning the token, in addition to the target tokens screened, the first preset number of reasoned tokens closest to the to-be-reasoned tokens can also be input into the large language model. It should be noted that the value of the first preset number can be less than 10% of the total sequence of the reasoned tokens. The 10% is determined according to experience or multiple tests, and in actual application, it can be adjusted.

[0086] For example, if the position of the token to be inferred is the 10th in the text sequence, and the first preset number is 5, then the part of tokens are the inferred tokens at positions from 5th to 9th. This is because the nearest tokens tend to have a strong semantic association with the token to be inferred, and can provide direct contextual information for the inference process. For example, when predicting the next word in a language model, the first few words usually have a greater impact on its meaning. They can provide local contextual information to help the model better understand the specific language environment in which the token to be inferred is located. For example, in a sentence, the words after "The cat sat on the" are likely to be "mat" or "chair", etc. The first few words "The cat sat on the" are the nearest inferred tokens to the position to be inferred, and they together form a local semantic scene, enabling the model to accurately predict the next suitable word.

[0087] The input into the large language model can also include the first second preset number of tokens in the entire sequence of inferred tokens. The second preset number can be determined according to the task requirements and model design. For example, in a long text sequence, the first second preset number of tokens can contain important information such as the theme and background. These information is very important for understanding the macro semantic and logical structure of the entire text. They can provide the model with global semantic background. For example, in a news report, the first few sentences usually introduce the theme and key information of the news. In the subsequent inference process, these previous tokens can help the model maintain a coherent understanding of the theme and avoid deviating from the theme in the generation or inference process. For example, when generating a news review, referring to the theme information at the beginning of the news can ensure that the review content is related to the news.

[0088] It should be noted that if there are repeated tokens between the tokens included in the target tokens and the tokens included in the tokens to be input, the repeated tokens can be processed.

[0089] In the embodiments of the present application, the composition of the tokens to be input comprehensively considers local and global semantic information. By including the nearest inferred tokens to the token to be inferred, the model can fully consider the current contextual details; at the same time, by adding the first part of the tokens in the inferred tokens, the model can grasp the overall semantic direction. This combination helps the model better balance the relationship between local details and global logic in the inference process, and improves the accuracy and rationality of the inference.

[0090] Figure 2A structural schematic diagram of a model reasoning device provided in an embodiment of the present application. The device can be a module, a program segment or code on an electronic device. It should be understood that the device is the same as the device described above Figure 1 The method embodiment corresponds to, and can execute Figure 1 The device corresponds to each step involved in the method embodiment, and the specific functions of the device can be referred to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes an attention calculation module 201, a screening module 202 and a reasoning module 203, wherein: The attention calculation module 201 is configured to perform attention calculation on the reasoned tokens by a large language model using a first numerical precision, to obtain a first attention score corresponding to each reasoned token; The screening module 202 is configured to screen target tokens from the reasoned tokens based on the first attention score; The reasoning module 203 is configured to perform reasoning on an input sequence by the large language model using a second numerical precision, to obtain a reasoning result output by the large language model; the input sequence includes target tokens and to-be-reasoned tokens corresponding to to-be-input tokens; the to-be-input tokens are tokens preselected from the reasoned tokens according to a preset rule; the first numerical precision is lower than the second numerical precision.

[0091] On the basis of the above embodiment, the screening module 202 is specifically configured to: screen reasoned tokens corresponding to the first attention score greater than a preset threshold as the target tokens; or, select a preset number of reasoned tokens from large to small as the target tokens based on the first attention score; or, screen the target tokens from the reasoned tokens based on the first attention score using a reinforcement learning algorithm.

[0092] On the basis of the above embodiment, the preset threshold is determined by the following method: calculate context information entropy based on the reasoned tokens; determine the preset threshold based on the context information entropy and a sequence length of the reasoned tokens.

[0093] On the basis of the above embodiment, the attention calculation module 201 is specifically configured to: run the large language model by a first calculation unit in a GPU to perform attention calculation on the reasoned tokens using the first numerical precision; the first numerical precision includes FP4 and INT4; The inference on the input sequence by the large language model comprises: The large language model is run by the second computing unit in the GPU to infer the input sequence by using the second numerical precision; the second numerical precision comprises FP16 and FP32.

[0094] On the basis of the above-mentioned embodiments, the attention calculation process by the first computing unit using the first numerical precision and the attention calculation process by the second computing unit using the second numerical precision are executed in parallel.

[0095] On the basis of the above-mentioned embodiments, the screening module 202 is specifically configured to: screen intermediate tokens from the inferred tokens based on the first attention scores; perform attention calculation on the intermediate tokens by using a third numerical precision to obtain second attention scores corresponding to the intermediate tokens respectively; screen the target tokens from the intermediate tokens based on the second attention scores.

[0096] On the basis of the above-mentioned embodiments, the to-be-input tokens comprise a first preset number of inferred tokens closest to the to-be-inferred tokens and a second preset number of tokens in front of the inferred tokens.

[0097] Figure 3 An electronic device entity structure diagram provided by the embodiments of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, the electronic device comprises a processor 301, a memory 302 and a bus 303; wherein: The processor 301 and the memory 302 complete mutual communication through the bus 303; The processor 301 is configured to invoke program instructions in the memory 302 to perform the method provided by each of the above method embodiments, for example, comprising: performing attention calculation on the inferred tokens by a large language model using a first numerical precision to obtain a first attention score corresponding to each inferred token; filtering target tokens from the inferred tokens based on the first attention score; performing inference on an input sequence by the large language model using a second numerical precision to obtain an inference result output by the large language model; the input sequence comprises target tokens and to-be-inferred tokens corresponding to to-be-input tokens; the to-be-input tokens are tokens preselected from the inferred tokens according to a preset rule; and the first numerical precision is lower than the second numerical precision.

[0098] The processor 301 can be an integrated circuit chip having a signal processing capability. The processor 301 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor.

[0099] The memory 302 can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0100] The embodiment discloses a computer program product, the computer program product comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the method provided by each method embodiment described above, for example, comprising: through a large language model, using a first numerical precision to perform attention calculation on the inferred tokens, obtaining a first attention score corresponding to each inferred token; filtering target tokens from the inferred tokens based on the first attention score; through the large language model, using a second numerical precision to infer an input sequence, obtaining an inference result output by the large language model; the input sequence comprises the target tokens and the to-be-input tokens corresponding to the to-be-inferred tokens; the to-be-input tokens are tokens preselected from the inferred tokens according to a preset rule; and the first numerical precision is lower than the second numerical precision.

[0101] The embodiment provides a non-transitory computer readable storage medium, the non-transitory computer readable storage medium stores computer instructions, the computer instructions enable the computer to execute the method provided by each method embodiment described above, for example, comprising: through a large language model, using a first numerical precision to perform attention calculation on the inferred tokens, obtaining a first attention score corresponding to each inferred token; filtering target tokens from the inferred tokens based on the first attention score; through the large language model, using a second numerical precision to infer an input sequence, obtaining an inference result output by the large language model; the input sequence comprises the target tokens and the to-be-input tokens corresponding to the to-be-inferred tokens; the to-be-input tokens are tokens preselected from the inferred tokens according to a preset rule; and the first numerical precision is lower than the second numerical precision.

[0102] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0103] In addition, the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0104] Furthermore, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0105] In this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0106] The above is only an embodiment of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model reasoning method, characterized in that, include: Using a large language model, attention is calculated on the inferred tokens with the first numerical precision to obtain the first attention score corresponding to each inferred token. Target tokens are selected from the inferred tokens based on the first attention score; The large language model is used to infer the input sequence using a second numerical precision to obtain the inference result output by the large language model; the input sequence includes the target tokens and the input tokens corresponding to the tokens to be inferred; the input tokens are tokens pre-selected from the inferred tokens according to preset rules; the first numerical precision is lower than the second numerical precision.

2. The method according to claim 1, characterized in that, The step of filtering target tokens from the reasoned tokens based on the first attention score includes: Select the inferred tokens corresponding to the first attention score that is greater than the preset threshold as the target tokens; or... Based on the first attention score, a preset number of reasoned tokens are selected as the target tokens, from largest to smallest; or... Based on the first attention score, the target tokens are selected from the inferred tokens using a reinforcement learning algorithm.

3. The method according to claim 2, characterized in that, The preset threshold is determined by the following method: Calculate the context information entropy based on the inferred tokens; The preset threshold is determined based on the context information entropy and the sequence length of the inferred tokens.

4. The method according to claim 1, characterized in that, The process of performing attention calculations on the inferred tokens using a large language model with first numerical precision includes: The large language model is run by a first computing unit in the GPU to perform attention computation on the inferred tokens with the first numerical precision; the first numerical precision includes FP4 and INT4. The step of reasoning about the input sequence using the large language model with second numerical precision includes: The large language model is run by a second computing unit in the GPU to infer the input sequence using the second numerical precision, which includes FP16 and FP32.

5. The method according to claim 4, characterized in that, The attention calculation process performed by the first computing unit using the first numerical precision and the attention calculation process performed by the second computing unit using the second numerical precision are executed in parallel.

6. The method according to claim 1, characterized in that, The step of filtering target tokens from the reasoned tokens based on the first attention score includes: Based on the first attention score, filter intermediate tokens from the reasoned tokens; Attention is calculated on the intermediate tokens using a third numerical precision to obtain the second attention score corresponding to each intermediate token. The target tokens are filtered from the intermediate tokens based on the second attention score.

7. The method according to any one of claims 1-6, characterized in that, The tokens to be input include a first preset number of already reasoned tokens that are closest to the tokens to be reasoned, and a second preset number of tokens among the already reasoned tokens.

8. A model reasoning device, characterized in that, include: The attention calculation module is used to perform attention calculation on the inferred tokens using a large language model with the first numerical precision, and obtain the first attention score corresponding to each inferred token. A filtering module is used to filter target tokens from the inferred tokens based on the first attention score; The inference module is used to infer the input sequence using the large language model with a second numerical precision to obtain the inference result output by the large language model; the input sequence includes the target tokens and the input tokens corresponding to the tokens to be inferred; the input tokens are tokens pre-selected from the inferred tokens according to preset rules; the first numerical precision is lower than the second numerical precision.

9. An electronic device, characterized in that, include: Processor, memory, and bus, among which: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-7 by calling the program instructions.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-7.

Images