Large language model reasoning method and device, electronic equipment, storage medium and program product

Abstract

The application provides a large language model reasoning method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: for each attention head in at least one transformer module, calculating an importance index corresponding to the attention head according to a current token; screening a target attention head from the plurality of attention heads according to the importance index; reasoning the current token based on the target attention head to obtain an intermediate reasoning result of the transformer module; and performing subsequent reasoning based on the intermediate reasoning result to obtain a final reasoning result output by the large model. Before the attention mechanism module in the large language model is used to reason the current token, the importance index of each attention head for the current token is calculated, the target attention head used for subsequent reasoning is screened based on the importance index, the pruning of the unimportant attention head is realized, the calculation amount in the reasoning process of the attention mechanism module is reduced, and thus the reasoning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a large language model reasoning method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Large language models are complex deep learning models in the field of artificial intelligence, primarily used for natural language-related tasks. They are typically based on neural network architectures, such as the Transformer architecture. The Transformer architecture, through its self-attention mechanism, can process sequential data in parallel and effectively capture long-range dependencies between different positions within a sequence.

[0003] As the parameter size of Large Language Models (LLMs) continues to increase, their inference computational overhead has become a major bottleneck in deployment and application. Therefore, how to improve the inference efficiency of large language models while maintaining a certain level of inference accuracy has become a direction that researchers have been focusing on and exploring. Summary of the Invention

[0004] The purpose of this application is to provide a large language model reasoning method, apparatus, electronic device, storage medium, and program product, so as to improve the model reasoning efficiency while maintaining a certain reasoning accuracy of the large language model.

[0005] In a first aspect, embodiments of this application provide a large language model inference method. The large language model includes multiple cascaded transformer modules, each transformer module includes an attention mechanism module, and the attention mechanism module includes multiple attention heads. The method includes:

[0006] For each attention head in at least one transformer module, calculate the importance metric corresponding to the attention head based on the current token;

[0007] Select a target attention head from multiple attention heads based on importance indicators;

[0008] Based on the target attention head, inference is performed on the current token to obtain the intermediate inference results of the transformer module;

[0009] Subsequent inferences are performed based on the intermediate inference results to obtain the final inference result output by the large model.

[0010] In this embodiment, before reasoning about the current token using the attention mechanism module in the large language model, the importance index of each attention head to the current token is calculated. Based on the importance index, target attention heads for subsequent reasoning are selected, and unimportant attention heads are pruned. This reduces the computational load in the reasoning process of the attention mechanism module, thereby improving reasoning efficiency. In addition, since unimportant attention heads are pruned, the large language model can meet the requirements of reasoning accuracy.

[0011] In one possible implementation of the first aspect, reasoning about the current token based on the target attention head includes:

[0012] For each target attention head, calculate the attention weight between the current token and each historical token;

[0013] Based on attention weights, select the target historical token corresponding to the target attention head from multiple historical tokens;

[0014] Based on the target history token, the context representation of the target attention head output is calculated using the corresponding target attention head;

[0015] The attention output of the attention mechanism module is obtained based on the contextual representation of all target attention heads;

[0016] Reasoning is performed based on attention output to obtain intermediate reasoning results.

[0017] In this embodiment, as reasoning continues, the number of historical tokens increases. Not all historical tokens are helpful for the reasoning of the current token. Therefore, this application calculates the attention weight between the current token and each historical token to filter historical tokens that are useful for the reasoning of the current token, thereby reducing the amount of computation, improving reasoning efficiency, and without significantly affecting the reasoning accuracy.

[0018] In one possible implementation of the first aspect, the transformer module further includes a feedforward neural network module, which comprises a first fully connected layer and a second fully connected layer; inference is performed based on the attention output to obtain intermediate inference results, including:

[0019] The activation value corresponding to the attention output is calculated using the first fully connected layer;

[0020] Calculate the performance metrics of each neuron in the feedforward neural network module based on activation values;

[0021] Target neurons are selected based on performance metrics;

[0022] The activation value corresponding to the target neuron is used as the input to the second fully connected layer to obtain intermediate inference results.

[0023] In this embodiment, before using the feedforward neural network module to reason about the current token, the performance index of each neuron for reasoning about the current token is calculated, and the neurons with better performance are selected to participate in subsequent reasoning. By reducing the participation of neurons with poor performance in reasoning, the large language model can improve the reasoning efficiency of the large language model while maintaining a certain reasoning accuracy.

[0024] In one possible implementation of the first aspect, subsequent inference is performed based on the intermediate inference results to obtain the final inference result output by the large model, including:

[0025] If the similarity between the intermediate inference result of the current transformer module and the intermediate inference result of the previous transformer module is greater than a preset similarity threshold, then subsequent transformer modules will no longer be used for inference. Instead, the intermediate inference result of the current transformer module will be used as the inference result of the last transformer module for subsequent inference to obtain the final inference result of the large model output.

[0026] In this embodiment, since the large language model includes multiple cascaded transformer modules, the output of the previous transformer module is used as the input of the next transformer module. If the output of the current transformer module is highly similar to the output of the previous transformer module, it indicates that the output of the current transformer module is relatively stable and accurate, and there is no need to use subsequent cascaded transformer modules for inference, thereby improving the efficiency of model inference.

[0027] In one possible implementation of the first aspect, a target attention head is selected from multiple attention heads based on an importance metric, including:

[0028] Calculate the first deviation between the importance indices corresponding to each attention head. If the first deviation is less than the first preset deviation, then all attention heads are taken as target attention heads.

[0029] In this embodiment, if the first deviation between the importance of each attention head is less than the first preset deviation, it means that each attention head is equally important to the current token. Therefore, in order to ensure the inference accuracy of the large language model, the attention head is not pruned.

[0030] In one possible implementation of the first aspect, the target historical token corresponding to the target attention head is selected from multiple historical tokens based on attention weights, including:

[0031] Calculate the second deviation between the attention weights corresponding to each historical token. If the second deviation is less than the second preset deviation, then all historical tokens are used as target historical tokens.

[0032] In this embodiment of the application, if the second deviation between the attention weights corresponding to each historical token is less than the second preset deviation, it indicates that the historical tokens are of equal importance to the reasoning of the current token. In order to ensure the reasoning accuracy of the large language model, the historical tokens are not pruned.

[0033] In one possible implementation of the first aspect, target neurons are selected based on performance metrics, including:

[0034] Calculate the third deviation between the performance indicators corresponding to each neuron. If the third deviation is less than the third preset deviation, then all neurons are taken as target neurons.

[0035] In this embodiment, if the third deviation between the performance indicators corresponding to each neuron is less than the third preset deviation, it indicates that each neuron is of equal importance to the reasoning of the current token. In order to ensure the reasoning accuracy of the large language model, the neurons are not pruned.

[0036] In one possible implementation of the first aspect, the attention mechanism module is one of a multi-head attention mechanism module, a multi-query attention mechanism module, a grouped query attention mechanism module, and a linear attention mechanism module.

[0037] The embodiments of this application can be adapted to various attention mechanism modules, and their application scenarios are wide-ranging.

[0038] In one possible implementation of the first aspect, the performance metrics of each neuron in the feedforward neural network module are calculated based on activation values, including:

[0039] The activation values ​​are calculated according to a preset algorithm to obtain the performance indicators of each neuron; the preset algorithm includes the absolute value algorithm, the square algorithm, or the gradient approximation algorithm.

[0040] The embodiments of this application can accurately evaluate the performance indicators of each neuron using absolute value algorithms, square algorithms, or gradient approximation algorithms.

[0041] Secondly, embodiments of this application provide another method for reasoning using a large language model. The large language model includes multiple cascaded transformer modules, each transformer module including an attention mechanism module and a feedforward neural network module; at least one transformer module includes an attention head pruning module, a token pruning module, a feedforward neural network sparse module, and an early exit module; the method includes:

[0042] For each attention head in the transformer module, the importance index corresponding to the attention head is calculated based on the current token, and the target attention head is filtered based on the importance index by the attention head pruning module;

[0043] The attention weight between the current token and each historical token is calculated based on the target attention head. The token pruning module then filters the target historical tokens corresponding to the target attention head from multiple historical tokens based on the attention weights.

[0044] By utilizing the target attention head, reasoning about the current token based on the target's historical tokens, and obtaining the attention output for the current token;

[0045] The activation value corresponding to the attention output is calculated using the first fully connected layer of the feedforward neural network module, and the target neuron is selected based on the activation value through the sparse module of the feedforward neural network.

[0046] The activation value corresponding to the target neuron is used as the input to the second fully connected layer to obtain the intermediate inference result output by the transformer module.

[0047] By calculating the similarity between the intermediate inference result and the intermediate inference result output by the previous level transformer module, the module exits early. If the similarity is greater than the preset similarity threshold, the subsequent transformer module will no longer be used for inference. Instead, the intermediate inference result of the current transformer module will be used as the inference result of the last transformer module for subsequent inference, thus obtaining the final inference result output by the large model.

[0048] Subsequent inferences are performed based on the intermediate inference results to obtain the final inference result output by the large model.

[0049] The embodiments of this application reduce the computational load of large language models and improve model inference efficiency by using an attention head pruning module, a token pruning module, a feedforward neural network sparse module, and an early exit module.

[0050] Thirdly, embodiments of this application provide a large language model inference apparatus. The large language model includes multiple cascaded transformer modules, each transformer module includes an attention mechanism module, and the attention mechanism module includes multiple attention heads. The apparatus includes:

[0051] The attention metric calculation module is used to calculate the importance metric corresponding to each attention head in each transformer module based on the current token.

[0052] The attention head pruning module is used to select a target attention head from multiple attention heads based on importance metrics;

[0053] The attention calculation module is used to reason about the current token based on the target attention head and obtain the intermediate reasoning results of the transformer module.

[0054] The inference module is used to perform subsequent inference based on intermediate inference results to obtain the final inference result output by the large model.

[0055] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus, wherein:

[0056] The processor and memory communicate with each other via a bus;

[0057] The memory stores program instructions that can be executed by the processor, and the processor can execute the first or second method by calling the program instructions.

[0058] Fifthly, embodiments of this application provide a non-transitory computer-readable storage medium, comprising:

[0059] A non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the methods in various possible implementations of the first or second aspect.

[0060] In a sixth aspect, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the methods in various possible implementations of the first or second aspect.

[0061] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This application provides a schematic flowchart of a large language model reasoning method.

[0064] Figure 2 This is a schematic diagram of another large language model inference method provided in an embodiment of this application;

[0065] Figure 3 A schematic diagram of a large language model inference device provided in an embodiment of this application;

[0066] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0069] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0070] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0071] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0072] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0073] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0074] Large language models are complex deep learning models in the field of artificial intelligence, primarily used for natural language-related tasks. They are typically based on neural network architectures, such as the transformer architecture. The transformer architecture, through its self-attention mechanism, can process sequential data in parallel and effectively capture long-range dependencies between different positions in a sequence. At the core of a large language model is a transformer decoder block with N stacked layers.

[0075] As the parameter scale of Large Language Models (LLMs) continues to increase, their inference computation overhead becomes a major bottleneck in deployment and application. To improve the inference accuracy of Large Language Models while ensuring they meet inference accuracy requirements, this application provides a Large Language Model inference method. The attention mechanism module includes multiple attention heads. Before inferring the current token, the attention mechanism module calculates the importance index of each attention head to the current token. Based on the importance index, attention heads are pruned; that is, only important attention heads are retained for subsequent inference for the current token. This reduces the overall computational load of the attention mechanism module. The pruned attention heads have very little impact on subsequent inference for the current token; therefore, even if they do not participate in subsequent inference, they will not significantly affect the final inference result. Therefore, the Large Language Model inference method provided in this application improves inference efficiency while meeting the inference accuracy requirements of the Large Language Model.

[0076] It is understood that the large language model inference method provided in this application embodiment can be applied to electronic devices, including terminals and servers; wherein the terminal can specifically be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; the server can specifically be an application server or a web server. A large language model runs in the electronic device.

[0077] This application improves upon at least one of four aspects: attention head pruning, historical token pruning, sparsification of neurons in the feedforward neural network, and determining whether the result transformer module needs to perform inference in advance. These improvements reduce the computational load of inference and thus improve inference efficiency. Each of these four aspects will be described in detail below.

[0078] Figure 1 This is a schematic diagram of a large language model reasoning method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes:

[0079] Step 101: For each attention head in at least one transformer module, calculate the importance index corresponding to the attention head based on the current token;

[0080] Step 102: Select the target attention head from multiple attention heads based on importance indicators;

[0081] Step 103: Perform inference on the current token based on the target attention head to obtain the intermediate inference results of the transformer module;

[0082] Step 104: Perform subsequent inference based on the intermediate inference results to obtain the final inference result output by the large model.

[0083] In practical implementation, a large language model is a large-scale language inference model oriented towards a transformer structure. Specifically, large language models can be GPT, LLaMA, Qwen, etc. A large language model includes multiple cascaded transformer modules, each performing specific transformations and processing on its input data. The output of one module serves as the input for the next. Through the cascading of multiple transformer modules, the large language model can progressively and deeply understand and generate language information. It is understandable that, in addition to multiple cascaded transformer modules, a large language model may also include other modules, such as input processing modules, positional encoding modules, and output generation modules.

[0084] The following describes a typical architecture for a large language model:

[0085] 1. The input processing module includes a tokenizer and an embedding layer. The tokenizer breaks down the original text into sub-word units that the model can process (e.g., BPE, WordPiece algorithms), and then outputs a token ID sequence (e.g., [2034, 508, 1025]). The embedding layer maps the token IDs to dense vectors (dimensional d_model, e.g., 4096), and outputs an embedding matrix of [seq_len, d_model].

[0086] 2. The position encoding module includes position injection, mainly comprising absolute position encoding and relative position encoding. Relative position encoding can be specifically implemented through rotational position encoding, injecting position information into the query (Q) / key (K) (LLaMA, GPT-NeoX) via a rotation matrix.

[0087] 3. The core layer of the transformer consists of N stacked transformer modules. Each transformer module includes an attention mechanism module, a feedforward network, residual connections, and a normalization layer.

[0088] The attention mechanism module can be a multi-head sub-attention mechanism. Its working principle is that the input vector is passed through a linear layer to generate Q / K / values ​​(V) (dimensions d_k, d_v). The formula for calculating the attention score for each attention head is:

[0089] .

[0090] .

[0091] in, The attention score is the score corresponding to the i-th attention head. Let Q be the weight in the i-th attention head. Let K be the weight of the i-th attention head. Let V be the weight of V in the i-th attention head. Mask is the mask.

[0092] Understandably, the attention mechanism module can also include multi-head attention (MHA), multi-query attention (MQA), grouped query attention (GQA), and linear attention (MLA), etc.

[0093] The outputs from multiple sources are concatenated and then fused using a w_0 linear layer.

[0094] The feedforward network consists of a two-layer fully connected structure: .

[0095] The intermediate layer dimension is expanded (d_{ff} = 4 times d_{model}).

[0096] The role of the feedforward network is to provide nonlinear transformation capability (the main body of model parameters).

[0097] The output of the residual connection in the normalization process equals the sublayer input plus the sublayer output. Normalization is performed before the sublayer is processed.

[0098] 4. The output generation module projects the final hidden state onto the vocabulary space, and then performs softmax calculation to obtain the final result.

[0099] In this embodiment of the application, the attention mechanism module in each transformer module prunes multiple attention heads before inference on the current token, as follows:

[0100] When reasoning about text data, the text data is broken down into tokens (usually word units). For each transformer module, when processing the current token, the performance of each attention head on that token needs to be evaluated. The current token refers to the token that the large language model is currently reasoning about.

[0101] Obtain the Q of the current token, and the K and V of historical tokens, and then split Q, K, and V along the attention head dimension. Importance metrics can be estimated in various ways, such as calculating attention scores, activation entropy, and attention norms. Taking the attention score as an example, calculate the concentration or variation of the attention score allocated to the current token by the attention head.

[0102] After calculating the importance index of each attention head, a screening criterion can be set to determine which attention heads are target attention heads. This screening criterion can be a fixed threshold, for example, only retaining a certain percentage (e.g., the top 70%) of attention heads based on the importance index, or the threshold can be dynamically adjusted according to the performance requirements of the model and the limitations of computing resources.

[0103] Based on the above selection criteria, the target attention head is obtained by selecting from multiple attention heads. This process is actually a dynamic pruning of attention heads, removing those attention heads that are relatively less important in the current token processing, thereby reducing the amount of computation in the subsequent reasoning process. At the same time, it retains as much information as possible about the understanding of the current token and the information contained in the most critical attention head for generation.

[0104] After inference calculations are performed using the target attention head, intermediate inference results for the current module can be generated by combining other components of the transformer (such as feedforward neural networks). These intermediate inference results not only contain information about the current token itself, but also incorporate contextual relationship information between the token and other tokens (i.e., historical tokens) captured by the target attention head, providing a foundation for further processing in subsequent modules.

[0105] The transformer module of the current layer can use the intermediate inference results as input to the transformer module of the next layer for subsequent inference, and obtain the final inference result of the output of the large language model.

[0106] It should be noted that each transformer module in the large language model can use the above process to filter the attention heads in the current transformer module before proceeding with subsequent inference.

[0107] The following pruning methods can be used to address different attention mechanisms:

[0108] MHA: After pruning, Q / K / V are filtered and calculated separately, and valid heads are retained. Concat is used to splice along the head dimension, and a linear layer is used to uniformly map back to the original dimension.

[0109] MQA: Due to shared key-value pairs, only the head portion of the Q part is filtered; the shared key-value pairs are still retained after the output is concatenated.

[0110] GQA: Pruning is performed on a group basis to ensure that pruning does not disrupt the shared relationships of the key / value structure;

[0111] MLA: In low-rank projection scenarios, only the most effective query direction is selected to maintain the invertibility of sparse matrix multiplication.

[0112] To maintain the same output dimension as the unpruned model, zero-padding is used for the unretained heads, or a soft-mask mechanism is used to skip their computation but keep the output shape consistent.

[0113] In this embodiment, before reasoning about the current token using the attention mechanism module in the large language model, the importance index of each attention head to the current token is calculated. Based on the importance index, target attention heads for subsequent reasoning are selected, and unimportant attention heads are pruned, reducing the amount of computation in the reasoning process of the attention mechanism module and thus improving reasoning efficiency.

[0114] Furthermore, the method provided in this application is compatible with current mainstream large language models, possessing strong versatility, low deployment cost, and requiring no modification to the internal parameters of the large language model or retraining. This application does not alter the internal data processing logic of the large language model; it merely determines which nodes' data should not be used as input for subsequent inference, and which nodes' output data should be used as input for subsequent inference.

[0115] In any embodiment, the method for pruning historical tokens is as follows:

[0116] The attention weight between the current token and each historical token is calculated using each attention head in the transformer module. Based on the attention weight, the target historical token corresponding to the target attention head is selected from multiple historical tokens. Based on the target historical token, the context representation of the target attention head output is calculated using the corresponding attention head. The attention output of the attention mechanism module is obtained based on the context representation of all attention heads. Inference is performed based on the attention output to obtain the intermediate inference result.

[0117] Based on the above embodiments, after pruning the attention head, the historical tokens can also be pruned. The specific method is as follows:

[0118] For each target attention head, calculate the attention weight between the current token and each historical token;

[0119] Based on attention weights, select the target historical token corresponding to the target attention head from multiple historical tokens;

[0120] Based on the target history token, the context representation of the target attention head output is calculated using the corresponding target attention head;

[0121] The attention output of the attention mechanism module is obtained based on the contextual representation of all target attention heads;

[0122] Reasoning is performed based on attention output to obtain intermediate reasoning results.

[0123] In practical implementation, historical tokens refer to tokens already inferred by the large language model and tokens initially input into the large language model. As the large language model performs inference, the number of historical tokens increases. If all of them were to participate in subsequent inference, the computational load would become increasingly large, thus affecting inference efficiency. In reality, some historical tokens in the sequence may not be of much use in inferring the current token. Therefore, this application embodiment selects target historical tokens that are helpful in inferring the current token, allowing these selected target historical tokens to participate in the subsequent inference of the current token.

[0124] In previous large language model inference, target attention heads were selected from multiple attention heads. These target attention heads are considered to be of high importance when processing the current token, as they can capture the relationship between the current token and historical tokens from different key perspectives.

[0125] For each target attention head, the attention score between the current token and each historical token is calculated. The formula for calculating the attention score is as follows: Attention scores can be used to identify historical tokens that are semantically weakly related to the current token, and these tokens can be masked before softmax calculation. It is the dot product; This is for square root calculation.

[0126] The selection threshold can be determined based on the specific task requirements and computational resource limitations of the large language model. For example, a fixed attention weight threshold can be set, such as starting from the maximum attention weight and only retaining the top 40% of historical tokens with the highest attention weight as target historical tokens. Alternatively, the threshold can be dynamically adjusted based on the semantic correlation between the current token and each historical token to ensure that the selected historical tokens can provide sufficient semantic information to support the reasoning of the current token.

[0127] For the selected target historical tokens, a boolean mask is generated. The score of unselected historical tokens is infinitesimal (-inf).

[0128] The selected target historical tokens are those historical tokens that are important for understanding and generating the current token. They contain key semantic and syntactic information related to the current token, which can help the model build a better contextual representation.

[0129] Using the corresponding target attention head, the value vector of the target historical token is weighted and summed with the corresponding attention weights to obtain the context representation output by the target attention head. This context representation integrates the information of the target historical tokens and is weighted according to their relevance to the current token, reflecting the semantic and syntactic environment of the current token within the context.

[0130] The context representations output by all target attention heads are concatenated or summed to integrate them into the attention output of the attention mechanism module. Concatenation can preserve the different perspectives provided by each target attention head, but it increases the dimensionality of the output; summation can fuse the information from each target attention head without increasing dimensionality. The specific operation method can be determined based on the design requirements of the model and the characteristics of the task.

[0131] By integrating the contextual representations of multiple target attention heads, the output of the attention mechanism module can be made richer and more comprehensive. This is because each target attention head focuses on different key relationships between the current token and historical tokens, and fusing their information can provide more complete contextual information for the subsequent reasoning process.

[0132] The attention output of the attention mechanism module is used as input and passed to other components in the Transformer module (such as feedforward neural networks) for further processing and transformation. Feedforward neural networks typically perform non-linear transformations on the attention output to enhance the model's expressive and learning capabilities, thereby extracting higher-level semantic features and patterns.

[0133] After processing by components such as the feedforward neural network, the intermediate inference result of the current Transformer module is generated. This intermediate result contains information about the current token itself, as well as contextual relationship information between other tokens captured by the target attention head. It will become the input of the next Transformer module, providing the foundation and basis for the entire model's inference process.

[0134] It should be noted that after removing some historical tokens, in order to ensure that the context token is not lost after pruning, the complete position signal can be retained when using absolute position encoding such as RoPE; pruning is only performed during the attention score calculation stage and does not modify the key / value cache structure; for decoder-only models, the priority of the most recent token is retained to ensure semantic consistency in generation.

[0135] In this embodiment, as reasoning continues, the number of historical tokens increases. Not all historical tokens are helpful for the reasoning of the current token. Therefore, this application calculates the attention weight between the current token and each historical token to filter historical tokens that are useful for the reasoning of the current token, thereby reducing the amount of computation, improving reasoning efficiency, and without significantly affecting the reasoning accuracy.

[0136] In any embodiment, the transformer module further includes a feedforward neural network module, which includes a first fully connected layer and a second fully connected layer; the method for sparsifying the neurons in the feedforward neural network module is as follows:

[0137] The activation value corresponding to the attention output is calculated using the first fully connected layer; where the attention output is the output of the attention mechanism module of the transformer module;

[0138] Calculate the performance metrics of each neuron in the feedforward neural network module based on activation values;

[0139] Target neurons are selected based on performance metrics;

[0140] The activation value corresponding to the target neuron is used as the input to the second fully connected layer to obtain intermediate inference results.

[0141] Based on the above embodiments, after pruning the attention head, the neurons in the feedforward neural network can also be sparsified, as follows:

[0142] The activation value corresponding to the attention output is calculated using the first fully connected layer; this attention output is the result of concatenating the outputs of the target attention head obtained through the above embodiments.

[0143] Calculate the performance metrics of each neuron in the feedforward neural network module based on activation values;

[0144] Target neurons are selected based on performance metrics;

[0145] The activation value corresponding to the target neuron is used as the input to the second fully connected layer to obtain intermediate inference results.

[0146] In the specific implementation process, after receiving the attention output from the attention mechanism module, the feedforward neural network module in the transformer module further transforms and refines the information through two fully connected layers to obtain richer feature representations, thereby providing a stronger foundation for the final inference result.

[0147] The first fully connected layer receives the attention output from the attention mechanism module as input. Its function is to perform a linear transformation on the input data, that is, to map the input data to a new feature space through the learned weight matrix and bias vector. The formula is expressed as Linear(x) = xW + b, where x is the input, W is the weight matrix, and b is the bias vector.

[0148] After linear transformations, nonlinear activation functions (such as ReLU and GELU) are typically applied to introduce nonlinearity, enabling large language models to learn and capture more complex features and patterns. Activation functions ensure that the network retains sufficient expressive power even after multiple layers are stacked, avoiding the problem of a model degenerating into a simple linear model that can occur with purely linear transformations. For example, the ReLU activation function is defined as ReLU(x) = max(0,x), which sets all negative values ​​to zero and retains the positive values, thereby introducing sparsity and nonlinearity into the feature space.

[0149] Performance metrics can be various indicators that measure the importance or contribution of neurons to inference of the current token. Common indicators include the magnitude of neuron activation values, the magnitude of gradients, and the degree of influence on model output. For example, the average activation value of each neuron can be calculated, or the magnitude of the gradients received by neurons during backpropagation can be calculated. These metrics can reflect the activity intensity of neurons in the current task and data and their contribution to model learning.

[0150] For each neuron, the corresponding performance index is calculated based on its activation value, the absolute value of the activation value, the square of the activation value, or the gradient of the activation value.

[0151] The selection criteria can be determined based on the model's inference requirements and computational resource constraints. For example, a performance metric threshold can be set, retaining only neurons with performance metrics above the threshold as target neurons. Alternatively, neurons can be ranked according to performance metrics, selecting the top 50% (e.g., the top percentage) of neurons as target neurons.

[0152] According to the established criteria, target neurons are selected from all neurons in the first fully connected layer. This process aims to remove neurons that are relatively unimportant in the current inference task, thereby reducing subsequent computation and improving inference efficiency. Simultaneously, the selected target neurons should retain most of the key feature information to ensure that the model's performance does not significantly decrease due to the reduction in the number of neurons.

[0153] From the output of the first fully connected layer, the activation values ​​corresponding to the target neuron are selected. These activation values ​​represent the most valuable feature information retained after filtering, which is crucial for the current inference task. These activation values ​​are then concatenated or recombined to form a new feature vector, which serves as the input to the second fully connected layer.

[0154] The second fully connected layer performs another linear transformation on the input feature vector, further integrating and refining the feature information. Similar to the first layer, the second fully connected layer also has its own weight matrix and bias vector. It maps the input features to the output space using the learned parameters, generating the final intermediate inference result. This intermediate inference result combines the output of the attention mechanism module and the processing of the feedforward neural network module, including the contextual information of the current token and the feature representation after nonlinear transformation. This provides richer and more advanced feature representations for subsequent transformer modules or the final model output.

[0155] In this embodiment, before using the feedforward neural network module to reason about the current token, the performance index of each neuron for reasoning about the current token is calculated. The neurons with better performance participate in subsequent reasoning. By reducing the participation of neurons with poor performance in reasoning, the large language model can improve the reasoning efficiency of the large language model while maintaining a certain reasoning accuracy.

[0156] Based on the above embodiments, subsequent inference is performed based on the intermediate inference results to obtain the final inference result output by the large model, including:

[0157] If the similarity between the intermediate inference result of the current transformer module and the intermediate inference result of the previous transformer module is greater than a preset similarity threshold, then subsequent transformer modules will no longer be used for inference. Instead, the intermediate inference result of the current transformer module will be used as the inference result of the last transformer module for subsequent inference to obtain the final inference result of the large model output.

[0158] In practice, the core idea of ​​this scheme is to optimize inference efficiency by detecting the similarity between the intermediate inference results of the transformer module to determine whether to terminate the inference process of subsequent modules in advance.

[0159] Similarity can be calculated in various ways, with common methods including cosine similarity and Euclidean distance. For example, cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them. Its value ranges from -1 to 1, with a value closer to 1 indicating a higher similarity.

[0160] During the inference process of each transformer module, the similarity calculation method described above can be used to calculate the similarity between the intermediate inference result of the current module and the intermediate inference result of the previous transformer module.

[0161] A preset similarity threshold is used to determine whether to stop inference in subsequent transformer modules. This preset similarity threshold can be set according to the specific task requirements, data characteristics, and performance requirements of the model. For example, for scenarios with high requirements for inference efficiency, the threshold can be set higher (e.g., 0.9 or higher) to terminate inference earlier; while for scenarios with high requirements for accuracy, the threshold can be set lower (e.g., 0.7).

[0162] After calculating the similarity between the current module and the previous module, it is compared with a preset similarity threshold. If the similarity is greater than or equal to the preset threshold, it means that the reasoning result of the current module is close enough to the reasoning result of the previous module. Further reasoning may not bring significant performance improvement, and may even lead to overfitting or unnecessary computational overhead.

[0163] If the similarity between the intermediate inference result of the current module and the intermediate inference result of the previous module is greater than a preset similarity threshold, then subsequent transformer modules will not be called for inference. The intermediate inference result of the current module will be passed to subsequent parts of the model (such as decoders, classifiers, etc.) to generate the final output result.

[0164] In this embodiment, since the large language model includes multiple cascaded transformer modules, the output of the previous transformer module is used as the input of the next transformer module. If the output of the current transformer module is highly similar to the output of the previous transformer module, it indicates that the output of the current transformer module is relatively stable and accurate, and there is no need to use subsequent cascaded transformer modules for inference, thereby improving the efficiency of model inference.

[0165] Based on the above embodiments, a target attention head is selected from multiple attention heads according to an importance index, including:

[0166] Calculate the first deviation between the importance indices corresponding to each attention head. If the first deviation is less than the first preset deviation, then all attention heads are taken as target attention heads.

[0167] In practice, the first deviation can be the absolute value of the difference between the importance indices of any two attention heads. If this first deviation is less than the first preset deviation, it indicates that the importance of each attention head is equivalent. In this case, it is not suitable to prune the attention heads. Alternatively, the first deviation can also be measured by calculating the standard deviation, variance, etc., of the attention heads.

[0168] If the first bias is less than or equal to the first preset bias, then all attention heads are used as target attention heads. This means that in the current inference task, all attention heads are considered important, and therefore all attention heads are retained for subsequent inference.

[0169] If the first deviation is greater than the first preset deviation, further screening of target attention heads is required, for example, selecting the top-ranked attention heads based on the magnitude of the importance index.

[0170] In this embodiment, if the first deviation between the importance of each attention head is less than the first preset deviation, it means that each attention head is equally important to the current token. Therefore, in order to ensure the inference accuracy of the large language model, the attention head is not pruned.

[0171] Based on the above embodiments, and based on attention weights, the target historical token corresponding to the target attention head is selected from multiple historical tokens, including:

[0172] Calculate the second deviation between the attention weights corresponding to each historical token. If the second deviation is less than the second preset deviation, then all historical tokens are used as target historical tokens.

[0173] In its implementation, the core idea of ​​this scheme is to determine whether to retain all historical tokens by evaluating the differences (i.e., the second bias) between the attention weights of each historical token, thereby dynamically adjusting the usage strategy of historical tokens during the inference process.

[0174] The second bias measures the difference in attention weights between different historical tokens. Specifically, it can be absolute bias, relative bias, or other statistics.

[0175] Based on the selected bias calculation method, a second bias is calculated between the attention weights of all historical tokens. This second preset bias is a threshold used to determine whether to retain all historical tokens. The specific value of the second preset threshold can be set according to the model's specific task requirements, data characteristics, and performance requirements. For example, if the task has high requirements for inference efficiency, the threshold can be set smaller to retain all historical tokens more frequently; if the task has high requirements for accuracy, the threshold can be set larger.

[0176] The calculated second deviation is compared with a second preset deviation. If the second deviation is less than or equal to the second preset deviation, it indicates that the differences in attention weights among all historical tokens are small, meaning that the importance of each historical token is relatively balanced, with no obvious distinction between "important" and "unimportant." If the second deviation is less than or equal to the second preset deviation, all historical tokens are selected as target historical tokens. This means that in the current inference task, all historical tokens are considered important, and therefore all historical tokens are retained for subsequent inference. If the second deviation is greater than the second preset deviation, further filtering of target historical tokens is required, for example, selecting historical tokens with higher rankings based on their attention weights.

[0177] In this embodiment of the application, if the second deviation between the attention weights corresponding to each historical token is less than the second preset deviation, it indicates that the historical tokens are of equal importance to the reasoning of the current token. In order to ensure the reasoning accuracy of the large language model, the historical tokens are not pruned.

[0178] Based on the above embodiments, target neurons are screened based on performance metrics, including:

[0179] Calculate the third deviation between the performance indicators corresponding to each neuron. If the third deviation is less than the third preset deviation, then all neurons are taken as target neurons.

[0180] In the specific implementation process, the core idea of ​​this scheme is to determine whether to retain all neurons by evaluating the differences in the performance indicators of each neuron (i.e., the third bias), thereby dynamically adjusting the neuron usage strategy during the inference process.

[0181] The third bias is used to measure the differences in performance metrics between different neurons. Specifically, it can be absolute bias, relative bias, or other statistics.

[0182] Based on the selected bias calculation method, a third bias is calculated among the performance metrics of all neurons. This third preset bias is a threshold used to determine whether to retain all neurons. This threshold can be set according to the specific task requirements, data characteristics, and performance requirements of the model. For example, if the task requires high inference efficiency, the threshold can be set smaller to retain all neurons more frequently; if the task requires high accuracy, the threshold can be set larger.

[0183] The calculated third bias is compared with the third preset bias. If the third bias is less than or equal to the third preset bias, it indicates that the differences in performance metrics among all neurons are small, meaning that the contributions of each neuron are relatively balanced, with no obvious distinction between "important" and "unimportant." If the third bias is less than or equal to the third preset bias, all neurons are considered target neurons. This means that in the current inference task, all neurons are considered important, and therefore all neurons are retained for subsequent inference. If the third bias is greater than the third preset bias, further screening of target neurons is required, for example, selecting neurons ranked higher based on the magnitude of their performance metrics.

[0184] In this embodiment, if the third deviation between the performance indicators corresponding to each neuron is less than the third preset deviation, it indicates that each neuron is of equal importance to the reasoning of the current token. In order to ensure the reasoning accuracy of the large language model, the neurons are not pruned.

[0185] Figure 2 This is a schematic diagram of another large language model inference method provided in this application embodiment. Based on the large language model, it mainly includes four modules: an attention head pruning module, a token pruning module, a feedforward network (FFN) sparse module, and an early exit module. These system modules can be enabled independently or deployed jointly, adapting to various mainstream large language models (such as LLaMA and Qwen series) and compatible with different types of attention mechanisms (including MHA, MQA, GQA, MLA, etc.).

[0186] The entire process is divided into four key stages, which involve dynamic compression and optimization from four dimensions: attention mechanism, input sequence, neural network parameters, and model depth.

[0187] The first stage is attention head pruning. Each attention head in the transformer model is responsible for capturing different aspects of information from the input sequence, but not all heads contribute significantly to the final output. This stage evaluates the historical output scores of each attention head (i.e., the importance index corresponding to the attention head), and based on a preset ratio `head_threshold`, selects and removes heads with lower contributions, thereby reducing redundant computation. It should be noted that during attention head pruning, the historical output scores can be obtained through calculations such as entropy or the average attention weights.

[0188] The second stage is token pruning. When processing long texts, some historical tokens (such as function words or redundant content) have little impact on semantic understanding. This stage uses attention weights to score historical tokens, setting a threshold (token_threshold). Historical tokens below this threshold are masked or directly removed, and only key historical tokens are retained for subsequent calculations, thereby significantly shortening the sequence length and saving memory and computing resources.

[0189] The third stage is FFN neuron sparsification. As a crucial component of the transformer layer, the feedforward neural network (FFN) contains many neurons that are not always active during actual inference. This stage identifies important neurons (i.e., target neurons) by statistically analyzing their activation frequency and amplitude, and activates only the top-K neurons, while the remaining neurons are either set to zero or skipped, achieving parameter-level compression and acceleration. The value of K can be a preset proportion (neuron_threshold) of the total number of neurons.

[0190] The fourth stage is the early exit mechanism. Transformer models typically consist of multiple stacked layers, but under certain input conditions, the output of an intermediate layer may be sufficiently stable, eliminating the need for further computation. This mechanism compares the similarity between the hidden states of the current layer and the previous layer of the transformer model (using methods such as cosine similarity or Euclidean distance) and sets an exit threshold (exit_threshold). If the similarity exceeds this threshold, the inference process is terminated early, and the current result is returned, thus saving significant computational resources.

[0191] It should be noted that the specific implementation methods of each stage can be found in the above embodiments, and will not be repeated here.

[0192] The entire optimization process is dynamic and adaptive, automatically determining how many attention heads to prune, how many tokens to retain, how many neurons to activate, and whether to exit prematurely at each layer, based on the different input content. This multi-layered, multi-dimensional compression strategy enables the model to significantly reduce computational overhead while maintaining high inference quality.

[0193] Without changing the model structure and parameters, each module significantly improves inference efficiency by performing real-time calculation and filtering of intermediate representations during the forward propagation stage, avoiding redundant calculations.

[0194] Figure 3 This is a schematic diagram of a large language model inference device provided in an embodiment of this application. The device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in each step of the method embodiment can be found in the description above; to avoid repetition, detailed descriptions are omitted here. The device includes: an attention index calculation module 301, an attention head pruning module 302, an attention calculation module 303, and an inference module 304, wherein:

[0195] The attention metric calculation module 301 is used to calculate the importance metric corresponding to each attention head in at least one transformer module based on the current token.

[0196] The attention head pruning module 302 is used to filter target attention heads from multiple attention heads based on importance indicators;

[0197] The attention calculation module 303 is used to reason about the current token based on the target attention head and obtain the intermediate reasoning results of the transformer module;

[0198] The reasoning module 304 is used to perform subsequent reasoning based on the intermediate reasoning results to obtain the final reasoning result output by the large model.

[0199] Based on the above embodiments, the attention calculation module 303 is specifically used for:

[0200] For each target attention head, calculate the attention weight between the current token and each historical token;

[0201] Based on the attention weight, the target historical token corresponding to the target attention head is selected from multiple historical tokens;

[0202] Based on the target history token, the context representation output by the target attention head is calculated using the corresponding target attention head;

[0203] The attention output of the attention mechanism module is obtained based on the contextual representation of all the target attention heads;

[0204] Reasoning is performed based on the attention output to obtain the intermediate reasoning result.

[0205] Based on the above embodiments, the transformer module further includes a feedforward neural network module, which includes a first fully connected layer and a second fully connected layer; the attention calculation module 303 is specifically used for:

[0206] The activation value corresponding to the attention output is calculated using the first fully connected layer.

[0207] Calculate the performance index of each neuron in the feedforward neural network module based on the activation value;

[0208] Target neurons are selected based on the aforementioned performance metrics;

[0209] The intermediate inference result is obtained by using the activation value corresponding to the target neuron as the input to the second fully connected layer.

[0210] Based on the above embodiments, the inference module 304 is specifically used for:

[0211] If the similarity between the intermediate inference result of the current transformer module and the intermediate inference result of the previous transformer module is greater than a preset similarity threshold, then subsequent transformer modules will no longer be used for inference. Instead, the intermediate inference result of the current transformer module will be used as the inference result of the last transformer module for subsequent inference, thereby obtaining the final inference result output by the large model.

[0212] Based on the above embodiments, the attention head pruning module 302 is specifically used for:

[0213] Calculate the first deviation between the importance indices corresponding to each attention head. If the first deviation is less than a first preset deviation, then all the attention heads are taken as the target attention heads.

[0214] Based on the above embodiments, the attention calculation module 303 is specifically used for:

[0215] Calculate the second deviation between the attention weights corresponding to each historical token. If the second deviation is less than the second preset deviation, then all the historical tokens are taken as the target historical token.

[0216] Based on the above embodiments, the attention calculation module 303 is specifically used for:

[0217] Calculate the third deviation between the performance indicators corresponding to each neuron. If the third deviation is less than the third preset deviation, then all the neurons are taken as the target neurons.

[0218] Based on the above embodiments, the attention mechanism module is one of a multi-head attention mechanism module, a multi-query attention mechanism module, a grouped query attention mechanism module, and a linear attention mechanism module.

[0219] Based on the above embodiments, the attention calculation module 303 is specifically used for:

[0220] The activation values ​​are calculated according to a preset algorithm to obtain the performance index of each neuron; the preset algorithm includes an absolute value algorithm, a square algorithm, or a gradient approximation algorithm.

[0221] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device includes: a processor 401, a memory 402, and a bus 403; wherein:

[0222] The processor 401 and the memory 402 communicate with each other through the bus 403;

[0223] The processor 401 is used to call program instructions in the memory 402 to execute the methods provided in the above-described method embodiments, including, for example, calculating the importance index corresponding to each attention head in at least one transformer module based on the current token; filtering target attention heads from multiple attention heads based on the importance index; performing inference on the current token based on the target attention head to obtain intermediate inference results of the transformer module; and performing subsequent inference based on the intermediate inference results to obtain the final inference result output by the large model.

[0224] Processor 401 can be an integrated circuit chip with signal processing capabilities. The processor 401 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0225] The memory 402 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0226] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: for each attention head in at least one transformer module, calculating an importance index corresponding to the attention head based on the current token; filtering target attention heads from multiple attention heads based on the importance index; performing inference on the current token based on the target attention head to obtain intermediate inference results of the transformer module; and performing subsequent inference based on the intermediate inference results to obtain the final inference result output by the large model.

[0227] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the methods provided in the above-described method embodiments. These instructions include, for example, calculating an importance index corresponding to each attention head in at least one transformer module based on the current token; selecting a target attention head from among multiple attention heads based on the importance index; performing inference on the current token based on the target attention head to obtain intermediate inference results from the transformer module; and performing subsequent inference based on the intermediate inference results to obtain the final inference result output by the large model.

[0228] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0229] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0230] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0231] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0232] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A large language model inference method, characterized in that, The large language model includes multiple cascaded transformer modules, each transformer module includes an attention mechanism module, and the attention mechanism module includes multiple attention heads. The input data of the large language model is text data; the method includes: For each attention head in at least one of the transformer modules, calculate the importance index corresponding to the attention head based on the current token; Target attention heads are selected from multiple attention heads based on the importance index; Based on the target attention head, inference is performed on the current token to obtain the intermediate inference result of the transformer module; Based on the intermediate inference results, subsequent inferences are performed to obtain the final inference results output by the large language model; The reasoning based on the target attention head for the current token includes: For each target attention head, calculate the attention weight between the current token and each historical token; Based on the attention weight, the target historical token corresponding to the target attention head is selected from multiple historical tokens; Based on the target history token, the context representation output by the target attention head is calculated using the corresponding target attention head; The attention output of the attention mechanism module is obtained based on the contextual representation of all the target attention heads; Reasoning is performed based on the attention output to obtain the intermediate reasoning result; The transformer module further includes a feedforward neural network module, which comprises a first fully connected layer and a second fully connected layer; the inference based on the attention output to obtain the intermediate inference result includes: The activation value corresponding to the attention output is calculated using the first fully connected layer. Calculate the performance index of each neuron in the feedforward neural network module based on the activation value; Target neurons are selected based on the aforementioned performance metrics; The intermediate inference result is obtained by using the activation value corresponding to the target neuron as the input to the second fully connected layer; The subsequent reasoning based on the intermediate reasoning results to obtain the final reasoning result output by the large language model includes: If the similarity between the intermediate inference result of the current transformer module and the intermediate inference result of the previous transformer module is greater than a preset similarity threshold, then subsequent transformer modules will no longer be used for inference. Instead, the intermediate inference result of the current transformer module will be used as the inference result of the last transformer module for subsequent inference, thereby obtaining the final inference result output by the large language model.

2. The method of claim 1, wherein, The step of selecting a target attention head from multiple attention heads based on the importance index includes: Calculate the first deviation between the importance indices corresponding to each attention head. If the first deviation is less than a first preset deviation, then all the attention heads are taken as the target attention heads.

3. The method of claim 1, wherein, The step of filtering the target historical token corresponding to the target attention head from multiple historical tokens based on the attention weight includes: Calculate the second deviation between the attention weights corresponding to each historical token. If the second deviation is less than the second preset deviation, then all the historical tokens are taken as the target historical token.

4. The method of claim 1, wherein, The process of selecting target neurons based on the performance metrics includes: Calculate the third deviation between the performance indicators corresponding to each neuron. If the third deviation is less than the third preset deviation, then all the neurons are taken as the target neurons.

5. The method according to claim 1, characterized in that, The attention mechanism module is one of the following: multi-head attention mechanism module, multi-query attention mechanism module, grouped query attention mechanism module, and linear attention mechanism module.

6. The method according to claim 1, characterized in that, The calculation of the performance metrics of each neuron in the feedforward neural network module based on the activation values ​​includes: The activation values ​​are calculated according to a preset algorithm to obtain the performance index of each neuron; the preset algorithm includes an absolute value algorithm, a square algorithm, or a gradient approximation algorithm.

7. A large language model reasoning method, characterized in that, The large language model includes multiple cascaded transformer modules, each including an attention mechanism module and a feedforward neural network module; at least one transformer module includes an attention head pruning module, a token pruning module, a feedforward neural network sparse module, and an early exit module. The input data for the large language model is text data; the method includes: For each attention head in the transformer module, the importance index corresponding to the attention head is calculated based on the current token, and the target attention head is filtered by the attention head pruning module according to the importance index; Based on the target attention head, the attention weight between the current token and each historical token is calculated. Then, the token pruning module filters the target historical tokens corresponding to the target attention head from multiple historical tokens based on the attention weights. Using the target attention head, reason about the current token based on the target historical token to obtain the attention output of the current token; The activation value corresponding to the attention output is calculated using the first fully connected layer of the feedforward neural network module, and the target neuron is selected based on the activation value by the sparse module of the feedforward neural network. The activation value corresponding to the target neuron is used as the input of the second fully connected layer to obtain the intermediate inference result output by the transformer module; The early exit module calculates the similarity between the intermediate inference result and the intermediate inference result output by the previous level transformer module. If the similarity is greater than a preset similarity threshold, the subsequent transformer module will no longer be used for inference. Instead, the intermediate inference result of the current transformer module will be used as the inference result of the last transformer module for subsequent inference to obtain the final inference result output by the large language model.

8. A large language model reasoning device, characterized in that, The large language model includes multiple cascaded transformer modules, each transformer module includes an attention mechanism module, and the attention mechanism module includes multiple attention heads. The input data of the large language model is text data; the device includes: An attention metric calculation module is used to calculate the importance metric corresponding to each attention head in at least one of the transformer modules, based on the current token. An attention head pruning module is used to filter target attention heads from a plurality of attention heads based on the importance index; An attention calculation module is used to reason about the current token based on the target attention head to obtain the intermediate reasoning result of the transformer module; The reasoning module is used to perform subsequent reasoning based on the intermediate reasoning results to obtain the final reasoning result output by the large language model. The attention calculation module is specifically used for: For each target attention head, calculate the attention weight between the current token and each historical token; Based on the attention weight, the target historical token corresponding to the target attention head is selected from multiple historical tokens; Based on the target history token, the context representation output by the target attention head is calculated using the corresponding target attention head; The attention output of the attention mechanism module is obtained based on the contextual representation of all the target attention heads; Reasoning is performed based on the attention output to obtain the intermediate reasoning result; The transformer module further includes a feedforward neural network module, which comprises a first fully connected layer and a second fully connected layer; the inference based on the attention output to obtain the intermediate inference result includes: The activation value corresponding to the attention output is calculated using the first fully connected layer. Calculate the performance index of each neuron in the feedforward neural network module based on the activation value; Target neurons are selected based on the aforementioned performance metrics; The intermediate inference result is obtained by using the activation value corresponding to the target neuron as the input to the second fully connected layer; The reasoning module is specifically used for: If the similarity between the intermediate inference result of the current transformer module and the intermediate inference result of the previous transformer module is greater than a preset similarity threshold, then subsequent transformer modules will no longer be used for inference. Instead, the intermediate inference result of the current transformer module will be used as the inference result of the last transformer module for subsequent inference, thereby obtaining the final inference result output by the large language model.

9. An electronic device, characterized in that, include: Processor, memory, and bus, among which: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-7 by calling the program instructions.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-7.

Images