Accelerated reasoning method of large language model and computing equipment

Abstract

The invention relates to a data processing technology, in particular to an accelerated reasoning method of a large language model, computing equipment for implementing the method, a non-instantaneous computer readable storage medium and a computer program product. According to one aspect of the invention, an accelerated reasoning method of a large language model is provided, the large language model comprises a plurality of sub-layers connected in a cascade mode, the sub-layers are multi-head attention layers or feedforward neural network layers, and the method comprises the following steps: A, determining a corresponding sparsity for each sub-layer by using a dynamic sparse optimizer; b, the reasoning request is processed through sparse sub-layers, and the sparse mode is that for each sub-layer, a dynamic sparse predictor associated with the sub-layer is used for predicating the reasoning request through the dynamic sparse predictor associated with the sub-layer; an attention head or neuron that needs to be activated is determined based on the sparseness of the sub-layer and the context to a previous sub-layer input to the sub-layer and the determined attention head or neuron is activated.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to an accelerated reasoning method for a large language model, a computing device for implementing the method, a non-transitory computer-readable storage medium, and a computer program product. Background Art

[0002] Large language models have demonstrated excellent performance on tasks such as text generation, question answering, and text translation. As the scale increases, the number of model parameters and context length also increase, making it possible to capture and understand more complex language patterns. However, as the model size increases, traditional dense computing methods become inefficient, resulting in increased inference latency and a sharp increase in resource consumption.

[0003] In order to reduce the amount of computation and memory usage of the model during the inference phase, dynamic sparsification technology can be introduced. This technology uses the sparse pattern learning mechanism to identify the most important inference units (such as neurons and attention heads) for the current input and dynamically adjusts the set of activated inference units to achieve higher computational efficiency. However, the introduction of sparsification technology will inevitably reduce the performance of the model, so it is crucial to select appropriate activated units to strike a balance between reducing computing resource consumption and avoiding excessive degradation of model performance. Summary of the invention

[0004] One object of the present application is to provide a method for accelerating reasoning of a large language model, a computing device for implementing the method, a non-transitory computer-readable storage medium, and a computer program product, which can reduce the computing resource consumption during reasoning of a large language model while ensuring that the model has higher performance.

[0005] According to one aspect of the present application, a method for accelerating reasoning of a large language model is provided, wherein the large language model comprises a plurality of sub-layers connected in a cascade manner, the sub-layers being a multi-head attention layer or a feedforward neural network layer, and the method comprising:

[0006] A. Use the dynamic sparse optimizer to determine the corresponding sparsity for each sub-layer;

[0007] B. Use the sparse sub-layers to process the inference request, where the sparse method is: for each sub-layer, use the dynamic sparse predictor associated with the sub-layer to determine the attention head or neuron that needs to be activated based on the sparsity of the sub-layer and the context of the previous sub-layer input to the sub-layer, and activate the determined attention head or neuron.

[0008] Optionally, in the above-mentioned accelerated inference method of the large language model, in step A, the dynamic sparse optimizer determines the sparsity of each sublayer based on the overall sparsity of the large language model and the importance of each sublayer.

[0009] Further optionally, in the above-mentioned accelerated reasoning method for a large language model, the overall sparsity is determined according to the type of the current reasoning task and the computing resources provided by the computing device that performs the reasoning task.

[0010] Further optionally, in the above-mentioned accelerated inference method of the large language model, the importance is measured by at least one of the following items: the type of reasoning task performed, the contribution of each sub-layer to the performance of the large language model.

[0011] Further optionally, in the above-mentioned accelerated inference method of the large language model, calibration data is used to determine the importance, and data generated during the training process or inference process of the large language model is used as the calibration data.

[0012] Further optionally, in the above-mentioned accelerated inference method of a large language model, the dynamic sparse optimizer uses one of the following items to determine the sparsity of each sub-layer: a dynamic programming algorithm, a greedy algorithm, a divide-and-conquer algorithm, and a branch-and-bound algorithm.

[0013] Optionally, in the above-mentioned accelerated inference method of a large language model, the dynamic sparse predictor is a neural network model including multiple fully connected layers, and the data generated during the training process or inference process of the large language model is used as training data of the neural network model.

[0014] According to another aspect of the present application, there is provided a computing device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively coupled to the at least one processor,

[0017] The memory stores a large language model and a computer program / instruction for accelerating reasoning of the large language model, wherein the large language model comprises a plurality of sub-layers connected in a cascade manner, wherein the sub-layers are multi-head attention layers or feed-forward neural network layers,

[0018] The computer program / instructions are executed by the at least one processor to implement the following operations:

[0019] A. Use the dynamic sparse optimizer to determine the corresponding sparsity for each sub-layer;

[0020] B. Use the sparse sub-layers to process the inference request, where the sparse method is: for each sub-layer, use the dynamic sparse predictor associated with the sub-layer to determine the attention head or neuron that needs to be activated based on the sparsity of the sub-layer and the context of the previous sub-layer input to the sub-layer, and activate the determined attention head or neuron.

[0021] Optionally, in the above computing device, in operation A, the dynamic sparse optimizer determines the sparsity of each sub-layer based on the overall sparsity of the large language model and the importance of each sub-layer.

[0022] Further optionally, in the above computing device, the overall sparsity is determined according to the type of the current reasoning task and the computing resources provided by the computing device.

[0023] Further optionally, in the above-mentioned computing device, the importance is measured by at least one of the following items: the type of reasoning task performed, the contribution of each sub-layer to the performance of the large language model.

[0024] Further optionally, in the above-mentioned computing device, calibration data is used to determine the importance, and data generated during the training process or the inference process of the large language model is used as the calibration data.

[0025] Further optionally, in the above-mentioned computing device, the dynamic sparse optimizer uses one of the following items to determine the sparsity of each sub-layer: a dynamic programming algorithm, a greedy algorithm, a divide-and-conquer algorithm, and a branch-and-bound algorithm.

[0026] Optionally, in the above-mentioned computing device, the dynamic sparse predictor is a neural network model including multiple fully connected layers, and the training data of the neural network model is taken from the data generated during the training process or inference process of the large language model.

[0027] Further optionally, in the above-mentioned computing device, the dynamic sparse optimizer or the dynamic sparse predictor is implemented by dedicated hardware in the computing device, or is implemented by executing the computer program / instructions on the at least one processor.

[0028] According to another aspect of the present application, a non-transitory computer-readable storage medium is provided, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the accelerated reasoning method for the large language model as described above are implemented.

[0029] According to another aspect of the present application, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for accelerating reasoning of a large language model as described above.

[0030] In some embodiments of the present application, by considering the overall sparsity and the importance of the sub-layer as a whole, the optimization effect of the sparsity allocation can be improved. In other embodiments, by dynamically adjusting the overall sparsity based on the task type and / or the currently available computing resources, the utilization efficiency of the computing resources is improved. In some other embodiments, by comprehensively considering the type of reasoning task and the contribution of the sub-layer, the importance of each sub-layer in the reasoning process can be more accurately described. In addition, considering the type of reasoning task in determining the overall sparsity and the importance of each sub-layer can unexpectedly improve the optimization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and / or other aspects and advantages of the present application will become clearer and easier to understand through the following description of various aspects in conjunction with the accompanying drawings, in which the same or similar units are represented by the same reference numerals. The accompanying drawings include:

[0032] Figure 1 A schematic diagram of a large language model accelerated reasoning architecture according to an embodiment of the present application.

[0033] Figure 2 A schematic diagram of a large language model based on the Transformer architecture.

[0034] Figure 3 A schematic diagram of accelerating the reasoning process of a large language model according to another embodiment of the present application.

[0035] Figure 4 The present invention is a flowchart of a method for accelerating reasoning of a large language model according to another embodiment of the present application.

[0036] Figure 5 is a schematic block diagram of an exemplary computing device that can be used to implement embodiments of the present application. DETAILED DESCRIPTION

[0037] The present application is described more fully below with reference to the accompanying drawings in which illustrative embodiments of the present application are illustrated. However, the present application may be implemented in different forms and should not be interpreted as being limited to the embodiments given herein. The above embodiments are given to make the disclosure herein comprehensive and complete, so as to more fully convey the scope of protection of the present application to those skilled in the art.

[0038] In this specification, terms such as "comprise" and "include" indicate that in addition to the units and steps directly and explicitly stated in the specification and claims, the technical solution of the present application does not exclude the situation where there are other units and steps that are not directly or explicitly stated.

[0039] Unless otherwise specified, terms such as "first" and "second" do not indicate the order of the units in terms of time, space, size, etc., but are merely used to distinguish the units.

[0040] Figure 1 This is a schematic diagram of a large language model accelerated reasoning architecture according to an embodiment of the present application, which can realize accelerated reasoning of a large language model. Figure 1 The illustrated architecture 110 includes a large language model inference module 111, a dynamic sparse optimizer 112, and a dynamic sparse predictor group 113. An inference request 120 is input into the large language model inference module 111, and an inference result 130 is obtained after processing. Figure 1 In the architecture shown, the large language model inference module 111 is dynamically sparsified using a dynamic sparse optimizer 112 and a dynamic sparse predictor 113. It should be noted that dynamic sparsification can be applied to all inference units in the large language model module 111 (for example, sub-layers such as multi-head attention layers, masked multi-head attention layers, and encoder-decoder multi-head attention layers to be described below), or only to some inference units, and the following description applies to both situations.

[0041] Figure 2 A schematic diagram of a large language model based on the Transformer architecture that can be used to implement Figure 1 The large language model inference module 111 is shown. The large language model based on the Transformer architecture shown includes an encoding component 201, a decoding component 202 and the connection relationship between them.

[0042] like Figure 2 As shown, the encoding component 201 consists of N encoder layers, each of which mainly includes two cascade-connected sub-layers: a multi-head attention layer and a feedforward neural network layer. The decoding component consists of N decoder layers, each of which mainly includes three cascade-connected sub-layers: a masked multi-head attention layer, an encoder-decoder multi-head attention layer, and a feedforward neural network layer. Figure 2 The encoder layer and decoder layer in the Transformer architecture shown are collectively referred to as Transformer units. In addition, unless otherwise specified, in the following description, the multi-head attention layer can also refer to the masked multi-head attention layer and the encoder-decoder multi-head attention layer in the decoder layer.

[0043] It should be noted that Figure 2 The symbol “N×” in represents a multi-layer structure in which the output of the previous encoder layer is the input of the next encoder layer, and the output of the previous decoder layer is the input of the next decoder layer. In other words, Figure 2In the Transformer architecture shown, the Transformer units in the encoding component 201 and the decoding component 202 are connected in a cascade manner.

[0044] See also Figure 2 , the embedding layer finds a corresponding embedding vector for each token. The token is then positionally encoded to mark the relative position of the token. The embedding vector along with the position encoding is then fed into the multi-head attention layer of the encoder layer, which allows the model to focus on different parts of the input sequence at the same time. Figure 2 In the encoder layer of , the output of the multi-head attention layer is added to the input (residual connection) and then normalized, which helps to avoid the problem of vanishing gradients. The normalized output of the multi-head attention layer is then sent to the feedforward neural network layer of the same encoder layer. The feedforward neural network layer applies a nonlinear activation layer to each embedding vector separately. When processing the input sequence layer by layer using multiple encoder layers, each layer can transform the embedding vector through the self-attention mechanism and the feedforward neural network to capture the dependencies within the sequence.

[0045] exist Figure 2 In the example, the masked multi-head attention layer in the decoder layer ensures that each embedding vector only focuses on earlier positions to prevent information from future tokens from leaking into the current position. In addition, the decoder layer uses a cross-attention mechanism (encoder-decoder multi-head attention layer) to focus on the embedding vector output by the decoder to align the context of the source language with the generated target language. Further, the embedding vector output by the decoding component is converted to the predicted probability of the next word in the target sequence through the SoftMax layer. The final output of the decoding component is converted into a vocabulary to form the prediction sequence of the model.

[0046] Back to Figure 1 In this embodiment, the dynamic sparse optimizer 112 is used to determine the corresponding sparsity for the multi-head attention layer and the feedforward neural network in the Transformer unit. In particular, for important sub-layers, the dynamic sparse optimizer 112 will assign lower sparsity to them (i.e., assign more attention heads or neurons to participate in the reasoning process) to maintain their contribution to the model performance; and for sub-layers with low importance, higher sparsity is assigned (i.e., assigning fewer attention heads or neurons to participate in the reasoning process) to reduce the amount of calculation.

[0047] In this embodiment, the dynamic sparse optimizer 112 is based on the large language model inference module 111 (for example Figure 2The corresponding sparsity is determined by the overall sparsity of the large language model (such as the large language model shown in the figure) and the importance of each sublayer of the Transformer unit in the reasoning process. The overall sparsity mentioned here refers to the proportional relationship between non-zero elements (such as attention heads and neurons involved in the reasoning process) and zero elements (such as attention heads and neurons not involved in the reasoning process) for all elements of the large language model (such as all attention heads and neurons contained in the model). The sparsity of the sublayer mentioned here refers to the proportional relationship between non-zero elements (such as attention heads and neurons involved in the reasoning process) and zero elements (such as attention heads and neurons not involved in the reasoning process) for all elements of the sublayer (such as attention heads in the multi-head attention layer and neurons in the feedforward neural network layer).

[0048] When allocating the sparsity of each sub-layer in the Transformer unit, by considering the overall sparsity and the importance of the sub-layer as a whole, a more optimized sparsity allocation result can be obtained, thereby achieving a better balance between model performance and inference efficiency.

[0049] The overall sparsity can be input by the user, or determined based on the type of reasoning task currently being performed (e.g., text translation, question answering, and text content summarization, etc.) and the computing resources provided by the computing device performing the reasoning task (e.g., computing power, storage space, and data transmission bandwidth, etc.). Dynamically adjusting the overall sparsity based on the task type and / or currently available computing resources is beneficial for optimizing sparsity allocation and improving computing resource utilization efficiency.

[0050] Typically, the sublayers of the Transformer unit are used for large language models (e.g. Figure 2 The influence of the output results or performance of the model shown in the figure is different. Some sub-layers have a greater influence because they can capture the features or patterns that are critical to the reasoning task. In addition, the inventors of the present applicant have found that in Figure 2In the deep learning model shown, different sublayers play different roles in feature extraction, some sublayers are better at capturing local features, while others are more sensitive to global features; and in sequence processing, different sublayers also have different efficiencies in information retention. The above characteristics mean that the importance of each sublayer may vary depending on the reasoning task. In some specific embodiments of the present application, a single parameter or a combination of multiple parameters is used to measure or characterize the aforementioned importance. These parameters include, for example, but are not limited to: the type of reasoning task currently being performed (such as text translation, question answering, and text content summary, etc.) and the contribution of each sublayer to the performance of the large language model. By introducing a combination of the type of reasoning task and the contribution of the sublayer, the importance of each sublayer in the reasoning process can be more accurately described. It should be noted that it is generally recognized that it is at least unnecessary to consider the type of reasoning task in determining the overall sparsity and the importance of each sublayer. This approach not only increases complexity, but also weakens the optimization effect of sparsity allocation in some cases. However, the research of the inventors of the present application shows that the above-mentioned dual use of the reasoning task type will unexpectedly improve the optimization effect, that is, compared with the situation where the reasoning task type is only considered in determining the overall sparsity or determining the importance of each sub-layer, for the same reasoning task, the above-mentioned dual use method enables the sparse large language model to complete the reasoning process at a faster speed and obtain better reasoning results.

[0051] In some embodiments, a calibration dataset (e.g. Figure 1 The importance of each sublayer to the performance of the large language model is determined by using a calibration data set 140 in the training dataset. Exemplarily, the importance of each sublayer can be determined by disabling or weakening each sublayer or a combination of multiple sublayers and evaluating the change in model performance on the calibration data set. As another example, the contribution of each sublayer to model performance can be evaluated by analyzing the gradient size of each sublayer during training. As another example, model interpretability tools such as SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) can be used to evaluate the contribution of each sublayer to model performance. As another example, the influence of each sublayer on information flow can be evaluated by statistically analyzing the changes in activation values ​​before and after layer normalization of each sublayer, thereby inferring its importance.

[0052] Calibration datasets are usually small datasets used to calibrate or fine-tune large language models, which help the model achieve better performance on specific tasks. There are many ways to build calibration datasets, such as using public data resources as calibration datasets, building calibration datasets based on specific reasoning task requirements, and using the data output by large language models during the pre-training model process or reasoning process as calibration datasets.

[0053] The determination of the sparsity of the sub-layer involves problems such as optimal resource allocation under limited resources and optimal solution under multiple constraints. To this end, when determining the corresponding sparsity for the sub-layers in the Transformer unit, the dynamic sparse optimizer 112 can use a dynamic programming algorithm, which decomposes a complex problem into simpler sub-problems and uses the solutions of these sub-problems to construct a solution to the original problem. In addition, other optimization algorithms are also available, such as but not limited to greedy algorithms, divide-and-conquer algorithms, and branch-and-bound algorithms.

[0054] Continue to see Figure 1 In this embodiment, after determining the sparsity of each sub-layer of the Transformer unit, the dynamic sparse predictor group 113 can be used to determine the attention heads or neurons that need to be activated for each sub-layer. The dynamic sparse predictor group 113 includes a plurality of dynamic sparse predictors, each of which is associated with one of the sub-layers of the large language model and is configured to determine the attention heads or neurons that need to be activated for the associated sub-layer.

[0055] Figure 3 FIG. 1 is a schematic diagram of accelerating the reasoning process of a large language model according to another embodiment of the present application. Figure 3 In the above example, Attention_k and FFN_k represent large language models (e.g. Figure 1 The multi-head attention layer and feedforward neural network layer in the k-th Transformer unit in the large language model inference module 111) shown in the figure, Attention_k+1 and FFN_k+1 respectively represent the multi-head attention layer and feedforward neural network layer in the (k+1)-th Transformer unit, P Attention_k , P FFN_k , P Attention_k+1 and P FFN_k+1 Respectively represent the dynamic sparse predictors associated with the multi-head attention layer Attention_k, the feedforward neural network layer FFN_k, the multi-head attention layer Attention_k+1, and the feedforward neural network layer FFN_k+1. For example, Figure 3Each sublayer in is subdivided into multiple grids, which represent the elements contained in the sublayer, i.e., attention heads or neurons, where gray grids represent activated elements and white grids represent inactivated elements. Figure 3 In the example, the Transformer unit represented as an encoder layer is only illustrative. In some other examples, the Transformer unit can also be a decoder. For example, the sublayer Attention_k can be a masked multi-head attention layer, and the sublayers FFN_k and Attention_k+1 are replaced by an encoder-decoder multi-head attention layer and a feedforward neural network layer, respectively. It should also be noted that for the sake of simplicity, model elements such as residual connections and layer normalization are not included in this paper. Figure 3 Appeared in.

[0056] See also Figure 3 , the dynamic sparse predictor P associated with the multi-head attention layer Attention_k Attention_k Based on the context Context_k-1 and the sparsity S_k, predict or determine which attention heads in the multi-head attention layer Attention_k will be activated (e.g. Figure 3 The gray grid in represents the activated attention head), where Context'_k-1 is the input to the previous sublayer (in Figure 3 In the example of , the previous sublayer is the context of the feedforward neural network layer in the (k-1)th Transformer unit), and S_k is the sparsity of the multi-head attention layer Attention_k determined by the dynamic sparse optimizer 102. Exemplarily, the dynamic sparse predictor P Attention_k The prediction results can be output in the form of configuration parameters (in Figure 3 China-Israel Q k , where the configuration parameter defines the attention heads that need to be activated in the multi-head attention layer Attention_k. Accordingly, by activating the attention heads defined by the configuration parameter Q in the multi-head attention layer Attention_k k The defined attention heads are used to sparse the multi-head attention layer Attention_k, so that the sparse multi-head attention layer Attention_k can be used to process the context Context_k from the (k-1)th Transformer unit.

[0057] Similarly, the dynamic sparse predictor P associated with the feed-forward neural network layer FFN_k FFN_k Based on the context Context_k and the sparsity S'_k, predict or determine which neurons in the feedforward neural network layer FFN_k will be activated (e.g. Figure 3The gray grid in represents the activated neurons), where Context_k is the input to the previous sublayer (in Figure 3 In the example of , the previous sublayer is the context of the multi-head attention layer Attention_k) in the k-th Transformer unit, and S'_k is the sparsity of the feedforward neural network layer FFN_k determined by the dynamic sparse optimizer 102. Dynamic sparse predictor P FFN_k The prediction results can be output in the form of configuration parameters (in Figure 3 China-Israel Q' k ), wherein the configuration parameter defines the neurons activated in the feedforward neural network layer FFN_k. Accordingly, the feedforward neural network layer FFN_k is composed of the configuration parameter Q' k The defined neurons are activated, so the context Context'_k from the multi-head attention layer Attention_k will be processed by the sparse feed-forward neural network layer FFN_k.

[0058] By analogy, other sub-layers and their associated dynamic sparse predictors also adopt similar working principles, so they will not be described here.

[0059] In some specific embodiments, the dynamic sparse predictor 113 can be a neural network model including multiple fully connected layers (also called dense layers), in which each neuron is connected to all neurons in the previous layer. It should be noted that neural network models of other structures are also available, for example, including but not limited to convolutional neural networks, recurrent neural networks, and long short-term memory networks. In some other specific embodiments, the data generated during the training process or reasoning process of the large language model can be used as training data for the dynamic sparse predictor 130.

[0060] In this embodiment, optionally, the dynamic sparse optimizer 112 and the dynamic sparse predictor 113 can be implemented by a software method. Alternatively, the dynamic sparse optimizer 112 and the dynamic sparse predictor 113 can be implemented by a hardware method, and the hardware includes, for example, but is not limited to, a graphics processing unit (GPU), a tensor processing unit (TPU), and a neural network processing unit (NPU). In addition, the above software method and hardware method can also be used in combination to achieve better sparseness effect and performance improvement.

[0061] Figure 4 FIG. 1 is a flowchart of a method for accelerating reasoning of a large language model according to another embodiment of the present application. For exemplary purposes, the following description will sometimes refer to Figure 1-3The present invention is carried out in the context of the large language model shown in the figure, and thus may include various features of the above-mentioned embodiments. However, it should be noted that the spirit and principle of the described method can be applied to a variety of large language models and are not limited to the specific details of the large language model described above. Exemplarily, the embodiments described below are applied to such a large language model, which includes a plurality of sub-layers L1 ... L1 connected in a cascade manner. n , these sub-layers can be, for example, multi-head attention layers or feed-forward neural network layers.

[0062] Figure 4 The method shown includes the following steps:

[0063] Step 410: Using a dynamic sparse optimizer (e.g. Figure 1 The dynamic sparse predictor 113 in the example is a multiple sub-layer L1...L n Determine the corresponding sparsity S1…S n The specific method for determining the sparsity has been described in detail above and will not be repeated here.

[0064] Step 420: Using a large language model (e.g. Figure 1 The large language model inference module 111) in the process processes the inference request.

[0065] This step includes a plurality of sub-processes 420_1 ... 420_n executed in sequence, each of which corresponds to a plurality of sub-layers L1 ... L n One of the processing processes. i (where i=1...n) as an example, the sublayer L i The associated dynamic sparse predictor is based on the sublayer L i The sparsity S i (which is determined by the dynamic sparse optimizer in step 410) and the previous sub-layer L input to this sub-layer i-1 Context i-1 To determine the sublayer L i The attention heads or neurons that need to be activated in the sub-layer L are realized by activating the attention heads or neurons determined by the dynamic sparse predictor. i The sparseness of the sub-layer L i To process the context of the sublayer i The method for determining the attention heads or neurons that need to be activated has been described in detail above and will not be repeated here.

[0066] It should be noted that the above Figure 1-4The described sparsification method can be applied to all sub-layers in a large language model, or only to some sub-layers (for example, only the sub-layers in the encoding component).

[0067] Figure 5 is a block diagram of an exemplary computing device that can be used to implement the embodiments of the present application. The computing device described herein covers various forms of devices with computing functions, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, mobile phones, tablet computers, and wearable devices.

[0068] like Figure 5 As shown, the computing device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. Various programs, data, and other data required for the operation of the computing device 500 can also be stored in the RAM 503. Figure 1-3 The computing unit 501 , the ROM 502 , and the RAM 503 are connected to each other via a bus 504 . An input / output (I / O) interface 505 is also connected to the bus 504 .

[0069] A number of components in the computing device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the computing device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0070] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods described above. For example, in some embodiments, the accelerated reasoning method of the large language model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method for training a generative large language model and / or the method for processing an image task described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the accelerated inference method of the large language model in any other appropriate manner (for example, by means of firmware).

[0071] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0073] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0074] Although specific embodiments of the present application are described in detail herein, they are provided for the purpose of explanation only and should not be considered to limit the scope of the present application. Various alternatives and modifications may be proposed without departing from the core and scope of the present application.

Claims

1. A method for accelerating reasoning of a large language model, wherein: The large language model comprises a plurality of sub-layers connected in a cascade manner, wherein the sub-layers are multi-head attention layers or feed-forward neural network layers, and the method comprises: A. Use the dynamic sparse optimizer to determine the corresponding sparsity for each sub-layer; B. Use the sparse sub-layers to process the inference request, where the sparse method is: for each sub-layer, use the dynamic sparse predictor associated with the sub-layer to determine the attention head or neuron that needs to be activated based on the sparsity of the sub-layer and the context of the previous sub-layer input to the sub-layer, and activate the determined attention head or neuron.

2. The accelerated reasoning method for a large language model as claimed in claim 1, wherein: In step A, the dynamic sparse optimizer determines the sparsity of each sub-layer based on the overall sparsity of the large language model and the importance of each sub-layer.

3. The accelerated reasoning method for a large language model as claimed in claim 2, wherein: The overall sparsity is determined according to the type of the current reasoning task and the computing resources provided by the computing device that executes the reasoning task.

4. The accelerated reasoning method for a large language model as claimed in claim 2, wherein: The importance is measured by at least one of the following items: the type of reasoning task performed, and the contribution of each sub-layer to the performance of the large language model.

5. The accelerated reasoning method for a large language model as claimed in claim 2, wherein: The importance is determined using calibration data, and data generated during the large language model training or reasoning process is used as the calibration data.

6. The accelerated reasoning method for a large language model as claimed in claim 2, wherein: The dynamic sparse optimizer uses one of the following to determine the sparsity of each sub-layer: a dynamic programming algorithm, a greedy algorithm, a divide-and-conquer algorithm, and a branch-and-bound algorithm.

7. The accelerated reasoning method for a large language model as claimed in claim 1, wherein: The dynamic sparse predictor is a neural network model including multiple fully connected layers, and uses the data generated during the training process or inference process of the large language model as training data of the neural network model.

8. A computing device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor, The memory stores a large language model and a computer program / instruction for accelerating reasoning of the large language model, wherein the large language model comprises a plurality of sub-layers connected in a cascade manner, wherein the sub-layers are multi-head attention layers or feed-forward neural network layers, The computer program / instructions are executed by the at least one processor to implement the following operations: A. Use the dynamic sparse optimizer to determine the corresponding sparsity for each sub-layer; B. Use the sparse sub-layers to process the inference request, where the sparse method is: for each sub-layer, use the dynamic sparse predictor associated with the sub-layer to determine the attention head or neuron that needs to be activated based on the sparsity of the sub-layer and the context of the previous sub-layer input to the sub-layer, and activate the determined attention head or neuron.

9. A non-transitory computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method for accelerating reasoning of a large language model as described in any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method for accelerating reasoning of a large language model according to any one of claims 1 to 7.

Images