A data reasoning method and apparatus
By segmenting and configuring the transformer unit of the multi-head attention mechanism of a large language model across multiple computing chips for parallel inference, the problem of slow inference speed caused by insufficient cache capacity is solved, improving efficiency and reducing hardware costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN MICROBT ELECTRONICS TECH CO LTD
- Filing Date
- 2024-04-29
- Publication Date
- 2026-07-28
AI Technical Summary
The reasoning speed of large language models is relatively slow, mainly because the model's weight parameter data is huge and the cache capacity of the computing chip is small, which leads to an increase in the time overhead of inputting weight parameters into the computing chip cache.
The transformer unit of the multi-head attention mechanism of the large language model is divided into a preset number of transformer sub-units, and these sub-units are configured in a preset number of computing chips. Parallel inference is performed through multiple computing chips, and the parameter cache capacity of multiple computing chips is used to accumulate the inference results of the multi-head attention mechanism.
It improves the inference speed of large language models, reduces hardware costs, and makes it easier to support larger amounts of parameter data by expanding the number of computing chips when the amount of parameter data increases.
Smart Images
Figure CN120893558B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of neural network technology, and in particular to a data reasoning method and apparatus. Background Technology
[0002] Large language models are an advanced artificial intelligence technology trained using deep learning algorithms. These models have demonstrated enormous potential in fields such as natural language processing, machine translation, and dialogue systems, achieving remarkable results in many tasks. First, their performance in natural language processing has broken through previous limitations, enabling machines to make significant progress in understanding and generating human language. This is of great significance for applications such as machine translation, text summarization, and question-answering systems. Second, large language models possess powerful expressive capabilities in reasoning and decision-making processes, providing users with more accurate and intelligent responses and suggestions. This makes interactive applications such as dialogue systems and virtual assistants more powerful and practical.
[0003] However, a major problem faced by current large language models during inference is their slow inference speed. A primary reason for this slowness is the massive amount of data involved in the model's weight parameters and the relatively small cache capacity of the computing chip, leading to increased time overhead in inputting the weight parameters into the chip's cache. Summary of the Invention
[0004] In view of this, this disclosure provides a data reasoning method and apparatus that combines the characteristics of multi-head reasoning in the multi-head attention mechanism of large language models to achieve parallel reasoning using multiple computing chips and improve the reasoning speed of large language models.
[0005] The technical solution disclosed herein is implemented as follows:
[0006] According to one aspect of the embodiments of this disclosure, a data reasoning method is provided, including:
[0007] A transformer unit for obtaining the multi-head attention mechanism of a large language model, wherein the transformer unit includes multiple heads;
[0008] The converter unit is divided into a predetermined number of converter sub-units, wherein each converter sub-unit includes at least one head;
[0009] A preset number of the converter sub-units are respectively configured in a preset number of computing chips, wherein each computing chip is configured with one converter sub-unit;
[0010] During the reasoning process of the large language model, the data to be reasoned in the reasoning process of the multi-head attention mechanism is input into a preset number of computing chips, and a preset number of temporary results of multi-head attention mechanism reasoning are obtained through the reasoning of the preset number of computing chips.
[0011] Based on a preset number of temporary inference results from the multi-head attention mechanism, the inference result of the multi-head attention mechanism is obtained.
[0012] In one possible implementation, the data reasoning method further includes:
[0013] Obtain the total number of heads in the converter unit used and the amount of data for the parameters of each head;
[0014] Obtain the parameter cache capacity of a single computing chip;
[0015] The preset number is obtained based on the total number of heads in the converter unit, the data volume of the parameters of each head, and the parameter cache capacity of a single computing chip;
[0016] The parameter cache capacity of each of the computing chips is not less than the data volume of the parameters of the configured converter subunit.
[0017] In one possible implementation, the data reasoning method further includes:
[0018] Establish a mapping relationship between the converter subunit and the computing chip.
[0019] In one possible implementation, the step of configuring a preset number of the converter sub-units into a preset number of computing chips includes:
[0020] According to the mapping relationship, each converter subunit is configured in the corresponding computing chip.
[0021] In one possible implementation, obtaining the inference result of the multi-head attention mechanism based on a preset number of temporary inference results includes:
[0022] The temporary inference results of the multi-head attention mechanism associated with each of the transformer sub-units are accumulated to obtain the inference result of the multi-head attention mechanism.
[0023] In one possible implementation, the step of inputting the data to be reasoned in the inference process of the multi-head attention mechanism into a preset number of computing chips, and obtaining a preset number of temporary inference results of the multi-head attention mechanism through the inference of the preset number of computing chips, includes:
[0024] The data to be inferred is synchronously input into a preset number of the computing chips;
[0025] Each computing chip performs inference on the data to be inferred based on the parameters of the converter subunit configured in itself, and obtains a temporary inference result of the multi-head attention mechanism associated with the converter subunit configured in each computing chip.
[0026] In one possible implementation, the multi-head attention mechanism includes a self-attention phase and a feedforward network phase;
[0027] The reasoning process of the multi-head attention mechanism includes a sequentially executed self-attention reasoning process and a feedforward network reasoning process;
[0028] When the inference process of the multi-head attention mechanism is a self-attention inference process, each transformer subunit in the multi-head attention mechanism is a transformer subunit divided from the transformer unit of the self-attention inference process, the preset number of computing chips is a first number of first computing chips used in the self-attention inference process, each temporary inference result of the multi-head attention mechanism is a temporary self-attention inference result associated with each transformer subunit in the self-attention inference process, and the inference result of the multi-head attention mechanism is the self-attention inference result of the self-attention inference process.
[0029] When the inference process of the multi-head attention mechanism is a feedforward network inference process, each transformer subunit in the multi-head attention mechanism is a transformer subunit divided from the transformer unit of the feedforward network inference process, the preset number of computing chips is a second number of second computing chips used in the feedforward network inference process, the data to be inferred is the final inference result of the self-attention inference process, each temporary inference result of the multi-head attention mechanism is a feedforward network inference temporary result associated with each transformer subunit in the feedforward network inference process, and the inference result of the multi-head attention mechanism is the feedforward network inference result of the feedforward network inference process.
[0030] According to another aspect of the embodiments of this disclosure, a data inference apparatus is provided, comprising:
[0031] A transformer block acquisition module is configured to execute a transformer unit for acquiring a multi-head attention mechanism for a large language model, wherein the transformer unit includes at least one head.
[0032] The segmentation module is configured to perform the division of the converter unit into a preset number of converter sub-units, wherein each converter sub-unit includes at least one head;
[0033] The configuration module is configured to configure a preset number of the converter subunits into a preset number of computing chips, wherein each computing chip is configured with one converter subunit;
[0034] The data inference module is configured to, during the inference process of the large language model, input the data to be inferred in the inference process of the multi-head attention mechanism into a preset number of computing chips, and obtain a preset number of temporary inference results of the multi-head attention mechanism through the inference of the preset number of computing chips.
[0035] The result acquisition module is configured to execute a preset number of temporary inference results from the multi-head attention mechanism to obtain the inference result of the multi-head attention mechanism.
[0036] According to another aspect of the embodiments of this disclosure, an electronic device is provided, comprising:
[0037] processor;
[0038] Memory for storing the executable instructions of the processor;
[0039] The processor is configured to execute the executable instructions to implement the data inference method as described in any of the preceding claims.
[0040] According to another aspect of the present disclosure, a computer-readable storage medium is provided such that, when at least one instruction in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is able to implement the data reasoning method as described in any of the preceding claims.
[0041] As can be seen from the above scheme, the data inference method and apparatus disclosed herein, combined with the characteristics of multi-head inference in the multi-head attention mechanism of large language models, divides the transformer unit of the multi-head attention mechanism of the large language model into a preset number of transformer sub-units according to the parameters of the head, and configures these transformer sub-units in a preset number of computing chips. During the inference process, each computing chip infers the temporary inference result of the multi-head attention mechanism corresponding to each transformer sub-unit, and then accumulates these temporary inference results to obtain the inference result of the multi-head attention mechanism. In the inference process, parallel inference is achieved by utilizing a preset number of computing chips, which helps to improve the inference efficiency of large language models. Furthermore, the preset number of transformer sub-units divided according to the parameters of the head in the transformer block also takes into account the parameter cache capacity of the computing chips, so that the storage space of each computing chip is within a controllable range, which helps to ensure the yield of computing chips. Furthermore, based on the data inference method and apparatus disclosed herein, as the amount of data of the head parameters continues to increase, it is easy to support large language models with larger parameter data volumes by expanding the number of computing chips. The data reasoning method and apparatus disclosed herein help reduce the hardware cost of large language models and improve the reasoning speed of large language models. Attached Figure Description
[0042] Figure 1 This is a schematic flowchart illustrating a data reasoning method according to an illustrative embodiment;
[0043] Figure 2 This is a schematic flowchart illustrating the determination of a preset quantity according to an illustrative embodiment;
[0044] Figure 3 This is a schematic diagram illustrating a process for obtaining temporary inference results using a multi-head attention mechanism by utilizing a preset number of computing chips, according to an illustrative embodiment.
[0045] Figure 4A This is a schematic diagram of the reasoning process of a converter subunit in a computing chip regarding a self-attention reasoning process, according to an illustrative embodiment.
[0046] Figure 4B This is a schematic diagram illustrating the reasoning process of another converter subunit in another computing chip regarding the self-attention reasoning process, according to an illustrative embodiment.
[0047] Figure 5A This is a schematic diagram illustrating the inference process of a converter subunit in a computing chip regarding the inference process of a feedforward network, according to an illustrative embodiment.
[0048] Figure 5BThis is a schematic diagram illustrating the inference process of another converter subunit in another computing chip regarding the feedforward network inference process, according to an illustrative embodiment.
[0049] Figure 6 This is a schematic diagram of the logic structure of a converter block that implements the full functionality of four computing chips, according to an illustrative embodiment.
[0050] Figure 7 This is a schematic diagram of the logic structure of a data reasoning device according to an illustrative embodiment;
[0051] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments.
[0053] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. They are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0054] Current inference algorithms for large language models mostly utilize general-purpose GPUs (Graphics Processing Units). The internal cache of these GPUs, such as SRAM (Static Random-Access Memory), is relatively small. Each time a token from the input large language model is processed, the entire model's weights need to be copied piece by piece into the GPU's internal cache for computation. Taking the classic large language model llama270B (fp16) as an example, copying the entire model requires 2 * 70 GB (gigabytes) of bandwidth. If 100 tokens are processed per second, this requires 14,000 GB / s of bandwidth.
[0055] To address this issue, the current trend is to gradually increase the capacity of the cache (such as SRAM) inside computing chips (e.g., GPU chips), such as to several hundred MB (megabytes) or even several GB, to accommodate more parameters. This is helpful in improving bandwidth, but it inevitably cannot completely solve the problem. On the one hand, large language models continue to grow, requiring even larger cache capacities to accommodate more parameters. Obviously, the cache space of computing chips cannot be increased indefinitely. On the other hand, larger cache capacities mean a larger number of semiconductor devices contained in the computing chip, leading to increasingly higher requirements for semiconductor processes and greater challenges to the yield of computing chips, resulting in a continuous increase in chip costs.
[0056] The transformer block is the core component of the Transformer model. It consists of multiple sub-layers, including a self-attention layer and a feed-forward neural network layer. In the self-attention layer, each input vector performs attention calculations with itself to obtain dependencies between different positions in the input sequence. This attention mechanism captures contextual information in the input sequence and achieves position-independent encoding. The output of the self-attention layer is a linear combination of the input vectors, where the weights are obtained by calculating the similarity between the input vector and all other positional vectors. The feed-forward neural network layer is a fully connected feed-forward neural network that takes the output of the self-attention layer as input and performs a non-linear transformation using a multilayer perceptron.
[0057] In view of this, the present disclosure provides a data reasoning method and apparatus that, combined with the characteristics of multi-head reasoning in the multi-head attention mechanism of large language models, utilizes the head in the multi-head attention mechanism of large language models to decompose the reasoning process of the multi-head attention mechanism into multiple computing chips for execution, thereby solving the problem that the reasoning speed of large language models is slow due to insufficient cache capacity of a single computing chip.
[0058] In the transformer model, "head" refers to a branch of multi-head attention. When self-attention in the transformer model is extended to multi-head attention, it decomposes the representation of the input data into multiple distinct representation subspaces, each corresponding to a head. Each head performs self-attention computation independently, meaning they can focus on different parts of the input data or learn features from different perspectives. For example, one head might focus on capturing positional information in the sequence, while another head might focus on capturing syntactic structure or semantic information. In multi-head attention, the outputs of each head are summed and integrated through a linear layer to form the final output. This approach enhances the model's representational power, allowing it to simultaneously capture multiple complex dependencies in the sequence data.
[0059] Based on the characteristics of the head mentioned above, this disclosure provides a data reasoning method. Figure 1 This is a flowchart illustrating a data reasoning method according to an illustrative embodiment, such as... Figure 1 As shown, this data reasoning method mainly includes the following steps 101 to 105.
[0060] Step 101: Obtain the transformer unit of the multi-head attention mechanism of the large language model, wherein the transformer unit includes at least one head.
[0061] In the illustrative embodiment, the parameters of the head refer to the weight parameters associated with the head. The parameters of each head in the multi-head attention mechanism are a part of the many parameters of the large language model, which can be obtained from the storage medium storing the large language model by querying.
[0062] In an illustrative embodiment, a transformer unit is part of a transformer block. Since a transformer block includes a self-attention layer and a feedforward neural network layer, one transformer block contains two transformer units. The transformer unit can be a transformer unit associated with the self-attention inference process of the self-attention layer, or it can be a transformer unit associated with the feedforward network inference process of the feedforward neural network layer.
[0063] Step 102: Divide the converter unit into a preset number of converter sub-units, wherein each converter sub-unit includes at least one head.
[0064] In the illustrative embodiment, the preset number relates to the number of computing chips used in subsequent steps. Therefore, the preset number can be obtained before deploying the large language model and performing inference. In this way, after determining the preset number of computing chips, it is convenient to deploy the hardware system for executing the large language model and perform the inference of the large language model.
[0065] Figure 2 This is a schematic flowchart illustrating the determination of a preset quantity according to an illustrative embodiment, such as... Figure 2 As shown in the illustrative embodiment, the data reasoning method of this disclosure further includes the process of determining a preset quantity in steps 201 to 203.
[0066] Step 201: Obtain the total number of heads in the converter unit used and the amount of data for the parameters of each head.
[0067] In the illustrative embodiment, the total number of heads and the amount of data for the parameters of each head can be obtained from the large language model.
[0068] Step 202: Obtain the parameter cache capacity of a single computing chip.
[0069] In this illustrative embodiment, the computing chip is used to perform large language model inference. Before building the hardware system for executing the large language model, it is necessary to determine the required computing chip model and other specifications based on requirements and conditions, thereby obtaining the relevant cache capacity. In this illustrative embodiment, the parameter cache capacity may specifically refer to the cache capacity used to store the parameters of the converter subunit.
[0070] Step 203: Based on the total number of heads in the converter unit, the data volume of parameters of each head, and the parameter cache capacity of a single computing chip, obtain the preset quantity.
[0071] In this illustrative embodiment, the parameter cache capacity of each computing chip is no less than the data volume of the parameters of the configured converter subunit, thus ensuring that the computing chip can store the parameters of the converter subunit. In this illustrative embodiment, the data volume of the parameters of the converter subblock includes the data volume of the parameters of the head contained within the converter subblock. In this illustrative embodiment, each computing chip is configured with at least one converter subunit, thus ensuring that each computing chip can be used for data inference of large language models.
[0072] In an illustrative embodiment, the preset quantity can be estimated using the following formula:
[0073] X = A * B / C
[0074] Where X is the preset number of computing chips, A is the total number of heads in the converter units used, B is the data volume of parameters for each head, and C is the parameter cache capacity of the computing chips. To ensure that the computing chips have sufficient redundancy in their storage space for parameter updates of large language models, the actual parameter cache capacity of the computing chips can be appropriately reduced and used as C to estimate the preset number during estimation. This ensures that the computing chips have enough storage space to store the parameters of the converter subunits.
[0075] Where D = C / B, then D rounded down represents how many heads a chip can hold, which is the number of heads in a converter subunit; for X = A / D, then rounded up represents how many converter subunits a converter block is divided into.
[0076] Step 103: Configure a preset number of converter sub-units in a preset number of computing chips, wherein each computing chip is configured with one converter sub-unit.
[0077] In the illustrative embodiment, it is also necessary to determine the relationship between the computing chips and the converter sub-units. This ensures that each converter sub-unit is configured in a predetermined number of computing chips in a one-to-one correspondence. This also helps to determine the association between the temporary results of the multi-head attention mechanism inference and the converter sub-units, thereby avoiding deviations in the position of each temporary result of the multi-head attention mechanism inference within the overall multi-head attention mechanism inference result. Therefore, before step 103, the data inference method of this embodiment may further include: establishing a mapping relationship between the converter sub-units and the computing chips.
[0078] Based on this, in an illustrative embodiment, step 103 may specifically include: configuring each converter subunit in the corresponding computing chip according to the mapping relationship.
[0079] Step 104: During the reasoning process of the large language model, the data to be reasoned in the reasoning process of the multi-head attention mechanism is input into a preset number of computing chips. Through the reasoning of the preset number of computing chips, a preset number of temporary results of the multi-head attention mechanism reasoning are obtained.
[0080] In the illustrative embodiment, each computing chip uses the parameters of its assigned converter subunit to perform inference on the data to be inferred. Each computing chip is only responsible for the inference of its assigned converter subunit. The inference process between computing chips does not affect each other. In this way, multiple computing chips complete the inference process associated with different converter subunits (i.e., associated with different heads), realizing parallel inference between computing chips, accelerating the inference speed of the multi-head attention mechanism, and shortening the inference time of the multi-head attention mechanism.
[0081] Figure 3 This is a schematic diagram illustrating a process for obtaining temporary inference results using a multi-head attention mechanism by utilizing a preset number of computing chips, according to an illustrative embodiment. Figure 3 As shown, in the illustrative embodiment, step 104 may further include steps 301 to 302.
[0082] Step 301: Synchronously input the data to be inferred into a preset number of computing chips;
[0083] Step 302: Each computing chip performs inference on the data to be inferred according to the parameters of its own configured converter subunit, and obtains a temporary inference result of the multi-head attention mechanism associated with the converter subunit configured by each computing chip.
[0084] In an illustrative embodiment, the parameters of the converter subunit include the parameters of the head and other parameters of the converter unit besides the parameters of the head.
[0085] Step 105: Based on a preset number of multi-head attention mechanism inference temporary results, obtain the inference results of the multi-head attention mechanism.
[0086] Because during the inference process, the head in each converter subunit is only responsible for inferring the data to be inferred associated with that head, the resulting temporary inference result of the multi-head attention mechanism is also only associated with the data to be inferred associated with the head in that converter subunit. Therefore, the inference result of the multi-head attention mechanism can be obtained by directly using an accumulation method to process all the obtained temporary inference results of the multi-head attention mechanism. Based on this, in the illustrative embodiment, step 105 may specifically include: accumulating the temporary inference results of the multi-head attention mechanism associated with each converter subunit to obtain the inference result of the multi-head attention mechanism.
[0087] Existing large language models in the field of Natural Language Processing (NLP) include the Transformer model, and multi-head attention (M&A) is a core component of the Transformer model. Based on the Transformer model, M&A includes a self-attention stage and a feedforward network stage. The inference process of M&A includes sequentially executed self-attention inference and feedforward network inference. Therefore, in an illustrative embodiment, when the inference process of M&A is a self-attention inference process, each transformer subunit in the M&A is a transformer subunit divided from the transformer subunit of the self-attention inference process. A preset number of computing chips is a first number of first computing chips used for the self-attention inference process. Each temporary inference result of M&A is a temporary self-attention inference result associated with each transformer subunit in the self-attention inference process. The inference result of M&A is the self-attention inference result of the self-attention inference process. In an illustrative embodiment, when the inference process of the multi-head attention mechanism is a feedforward network inference process, each transformer subunit in the multi-head attention mechanism is a transformer subunit divided from the transformer unit of the feedforward network inference process. A preset number of computing chips is a second number of second computing chips used in the feedforward network inference process. The data to be inferred is the final inference result of the self-attention inference process. Each temporary inference result of the multi-head attention mechanism is a feedforward network inference result associated with each transformer subunit in the feedforward network inference process. The inference result of the multi-head attention mechanism is the feedforward network inference result of the feedforward network inference process. The final inference result of the self-attention inference process is obtained by adding the self-attention inference result to the data to be inferred. The transformer block of the multi-head attention mechanism includes transformer units for the self-attention inference process and transformer units for the feedforward network inference process. The inference result of the transformer block is obtained by adding the final inference result of the self-attention inference process to the feedforward network inference result.
[0088] Based on this, in the illustrative embodiment, step 103 may specifically include:
[0089] The parameters of each transformer subunit in the self-attention inference process are configured in a first number of first computing chips; and,
[0090] The parameters of each converter subunit in the feedforward network inference process are configured in a second number of second computing chips.
[0091] Based on this, in an illustrative embodiment, when the reasoning process of the multi-head attention mechanism is a self-attention reasoning process, step 104 may specifically include:
[0092] The normalized data to be inferred is input into a first number of first computing chips. Through the inference of the first number of first computing chips, the temporary self-attention inference results of each converter subunit associated with the self-attention inference process are obtained.
[0093] Based on this, in the illustrative embodiment, when the reasoning process of the multi-head attention mechanism is a self-attention reasoning process, step 105 may specifically include:
[0094] The temporary results of all heads associated with each transformer subunit in the self-attention reasoning process are summed to obtain the self-attention reasoning result. The self-attention reasoning result is then added to the data to be reasoned to obtain the final reasoning result of the self-attention reasoning process. The temporary results of all heads in each transformer subunit are the temporary self-attention reasoning results of that respective transformer subunit.
[0095] Based on this, in the illustrative embodiment, when the inference process of the multi-head attention mechanism is a feedforward network inference process, step 104 may specifically include:
[0096] The normalized self-attention reasoning process's final reasoning result is input into a second number of second computing chips. Through the reasoning of the second number of second computing chips, the feedforward network reasoning temporary results associated with each converter subunit of the feedforward network reasoning process are obtained.
[0097] Based on this, in the illustrative embodiment, when the inference process of the multi-head attention mechanism is a feedforward network inference process, step 105 may specifically include:
[0098] The temporary results of all heads associated with each converter subunit in the feedforward network inference process are summed to obtain the feedforward network inference result. Specifically, the temporary results of all heads in each converter subunit are the temporary feedforward network inference results for that respective converter subunit.
[0099] After that, the inference result of the feedforward network is added to the final inference result of the self-attention inference process to obtain the final inference result of the multi-head attention mechanism.
[0100] The data inference method of this disclosure, combined with the multi-head inference feature of the multi-head attention mechanism in large language models, divides the transformer block of the multi-head attention mechanism of the large language model into a preset number of transformer sub-units according to the parameters of the head. These transformer sub-units are then configured in a preset number of computing chips. During inference, each computing chip infers a temporary inference result corresponding to its respective transformer sub-unit. The inference result is obtained by accumulating these temporary results. Parallel inference is achieved using a preset number of computing chips, which helps improve the inference efficiency of large language models. Furthermore, the preset number of transformer sub-units, divided according to the parameters of the head in the transformer block, also considers the parameter cache capacity of the computing chips, thus ensuring that the storage space of each computing chip is within a controllable range and helping to ensure the yield of the computing chips. Moreover, based on the data inference method of this disclosure, as the amount of data in the head parameters continues to increase, it is easy to support large language models with larger parameter data volumes by expanding the number of computing chips. The data inference method of this disclosure helps reduce the hardware cost of large language models and improve their inference speed.
[0101] The data reasoning method based on the embodiments of this disclosure can execute different heads in different computing chips.
[0102] Figure 4A This is a schematic diagram illustrating the reasoning process of a converter subunit in a computing chip regarding a self-attention reasoning process, according to an illustrative embodiment. The converter subunit is a subunit of the converter unit used in the self-attention reasoning process. Figure 4A As shown, the inference process of this converter subunit in the computing chip generally includes the following steps:
[0103] The normalized inference data x1 is multiplied by the weight matrices associated with the head h1 part of the fully connected layer wk, the weight matrices associated with the head h1 part of the fully connected layer wq, and the weight matrices associated with the head h1 part of the fully connected layer wv, respectively, to obtain the head h1 parts of the results xk, xq, and xv. The parts other than the head h1 part are not processed in the computation chip of this converter subunit. The head h1 parts of the results xk and xq are multiplied by the position codes freqs_cis associated with the head h1 part. The first relevance calculation matmul1 is then performed on the head h1 parts of the results xk and xq after the dot product with the position codes freqs_cis associated with the head h1 part to obtain the first relevance. The first correlation value scores1 is obtained by adding the first correlation value scores1 to the mask matrix mask associated with the first correlation value scores2; the first correlation value scores2 is obtained by calculating the first correlation value scores3 using the softmax function; the first correlation value scores3 is obtained by performing the second correlation calculation matmul2 on the first correlation value scores2 and the first correlation value xv; the first correlation value results result is obtained by multiplying the first correlation value results result result with the weight matrix associated with the first correlation value scores1 of the fully connected layer wo to obtain the temporary self-attention inference result y11 associated with the first correlation value scores1.
[0104] Figure 4B This is a schematic diagram illustrating the inference process of another converter subunit in another computing chip regarding the self-attention inference process, according to an illustrative embodiment. This other converter subunit is a converter subunit divided from the converter unit of the self-attention inference process. Figure 4B As shown, the inference process of this other converter subunit in another computing chip generally includes the following steps:
[0105] The normalized inference data x1 is multiplied by the weight matrices associated with the head h2 part of the fully connected layer wk, the weight matrices associated with the head h2 part of the fully connected layer wq, and the weight matrices associated with the head h2 part of the fully connected layer wv, respectively, to obtain the head h2 parts of the results xk, xq, and xv. The parts other than the head h2 part are not processed in the computation chip of this converter subunit. The head h2 parts of the results xk and xq are multiplied by the position codes freqs_cis associated with the head h2 part. The first relevance calculation matmul1 is then performed on the head h2 parts of the results xk and xq after the dot product with the position codes freqs_cis associated with the head h2 part to obtain the first relevance. The first correlation value scores1 is obtained by adding the first correlation value scores1 to the mask matrix mask associated with the first correlation value scores2; the first correlation value scores2 is obtained by calculating the first correlation value scores3 using the softmax function; the first correlation value scores3 is obtained by performing the second correlation calculation matmul2 on the first correlation value scores3 and the first correlation value xv; the correlation result result is obtained by multiplying the first correlation value result result with the weight matrix associated with the first correlation value xv of the fully connected layer wo to obtain the temporary self-attention inference result y12 associated with the first correlation value scores2.
[0106] Figure 4A and Figure 4B This is just an illustrative example; in reality, the total number of heads may be much greater. Based on the temporary results of self-attention reasoning y11, y12, and the temporary results of self-attention reasoning associated with other heads, all temporary results of self-attention reasoning are summed to obtain the self-attention reasoning result y1 of the self-attention reasoning process.
[0107] Figure 5A This is a schematic diagram illustrating the inference process of a converter subunit in a computing chip regarding the feedforward network inference process, according to an illustrative embodiment. The converter subunit is a subunit divided from the converter unit of the feedforward network inference process. Figure 5A As shown, the inference process of this converter subunit in the computing chip generally includes the following steps:
[0108] The normalized data x2 to be inferred is multiplied by the weight matrix of the fully connected layer w1 associated with the head h1 part and the weight matrix of the fully connected layer w2 associated with the head h1 part, respectively, to obtain the head h1 part of the result x21 and the head h1 part of the result x22. The data x2 to be inferred is the final inference result of the self-attention inference process. Except for the head h1 part, the rest is not processed in the computing chip where this converter sub-unit is located. The head h1 part of the result x21 is calculated by the SILU (Sigmoid Activation Linear Unit) function and then multiplied by the head h1 part of the result x22 to obtain the head h1 part of the result x23. The head h1 part of the result x23 is multiplied by the weight matrix of the fully connected layer w3 associated with the head h1 part to obtain the feedforward network inference temporary result y21 associated with the head h1.
[0109] Figure 5B This is a schematic diagram illustrating the inference process of another converter subunit in another computing chip regarding the feedforward network inference process, according to an illustrative embodiment. The converter subunit is a subunit divided from the converter unit of the feedforward network inference process. Figure 5B As shown, the inference process of this other converter subunit in another computing chip generally includes the following steps:
[0110] The normalized data x2 to be inferred is multiplied by the weight matrix of the fully connected layer w1 associated with the head h2 part and the weight matrix of the fully connected layer w2 associated with the head h2 part, respectively, to obtain the head h2 part of the result x21 and the head h2 part of the result x22. The data x2 to be inferred is the final inference result of the self-attention inference process. The parts other than the head h2 part are not processed in the computing chip where this converter sub-unit is located. The head h2 part of the result x21 is calculated by the SILU (Sigmoid Activation Linear Unit) function and then multiplied by the head h2 part of the result x22 to obtain the head h2 part of the result x23. The head h2 part of the result x23 is multiplied by the weight matrix of the fully connected layer w3 associated with the head h2 part to obtain the feedforward network inference temporary result y22 associated with the head h1.
[0111] Figure 5A and Figure 5BThis is merely an illustrative example; in reality, the total number of heads may be much greater. Based on the temporary feedforward inference results y21, y22, and y2 associated with other heads, all temporary feedforward inference results are summed to obtain the feedforward inference result y2 of the feedforward inference process. After obtaining the feedforward inference result y2, the data to be inferred x2 (i.e., the final inference result of the self-attention inference process) is added to the feedforward inference result y2 to obtain the final inference result of the feedforward inference process, i.e., the inference result of the transformer block.
[0112] Figure 6 This is a schematic diagram illustrating the logic structure of a converter block that implements the complete functionality of four computing chips, according to an illustrative embodiment. Figure 6 and combined Figure 4A , Figure 4B , Figure 5A , Figure 5B As shown, Figure 4A The reasoning process can be executed in the first self-attention reasoning operation chip 601. Figure 4B The reasoning process can be executed in the second self-attention reasoning operation chip 602. Figure 5A The inference process can be executed in the first feedforward network inference operation chip 603. Figure 5B The inference process can be executed in the second feedforward network inference operation chip 604. The four operation chips, namely the first self-attention inference operation chip 601, the second self-attention inference operation chip 602, the first feedforward network inference operation chip 603, and the second feedforward network inference operation chip 604, can realize the complete function of a converter block.
[0113] Taking the llama2 70B large language model as an example, the number of computing chips required can be calculated based on the set SRAM size of the computing chip, as shown in the table below.
[0114] SRAM capacity (MB) 32 64 128 256 512 1024 2048 Number of computing chips 2720 1440 720 400 240 80 40
[0115] As shown in the table above, 2720 computing chips are needed when the SRAM of the computing chip is 32MB (megabytes); 1440 computing chips are needed when the SRAM of the computing chip is 64MB; 720 computing chips are needed when the SRAM of the computing chip is 128MB; 400 computing chips are needed when the SRAM of the computing chip is 256MB; 240 computing chips are needed when the SRAM of the computing chip is 512MB; 80 computing chips are needed when the SRAM of the computing chip is 1024MB; and 40 computing chips are needed when the SRAM of the computing chip is 2048MB.
[0116] For example, a single converter block originally required 1024MB of SRAM to store parameters. According to the embodiments of this disclosure, by increasing the number of chips, each chip only needs 256MB of SRAM. While achieving the same functionality, the SRAM requirement per computing chip is significantly reduced, which helps improve chip yield. Larger oracle models can also be supported by increasing the number of chips.
[0117] It is evident that the scheme of performing inference on a large language model by reducing the SRAM of a single computing chip and increasing the number of computing chips has advantages. Smaller SRAM computing chips are easier to manufacture, have higher yields, and lower costs than larger SRAM computing chips. Furthermore, multiple computing chips can execute the inference process of the converter block in parallel, which is faster. Therefore, the data inference method of this disclosure helps to reduce the hardware cost of large language models and improve the inference speed of large language models.
[0118] Figure 7 This is a schematic diagram of the logic structure of a data inference device according to an illustrative embodiment, such as... Figure 7 As shown, the data inference device mainly includes a transformer block acquisition module 701, a segmentation module 702, a configuration module 703, a data inference module 704, and a result acquisition module 705.
[0119] The system includes the following modules: a transformer block acquisition module 701, configured to acquire transformer units for the multi-head attention mechanism of a large language model, wherein each transformer unit includes at least one head; a segmentation module 702, configured to segment the transformer unit into a preset number of transformer sub-units, wherein each transformer sub-unit includes at least one head; a configuration module 703, configured to configure the preset number of transformer sub-units into a preset number of computing chips, wherein each computing chip is configured with one transformer sub-unit; a data inference module 704, configured to input the data to be inferred in the multi-head attention mechanism inference process into the preset number of computing chips during the inference process of the large language model, and obtain a preset number of temporary inference results for the multi-head attention mechanism through inference by the preset number of computing chips; and a result acquisition module 705, configured to obtain the inference result of the multi-head attention mechanism based on the preset number of temporary inference results for the multi-head attention mechanism.
[0120] In an illustrative embodiment, the data inference device further includes:
[0121] The preset quantity determination module is configured to perform the following actions: obtain the total number of heads in the converter unit and the data volume of the parameters of each head; obtain the parameter cache capacity of a single computing chip; and obtain the preset quantity based on the total number of heads in the converter unit, the data volume of the parameters of each head, and the parameter cache capacity of a single computing chip; wherein the parameter cache capacity of each computing chip is not less than the data volume of the parameters of the configured converter subunit.
[0122] In an illustrative embodiment, the data inference device further includes:
[0123] The mapping module is configured to establish the mapping relationship between the converter subunit and the computing chip.
[0124] In an illustrative embodiment, the configuration module 703 is further configured to perform: configuring each converter subunit in the corresponding computing chip according to the mapping relationship.
[0125] In an illustrative embodiment, the result acquisition module 705 is further configured to perform: summing the temporary inference results of the multi-head attention mechanism associated with each converter subunit to obtain the inference result of the multi-head attention mechanism.
[0126] In an illustrative embodiment, the data inference module 704 is further configured to perform: synchronously inputting the data to be inferred into a preset number of computing chips; each computing chip infers the data to be inferred according to the parameters of its own configured converter subunit, and obtains a temporary inference result of the multi-head attention mechanism associated with the converter subunit configured by each computing chip.
[0127] In an illustrative embodiment, the multi-head attention mechanism includes a self-attention phase and a feedforward network phase;
[0128] The reasoning process of multi-head attention mechanisms includes sequentially executed self-attention reasoning and feedforward network reasoning.
[0129] When the reasoning process of the multi-head attention mechanism is a self-attention reasoning process, each transformer sub-unit in the multi-head attention mechanism is a transformer sub-unit divided by the transformer unit of the self-attention reasoning process, the preset number of computing chips is the first number of computing chips used for the self-attention reasoning process, each temporary reasoning result of the multi-head attention mechanism is the temporary reasoning result of the self-attention reasoning associated with each transformer sub-unit in the self-attention reasoning process, and the reasoning result of the multi-head attention mechanism is the self-attention reasoning result of the self-attention reasoning process.
[0130] When the inference process of the multi-head attention mechanism is a feedforward network inference process, each transformer subunit in the multi-head attention mechanism is a transformer subunit divided by the transformer unit of the feedforward network inference process, the preset number of computing chips is the second number of second computing chips used for the feedforward network inference process, the data to be inferred is the final inference result of the self-attention inference process, each temporary inference result of the multi-head attention mechanism is the temporary inference result of the feedforward network inference associated with each transformer subunit in the feedforward network inference process, and the inference result of the multi-head attention mechanism is the feedforward network inference result of the feedforward network inference process.
[0131] The final reasoning result of the self-attention reasoning process is obtained by adding the self-attention reasoning result to the data to be reasoned.
[0132] The transformer block of the multi-head attention mechanism includes transformer units for the self-attention reasoning process and transformer units for the feedforward network reasoning process;
[0133] The inference result of the transformer block is obtained by adding the final inference result of the self-attention inference process to the inference result of the feedforward network.
[0134] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.
[0135] Regarding the data inference apparatus in the above embodiments, the specific manner in which each unit performs its operations has been described in detail in the embodiments related to the data inference method, and will not be elaborated upon here.
[0136] It should be noted that the above embodiments are only examples of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0137] Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. In some embodiments, the electronic device is a server. The electronic device 800 can vary considerably due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 801 and one or more memories 802, wherein the memory 802 stores at least one line of program code, which is loaded and executed by the processor 801 to implement the data reasoning methods provided in the various embodiments described above. Of course, the electronic device 800 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The electronic device 800 may also include other components for implementing device functions, which will not be elaborated here.
[0138] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction, which can be executed by a processor in a computer device to perform the data reasoning method in the above embodiments.
[0139] Optionally, the aforementioned computer-readable storage medium may be a non-transitory computer-readable storage medium, such as ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices.
[0140] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A data reasoning method, comprising: A transformer unit for acquiring the multi-head attention mechanism of a large language model, wherein the transformer unit includes at least one head; The converter unit is divided into a predetermined number of converter sub-units, wherein each converter sub-unit includes at least one head; A preset number of the converter sub-units are respectively configured in a preset number of computing chips, wherein each computing chip is configured with one converter sub-unit; During the reasoning process of the large language model, the data to be reasoned in the reasoning process of the multi-head attention mechanism is input into a preset number of computing chips, and a preset number of temporary results of multi-head attention mechanism reasoning are obtained through the reasoning of the preset number of computing chips. Based on a preset number of temporary inference results from the multi-head attention mechanism, the inference result of the multi-head attention mechanism is obtained; The data reasoning method also includes: Obtain the total number of heads in the converter unit used and the amount of data for the parameters of each head; Obtain the parameter cache capacity of a single computing chip; The preset number is obtained based on the total number of heads in the converter unit, the data volume of the parameters of each head, and the parameter cache capacity of a single computing chip; The parameter cache capacity of each of the computing chips is not less than the data volume of the parameters of the configured converter subunit.
2. The data reasoning method according to claim 1, characterized in that, The data reasoning method also includes: Establish a mapping relationship between the converter subunit and the computing chip.
3. The data reasoning method according to claim 2, characterized in that, The step of configuring a preset number of converter subunits into a preset number of computing chips includes: According to the mapping relationship, each converter subunit is configured in the corresponding computing chip.
4. The data reasoning method according to claim 1, characterized in that, The step of obtaining the inference result of the multi-head attention mechanism based on a preset number of temporary inference results of the multi-head attention mechanism includes: The temporary inference results of the multi-head attention mechanism associated with each of the transformer sub-units are accumulated to obtain the inference result of the multi-head attention mechanism.
5. The data reasoning method according to claim 1, characterized in that, The step of inputting the data to be reasoned into a preset number of computing chips during the reasoning process of the multi-head attention mechanism, and obtaining a preset number of temporary inference results of the multi-head attention mechanism through the reasoning of the preset number of computing chips, includes: The data to be inferred is synchronously input into a preset number of the computing chips; Each computing chip performs inference on the data to be inferred based on the parameters of the converter subunit configured in itself, and obtains a temporary inference result of the multi-head attention mechanism associated with the converter subunit configured in each computing chip.
6. The data reasoning method according to claim 1, characterized in that: The multi-head attention mechanism includes a self-attention phase and a feedforward network phase; The reasoning process of the multi-head attention mechanism includes a sequentially executed self-attention reasoning process and a feedforward network reasoning process; When the inference process of the multi-head attention mechanism is a self-attention inference process, each transformer subunit in the multi-head attention mechanism is a transformer subunit divided from the transformer unit of the self-attention inference process, the preset number of computing chips is a first number of first computing chips used in the self-attention inference process, each temporary inference result of the multi-head attention mechanism is a temporary self-attention inference result associated with each transformer subunit in the self-attention inference process, and the inference result of the multi-head attention mechanism is the self-attention inference result of the self-attention inference process. When the inference process of the multi-head attention mechanism is a feedforward network inference process, each transformer subunit in the multi-head attention mechanism is a transformer subunit divided from the transformer unit of the feedforward network inference process, the preset number of computing chips is a second number of second computing chips used in the feedforward network inference process, the data to be inferred is the final inference result of the self-attention inference process, each temporary inference result of the multi-head attention mechanism is a feedforward network inference temporary result associated with each transformer subunit in the feedforward network inference process, and the inference result of the multi-head attention mechanism is the feedforward network inference result of the feedforward network inference process.
7. A data inference device, characterized in that, include: A transformer block acquisition module is configured to execute a transformer unit for acquiring a multi-head attention mechanism for a large language model, wherein the transformer unit includes at least one head. The segmentation module is configured to perform the division of the converter unit into a preset number of converter sub-units, wherein each converter sub-unit includes at least one head; The configuration module is configured to configure a preset number of the converter subunits into a preset number of computing chips, wherein each computing chip is configured with one converter subunit; The data inference module is configured to, during the inference process of the large language model, input the data to be inferred in the inference process of the multi-head attention mechanism into a preset number of computing chips, and obtain a preset number of temporary inference results of the multi-head attention mechanism through the inference of the preset number of computing chips. The result acquisition module is configured to execute a preset number of temporary inference results from the multi-head attention mechanism to obtain the inference result of the multi-head attention mechanism; The preset quantity determination module is configured to perform the following operations: obtain the total number of heads in the converter unit and the data volume of parameters of each head; obtain the parameter cache capacity of a single computing chip; and obtain the preset quantity based on the total number of heads in the converter unit, the data volume of parameters of each head, and the parameter cache capacity of a single computing chip; wherein the parameter cache capacity of each computing chip is not less than the data volume of parameters of the configured converter subunit.
8. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the data inference method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, When at least one instruction in the computer-readable storage medium is executed by the processor of the electronic device, the electronic device is able to implement the data reasoning method as described in any one of claims 1 to 6.