Data processing method, computing device, and storage medium

Abstract

The application provides a data processing method, a computing device and a storage medium. The method is applied to the computing device, and the method comprises the following steps: for N prompt reasoning sub-tasks, a computing unit performs matrix multiplication operation on a third matrix and a first sub-matrix in a first matrix to obtain a first intermediate matrix; the first intermediate matrix and a first sub-matrix in a second matrix are subjected to matrix multiplication operation to obtain a second intermediate matrix; for any sub-task in the N word element reasoning sub-tasks and a first vector corresponding to the sub-task, the first vector is subjected to multiplication operation on one of N second sub-matrices of the first matrix to obtain a second vector, and a third vector is obtained based on the second vector and the second sub-matrix of the second matrix; and the third intermediate matrix obtained by stacking N third vectors is subjected to superposition operation on the second intermediate matrix to obtain a result matrix. In the above scheme, the space occupation of the display memory unit is reduced, the operation speed is improved, and the overall reasoning speed of the model is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, computing device, and storage medium. Background Technology

[0002] Large Language Models (LLMs) are developing rapidly. As a representative of this type of model, Generative Pre-trained Transformer (GPT) models contain a large number of FeedForward Neural Network (FFN) operations and attention operations, which require a lot of computing resources.

[0003] Currently, beam search is commonly used in GPT inference to improve the accuracy of the inference results. For example... Figure 1As shown, for any sequence during token inference, a new token prediction is generated at each time step. The beam search mechanism selects N tokens (where N represents the beam size, which can be user-defined) from these predictions to proceed to the next time step for token inference. Therefore, except for the first time step, for any hint in a batch (of size M, where M is the batch size), N branches of token inference are performed. The total number of branches in a batch is M × N, which is batch size × beam size. To maintain consistency, each hint in the batch is typically copied and expanded by a multiple of the beam size during the hint inference stage, ensuring that the total number of hint branches in each subsequent step remains consistent (i.e., batch size × beam size). However, the above scheme significantly increases the inference task. During the attention calculation between these inference branches in the token inference part, each branch has its own k_cache and v_cache. Here, k_cache and v_cache represent the cache corresponding to the key matrix and the cache corresponding to the value matrix in the attention mechanism, respectively. Therefore, the operation between the query matrix corresponding to the hint and k_cache and v_cache is a matrix-vector multiplication, which is less efficient than matrix multiplication. Moreover, since each branch's k_cache and v_cache must be stored in the GPU memory space, the duplicate GPU memory space occupied is large, resulting in a lot of resource waste. Summary of the Invention

[0004] This invention provides a data processing method, computing device, and storage medium to address the shortcomings of low computing efficiency and large memory usage in existing technologies, thereby achieving a data processing method with high computing efficiency and small storage usage.

[0005] This invention provides a data processing method applied to a computing device. The computing device includes a computing unit and a video memory unit. The video memory unit stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of the second sub-matrix of the first matrix, and N copies of the second sub-matrix of the second matrix, where N is an integer greater than 0. The width of the first sub-matrix is ​​the prompt length, and the width of the second sub-matrix is ​​the number of tokens generated by reasoning. The method includes:

[0006] For N prompt reasoning subtasks with the same prompt, the computing unit performs matrix multiplication on the third matrix and the first submatrix in the first matrix to obtain a first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors.

[0007] The calculation unit performs matrix multiplication based on the first intermediate matrix and the first submatrix in the second matrix to obtain the second intermediate matrix;

[0008] For any of the N word inference subtasks mentioned in the prompt, and a first vector among the N first vectors corresponding to the word inference subtask, the computing unit performs a product operation between the first vector and a second submatrix among the N second submatrices of the first matrix to obtain a second vector, and obtains a third vector based on the second vector and the second submatrix in the second matrix;

[0009] The computing unit performs a superposition operation on the third intermediate matrix obtained by stacking N third vectors and the second intermediate matrix to obtain the result matrix.

[0010] According to a data processing method provided by the present invention, the computing unit performs matrix multiplication based on the first intermediate matrix and the first submatrix in the second matrix to obtain a second intermediate matrix, including:

[0011] The calculation unit updates each element in the first intermediate matrix based on the maximum value of each row in the first intermediate matrix;

[0012] The calculation unit performs a matrix multiplication operation on the updated first intermediate matrix and the first submatrix in the second matrix to obtain the second intermediate matrix.

[0013] According to a data processing method provided by the present invention, the calculation unit obtains a third vector based on the second vector and the second submatrix in the second matrix, including:

[0014] The calculation unit updates each element of the second vector based on the maximum value in the second vector to obtain the updated second vector;

[0015] The calculation unit performs a product operation on the updated second vector and the second submatrix in the second matrix to obtain the third vector.

[0016] According to a data processing method provided by the present invention, the method further includes:

[0017] The computing unit caches the sum of the elements in each row of the updated first intermediate matrix and the maximum value of each row of the first intermediate matrix locally in the computing unit.

[0018] The computing unit caches the sum of elements in the updated second vector corresponding to each of the N word inference subtasks, as well as the maximum value in the unupdated second vector corresponding to each word inference subtask, locally within the computing unit.

[0019] According to a data processing method provided by the present invention, the computing unit performs a superposition operation on a third intermediate matrix obtained by superimposing N third vectors and a second intermediate matrix to obtain a result matrix, including:

[0020] The calculation unit scales the third intermediate matrix and the second intermediate matrix, and then performs a superposition operation on the two scaled intermediate matrices to obtain a fourth intermediate matrix.

[0021] The calculation unit scales the fourth and fifth vectors and performs a bitwise superposition operation on the two vectors obtained after scaling to obtain the sixth vector; the fourth vector is the vector formed by the sum of the elements in each row of the updated first intermediate matrix, and the fifth vector is the vector formed by the sum of the elements in the updated second vector corresponding to each of the N word inference subtasks;

[0022] For each row of the fourth intermediate matrix, the calculation unit divides each element of each row of the fourth intermediate matrix by the element at the corresponding position in the sixth vector to obtain the result matrix.

[0023] According to a data processing method provided by the present invention, the computing unit performs scaling processing on the third intermediate matrix and the second intermediate matrix, including:

[0024] For any row of the second intermediate matrix and the corresponding row of the third intermediate matrix, if the maximum value of the i1th row of the first intermediate matrix is ​​less than the i1th element of the seventh vector, then based on the maximum value of the i1th row of the first intermediate matrix and the i1th element of the seventh vector, each element of the i1th row of the second intermediate matrix is ​​scaled; the seventh vector is the vector formed by the maximum value of the second vector before the update corresponding to each of the N word inference subtasks; the value of i1 ranges from 1 to N;

[0025] If the maximum value of the i1th row in the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, each element of the i1th row in the third intermediate matrix is ​​scaled.

[0026] The calculation unit scales the fourth and fifth vectors, including:

[0027] For any element in the fourth vector and the corresponding element in the fifth vector, if the maximum value of the i1th row in the first intermediate matrix is ​​less than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, for the element s in the fourth vector... i1 Perform scaling processing;

[0028] If the maximum value in the i1th row of the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value in the i1th row of the first intermediate matrix and the i1th element in the seventh vector, the element s in the fifth vector is... i1 Perform scaling.

[0029] According to a data processing method provided by the present invention, the calculation unit updates each element in the first intermediate matrix based on the maximum value of each row in the first intermediate matrix, including:

[0030] For any element in the first intermediate matrix, the calculation unit updates the element using the following formula (1):

[0031] qk_(ij)'=exp(x_ij-max_i); (1)

[0032] Where qk_(ij)' represents the element in the i-th row and j-th column of the first intermediate matrix after the update, x_ij represents the element in the i-th row and j-th column of the first intermediate matrix before the update, max_i represents the maximum value in the i-th row of the first intermediate matrix, and exp represents the exponential operation with base e.

[0033] According to a data processing method provided by the present invention, the calculation unit updates each element of the second vector based on the maximum value in the second vector to obtain an updated second vector, including:

[0034] For any element in the second vector, the calculation unit updates the element using the following formula (2):

[0035] qk_(m)'=exp(x_m-max2 k (2)

[0036] Where qk_(m)' represents the m-th element in the second vector after the update, x_m represents the m-th element in the second vector before the update, and max2 k represents the maximum value in the second vector, and exp represents the exponential operation with base e.

[0037] According to a data processing method provided by the present invention, the first matrix and the second matrix are matrices in a target model based on an attention mechanism.

[0038] The present invention also provides a computing device, comprising:

[0039] The memory includes: a video memory unit; the video memory unit stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of the second sub-matrix of the first matrix, and N copies of the second sub-matrix of the second matrix, where N is an integer greater than 0, the width of the first sub-matrix is ​​the prompt length, and the width of the second sub-matrix is ​​the number of tokens generated by reasoning;

[0040] A processor, coupled to the memory, is configured to:

[0041] For N prompt reasoning subtasks with the same prompt, the third matrix is ​​multiplied by the first submatrix in the first matrix to obtain the first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors.

[0042] Based on the first intermediate matrix and the first submatrix in the second matrix, perform matrix multiplication to obtain the second intermediate matrix;

[0043] For any of the N word inference subtasks mentioned in the prompt, and for one of the N first vectors corresponding to the word inference subtask, the first vector is multiplied by one of the N second submatrices of the first matrix to obtain a second vector, and a third vector is obtained based on the second vector and the second submatrices of the second matrix.

[0044] The third intermediate matrix obtained by superimposing N third vectors is then superimposed with the second intermediate matrix to obtain the result matrix.

[0045] The present invention also provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the data processing methods described above.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above.

[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data processing methods described above.

[0048] The data processing method, computing device, and storage medium provided by this invention are applied to a computing device. The computing device includes a computing unit and a video memory unit. The video memory unit stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of the second sub-matrix of the first matrix, and N copies of the second sub-matrix of the second matrix, where N is an integer greater than 0. The width of the first sub-matrix is ​​the prompt length, and the width of the second sub-matrix is ​​the number of tokens generated by inference. For N prompt inference subtasks, the computing unit performs matrix multiplication on the third matrix and the first sub-matrix of the first matrix to obtain a first intermediate matrix; and performs matrix multiplication on the first intermediate matrix and the first sub-matrix of the second matrix to obtain a second intermediate matrix; for any subtask in the N token inference subtasks and the subtask's corresponding sub-task... The first vector is multiplied by one of the N second sub-matrices of the first matrix to obtain the second vector, and a third vector is obtained based on the second vector and the second sub-matrice of the second matrix. The third intermediate matrix obtained by stacking the N third vectors is superimposed with the second intermediate matrix to obtain the result matrix. In the above scheme, since the first sub-matrice used for each bundle in the prompt reasoning process is the same, only one copy of the first sub-matrice of the first matrix and one copy of the first sub-matrice of the second matrix are stored, which reduces the space occupied by the video memory unit. Moreover, the prompt reasoning subtask and the token reasoning subtask are calculated separately. The calculation of the prompt reasoning subtask uses matrix multiplication, which can improve the calculation speed and thus improve the overall reasoning speed of the model. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram illustrating the principle of beam search provided by the present invention;

[0051] Figure 2 This is a schematic diagram of the storage principle corresponding to the cluster search provided by the present invention;

[0052] Figure 3 This is one of the flowcharts illustrating the data processing method provided by the present invention;

[0053] Figure 4 This is a schematic diagram illustrating the storage principle of the data processing method provided by the present invention;

[0054] Figure 5This is one of the schematic diagrams illustrating the calculation principle of the data processing method provided by the present invention;

[0055] Figure 6 This is the second schematic diagram illustrating the calculation principle of the data processing method provided by this invention;

[0056] Figure 7 This is the third schematic diagram illustrating the calculation principle of the data processing method provided by this invention;

[0057] Figure 8 This is one of the structural schematic diagrams of the computing device provided by the present invention;

[0058] Figure 9 This is the second schematic diagram of the computing device provided by the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] First, the terms and application scenarios involved in the embodiments of the present invention will be introduced:

[0061] A token, or lexical unit, in large language models typically refers to a discrete unit of text. It can be a word, punctuation mark, number, or other language element, which is used as the basic unit for training and generating text.

[0062] like Figure 1 As shown, after adopting the beam search mechanism, the number of inference branches has increased from M to M×N, which is batch size×beam size, significantly increasing the inference task. During the attention calculation in the token inference part, each branch has its own k_cache and v_cache. Therefore, the operations between the query (q) and k_cache and v_cache, such as q×k_cache and softmax(q×k_cache)×v_cache, are performed as separate matrix-vector multiplications, resulting in low computational efficiency. Figure 2 As shown, the k_cache and v_cache of each branch are stored in the video memory space. However, the k_cache and v_cache parts with the specified length are the same for each branch. Therefore, it is equivalent to saving the same content N times, which causes a great waste of video memory space.

[0063] To address the aforementioned issues, this embodiment of the invention utilizes the principle that the reasoning portion of all branches corresponding to the same prompt is identical. The calculation of q×k_cache and q×k_cache×v_cache (e.g., softmax(q×k_cache)×v_cache) in the attention portion of token reasoning is divided into two parts. The part corresponding to the prompt length can be implemented using matrix-matrix multiplication, while the part corresponding to the token length continues to use matrix-vector multiplication, improving overall computational efficiency. Furthermore, only one copy of the k_cache and v_cache corresponding to the prompt portion belonging to the same prompt is stored instead of a beamsize copy, reducing the storage space occupied by k_cache and v_cache.

[0064] The following is combined Figures 1-9 The technical solutions of the embodiments of the present invention will be described in detail with reference to specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some examples.

[0065] Figure 3 This is a flowchart illustrating the data processing method provided by the present invention. Figure 3 , Figure 4 As shown, the method of this embodiment is applied to a computing device, which includes a computing unit and a video memory unit. The video memory unit stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of the second sub-matrix of the first matrix, and N copies of the second sub-matrix of the second matrix, where N is an integer greater than 1. The width of the first sub-matrix is ​​the prompt length L1, and the width of the second sub-matrix is ​​the number of tokens generated by reasoning L2. The method provided in this embodiment includes:

[0066] Step 101: For N clue reasoning subtasks with the same clue, the computing unit performs matrix multiplication on the third matrix and the first submatrix in the first matrix to obtain the first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors.

[0067] Specifically, step 101 involves performing a matrix multiplication operation between the first submatrix of the first matrix and the third matrix, where the first submatrix is ​​a submatrix of the first matrix. For example... Figure 4As shown, for any cue in a batch, during cue reasoning, only one copy of the k_cache and v_cache corresponding to the cue length L1 for each head is stored in the video memory. However, during token reasoning, due to the beam search mechanism, beam-size k_cache and v_cache are generated for each head of any cue, thus requiring N copies to be stored, where N represents the beam size. The k_cache and v_cache corresponding to cue reasoning and token reasoning are stored independently due to their different quantities. The token is the word segment obtained by word segmentation of the cue.

[0068] For a batch inference task, the weight matrix multiplication operation preceding attention passes in M×H×N subtasks that need to perform attention operations, which is the batch size × number of heads × beamsize. At this point, N subtasks belonging to the same head in the same cue are assigned to a single computation unit, for example, a unit equipped with an independent matrix multiplication accelerator. Figure 5 As shown, the third matrix corresponding to this subtask is a matrix q obtained by combining N first vectors (e.g., the Query in the attention mechanism, which can be the Query corresponding to the token in the prompt). The size of this third matrix is, for example, the width is the hidden dimension in the attention mechanism and the height is the beam size. The key is a matrix with a width of L1 and a height of the hidden dimension. First, perform a matrix product operation on these two matrices, and the result of the product is qk1.

[0069] It should be noted that the width of the first submatrix is ​​the hint length L1, for example, the number of columns in the first submatrix is ​​equal to the number of words corresponding to the hint length L1. The width of the second submatrix is ​​the number of tokens generated by reasoning L2. The width of the second submatrix is ​​represented by the number of columns, that is, the number of columns in the second submatrix is ​​equal to L2. The number of tokens generated by reasoning refers to the number of newly generated tokens when reasoning to the current step.

[0070] Step 102: The calculation unit performs matrix multiplication based on the first submatrix in the first intermediate matrix and the second matrix to obtain the second intermediate matrix;

[0071] Specifically, the first intermediate matrix qk1 is multiplied by the first submatrix in the second matrix, i.e. Figure 5In the middle, v_cache, the width of the v_cache matrix is ​​the hidden dimension and the height is L1, thus obtaining the second intermediate matrix o1.

[0072] It should be noted that the height of the v_cache matrix is ​​L1, which means that the number of rows in the v_cache matrix is ​​equal to the number of tokens corresponding to L1.

[0073] Step 103: For any one of the N word reasoning subtasks in the prompt, and one of the N first vectors corresponding to the word reasoning subtask, the calculation unit performs a product operation on the first vector and one of the N second submatrices of the first matrix to obtain a second vector, and obtains a third vector based on the second vector and the second submatrices of the second matrix.

[0074] Specifically, such as Figure 6 As shown, the computing unit processes each of the beam-size token inference subtasks separately. In each subtask, the q vector of length hidden dimension is multiplied by the corresponding k_cache (one of the N) with width L2 and height hidden dimension to obtain a second vector qk2 of length L2.

[0075] Then, a matrix-vector product operation is performed on the second vector qk2 and the v_cache matrix with a width of hiddendimension and a height of L2 for the token inference part of each corresponding beam branch to obtain the third vector o. 2_k The value of k ranges from 1 to beam size, and then beam size is used to represent these values. 2_k The vectors are concatenated to form a third intermediate matrix o2, which has the same size as the second intermediate matrix o1 output in the previous step.

[0076] Optionally, the execution order of the beam-size subtasks corresponding to this prompt depends on the processing power of the computing unit; they can be processed sequentially in a loop, or partially or entirely in parallel.

[0077] Step 104: The calculation unit performs a superposition operation on the third intermediate matrix obtained by stacking N third vectors and the second intermediate matrix to obtain the result matrix.

[0078] Specifically, the N third vectors are stacked to obtain the third intermediate matrix, and the second and third intermediate matrices are superimposed, for example, by element-wise, to obtain the result matrix.

[0079] Optionally, the first and second matrices are matrices in the target model based on the attention mechanism. The target model is, for example, the GPT model. For instance, the first matrix is ​​the key matrix, and the second matrix is ​​the value matrix.

[0080] The method of this embodiment is applied to a computing device, which includes a computing unit and a video memory unit. The video memory unit stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of a second sub-matrix of the first matrix, and N copies of a second sub-matrix of the second matrix, where N is an integer greater than 0. The width of the first sub-matrix is ​​the prompt length, and the width of the second sub-matrix is ​​the number of tokens generated by inference. For N prompt inference subtasks, the computing unit performs matrix multiplication on the third matrix and the first sub-matrix of the first matrix to obtain a first intermediate matrix; performs matrix multiplication on the first intermediate matrix and the first sub-matrix of the second matrix to obtain a second intermediate matrix; and performs matrix multiplication on any subtask among the N token inference subtasks and the first vector corresponding to that subtask. The first vector is multiplied by one of the N second sub-matrices of the first matrix to obtain the second vector, and the third vector is obtained based on the second vector and the second sub-matrice of the second matrix. The third intermediate matrix obtained by stacking the N third vectors is superimposed with the second intermediate matrix to obtain the result matrix. In the above scheme, since the first sub-matrice used for each bundle in the prompt reasoning process is the same, only one copy of the first sub-matrice of the first matrix and one copy of the first sub-matrice of the second matrix are stored, which reduces the space occupied by the video memory unit. Moreover, the prompt reasoning subtask and the token reasoning subtask are calculated separately. The calculation of the prompt reasoning subtask uses matrix multiplication, which can improve the calculation speed and thus improve the overall reasoning speed of the model.

[0081] Optionally, step 102 can be implemented in the following way:

[0082] The calculation unit updates each element in the first intermediate matrix based on the maximum value of each row in the first intermediate matrix;

[0083] The calculation unit performs matrix multiplication on the first submatrix of the updated first intermediate matrix and the second matrix to obtain the second intermediate matrix.

[0084] Optionally, for any element in the first intermediate matrix, the calculation unit updates the element using the following formula (1):

[0085] qk_(ij)'=exp(x_ij-max_i); (1)

[0086] Where qk_(ij)' represents the element in the i-th row and j-th column of the first intermediate matrix after the update, x_ij represents the element in the i-th row and j-th column of the first intermediate matrix before the update, max_i represents the maximum value in the i-th row of the first intermediate matrix, and exp represents the exponential operation with base e.

[0087] Specifically, the computing unit calculates the maximum value max_i for each row of the first intermediate matrix qk1, where i is the row index and max_i represents the maximum value of the i-th row. Then, based on this maximum value, each element in the first intermediate matrix qk1 is updated. For example, for the element x_ij in the i-th row and j-th column, exp(x_ij-max_i) is calculated, and x_ij is updated to the value of exp(x_ij-max_i).

[0088] Optionally, sum all elements in each row, i.e., calculate s1_i = sum_i(exp(x_ij-max_i)), where sum_i() represents the summation function for summing the elements in the i-th row.

[0089] The maximum value max_i corresponding to each row is combined into a vector max1 with a length of beam size. The summation result s1_i corresponding to each row is combined into a vector s1 with a length of beam size. Vectors max1 and s1 are cached in the local storage of the computing unit.

[0090] Optionally, step 103, "obtaining the third vector based on the second vector and the second submatrix of the second matrix," can be achieved as follows:

[0091] The calculation unit updates each element of the second vector based on the maximum value in the second vector to obtain the updated second vector;

[0092] The calculation unit performs a product operation on the updated second vector and the second submatrix of the second matrix to obtain the third vector.

[0093] Optionally, for any element in the second vector, the calculation unit updates the element using the following formula (2):

[0094] qk_(m)'=exp(x_m-max2 k (2)

[0095] Where qk_(m)' represents the m-th element in the second vector after the update, x_m represents the m-th element in the second vector before the update, and max2 k represents the maximum value in the second vector, and exp represents the exponential operation with base e.

[0096] Specifically, for any subtask in the beam-size token inference subtasks, the computation unit calculates the maximum value max2 of the second vector qk2. k It is stored locally in the computation unit and updated for each element in the second vector qk2, such as when calculating exp(x_m-max2). k Let x_m represent the m-th element in the second vector before the update, and update x_m to the value of exp(x_m - max2). k )value.

[0097] Optionally, summate all elements in the updated second vector: s2_k = sum_m(exp(x_m - max2)) k The summation is stored locally in the computation unit, where sum_m() represents the summation function that sums the elements of the second vector.

[0098] Optionally, for beam-size token inference subtasks, the max2 obtained from each token inference subtask is... k Combined with s2_k, they form vectors max2 and s2, which are vectors of length beam size and stored locally in the computation unit.

[0099] In the above embodiments, the sum of elements and the maximum value are stored and calculated locally in the computing unit, which reduces data handling and increases the calculation speed.

[0100] Optionally, step 104 can be implemented in the following way:

[0101] The calculation unit scales the third intermediate matrix and the second intermediate matrix, and then performs a superposition operation on the two scaled intermediate matrices to obtain the fourth intermediate matrix.

[0102] The computing unit scales the fourth and fifth vectors and performs a bitwise superposition operation on the two vectors obtained after scaling to obtain the sixth vector; the fourth vector is the vector formed by the sum of the elements in each row of the updated first intermediate matrix, and the fifth vector is the vector formed by the sum of the elements in the updated second vector corresponding to each of the N word inference subtasks;

[0103] For each row of the fourth intermediate matrix, the calculation unit divides each element of each row of the fourth intermediate matrix by the element at the corresponding position in the sixth vector to obtain the result matrix.

[0104] Specifically, such as Figure 7 As shown, the third intermediate matrix o2 and the second intermediate matrix o1 are scaled and then superimposed to obtain the fourth intermediate matrix;

[0105] The vector s1, formed by the sum of the elements in each row of the first intermediate matrix, and the vector s2, formed by the sum of the elements in the updated second vector corresponding to each of the N word inference subtasks, are scaled and then superimposed to obtain the sixth vector.

[0106] For each row in the fourth intermediate matrix, the computation unit divides each element of each row in the fourth intermediate matrix by the element at the corresponding position in the sixth vector to obtain the result matrix o, which is the final output matrix of the attention calculation. This result matrix can be used for other subsequent calculation processes.

[0107] Optionally, the third intermediate matrix and the second intermediate matrix can be scaled, which can be achieved in the following way:

[0108] For any row of the second intermediate matrix and the corresponding row of the third intermediate matrix, if the maximum value of the i1th row of the first intermediate matrix is ​​less than the i1th element of the seventh vector, then based on the maximum value of the i1th row of the first intermediate matrix and the i1th element of the seventh vector, each element of the i1th row of the second intermediate matrix is ​​scaled; the seventh vector is the vector formed by the maximum value of the second vector before the update corresponding to each of the N word inference subtasks; the value of i1 ranges from 1 to N;

[0109] If the maximum value of the i1th row in the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, each element of the i1th row in the third intermediate matrix is ​​scaled.

[0110] Optionally, the fourth and fifth vectors can be scaled, which can be achieved as follows:

[0111] For any element in the fourth vector and the corresponding element in the fifth vector, if the maximum value of the i1th row in the first intermediate matrix is ​​less than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, for the element s in the fourth vector... i1 Perform scaling processing;

[0112] If the maximum value in the i1th row of the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value in the i1th row of the first intermediate matrix and the i1th element in the seventh vector, the element s in the fifth vector is... i1 Perform scaling.

[0113] Specifically, before superimposing, the second and third intermediate matrices need to be scaled. The scaling rule is to compare the size of the two elements at corresponding positions in the vectors max1 and max2, and scale the corresponding row elements of the intermediate matrix corresponding to the smaller element. For example, if the i1-th element in the max1 vector is less than the i1-th element in the max2 vector, then the second intermediate matrix is ​​scaled. This is done by multiplying any element in the i1-th row of the second intermediate matrix by exp(max_a - max_b), where max_a is the larger of the i1-th elements in both vectors max1 and max2, and max_b is the smaller. Similarly, if the i1-th element in the max1 vector is greater than the i1-th element in the max2 vector, then the third intermediate matrix is ​​scaled. After scaling, the two intermediate matrices are superimposed.

[0114] Optionally, such as Figure 6 As shown, vectors s1 and s2 are also scaled, and the two scaled vectors are superimposed bit by bit. The scaling principle is the same as the principle described above. For example, if the i1th element in vector max1 is less than the i1th element in vector max2, then the i1th element in vector s1 is scaled. If the i1th element in vector max1 is greater than the i1th element in vector max2, then the i1th element in vector s2 is scaled.

[0115] The method described in this invention first avoids the repeated storage of the portion corresponding to the cue length in the k_cache and v_cache tensors, reducing the space occupied by the GPU. Then, a segmented calculation method is adopted for the attenuation operation in each time step of inference (i.e., it is calculated in two parts: cue inference and token inference). Furthermore, the matrix-vector multiplication operation that originally required 2 × beam size operations in cue inference is reduced to 2 matrix multiplication operations. Utilizing the tensor accelerator in the computing unit can significantly improve the computation speed of this step, thereby improving the overall inference speed of the model. This invention has a better inference acceleration effect on large GPT-type models with long cues.

[0116] The data processing apparatus provided by the present invention will be described below. The data processing apparatus described below can be referred to in correspondence with the data processing method described above.

[0117] Figure 8 This is a schematic diagram of the structure of the computing device provided by the present invention. Figure 8As shown, the computing device provided in this embodiment includes: a processor 801 and a memory 802. The memory includes a video memory unit, which stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of the second sub-matrix of the first matrix, and N copies of the second sub-matrix of the second matrix, where N is an integer greater than 0. The width of the first sub-matrix is ​​the prompt length, and the width of the second sub-matrix is ​​the number of tokens generated by reasoning. The processor 801 is coupled to the memory 802, and the processor 801 is configured to:

[0118] For N prompt reasoning subtasks with the same prompt, the third matrix is ​​multiplied by the first submatrix in the first matrix to obtain the first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors.

[0119] Based on the first intermediate matrix and the first submatrix in the second matrix, perform matrix multiplication to obtain the second intermediate matrix;

[0120] For any of the N word inference subtasks mentioned in the prompt, and for one of the N first vectors corresponding to the word inference subtask, the first vector is multiplied by one of the N second submatrices of the first matrix to obtain a second vector, and a third vector is obtained based on the second vector and the second submatrices of the second matrix.

[0121] The third intermediate matrix obtained by stacking N third vectors is superimposed on the second intermediate matrix to obtain the result matrix.

[0122] Optionally, the processor 301 is specifically configured to:

[0123] Update each element in the first intermediate matrix based on the maximum value of each row in the first intermediate matrix;

[0124] Perform matrix multiplication on the updated first intermediate matrix and the first submatrix in the second matrix to obtain the second intermediate matrix.

[0125] Optionally, the processor 301 is specifically configured to:

[0126] The updated second vector is obtained by updating each element of the second vector based on the maximum value in the second vector.

[0127] The updated second vector and the second submatrix in the second matrix are multiplied to obtain the third vector.

[0128] Optionally, the processor 301 is further configured to:

[0129] The sum of the elements in each row of the updated first intermediate matrix, and the maximum value in each row of the first intermediate matrix, are cached locally.

[0130] The sum of the elements in the updated second vector corresponding to each of the N word inference subtasks, and the maximum value in the unupdated second vector corresponding to each word inference subtask, are cached locally.

[0131] Optionally, the processor 301 is specifically configured to:

[0132] The third intermediate matrix and the second intermediate matrix are scaled, and the two scaled intermediate matrices are superimposed to obtain the fourth intermediate matrix.

[0133] The fourth and fifth vectors are scaled, and the two vectors obtained after scaling are superimposed bitwise to obtain the sixth vector; the fourth vector is the vector formed by the sum of the elements of each row in the updated first intermediate matrix, and the fifth vector is the vector formed by the sum of the elements of the updated second vector corresponding to each of the N word inference subtasks.

[0134] For each row of the fourth intermediate matrix, each element of each row of the fourth intermediate matrix is ​​divided by the element at the corresponding position in the sixth vector to obtain the result matrix.

[0135] Optionally, the processor 301 is specifically configured to:

[0136] For any row of the second intermediate matrix and the corresponding row of the third intermediate matrix, if the maximum value of the i1th row of the first intermediate matrix is ​​less than the i1th element of the seventh vector, then based on the maximum value of the i1th row of the first intermediate matrix and the i1th element of the seventh vector, each element of the i1th row of the second intermediate matrix is ​​scaled; the seventh vector is the vector formed by the maximum value of the second vector before the update corresponding to each of the N word inference subtasks; the value of i1 ranges from 1 to N;

[0137] If the maximum value of the i1th row in the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, each element of the i1th row in the third intermediate matrix is ​​scaled.

[0138] Optionally, the processor 301 is specifically configured to:

[0139] For any element in the fourth vector and the corresponding element in the fifth vector, if the maximum value of the i1th row in the first intermediate matrix is ​​less than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, for the element s in the fourth vector... i1 Perform scaling processing;

[0140] If the maximum value in the i1th row of the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value in the i1th row of the first intermediate matrix and the i1th element in the seventh vector, the element s in the fifth vector is... i1 Perform scaling.

[0141] Optionally, the processor 301 is specifically configured to:

[0142] For any element in the first intermediate matrix, the element is updated using the following formula (1):

[0143] qk_(ij)'=exp(x_ij-max_i); (1)

[0144] Where qk_(ij)' represents the element in the i-th row and j-th column of the first intermediate matrix after the update, x_ij represents the element in the i-th row and j-th column of the first intermediate matrix before the update, max_i represents the maximum value in the i-th row of the first intermediate matrix, and exp represents the exponential operation with base e.

[0145] Optionally, the processor 301 is specifically configured to:

[0146] For any element in the second vector, the element is updated using the following formula (2):

[0147] qk_(m)'=exp(x_m-max2 k (2)

[0148] Where qk_(m)' represents the m-th element in the second vector after the update, x_m represents the m-th element in the second vector before the update, and max2 k represents the maximum value in the second vector, and exp represents the exponential operation with base e.

[0149] Optionally, the first matrix and the second matrix are matrices in the target model based on the attention mechanism.

[0150] The computing device of this invention is used to execute the method in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0151] Figure 9 An example is a schematic diagram of the physical structure of a computing device, such as... Figure 9 As shown, the computing device may include a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a data processing method, which includes:

[0152] For N prompt reasoning subtasks with the same prompt, the third matrix is ​​multiplied by the first submatrix in the first matrix to obtain the first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors.

[0153] Based on the first intermediate matrix and the first submatrix in the second matrix, perform matrix multiplication to obtain the second intermediate matrix;

[0154] For any of the N word inference subtasks mentioned in the prompt, and for one of the N first vectors corresponding to the word inference subtask, the first vector is multiplied by one of the N second submatrices of the first matrix to obtain a second vector, and a third vector is obtained based on the second vector and the second submatrices of the second matrix.

[0155] The third intermediate matrix obtained by stacking N third vectors is superimposed on the second intermediate matrix to obtain the result matrix.

[0156] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the data processing method provided by the above methods, the method including: for N prompt reasoning subtasks of the same prompt, performing a matrix multiplication operation between a third matrix and a first submatrix in the first matrix to obtain a first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors;

[0158] Based on the first intermediate matrix and the first submatrix in the second matrix, perform matrix multiplication to obtain the second intermediate matrix;

[0159] For any of the N word inference subtasks mentioned in the prompt, and for one of the N first vectors corresponding to the word inference subtask, the first vector is multiplied by one of the N second submatrices of the first matrix to obtain a second vector, and a third vector is obtained based on the second vector and the second submatrices of the second matrix.

[0160] The third intermediate matrix obtained by stacking N third vectors is superimposed on the second intermediate matrix to obtain the result matrix.

[0161] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data processing method provided by the above methods, the method comprising: for N prompt reasoning subtasks of the same prompt, performing a matrix multiplication operation between a third matrix and a first submatrix in the first matrix to obtain a first intermediate matrix; wherein the third matrix is ​​obtained based on a combination of N first vectors;

[0162] Based on the first intermediate matrix and the first submatrix in the second matrix, perform matrix multiplication to obtain the second intermediate matrix;

[0163] For any of the N word inference subtasks mentioned in the prompt, and for one of the N first vectors corresponding to the word inference subtask, the first vector is multiplied by one of the N second submatrices of the first matrix to obtain a second vector, and a third vector is obtained based on the second vector and the second submatrices of the second matrix.

[0164] The third intermediate matrix obtained by stacking N third vectors is superimposed on the second intermediate matrix to obtain the result matrix.

[0165] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, The method is applied to a computing device, which includes a computing unit and a video memory unit. The video memory unit stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of the second sub-matrix of the first matrix, and N copies of the second sub-matrix of the second matrix, where N is an integer greater than 0. The width of the first sub-matrix is ​​the prompt length, and the width of the second sub-matrix is ​​the number of tokens generated by inference. The method includes: For N prompt reasoning subtasks with the same prompt, the computing unit performs matrix multiplication on the third matrix and the first submatrix in the first matrix to obtain a first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors. The calculation unit performs matrix multiplication based on the first intermediate matrix and the first submatrix in the second matrix to obtain the second intermediate matrix; For any of the N word inference subtasks mentioned in the prompt, and a first vector among the N first vectors corresponding to the word inference subtask, the computing unit performs a product operation between the first vector and a second submatrix among the N second submatrices of the first matrix to obtain a second vector, and obtains a third vector based on the second vector and the second submatrix in the second matrix; The computing unit performs a superposition operation on the third intermediate matrix obtained by stacking N third vectors and the second intermediate matrix to obtain the result matrix.

2. The data processing method according to claim 1, characterized in that, The calculation unit performs matrix multiplication based on the first intermediate matrix and the first submatrix in the second matrix to obtain the second intermediate matrix, including: The calculation unit updates each element in the first intermediate matrix based on the maximum value of each row in the first intermediate matrix; The calculation unit performs a matrix multiplication operation on the updated first intermediate matrix and the first submatrix in the second matrix to obtain the second intermediate matrix.

3. The data processing method according to claim 1 or 2, characterized in that, The calculation unit obtains a third vector based on the second vector and the second submatrix in the second matrix, including: The calculation unit updates each element of the second vector based on the maximum value in the second vector to obtain the updated second vector; The calculation unit performs a product operation on the updated second vector and the second submatrix in the second matrix to obtain the third vector.

4. The data processing method according to claim 3, characterized in that, The method further includes: The computing unit caches the sum of the elements in each row of the updated first intermediate matrix and the maximum value of each row of the first intermediate matrix locally in the computing unit. The computing unit caches the sum of elements in the updated second vector corresponding to each of the N word inference subtasks, as well as the maximum value in the unupdated second vector corresponding to each word inference subtask, locally within the computing unit.

5. The data processing method according to claim 2, characterized in that, The computing unit performs a superposition operation on the third intermediate matrix obtained by stacking the N third vectors and the second intermediate matrix to obtain a result matrix, including: The calculation unit scales the third intermediate matrix and the second intermediate matrix, and then performs a superposition operation on the two scaled intermediate matrices to obtain a fourth intermediate matrix. The calculation unit scales the fourth and fifth vectors and performs a bitwise superposition operation on the two vectors obtained after scaling to obtain the sixth vector; the fourth vector is the vector formed by the sum of the elements in each row of the updated first intermediate matrix, and the fifth vector is the vector formed by the sum of the elements in the updated second vector corresponding to each of the N word inference subtasks; For each row of the fourth intermediate matrix, the calculation unit divides each element of each row of the fourth intermediate matrix by the element at the corresponding position in the sixth vector to obtain the result matrix.

6. The data processing method according to claim 5, characterized in that, The scaling process between the third intermediate matrix and the second intermediate matrix includes: For any row of the second intermediate matrix and the corresponding row of the third intermediate matrix, if the maximum value of the i1th row of the first intermediate matrix is ​​less than the i1th element of the seventh vector, then based on the maximum value of the i1th row of the first intermediate matrix and the i1th element of the seventh vector, each element of the i1th row of the second intermediate matrix is ​​scaled; the seventh vector is the vector formed by the maximum value of the second vector before the update corresponding to each of the N word inference subtasks; the value of i1 ranges from 1 to N; If the maximum value of the i1th row in the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, each element of the i1th row in the third intermediate matrix is ​​scaled. The scaling process for the fourth and fifth vectors includes: For any element in the fourth vector and the corresponding element in the fifth vector, if the maximum value of the i1th row in the first intermediate matrix is ​​less than the i1th element in the seventh vector, then based on the maximum value of the i1th row in the first intermediate matrix and the i1th element in the seventh vector, for the element s in the fourth vector... i1 Perform scaling processing; If the maximum value in the i1th row of the first intermediate matrix is ​​greater than the i1th element in the seventh vector, then based on the maximum value in the i1th row of the first intermediate matrix and the i1th element in the seventh vector, the element s in the fifth vector is... i1 Perform scaling.

7. The data processing method according to claim 2, characterized in that, The calculation unit updates each element in the first intermediate matrix based on the maximum value of each row in the first intermediate matrix, including: For any element in the first intermediate matrix, the calculation unit updates the element using the following formula (1): qk_(ij)'=exp(x_ij-max_i); (1) Where qk_(ij)' represents the element in the i-th row and j-th column of the first intermediate matrix after the update, x_ij represents the element in the i-th row and j-th column of the first intermediate matrix before the update, max_i represents the maximum value in the i-th row of the first intermediate matrix, and exp represents the exponential operation with base e.

8. The data processing method according to claim 3, characterized in that, The calculation unit updates each element of the second vector based on the maximum value in the second vector to obtain the updated second vector, including: For any element in the second vector, the calculation unit updates the element using the following formula (2): qk_(m)’=exp(x_m-max2 k );(2) Where qk_(m)' represents the m-th element in the second vector after the update, x_m represents the m-th element in the second vector before the update, and max2 k represents the maximum value in the second vector, and exp represents the exponential operation with base e.

9. The data processing method according to claim 1 or 2, characterized in that, The first matrix and the second matrix are matrices in the target model based on the attention mechanism.

10. A computing device, characterized in that, include: Memory; The memory includes: a video memory unit; the video memory unit stores a first sub-matrix of a first matrix, a first sub-matrix of a second matrix, N copies of the second sub-matrix of the first matrix, and N copies of the second sub-matrix of the second matrix, where N is an integer greater than 0, the width of the first sub-matrix is ​​the prompt length, and the width of the second sub-matrix is ​​the number of tokens generated by reasoning; A processor, coupled to the memory, is configured to: For N prompt reasoning subtasks with the same prompt, the third matrix is ​​multiplied by the first submatrix in the first matrix to obtain the first intermediate matrix; the third matrix is ​​obtained based on the combination of N first vectors. Based on the first intermediate matrix and the first submatrix in the second matrix, perform matrix multiplication to obtain the second intermediate matrix; For any of the N word inference subtasks mentioned in the prompt, and for one of the N first vectors corresponding to the word inference subtask, the first vector is multiplied by one of the N second submatrices of the first matrix to obtain a second vector, and a third vector is obtained based on the second vector and the second submatrices of the second matrix. The third intermediate matrix obtained by stacking N third vectors is superimposed on the second intermediate matrix to obtain the result matrix.

11. A computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data processing method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 9.

Images