Data processing method and apparatus

Abstract

The application provides a data processing method and device, the method comprising: obtaining input data of a model, the model comprising a low-rank adaptive LoRA module; constructing a first calculation graph according to the input data and a script of the model, the first calculation graph comprising a plurality of batched matrix multiplication BMM nodes; and running the model on a processor according to the input data and the first calculation graph to obtain output data, an optimal performance of the processor corresponding to a matrix rank M, an incremental memory occupation of the model being proportional to the square of M, the incremental memory occupation comprising an incremental memory usage of the model using t+1 LoRA modules relative to using t LoRA modules, M>1, and t≥1. The data processing method provided by the application can reduce the waste of memory in the model training process.

Description

[0001] This application claims priority to Russian patent application filed on March 3, 2025, with application number 2025104837 and entitled "Data Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and more specifically to a data processing method and apparatus. Background Technology

[0003] Low-rank adaptation (LoRA) is a technique that enhances the performance of large pre-trained models by introducing low-rank updates during fine-tuning. This technique can efficiently adapt to the training task while maintaining a manageable number of training parameters. LoRA is advantageous in scenarios with limited computational resources or where rapid model tuning is required.

[0004] Neural network processing units (NPUs) and graphics processing units (GPUs) are designed for large matrix multiplications, and there exists an optimal matrix shape for multiplication operations. Therefore, to achieve optimal performance, LoRA modules need to be padded to the optimal size. This implementation detail reduces the advantage of the low rank of LoRA modules and forces a significant amount of memory to be wasted. Summary of the Invention

[0005] This application provides a data processing method and apparatus that can reduce memory waste during model training or inference.

[0006] In a first aspect, a data processing method is provided, comprising: acquiring input data of a model, the model including low-rank adaptive LoRA modules; constructing a first computation graph based on the input data and the script of the model, the first computation graph including multiple batch matrix multiplication (BMM) nodes; running the model on a processor based on the input data and the first computation graph to obtain output data, wherein the matrix rank corresponding to the optimal performance of the processor is M, the memory usage increment of the model is proportional to the square of M, and the memory usage increment includes the increase in memory usage when the model uses t+1 LoRA modules compared to using t LoRA modules, where M>1 and t≥1.

[0007] In existing technologies, the constructed computation graph typically includes grouped matrix multiplication (GMM) nodes. Each GMM node includes N sets of matrix multiplication calculations, where N is the number of LoRA modules in the model, and each set of matrix multiplications corresponds to one LoRA module. Due to the efficiency requirements of NPUs and GPUs, during computation, the rank of the two multiplied matrices in each LoRA module must first be padded to a hardware-friendly size M. For example, 0 is used as padding data to padded the rank of the two multiplied matrices to M, where 1≤i≤N. To avoid storing unnecessary padding data, this application provides a data processing method that uses BMM nodes instead of GMM nodes. Instead of using additional padding data to padded each matrix and then storing each LoRA module separately, it stores a matrix whose rank is padded to M after merging the matrices of N LoRA modules, plus an additional M×M square matrix. Since the overall dimension of the square matrix is ​​much smaller than the overall dimension of the two multiplied matrices in the LoRA module after padding, the data processing method provided in this application can reduce the model's memory usage and reduce memory waste during model training or inference.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the memory footprint increment of the model is proportional to the square of M, including: P = DM 2 P represents the memory usage increment of the model, and D represents the size of the scalar data type.

[0009] The size of a scalar data type refers to the number of bytes occupied by a specific data type in memory. A matrix can include different data types, and this application does not restrict the data types stored in the matrix.

[0010] This application provides a data processing method in which the memory usage increment of the model is proportional to the square of M. Since the overall dimension of the square matrix is ​​much smaller than the overall dimension of the two multiplied matrices in the LoRA module after being filled, the data processing method provided by this application can reduce the memory usage of the model and reduce the waste of memory during model training or inference.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, constructing the first computation graph based on the input data and the model's script includes: constructing a second computation graph based on the input data and the model's script, the second computation graph including a first grouped matrix multiplication node and a second grouped matrix multiplication node, the first grouped matrix multiplication node including the product of N first matrices and N second matrices, the second grouped matrix multiplication node including the product of N third matrices and N fourth matrices, wherein the i-th third matrix is ​​the product of the i-th first matrix and the i-th second matrix, N is the number of LoRA modules included in the model, 1≤i≤N; replacing the first grouped matrix multiplication node and the second grouped matrix multiplication node in the second computation graph with a first batching... The matrix multiplication node, the third group matrix multiplication node, and the second batch matrix multiplication node are used to obtain the first computation graph. The first batch matrix multiplication node includes the product of the N first matrices and the fifth matrix. The third group matrix multiplication node includes the product of the N sixth matrices and the N seventh matrices. The second batch matrix multiplication node includes the product of the N eighth matrices and the ninth matrix. The fifth matrix includes the N second matrices. The i-th sixth matrix includes the product of the i-th first matrix and the fifth matrix. The seventh matrix is ​​an M x M square matrix. The i-th eighth matrix is ​​the product of the i-th sixth matrix and the i-th seventh matrix. The ninth matrix includes the N fourth matrices. The first batch matrix multiplication node and the second batch matrix multiplication node belong to the BMM node.

[0012] The first and second grouped matrix multiplication nodes in the second computation graph both include N groups of matrix multiplication calculations, where N is the number of LoRA modules included in the model. The first grouped matrix multiplication node GMM1 includes N first matrices X. i and N second matrices A i The product operation, the second grouped matrix multiplication node GMM2 includes N third matrices C i and N fourth matrices B i The product operation, where the i-th third matrix C i For the i-th first matrix X i and the i-th second matrix A i The product of C i =X i A i , 1≤i≤N.

[0013] By replacing the first grouped matrix multiplication node GMM1 and the second grouped matrix multiplication node GMM2 in the second computation graph with the first batched matrix multiplication node BMM1, the third grouped matrix multiplication node GMM3, and the second batched matrix multiplication node BMM2, the first computation graph can be obtained.

[0014] The first batch matrix multiplication node BMM1 includes N first matrices X. i The product of the fifth matrix A and the third grouped matrix multiplication node GMM3 includes N sixth matrices E. i and N seventh matrices D i The product, the second batch matrix multiplication node BMM2 includes N eighth matrices G i The product of the fifth matrix A and the ninth matrix B. The fifth matrix A consists of N second matrices A. i For example, N second matrices A i The merged matrix is ​​filled with a rank of M to form the i-th sixth matrix E. i Including the i-th first matrix X i The product of the fifth matrix A and E i =X i A. The seventh matrix D i Let G be an M×M square matrix, and let G be the i-th octet. i For the i-th sixth matrix E i and the i-th seventh matrix D i The product result, i.e., G i =X i AD i The ninth matrix B comprises N fourth matrices B. i For example, N second matrices B i The merged rank is filled with M and then formed.

[0015] Due to the efficiency requirements of NPUs and GPUs, during computation, the rank of the two matrices being multiplied in each LoRA module of the model must first be padded to a hardware-friendly size M. For example, 0 is used as padding data to padded the first matrix X. i Second matrix A i and the fourth matrix B i The rank is filled into M, 1≤i≤N. Since A i =A×D i B i =D i ×B, therefore X i ×A i ×B i =X i ×A×D i ×B, D i For a square matrix of M rows and M columns, two GMM nodes can be replaced with two smaller BMM nodes and one GMM node. To avoid storing unnecessary padding data, the data processing method provided in this application uses the second matrix A in the GMM node. i Merge the matrices, then fill the rank of the merged matrix into M to obtain the fifth matrix A; then merge the fourth matrix B from the GMM nodes. iMerge the matrices, then fill the rank of the merged matrix into M to obtain the ninth matrix B; additionally store N square matrices D. i The model's memory usage increment is proportional to the square of M, since D i The overall dimension is much smaller than that of the second matrix A. i and the fourth matrix B i Therefore, the first computational graph constructed in this embodiment uses A×D as its overall dimension. i ×B replaces multiple A's i ×B i This can reduce the memory usage of the model and reduce memory waste during model training or inference.

[0016] The data processing method provided in this application embodiment has a memory footprint of F = DM(L) A +L B )+DNM 2 F represents the memory occupied by the LoRA modules in the model, D represents the size of the scalar data type, N represents the number of LoRA modules included in the model, and L represents the memory occupied by the LoRA modules in the model. A L represents the other dimension of matrix A besides M. B This represents the other dimension of matrix B besides M. Since the memory usage increment P includes the increase in memory usage compared to using t+1 LoRA modules, P = DM(L... A +L B )+D(t+1)M 2 -DM(L A +L B )-DtM 2 =DM 2 .

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the square matrix includes a diagonal matrix.

[0018] A diagonal matrix is ​​a special type of square matrix where all elements off-diagonal are zero, and only the elements on the main diagonal can be non-zero. This application allows for the setting of a diagonal matrix. D i Let r represent the diagonal matrix corresponding to the i-th LoRA module. i For A i column width or B i line width, d jj D represents i The value of the element in the j-th row and j-th column.

[0019] This application provides a data processing method that, instead of using additional padding data to fill each matrix and then storing each LoRA module separately, stores the matrices of N LoRA modules, merges them, fills the rank of the resulting M-matrix, and then adds an additional M×M square matrix. Since the overall dimension of the square matrix is ​​much smaller than the overall dimension of the two multiplied matrices in the LoRA module after individual padding, the data processing method provided in this application can reduce the model's memory footprint and minimize memory waste during model training or inference.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the N seventh matrices are three-dimensional matrices with N layers, M rows, and M columns.

[0021] Matrix D (Seventh Matrix) i For an M×M square matrix, in the embodiments of this application, N seventh matrices D can be used. i The data can be merged into a single N×M×M three-dimensional matrix for storage or use, thereby improving computational performance.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, M equals a power of 2.

[0023] For example, r i Typical values ​​for r are 8, 16, 32, 64, or 128. Generally, r... i It is 8 or 16. For processors that use tensor cores, the processor can achieve optimal performance when M is a multiple of 16 or 32, and it is generally not less than 64.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the processor includes an NPU or a GPU.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the model includes any one or more of the following: text generation tools, content creation tools, text classification systems, machine translation systems, or inference systems.

[0026] The data processing method provided in this application can be used to handle various tasks, such as sentiment analysis, summarization, or question answering. The model can be implemented as a software or hardware / software hybrid product, such as a text generation tool, content creation tool, text classification system, machine translation system, or any other product or framework for other tasks, such as an inference system. The inference system can include an inference engine and an inference scheduling framework, and multiple LoRA modules can be deployed simultaneously to achieve efficient task management without retraining the basic model. For systems with limited computing resources, the data processing method provided in this application can maximize the performance of existing infrastructure and reduce the need for additional hardware by using a single model to serve multiple tasks.

[0027] In a second aspect, a computer device is provided, comprising: an acquisition module for acquiring input data of a model, the model including low-rank adaptive LoRA modules; a processing module for constructing a first computation graph based on the input data and the script of the model, the first computation graph including multiple batch matrix multiplication (BMM) nodes; the processing module is further configured to run the model on a processor based on the input data and the first computation graph to obtain output data, wherein the optimal performance of the processor corresponds to a matrix rank of M, the memory usage increment of the model is proportional to the square of M, the memory usage increment includes the increase in memory usage when the model uses t+1 LoRA modules compared to using t LoRA modules, where M>1, t≥1.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, the memory footprint increment of the model is proportional to the square of M, including: P = DM 2 P represents the memory usage increment of the model, and D represents the size of the scalar data type.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, the processing module is specifically used to: construct a second computation graph based on the input data and the script of the model, the second computation graph including a first grouped matrix multiplication node and a second grouped matrix multiplication node, the first grouped matrix multiplication node including the product of N first matrices and N second matrices, the second grouped matrix multiplication node including the product of N third matrices and N fourth matrices, wherein the i-th third matrix is ​​the product of the i-th first matrix and the i-th second matrix, N is the number of LoRA modules included in the model, 1≤i≤N; replace the first grouped matrix multiplication node and the second grouped matrix multiplication node in the second computation graph with a first batched matrix multiplication node and a third... The first computation graph is obtained by grouping matrix multiplication nodes and second batch matrix multiplication nodes. The first batch matrix multiplication node includes the product of the N first matrices and the fifth matrix. The third group matrix multiplication node includes the product of the N sixth matrices and the N seventh matrices. The second batch matrix multiplication node includes the product of the N eighth matrices and the ninth matrix. The fifth matrix includes the N second matrices. The i-th sixth matrix includes the product of the i-th first matrix and the fifth matrix. The seventh matrix is ​​an M-row M-column square matrix. The i-th eighth matrix is ​​the product of the i-th sixth matrix and the i-th seventh matrix. The ninth matrix includes the N fourth matrices. The first batch matrix multiplication node and the second batch matrix multiplication node belong to the BMM node.

[0030] In conjunction with the second aspect, in some implementations of the second aspect, the square matrix includes a diagonal matrix.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, the N seventh matrices are three-dimensional matrices with N layers, M rows, and M columns.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, M equals a power of 2.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the processor includes an NPU or a GPU.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, the model includes any one or more of the following: text generation tools, content creation tools, text classification systems, machine translation systems, or inference systems.

[0035] The beneficial effects of the second aspect and any possible implementation of the second aspect correspond to the beneficial effects of the first aspect and any possible implementation of the first aspect, which will not be elaborated further.

[0036] Thirdly, embodiments of this application provide a computer device including a processor for coupling with a memory to read and execute instructions and / or program code in the memory to perform the first aspect or any possible implementation of the first aspect.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code that, when executed on a computer, causes the computer to perform the first aspect or any possible implementation thereof.

[0038] Fifthly, embodiments of this application provide a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform as in the first aspect or any possible implementation thereof. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a LoRA implementation provided in an embodiment of this application.

[0040] Figure 2 This is a schematic diagram of LoRA revenue provided in an embodiment of this application.

[0041] Figure 3 This is an exemplary flowchart of a data processing method provided in an embodiment of this application.

[0042] Figure 4 This is an exemplary flowchart of another data processing method provided in the embodiments of this application.

[0043] Figure 5 This is a schematic diagram of a second calculation graph provided in an embodiment of this application.

[0044] Figure 6 This is a schematic diagram of LoRA module factorization provided in the embodiments of this application.

[0045] Figure 7 This is a schematic diagram of an equivalent LoRA module provided in an embodiment of this application.

[0046] Figure 8 This is a schematic diagram of a first computational graph provided in an embodiment of this application.

[0047] Figure 9 This is a schematic diagram of the storage scheme provided in the embodiments of this application.

[0048] Figure 10 This is a structural example diagram of a computer device provided in an embodiment of this application.

[0049] Figure 11 This is a structural example diagram of another computer device provided in the embodiments of this application.

[0050] Figure 12 This is an example diagram of a computer program product provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort should fall within the scope of protection of this application.

[0052] In the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design options. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0053] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0054] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0055] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0056] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0057] Figure 1 This is a schematic diagram of a LoRA implementation provided in an embodiment of this application.

[0058] Low-rank adaptation (LoRA) is a technique that enhances the performance of large pre-trained models by introducing low-rank updates during fine-tuning. This technique can efficiently adapt to the training task while maintaining a manageable number of training parameters. LoRA is advantageous in scenarios with limited computational resources or where rapid model tuning is required.

[0059] LoRA reduces the storage and computational resources required during training by performing a low-rank decomposition on the weight update matrix ΔW, representing it as the product of two low-rank matrices J and K. In traditional full-parameter fine-tuning, all weight matrices W are updated, leading to huge parameter storage requirements, especially with large-scale pre-trained models. LoRA, by introducing low-rank decomposition, optimizes only a low-rank approximation of the weight update matrix, thereby reducing computational costs and storage requirements.

[0060] Specifically, LoRA adjusts the model weights as follows: Let the original weight matrix be W, and its update amount be denoted as ΔW, which is approximately equal to JK. The updated weight matrix (W') then becomes W' = W + JK. Matrix J has a size of d × r, and matrix K has a size of r × d, where r is much smaller than d. Matrix J and matrix K have low dimensions, and their product JK represents a low-rank approximation of ΔW. W is the original pre-trained weight matrix, which remains unchanged. JK is the learned low-rank compensation term used to adjust W. Because J and K have low ranks (i.e., small r), LoRA only needs to optimize matrices J and K, without directly optimizing the complete W, thus greatly reducing the number of parameters and computational cost required for training.

[0061] Figure 2 This is a schematic diagram of LoRA revenue provided in an embodiment of this application.

[0062] During inference, for different LoRA modules trained on the same base model, the current mainstream approach is servicing low-rank adaptation (sLoRA), which merges multiple requests into batch processing to improve the resource utilization of the neural network processing unit (NPU). sLoRA can serve thousands of concurrent LoRA modules, aiming to facilitate the simultaneous deployment of multiple LoRA modules for large language models (LLMs). By utilizing a single pre-trained model, sLoRA can efficiently adapt to various tasks without the overhead of maintaining multiple complete models. To achieve this computation, grouped matrix multiplication (GMM) is required. Through GMM, the correspondence between different input data (X) and different LoRA weights is found, ensuring correct computation.

[0063] However, NPUs, GPUs, and other chips are designed for large matrix multiplications, and there exists an optimal matrix shape for multiplication operations. Therefore, to achieve optimal performance, LoRA modules need to be padded to the optimal size. Table 1 shows an example of different performance characteristics of LoRA modules with different ranks, as provided in the embodiments of this application. Generally, the rank of a LoRA module is 8 or 16, but for computing devices using tensor cores, the rank dimension of the LoRA module needs to be padded to 64 to achieve optimal performance. As shown in Table 1, matrix multiplication is inefficient at low rank, and optimal performance is achieved only when the rank is 64. This implementation detail diminishes the advantage of low-rank LoRA modules and forces a significant waste of memory.

[0064] Table 1

[0065]

[0066] Some characteristics of NPUs and GPUs make it difficult to fully utilize the low-rank properties of LoRA modules. Furthermore, GMM requires merging requests from different LoRA modules into a single batch, making it more complex and slower than BMM. GMM is approximately 20% slower than batched matrix multiplication (BMM), resulting in lower computational performance. This application provides a data processing method that can reduce the memory usage and execution time overhead of GMM.

[0067] The data processing method provided in this application can be used to handle various tasks, such as sentiment analysis, summarization, or question answering. It can be implemented as a software or hardware / software combined product, such as a text generation tool, content creation tool, text classification system, machine translation system, or any other product or framework for other tasks, such as an inference system. The inference system can include an inference engine and an inference scheduling framework, and multiple LoRA modules can be deployed simultaneously to achieve efficient task management without retraining the basic model. For systems with limited computing resources, the data processing method provided in this application can maximize the performance of existing infrastructure and reduce the need for additional hardware by using a single model to serve multiple tasks.

[0068] Figure 3 This is an exemplary flowchart of a data processing method provided in an embodiment of this application.

[0069] 310. Obtain the input data for the model.

[0070] The model includes LoRA modules, and users can set the number of LoRA modules according to their actual needs. Input data can be, for example, questions entered by users in artificial intelligence (AI) question-answering systems.

[0071] The data processing method provided in this application can be used to process various tasks, such as sentiment analysis, summarization, or question answering. The model can be implemented as a software product or a combination of software and hardware, such as a text generation tool, content creation tool, text classification system, machine translation system, or any other product or framework for other tasks, such as an inference system.

[0072] 320. Construct the first computation graph.

[0073] The first computational graph is constructed based on the input data and the model's script. The first computational graph includes multiple BMM nodes.

[0074] In some possible implementations, a second computation graph can be constructed first by inputting data and model scripts, and then the first grouped matrix multiplication node GMM1 and the second grouped matrix multiplication node GMM2 in the second computation graph can be replaced with the first batched matrix multiplication node BMM1, the third grouped matrix multiplication node GMM3 and the second batched matrix multiplication node BMM2 to obtain the first computation graph.

[0075] In another possible implementation, the first computation graph can be constructed directly through the input data and the model's script. For example, during the construction of the first computation graph, the first group matrix multiplication node GMM1 and the second group matrix multiplication node GMM2 can be identified, and the first group matrix multiplication node GMM1 and the second group matrix multiplication node GMM2 can be replaced with the first batch matrix multiplication node BMM1, the third group matrix multiplication node GMM3, and the second batch matrix multiplication node BMM2.

[0076] Each GMM node, such as GMM1 and GMM2, includes N sets of matrix multiplication calculations, where N is the number of LoRA modules in the model, and each set of matrix multiplications corresponds to one LoRA module. Due to the efficiency requirements of NPUs and GPUs, during the computation process, the rank of the two multiplied matrices in each LoRA module of the model must first be padded to a hardware-friendly size M. For example, using 0 as padding data to padded the rank of the two multiplied matrices to M, 1≤i≤N, would waste a lot of memory. To avoid storing unnecessary padding data, the data processing method provided in this application separates the N matrices A in the N sets of matrix multiplication calculations in the GMM node. i Merge the matrices, then fill the rank of the merged matrix into M to obtain matrix A; calculate the N matrices B in the N sets of matrix multiplications. iMerge, then fill the rank of the merged matrix into M to obtain matrix B. Since A i =A×D i B i =D i ×B, therefore A i ×B i =A×D i ×B, D i Given an M x M square matrix, the memory usage increment of the model is proportional to the square of M. Since D... i The overall dimension is much smaller than that of matrix A i Sum matrix B i The dimensions are filled, therefore the first computational graph constructed in this embodiment uses A×D. i ×B replaces multiple A's i ×B i This can reduce model memory usage and minimize memory waste during model training or inference. For specific steps on replacing GMM nodes, please refer to [link / reference needed]. Figure 4 .

[0077] In some possible implementations, D i It can be a diagonal matrix, for example D i Let r represent the diagonal matrix corresponding to the i-th LoRA module. i For A i column width or B i line width, d jj D represents i The value of the element in the j-th row and j-th column.

[0078] It should be understood that the above-described acceleration of GMM nodes in the computation graph is merely an example. The data processing method provided in this application embodiment can also accelerate similar GMM nodes, such as serving grouped matrix-vector (SGMV) nodes, and this application does not impose any limitations on this.

[0079] 330. Run the model and obtain the output data.

[0080] The model is run on a processor based on the input data and the first computational graph to obtain output data. The model runs on a processor, such as an NPU or GPU, where the optimal performance of the processor corresponds to a matrix rank of M; that is, the processor processes matrices of rank M most efficiently. The increase in model memory usage is proportional to the square of M. This increase includes the additional memory usage when using t+1 LoRA modules compared to using t LoRA modules, where t ≥ 1, assuming other parameters remain constant. For example, P = DM. 2P represents the incremental memory usage of the model, and D represents the size of the scalar data type. That is, for each additional LoRA module used in the model, the memory usage increment is DM. 2 .

[0081] In some possible scenarios, the matrix in the embodiments of this application may also be called a vector, and this application does not limit this.

[0082] Figure 4 This is an exemplary flowchart of another data processing method provided in the embodiments of this application.

[0083] 410. Construct a second computational graph based on the input data and model script.

[0084] The process involves acquiring input data X and parsing the model script to construct a second computational graph. A model script typically refers to a code file or program used to define, train, and deploy a machine learning or deep learning model. The second computational graph can, for example, include a directed acyclic graph (DAG) describing the computational operations and dependencies within the machine learning or deep learning model. This process helps identify computational patterns, optimization opportunities, and dependency paths within the model.

[0085] The second computation graph includes multiple Grouped Matrix Multiplication (GMM) nodes. A GMM node is an operation node used to efficiently perform grouped matrix multiplication, commonly found in deep learning frameworks or high-performance computing libraries. Its main purpose is to group multiple independent matrix multiplication operations and complete the computation in a single call, thereby improving computational efficiency. For example, there are multiple matrix pairs (A1, B1), (A2, B2), ..., (A... n B n Each pair of matrices requires matrix multiplication A. i ×B i n is an integer greater than 1, 1≤i≤n. GMM nodes group these independent matrix multiplication operations and complete all calculations in a single call.

[0086] Figure 5 This is a schematic diagram of a second computation graph provided in an embodiment of this application. The diagram includes two GMM nodes: a first grouped matrix multiplication node GMM1 and a second grouped matrix multiplication node GMM2. The model includes N LoRA modules, therefore each GMM node includes N groups of matrix multiplication calculations, where N equals 3 in this example. The first grouped matrix multiplication node GMM1 includes N first matrices X... i and N second matrices A i The product operation includes matrix multiplication of X1A1, X2A2, and X3A3. The second grouped matrix multiplication node GMM2 includes N third matrices C. iand N fourth matrices B i The product operation, where the i-th third matrix C i For the i-th first matrix X i and the i-th second matrix A i The product of C i =X i A i , 1≤i≤N, therefore, Figure 5 In GMM2, matrix multiplication calculations of X1A1B1, X2A2B2, and X3A3B3 are included.

[0087] 420, Identify nodes that can be accelerated.

[0088] In the second computation graph, GMM nodes are identified as candidate nodes that need to be accelerated, especially those computation nodes that need to be populated with a large number of values, as these nodes would cause a waste of computational resources.

[0089] In one possible implementation scenario, GMM nodes can be located by traversing the second computation graph. The weight size of each GMM node is examined to identify nodes that contain padding data. For example, if the weight matrix dimension is aligned to a fixed size (such as the block size required by the hardware), resulting in additional zero padding, nodes with input padding sizes exceeding a preset threshold are marked as optimization candidate nodes.

[0090] 430, Replace the GMM node.

[0091] Figure 6 This is a schematic diagram of LoRA module factorization provided in an embodiment of this application. This embodiment introduces a new square matrix D. i A can be obtained i =A×D i B i =D i ×B. Among them... D i Let r represent the diagonal matrix corresponding to the i-th LoRA module. i For A i column width or B i line width, d jj D represents i The value of the element in the j-th row and j-th column. It should be understood that D in the embodiments of this application... i It can be a diagonal matrix or not; this application does not impose any restrictions on this.

[0092] At the same time, exact non-zero indexes of columns and rows are not important in the LoRA module, such as Figure 7 As shown, Figure 7 This is a schematic diagram of an equivalent LoRA module provided in an embodiment of this application. Figure 7 All LoRA module matrices shown in (a), (b), (c), and (d) are equivalent.

[0093] Furthermore, due to A i =A×D i B i =D i ×B, therefore X i ×A i ×B i =X i ×(A×D i )×(D i ×B), because D i ×D i =D i Therefore, X i ×A i ×B i =X i ×A×D i ×B, X i This is the input data for the model.

[0094] According to X i ×A i ×B i =X i ×A×D i ×B, it can be concluded that more computations can be performed using more efficient BMM operations. This embodiment of the application can achieve this by modifying the second computation graph. In this embodiment, the first grouped matrix multiplication node GMM1 and the second grouped matrix multiplication node GMM2 in the second computation graph are replaced with the first batched matrix multiplication node BMM1, the third grouped matrix multiplication node GMM3, and the second batched matrix multiplication node BMM2, resulting in the first computation graph. The more efficient BMM nodes will serve as input matrices and perform the necessary computations.

[0095] Figure 8 This is a schematic diagram of a first computational graph provided in an embodiment of this application, where GMM1 and GMM2 are replaced with BMM1, GMM3 and BMM2.

[0096] BMM1 includes N first matrices X i The product of the second matrix A and the fifth matrix A, where the fifth matrix A consists of N second matrices A. i For example, N second matrices A i The merged structure is filled with a rank of M. Figure 8 BMM1 includes three product operations between the first matrix X1, X2, and X3 and the fifth matrix A. GMM3 includes N sixth matrices E. i and N seventh matrices D i The product of the i-th sixth matrix Ei Including the i-th first matrix X i The product of the fifth matrix A and E i =X i A, Figure 8 GMM3 includes product operations X1AD1, X2AD2, and X3AD3. BMM2 includes N eighth matrices G. i The product of the ninth matrix B and the i-th eighth matrix G i For the i-th sixth matrix E i and the i-th seventh matrix D i The product result, i.e., G i =X i AD i , Figure 8 BMM2 includes product operations X1AD1B, X2AD2B, and X3AD3B.

[0097] Among them, N seventh matrices D i It can be stored and used as N M×M square matrices, or it can be combined into an N×M×M three-dimensional matrix for storage and use.

[0098] Figure 9 This is a schematic diagram of the storage scheme provided in an embodiment of this application. In the prior art, due to the efficiency requirements of NPU and GPU, X... i A i and B i The rank needs to be padded to a hardware-friendly size M. This reduces some of the advantages of the low-rank characteristics of LoRA modules and introduces more zeros. For example, A i The dimension is [L A ,r i ], B i The dimension is [r i ,L B After filling, matrix A i B i The sizes are [L] A [M,L] and [M,L] B To avoid storing unnecessary padding data, this embodiment of the application uses the second matrix A in the GMM node. i Merge the matrices, then fill the rank of the merged matrix into M to obtain the fifth matrix A; then merge the fourth matrix B from the GMM nodes. i Merge the matrices, then fill the rank of the merged matrix into M to obtain the ninth matrix B; additionally store N square matrices D. i Among them, there are N square matrices D. i It can be stored and used as N M×M square matrices, or it can be combined into an N×M×M three-dimensional matrix for storage and use.

[0099] The memory usage of existing technologies is shown in formula (1), where F represents the memory occupied by the LoRA module in the model, D represents the size of the scalar data type, N represents the number of LoRA modules included in the model, M represents the matrix rank corresponding to the optimal performance of the processor, and the model runs on this processor. A L represents the other dimension of matrix A besides M. B This represents the other dimension of matrix B besides M.

[0100] F = DNM(L) A +L B (1)

[0101] like Figure 9 As shown, the storage scheme used in the data processing method provided in this application is more compact than that of the prior art. In this embodiment, the second matrix A in the GMM node... i Merge the matrices, then fill the rank of the merged matrix into M to obtain the fifth matrix A; then merge the fourth matrix B from the GMM nodes. i Merge the matrices, then fill the rank of the merged matrix into M to obtain the ninth matrix B; additionally store N square matrices D. i The memory usage of the data processing method provided in this application embodiment is shown in formula (2), where F represents the memory occupied by the LoRA module in the model, D represents the size of the scalar data type, N represents the number of LoRA modules included in the model, M represents the matrix rank corresponding to the optimal performance of the processor, and the model runs on this processor. A L represents the other dimension of matrix A besides M. B This represents the other dimension of matrix B besides M.

[0102] F = DM(L) A +L B )+DNM 2 (2)

[0103] Because of L A L B Typical values ​​are in the thousands, while M is usually much smaller than L. A and L B For example, the number of matrices is on the order of 64, thus saving a considerable amount of memory. Furthermore, in the data processing method provided in this application embodiment, all matrices have optimal dimensions.

[0104] Formula (3) represents the memory usage increment of the data processing method provided in this application embodiment, where P is the memory usage increment of the model. The memory usage increment includes the increase in memory usage when the model uses t+1 LoRA modules compared to using t LoRA modules, assuming other parameters remain unchanged, where t≥1. D is the size of the scalar data type, and M represents the matrix rank corresponding to the optimal performance of the processor, on which the model runs.

[0105] P=DM 2 (3)

[0106] Because of L A and L B Much larger than M, L in general scenarios A +L B >M, therefore the matrix processed by the first computation graph is much smaller.

[0107] This application's embodiments replace two GMM nodes with two smaller BMM nodes and one GMM node, allowing most computations to be performed by the BMM nodes, with only a small portion of processing delegated to the GMM nodes. In terms of performance, processing N LoRA modules in this application is almost equivalent to the processing speed of a single LoRA module.

[0108] 440, Reorganize the storage method of the LoRA module.

[0109] Reorganize the storage method of LoRA modules, and put A in N LoRA modules. i and B i Instead of using extra padding data to fill each LoRA module separately, the two LoRA modules are merged into single matrices A and B, and N smaller square matrices are added to accommodate the two new BMM nodes, thus eliminating the redundancy caused by independent storage.

[0110] 450, load the weights into the model.

[0111] After adjusting the computation graph and storage scheme, the weights and LoRA modules are loaded from the CPU, the weights of the LoRA modules are reorganized, and the optimized weights are loaded into the model to ensure the correctness and consistency of the calculation results.

[0112] 460, running.

[0113] The first computation graph is deployed to an NPU or GPU, runs based on the input data, and output data is obtained.

[0114] In existing technologies, due to the efficiency requirements of NPUs and GPUs, during computation, the rank of the two multiplied matrices in each LoRA module of the model must first be padded to a hardware-friendly size M. For example, 0 is used as padding data to padded the rank of the two multiplied matrices to M, where 1≤i≤N. To avoid storing unnecessary padding data, this application provides a data processing method that uses BMM nodes instead of GMM nodes. Instead of using additional padding data to padded each matrix and then storing each LoRA module separately, it stores a matrix whose rank is padded to M after merging the matrices of N LoRA modules, plus an additional M×M square matrix. Since the overall dimension of the square matrix is ​​much smaller than the overall dimension of the two multiplied matrices in the LoRA module after padding, the data processing method provided in this application can reduce the model's memory usage and reduce memory waste during model training or inference.

[0115] It should be understood that the acceleration of GMM nodes in the computation graph in the above embodiments is only an example. The data processing method provided in this application can also accelerate similar GMM nodes, such as SGMV nodes, and this application does not limit it.

[0116] The following experimental data will illustrate the beneficial effects of the technical solution in this application.

[0117] Table 2 shows the test data of the data processing method provided in the embodiments of this application on the large language model LLAMA3.1 70B. After applying the data processing method provided in the embodiments of this application, the overall performance of the model can be improved by 4%.

[0118] Table 2

[0119]

[0120]

[0121] The data processing method according to the embodiments of this application has been described above. The following will be combined with... Figure 10 and Figure 11 This application describes apparatus and devices according to embodiments thereof.

[0122] This application also provides a computer storage medium storing program instructions, which, when executed, may include, for example... Figure 3 and Figure 4 Some or all of the steps of the data processing method in the corresponding embodiments.

[0123] Figure 10This is a structural example diagram of a computer device 1000 provided in an embodiment of this application. The computer device 1000 includes an acquisition module 1010 and a processing module 1020. The acquisition module 1010 and the processing module 1020 can be implemented in software, hardware, or a combination of both.

[0124] The acquisition module 1010 is used to acquire the input data of the model in order to perform... Figure 3 310 in the method, Figure 4 Method 410.

[0125] Processing module 1020 is used to construct a first computational graph based on the input data and the model's script, run the model on the processor, obtain output data, and execute... Figure 3 and Figure 4 Some or all of the steps in the method.

[0126] Figure 11 This is a structural example diagram of another computer device 1100 provided in an embodiment of this application. The computer device 1100 includes a processor 1102, a communication interface 1103, and a memory 1104. One example of the computer device 1100 is a computing device, such as a server for running models.

[0127] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1102. Processor 1102 can be a central processing unit (CPU), GPU, or NPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of processor 1102 or by instructions in software form. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.

[0128] The memory 1104 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0129] The processor 1102, memory 1104, and communication interface 1103 can communicate via a bus. The memory 1104 stores executable code, and the processor 1102 reads the executable code from the memory 1104 to execute the corresponding method. The memory 1104 may also include other software modules required for running processes, such as an operating system. The operating system can be Linux. TM UNIX TM WINDOWS TM wait.

[0130] For example, the executable code in memory 1104 is used to implement Figure 3 and Figure 4 The method shown involves processor 1102 reading the executable code from memory 1104 to execute it. Figure 3 and Figure 4 The method shown.

[0131] In some embodiments of this application, the disclosed methods can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art. Figure 12 A conceptual partial view schematically illustrates an example computer program product arranged according to at least some embodiments shown herein, the example computer program product including a computer program for executing computer processes on a computing device. In one embodiment, the example computer program product 1200 is provided using a signal carrying medium 1201. The signal carrying medium 1201 may include one or more program instructions 1202 that, when executed by one or more processors, can provide the above-described... Figure 3 and Figure 4 The functions or parts thereof described in the methods shown. Therefore, for example, refer to... Figure 3 and Figure 4 In the embodiments shown, one or more features may be provided by one or more instructions associated with the signal carrying medium 1201.

[0132] In some examples, signal-bearing medium 1201 may comprise computer-readable medium 1203, such as, but not limited to, hard disk drives, CDs, digital video optical discs (DVDs), digital magnetic tapes, memory, read-only memory (ROM), or random access memory (RAM), etc. In some embodiments, signal-bearing medium 1201 may comprise computer-recordable medium 1204, such as, but not limited to, memory, read / write (R / W) CDs, R / W DVDs, etc. In some embodiments, signal-bearing medium 1201 may comprise communication medium 1205, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, signal-bearing medium 1201 may be conveyed by wireless communication medium 1205 (e.g., wireless communication media conforming to the IEEE 802.11 standard or other transmission protocols). One or more program instructions 1202 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the aforementioned computing device can be configured to provide various operations, functions, or actions in response to program instructions 1202 transmitted to the computing device via one or more of computer-readable media 1203, computer-recordable media 1204, and / or communication media 1205. It should be understood that the arrangements described herein are merely illustrative. Therefore, those skilled in the art will understand that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements can be omitted depending on the desired result. Furthermore, many of the described elements are functional entities that can be implemented as discrete or distributed components, or in any suitable combination and location in conjunction with other components.

[0133] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0138] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, include: Obtain the input data for the model, which includes a low-rank adaptive LoRA module; A first computation graph is constructed based on the input data and the script of the model. The first computation graph includes multiple batch matrix multiplication (BMM) nodes. The model is run on the processor based on the input data and the first computation graph to obtain output data. The matrix rank corresponding to the optimal performance of the processor is M. The memory usage increment of the model is proportional to the square of M. The memory usage increment includes the amount of memory used by the model when using t+1 LoRA modules compared to when using t LoRA modules, where M>1 and t≥1.

2. The method according to claim 1, characterized in that, The memory usage increment of the model is proportional to the square of M, including: P = DM 2 P represents the memory usage increment of the model, and D represents the size of the scalar data type.

3. The method according to claim 1 or 2, characterized in that, The step of constructing the first computational graph based on the input data and the model's script includes: A second computation graph is constructed based on the input data and the script of the model. The second computation graph includes a first grouped matrix multiplication node and a second grouped matrix multiplication node. The first grouped matrix multiplication node includes the product of N first matrices and N second matrices. The second grouped matrix multiplication node includes the product of N third matrices and N fourth matrices. The i-th third matrix is ​​the product of the i-th first matrix and the i-th second matrix. N is the number of LoRA modules included in the model, 1≤i≤N. The first grouped matrix multiplication node and the second grouped matrix multiplication node in the second computation graph are replaced with the first batched matrix multiplication node, the third grouped matrix multiplication node, and the second batched matrix multiplication node to obtain the first computation graph. The first batched matrix multiplication node includes the product of the N first matrices and the fifth matrix. The third grouped matrix multiplication node includes the product of the N sixth matrices and the N seventh matrices. The second batched matrix multiplication node includes the product of the N eighth matrices and the ninth matrix. The fifth matrix includes the N second matrices. The i-th sixth matrix includes the product of the i-th first matrix and the fifth matrix. The seventh matrix is ​​an M x M square matrix. The i-th eighth matrix is ​​the product of the i-th sixth matrix and the i-th seventh matrix. The ninth matrix includes the N fourth matrices. The first batched matrix multiplication node and the second batched matrix multiplication node belong to the BMM node.

4. The method according to claim 3, characterized in that, The square matrix includes a diagonal matrix.

5. The method according to claim 3 or 4, characterized in that, The N seventh matrices are three-dimensional matrices with N layers, M rows, and M columns.

6. The method according to any one of claims 1 to 5, characterized in that, M equals a power of 2.

7. The method according to any one of claims 1 to 6, characterized in that, The processor includes a neural network processor (NPU) or a graphics processing unit (GPU).

8. The method according to any one of claims 1 to 7, characterized in that, The model includes any one or more of the following: text generation tools, content creation tools, text classification systems, machine translation systems, or reasoning systems.

9. A computer device, characterized in that, include: An acquisition module is used to acquire the input data of the model, wherein the model includes a low-rank adaptive LoRA module; The processing module is used to construct a first computation graph based on the input data and the script of the model, the first computation graph including multiple batch matrix multiplication (BMM) nodes; The processing module is further configured to run the model on the processor according to the input data and the first computation graph to obtain output data. The matrix rank corresponding to the optimal performance of the processor is M. The memory usage increment of the model is proportional to the square of M. The memory usage increment includes the amount of memory used by the model using t+1 LoRA modules compared to using t LoRA modules, where M>1 and t≥1.

10. The apparatus according to claim 9, characterized in that, The memory usage increment of the model is proportional to the square of M, including: P = DM 2 P represents the memory usage increment of the model, and D represents the size of the scalar data type.

11. The apparatus according to claim 9 or 10, characterized in that, The processing module is specifically used for: A second computation graph is constructed based on the input data and the script of the model. The second computation graph includes a first grouped matrix multiplication node and a second grouped matrix multiplication node. The first grouped matrix multiplication node includes the product of N first matrices and N second matrices. The second grouped matrix multiplication node includes the product of N third matrices and N fourth matrices. The i-th third matrix is ​​the product of the i-th first matrix and the i-th second matrix. N is the number of LoRA modules included in the model, 1≤i≤N. The first grouped matrix multiplication node and the second grouped matrix multiplication node in the second computation graph are replaced with the first batched matrix multiplication node, the third grouped matrix multiplication node, and the second batched matrix multiplication node to obtain the first computation graph. The first batched matrix multiplication node includes the product of the N first matrices and the fifth matrix. The third grouped matrix multiplication node includes the product of the N sixth matrices and the N seventh matrices. The second batched matrix multiplication node includes the product of the N eighth matrices and the ninth matrix. The fifth matrix includes the N second matrices. The i-th sixth matrix includes the product of the i-th first matrix and the fifth matrix. The seventh matrix is ​​an M x M square matrix. The i-th eighth matrix is ​​the product of the i-th sixth matrix and the i-th seventh matrix. The ninth matrix includes the N fourth matrices. The first batched matrix multiplication node and the second batched matrix multiplication node belong to the BMM node.

12. The apparatus according to claim 11, characterized in that, The square matrix includes a diagonal matrix.

13. The apparatus according to claim 11 or 12, characterized in that, The N seventh matrices are three-dimensional matrices with N layers, M rows, and M columns.

14. The apparatus according to any one of claims 9 to 13, characterized in that, M equals a power of 2.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The processor includes a neural network processor (NPU) or a graphics processing unit (GPU).

16. The apparatus according to any one of claims 9 to 15, characterized in that, The model includes any one or more of the following: text generation tools, content creation tools, text classification systems, machine translation systems, or reasoning systems.

17. A computer device, characterized in that, include: A processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform the method as described in any one of claims 1 to 8.

18. A computer-readable medium, characterized in that, The computer-readable medium stores computer program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

Images