Data processing method for neural network model, server, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, particularly provides a data processing method for a neural network model, a server, equipment and a medium, and aims to solve the problem of how to carry out attention calculation on the neural network model based on a CPU of mobile equipment. The method provided by the invention comprises the steps of obtaining a first calculation graph according to a model file of a neural network model, wherein the first calculation graph is used for describing all operators of the neural network model and a connection relationship among the operators; obtaining a plurality of target operators in the first calculation graph, wherein the plurality of target operators are a plurality of operators used for carrying out attention calculation by a Transform module; the calculation operations corresponding to the multiple target operators are fused to form a fusion operator, and the fusion operator is used for a CPU of the mobile device to execute the calculation operations; and replacing the plurality of target operators with the fusion operator to obtain a second calculation graph. Through the method, the calculation operation corresponding to the fusion operator can be executed based on the CPU, the attention calculation speed is increased, and therefore the model reasoning speed is increased.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, server, device and medium for a neural network model. Background Art

[0002] With the rapid development of neural network models, neural network models have gradually been deployed on mobile devices such as mobile phones and car computers. Using neural network models on mobile devices can realize functions such as text generation and intelligent reply.

[0003] For neural network models that use the Transformer network architecture, the neural network model usually has multiple Transformer modules. Among the operations involved in the Transformer module, the attention calculation is the most core and has the largest amount of calculation. Therefore, the attention calculation is usually optimized to reduce the amount of attention calculation, thereby improving the reasoning speed of the neural network model. However, currently, attention calculation is mainly implemented by GPU (Graphics Processing Unit). Therefore, the optimization method for attention calculation is also based on GPU implementation. There are few methods that use CPU (Central Processing Unit) to implement attention calculation, and there is no method based on CPU to optimize attention calculation to improve the reasoning speed of the neural network model. For example, the Flash Attention method provided in the paper "Fast and Memory-Efficient Exact Attention with IO-Awareness" can improve the speed of attention calculation and reduce memory access overhead, but this method is implemented based on GPU, not CPU.

[0004] Accordingly, the art needs a new technical solution to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the following technical problem: for a neural network model deployed on a mobile device, how to perform attention calculation based on the CPU of the mobile device, thereby improving the reasoning speed of the neural network model.

[0006] In a first aspect, a data processing method for a neural network model is provided, which is applied to a server, wherein the neural network model is a neural network model for deployment on a mobile device, and the neural network model includes a Transformer module, and the method includes: According to the model file of the neural network model, a first calculation graph of the neural network model is obtained, where the first calculation graph is used to describe all operators of the neural network model and connection relationships between the operators; Acquire multiple target operators in the first computation graph, where the multiple target operators are multiple operators used by the Transformer module to perform attention calculation; Fusing the computing operations corresponding to the multiple target operators to form a fused operator, wherein the fused operator is used for a CPU of the mobile device to perform the computing operation; The multiple target operators in the first computation graph are replaced with the fusion operator to obtain a second computation graph of the neural network model, and the second computation graph is used by the mobile device to perform model inference on the input data of the neural network model to obtain an inference result.

[0007] In a technical solution of the above data processing method, the connection relationship between the operators is used to describe the input and output relationship between the operators, and the obtaining of the multiple target operators in the first calculation graph includes: Determining a target arrangement order of the multiple target operators according to a connection relationship between the multiple target operators; The operators in the first computation graph are traversed in sequence. If all the multiple target operators are traversed and the traversal order of the multiple target operators is the same as the target arrangement order, the multiple target operators are obtained.

[0008] In a technical solution of the above data processing method, the multiple target operators include a matrix transposition operator, a first matrix multiplication operator, a division operator, an addition operator, a normalization operator and a second matrix multiplication operator connected in sequence; The first calculation operation corresponding to the matrix transposition operator is: transposing the Key matrix in the attention mechanism to obtain a transposed matrix; The second calculation operation corresponding to the first matrix multiplication operator is: multiplying the Query matrix in the attention mechanism by the transposed matrix to obtain a first attention score; The third calculation operation corresponding to the division operator is: Divide to obtain a second attention score, where d is the dimension of the Key matrix or the Query matrix; The fourth calculation operation corresponding to the addition operator is: adding the Attention mask matrix of the attention mechanism to the second attention score to obtain a third attention score; The fifth calculation operation corresponding to the normalization operator is: normalizing the third attention score to obtain an attention weight; The sixth calculation operation corresponding to the second matrix multiplication operator is: multiplying the Value matrix in the attention mechanism by the attention weight to obtain the attention calculation result.

[0009] In a technical solution of the above data processing method, the step of fusing the computing operations corresponding to the multiple target operators to form a fused operator includes: The second calculation operation is replaced by a seventh calculation operation, wherein the seventh calculation operation is to multiply the Query matrix by the Key matrix, and the row elements and column elements of the Key matrix are used as column elements and row elements respectively when performing the multiplication; fusing the seventh computing operation, the third computing operation, the fourth computing operation, the fifth computing operation, and the sixth computing operation to form the fused operator; The input tensor of the fusion operator represents the Key matrix, the Query matrix, the Attention mask matrix and the Value matrix; The output tensor of the fusion operator represents the attention calculation result.

[0010] In a technical solution of the above data processing method, replacing the multiple target operators in the first calculation graph with the fusion operator to obtain a second calculation graph of the neural network model includes: The second matrix multiplication operator in the first computation graph is replaced with the fusion operator, and the matrix transposition operator, the first matrix multiplication operator, the division operator, the addition operator and the normalization operator are deleted from the first computation graph to obtain the second computation graph.

[0011] In a second aspect, a data processing method for a neural network model is provided, which is applied to a mobile device, wherein the neural network model is a neural network model deployed on the mobile device, and the neural network model includes a Transformer module, and the method includes: According to the model file of the neural network model, obtaining a target calculation graph of the neural network model, wherein the target calculation graph is a second calculation graph obtained by using the data processing method for the neural network model provided by the first aspect; In response to the neural network model receiving input data, each operator in the target calculation graph is traversed in turn and the calculation operation of each operator is executed to obtain the inference result of the neural network model performing model inference on the input data.

[0012] In a technical solution of the above data processing method, the input tensor of the fused operator in the target computation graph is stored in the stack memory of the mobile device, and the sequentially traversing each operator in the target computation graph and performing the computation operation of each operator includes: The CPU of the mobile device obtains the channel data of the input tensor from the stack memory, and performs the calculation operation of the fusion operator according to the channel data.

[0013] In a technical solution of the above data processing method, the CPU of the mobile device obtains the channel data of the input tensor from the stack memory, including: Before traversing to the fusion operator, the CPU of the mobile device obtains each channel data of the input tensor from the stack memory, and stores the each channel data in the cache of the mobile device; In response to traversing to the fusion operator, the CPU of the mobile device obtains the channel data from the cache.

[0014] In a technical solution of the above data processing method, the cache is an L1 cache.

[0015] In a technical solution of the above data processing method, the input tensor is stored in the stack memory in the following manner: If the data type of the input tensor is FP32 and the number of channels is 4, the input tensor is stored in the stack memory according to the memory arrangement of NC4HW4; If the data type of the input tensor is FP16 and the number of channels is 8, the input tensor is stored in the stack memory according to the memory arrangement of NC8HW8; Wherein, N, C, H, and W represent the batch number, channel number, height, and width of the input tensor respectively; The memory arrangement of NC4HW4 is as follows: the data of the jth row and the kth column of channel i in the input tensor is used as the data of the jth row and the i+C×(k-1)th column of the input tensor when it is stored in the stack memory, i=1,…,C, j=1,…,H, k=1,…,W, and the shape of the input tensor when it is stored in the stack memory is [N, C / 4,H, W×4]; The memory arrangement of NC8HW8 is as follows: the data of the jth row and kth column of channel i in the input tensor is used as the data of the jth row and i+C×(k-1)th column of the input tensor when it is stored in the stack memory, i=1,…,C, j=1,…,H, k=1,…,W, and the shape of the input tensor when it is stored in the stack memory is [N, C / 8,H, W×8].

[0016] In a technical solution of the above data processing method, performing the calculation operation corresponding to the fusion operator according to the channel data includes: The CPU of the mobile device executes the calculation operations corresponding to the fusion operator in parallel according to the channel data.

[0017] In a technical solution of the above data processing method, the step of performing the calculation operation of the fusion operator according to the channel data includes: Obtaining the number n of valid threads, where the number n of valid threads is the maximum number of threads that can be used in a target process, and the target process is a process used to execute a computing operation of the fusion operator; Performing matrix segmentation on the Query matrix in each channel data according to the number n to form n Query sub-matrices; Each of the effective threads executes the calculation operation corresponding to the fusion operator according to the corresponding data to be processed; The data to be processed includes a Query sub-matrix and a Key matrix, a Value matrix and an Attention mask matrix in each channel data, and the Query sub-matrices in the data to be processed corresponding to different valid threads are different.

[0018] In a technical solution of the above data processing method, each of the valid threads executes the calculation operation corresponding to the fusion operator according to the corresponding data to be processed, including: The Query sub-matrix, the Key matrix, and the Value matrix in the data to be processed are respectively divided into a plurality of matrix blocks, wherein the number of rows and the number of columns of the matrix blocks are both 1; The valid thread executes the calculation operation corresponding to the fusion operator according to the matrix block of the Query sub-matrix, the matrix block of the Key matrix, and the matrix block of the Value matrix.

[0019] In a third aspect, a server is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions provided in the first aspect is implemented.

[0020] In a fourth aspect, a mobile device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions provided in the second aspect is implemented.

[0021] In a fifth aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions provided in the first aspect or the second aspect.

[0022] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects: In a technical solution for implementing the data processing method for a neural network model provided in the present application, the method is applied to a server, the neural network model is a neural network model for deployment on a mobile device, the neural network model includes a Transformer module, and the method includes the following steps: according to the model file of the neural network model, a first calculation graph of the neural network model is obtained, the first calculation graph is used to describe all operators of the neural network model and the connection relationship between each of the operators; multiple target operators in the first calculation graph are obtained, the multiple target operators are multiple operators used for the Transformer module to perform attention calculations; the calculation operations corresponding to the multiple target operators are fused to form a fused operator, and the fused operator is used for the CPU to perform calculation operations; the multiple target operators in the first calculation graph are replaced with the fused operator to obtain a second calculation graph of the neural network model, and the second calculation graph is used for the mobile device to perform model inference on the input data of the neural network model to obtain inference results.

[0023] After the neural network model is deployed on the mobile device, the mobile device can perform model reasoning according to the second calculation graph. For each operator, when executing the calculation operation corresponding to the operator, it is necessary to access the memory to obtain the input data of the operator, perform the calculation operation on the input data, and then access the memory to store the operation result in the memory. The above implementation scheme fuses multiple operators used for attention calculation operations into one operator, which can greatly reduce the number of memory accesses, thereby increasing the speed of attention calculation. In addition, the fused operator can be used by the CPU to perform calculation operations, which not only overcomes the defect that the prior art can only implement attention calculation based on GPU, but also realizes the acceleration of attention calculation, thereby increasing the reasoning speed of the neural network model.

[0024] In a technical solution for implementing the data processing method for a neural network model provided in the present application, the method is applied to a mobile device, the neural network model is a neural network model deployed on the mobile device, the neural network model includes a Transformer module, and the method includes the following steps: according to the model file of the neural network model, obtaining a target calculation graph of the neural network model, the target calculation graph is a second calculation graph obtained by using the data processing method for a neural network model provided in the above implementation scheme; in response to the neural network model receiving input data, traversing each operator in the target calculation graph in turn and executing the calculation operation of each operator to obtain an inference result of the neural network model performing model inference on the input data.

[0025] Since the target computation graph is obtained by using the data processing method provided by the aforementioned implementation scheme, when the mobile device performs model inference according to the target computation graph, it can execute the computational operations of the fusion operators in the target computation graph based on the CPU, thereby improving the attention calculation speed and thus improving the inference speed of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them: Figure 1 It is a flowchart of main steps of a data processing method applied to a server according to an embodiment of the present application; Figure 2 is a schematic diagram of a target operator according to an embodiment of the present application; Figure 3 is a schematic diagram of a fusion operator according to an embodiment of the present application; Figure 4 It is a schematic diagram of the main steps of obtaining multiple target operators in the first calculation graph according to an embodiment of the present application; Figure 5 It is a schematic flow chart of the main steps of fusing the computing operations of multiple target operators according to an embodiment of the present application; Figure 6 It is a schematic diagram of the overall process of acquiring and fusing target operators, and replacing fusion operators according to an embodiment of the present application; Figure 7 is a flowchart of main steps of a data processing method applied to a mobile device according to an embodiment of the present application; Figure 8 is a schematic diagram of an existing arrangement of input tensors in memory according to an embodiment of the present application; Fig. 9is a schematic diagram of the memory arrangement of NC4HW4 according to an embodiment of the present application; Fig.10 is a schematic diagram of the memory arrangement of NC8HW8 according to an embodiment of the present application; Fig.11 is a schematic diagram of a flow chart of executing a computing operation corresponding to a fusion operator according to an embodiment of the present application; Fig.12 It is a schematic flow chart of the main steps of performing the calculation operation of the fusion operator according to each channel data according to an embodiment of the present application; Fig.13 It is a schematic diagram of the overall data processing process including operator fusion and model reasoning process according to an embodiment of the present application; Fig.14 is a schematic diagram of the main structure of a server according to an embodiment of the present application; Fig.15 It is a schematic diagram of the main structure of a mobile device according to an embodiment of the present application.

[0027] Reference numerals: 11: memory; 12: processor; 21: memory; 22: processor. DETAILED DESCRIPTION

[0028] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0029] In the description of this application, "processor" may include hardware, software or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc.

[0030] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0031] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0032] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0033] Here we first explain some terms involved in this application.

[0034] (1) Neural network model.

[0035] The neural network model may be a neural network model with a large number of parameters and a complex structure, and the neural network model may be deployed on a mobile device. In some embodiments, the neural network model may be a model built based on a Transformer model.

[0036] (2) Computational graph.

[0037] The computational logic of the neural network model can be determined based on the parameters and structure of the neural network model. The computational logic can realize the reasoning of the neural network model, and the computational logic is represented by a computational graph. Specifically, the computational graph describes all operators of the neural network model and the connection relationship between the operators. The connection relationship between the operators is used to describe the input and output relationship between the operators. For example, in the computational graph, the output of operator A is connected to the input of operator B, and the connection relationship between the two operators is used to describe the connection between the output of operator A and the input of operator B.

[0038] In addition, each operator corresponds to a calculation operation. By traversing each operator in the calculation graph and executing the calculation operation of each operator in turn, the above calculation logic can be realized, that is, the reasoning of the neural network model can be realized. For example, operator 3 is a matrix multiplication operator, and the input of operator 3 is connected to the output of operator 1 and operator 2 respectively, that is, operator 3 has two input data, and the calculation operation corresponding to operator 3 is to perform matrix multiplication calculation on the two input data.

[0039] (3) Transformer module.

[0040] The Transformer module is the Transformer Block in the Transformer model (converter model).

[0041] (4) Attention calculation.

[0042] The Transformer model uses the Attention Mechanism. When processing input data, the Transformer module will perform attention calculation based on the input data and complete the corresponding processing based on the calculation results. The following explains the principle of attention calculation.

[0043] Specifically, the attention calculation includes the following steps 11 to 15.

[0044] Step 11: Calculate the first attention score by the following formula (1): (1) Among them, Attention Scores is the first attention score, K is the Key matrix in the attention mechanism, Key represents the key; Q represents the Query matrix in the attention mechanism, Query represents the query; T represents the transpose of the matrix.

[0045] Step 12: Scale the first attention score Attention Scores using the following formula (2) to obtain the second attention score: (2) Among them, Scaled Score represents the second attention score, and d represents the dimension of the Key matrix or the Query matrix.

[0046] Step 13: Add the second attention score to the Attention mask matrix in the attention mechanism to obtain the third attention score.

[0047] Step 14: Normalize the third attention score to obtain the attention weight. For example, the attention weight can be obtained by the following formula (3): (3) Among them, Attention Weights represents the attention weight is the third attention score, and softmax is the softmax function.

[0048] Step 15: Obtain the attention calculation result through the following formula (4): (4) Among them, Output represents the attention calculation result, V represents the Value matrix in the attention mechanism, and Value represents the value.

[0049] The Key matrix, Query matrix, Attention mask matrix and Value matrix in the above steps are all conventional matrices in the attention mechanism. The embodiments of this application do not elaborate on the acquisition methods and functions of these matrices.

[0050] (5) Tensor.

[0051] The data in the neural network model is represented by a tensor. In some embodiments of the present application, the tensor is four-dimensional, and the shape of the tensor is Shape = [N, C, H, W], where N, C, H, and W represent the batch size, channel number, height, and width of the tensor, respectively.

[0052] The batch number can be understood as the number of input data of the neural network model, the number of channels can be understood as the number of feature dimensions of the input data, the height can be understood as the height of the input data, and the width can be understood as the width of the input data. In a tensor, all input data have the same number of channels, the same height, and the same width, so that all input data can be represented by one tensor.

[0053] Taking the input data as an example, one or more images can be input to the neural network model for model inference. Therefore, the input data of the neural network model can be understood as a batch of images, in which the number of images is greater than or equal to 1. The number of images can be understood as the batch number of this batch of images, which has the same meaning as the batch number of tensors. The number of channels can be understood as the number of feature dimensions of the image. For example, if an image has three feature dimensions of R (red), G (green), and B (blue), then the number of channels is 3. If the batch number of images is 2, and the height and width are 128 respectively, then the shape of the tensor used to represent the input data is Shape=[2,3,128,128].

[0054] The following is a description of an embodiment of a data processing method applied to a server provided in the present application. The method is a data processing method for a neural network model. The neural network model is a neural network model for deployment on a mobile device. The neural network model includes a Transformer module.

[0055] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of a data processing method for a neural network model according to an embodiment of the present application, and the method is applied to a server. Figure 1 As shown, the data processing method for the neural network model in the embodiment of the present application mainly includes the following steps S101 to S104.

[0056] Step S101: According to the model file of the neural network model, obtain the first calculation graph of the neural network model, the first calculation graph is used to describe all operators of the neural network model and the connection relationship between the operators. These operators are unfused operators and can be understood as the original operators of the neural network model.

[0057] Step S102: Acquire multiple target operators in the first computation graph, where the multiple target operators are multiple operators used by the Transformer module to perform attention calculations.

[0058] See attached Figure 2 The above-mentioned multiple target operators may include a matrix transposition operator, a first matrix multiplication operator, a division operator, an addition operator, a normalization operator and a second matrix multiplication operator connected in sequence. The calculation operations corresponding to these operators are described below.

[0059] The first calculation operation corresponding to the matrix transposition operator is: transpose the key matrix in the attention mechanism to obtain the transposed matrix, that is, the transposed matrix is ​​the same as in the above formula (1) .

[0060] The second calculation operation corresponding to the first matrix multiplication operator is: multiplying the query matrix in the attention mechanism by the transposed matrix to obtain the first attention score. The first attention score can be understood as the Attention Scores calculated by the above formula (1).

[0061] The third calculation operation corresponding to the division operator is: Divide to get the second attention score, d is the dimension of the Key matrix or the Query matrix. The second attention score can be understood as the result of the calculation of the above formula (2): .

[0062] The fourth calculation operation corresponding to the addition operator is: adding the attention mask matrix of the attention mechanism to the second attention score to obtain a third attention score. The third attention score can be understood as the third attention score in step 14 in the above terminology explanation section. .

[0063] The fifth calculation operation corresponding to the normalization operator is: normalizing the third attention score to obtain the attention weight. In some embodiments, a Softmax function may be used for normalization, and the attention weight may be understood as the Attention Weights obtained by the aforementioned formula (3).

[0064] The sixth calculation operation corresponding to the second matrix multiplication operator is: multiplying the value matrix in the attention mechanism by the attention weight to obtain the attention calculation result. The attention calculation result can be understood as the Output obtained by the aforementioned formula (4).

[0065] Step S103: Fusion the computing operations corresponding to the multiple target operators to form a fused operator, which is used by the CPU to perform computing operations.

[0066] After the neural network model is deployed on the mobile device, the CPU of the mobile device can perform the computing operations corresponding to the fusion operator. The type of the fusion operator can be set to attention, that is, the fusion operator can also be described as an attention operator.

[0067] See attached Figure 3 , the input of the fusion operator includes the key matrix, the query matrix, the attention mask matrix and the value matrix. The calculation operation corresponding to the fusion operator may include the first, second, third, fourth, fifth and sixth operations mentioned above. The output of the fusion operator is the above Output.

[0068] Step S104: Replace multiple target operators in the first computation graph with fusion operators to obtain a second computation graph of the neural network model.

[0069] The second calculation graph can be used as the final calculation graph of the neural network model. After the neural network model is deployed on the mobile device, the calculation graph obtained according to the model file of the neural network model is the second calculation graph, and the mobile device can perform model reasoning according to the second calculation graph. That is, the second calculation graph is used by the mobile device to perform model reasoning on the input data of the neural network model to obtain the reasoning result. The reasoning result can be used as the output result of the system application (application, APP) or third-party application of the mobile device, or as the input data of the system application or third-party application, and further processed and the output result is obtained. In some examples, the above reasoning results may include: voice information (such as the user's voice instructions or content), image information (such as redundant image content such as people or objects, or image content that is desired to be expanded), life information (such as a set of inferred restaurants or hotels), navigation information (such as a set of possible destinations), intelligent driving control information (such as control instructions for controlling the longitudinal movement and / or lateral movement of the vehicle), etc. It should be understood that the above reasoning results can also be the results obtained by applying the neural network model in other scenarios, and this application is not limited to this.

[0070] For each operator, when executing the calculation operation of the operator, it is necessary to access the memory to obtain the input data of the operator, perform the calculation operation on the input data, and then access the memory to store the operation result in the memory. The above method embodiment fuses multiple operators used for attention calculation operations into one operator, which can greatly reduce the number of memory accesses, thereby improving the speed of attention calculation. In addition, the fused operator can be used by the CPU to perform calculation operations, which not only overcomes the defect that the prior art can only implement attention calculation based on GPU, but also realizes the acceleration of attention calculation, thereby improving the reasoning speed of the neural network model.

[0071] The above steps S102 to S104 are described below.

[0072] 1. Explain step S102.

[0073] In some implementations of the above step S102, Figure 4 The following steps S1021 to S1022 shown in the figure obtain multiple target operators in the first computation graph.

[0074] Step S1021: Determine the target arrangement order of the multiple target operators according to the connection relationship between the multiple target operators, wherein the connection relationship between the operators is used to describe the input and output relationship between the operators.

[0075] like Figure 2 As shown, the target arrangement order is: matrix transposition operator, first matrix multiplication operator, division operator, addition operator, normalization operator and second matrix multiplication operator.

[0076] Step S1022: traverse the operators in the first computation graph in sequence. If all the multiple target operators are traversed and the traversal order of the multiple target operators is the same as the target arrangement order, then obtain the multiple target operators.

[0077] Specifically, the first computational graph describes the connection relationship between operators, and when traversing operators, the traversal is performed according to the connection relationship, that is, according to the input and output relationship between operators. Figure 2 As shown, if the operator currently traversed is the matrix transposition operator, since the output of the matrix transposition operator is connected to the first matrix multiplication operator, the next traversal is the first matrix multiplication operator; and the output of the first matrix multiplication operator is connected to the division operator, then after traversing the first matrix multiplication operator, the division operator is traversed.

[0078] The first calculation graph may not mark which operators are combined for attention calculation. In this case, if each target operator among multiple target operators is traversed in sequence, it means that several target operators required for attention calculation are traversed continuously, but the traversal order of these target operators may be the same as the above target arrangement order, or it may be different. If they are the same, it means that these target operators are combined for attention calculation, so these traversed target operators can be obtained; otherwise, these target operators do not need to be obtained.

[0079] Based on the method described in steps S1021 to S1022 above, when the first calculation graph does not mark which operators are combined for attention calculation, multiple target operators for attention calculation can be accurately obtained according to the type of operators and the traversal order.

[0080] 2. Explain step S103.

[0081] In some implementations of the above step S103, Figure 5 The following steps S1031 to S1032 shown are used to fuse the calculation operations of multiple target operators.

[0082] Step S1031: Replace the second calculation operation with the seventh calculation operation. The seventh calculation operation is to multiply the query matrix and the key matrix, and the row elements and column elements of the key matrix are used as column elements and row elements respectively during the multiplication.

[0083] Below is the Query matrix , Key matrix , the transpose of the Key matrix Taking as an example, the second and seventh calculation operations are explained.

[0084] 1. The second calculation operation is described.

[0085] Q and Multiplying them, the result is: .

[0086] 2. Explain the seventh calculation operation.

[0087] The row and column elements of Q and K are shown in Table 1 below.

[0088] Table 1

[0089] The seventh calculation operation is: multiply Q and K, and the row elements and column elements of K are used as column elements and row elements respectively when performing the multiplication.

[0090] (1) The first row element of Q [a 11 a 12 ] is multiplied by the first column element of K and then added. However, since the row elements of K are used as column elements during the multiplication, the first row element of Q [a 11 a 12 ] will be combined with the first row element of K [b 11 b 12 ] multiplied and then added, that is: a 11 b 11 + a 12 b 12 , a 11 b 11 + a 12 b 12 It will be the first row and first column element in the product of Q and K.

[0091] (2) The first row element of Q [a 11 a 12 ] is multiplied by the second column element of K and then added. However, since the row elements of K are used as column elements during the multiplication, the first row element of Q [a 11 a 12 ] will be combined with the second row element of K [b 21 b 22 ] multiplied and then added, that is: a 11 b 21 + a 12 b 22 , a 11 b 21 + a 12 b 22 It will be the first row and second column element in the product of Q and K.

[0092] (3) The second row element of Q [a 21 a 22 ] is multiplied by the first column element of K and then added. However, since the row elements of K are used as column elements during the multiplication, the second row element of Q [a 21 a 22 ] will be combined with the first row element of K [b 11 b 12 ] multiplied and then added, that is: a 21 b 11 + a 22 b 12 , a 21 b 11 + a 22 b 12 It will be the 2nd row and 1st column element in the product of Q and K.

[0093] (4) The second row element of Q [a 21a 22 ] is multiplied by the second column element of K and then added. However, since the row elements of K are used as column elements during the multiplication, the second row element of Q [a 21 a 22 ] will be combined with the second row element of K [b 21 b 22 ] multiplied and then added, that is: a 21 b 21 + a 22 b 22 , a 21 b 21 + a 22 b 22 It will be the 2nd row and 2nd column element in the product of Q and K.

[0094] Finally, it can be determined that the result of the seventh calculation operation is: .

[0095] The result of the seventh calculation operation is the same as the result of the second calculation operation. That is, the second and seventh calculation operations have different operation contents but the same results.

[0096] Compared with the second calculation operation, the seventh calculation operation has the following technical effects: When a mobile device performs model inference, the input data of the fusion operator (including the Key matrix, Query matrix, Attention mask matrix, and Value matrix) will be stored in the memory of the mobile device. The CPU of the mobile device will obtain the input data from the memory and then perform the calculation operation of the fusion operator based on the input data. When the matrix is ​​stored in the memory of the mobile device, it is stored in the row major order (Row Major), that is, the elements in the matrix are stored row by row in the memory. Based on this, K, The storage address in the memory may be as shown in Table 2 below.

[0097] Table 2

[0098] The second calculation operation: When the first row element of the Q matrix [a 11 a 12 ]and The first element of When multiplying and then adding, if we calculate a 11 b 11 + a 12 b 12 , you need to traverse b separately 11 、b 12 According to Table 2, When storing in memory b 11 The storage address 01 and b12 The storage address 03 is not continuous. 11 After that, jump addressing is needed to obtain b 12 .

[0099] The seventh computational operation: When the first row element of Q [a 11 a 12 ] is multiplied by the first column element of K and then added. Since the row elements of K are used as column elements during the multiplication, the first row element of Q [a 11 a 12 ] will be combined with the first row element of K [b 11 b 12 ] are multiplied and then added. According to Table 2, K is stored in memory. 11 The storage address 01 and b 12 The storage address 02 is continuous, and the sequential addressing method can be used to obtain b in sequence. 11 、b 12 .

[0100] Compared with obtaining matrix elements by using a jump addressing method, obtaining matrix elements by using a sequential addressing method can significantly increase the speed of obtaining matrix elements, thereby improving the computational efficiency of matrix multiplication.

[0101] Step S1032: Fusing the seventh calculation operation, the third calculation operation, the fourth calculation operation, the fifth calculation operation, and the sixth calculation operation to form a fused operator.

[0102] That is to say, the first calculation operation is removed, the second calculation operation is replaced by the seventh calculation operation, and the third to sixth calculation operations are retained. The input of the fusion operator still includes the Key matrix, Query matrix, Attention mask matrix and Value matrix, and the output of the fusion operator is still the final result Output of the attention calculation. That is, the input tensor of the fusion operator represents the Key matrix, Query matrix, Attention mask matrix and Value matrix, and the output tensor represents Output.

[0103] Based on the method described in steps S1031 to S1032, the six calculation operations before fusion (the first to sixth calculation operations) can be reduced to five calculation operations after fusion (the seventh, third to sixth calculation operations). Since the number of calculation operations is reduced, the execution efficiency of the fusion operator can be improved. In addition, when executing the seventh calculation operation, the matrix elements can be obtained in sequence by sequential addressing, which improves the execution efficiency of the seventh calculation operation, thereby further improving the execution efficiency of the entire fusion operator.

[0104] 3. Explain step S104.

[0105] In some implementations of the above step S104, when multiple target operators in the first computation graph are replaced with fusion operators, it can be achieved through the following steps.

[0106] The second matrix multiplication operator in the first computation graph is replaced with the fusion operator, and the matrix transposition operator, the first matrix multiplication operator, the division operator, the addition operator, and the normalization operator are deleted from the first computation graph to obtain a second computation graph.

[0107] Based on this, the output of the fusion operator is still the output of the second matrix multiplication operator in the first calculation graph, and the input of the fusion operator still includes the input of the matrix transposition operator in the first calculation graph (Key matrix), the input of the first matrix multiplication operator (Query matrix), the input of the addition operator (Attention mask matrix), and the input of the second matrix multiplication operator (Value matrix). That is, the fusion operator still uses the original input and output data in the first calculation graph, but only changes the local structure of the first calculation graph, and does not affect the structure of other positions in the first calculation graph.

[0108] The following is combined with Figure 6 The overall process of obtaining and fusing target operators and replacing fusion operators is described. Figure 6 As shown, it can be achieved through the following steps S201 to S214.

[0109] Step S201: Determine whether the traversal index (index) is less than the length of the operator list.

[0110] The operator list can be understood as representing the operators in the first computation graph in the form of a list, and the length of the operator list represents the total number of operators in the first computation graph, N. Each operator in the operator list has an index, and the index value ranges from 1 to N. The traversal index index can be understood as the index value of the operator currently being traversed.

[0111] If the traversal index index is less than the length of the operator list, it indicates that there are other operators in the operator list that have not been traversed, so go to step S202; otherwise, it indicates that the operator list has been traversed and no target operator will be obtained for fusion, so go to step S212.

[0112] Step S202: Determine whether the first operator has been accessed; if it has been accessed, go to step S201; otherwise, go to step S203.

[0113] The first operator can be expressed as an ops[index] operator, where the ops[index] operator represents an operator whose index value is index.

[0114] Step S203: Determine whether the first operator is a transpose operator and whether the fuse size is 0; if so, add 1 to the value of the fuse size and go to step S204; otherwise, go to step S201.

[0115] transpose represents the matrix transpose operator.

[0116] Step S204: Determine whether the second operator is a multiplication (matmul) operator and whether the fuse size is 1; if so, add 1 to the value of the fuse size and go to step S205; otherwise, go to step S201.

[0117] The second operator can be represented as the ops[index].output operator, which represents the operator connected to the output of the ops[index] operator.

[0118] matmul represents the matrix multiplication operator.

[0119] Step S205: Determine whether the third operator is a division (div) operator and whether the fuse size is 2; if so, add 1 to the value of the fuse size and go to step S206; otherwise, go to step S201.

[0120] The third operator can be represented as a matmul.output operator, which represents the operator connected to the output of the matmul operator.

[0121] div represents the division operator.

[0122] Step S206: Determine whether the fourth operator is an addition operator and whether the fuse size is 3; if so, add 1 to the value of the fuse size and go to step S207; otherwise, go to step S201.

[0123] The fourth operator can be expressed as a div.output operator, which represents an operator connected to the output of the div operator.

[0124] add represents the addition operator.

[0125] Step S207: Determine whether the fifth operator is a normalization operator and whether the fuse size is 4; if so, add 1 to the value of the fuse size and go to step S208; otherwise, go to step S201.

[0126] The fifth operator may be represented as an add.output operator, where the add.output operator represents an operator connected to the output of the add operator.

[0127] The calculation operation of the normalization operator is to normalize the input data using the Softmax function.

[0128] Step S208: Determine whether the sixth operator is a multiplication (matmul) operator and whether the fuse size is 5; if so, go to step S209; otherwise, go to step S201.

[0129] The sixth operator can be expressed as softmax.output, and the softmax.output operator represents the operator connected to the output of the Softmax operator.

[0130] Step S209: Generate a fusion operator. In addition, the type of the fusion operator can be set to attention.

[0131] Specifically, the method described in steps S1031 to S1032 in the aforementioned method embodiment is used to fuse the calculation operations of the matrix transposition operator, matrix multiplication operator, division operator, addition operator, normalization operator, and matrix multiplication operator in steps S203 to S208 to form a fused operator.

[0132] Step S210: Setting the input and output of the fusion operator.

[0133] The input of the fusion operator includes: the input of the first operator (Key matrix), the input of the second operator (Query matrix), the input of the fourth operator (Attention mask matrix), and the input of the sixth operator (Value matrix).

[0134] The output of the fusion operator includes: the output of the sixth operator.

[0135] Step S211: Save the fusion operator.

[0136] In addition, the traversal index index is increased by 1 and the process goes to step S201.

[0137] Step S212: traverse the operator list and fusion operators.

[0138] Step S213: Replace the sixth operator with the fusion operator.

[0139] Step S214: Delete the remaining five fused operators.

[0140] Specifically, the first operator, the second operator, the third operator, the fourth operator, and the fifth operator are deleted.

[0141] Based on the above steps S201 to S214, the acquisition and fusion of the target operator and the replacement of the fused operator are realized.

[0142] The following is a description of an embodiment of a data processing method for a mobile device provided in the present application. The method is a data processing method for a neural network model. The neural network model is a neural network model deployed on a mobile device. The neural network model includes a Transformer module.

[0143] See attached Figure 7 , Figure 7 1 is a flow chart of the main steps of a data processing method for a neural network model according to an embodiment of the present application, and the method is applied to a mobile device. Figure 7 As shown, the data processing method for the neural network model in the embodiment of the present application mainly includes the following steps S301 to S302.

[0144] Step S301: According to the model file of the neural network model, obtain the target calculation graph of the neural network model. The target calculation graph is the second calculation graph obtained by the data processing method of the aforementioned method embodiment. It should be noted that after the neural network model is deployed to the mobile device, the target calculation graph of the neural network model is the second calculation graph, rather than firstly executing the data processing method of the aforementioned method embodiment by the mobile device to obtain the second calculation graph after the neural network model is deployed to the mobile device, and then replacing the actual calculation graph of the neural network model with the second calculation graph.

[0145] Step S302: In response to the neural network model receiving input data, each operator in the target calculation graph is traversed in turn and the calculation operation of each operator is executed to obtain the inference result of the neural network model performing model inference on the input data.

[0146] The functions of the neural network model are different, and the types of input data it can receive may also be different. For example, if the neural network model is a large language model, its input data may include text. The embodiment of the present application does not specifically limit the input data of the neural network model, and the input data may be data of types such as text, images, etc.

[0147] In the method described in the above steps S301 to S302, since the target computation graph is obtained by using the data processing method of the method embodiment described in the above steps S101 to S104, when the mobile device performs model inference according to the target computation graph, the calculation operation of the fusion operator in the target computation graph can be executed based on the mobile device's own CPU, thereby improving the attention calculation speed and thus improving the reasoning speed of the neural network model.

[0148] The above step S302 is further explained below.

[0149] In some implementations of step S302, the input tensor of the fusion operator in the target calculation graph is stored in the stack memory of the mobile device. When traversing each operator in the target calculation graph in turn and performing the calculation operation of each operator, the CPU of the mobile device can obtain the channel data of the input tensor from the stack memory, and perform the calculation operation of the fusion operator according to each channel data. In some implementations, the CPU of the mobile device performs the calculation operation of the fusion operator in parallel according to each channel data. By performing the calculation operation on each channel data in parallel, the calculation efficiency can be improved, which is conducive to improving the speed of model reasoning. The CPU of the mobile device can use instructions that can process multiple groups of data simultaneously, and perform the calculation operation of the fusion operator in parallel according to each channel data.

[0150] After the CPU obtains the channel data of the input tensor from the stack memory, it will store the channel data in the register, and then use the data stored in the register to perform the calculation operation. In this regard, in some embodiments, the CPU of the mobile device can obtain the channel data of the input tensor from the stack memory in the following manner: Before traversing to the fusion operator, the CPU of the mobile device obtains the channel data of each input tensor from the stack memory and stores the channel data in the cache of the mobile device. In response to traversing to the fusion operator, the CPU of the mobile device obtains the channel data from the cache. Since the access speed of the cache is relatively fast, the CPU can quickly store the channel data in the cache to the register, so that the calculation operation can be completed quickly, further improving the speed of model reasoning. In some embodiments, L1 cache (Level 1 Cache) can be used to further improve the speed of storing channel data in registers.

[0151] The following describes how the input tensors are arranged in the stack memory.

[0152] When the CPU of the mobile device performs the calculation operation of the fusion operator in parallel based on the data of each channel, the register operated by the CPU instruction can be a 128-bit register, which can store 128-bit data. If the data type of the input tensor is FP32 (32-bit floating point number), one data in the input tensor is 32 bits, and the 128-bit register can store 4 data, so the CPU instruction can process 4 data at the same time; if the data type of the input tensor is FP16 (16-bit floating point number), one data in the input tensor is 16 bits, and the 128-bit register can store 8 data, so the CPU instruction can process 8 data at the same time.

[0153] The following is combined with Figure 8, taking the input tensor with data type FP32 and 4 channels as an example, the existing arrangement method is explained. Figure 8 The shape of the input tensor is Shape=[N, C,H, W]= [1,4,4,4], where N, C, H, and W represent the batch size, channel number, height, and width of the input tensor respectively.

[0154] Taking channel 1 as an example, its data is shown in Table 3 below.

[0155] Table 3

[0156] The elements in channel 2, channel 3, and channel 4 are as follows Figure 8 As shown, no further elaboration is required.

[0157] The existing arrangement method is NCHW arrangement, which arranges the data in channel 1 row by row first, then arranges the data in channel 2 row by row, then arranges the data in channel 3 row by row, and finally arranges the data in channel 4 row by row. Figure 8 As shown, after NCHW arrangement, data 1 to data 16 in channel 1 are located in the same row, data 17 to data 32 in channel 2 are located in the same row, data 33 to data 48 in channel 3 are located in the same row, data 17 to data 32 in channel 4 are located in the same row, and the storage addresses of data 1 to data 64 are continuous.

[0158] When performing the calculation operation of the fusion operator in parallel according to the data of each channel, the CPU can store one data from channel 1 to channel 4 in a 128-bit register respectively, and then use the data of each channel stored in the 128-bit register to perform the calculation operation. Taking the data in the first row and the first column as an example, it is necessary to store data 1 of channel 1, data 17 of channel 2, data 33 of channel 3, and data 49 of channel 4 in a 128-bit register. However, if arranged in NCHW, the four data continuously obtained from the memory by the CPU using sequential addressing are data 1, 2, 3 and 4 of channel 1, not data 1, 17, 33 and 49, which makes it impossible to perform parallel calculations on the data of each channel.

[0159] The following is combined with Fig. 9 and attached Fig.10, the memory arrangement provided by the present application is described. In the embodiment of the present application, if the data type of the input tensor is FP32 and the number of channels is 4, the input tensor is stored in the stack memory according to the memory arrangement of NC4HW4; if the data type of the input tensor is FP16 and the number of channels is 8, the input tensor is stored in the stack memory according to the memory arrangement of NC8HW8. FP32 represents a 32-bit floating point number, FP16 represents a 16-bit floating point number, and N, C, H, and W represent the batch number, number of channels, height, and width of the input tensor, respectively.

[0160] (1) Explain the memory layout of NC4HW4.

[0161] The memory arrangement of NC4HW4 is as follows: the data of the j-th row and k-th column of channel i in the input tensor is used as the data of the j-th row and i+C×(k-1)-th column of the input tensor when it is stored in the stack memory, i=1,…,C, j=1,…,H, k=1,…,W, and the shape of the input tensor when it is stored in the stack memory is [N, C / 4,H, W×4].

[0162] See attached Fig. 9 , the shape of the input tensor Shape=[N, C,H, W]= [1,4,4,4].

[0163] Take the first row of data of channel 1 as an example for explanation.

[0164] For the data in the 1st row and 1st column of channel 1: i=1, j=1, k=1, i+C×(k-1)=1+4×(1-1)=1, therefore, the data is stored in the 1st row and 1st column when stored in the stack memory.

[0165] For the data in the 1st row and 2nd column of channel 1: i=1, j=1, k=2, i+C×(k-1)=1+4×(2-1)=5, therefore, the data is stored in the 1st row and 5th column when stored in the stack memory.

[0166] For the data in the 1st row and 3rd column of channel 1: i=1, j=1, k=3, i+C×(k-1)=1+4×(3-1)=7, therefore, the data is stored in the 1st row and 7th column when stored in the stack memory.

[0167] For the data in the 1st row and 4th column of channel 1: i=1, j=1, k=4, i+C×(k-1)=1+4×(4-1)=13, therefore, the data is stored in the 1st row and 13th column when stored in the stack memory.

[0168] The data of other rows of channel 1, as well as channels 2 to 4 are arranged in the same way. The arrangement results are as follows: Fig. 9Finally, it can be determined that the shape of the input tensor when stored in the stack memory is [1,1,4,16].

[0169] When executing the calculation operation of the fusion operator in parallel based on the data of each channel, the CPU uses sequential addressing to continuously obtain four data from the memory, which are the 1st row and 1st column data of channels 1 to channel 4 (that is, data 1, 17, 33 and 49). Therefore, the data of each channel can be calculated in parallel normally.

[0170] (2) Explain the memory layout of NC8HW8.

[0171] The memory arrangement of NC8HW8 is as follows: the data of the jth row and kth column of channel i in the input tensor is used as the data of the jth row and i+C×(k-1)th column of the input tensor when it is stored in the stack memory, i=1,…,C, j=1,…,H, k=1,…,W, and the shape of the input tensor when it is stored in the stack memory is [N, C / 8,H, W×8].

[0172] The arrangement principle of NC8HW8 and NC4HW4 is the same, the only difference is that the shape of the input tensor is different when it is stored in the stack memory.

[0173] See attached Fig.10 , the shape of the input tensor Shape=[N, C,H, W]= [1,8,4,4]. After memory arrangement, the shape of the input tensor when stored in the stack memory is [1,1,4,32].

[0174] In some implementations of step S302 above, the default data type stored in the stack memory of the mobile device is FP32, and the data type of the input tensor may be FP32 or FP16. In order to ensure the reliable performance of attention calculation, Fig.11 The following steps S401 to S411 shown perform the calculation operation of the fusion operator to realize the attention calculation.

[0175] Step S401: Start model reasoning.

[0176] Step S402: traverse each operator in the operator list in sequence.

[0177] The operator list represents the operators in the first computation graph in the form of a list.

[0178] Step S403: Determine whether the currently traversed operator is an attention operator; if so, go to step S404; otherwise, continue to determine whether the next operator is an attention operator when traversing to the next operator.

[0179] According to step S209 in the aforementioned method embodiment, the type of the fusion operator is set to attention, and therefore, the attention operator also represents the fusion operator.

[0180] Step S404: Determine whether the memory layout is NC4HW4 and whether the data type of the input tensor is FP32; if so, go to step S405; if not, go to step S406.

[0181] Step S405: Run the first function, and after the running is completed, go to step S411 to return the calculation result of the attention calculation.

[0182] The first function represents a computer function that performs a computational operation on an input tensor that uses the NC4HW4 memory layout and has a data type of FP32.

[0183] Step S406: Determine whether the memory layout is NC8HW8 and whether the data type of the input tensor is FP16; if so, go to step S407; if not, go to step S410.

[0184] Step S407: Apply memory for the input tensor.

[0185] Since the default data type stored in the stack memory of the mobile device is FP32, you need to apply for a memory space in which the input tensor is stored using the NC8HW8 memory layout. If you still use the default FP32 memory space in the stack memory to store FP16 input tensors, a storage error will occur.

[0186] Step S408: Use the NC8HW8 memory layout to store the input tensor in the requested memory space.

[0187] Step S409: Run the second function, and after the run is completed, go to step S411 to return the calculation result of the attention calculation.

[0188] The second function represents a computer function that performs a computational operation on an input tensor that uses the NC8HW8 memory layout and has a data type of FP16.

[0189] The operations of the first function and the second function are basically the same, and both can be implemented by the method described in step S3021 to step S3023. The only difference is that: when running the first function, the 128-bit register operated by the CPU instructions stores data of 4 channels, and the instructions are used to process the data of the 4 channels in parallel at the same time; when running the second function, the 128-bit register operated by the CPU instructions stores data of 8 channels, and the instructions are used to process the data of the 8 channels in parallel at the same time.

[0190] Step S410: Not supported.

[0191] Not supported indicates that the mobile device cannot perform the calculation operation of the attention operator. In this case, the process can go to step S411 and return the result of unsuccessful calculation.

[0192] Based on the method described in steps S401 to S411 above, when the default data type stored in the stack memory of the mobile device is FP32, the attention calculation can be reliably completed regardless of whether the data type of the input tensor is FP32 or FP16.

[0193] In some implementations of the aforementioned step S302, when performing the calculation operation of the fusion operator according to each channel data, it is possible to Fig.12 The following steps S3021 to S3023 are implemented as shown.

[0194] Step S3021: Obtain the number n of valid threads, where the number n of valid threads is the maximum number of threads that can be used in the target process, each used thread is a valid thread, and the target process is a process used to execute the computing operation of the fusion operator.

[0195] In some implementations, the number of active threads n=4.

[0196] Step S3022: splitting the query matrix in each channel data into n sub-matrices according to the number n, so as to form n query sub-matrices.

[0197] In this embodiment, a conventional matrix segmentation method can be used to segment the Query matrix according to the number n, and this embodiment does not specifically limit this. For example, the matrix block segmentation method in the Flash Attention method can be used, the segmentation number is set to the above number n, and the Query matrix is ​​segmented according to the number n.

[0198] Step S3023: Each valid thread performs the calculation operation of the fusion operator according to its corresponding data to be processed. The data to be processed includes a Query sub-matrix and the Key matrix, Value matrix and Attention mask matrix in each channel data. The Query sub-matrix in the data to be processed corresponding to different valid threads is different.

[0199] Each valid thread can run in parallel within the target process.

[0200] Based on the methods described in steps S3021 to S3023 above, multiple valid threads can be used in parallel to execute the calculation operations of the fusion operator within a target process, further improving the calculation speed of the attention calculation based on the CPU.

[0201] The above step S3023 is further explained below.

[0202] In some implementations of step S3023, each valid thread executes the calculation operation according to its corresponding data to be processed in the following way: the Query sub-matrix, Key matrix, and Value matrix in the data to be processed are respectively divided into multiple matrix blocks, and the number of rows and columns of the matrix blocks are both 1, and the valid thread executes the calculation operation of the fusion operator according to the matrix blocks of the Query sub-matrix, the matrix blocks of the Key matrix, and the matrix blocks of the Value matrix.

[0203] If the number of rows and columns are both 1, it means that the matrix block is a data in the above-mentioned Query submatrix, Key matrix, and Value matrix. In this way, when the CPU performs attention calculation based on the CPU, the CPU can use the data in the above-mentioned Query submatrix, Key matrix, and Value matrix to complete the attention calculation. The Flash Attention method provided in the paper "Fast and Memory-Efficient ExactAttention with IO-Awareness" is suitable for GPU. This method divides the Query matrix, Key matrix, and Value matrix into multiple matrix blocks. However, due to the different operating principles of the CPU and GPU, the CPU cannot calculate a matrix or a matrix block. The CPU calculates the data in the matrix one by one. The embodiment of the present application can divide the Query submatrix, Key matrix, and Value matrix into matrix blocks with the number of rows and columns of 1 respectively. In this way, when the CPU calculates a matrix block, it actually calculates a data in the matrix. Therefore, the embodiment of the present application realizes the extreme block division of the matrix, so that the CPU can reliably complete the attention calculation.

[0204] The following is combined with Fig.13 , and explain the process of operator fusion and model reasoning. Fig.13 As shown, operator fusion and model inference can be performed through the following steps S501 to S504.

[0205] Step S501: Operator fusion.

[0206] Specifically, the server may execute the method described in the aforementioned steps S101 to S104 to obtain a first computational graph of the neural network model, fuse the target operator in the first computational graph to form a fusion operator, and the type of the fusion operator is set to attention, so the fusion operator can also be understood as an attention operator. In addition, the target operator in the first computational graph is replaced with the attention operator to obtain a second computational graph of the neural network model. When the neural network model is deployed to a mobile device, the target computational graph contained in the model file of the neural network model is the second computational graph.

[0207] Step S502: The attention operator receives the input tensor.

[0208] Specifically, when the neural network model is deployed to a mobile device for model inference, the mobile device can respond to the neural network model receiving input data, traverse each operator in the target calculation graph in turn and perform the calculation operation of each operator to obtain the model result of the neural network model performing model inference on the input data. In this process, the attention operator will receive the input tensor, and the mobile device will perform the calculation operation of the attention operator according to the input tensor.

[0209] Step S503: Execute the calculation operation of the attention operator.

[0210] Specifically, the CPU of the mobile device can perform the calculation operation of the attention operator according to the input tensor of the attention operator, and the execution method is the same as the method described in the aforementioned steps S301 to S302.

[0211] Step S504: Output the result.

[0212] Specifically, the execution result of the above calculation operation is output, and the execution result can be the Output calculated by formula (4) in the term explanation part.

[0213] Based on the method described in steps S501 to S504 above, the server can complete the operator fusion of the neural network model, and then deploy the neural network model to the mobile device. During the model inference process, the CPU of the mobile device executes the calculation operation of the attention operator.

[0214] The following takes the Tongyi Qianwen 0.5B model as an example of the neural network model to illustrate the technical effects of the data processing method for the neural network model provided by the present application, wherein the Shape of the Key matrix, the Query matrix and the Value matrix is ​​[1, 32, 128, 128].

[0215] (1) Explain the inference speed of a single Transformer module.

[0216] A Transformer module in the Tongyi Qianwen 0.5B model is used as the first Transformer module. The GPU uses the Flash Attention method to perform 10 calculations on the attention calculation in the first Transformer module, and the average time consumed for the 10 calculations is 60-70ms.

[0217] The method provided in this application is used to perform operator fusion on a Transformer module in the Tongyi Qianwen 0.5B model, and the Transformer module that completes the operator fusion is used as the second Transformer module. The CPU uses the method provided in this application to perform 10 calculations on the attention calculation in the second Transformer module, and obtains the average time consumption of the 10 calculations. Among them, the average time consumption is 4.3ms when the data type of the input tensor is FP32, and the average time consumption is 3.19ms when the data type of the input tensor is FP16.

[0218] Based on the above average time consumption, it can be determined that using the method provided in this application for operator fusion and model reasoning can significantly improve the speed of attention calculation.

[0219] (2) Explain the inference speed of the entire model.

[0220] The Tongyi Qianwen 0.5B model that does not use the method provided in this application to perform operator fusion is used as the first neural network model, and the Tongyi Qianwen 0.5B model that uses the method provided in this application to perform operator fusion is used as the second neural network model.

[0221] The prompts during the test were: "Hello, who are you?", "hi", "Write a message for your circle of friends, within 15 words".

[0222] The expected answers during the test are: "I am a large-scale language model from Alibaba Cloud. My name is Tongyi Qianwen. Hello! How can I assist you today?" and "Life is like a poem. May you and I face life with a smile."

[0223] The test results of the first neural network model and the second neural network model are shown in Table 4 below: Table 4

[0224] prompt tokens num indicates the number of prompt tokens.

[0225] decode tokens num indicates the number of tokens decoded by the model.

[0226] Prefill time and prefill speed represent the time and speed of the prefilling phase, respectively. The prefilling phase is the time from the start of inference to the decoding of the first token.

[0227] Decode time and decode speed represent the time and speed of the decoding phase respectively. The decoding phase is the time from decoding the first token to decoding all tokens.

[0228] The prefill speed of the second neural network model is improved by (27.74-17.13) / 17.13=61.93% compared with the prefill speed of the first neural network model; the decoding speed of the second neural network model is improved by (9.83-6.42) / 6.42=53.11% compared with the decoding speed of the first neural network model.

[0229] The results of two tests on the second neural network model are shown in Table 5 below.

[0230] Table 5

[0231] According to Table 4 and Table 5, it can be determined that: According to the results of the first test, the prefill speed of the second neural network model is improved by (27.35-17.13) / 17.13=59.66% compared with the first neural network model; the decoding speed of the second neural network model is improved by (11.35-6.42) / 6.42=76.79% compared with the first neural network model.

[0232] According to the results of the first test, the prefill speed of the second neural network model is improved by (27.39-17.13) / 17.13=59.89% compared with the first neural network model; the decoding speed of the second neural network model is improved by (11.71-6.42) / 6.42=82.39% compared with the first neural network model.

[0233] It can be seen that the two tests again proved that the prefill speed and decoding speed of the second neural network model have been significantly improved.

[0234] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order, and they can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

[0235] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0236] Another aspect of the present application also provides a computer-readable storage medium.

[0237] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium may be configured to store a program for executing the data processing method for a neural network model of the above-mentioned method embodiment, and the program may be loaded and run by a processor to implement the above-mentioned data processing method for a neural network model. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium may be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0238] Another aspect of the present application provides a server, which may include at least one processor; and a memory connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any embodiment of the above-mentioned application to the server is implemented. Fig.14 , Fig.14In the example, the memory 11 and the processor 12 are connected to each other through a bus. In addition, the server may also be a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, etc., which is not limited in the embodiment of the present application.

[0239] Another aspect of the present application also provides a mobile device.

[0240] In an embodiment of a mobile device according to the present application, the mobile device may include at least one processor; and a memory connected to the at least one processor in communication; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any embodiment of the mobile device is implemented. Fig.15 , Fig.15 2 shows that the memory 21 and the processor 22 are connected by bus communication. The mobile device may be a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, a netbook, a personal digital assistant (PDA), an augmented reality (AR) or a virtual reality (VR) device, etc., which is not limited in the embodiments of the present application.

[0241] So far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A data processing method for a neural network model, characterized in that: Applied to a server, the neural network model is a neural network model for deployment on a mobile device, the neural network model includes a Transformer module, and the method includes: According to the model file of the neural network model, a first calculation graph of the neural network model is obtained, where the first calculation graph is used to describe all operators of the neural network model and connection relationships between the operators; Acquire multiple target operators in the first computation graph, where the multiple target operators are multiple operators used by the Transformer module to perform attention calculation; Fusing the computing operations corresponding to the multiple target operators to form a fused operator, wherein the fused operator is used for a CPU of the mobile device to perform the computing operation; The multiple target operators in the first computation graph are replaced with the fusion operator to obtain a second computation graph of the neural network model, and the second computation graph is used by the mobile device to perform model inference on the input data of the neural network model to obtain an inference result.

2. The data processing method for a neural network model according to claim 1, characterized in that: The connection relationship between the operators is used to describe the input and output relationship between the operators. The obtaining of the multiple target operators in the first computation graph includes: Determining a target arrangement order of the multiple target operators according to a connection relationship between the multiple target operators; The operators in the first computation graph are traversed in sequence. If all the multiple target operators are traversed and the traversal order of the multiple target operators is the same as the target arrangement order, the multiple target operators are obtained.

3. The data processing method for a neural network model according to claim 1, characterized in that: The multiple target operators include a matrix transposition operator, a first matrix multiplication operator, a division operator, an addition operator, a normalization operator, and a second matrix multiplication operator connected in sequence; The first calculation operation corresponding to the matrix transposition operator is: transposing the Key matrix in the attention mechanism to obtain a transposed matrix; The second calculation operation corresponding to the first matrix multiplication operator is: multiplying the Query matrix in the attention mechanism by the transposed matrix to obtain a first attention score; The third calculation operation corresponding to the division operator is: Divide to obtain a second attention score, where d is the dimension of the Key matrix or the Query matrix; The fourth calculation operation corresponding to the addition operator is: adding the Attention mask matrix of the attention mechanism to the second attention score to obtain a third attention score; The fifth calculation operation corresponding to the normalization operator is: normalizing the third attention score to obtain an attention weight; The sixth calculation operation corresponding to the second matrix multiplication operator is: multiplying the Value matrix in the attention mechanism by the attention weight to obtain the attention calculation result.

4. The data processing method for a neural network model according to claim 3, characterized in that: The fusing the computing operations corresponding to the multiple target operators to form a fused operator includes: The second calculation operation is replaced by a seventh calculation operation, wherein the seventh calculation operation is to multiply the Query matrix by the Key matrix, and the row elements and column elements of the Key matrix are used as column elements and row elements respectively when performing the multiplication; fusing the seventh computing operation, the third computing operation, the fourth computing operation, the fifth computing operation, and the sixth computing operation to form the fused operator; in, The input tensor of the fusion operator represents the Key matrix, the Query matrix, the Attention mask matrix and the Value matrix; The output tensor of the fusion operator represents the attention calculation result.

5. The data processing method for a neural network model according to claim 4, characterized in that: The step of replacing the multiple target operators in the first computation graph with the fusion operator to obtain a second computation graph of the neural network model includes: The second matrix multiplication operator in the first computation graph is replaced with the fusion operator, and the matrix transposition operator, the first matrix multiplication operator, the division operator, the addition operator and the normalization operator are deleted from the first computation graph to obtain the second computation graph.

6. A data processing method for a neural network model, characterized in that: Applied to a mobile device, the neural network model is a neural network model deployed on the mobile device, the neural network model includes a Transformer module, and the method includes: According to the model file of the neural network model, obtaining a target calculation graph of the neural network model, wherein the target calculation graph is a second calculation graph obtained by using the data processing method for a neural network model according to any one of claims 1 to 5; In response to the neural network model receiving input data, each operator in the target calculation graph is traversed in turn and the calculation operation of each operator is performed to obtain the inference result of the neural network model performing model inference on the input data.

7. The data processing method for a neural network model according to claim 6, characterized in that: The input tensor of the fused operator in the target computation graph is stored in the stack memory of the mobile device, and the sequentially traversing each operator in the target computation graph and performing the computation operation of each operator includes: The CPU of the mobile device obtains the channel data of the input tensor from the stack memory, and performs the calculation operation of the fusion operator according to the channel data.

8. The data processing method for a neural network model according to claim 7, characterized in that: The CPU of the mobile device obtains channel data of the input tensor from the stack memory, including: Before traversing to the fusion operator, the CPU of the mobile device obtains each channel data of the input tensor from the stack memory, and stores the each channel data in the cache of the mobile device; In response to traversing to the fusion operator, the CPU of the mobile device obtains the channel data from the cache.

9. The data processing method for a neural network model according to claim 8, characterized in that: The cache is an L1 cache.

10. The data processing method for a neural network model according to claim 7, characterized in that: The input tensor is stored in the stack memory in the following way: If the data type of the input tensor is FP32 and the number of channels is 4, the input tensor is stored in the stack memory according to the memory arrangement of NC4HW4; If the data type of the input tensor is FP16 and the number of channels is 8, the input tensor is stored in the stack memory according to the memory arrangement of NC8HW8; in, N, C, H, and W represent the batch number, channel number, height, and width of the input tensor, respectively; The memory arrangement of NC4HW4 is as follows: the data of the jth row and the kth column of channel i in the input tensor is used as the data of the jth row and the i+C×(k-1)th column of the input tensor when it is stored in the stack memory, i=1,…,C, j=1,…,H, k=1,…,W, and the shape of the input tensor when it is stored in the stack memory is [N, C / 4,H, W×4]; The memory arrangement of NC8HW8 is as follows: the data of the jth row and kth column of channel i in the input tensor is used as the data of the jth row and i+C×(k-1)th column of the input tensor when it is stored in the stack memory, i=1,…,C, j=1,…,H, k=1,…,W, and the shape of the input tensor when it is stored in the stack memory is [N, C / 8,H, W×8].

11. The data processing method for a neural network model according to claim 7, characterized in that: The performing the calculation operation corresponding to the fusion operator according to the channel data includes: The CPU of the mobile device executes the calculation operations corresponding to the fusion operator in parallel according to the channel data.

12. The data processing method for a neural network model according to claim 7, characterized in that: The performing the calculation operation of the fusion operator according to the channel data includes: Obtaining the number n of valid threads, where the number n of valid threads is the maximum number of threads that can be used in a target process, and the target process is a process used to execute a computing operation of the fusion operator; Performing matrix segmentation on the Query matrix in each channel data according to the number n to form n Query sub-matrices; Each of the effective threads executes the calculation operation corresponding to the fusion operator according to the corresponding data to be processed; The data to be processed includes the Query sub-matrix and the Key matrix, the Value matrix and the Attention mask matrix in each channel data, and the Query sub-matrices in the data to be processed corresponding to different valid threads are different.

13. The data processing method for a neural network model according to claim 12, characterized in that: Each of the valid threads executes the calculation operation corresponding to the fusion operator according to the corresponding data to be processed, including: The Query sub-matrix, the Key matrix, and the Value matrix in the data to be processed are respectively divided into a plurality of matrix blocks, wherein the number of rows and the number of columns of the matrix blocks are both 1; The valid thread executes the calculation operation corresponding to the fusion operator according to the matrix block of the Query sub-matrix, the matrix block of the Key matrix, and the matrix block of the Value matrix.

14. A server, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the data processing method for a neural network model described in any one of claims 1 to 5 is implemented.

15. A mobile device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the data processing method for a neural network model described in any one of claims 6 to 13 is implemented.

16. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the data processing method for a neural network model described in any one of claims 1 to 5, or the data processing method for a neural network model described in any one of claims 6 to 13.

Images