Reasoning method and device based on softmax neural network model
By performing behavior unit sparse processing and softmax operator optimization on the input features of the neural network model, the problems of accuracy and calculation amount in the existing sparse processing are solved, and the model inference speed is improved and the effect is guaranteed.
Patent Information
- Application Number
- CN202411017271.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-29
AI Technical Summary
In the sparse processing of existing neural network models based on Transformer structure, low-precision calculation leads to insufficient accuracy, high-precision calculation leads to large calculation amount, which cannot effectively reduce the calculation overhead and affect the model inference effect.
Using the inference method of the neural network model based on softmax, the input features are sparsely processed in behavioral units, the largest first N elements of each row are retained, and subsequent features are processed using the softmax operator in the artificial intelligence chip, combined with the CPU Or the VPU performs sparse processing and matrix multiplication to optimize the linear layer weights to reduce the differences before and after sparse processing.
It effectively improves the inference speed of neural network models, reduces the time-consuming data handling and calculation processing, ensures the model inference effect, and controls performance losses at a sparseness of 75%.
Smart Images

Figure CN118551848B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to artificial intelligence technology, and in particular to a reasoning method and device based on a softmax neural network model. Background Art
[0002] With the advancement of artificial intelligence technology, systems using neural network models have been increasingly used.
[0003] In systems that use neural network models, processing using neural network models usually involves a lot of intensive computing and requires a lot of processing time to implement. In order to reduce time overhead and increase the processing speed of the model, some solutions have proposed to simplify some calculations in the neural network model through sparse processing.
[0004] Specifically, this solution is for the neural network model based on the Transformer structure. The low-precision matrix Q and matrix K are determined through quantization processing. The structured mask is determined by multiplying these two low-precision matrices. Then, the high-precision matrix Q and matrix K are used to calculate again at the corresponding position according to the non-zero value of the mask. Although this sparse processing method can simplify some calculations, it is impossible to balance the benefits of low-precision calculations and sparse calculations. If the accuracy of low-precision calculations is too low, the mask will be inaccurate, and the performance loss of the neural network model after sparse processing will be large; if the accuracy of low-precision calculations is too high, the calculation overhead of the mask will be large, and the effect of effectively reducing the calculation overhead cannot be achieved; at the same time, the above-mentioned sparse processing method will also destroy the original calculation results of the neural network model and affect the model reasoning effect. Summary of the invention
[0005] The present application provides an inference method, device and storage medium for a softmax-based neural network model, which can effectively improve the model inference speed and ensure the model inference effect.
[0006] To achieve the above objectives, this application adopts the following technical solutions:
[0007] A method for reasoning a neural network model based on softmax, wherein the reasoning of the neural network model includes processing of a softmax operator, and the method comprises:
[0008] The artificial intelligence chip uses the neural network model to infer the input data of the neural network model until the input features of the softmax operator are determined;
[0009] The central processing unit CPU or the vector processing unit VPU performs sparse processing on the input features in units of lines, and processes the sparsely processed input features using a softmax operator;
[0010] The artificial intelligence chip completes the reasoning of the neural network model based on the output result processed by the softmax operator;
[0011] The step of performing sparse processing on the input features in units of lines includes:
[0012] In each row of the input feature, M consecutive elements are sorted, and the largest first N elements are selected, and their values and indexes are retained, and the number of elements with retained values in each row is the same.
[0013] Preferably, when the neural network model is a neural network model based on an attention mechanism,
[0014] After the input features are subjected to sparse processing, the method further comprises: the CPU or VPU marks index information of retained elements in the input features subjected to sparse processing;
[0015] After the input features after the sparse processing are processed by the softmax operator, the method further includes: the artificial intelligence chip determines the element position of the retained value in the softmax output feature based on the index information of the retained element in the input features after the sparse processing, and searches for the element at the same position as the determined element position in the matrix V, retains the value of the element queried in the matrix V, and discards other elements in the matrix V;
[0016] The artificial intelligence chip completes the reasoning of the neural network model based on the updated matrix V.
[0017] Preferably, before determining the input features of softmax through reasoning of the neural network model, the method further includes: the artificial intelligence chip pre-fine-tunes the linear layer weights of the module where the softmax operator in the neural network model is located, and the goal of the fine-tuning is to minimize the difference between the output features of the module before and after the sparsification processing.
[0018] Preferably, when the neural network model is a neural network model based on an attention mechanism, the module where the softmax operator is located is: an attention module.
[0019] Preferably, the sparse processing of the input features comprises:
[0020] In the input features, sparse processing is performed in units of lines;
[0021] In each row of the input feature, M consecutive elements are sorted, and the largest first N elements are selected, and their values and indexes are retained, and the number of elements with retained values in each row is the same.
[0022] An inference device based on a softmax neural network model, comprising: a first inference unit, a sparse processing unit, a softmax operator processing unit and a second inference unit;
[0023] The first reasoning unit is used to use the neural network model to reason on the input data of the neural network model until the input features of the softmax operator are determined; wherein the reasoning of the neural network model includes the processing of the softmax operator;
[0024] The sparse processing unit is used to perform sparse processing on the input feature in units of rows; wherein the sparse processing on the input feature in units of rows includes: in each row of the input feature, sorting M consecutive elements, selecting the largest first N elements therein, retaining their values, and the number of elements with retained values in each row is the same;
[0025] The softmax operator processing unit is used to process the input features after the sparse processing using the softmax operator;
[0026] The second reasoning unit is used to complete the reasoning of the neural network model and determine the output of the neural network model.
[0027] Preferably, when the neural network is a neural network model based on an attention mechanism,
[0028] The device further includes an index information processing unit, which is used to mark the index information of each element in the input feature after the sparse processing;
[0029] The softmax operator processing unit, after processing the input feature after the sparse processing by using the softmax operator, is further used to determine the element position of the softmax output feature that retains the original value based on the index information of each element in the input feature after the sparse processing, and to query the element at the same position as the determined element position in the matrix V, retain the value of the element queried in the matrix V, and discard other elements in the matrix V;
[0030] The second reasoning unit is used to complete the reasoning of the neural network model based on the updated matrix V.
[0031] Preferably, the device further includes a fine-tuning unit for pre-fine-tuning the linear layer weights of the module where the softmax operator in the neural network model is located, and the goal of the fine-tuning is to minimize the difference between the output features of the module before and after the sparsification processing.
[0032] Preferably, when the neural network model is a neural network model based on an attention mechanism, the module where the softmax operator is located is: an attention module.
[0033] A computer-readable storage medium having computer instructions stored thereon, characterized in that when the instructions are executed by a processor, any one of the above-mentioned softmax-based neural network model inference methods can be implemented.
[0034] It can be seen from the above technical scheme that the present application is aimed at a neural network model including softmax operator processing. When performing reasoning, the artificial intelligence chip uses the neural network model to reason on the input data to determine the input features of the softmax operator; then, the central processing unit (CPU) or vector processing unit (VPU) performs sparse processing on the determined input features in line units, and then uses the softmax operator to process the sparsely processed input features; finally, the artificial intelligence chip completes the reasoning of the neural network model based on the output result after the softmax operator processing. The softmax operator is used in most neural network models, and the data transfer between the CPU and the artificial intelligence chip and the calculation processing in the CPU of the softmax operator account for a relatively large proportion of the time consumed in the entire model reasoning. Therefore, the present application effectively reduces the time consumed in data transfer and calculation processing of the softmax operator by sparsely processing the input features of the softmax operator in line units, and can effectively speed up the model reasoning speed; at the same time, since the present application performs sparse processing on the operator, the model structure is not affected, and the model reasoning is consistent. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a schematic diagram of the time consumption ratio of different modules in the Transformer model and the time consumption ratio of different operators in the Attention module;
[0036] Figure 2 This is a schematic diagram of the calculation process of the softmax operator;
[0037] Figure 3 A schematic diagram of the basic flow of the reasoning method based on the softmax neural network model in this application;
[0038] Figure 4 It is a specific flow chart of the reasoning method based on the softmax neural network model in the specific embodiment of the present application;
[0039] Figure 5 This is a flowchart of the attention module in the Transformer model;
[0040] Figure 6 Schematic diagram of input features before NM sparse processing and input features and index information after sparse processing in a specific embodiment of the present application;
[0041] Figure 7 A schematic diagram of the basic structure of a chip provided in a specific embodiment of the present application;
[0042] Figure 8 This is a basic structural diagram of the softmax-based neural network model inference device in this application. DETAILED DESCRIPTION
[0043] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.
[0044] In order to solve the problem that the neural network model takes a long time to reason about the input data, the applicant analyzed the time consumption of each module in the neural network model. Figure 1 The figure shows the time consumption ratio of different modules in the Transformer model and the time consumption ratio of different operators in the attention module. Among them, in the corresponding model, although the matrix multiplication operation in the artificial intelligence chip accounts for the vast majority of the computational overhead, the high efficiency of matrix multiplication in the artificial intelligence chip makes its time consumption ratio in reasoning not high. As for the softmax operator processing in the attention module, since the operator is a nonlinear operator, its processing is generally completed in the CPU, that is, the input data of the operator needs to be transferred from the artificial intelligence chip to the storage unit of the CPU, and then the CPU processes the softmax operator, and finally the CPU moves it back to the artificial intelligence chip; at the same time, the processing of the softmax operator includes complex operations such as exp calculation and div division, so it is also time-consuming, such as Figure 1 As shown, softmax processing accounts for 33.3% of the time-consuming overhead. It can be seen that in the neural network model based on softmax, the data processing (including data handling and calculation) related to softmax is very time-consuming. At the same time, softmax processing is used in many neural network models. Based on this, in this application, sparse processing is performed starting from softmax processing to improve the reasoning speed of the neural network model. Accordingly, the neural network model of this application is based on softmax, that is, the operation of the softmax operator is included in the processing of the neural network model.
[0045] Before introducing this application, let's first introduce the specific processing flow of the softmax operator. Assume that the softmax input is X, whose dimension is M×N, and the second dimension is the calculation dimension of Softmax. Figure 2 The figure shows the calculation process of the softmax operator. The specific processing includes:
[0046] 1. Calculate the maximum value along the second dimension, Y=Max(X). The dimension of the feature after calculation is M×1.
[0047] That is, the output Y is a column vector, each element of which is the maximum value found in the corresponding row of X;
[0048] 2. For each element of input X, subtract the maximum value of the corresponding row calculated in step 1, that is, Z = XY. After calculation, the dimension of feature Z is M×N;
[0049] 3. For each element of feature Z, calculate the E index to obtain feature E, E=exp(Z). After calculation, the dimension of feature E is M×N.
[0050] 4. Calculate the sum along the second dimension, S=sum(E). After calculation, the dimension of feature S is M×1.
[0051] That is, the output S is a column vector, and each element in the vector is the sum of the elements in the corresponding row of E;
[0052] 5. For feature E, remove the value corresponding to the corresponding row in feature S from each element in each row, O=E / S, and the dimension of feature O is calculated to be M×N.
[0053] The above is the processing flow of the softmax operator. The following is an introduction to the reasoning method of the neural network model based on the softmax in this application.
[0054] Figure 3 The following is a basic flow chart of the inference method based on the softmax neural network model in this application. Figure 3 As shown, the method includes:
[0055] Step 301: The artificial intelligence chip uses the neural network model to infer the input data of the neural network model until the input features of the softmax operator are determined.
[0056] The neural network model improved in this application is a softmax-based neural network model. This model can be applied to various scenarios and fields, such as license plate recognition scenarios, OCR text recognition scenarios, and pedestrian retrieval scenarios, and the input data of the model can be various types of data such as image data and text data.
[0057] In the present application, the softmax processing is sparse, specifically, the softmax input features are sparse, and then the sparse input features are processed using the softmax operator to obtain the sparse softmax processing results.
[0058] In this step, the artificial intelligence chip uses the neural network model to infer the input data until the input features of the softmax operator are obtained. In this step, the artificial intelligence chip does not need to perform the complete inference process of the neural network model, as long as the inference reaches before the processing of the softmax operator, the input features of the softmax operator are obtained. The specific inference process is the same as the existing model process.
[0059] In addition, the processing before the softmax operator is mainly completed in artificial intelligence chips (such as GPU, NPU, etc.). Of course, in some application scenarios, some nonlinear processing may also be involved and need to be completed in the CPU or vector processor (VPU), but the linear processing of the neural network model is basically completed in the artificial intelligence chip.
[0060] In step 302, the CPU or VPU performs a sparse processing on the input features determined in step 301 in units of lines.
[0061] In artificial intelligence chips such as GPU and GPGPU that mainly perform linear computing processing, nonlinear processing such as softmax cannot be implemented, and the corresponding processing is completed in the CPU. In artificial intelligence chips such as NPU that can perform complex calculations, in addition to ordinary computing processing units for linear computing processing, vector processing units (VPUs) can also be included. Because VPUs can perform nonlinear processing, softmax operations can also be performed in VPUs.
[0062] Before performing the sparse processing, the input features are moved from the artificial intelligence chip to the CPU or VPU. Next, the sparse processing is performed. Existing sparse processing methods mainly include threshold sparse, sorted sparse, etc. Specifically, threshold sparse means giving a threshold T, sparse out the elements in the feature whose values are less than T, and retain the values greater than T. Sorted sparse means sorting the features by row or column, retaining only the largest top K elements. This application does not limit the specific sparse processing method.
[0063] This application takes into account that the threshold sparse method is fast, but the sparsity between rows is unbalanced, which affects the load balance of subsequent sparse matrix calculations; while the sorting sparse method can ensure the same sparsity between rows, but the calculation speed is slow when the number of sorted elements is large. Based on this, this application adopts the NM sparse method to sparse the input features of Softmax to combine the advantages and disadvantages of the two sparse methods. Among them, the NM sparse method refers to sparse processing of the matrix in units of rows, and in each row, a set number of matrix elements are selected for retention according to the sorting sparse method. Specifically, in each row of the input feature, the M consecutive elements are sorted, and the largest first N elements are selected, and their values and indexes are retained, and the number of elements with retained values in each row is the same.
[0064] Step 303: The CPU or VPU processes the input features after the sparse processing using a softmax operator.
[0065] Through the processing of step 302, the input features of the softmax operator are sparsely processed, thereby greatly reducing the amount of data in the input features. In this step, the softmax operator is used to process the input features after sparse processing. The specific processing flow of the softmax operator is the same as the existing processing, that is, Figure 2 The processing flow shown in the figure only differs in that the input features are sparsely processed.
[0066] Specifically, in the operator calculation process, the minus Max in step 2 is used to prevent numerical overflow. For the input features after sparse processing, this part of the calculation cannot be omitted, but the number of elements for calculating Max is reduced, which can also have a certain speed improvement effect. However, since many elements in the input features are discarded after sparse processing, the number of elements involved in the EXP operation in step 3 and the div division operation in step 5 is reduced, so the calculation time of the corresponding elements can be saved. According to the calculation time information, Exp is time-consuming as a nonlinear operator. Therefore, the input features of sparse Softmax can achieve Softmax calculation acceleration. At the same time, since Softmax is monotonic, the mask of the input features after sparse processing can be applied to the output features accordingly, that is, the position of the sparse elements in the Softmax input features is 0 in the output features. Furthermore, since many elements in the input features after sparse processing are discarded, the number of elements in the output features after softmax processing is also reduced accordingly, saving the time of data handling.
[0067] Step 304: The artificial intelligence chip completes the reasoning of the neural network model based on the output result processed by the softmax operator.
[0068] After the softmax processing is performed, the output result obtained after the processing is moved from the CPU or VPU to the artificial intelligence chip. Among them, when the VPU is included in an artificial intelligence chip such as the NPU that can realize nonlinear operations, after the VPU completes the softmax processing, the output result is moved from the VPU to the ordinary computing unit. The artificial intelligence chip continues to complete the reasoning of the neural network model based on the output result after the softmax operator processing. As above, one reasoning of the neural network model is completed. Next, you can return to step 301 to perform the next reasoning, and so on until the output of the neural network model is obtained. The specific reasoning and output determination process is the same as the existing method, so it will not be repeated here.
[0069] So far, Figure 3 The basic method flow shown ends. The above-mentioned neural network model reasoning method can be performed for various softmax-based neural network models, such as CNN models, RNN models or models based on attention mechanisms. Among them, compared with the CNN model and the RNN model, the softmax operator is usually located before determining the model output. In the model based on the attention mechanism, the softmax operator is located in the attention module. After the softmax operator is processed, other calculations such as matrix multiplication are required. These calculations are based on the results after sparsification, which can further improve the processing speed. Therefore, for the model of the attention mechanism, the neural network model reasoning method of this application can obtain better processing acceleration effect.
[0070] The specific implementation of the inference method of the softmax-based neural network model in this application is described below through a specific embodiment. In this specific embodiment, a neural network model with a Transformer structure is used as an example for description, and the artificial intelligence chip is specifically a GPU. Figure 4 Schematic diagram of the specific process of the inference method based on the softmax neural network model in the specific embodiment of the present application. Figure 4 As shown, the method includes:
[0071] Step 401, pre-fine-tune the linear layer weights of the module where the softmax operator in the neural network model is located.
[0072] This step is to Figure 3 A supplement to the basic method in the process shown. Its purpose is to optimize the model to improve the performance of the entire neural network model after sparse processing.
[0073] This optimization is performed on the module where the softmax operator is located. In this embodiment, the neural network model is a model of the attention mechanism, and the module where the corresponding softmax operator is located is the attention module. In this step, the linear layer weight of the attention module is fine-tuned. In the specific Transformer model, the flowchart of the attention module is as follows: Figure 5 As shown in the figure, it specifically includes: a) the input is linearly transformed to generate Q, K, and V; b) the features Q and K are matrix multiplied to generate feature S; c) the feature S is passed through the Softmax operator to generate feature A; d) the features A and V are matrix multiplied and then linearly transformed to obtain the final output.
[0074] The fine-tuning and optimization in this step are based on the difference in output features of the attention module before and after sparse processing. The specific constraints of fine-tuning and optimization are: to minimize the difference between the output features of the attention module before and after sparse processing.
[0075] In more detail, the attention module is used to perform a first processing on the input to obtain a first output feature, and the first processing refers to the input of the softmax operator in the attention module as the input feature before the sparse processing; the attention module is used to perform a second processing on the same input to obtain a second output feature, and the second processing refers to the input of the softmax operator in the attention module as the input feature after the sparse processing; next, the first output feature and the second output feature are compared, and the linear layer weights in the attention module are adjusted with the goal of minimizing the difference between the two features. The above optimization objective can be expressed as follows:
[0076]
[0077] Among them, F is the function corresponding to the calculation of the attention module, x is the input feature of the attention module, W is the weight feature, and δ is the optimization parameter. In order to prevent overfitting during the optimization process, the range of δ can be constrained as above, and a is a hyperparameter, whose value can be set based on experience.
[0078] The fine-tuning process in this step is an optional operation, which can usually be achieved through software simulation, and then the fine-tuning result obtained by simulation is set in the attention module.
[0079] In step 402, the GPU uses the neural network model to infer the input data of the model until the input features of the softmax operator are determined.
[0080] The processing of this step is the same as step 301, and will not be repeated here. After obtaining the input features of the softmax operator, the input features are transferred from the GPU to the CPU.
[0081] In step 403, the CPU performs a sparse processing on the input features determined in step 402 in units of lines.
[0082] In this embodiment, the NM sparse method is used for sparse processing. NM sparse means that among the consecutive M elements, only N elements are non-zero. After sparse processing, only non-zero elements and corresponding indexes are retained.
[0083] Specifically in this embodiment, the NM sparse method is used to sparse the input features of Softmax, such as Figure 5 As shown in the figure, this is the output feature of the QK matrix multiplication. The NM sparse method is used to reduce the number of elements involved in the Softmax calculation, which improves the calculation speed and data transfer speed compared to the original dense features.
[0084] Specifically, let S = QK be the output feature of the multiplication of the Q and K matrices, that is, the input feature of the softmax operator. The calculation process of the NM sparse method (NM SparseSoftmax) is as follows:
[0085] a. Perform sparse processing on the input feature S in units of lines;
[0086] b. For each row of the input feature S, sort the M consecutive elements in the row, select the largest first N elements, retain their values, and discard the other elements;
[0087] c. For the input feature S, save the index of the retained element.
[0088] Through the above NM sparse method, for each row of the input feature S, the number of elements with the same value is retained, which can ensure the same sparsity between rows. At the same time, after sparse processing, the features can still be represented by a structured method. The input feature after sparse processing is recorded as S_sval (sparse feature), where the index information corresponding to the retained element is recorded as S_sidx (index), as follows Figure 6 As shown in the figure, the number of elements of the sparse feature S_sval is only N / M of the dense feature S, for example, M=8 and N=2, which significantly reduces the number of elements of the input feature.
[0089] Step 404: The CPU processes the input features after the sparse processing using the softmax operator.
[0090] In this step, Softmax is calculated for the S_sval feature to obtain the feature A_sval. The output feature A_sval and the corresponding index information are transferred to the GPU.
[0091] Step 405: The GPU performs sparsification on the matrix V based on the index values of the retained elements corresponding to the input features.
[0092] like Figure 5 As shown, the output feature after the softmax operator processing needs to be matrix multiplied with the matrix V. Since the output feature A_sval of the softmax operator is obtained by the feature S_sval after sparse processing, the output feature A_sval is also equivalent to being sparsely processed, and accordingly, the matrix V that performs the matrix multiplication operation with it also needs to be sparsely processed.
[0093] Specifically, the CPU transfers the output result after processing by the softmax operator to the GPU. The GPU can determine the position of the retained elements in the input feature S based on the index information S_sidx, and further determine the position of the retained elements in the output feature S_sval. By comparing with the matrix V, the elements in the same position are found, and the values of the elements in these positions in the matrix V are retained, and other elements are discarded, so that the sparsely processed matrix V_sval can be determined.
[0094] In step 406, the GPU performs a multiplication operation on the output feature A_sval and the matrix V_sval.
[0095] Here, A_sva and V_sval are both matrices that have been sparsely processed by NM. Therefore, a special tensor computing core can be used to accelerate the matrix multiplication operation of the two, and its acceleration effect is more obvious than that of traditional sparse matrices. The specific processing can be implemented in the existing way, which will not be repeated here.
[0096] Step 407: The GPU completes the reasoning of the neural network model.
[0097] After completing the matrix multiplication operation of step 406, other reasoning in the neural network model can be completed according to the processing method of the existing neural network model, and the process can return to step 401 for the next model reasoning, and so on, until the output of the neural network model is determined. The processing here can be performed in the existing manner, and will not be repeated here.
[0098] So far, Figure 4 The method flow shown ends. In the above method flow, the artificial intelligence chip is specifically a GPU, and of course the corresponding processing can also be performed in Figure 7 The chip shown is implemented as a neural network processor NPU, which is mounted on the main CPU (Host CPU) as a coprocessor, and the CPU assigns tasks. The NPU includes a VPU that can implement nonlinear processing.
[0099] like Figure 7 As shown, the chip mainly includes the following components:
[0100] 1. Main CPU: completes various computing operations and instructions;
[0101] 2. External memory: used to store data required for calculation;
[0102] 3. Bus: used for interaction between AXI bus, DMAC and instruction fetch memory;
[0103] 4. Neural network processor: mounted on the main CPU as a coprocessor, and the main CPU assigns tasks. A processing unit used to complete computationally intensive operations;
[0104] 5. Storage unit access controller: used to transfer input data in the external memory DDR to the unified memory, or transfer weight data to the weight memory, or transfer input data to the input memory;
[0105] 6. Unified memory: used to store input data and output data of calculation;
[0106] 7. Input memory: used to store input data related to calculation;
[0107] 8. Weight memory: used to store the weights of the convolutional layer or linear layer of the neural network;
[0108] 9. Ordinary arithmetic circuit: a one-dimensional or two-dimensional pulsating array, an electronic circuit capable of performing mathematical operations such as multiplication and addition, mainly used to complete instructions related to matrix processing;
[0109] 10. Accumulator: used to accumulate the output of a single row-column matrix calculation that is divided into multiple row-column multiplications due to the limitation of the computing unit size;
[0110] 11. Vector Processing Unit VPU: It includes multiple processing units, which can further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer calculations in neural networks, such as Softmax calculations;
[0111] 12. Instruction fetch memory: used to fetch instructions from external memory;
[0112] 13. Controller: controls the operation circuit to extract the matrix data in the memory and perform multiplication operations.
[0113] In the above chip, when implementing matrix multiplication, assume that the input matrix A, the weight matrix B, and the output matrix C = A × B. The operation circuit takes the corresponding data of matrix B from the weight memory and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory and performs matrix operations with matrix B. The partial results or final results of the matrix are stored in the accumulator. If nonlinear calculations are required after obtaining matrix C, such as RELU function, then this part of the operation is completed in the vector calculation unit.
[0114] When VPU implements Softmax calculation, it inputs matrix A and outputs matrix B. Specifically, the instruction fetch memory controls the storage unit access controller to store input A into the unified memory. Softmax calculation is completed in vector operations, including taking Max, calculating Exp, etc. Sparse Softmax is different from conventional Softmax in that the instructions obtained by the instruction fetch memory are different.
[0115] It can be seen from the reasoning process of the Transformer structure model in the above specific embodiment that the reasoning method in the embodiment of the present application proposes to use the NM sparse method to sparse the input features of Softmax during the Attention calculation process. On the one hand, NM sparse reduces the number of elements involved in the Softmax calculation, thereby improving the calculation speed and data handling speed, and ensures the same sparsity between rows, so that the subsequent multiplication with the V matrix can also be accelerated by the NM sparse matrix multiplication, and a better reasoning acceleration effect can be obtained; on the other hand, the optimization and fine-tuning method of step 401 can effectively improve the performance of the sparse model. According to experiments, the point drop can be controlled within 1 point at a sparsity of 75%; on the other hand, NM SparseSoftmax directly sparses the input features, and does not need to perform secondary calculations of coarse precision and fine precision like the solution mentioned in the background technology. In the Transformer model, it can equivalently replace Softmax, improve the calculation speed, and ensure the model reasoning effect.
[0116] The neural network model inference method in this application can be applied in various application scenarios to process various types of data such as text and images. For example, it can be applied in the fields of target detection, classification, target tracking, etc. The following are some typical application scenarios:
[0117] Application scenario 1: License plate recognition scenario
[0118] License plate recognition refers to the technology that can detect vehicles on the monitored road and automatically extract vehicle license plate information (including Chinese characters, English letters, Arabic numerals and license plate color) for processing. License plate recognition is one of the important components of modern intelligent transportation systems and is widely used. It is based on digital image processing, pattern recognition, computer vision and other technologies, analyzes the vehicle images or video sequences taken by the camera, and obtains the unique license plate number of each car, thereby completing the recognition process. Through some subsequent processing methods, parking fee management, traffic flow control index measurement, vehicle positioning, car anti-theft, highway speeding automatic supervision, red light electronic police, highway toll stations and other functions can be realized. The steps of license plate recognition are mainly to locate the license plate position in the picture first, then segment the characters in the license plate, and finally recognize the segmented characters to form the license plate number. Among them, license plate character recognition is based on neural network implementation, and the method of this application can quickly and efficiently complete the recognition task.
[0119] Application scenario 2: OCR text recognition scenario
[0120] OCR (Optical Character Recognition) is the process of an electronic device (such as a scanner or digital camera) checking the characters printed on paper and then translating the shapes into computer text using character recognition methods; that is, the process of scanning text materials and then analyzing and processing the image files to obtain text and layout information. Recognition speed is one of the main indicators for measuring the performance of an OCR system. Using the method of this application to extract text features can improve recognition speed, thereby improving the practicality of OCR products.
[0121] Application scenario 3: Pedestrian retrieval scenario
[0122] Pedestrian retrieval is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is an image retrieval problem. Given a monitored pedestrian image, retrieve the pedestrian image across devices. The core of pedestrian retrieval is how to find discriminative pedestrian expressions. Many recent methods use deep learning models to extract visual features. The method of this application can achieve fast feature extraction and reduce time overhead.
[0123] The above is the specific implementation of the neural network model reasoning method in this application. This application also provides a reasoning device based on a softmax neural network model, which can be used to implement the above reasoning method. Figure 8 Schematic diagram of the basic structure of the neural network model inference device in this application. Figure 8 As shown, the device includes: a first reasoning unit, a sparse processing unit, a softmax operator processing unit and a second reasoning unit.
[0124] The first reasoning unit is used to use the neural network model to reason about the input data of the neural network model to determine the input features of the softmax operator; wherein the reasoning of the neural network model includes the processing of the softmax operator;
[0125] A sparse processing unit, used for sparse processing the input features in units of rows; wherein the sparse processing of the input features in units of rows specifically includes: in each row of the input features, sorting the continuous M elements, and selecting the largest first N elements therein, retaining their values, and the number of elements with retained values in each row is the same;
[0126] A softmax operator processing unit, used to process the input features after the sparse processing using the softmax operator;
[0127] The second reasoning unit is used to complete the reasoning of the neural network model and determine the output of the neural network model.
[0128] The first reasoning unit and the second reasoning unit of the above-mentioned device can be located in an artificial intelligence chip, and the sparsification processing unit and the softmax operator processing unit can be located in a CPU or a VPU.
[0129] Optionally, the neural network model may be a CNN model, an RNN model, or a neural network model based on an attention mechanism.
[0130] Optionally, when the neural network is a neural network model based on an attention mechanism,
[0131] The device may further include an index information processing unit for marking index information of each element in the input feature after the sparse processing;
[0132] A softmax operator processing unit, after processing the input features after the sparse processing by using the softmax operator, is further used to determine the element position of the retained element in the input features after the sparse processing based on the index information of each retained element in the input features after the sparse processing, query the element at the same position as the determined element position in the matrix V, and retain the value of the queried element in the matrix V, and discard other elements in the matrix V;
[0133] The second reasoning unit is used to perform reasoning based on the updated matrix V when completing the reasoning of the neural network model.
[0134] Optionally, the device may further include a fine-tuning unit for pre-fine-tuning the linear layer weights of the module where the softmax operator is located in the neural network model, and the goal of the fine-tuning is to minimize the difference between the output features of the module before and after the sparsification processing.
[0135] Optionally, when the neural network model is a neural network model based on an attention mechanism, the module where the softmax operator is located is: an attention module.
[0136] The present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, the steps in the inference method for implementing a neural network model based on softmax as described above can be executed. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above-mentioned embodiments, or it can exist independently without being assembled into the device / apparatus / system. Among them, the instructions are stored in the computer-readable storage medium, and when the instructions stored therein are executed by a processor, the steps in the inference method for implementing a neural network model based on softmax as described above can be executed.
[0137] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.
[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A reasoning method based on a softmax neural network model, characterized in that: The neural network model is based on an attention mechanism and includes processing of a softmax operator, and the method includes: The artificial intelligence chip uses the neural network model to infer the input data of the neural network model until the input features of the softmax operator are determined; The central processing unit CPU or the vector processing unit VPU performs sparse processing on the input features in units of lines, and processes the input features after the sparse processing using a softmax operator; the CPU or VPU marks the index information of the retained elements in the input features after the sparse processing; The artificial intelligence chip determines the element position of the retained value in the softmax output feature based on the index information of the retained element in the input feature after the sparse processing, and searches for the element at the same position as the determined element position in the matrix V, retains the value of the element searched in the matrix V, and discards other elements in the matrix V; The artificial intelligence chip completes the reasoning of the neural network model based on the output result processed by the softmax operator and the updated matrix V; The step of performing sparse processing on the input features in units of lines includes: In each row of the input feature, M consecutive elements are sorted, and the largest first N elements are selected, and their values and indexes are retained, and the number of elements with retained values in each row is the same.
2. The method according to claim 1, characterized in that Before determining the input features of the softmax operator, the method further includes: pre-fine-tuning the linear layer weights of the module where the softmax operator is located in the neural network model, wherein the goal of the fine-tuning is to minimize the difference between the output features of the module before and after the sparse processing.
3. The method according to claim 1, characterized in that: The module where the softmax operator is located is: attention module.
4. An inference device based on a softmax neural network model, characterized in that: include: A first reasoning unit, a sparse processing unit, a softmax operator processing unit, an index information processing unit, and a second reasoning unit; The first reasoning unit is used to use the neural network model to reason on the input data of the neural network model until the input feature of the softmax operator is determined; wherein the neural network model is based on an attention mechanism and includes processing of the softmax operator; The sparse processing unit is used to perform sparse processing on the input feature in units of rows; wherein the sparse processing on the input feature in units of rows includes: in each row of the input feature, sorting M consecutive elements, selecting the largest first N elements therein, retaining their values, and the number of elements with retained values in each row is the same; The index information processing unit is used to mark the index information of each element in the input feature after the sparse processing; The softmax operator processing unit is used to process the input features after the sparse processing using the softmax operator; it is also used to determine the element position that retains the original value in the softmax output feature based on the index information of each element in the input features after the sparse processing, and query the element at the same position as the determined element position in the matrix V, retain the value of the element queried in the matrix V, and discard other elements in the matrix V; The second reasoning unit is used to complete the reasoning of the neural network model based on the output result after processing by the softmax operator and the updated matrix V, and determine the output of the neural network model.
5. The device according to claim 4, characterized in that The device further includes a fine-tuning unit for pre-fine-tuning the linear layer weights of the module where the softmax operator in the neural network model is located, and the goal of the fine-tuning is to minimize the difference between the output features of the module before and after the sparse processing.
6. The device according to claim 4, characterized in that The module where the softmax operator is located is: attention module.
7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the inference method of the softmax-based neural network model described in any one of claims 1 to 3 can be implemented.
Citation Information
Patent Citations
Dynamic sparse method, intelligent analysis method, device, equipment and storage medium
CN118247515A
