A neural network inference acceleration method based on shallow feature pre-computation
By constructing shallow feature lookup table and replacing the inference calculation of shallow neural networks, the problem of high computational cost and time overhead in natural language processing tasks is solved, and the inference acceleration and accuracy maintenance is achieved.
Patent Information
- Application Number
- CN202210223992.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-09
- Publication Date
- 2025-04-01
- Estimated Expiration
- 2042-03-09
AI Technical Summary
The calculation cost and time overhead of pre-trained neural networks in natural language processing tasks is high, resulting in difficulty in inference operations and a large amount of resources.
The neural network inference acceleration method based on shallow feature pre-calculation is adopted. By constructing a shallow feature lookup table, replacing the inference calculation of the shallow neural network, reducing the operation overhead.
It significantly reduces the computational cost and time overhead of neural network inference, and maintains the accuracy of the original pre-trained neural network on multiple tasks.
Smart Images
Figure CN114580352B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology under the computer field, and in particular to a neural network reasoning acceleration method for the field of natural language processing technology. Background Art
[0002] With the rapid development of artificial intelligence technology in recent years, especially the emergence of extremely large-scale pre-trained neural networks in the field of natural language processing technology, the accuracy of many downstream tasks of natural language processing has been significantly improved, especially in the tasks of understanding and reasoning of human natural language, and in some tasks, it has even achieved a level close to or even exceeding that of humans. However, the application of pre-trained neural networks also brings about a rapid increase in model size and computational complexity. Even with the most advanced hardware computer equipment, it is very difficult to use these ultra-large-scale pre-trained neural networks for reasoning operations, and it requires a lot of computing costs and time. Summary of the invention
[0003] The purpose of the present invention is to provide a neural network reasoning acceleration method based on shallow feature pre-computation to address the deficiencies of the prior art. The present invention can effectively reduce the computational cost and time overhead of pre-trained neural networks, and maintain performance close to that of the original pre-trained neural networks in multiple artificial intelligence tasks such as news classification, sentiment analysis, and question-answering systems.
[0004] The object of the present invention is to achieve the following technical solution: a neural network reasoning acceleration method based on shallow feature pre-computation, comprising the following steps:
[0005] (1) obtaining a pre-trained neural network, and encoding the input text using the pre-trained neural network according to the input text;
[0006] (2) constructing a shallow feature precomputable neural network based on the pretrained neural network, and encoding the input text using the shallow feature precomputable neural network based on the input text;
[0007] (3) using the shallow feature pre-calculated neural network to fit the intermediate layer features of the pre-trained neural network, and training the shallow feature pre-calculated neural network;
[0008] (4) A neural network can be pre-calculated based on the shallow features to construct a shallow feature query table;
[0009] (5) Using the above shallow feature query table to replace the shallow feature can precompute the shallow neural network of the neural network.
[0010] Further, in the step (1), the pre-trained neural network is composed of L (L>1) layers of Transformer, CNN, LSTM or GRU neural networks; the l-th (0≤l<L) layer neural network of the pre-trained neural network is denoted as Layer (l) ; Given the input text X = [x1, x2, …, x n , where x i (i = 1, …, n) represents the words in the input text, and the input text contains n words. Use the pre-trained neural network to encode the input text and obtain the encoded text sequence features.
[0011] Further, the process of using the pre-trained neural network to encode the input text includes the following steps:
[0012] (1.1) Map the words in the input text to word vectors to obtain the initial text sequence features where correspond to the word vectors of the words x1, x2, …, x n respectively;
[0013] (1.2) Encode the text sequence features U (l) using the l-th (0≤l<L) layer neural network of the pre-trained neural network to obtain the encoded text sequence features U (l+1) = Layer (l) (U (l) );
[0014] (1.3) Repeat step (1.2) until the encoded text sequence features using the last layer neural network are obtained
[0015] Further, in the step (2), according to the pre-trained neural network, use a neural network of the same type and the same number of layers as the pre-trained neural network to construct a shallow feature pre-computable neural network; the first L loc (1≤L loc <L) layer neural networks of the shallow feature pre-computable neural network are denoted as the shallow neural networks of the shallow feature pre-computable neural network; use the above shallow feature pre-computable neural network to encode the input text.
[0016] Further, the specific process of using the above shallow feature pre-computable neural network to encode the input text includes the following steps:
[0017] (2.1) Given the input text X = [x1, x2, …, x n , divide the input text into multiple text regions X i = [x i-1 , xi , x i+1 ](i=2,…,n-1);
[0018] (2.2) Map the words contained in the above text area into word vectors to obtain the initial text area features
[0019] in Corresponding to x i-1 , x i , x i+1 The word vector of
[0020] (2.3) Using shallow features, we can pre-calculate the lth (0≤l<L) loc ) layer of shallow neural network for text region features Encode and obtain the encoded shallow text area features
[0021] (2.4) Repeat step (2.3) until the shallow text region features encoded by the last layer of shallow neural network are obtained.
[0022] (2.5) The shallow text region features after the above encoding Perform aggregation to obtain shallow text sequence features The text features of a single word are Where Norm is the layer normalization function; Gate is a gating mechanism based on the sigmoid function, and the specific calculation method is Gate(x)=sigmoid(v G ·x+b G ), v G and b G is the trainable network parameter; for the out-of-bounds case, i.e., i+j≤0 or i+j>n, we will Set it to zero vector, that is
[0023] (2.6) Using shallow features, we can pre-calculate the lth (L) loc ≤l<L) layers of neural network for text sequence features H (l) Encode and obtain the encoded text sequence feature H( l+1 )=Layer( l )(H (l) );
[0024] (2.7) Repeat step (2.6) until the text sequence feature H encoded by the last layer of neural network is obtained. (L) .
[0025] Furthermore, in step (3), given the input text X = [x1, x2, ..., x n ], using the above shallow features, the neural network can be pre-calculated to fit the intermediate layer features of the pre-trained neural network; specifically, the following steps are included:
[0026] (3.1) For a given input text, use a pre-trained neural network to encode the input text and obtain the text sequence features of the intermediate layer of the input text
[0027] (3.2) For a given input text, shallow features can be used to pre-calculate the neural network encoding input text and obtain the text sequence features of the intermediate layer of the input text.
[0028] (3.3) Based on the above pre-trained neural network and shallow features, the text sequence features of the intermediate layer of the neural network can be pre-calculated, and the loss value can be calculated:
[0029]
[0030] (3.4) Use SGD, Adam and other gradient descent-based optimization algorithms to minimize the above loss values, so as to train shallow features and pre-compute neural networks.
[0031] Furthermore, in the step (4), a neural network can be pre-calculated according to the shallow features to construct a shallow feature query table, which specifically includes the following steps:
[0032] (4.1) constructing a shallow feature query table based on key-value query; the shallow feature query table is used to store a large number of key-value pairs; given a key, the shallow feature query table can return the value corresponding to the given key;
[0033] (4.2) Enumerate text area X i =[x i-1 , x i , x i+1 ];
[0034] (4.3) Using shallow features to precompute the shallow neural network of the neural network to calculate the shallow text region features of the above text region
[0035] (4.4) The above text area X i As key, shallow text region features As the value, construct a key-value pair and store the key-value pair in the shallow feature query table; given a text area X i As a key, the shallow feature query table can return the value corresponding to the given key, that is, the shallow text region feature
[0036] (4.5) Repeat steps (4.2)-(4.4) until the shallow feature query table stores enough key-value pairs.
[0037] Furthermore, in step (5), after the shallow neural network of the shallow feature pre-calculated neural network is replaced by the shallow feature query table, the specific steps of the reasoning process of the shallow feature pre-calculated neural network are as follows:
[0038] (5.1) Given an input text X = [x1, x2, ..., x n ], split the input text into multiple text regions X containing three words i =[x i-1 , x i , x i+1 ], (i=2, ..., n-1);
[0039] (5.2) Using the above text region as a key, retrieve the shallow text region feature corresponding to the given input text region from the shallow feature query table
[0040] (5.3) The above shallow text region features Perform aggregation to obtain shallow text sequence features:
[0041]
[0042] The specific aggregation process is defined as follows: Among them, Norm is the layer normalization function; Gate is a gating mechanism based on the sigmoid function, and the specific calculation method is Gate(x)=sigmoid(v G ·x+b G ), v G and b G is a trainable network parameter; for out-of-bounds situations, i.e., i+j≤0 or i+j>n, is set to the zero vector, i.e.
[0043] (5.4) Using shallow features, we can pre-calculate the lth (L) loc ≤l<L) layers of neural network for text sequence features H (l) Encode and obtain the encoded text sequence feature H (l+1) =Layer (l) (H (l) );
[0044] (5.5) Repeat step (5.4) until the text sequence feature H encoded by the last layer of neural network is obtained. (L) .
[0045] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a neural network reasoning acceleration method based on shallow feature pre-computation, which avoids directly using shallow neural networks for reasoning calculations by constructing shallow feature query tables, thereby significantly reducing reasoning overhead. Experiments show that the neural network reasoning acceleration method can significantly improve the reasoning speed and effectively maintain the accuracy of the original neural network on multiple tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flowchart of the neural network construction and training of the present invention;
[0047] Figure 2 It is a schematic diagram of the reasoning process of a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.
[0049] The neural network reasoning acceleration method based on shallow feature pre-computation adopted by the present invention is as follows Figure 1 As shown, the key steps include:
[0050] (1) obtaining a pre-trained neural network, and encoding the input text using the pre-trained neural network according to the input text;
[0051] (2) constructing a shallow feature precomputable neural network based on the pretrained neural network, and encoding the input text using the shallow feature precomputable neural network based on the input text;
[0052] (3) using the shallow feature pre-calculated neural network to fit the intermediate layer features of the pre-trained neural network, and training the shallow feature pre-calculated neural network;
[0053] (4) A neural network can be pre-calculated based on the shallow features to construct a shallow feature query table;
[0054] (5) Using the above shallow feature query table to replace the shallow feature can precompute the shallow neural network of the neural network.
[0055] In step (1), the pre-trained neural network can be obtained from a public network, such as BERT, RoBERTa and other models, or it can be pre-trained using private training data. The pre-trained neural network consists of L (L>1) layers of Transformer, CNN, LSTM or GRU neural networks. The lth (0≤l<L) layer of the pre-trained neural network is denoted as Layer (l) . This embodiment uses the BERT-Base model trained by Google as a pre-trained neural network, which is composed of a 12-layer Transformer neural network. Each layer of the Transformer neural network contains a multi-head attention network and a feedforward neural network, wherein the number of attention heads of the multi-head attention network is 12, the number of dimensions of the multi-head attention network is 768, and the number of dimensions of the feedforward neural network is 768.
[0056] Given an input text X = [x1, x2, ..., x n ], where x i (i=1, ..., n) represents the words in the input text. The input text is encoded using a pre-trained neural network, and the encoded text sequence features are obtained. The encoded text sequence features can be used for artificial intelligence tasks such as news classification, sentiment analysis, and question-answering systems. The process of encoding the input text using a pre-trained neural network includes the following steps.
[0057] a. Map the words in the input text into word vectors to obtain the initial text sequence features in Corresponding to words x1, x2, ..., x n The word vector of , the dimension of the word vector is 768.
[0058] b. Use the lth (0≤l<L) layer of the pre-trained neural network to train the text sequence feature U (l) Encode and obtain the encoded text sequence feature U (l+1) =Layer (l) (U (l) ).
[0059] c. Repeat step b until the text sequence features encoded using the last layer of neural network are obtained
[0060] In the step (2), based on the pre-trained neural network, a neural network of the same type and number of layers as the pre-trained neural network is used to construct a shallow feature pre-calculated neural network. loc (1≤L locThe present embodiment adopts a 12-layer Transformer neural network, in which the first 6 neural network layers are recorded as shallow neural networks of which shallow features can be pre-computed.
[0061] Using the above shallow features, the neural network can be pre-calculated to encode the input text, which includes the following steps:
[0062] a. Given an input text X = [x1, x2, ..., x n ], split the input text into multiple text regions X containing three words i =[x i-1 , x i , x i+1 ](i=2,…,n-1).
[0063] b. Map the words contained in the above text area into word vectors to obtain the initial text area features in Corresponding to x i-1 , x i , x i+1 The word vector of , the dimension of the word vector is 768.
[0064] c. Using shallow features can pre-calculate the lth (0≤l<L loc ) layer of shallow neural network for text region features Encode and obtain the encoded shallow text area features
[0065] d. Repeat step c until the shallow text region features encoded by the last shallow neural network layer are obtained
[0066] e. The shallow text area features after the above encoding Perform aggregation to obtain shallow text sequence features The text features of a single word are
[0067] Norm is the layer normalization function, which is calculated as follows:
[0068]
[0069] Among them, E[x] represents the mean of x, Var[x] represents the variance of x, and γ and β are learnable parameters.
[0070] Gate is a gating mechanism based on the sigmoid function. The specific calculation method is as follows:
[0071] Gate(x) = sigmoid(v G ·x+b G )
[0072] Among them, vG and bG are trainable network parameters. Where e is the natural base.
[0073] For the out-of-bounds case, that is, when i+j≤0 or i+j>n, we let Set it to zero vector, that is
[0074] f. Using shallow features can pre-calculate the lth (L) loc ≤l<L) layers of neural network for text sequence features H (l) Encode and obtain the encoded text sequence feature H (l+1) =Layer (l) (H (l) ).
[0075] g. Repeat step f until the text sequence feature H encoded by the last layer of neural network is obtained (L) .
[0076] In step (3), given an input text X = [x1, x2, ..., x n ], use the above shallow feature precomputable neural network to fit the intermediate layer features of the pretrained neural network, so as to train the shallow feature precomputable neural network. Through the above training process, the shallow feature precomputable neural network can learn the knowledge contained in the pretrained neural network, so that when processing tasks in the field of artificial intelligence such as news classification, sentiment analysis, question-answering systems, etc., the shallow feature precomputable neural network can achieve an accuracy comparable to that of the pretrained neural network. The above training specifically includes the following steps:
[0077] a. For a given input text, use a pre-trained neural network to encode the input text and obtain the text sequence features of the intermediate layer of the input text
[0078] b. For a given input text, shallow features can be used to pre-calculate the neural network encoding input text and obtain the text sequence features of the intermediate layer of the input text
[0079] c. Based on the above pre-trained neural network and shallow features, the text sequence features of the middle layer of the neural network can be pre-calculated to calculate the loss value:
[0080]
[0081] d. Use SGD, Adam and other gradient descent-based optimization algorithms to minimize the above loss values, so as to train shallow features and pre-compute neural networks.
[0082] In step (4), a shallow feature query table is constructed based on the shallow feature precalculated neural network. The shallow feature query table is used to replace the shallow neural network of the shallow feature precalculated neural network, and the output of the shallow neural network is obtained by table lookup, thereby greatly accelerating the reasoning speed. Constructing the shallow feature query table specifically includes the following steps:
[0083] a. Build a shallow feature query table based on key-value query;
[0084] The shallow feature query table is used to store a large number of key-value pairs;
[0085] Given a key, this shallow feature query table can return the value corresponding to the given key.
[0086] b. Enumerate text area X i =[x i-1 , x i , x i+1 ].
[0087] c. Use shallow features to precompute the shallow neural network of the neural network to calculate the shallow text region features of the above text region
[0088] d. Change the text area X i As key, shallow text region features As the value, a key-value pair is constructed and the key-value pair is stored in the shallow feature query table.
[0089] Given a text area X i As a key, the shallow feature query table can return the value corresponding to the given key, that is, the shallow text region feature
[0090] e. Repeat steps bd until the shallow feature query table stores enough key-value pairs.
[0091] In the step (5), if Figure 2 As shown, after the shallow neural network of the shallow feature precomputable neural network is replaced by the shallow feature query table, the specific steps of the reasoning process of the shallow feature precomputable neural network are as follows:
[0092] a. Given an input text X = [x1, x2, ..., x n ], split the input text into multiple text regions X containing three words i =[xi-1 , x i , x i+1 ], (i=2,…,n-1).
[0093] b. Using the above text region as a key, retrieve the shallow text region features corresponding to the given input text region from the shallow feature query table
[0094] c. The above shallow text area features Perform aggregation to obtain shallow text sequence features The specific aggregation process is defined as follows:
[0095]
[0096] Among them, Norm is the layer normalization function; Gate is a gating mechanism based on the sigmoid function, and the specific calculation method is Gate(x)=sigmoid(v G ·x+b G ), v G and b G are trainable network parameters.
[0097] For out-of-bounds situations, that is, when i+j≤0 or i+j>n, is set to the zero vector, i.e.
[0098] d. Using shallow features can pre-calculate the lth (L loc ≤l<L) layers of neural network for text sequence features H (l) Encode and obtain the encoded text sequence feature H (l+1) =Layer (l) (H (l) ).
[0099] e. Repeat step d until the text sequence feature H encoded by the last layer of neural network is obtained (L) .
[0100] Based on the same inventive concept, another embodiment of the present invention provides a computer / server, which includes a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.
[0101] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.
[0102] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the principle and scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. A neural network reasoning acceleration method based on shallow feature pre-computation, characterized in that: The following steps are involved: (1) obtaining a pre-trained neural network, and encoding the input text using the pre-trained neural network according to the input text; (2) constructing a shallow feature precomputable neural network based on the pretrained neural network, and encoding the input text using the shallow feature precomputable neural network based on the input text; (3) using the shallow feature pre-calculated neural network to fit the intermediate layer features of the pre-trained neural network, and training the shallow feature pre-calculated neural network; (4) A neural network can be pre-calculated based on the shallow features to construct a shallow feature query table; (5) Using the above shallow feature query table to replace the shallow feature can precompute the shallow neural network of the neural network.
2. The neural network reasoning acceleration method based on shallow feature pre-computation according to claim 1, characterized in that: In the step (1), the pre-trained neural network is composed of L (L>1) layers of Transformer, CNN, LSTM or GRU neural network; the first The layer neural network is denoted as Given an input text X = [x1, x2, ..., x n ], where x i (i=1, ..., n) represents a word in an input text. The input text contains n words. A pre-trained neural network is used to encode the input text and obtain the encoded text sequence features.
3. The neural network reasoning acceleration method based on shallow feature pre-computation according to claim 1, characterized in that: The process of encoding the input text using a pre-trained neural network comprises the following steps: (1.1) Map the words in the input text into word vectors to obtain the initial text sequence features in Corresponding to words x1, x2, ..., x n The word vector of (1.2) Using the pre-trained neural network Layer neural network for text sequence features Encode and obtain the encoded text sequence features (1.3) Repeat step (1.2) until the text sequence features encoded by the last layer of neural network are obtained.
4. The neural network reasoning acceleration method based on shallow feature pre-computation according to claim 1, characterized in that: In the step (2), according to the pre-trained neural network, a neural network of the same type and number of layers as the pre-trained neural network is used to construct a shallow feature pre-calculated neural network; the shallow feature pre-calculated neural network has a front L loc (1≤L loc <L) layer neural network is recorded as a shallow neural network of a shallow feature precomputable neural network; the input text is encoded using the above shallow feature precomputable neural network.
5. The neural network reasoning acceleration method based on shallow feature pre-computation according to claim 4 is characterized in that: The use of the shallow features to pre-calculate the neural network to encode the input text specifically includes the following steps: (2.1) Given an input text X = [x1, x2, ..., x n ], split the input text into multiple text regions X containing three words i =[x i-1 , x i , x i+1 ](i=2,…,n-1); (2.2) Map the words contained in the above text area into word vectors to obtain the initial text area features in Corresponding to x i-1 , x i , x i+1 The word vector of (2.3) Using shallow features can pre-calculate the first Text region features using shallow neural networks Encode and obtain the encoded shallow text area features (2.4) Repeat step (2.3) until the shallow text region features encoded by the last layer of shallow neural network are obtained. (2.5) The shallow text region features after the above encoding Perform aggregation to obtain shallow text sequence features The text features of a single word are Where Norm is the layer normalization function; Gate is a gating mechanism based on the sigmoid function, and the specific calculation method is Gate(x)=sigmoid(v G ·x+b G ), v G and b G is the trainable network parameter; for the out-of-bounds case, i.e., i+j≤0 or i+j>n, we will Set it to zero vector, that is (2.6) Using shallow features can pre-calculate the first Layer neural network for text sequence features Encode and obtain the encoded text sequence features (2.7) Repeat step (2.6) until the text sequence feature H encoded by the last layer of neural network is obtained. (L) .
6. The neural network reasoning acceleration method based on shallow feature pre-computation according to claim 1, characterized in that: In step (3), given an input text X = [x1, x2, ..., x n ], using the above shallow features, the neural network can be pre-calculated to fit the intermediate layer features of the pre-trained neural network; The specific steps include: (3.1) For a given input text, use a pre-trained neural network to encode the input text and obtain the text sequence features of the intermediate layer of the input text (3.2) For a given input text, shallow features can be used to pre-calculate the neural network encoding input text and obtain the text sequence features of the intermediate layer of the input text. (3.3) Based on the above pre-trained neural network and shallow features, the text sequence features of the intermediate layer of the neural network can be pre-calculated, and the loss value can be calculated: (3.4) Use SGD, Adam and other gradient descent-based optimization algorithms to minimize the above loss values, so as to train shallow features and pre-compute neural networks.
7. The neural network reasoning acceleration method based on shallow feature pre-computation according to claim 1, characterized in that: In the step (4), a neural network can be pre-calculated based on the shallow features to construct a shallow feature query table, which specifically includes the following steps: (4.1) constructing a shallow feature query table based on key-value query; the shallow feature query table is used to store a large number of key-value pairs; given a key, the shallow feature query table can return the value corresponding to the given key; (4.2) Enumerate text area X i =[x i-1 , x i , x i+1 ]; (4.3) Using shallow features to precompute the shallow neural network of the neural network to calculate the shallow text region features of the above text region (4.4) The above text area X i As key, shallow text region features As the value, construct a key-value pair, and store the key-value pair in the shallow feature query table; Given a text area X i As a key, the shallow feature query table can return the value corresponding to the given key, that is, the shallow text region feature (4.5) Repeat steps (4.2)-(4.4) until the shallow feature query table stores enough key-value pairs.
8. The neural network reasoning acceleration method based on shallow feature pre-computation according to claim 1, characterized in that: In the step (5), after the shallow neural network of the shallow feature pre-calculated neural network is replaced by the shallow feature query table, the specific steps of the reasoning process of the shallow feature pre-calculated neural network are as follows: (5.1) Given an input text X = [x1, x2, ..., x n ], split the input text into multiple text regions X containing three words i =[x i-1 , x i , x i+1 ], (i=2, ..., n-1); (5.2) Using the above text region as a key, retrieve the shallow text region feature corresponding to the given input text region from the shallow feature query table (5.3) The above shallow text region features Perform aggregation to obtain shallow text sequence features: The specific aggregation process is defined as follows: Among them, Norm is the layer normalization function; Gate is a gating mechanism based on the sigmoid function, and the specific calculation method is Gate(x)=sigmoid(v G ·x+b G ), v G and b G is a trainable network parameter; for out-of-bounds situations, i.e., i+j≤0 or i+j>n, is set to the zero vector, i.e. (5.4) Using shallow features, we can pre-calculate the first Layer neural network for text sequence features Encode and obtain the encoded text sequence features (5.5) Repeat step (5.4) until the text sequence feature H encoded by the last layer of neural network is obtained. (L) .