Model reasoning acceleration method, device, computer equipment and storage medium
By judging and extracting the word vector sequence from the attention layer output in the neural network model, the problem of large amount of model calculation is solved, and the deployment and cost reduction in low-computing terminals are achieved.
Patent Information
- Application Number
- CN202011328165.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-24
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2040-11-24
AI Technical Summary
The existing neural network models are computationally expensive and cannot be effectively deployed on mobile terminals or reduce the computing cost of mobile terminals.
By judging whether the word vector sequence output by the self-attention layer exceeds the preset threshold, if it exceeds, some word vectors will be extracted from it to form a new word vector sequence and input it into the feedforward network layer to reduce the calculation amount.
It effectively reduces the computing volume of the feedforward network layer, allowing the neural network model to be deployed in terminals with low computing capabilities, and reducing the deployment cost.
Smart Images
Figure CN112434525B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model reasoning acceleration method, device, computer equipment and storage medium. Background Art
[0002] Model deployment is the "last mile" from AI application to production. In the field of natural language processing, BERT (Bidirectional Encoder Representation from Transformers) is widely popular as an effective method, and the improvement in accuracy it brings is unmatched by other small models. Engineers will give priority to using this large model when designing models. However, this model requires a lot of calculations. In actual production applications, BERT is often limited by computing power and cannot be well deployed on mobile terminals or reduce the computing costs of mobile terminals. Summary of the invention
[0003] The embodiments of the present invention provide a model reasoning acceleration method, device, computer equipment and storage medium, which aim to solve the problem that the existing neural network model has a large amount of calculation and cannot be well deployed on the mobile terminal or cannot reduce the calculation cost of the mobile terminal.
[0004] In a first aspect, an embodiment of the present invention provides a model reasoning acceleration method, wherein the neural network model includes an encoding layer, the encoding layer includes a self-attention layer and a feedforward network layer, and the model reasoning acceleration method includes:
[0005] Determine whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold;
[0006] If the number of word vectors contained in the first word vector sequence output by the self-attention layer is greater than a preset number threshold, extract some word vectors from the first word vector sequence to obtain a second word vector sequence;
[0007] The second word vector sequence is input into the feed-forward network layer of the encoding layer.
[0008] In a second aspect, an embodiment of the present invention further provides a model reasoning acceleration device, wherein the neural network model includes a coding layer, the coding layer includes a self-attention layer and a feedforward network layer, and the model reasoning acceleration device includes:
[0009] A judging unit, used to judge whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold;
[0010] An extraction unit, configured to extract part of the word vectors from the first word vector sequence output from the attention layer to obtain a second word vector sequence if the number of word vectors contained in the first word vector sequence output from the attention layer is greater than a preset number threshold;
[0011] The first input unit inputs the first word vector sequence into the feedforward network layer of the encoding layer.
[0012] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.
[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.
[0014] The embodiment of the present invention provides a model reasoning acceleration method, device, computer equipment and storage medium. The method includes: determining whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold; if the number of word vectors contained in the first word vector sequence output by the self-attention layer is greater than the preset number threshold, extracting part of the word vectors from the first word vector sequence to obtain a second word vector sequence; inputting the second word vector sequence into the feedforward network layer of the encoding layer. Since the number of word vectors in the second word vector sequence is less than the number of word vectors in the first word vector sequence, the amount of calculation in the feedforward network layer can be effectively reduced, so that the neural network model can be deployed in a terminal with low computing power, effectively reducing the deployment cost of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0016] Figure 1 A schematic diagram of a flow chart of a model reasoning acceleration method provided by an embodiment of the present invention;
[0017] Figure 2 A schematic diagram of a sub-process of a model reasoning acceleration method provided by an embodiment of the present invention;
[0018] Figure 3 A schematic diagram of a flow chart of a model reasoning acceleration method provided by another embodiment of the present invention;
[0019] Figure 4 A schematic block diagram of a model reasoning acceleration device provided by an embodiment of the present invention;
[0020] Figure 5 A schematic block diagram of a model reasoning acceleration device provided by another embodiment of the present invention;
[0021] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0024] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0025] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0026] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0027] See also Figure 1 , Figure 1It is a flow chart of the model reasoning acceleration method provided by an embodiment of the present invention. The present invention can be applied to scenarios such as smart government affairs / smart urban management / smart community / smart security / smart logistics / smart medical care / smart education / smart environmental protection / smart transportation that require the use of neural network models, thereby promoting the construction of smart cities. The model reasoning acceleration method proposed in an embodiment of the present invention can reduce the amount of calculation of the neural network model. The method is universal to models that use a self-attention mechanism during encoding. This type of neural network model includes a coding layer, which includes a self-attention layer and a feedforward network layer. A typical example of this type of model is the BERT model. As shown in the figure, the method includes the following steps S1-S3.
[0028] S1, determine whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold.
[0029] In a specific implementation, after receiving the first word vector sequence output by the self-attention layer of the encoding layer, it is determined whether the number of word vectors contained in the first word vector sequence is greater than a preset number threshold.
[0030] It should be noted that the quantity threshold can be set by those skilled in the art based on experience, and the present invention does not specifically limit this. For example, in one embodiment, the quantity threshold is set to 10. If the number of word vectors included in the first word vector sequence is 12, it is determined that the number of word vectors included in the first word vector sequence is greater than the preset quantity threshold.
[0031] According to the common practice in the prior art, the first word vector sequence output by the self-attention layer is usually directly input into the feedforward network layer. If the first word vector sequence contains too many word vectors, the amount of calculation in the feedforward network layer will be very large (the amount of calculation of the neural network model is mainly concentrated in the feedforward network layer. The more word vectors there are in the word vector sequence, the higher the amount of calculation in the feedforward network layer). The present invention is aimed at the above-mentioned problem of large amount of calculation, and determines whether the number of word vectors contained in the first word vector sequence is greater than a preset number threshold. If so, the following step S2 is executed to reduce the amount of calculation by reducing the number of word vectors.
[0032] S2: If the number of word vectors contained in the first word vector sequence output from the self-attention layer is greater than a preset number threshold, some word vectors are extracted from the first word vector sequence to obtain a second word vector sequence.
[0033] In a specific implementation, if the number of word vectors contained in the first word vector sequence output from the attention layer is greater than a preset threshold value, the present invention does not directly input the first word vector sequence into the feedforward network layer, but extracts part of the word vectors from the first word vector sequence to form a second word vector sequence.
[0034] The number of word vectors extracted from the first word vector sequence is less than or equal to a quantity threshold. For example, in one embodiment, the quantity threshold is set to 10. In this embodiment, the number of word vectors extracted from the first word vector sequence is less than or equal to 10.
[0035] It should be noted that the number of word vectors extracted from the first word vector sequence can be determined by those skilled in the art according to specific calculation accuracy requirements, and the present invention does not specifically limit this. The more word vectors there are, the higher the calculation accuracy.
[0036] In one embodiment, the above step S2 specifically includes: extracting a preset number of word vectors from the first word vector sequence in sequence to obtain a second word vector sequence.
[0037] It should be noted that the preset number is less than or equal to the number threshold, which can be determined by those skilled in the art according to the specific calculation accuracy requirements, and the present invention does not specifically limit this. For example, in one embodiment, the preset number is set to 10. When the preset number is set to 10, the above step S2 specifically includes: extracting 10 word vectors from the first word vector sequence in order to obtain a second word vector sequence.
[0038] In order to better illustrate the technical solution of this embodiment, a specific example is provided below. In this example, the quantity threshold is set to 10, the preset number is 10, and the first word vector sequence is (w1, w2, w3, w4, w5, w6, w7, w8, w9, w10, w11, w12).
[0039] The first word vector sequence includes a total of 12 word vectors, which exceeds the quantity threshold of 10. Therefore, the first word vector sequence obtained by extracting 10 word vectors in sequence from the first word vector sequence is (w1, w2, w3, w4, w5, w6, w7, w8, w9, w10).
[0040] In one embodiment, the above step S2 specifically includes: randomly extracting a preset number of word vectors from the first word vector sequence to obtain a second word vector sequence.
[0041] It should be noted that the preset number is less than or equal to the number threshold, which can be determined by those skilled in the art according to the specific calculation accuracy requirements, and the present invention does not specifically limit this. For example, in one embodiment, the preset number is set to 10. When the preset number is set to 10, the above step S2 specifically includes: randomly extracting 10 word vectors from the first word vector sequence to obtain a second word vector sequence.
[0042] In order to better illustrate the technical solution of this embodiment, a specific example is provided below. In this example, the quantity threshold is set to 10, the preset number is 10, and the first word vector sequence is (w1, w2, w3, w4, w5, w6, w7, w8, w9, w10, w11, w12).
[0043] The first word vector sequence includes a total of 12 word vectors, which exceeds the quantity threshold of 10. In this embodiment, the first word vector sequence obtained by randomly extracting 10 word vectors from the first word vector sequence is (w1, w3, w4, w5, w6, w7, w8, w9, w11, w12).
[0044] In one embodiment, the self-attention layer includes multiple attention heads, and for each word vector in the first word vector sequence, each of the attention heads calculates an importance score for it. It should be noted that each attention head in the self-attention layer will calculate an importance score for the word vector in the first word vector sequence, and the importance score is used to characterize the importance of the word vector in the word vector sequence. In the prior art, the importance score is usually used as a parameter of the feedforward network layer. In an embodiment of the present invention, the importance score is also used to screen word vectors, which is specifically described as follows:
[0045] See also Figure 2 In this embodiment, the above step S2 specifically includes the following steps:
[0046] S21, for each word vector in the first word vector sequence, its overall importance score is determined according to the importance score calculated for it by each attention head.
[0047] In the specific implementation, the formula Sig(w)=∑ h Sig h (w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h (w) is the importance score calculated by the attention head h for the word vector w.
[0048] The above method directly accumulates the importance scores calculated by each attention head for the word vector to obtain the overall importance score.
[0049] Furthermore, the weight value of each attention head can be set, and the importance score calculated by each attention head for the word vector is weighted and summed to obtain the overall importance score.
[0050] Specifically, by the formula sig(w)=∑ h K h sig h(w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h (w) is the importance score calculated by the attention head h for the word vector w, K h is the weight of the attention head h.
[0051] It should be noted that the above-mentioned method for calculating the overall importance score is only a specific example provided by the present invention. Those skilled in the art may adopt other methods for calculating the overall importance score, which will not exceed the protection scope of the present invention.
[0052] S22, extracting a preset number of word vectors from the first word vector sequence in descending order of overall importance scores to obtain a second word vector sequence.
[0053] In the specific implementation, it should be noted that the preset number is less than or equal to the number threshold, which can be determined by those skilled in the art according to the specific calculation accuracy requirements, and the present invention does not specifically limit this. For example, in one embodiment, the preset number is set to 10. When the preset number is set to 10, the above step S22 is specifically: extract 10 word vectors from the first word vector sequence in order of overall importance scores from high to low to obtain a second word vector sequence.
[0054] In order to better illustrate the technical solution of this embodiment, a specific example is provided below. In this example, the quantity threshold is set to 10, the preset quantity is 10, and the first word vector sequence is (w1, w2, w3, w4, w5, w6, w7, w8, w9, w10, w11, w12). The word vectors in the first word vector sequence are sorted in descending order according to the overall importance score: w10, w12, w3, w5, w8, w6, w7, w9, w1, w4, w11, w2.
[0055] Then, the 10 word vectors extracted from the first word vector sequence in the order of overall importance scores from high to low are w10, w12, w3, w5, w8, w6, w7, w9, w1, w4. After that, the 10 word vectors extracted above are sorted according to the order in the first word vector sequence to obtain the second word vector sequence (w1, w3, w4, w5, w6, w7, w8, w9, w10, w12).
[0056] S3: input the second word vector sequence into the feedforward network layer of the encoding layer.
[0057] In a specific implementation, since the number of word vectors in the second word vector sequence is less than the number of word vectors in the first word vector sequence, the amount of computation in the feedforward network layer can be effectively reduced.
[0058] It can be understood that if the neural network model includes multiple coding layers, the model reasoning acceleration method proposed in the present invention can be used in each coding layer of the neural network model. For example, the BERT model includes 12 coding layers, and the model reasoning acceleration method proposed in the present invention can be used in the 12 coding layers.
[0059] The technical solution of the present invention determines whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold; if the number of word vectors contained in the first word vector sequence output by the self-attention layer is greater than the preset number threshold, extract part of the word vectors from the first word vector sequence to obtain a second word vector sequence; and input the second word vector sequence into the feedforward network layer of the encoding layer. Since the number of word vectors in the second word vector sequence is less than the number of word vectors in the first word vector sequence, the amount of calculation in the feedforward network layer can be effectively reduced, so that the neural network model can be deployed in a terminal with low computing power, effectively reducing the deployment cost of the neural network model.
[0060] Figure 3 FIG. 1 is a flow chart of a model reasoning acceleration method provided by another embodiment of the present invention. Figure 3 As shown, the model reasoning acceleration method of this embodiment includes steps S31-S38. Generally speaking, a neural network model usually includes multiple coding layers, and each coding layer is connected in sequence. In the technical solution of this example, steps S31-S34 need to be executed for the first encoder of the neural network model. For other encoders of the neural network model (encoders other than the first encoder), there is no need to execute steps S31-S34, and steps S35-S38 can be directly executed.
[0061] S31, performing word segmentation processing on the input corpus to obtain a first word segmentation sequence.
[0062] In a specific implementation, for the first encoding layer of the neural network model, its input word vector sequence is converted from the input corpus. When the input corpus is received, the input corpus is first segmented to obtain a first segmentation sequence, for example, by a preset segmentation tool (for example, jieba segmentation tool) to obtain the first segmentation sequence.
[0063] Word segmentation refers to dividing the input corpus into a sequence of multiple words. For example, in one embodiment, the input corpus is: Spring flowers are so beautiful! The first word segmentation sequence obtained after word segmentation is: Spring / of / flowers / are / beautiful / ah.
[0064] S32: Delete the stop words in the first word segmentation sequence to obtain a second word segmentation sequence.
[0065] In specific implementation, stopwords are often prepositions, adverbs, conjunctions, etc. For example, "in", "inside", "also", "of", "it", "for", etc. are all stopwords. Stopwords themselves have no actual meaning. Therefore, in the technical solution of this embodiment, the stopwords in the first word segmentation sequence are deleted, so that the word vectors in the input word vector sequence are reduced, thereby further reducing the amount of calculation.
[0066] For example, in one embodiment, the first word segmentation sequence is: spring / of / flower / so / beautiful / ah. Among them, "of" and "ah" are stopwords. Therefore, the second word segmentation sequence obtained after deleting the stopwords in the first word segmentation sequence is: spring / flower / so / beautiful.
[0067] S33. Perform word vector training on the words in the second word segmentation sequence to obtain an input word vector sequence.
[0068] In specific implementation, word2vec is used as a word vector training tool, and word2vec is used to perform word vector training on the words in the second word segmentation sequence to obtain an input word vector sequence.
[0069] Word2vec is a word vector training tool, and its function is to convert words in natural language into word vectors that can be understood by a computer. Traditional word vector training tools are easily troubled by the curse of dimensionality, and any two words are isolated from each other and cannot reflect the relationship between words. Therefore, this embodiment uses word2vec to train word vectors, which can reflect the similarity between words by calculating the distance between vectors.
[0070] For example, in one embodiment, the second word segmentation sequence is: spring / flower / so / beautiful. After training by word2vec, the word vector of "spring" is w11, the word vector of "flower" is w12, the word vector of "so" is w13, and the word vector of "beautiful" is w14. Then the input word vector sequence is (w11, w12, w13, w14).
[0071] It should be noted that word2vec is only an example of a word vector training tool provided by the present invention. Those skilled in the art can also use other word vector training tools, which will not exceed the protection scope of the present invention.
[0072] S34. Input the input word vector sequence into the self-attention layer of the first encoding layer of the neural network model.
[0073] In specific implementation, by deleting stopwords, the number of word vectors in the input word vector sequence can be reduced, thereby reducing the calculation amount of the self-attention layer of the first encoding layer of the neural network model.
[0074] S35, determining whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold.
[0075] In a specific implementation, after receiving the first word vector sequence output by the self-attention layer of the encoding layer, it is determined whether the number of word vectors contained in the first word vector sequence is greater than a preset number threshold.
[0076] It should be noted that the quantity threshold can be set by those skilled in the art based on experience, and the present invention does not specifically limit this. For example, in one embodiment, the quantity threshold is set to 10. If the number of word vectors included in the first word vector sequence is 12, it is determined that the number of word vectors included in the first word vector sequence is greater than the preset quantity threshold.
[0077] According to the common practice in the prior art, the first word vector sequence output by the self-attention layer is usually directly input into the feedforward network layer. If the first word vector sequence contains too many word vectors, the amount of calculation in the feedforward network layer will be very large (the amount of calculation of the neural network model is mainly concentrated in the feedforward network layer. The more word vectors there are in the word vector sequence, the higher the amount of calculation in the feedforward network layer). The present invention is aimed at the above-mentioned problem of large amount of calculation, and determines whether the number of word vectors contained in the first word vector sequence is greater than a preset number threshold. If so, the following step S2 is executed to reduce the amount of calculation by reducing the number of word vectors.
[0078] S36: If the number of word vectors contained in the first word vector sequence output from the self-attention layer is greater than a preset number threshold, some word vectors are extracted from the first word vector sequence to obtain a second word vector sequence.
[0079] In a specific implementation, if the number of word vectors contained in the first word vector sequence output from the attention layer is greater than a preset threshold value, the present invention does not directly input the first word vector sequence into the feedforward network layer, but extracts part of the word vectors from the first word vector sequence to form a second word vector sequence.
[0080] The number of word vectors extracted from the first word vector sequence is less than or equal to a quantity threshold. For example, in one embodiment, the quantity threshold is set to 10. In this embodiment, the number of word vectors extracted from the first word vector sequence is less than or equal to 10.
[0081] It should be noted that the number of word vectors extracted from the first word vector sequence can be determined by those skilled in the art according to specific calculation accuracy requirements, and the present invention does not specifically limit this. The more word vectors there are, the higher the calculation accuracy.
[0082] S37, input the second word vector sequence into the feedforward network layer of the encoding layer.
[0083] In a specific implementation, since the number of word vectors in the second word vector sequence is less than the number of word vectors in the first word vector sequence, the amount of computation in the feedforward network layer can be effectively reduced.
[0084] S38: If the number of word vectors contained in the first word vector sequence output from the attention layer is not greater than a preset number threshold, the first word vector sequence is input into the feedforward network layer of the encoding layer.
[0085] It should be noted that the quantity threshold can be set by those skilled in the art based on experience, and the present invention does not specifically limit this. For example, in one embodiment, the quantity threshold is set to 10. The number of word vectors included in the first word vector sequence is 8, and it is determined that the number of word vectors included in the first word vector sequence is not greater than the preset quantity threshold, and therefore, the first word vector sequence is directly input into the feedforward network layer of the encoding layer.
[0086] See also Figure 4 , Figure 4 It is a schematic block diagram of a model reasoning acceleration device 70 provided by an embodiment of the present invention. Corresponding to the above model reasoning acceleration method, the present invention also provides a model reasoning acceleration device 70. The model reasoning acceleration device 70 includes a unit for executing the above model reasoning acceleration method, and the model reasoning acceleration device 70 can be configured in a desktop computer, a tablet computer, a laptop computer, and other terminals. Specifically, the model reasoning acceleration device 70 is used to reduce the amount of calculation of the neural network model, the neural network model includes a coding layer, the coding layer includes a self-attention layer and a feedforward network layer, and the model reasoning acceleration device 70 includes a judgment unit 71, an extraction unit 72 and a first input unit 73.
[0087] A judging unit 71 is used to judge whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold;
[0088] An extraction unit 72 is configured to extract part of the word vectors from the first word vector sequence output from the attention layer to obtain a second word vector sequence if the number of word vectors contained in the first word vector sequence output from the attention layer is greater than a preset number threshold;
[0089] The first input unit 73 inputs the first word vector sequence into the feedforward network layer of the encoding layer.
[0090] In one embodiment, extracting some word vectors from the first word vector sequence to obtain a second word vector sequence includes:
[0091] Extracting a preset number of word vectors from the first word vector sequence in order to obtain a second word vector sequence;
[0092] or,
[0093] Randomly extracting a preset number of word vectors from the first word vector sequence to obtain a second word vector sequence;
[0094] The preset number is less than or equal to the number threshold.
[0095] In one embodiment, the self-attention layer includes a plurality of attention heads; for each word vector in the first word vector sequence, each of the attention heads calculates an importance score for it; and extracting part of the word vectors from the first word vector sequence to obtain a second word vector sequence includes:
[0096] For each word vector in the first word vector sequence, its overall importance score is determined according to the importance scores calculated for it by each attention head;
[0097] Extracting a preset number of word vectors from the first word vector sequence in descending order of overall importance scores to obtain a second word vector sequence;
[0098] The preset number is less than or equal to the number threshold.
[0099] In one embodiment, for each word vector in the first word vector sequence, determining its overall importance score according to the importance score calculated for it by each attention head includes:
[0100] By the formula Sig(w) = ∑ h Sig h (w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h (w) is the importance score calculated by the attention head h for the word vector w.
[0101] In one embodiment, for each word vector in the first word vector sequence, determining its overall importance score according to the importance score calculated for it by each attention head includes:
[0102] By the formula sig(w) = ∑ h K h sig h (w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h(w) is the importance score calculated by the attention head h for the word vector w, K h is the weight of the attention head h.
[0103] Figure 5 FIG. 7 is a schematic block diagram of a model reasoning acceleration device 70 provided by another embodiment of the present invention. Figure 5 As shown, the model reasoning acceleration device 70 of this embodiment is based on the above embodiment and adds a word segmentation unit 74, a deletion unit 75, a training unit 76, a second input unit 77 and a third input unit 78.
[0104] A word segmentation unit 74, used for performing word segmentation processing on the input corpus to obtain a first word segmentation sequence;
[0105] A deleting unit 75, configured to delete stop words in the first word segmentation sequence to obtain a second word segmentation sequence;
[0106] A training unit 76, configured to perform word vector training on the words in the second word segmentation sequence to obtain an input word vector sequence;
[0107] A second input unit 77, used to input the input word vector sequence into the self-attention layer of the first encoding layer of the neural network model;
[0108] The third input unit 78 is used to input the first word vector sequence output from the attention layer into the feedforward network layer of the encoding layer if the number of word vectors contained in the first word vector sequence output from the attention layer is not greater than a preset number threshold.
[0109] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned model reasoning acceleration device 70 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.
[0110] The above-mentioned model reasoning acceleration device 70 can be implemented in the form of a computer program. The computer program can be used in Figure 6 Runs on the computer device shown.
[0111] See also Figure 6 , Figure 6 500 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a terminal, wherein the terminal may be an electronic device with communication function such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device.
[0112] The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .
[0113] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute a model reasoning acceleration method.
[0114] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .
[0115] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a model reasoning acceleration method.
[0116] The network interface 505 is used to communicate with other devices over the network. Those skilled in the art will appreciate that the above structure is only a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the computer device 500 to which the present application solution is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0117] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:
[0118] Determine whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold;
[0119] If the number of word vectors contained in the first word vector sequence output by the self-attention layer is greater than a preset number threshold, extract some word vectors from the first word vector sequence to obtain a second word vector sequence;
[0120] The second word vector sequence is input into the feed-forward network layer of the encoding layer.
[0121] In one embodiment, extracting some word vectors from the first word vector sequence to obtain a second word vector sequence includes:
[0122] Extracting a preset number of word vectors from the first word vector sequence in order to obtain a second word vector sequence;
[0123] or,
[0124] Randomly extracting a preset number of word vectors from the first word vector sequence to obtain a second word vector sequence;
[0125] The preset number is less than or equal to the number threshold.
[0126] In one embodiment, the self-attention layer includes a plurality of attention heads; for each word vector in the first word vector sequence, each of the attention heads calculates an importance score for it; and extracting part of the word vectors from the first word vector sequence to obtain a second word vector sequence includes:
[0127] For each word vector in the first word vector sequence, its overall importance score is determined according to the importance scores calculated for it by each attention head;
[0128] Extracting a preset number of word vectors from the first word vector sequence in descending order of overall importance scores to obtain a second word vector sequence;
[0129] The preset number is less than or equal to the number threshold.
[0130] In one embodiment, for each word vector in the first word vector sequence, determining its overall importance score according to the importance score calculated for it by each attention head includes:
[0131] By the formula Sig(w) = ∑ h Sig h (w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h (w) is the importance score calculated by the attention head h for the word vector w.
[0132] In one embodiment, for each word vector in the first word vector sequence, determining its overall importance score according to the importance score calculated for it by each attention head includes:
[0133] By the formula sig(w) = ∑ h K h sig h (w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h (w) is the importance score calculated by the attention head h for the word vector w, K h is the weight of the attention head h.
[0134] In one embodiment, if the encoding layer is the first encoding layer of the neural network model, before determining whether the number of word vectors included in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold, the processor 502 further implements the following steps:
[0135] Perform word segmentation processing on the input corpus to obtain a first word segmentation sequence;
[0136] Deleting stop words in the first word segmentation sequence to obtain a second word segmentation sequence;
[0137] Performing word vector training on the words in the second word segmentation sequence to obtain an input word vector sequence;
[0138] Inputting the input word vector sequence into the self-attention layer of the first encoding layer of the neural network model;
[0139] If the number of word vectors contained in the first word vector sequence output from the attention layer is not greater than a preset number threshold, the first word vector sequence is input into the feedforward network layer of the encoding layer.
[0140] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0141] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.
[0142] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the following steps:
[0143] Determine whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold;
[0144] If the number of word vectors contained in the first word vector sequence output by the self-attention layer is greater than a preset number threshold, extract some word vectors from the first word vector sequence to obtain a second word vector sequence;
[0145] The second word vector sequence is input into the feed-forward network layer of the encoding layer.
[0146] In one embodiment, extracting some word vectors from the first word vector sequence to obtain a second word vector sequence includes:
[0147] Extracting a preset number of word vectors from the first word vector sequence in order to obtain a second word vector sequence;
[0148] or,
[0149] Randomly extracting a preset number of word vectors from the first word vector sequence to obtain a second word vector sequence;
[0150] The preset number is less than or equal to the number threshold.
[0151] In one embodiment, the self-attention layer includes a plurality of attention heads; for each word vector in the first word vector sequence, each of the attention heads calculates an importance score for it; and extracting part of the word vectors from the first word vector sequence to obtain a second word vector sequence includes:
[0152] For each word vector in the first word vector sequence, its overall importance score is determined according to the importance scores calculated for it by each attention head;
[0153] Extracting a preset number of word vectors from the first word vector sequence in descending order of overall importance scores to obtain a second word vector sequence;
[0154] The preset number is less than or equal to the number threshold.
[0155] In one embodiment, for each word vector in the first word vector sequence, determining its overall importance score according to the importance score calculated for it by each attention head includes:
[0156] By the formula Sig(w) = ∑ h Sig h (w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h (w) is the importance score calculated by the attention head h for the word vector w.
[0157] In one embodiment, for each word vector in the first word vector sequence, determining its overall importance score according to the importance score calculated for it by each attention head includes:
[0158] By the formula sig(w) = ∑ h Kh sig h (w) Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector, h is the attention head, sig(w) is the overall importance score of the word vector w, sig h (w) is the importance score calculated by the attention head h for the word vector w, K h is the weight of the attention head h.
[0159] In one embodiment, if the encoding layer is the first encoding layer of the neural network model, before determining whether the number of word vectors included in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold, the processor executes the computer program to further implement the following steps:
[0160] Perform word segmentation processing on the input corpus to obtain a first word segmentation sequence;
[0161] Deleting stop words in the first word segmentation sequence to obtain a second word segmentation sequence;
[0162] Performing word vector training on the words in the second word segmentation sequence to obtain an input word vector sequence;
[0163] Inputting the input word vector sequence into the self-attention layer of the first encoding layer of the neural network model;
[0164] If the number of word vectors contained in the first word vector sequence output from the attention layer is not greater than a preset number threshold, the first word vector sequence is input into the feedforward network layer of the encoding layer.
[0165] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which can store program codes. The computer-readable storage medium can be non-volatile or volatile.
[0166] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0167] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0168] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.
[0170] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0171] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
[0172] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A model reasoning acceleration method, characterized in that: The neural network model includes an encoding layer, the encoding layer includes a self-attention layer and a feedforward network layer, and the model reasoning acceleration method includes: Determine whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold; If the number of word vectors contained in the first word vector sequence output by the self-attention layer is greater than a preset number threshold, extract some word vectors from the first word vector sequence to obtain a second word vector sequence, wherein the self-attention layer includes a plurality of attention heads; for each word vector in the first word vector sequence, each of the attention heads calculates an importance score for it; The step of extracting part of the word vectors from the first word vector sequence to obtain the second word vector sequence includes: for each word vector in the first word vector sequence, determining its overall importance score according to the importance score calculated for it by each attention head; Extracting a preset number of word vectors from the first word vector sequence in descending order of overall importance scores to obtain a second word vector sequence, wherein the preset number is less than or equal to the number threshold; Or, randomly extracting a preset number of word vectors from the first word vector sequence to obtain a second word vector sequence, wherein the preset number is less than or equal to the quantity threshold; The step of determining the overall importance score of each word vector in the first word vector sequence according to the importance scores calculated for it by each attention head comprises: By formula Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector and h is the attention head. is the overall importance score of the word vector w, is the importance score computed by attention head h for word vector w; or By formula Calculate the overall importance score of the word vectors in the first word vector sequence, where w is the word vector and h is the attention head. is the overall importance score of the word vector w, is the importance score calculated by the attention head h for the word vector w, is the weight of the attention head h; The second word vector sequence is input into the feed-forward network layer of the encoding layer.
2. The model reasoning acceleration method according to claim 1, characterized in that: If the encoding layer is the first encoding layer of the neural network model, before determining whether the number of word vectors included in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold, the model reasoning acceleration method further includes: Perform word segmentation processing on the input corpus to obtain a first word segmentation sequence; Deleting stop words in the first word segmentation sequence to obtain a second word segmentation sequence; Performing word vector training on the words in the second word segmentation sequence to obtain an input word vector sequence; The input word vector sequence is input into the self-attention layer of the first encoding layer of the neural network model.
3. The model reasoning acceleration method according to claim 1, characterized in that: The model reasoning acceleration method also includes: If the number of word vectors contained in the first word vector sequence output from the attention layer is not greater than a preset number threshold, the first word vector sequence is input into the feedforward network layer of the encoding layer.
4. A model reasoning acceleration device, the device is used to implement the method according to any one of claims 1 to 3, characterized in that: The neural network model includes a coding layer, the coding layer includes a self-attention layer and a feedforward network layer, and the model reasoning acceleration device includes: A judging unit, used to judge whether the number of word vectors contained in the first word vector sequence output by the self-attention layer of the encoding layer is greater than a preset number threshold; An extraction unit, configured to extract part of the word vectors from the first word vector sequence output from the attention layer to obtain a second word vector sequence if the number of word vectors contained in the first word vector sequence output from the attention layer is greater than a preset number threshold; The first input unit inputs the first word vector sequence into the feedforward network layer of the encoding layer.
5. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 3 when executing the computer program.
6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 can be implemented.