Embedded layer access acceleration system and method of neural network model

Abstract

The invention provides an embedded layer access collaborative acceleration system and method of a neural network model, a processing unit layer, a control layer and a routing layer cooperatively work, and the control layer can predict a candidate item of a next lexical element before a processing unit by querying a lexical element statistical table. A routing layer reads word embedding vectors of candidate lexical elements from a memory bank in advance and stores the word embedding vectors in a prefetching buffer area corresponding to the memory bank, and after a processing unit inferes a next lexical element by using a neural network model, if the next lexical element hits the candidate lexical element, the next lexical element is not hit by the candidate lexical element. If not, the word embedding vector of the hit candidate lexical element is read from the prefetching buffer area, and compared with the mode that the processing unit inferes the next lexical element and then reads the corresponding word embedding vector from the storage bank, lexical element prefetching processing is increased, and the access speed of an embedding layer of the neural network model can be increased.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to, but is not limited to, artificial intelligence hardware acceleration and computing optimization technology, and more particularly to a neural network model embedding layer access collaborative acceleration system and method. BACKGROUND

[0002] Neural network models (such as Transformer models) have a wide range of applications in natural language processing, multi-modal reasoning, etc. Any text needs to be converted into a vector sequence by an Embedding layer before entering a neural network model. In large-scale models, the parameter quantity of the Embedding layer is extremely large. The vocabulary table can have tens of thousands or even hundreds of thousands of words, and each word corresponds to a vector of several hundred or several thousand dimensions. This makes the Embedding layer account for a considerable part of the total parameter quantity of the Transformer model (up to 20-30% in some models), and therefore, the memory access efficiency of the Embedding layer becomes a key constraint on system performance.

[0003] Traditional Embedding access architecture adopts static hash mapping (such as linear hash or modulo operation) to map Token (word element) IDs to fixed storage banks, resulting in queuing of PEs (Processing Elements) when accessing high-frequency Tokens due to centralized access to the same bank. According to Zipf's law, the top 1% of Tokens can account for more than 60% of the access volume, so static mapping cannot adapt to dynamic loads.

[0004] In addition, due to semantic discontinuity, the co-occurrence law of natural language Token sequences (such as "deep → degree → study → learn") cannot be captured by fixed step or simple state machine, so traditional cache prefetching mechanisms (such as Stride prefetching) are inefficient in the Embedding scenario. SUMMARY

[0005] The embodiment of the present disclosure provides an embedding layer access collaborative acceleration system of a neural network model, comprising: a processing unit layer, a control layer, a routing layer and a storage layer; the processing unit layer comprises at least one processing unit; the storage layer comprises: a cache and a plurality of storage banks; the storage bank is used for storing the word embedding vector of each word element in the word element dictionary, and the cache comprises a plurality of prefetch buffers, different storage banks correspond to different prefetch buffers; The processing unit is configured to send a prefetch request to the control layer, wherein the prefetch request carries an identifier of the processing unit and an identifier of a last word piece inferred by the neural network model; and after inferring a next word piece by using the neural network model, if a prefetch success response is received, it is determined whether the next word piece hits a candidate word piece, if yes, a word embedding vector of the hit candidate word piece is read from a prefetch buffer; if not, a word embedding vector of the next word piece is read from a storage according to the identifier of the next word piece; The control layer is configured to receive the prefetch request, query a word piece statistic table according to the identifier of the last word piece, if a front word piece of at least one word piece sequence is hit, select part of the word piece sequences from the hit word piece sequences according to a prefetch strategy, take a subsequent word piece in the selected word piece sequence as a candidate word piece, and send a first access request to the routing layer, wherein the first access request carries the identifier of the processing unit and the identifier of the candidate word piece; wherein any one word piece sequence comprises a front word piece and a subsequent word piece. The routing layer is configured to receive the first access request, read a word embedding vector of the candidate word piece from the storage according to the identifier of the candidate word piece and save the word embedding vector in a prefetch buffer corresponding to the storage, and send a prefetch success response to the processing unit, wherein the prefetch success response carries the identifier of the candidate word piece and a storage address of the candidate word piece in the prefetch buffer. The word piece statistic table is arranged in the cache.

[0006] The embodiment of the present disclosure provides a neural network model embedding layer access cooperative acceleration method, which comprises the following steps: The processing unit sends a prefetch request to the control layer, wherein the prefetch request carries an identifier of the processing unit and an identifier of a last word piece inferred by the neural network model. The control layer receives the prefetch request, queries a word piece statistic table according to the identifier of the last word piece, if a front word piece of at least one word piece sequence is hit, selects part of the word piece sequences from the hit word piece sequences according to a prefetch strategy, takes a subsequent word piece in the selected word piece sequence as a candidate word piece, and sends a first access request to the routing layer, wherein the first access request carries the identifier of the processing unit and the identifier of the candidate word piece; wherein any one word piece sequence comprises a front word piece and a subsequent word piece. The routing layer receives the first access request, reads a word embedding vector of the candidate word piece from the storage according to the identifier of the candidate word piece and saves the word embedding vector in a prefetch buffer corresponding to the storage, and sends a prefetch success response to the processing unit, wherein the prefetch success response carries the identifier of the candidate word piece and a storage address of the candidate word piece in the prefetch buffer. The processing unit, after inferring the next word piece by using the neural network model, if a pre-fetch success response is received, judges whether the next word piece hits a candidate word piece, if yes, reads the word embedding vector of the hit candidate word piece from the pre-fetch buffer, if not, reads the word embedding vector of the next word piece from the storage according to the identification of the next word piece; wherein the storage is used to store the word embedding vector of each word piece in the word piece dictionary, each storage corresponds to a pre-fetch buffer, and the word piece statistics table and the pre-fetch buffer of each storage are arranged in the cache.

[0007] The embedding layer access collaborative acceleration system and method of the neural network model provided by the embodiments of the present disclosure, the processing unit layer, the control layer and the routing layer work collaboratively, the control layer can predict the candidate item (candidate word piece) of the next word piece in advance by querying the word piece statistics table, the routing layer reads the word embedding vector of the candidate word piece from the storage in advance and saves it in the pre-fetch buffer corresponding to the storage, and the processing unit, after inferring the next word piece by using the neural network model, if the next word piece hits the candidate word piece, reads the word embedding vector of the hit candidate word piece from the pre-fetch buffer, compared with reading the corresponding word embedding vector from the storage after the processing unit infers the next word piece, increasing the word piece pre-fetching process can speed up the access speed of the embedding layer of the neural network model.

[0008] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present application. Other advantages of the present application will be realized and attained by those skilled in the art based on the description provided herein and will be particularly pointed out in the description. BRIEF DESCRIPTION OF DRAWINGS

[0009] The accompanying drawings are included to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0010] Figure 1 A structural schematic diagram of an embedding layer access collaborative acceleration system of a neural network model according to an embodiment of the present disclosure; Figure 2 A schematic diagram of word piece dictionary multi-copy load balancing for a plurality of (4*8) processing units according to an example of the present disclosure; Figure 3 A flowchart of an embedding layer access collaborative acceleration method of a neural network model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] The present disclosure describes a number of embodiments, but this description is illustrative rather than limiting and thus, many more embodiments and implementations than described herein will become apparent to those skilled in the art from the following description. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Any feature or element of any embodiment can be used with any other feature or element from any other embodiment, or can replace any other feature or element in any other embodiment, unless specifically stated otherwise.

[0012] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The presently disclosed embodiments, features and elements can also be combined with any conventional features or elements to form unique inventive solutions. Any feature or element of any embodiment can also be combined with features or elements from other inventive solutions to form another unique inventive solution. Thus, it should be understood that any of the features shown and / or discussed in the present disclosure can be implemented alone or in any combination. Accordingly, except as otherwise restricted, embodiments are not limited to the other restrictions other than those limitations made explicitly in the appended claims and their equivalents. Moreover, various modifications and changes can be made within the scope of the appended claims.

[0013] Furthermore, in describing representative embodiments, the specification can have presented the method and / or process as a particular sequence of steps. However, to the extent that the method or process depends on more than one step, the method or process should not be limited to the particular sequence of steps described. Other sequences of steps can be possible, and are within the scope of the present disclosure. Therefore, the particular order in which the steps are presented in the specification is not an implied limitation on the order in which the steps can be performed. Moreover, the specification can present the steps of the method and / or process in a particular order, which is not necessarily the order in which the steps are performed. The steps of the method and / or process can be performed in any order that is practical and / or suitable.

[0014] Figure 1 A structural diagram of an embedding layer access collaborative acceleration system of a neural network model is shown. As shown in Figure 1 An embedding layer access collaborative acceleration system of a neural network model includes a processing unit layer 10, a control layer 20, a routing layer 30, and a storage layer 40. The processing unit layer 10 includes at least one processing unit 101. The storage layer 40 includes a cache 401 and a plurality of storage banks 402. The storage banks are used to store word embedding vectors of each word in a word dictionary, and the cache includes a plurality of prefetch buffers. Different storage banks correspond to different prefetch buffers. The processing unit is configured to send a prefetch request to the control layer, wherein the prefetch request carries an identifier of the processing unit and an identifier of a last word piece inferred by the neural network model; and after inferring a next word piece by using the neural network model, if a prefetch success response is received, it is determined whether the next word piece hits a candidate word piece, if yes, a word embedding vector of the hit candidate word piece is read from a prefetch buffer; if not, a word embedding vector of the next word piece is read from a storage according to the identifier of the next word piece; The control layer is configured to receive the prefetch request, query a word piece statistics table according to the identifier of the last word piece, if a front word piece of at least one word piece sequence is hit, select part of the word piece sequences from the hit word piece sequences according to a prefetch strategy, take the subsequent word pieces in the selected word piece sequences as candidate word pieces, and send a first access request to the routing layer, wherein the first access request carries the identifier of the processing unit and the identifier of the candidate word piece; wherein any one word piece sequence comprises a front word piece and a subsequent word piece. The routing layer is configured to receive the first access request, read a word embedding vector of the candidate word piece from the storage according to the identifier of the candidate word piece and save the word embedding vector in a prefetch buffer corresponding to the storage, and send a prefetch success response to the processing unit, wherein the prefetch success response carries the identifier of the candidate word piece and a storage address of the candidate word piece in the prefetch buffer. The word piece statistics table is arranged in the cache.

[0015] The embedding layer access collaborative acceleration system for the neural network model provided in the embodiment is characterized in that the processing unit layer, the control layer and the routing layer work collaboratively, the control layer can predict a candidate item (a candidate word piece) of a next word piece in advance by querying a word piece statistics table, the routing layer reads a word embedding vector of the candidate word piece from a storage in advance and saves the word embedding vector in a prefetch buffer corresponding to the storage, and the processing unit reads the word embedding vector of the hit candidate word piece from the prefetch buffer after inferring the next word piece by using the neural network model, if the next word piece hits the candidate word piece. Compared with reading the corresponding word embedding vector from the storage after the processing unit infers the next word piece, increasing the word piece prefetching process can accelerate the access speed of the embedding layer of the neural network model.

[0016] A token is a smallest semantic unit segmented from a piece of text. Through a lexical analysis process, a continuous character sequence can be segmented into a meaningful sequence. A token can be a character, a word, or other language units. A Chinese token can be a single Chinese character (character), a common word (such as "like"), or a part of a word (such as "Jiuzhaigou") ( "Jiuzhaigou"). An English token can be a character, a subword, or a word. A subword can be a complete short word (such as the), a part of a long word (such as "ation"), an affix (such as "un"), or a single character (such as when splitting acronyms or very rare words). An N-Gram is a token sequence composed of N consecutive tokens. A Bi-Gram is a token sequence composed of two tokens, including a preceding token and a following token, such as: "I like", "like apple". In the token sequence "I like", "I" is the preceding token and "like" is the following token. In the token sequence "like apple", "like" is the preceding token and "apple" is the following token.

[0017] The token dictionary includes a plurality of token records, each token record including a token and a token ID, the token ID being used to represent the index position of the record in the token dictionary.

[0018] In an example embodiment of the present disclosure, the neural network model includes a Transformer model. The Transformer model has a wide range of applications in natural language processing, multi-modal reasoning, and other fields. In other embodiments, the neural network model can also be other models.

[0019] In an example embodiment of the present disclosure, the cache further includes a first-level cache and a second-level cache; the access speed of the first-level cache is faster than the access speed of the second-level cache. The token statistics table includes a first token statistics table and a second token statistics table; the first token statistics table is stored in the first-level cache, and the second token statistics table is stored in the second-level cache.

[0020] In an example embodiment of the present disclosure, the cache further includes a third-level cache; the access speed of the second-level cache is faster than the access speed of the third-level cache. The pre-fetch buffer corresponding to each memory bank is set in the third-level cache.

[0021] In an example embodiment of the present disclosure, the cache is set in an on-chip SRAM (Static Random-Access Memory).

[0022] In an example embodiment of the present disclosure, the memory bank is a core structure unit of a DRAM (Dynamic Random-Access Memory).

[0023] In the embodiment, by setting a multi-level cache mechanism, balance and optimization can be achieved in terms of access speed and storage space. The high-frequency access word sequence is placed in the first-level cache with the fastest access speed, but the capacity of the first-level cache is small. The less frequently used word sequence is placed in the second-level cache with slower access speed, but the second-level cache can have larger capacity. The third-level cache is used to store the word embedding vectors of the candidate words. Since the word embedding vectors occupy a larger space, the capacity of the third-level cache is larger than that of the second-level cache.

[0024] In an example embodiment of the present disclosure, the first word sequence statistical table includes a1 word sequence records; Each word sequence record includes: an identifier of a preceding word, a first index generated from the identifier of the preceding word, an identifier of a subsequent word, a second index generated from the identifier of the subsequent word, a hotness score of the word sequence, and a timestamp of the word sequence. The timestamp is the creation time or the latest access time of the word sequence. The latest access time refers to the time when the subsequent word in the word sequence is read into the prefetch buffer and hits the next word inferred by the neural network model for the last time.

[0025] In an example embodiment of the present disclosure, the first index is a numerical value obtained by performing first hash operation on the identifier of the preceding word, and the second index is a numerical value obtained by performing first hash operation on the identifier of the subsequent word.

[0026] In an example embodiment of the present disclosure, the first hash operation includes CRC32 (Cyclic Redundancy Check 32) operation. In other embodiments, the first hash operation can also use other algorithms.

[0027] In an example embodiment of the present disclosure, the second word sequence statistical table includes a2 word sequence records; Each word sequence record includes: an identifier of a preceding word, a third index generated from the identifier of the preceding word, an identifier of a subsequent word, a fourth index generated from the identifier of the subsequent word, a hotness score of the word sequence, and a timestamp of the word sequence. The timestamp is the creation time or the latest access time of the word sequence. The latest access time refers to the time when the subsequent word in the word sequence is read into the prefetch buffer and hits the next word inferred by the neural network model for the last time. .

[0028] In an example embodiment of the present disclosure, the third index is a value obtained by performing a second hash operation on the identification of the previous word element, and the fourth index is a value obtained by performing a second hash operation on the identification of the subsequent word element.

[0029] In an example embodiment of the present disclosure, the second hash operation includes a MurmurHash3 operation. In other embodiments, the second hash operation can also use other algorithms.

[0030] In an example embodiment of the present disclosure, the first word element statistical table and the second word element statistical table are pre-set in the cache. In the first word element statistical table, a1 word element sequences with the highest frequency of occurrence in the neural network model training process are included. The value of a1 is much smaller than the value of a2. For example, a1 can be a value in the order of thousands, and a2 can be a value in the order of tens of thousands or millions.

[0031] In an example embodiment of the present disclosure, because the number of records in the first word element statistical table is much smaller than the number of records in the second word element statistical table, in order to improve the query efficiency and save storage space, the complexity of the first hash operation is lower than the complexity of the second hash operation, and the length of the numerical result obtained by the first hash operation is shorter than the length of the numerical result obtained by the second hash operation.

[0032] In an example embodiment of the present disclosure, according to the identification of the previous word element, the word element statistical table is queried, and if the previous word element of at least one word element sequence is hit, part of the word element sequences are selected from the hit word element sequences according to a pre-fetching strategy, and the subsequent word elements in the selected word element sequences are taken as candidate word elements, including: performing a first hash operation on the identification of the previous word element to obtain a first target index, querying the first word element statistical table according to the first target index, and if the first target index hits the first index of m1 word element sequence records and then selecting m1 word element sequence records with the highest heat score from the hit word element sequences, and taking the subsequent word elements in the selected word element sequences as m1 candidate word elements; if the first target index hits the first index of m1 word element sequence records and then taking the subsequent word elements in the m1 word element sequences as m1 candidate word elements, performing a second hash operation on the identification of the previous word element to obtain a second target index, querying the second word element statistical table according to the second target index, and if the second target index hits the second index of m2 word element sequence records and then selecting m2 word element sequence records with the highest heat score from the hit word element sequences, and taking the subsequent word elements in the selected word element sequences as m2 candidate word elements. ​m1 candidate word items that hit the first word item statistical table and m2 candidate word items that hit the second word item table together as n candidate word items; if the second target index hits the second index of m2 word sequence records and m2 candidate word items that hit the second word item table together as n candidate word items; if the second target index hits the second index of m2 word sequence records and m2 candidate word items that hit the second word item table together as n candidate word items; if the second target index hits the second index of m2 word sequence records and if the first target index does not hit any word sequence record in the first word item statistical table, performing a second hash operation on the identification of the last word item to obtain a second target index, querying the second word item statistical table according to the second target index, if the second target index hits the second index of m2 word sequence records and m2 candidate word items that hit the second word item table together as n candidate word items; if the second target index hits the second index of m2 word sequence records and m2 candidate word items that hit the second word item table together as n candidate word items; if the second target index hits the second index of m2 word sequence records and m2 candidate word items that hit the second word item table together as n candidate word items; if the second target index hits the second index of m2 word sequence records and m2 candidate word items that hit the second word item table together as n candidate word items; if the second target index hits the second index of m2 word sequence records and n is a preset quantity threshold of candidate word items.

[0033] In the embodiment, by using two-level word item statistical tables, the first word item statistical table is used to place the most active word sequence, and the second word item statistical table is used to place the remaining word sequence. When pre-fetching candidate word items, the first word item statistical table is queried first. If enough skilled candidate word items are found, the second word item statistical table does not need to be accessed, thereby accelerating the speed of pre-fetching candidate word items.

[0034] In an example embodiment of the present disclosure, the processing unit is further configured to, if a pre-fetch failure response is received, read a word embedding vector of a next word item from the storage according to an identification of the next word item, and update the second word item statistical table in the following manner: add a word sequence record to the second word item statistical table, the added word sequence taking the last word item of the neural network model inference as a previous word item and taking the next word item of the neural network model inference as a subsequent word item; perform a second hash operation on the identification of the previous word item to generate a third index; perform a second hash operation on the identification of the subsequent word item to generate a fourth index; set an initial value for the heat score of the word sequence, the initial value being the maximum value of the heat scores in the current second word item statistical table; and set a timestamp of the word sequence as a creation time of the word sequence; The control layer is further configured to query the word statistic table according to the identifier of the previous word, and send a pre-fetch failure response to the processing unit if no previous word of any word sequence is hit.

[0035] In the embodiment, by continuously adding the newly appearing word combination to the second word statistic table and setting a larger initial value for the heat score of the word sequence, the word combination has a chance to quickly rise to the first word statistic table, thereby making the pre-fetch candidate word operation adapt to the context of the current text, and thus speeding up the pre-fetch candidate word.

[0036] In an example embodiment of the present disclosure, the processing unit is further configured to, after inferring the next word using the neural network model, update the first word statistic table and the second word statistic table as follows: If a pre-fetch success response is received, the first word statistic table and the second word statistic table are queried according to the identifier of the previous word and the identifier of the candidate word, and the heat score of each word sequence hit is increased by a first step value; If the next word hits the candidate word, the first word statistic table and the second word statistic table are queried according to the identifier of the previous word and the identifier of the hit candidate word, and the heat score of the word sequence hit is increased by a second step value; wherein the second step value is greater than the first step value.

[0037] In the embodiment, by updating the first word statistic table and the second word statistic table for reward purposes after inferring the next word each time, the high-frequency words in the current text can be discovered, counted and pre-fetched in time, thereby speeding up the pre-fetch candidate word.

[0038] In an example embodiment of the present disclosure, the processing unit is further configured to periodically update the first word statistic table and the second word statistic table, and perform at least one of the following processes: The heat score of the word sequence record in the first word statistic table whose time stamp is more than the aging time threshold from the current time is multiplied by a decay coefficient p; ; The word sequence record with the hotness score of the word sequence in the first word statistic table less than or equal to the elimination threshold is removed from the first word statistic table to the second word statistic table, and a second hash operation is performed on the identification of the previous word of the word sequence to generate a third index, the third index is used to replace the first index in the original record, a second hash operation is performed on the identification of the subsequent word of the word sequence to generate a fourth index, and the fourth index is used to replace the second index in the original record; after removing a word sequence record from the first word statistic table each time, the word sequence record with the highest hotness score in the second word statistic table is removed from the second word statistic table to the first word statistic table, and a first hash operation is performed on the identification of the previous word of the word sequence to generate a first index, the first index is used to replace the third index in the original record, a first hash operation is performed on the identification of the subsequent word of the word sequence to generate a second index, and the second index is used to replace the fourth index in the original record.

[0039] In the embodiment, by periodically updating the first word statistic table and the second word statistic table, the elimination / promotion mechanism between the first word statistic table and the second word statistic table is improved, the old records and the records with inactive access in the first word statistic table can be timely eliminated to the second word statistic table, and the new records and the records with active access in the second word statistic table can be timely promoted to the first word statistic table, so that the timeliness and effectiveness of the first word statistic table can be improved.

[0040] In an example embodiment of the present disclosure, the storage layer is configured to classify the word in the word dictionary with the access frequency greater than or equal to the first frequency threshold into the first word set, classify the word in the word dictionary with the access frequency greater than or equal to the second frequency threshold and less than the first frequency threshold into the second word set, and classify the word in the word dictionary with the access frequency less than the second frequency threshold into the third word set. The word embedding vector corresponding to each word in the first word set is copied k copies, and the original word embedding vector and the k copies are respectively stored in k+1 different banks; wherein k is greater than 1; the word embedding vector corresponding to each word in the second word set is copied one copy, and the original word embedding vector and the one copy are respectively stored in two different banks; the word embedding vector corresponding to each word in the third word set is stored in a bank.

[0041] In the embodiment, by dividing the words in the word dictionary into three levels (high, medium and low) according to the access frequency, different numbers of copies are set for the word embedding vectors with different levels (high frequency or medium frequency) of access frequency, and the original vector and the copy are stored in different banks, so that the multi-copy load balancing can be realized, and the bank conflict of the high frequency or medium frequency access word can be reduced.

[0042] In an example embodiment of the present disclosure, when the processing unit layer includes M rows and N columns of processing units; k = M - 1; Each word embedding vector corresponding to each word in the first word set is stored in a different memory bank, and processing units in different rows access different memory banks. Each word embedding vector corresponding to each word in the second word set is stored in a different memory bank, and processing units in the first M / 2 rows access one memory bank and processing units in the last M / 2 rows access another memory bank.

[0043] In one example, as shown in Figure 2 When the processing unit layer includes 4 rows and 8 columns of processing units (PEs), 8 processing units in the same row read word embedding vector data of the same word in parallel through data slicing, so each processing unit only loads embeddings_dim / 8 fragments of the word, and the total data amount is embeddings_dim (cooperatively completed by all PEs). Using the hierarchical multi-copy strategy of the present application, assuming that the first word set includes 1% of the words in the entire word dictionary, the second word set includes 4% of the words in the entire word dictionary, and the third word set includes 95% of the words in the entire word dictionary, only 1% of the high-frequency data needs to be copied 3 times, and each row of PEs accesses its own word embedding vector data; 4% of the medium-frequency data needs to be copied 1 time, and PEs(0,x) and PEs(1,x) share one copy of the word embedding vector data, and PEs(2,x) and PEs(3,x) share another copy of the word embedding vector data; the remaining 95% of the low-frequency data only needs to be stored once, and the four rows of PEs share.

[0044] The total data amount (embeddings_dim) is: wherein, is the number of words in the word dictionary, is the number of columns of word embedding vectors (column vectors) in the word dictionary, is the number of bytes of data per column of word embedding vectors.

[0045] In this example, by dividing the words in the word dictionary into three levels (high, medium, and low) according to the access frequency, different numbers of copies are set for word embedding vectors with different levels (high or medium) of access frequency, achieving multi-copy load balancing, which can reduce bank conflicts for high or medium access words on the one hand, and can also greatly reduce data storage capacity compared to the simple copy strategy without classification ( ).

[0046] Figure 3 A flowchart of a method for accessing and accelerating a neural network model embedding layer is shown. As shown inFigure 3 The embodiment provides a neural network model embedding layer access cooperative acceleration method, which comprises the following steps: In step S10, the processing unit sends a pre-fetch request to the control layer, wherein the pre-fetch request carries an identifier of the processing unit and an identifier of a last word element inferred by the neural network model; In step S20, the control layer receives the pre-fetch request, queries a word element statistical table according to the identifier of the last word element, and if a front word element of at least one word element sequence is hit, selects part of the word element sequence according to a pre-fetch strategy, takes a subsequent word element in the selected word element sequence as a candidate word element, and sends a first access request to the routing layer, wherein the first access request carries the identifier of the processing unit and the identifier of the candidate word element; wherein any one word element sequence comprises a front word element and a subsequent word element. In step S30, the routing layer receives the first access request, reads a word embedding vector of the candidate word element from a storage body according to the identifier of the candidate word element and saves the word embedding vector in a pre-fetch buffer corresponding to the storage body, and sends a pre-fetch success response to the processing unit, wherein the pre-fetch success response carries the identifier of the candidate word element and a storage address of the candidate word element in the pre-fetch buffer. In step S40, after the processing unit infers a next word element by using the neural network model, if the pre-fetch success response is received, it is judged whether the next word element hits the candidate word element, and if the next word element hits the candidate word element, the word embedding vector of the hit candidate word element is read from the pre-fetch buffer; if the next word element does not hit the candidate word element, a word embedding vector of the next word element is read from the storage body according to the identifier of the next word element; wherein the storage body is used for storing a word embedding vector of each word element in a word element dictionary, each storage body corresponds to a pre-fetch buffer, and the word element statistical table and the pre-fetch buffer of each storage body are arranged in a cache.

[0047] The neural network model embedding layer access cooperative acceleration method provided by the embodiment is characterized in that the processing unit, the control layer and the routing layer work cooperatively, the control layer can predict a candidate item (a candidate word element) of a next word element in advance by querying the word element statistical table, the routing layer reads a word embedding vector of the candidate word element from a storage body in advance and saves the word embedding vector in a pre-fetch buffer corresponding to the storage body, and after the processing unit infers the next word element by using the neural network model, if the next word element hits the candidate word element, the word embedding vector of the hit candidate word element is read from the pre-fetch buffer; compared with reading a corresponding word embedding vector from the storage body after the processing unit infers the next word element, the increase of the word element pre-fetching process can accelerate the access speed of the embedding layer of the neural network model.

[0048] In an example embodiment of the present disclosure, the neural network model comprises a Transformer model. The Transformer model has a wide range of applications in natural language processing, multi-modal reasoning, etc. In other embodiments, the neural network model can also be other models.

[0049] In an example embodiment of the present disclosure, the cache comprises a first-level cache and a second-level cache; the access speed of the first-level cache is faster than that of the second-level cache. The word statistics table comprises a first word statistics table and a second word statistics table; the first word statistics table is stored in the first-level cache, and the second word statistics table is stored in the second-level cache.

[0050] In an example embodiment of the present disclosure, the cache further comprises a third-level cache; the access speed of the second-level cache is faster than that of the third-level cache. The pre-fetch buffer corresponding to each memory bank is arranged in the third-level cache.

[0051] In an example embodiment of the present disclosure, the cache is arranged in an on-chip SRAM (Static Random-Access Memory).

[0052] In an example embodiment of the present disclosure, the memory bank is a core structure unit of a DRAM (Dynamic Random-Access Memory).

[0053] In an example embodiment of the present disclosure, the first word statistics table comprises a1 word sequence records. Each word sequence record comprises an identifier of a preceding word, a first index generated from the identifier of the preceding word, an identifier of a subsequent word, a second index generated from the identifier of the subsequent word, a hotness score of the word sequence, and a timestamp of the word sequence; the timestamp is a creation time or a latest access time of the word sequence; the latest access time refers to a time when the subsequent word in the word sequence is read into the pre-fetch buffer and hits the next word inferred by the neural network model.

[0054] In an example embodiment of the present disclosure, the first index is a numerical value obtained by performing a first hash operation on the identifier of the preceding word, and the second index is a numerical value obtained by performing a first hash operation on the identifier of the subsequent word.

[0055] In an example embodiment of the present disclosure, the first hash operation comprises a CRC32 (Cyclic Redundancy Check 32) operation. In other embodiments, the first hash operation can also use other algorithms.

[0056] In an example embodiment of the present disclosure, the second wordpiece statistics table includes a2 wordpiece sequence records. Each wordpiece sequence record includes: an identification of a preceding wordpiece, a third index generated from the identification of the preceding wordpiece, an identification of a subsequent wordpiece, a fourth index generated from the identification of the subsequent wordpiece, a hotness score of the wordpiece sequence, and a timestamp of the wordpiece sequence; the timestamp is a creation time or a latest access time of the wordpiece sequence; the latest access time refers to a time when the subsequent wordpiece in the wordpiece sequence is read into the prefetch buffer and hits the next wordpiece inferred by the neural network model for the last time. .

[0057] In an example embodiment of the present disclosure, the third index is a numerical value obtained by performing a second hash operation on the identification of the preceding wordpiece, and the fourth index is a numerical value obtained by performing a second hash operation on the identification of the subsequent wordpiece.

[0058] In an example embodiment of the present disclosure, the second hash operation includes a MurmurHash3 operation. In other embodiments, the second hash operation can also use other algorithms.

[0059] In an example embodiment of the present disclosure, the first wordpiece statistics table and the second wordpiece statistics table are pre-set in the cache. In the first wordpiece statistics table, a1 wordpiece sequences with the highest frequency in the neural network model training process are included. The value of a1 is much smaller than the value of a2, for example, a1 can be a value in the order of thousands, and a2 can be a value in the order of tens of thousands or millions.

[0060] In an example embodiment of the present disclosure, the control layer queries the wordpiece statistics table according to the identification of the last wordpiece, and if the preceding wordpiece of at least one wordpiece sequence is hit, it selects part of the wordpiece sequences according to the prefetch strategy, and the subsequent wordpiece in the selected wordpiece sequence is taken as a candidate wordpiece, including: The control layer performs a first hash operation on the identification of the last wordpiece to obtain a first target index, and queries the first wordpiece statistics table according to the first target index, if the first target index hits the first index of m1 wordpiece sequence records and then selects the n wordpiece sequence records with the highest hotness score from the hit wordpiece sequences, and takes the subsequent wordpiece in the selected wordpiece sequence as the n candidate wordpieces; if the first target index hits the first index of m1 wordpiece sequence records and If the first target index hits the first index of m1 word sequence records and If the first target index hits the first index of m1 word sequence records and m1 word sequence records, the subsequent word in the selected word sequence is taken as m1 candidate words. m1 candidate words hitting the first word statistics table and m2 candidate words hitting the second word statistics table are taken as n candidate words. m1 candidate words hitting the first word statistics table and m2 candidate words hitting the second word statistics table are taken as m1+m2 candidate words. m1 candidate words hitting the first word statistics table and m2 candidate words hitting the second word statistics table are taken as m1+m2 candidate words. If the first target index hits the first index of m1 word sequence records and If the first target index hits the first index of m1 word sequence records and m1 word sequence records, the subsequent word in the selected word sequence is taken as m1 candidate words. m1 word sequence records, the subsequent word in the selected word sequence is taken as m1 candidate words. m1 word sequence records, the subsequent word in the selected word sequence is taken as m1 candidate words. Wherein, n is a preset threshold of candidate words.

[0061] In an example embodiment of the present disclosure, the method further comprises: The control layer queries the word statistics table according to the identification of the last word, and sends a pre-fetch failure response to the processing unit if the pre-sequence word of any word sequence is not hit. If the processing unit receives the pre-fetch failure response, the processing unit reads the word embedding vector of the next word from the memory bank according to the identification of the next word, and updates the second word sequence statistics table in the following manner: adds a word sequence record in the second word sequence statistics table, the added word sequence takes the previous word of the neural network model inference as the previous word, and takes the next word of the neural network model inference as the subsequent word; performs second hash operation on the identification of the previous word to generate a third index; performs second hash operation on the identification of the subsequent word to generate a fourth index; sets an initial value for the heat score of the word sequence, the initial value being the maximum value of the heat score in the current second word sequence statistics table; and sets the timestamp of the word sequence as the creation time of the word sequence.

[0062] In an example embodiment of the present disclosure, the method further comprises: After the processing unit infers the next word using the neural network model, the processing unit updates the first word sequence statistics table and the second word sequence statistics table as follows: If the pre-fetch success response is received, the first word sequence statistics table and the second word sequence statistics table are queried according to the identification of the previous word and the identification of the candidate word, and the heat score of each word sequence in the hit is increased by a first step value; If the next word hits the candidate word, the first word sequence statistics table and the second word sequence statistics table are queried according to the identification of the previous word and the identification of the hit candidate word, and the heat score of the word sequence in the hit is increased by a second step value; wherein the second step value is greater than the first step value.

[0063] In an example embodiment of the present disclosure, the method further comprises: The processing unit periodically updates the first word sequence statistics table and the second word sequence statistics table, and performs at least one of the following processing: The heat score of the word sequence record in the first word sequence statistics table whose timestamp is more than the aging time threshold from the current time is multiplied by a decay coefficient p; ; The word sequence record with the hotness score of the word sequence in the first word statistic table less than or equal to the elimination threshold is removed from the first word statistic table to a second word statistic table, and a second hash operation is performed on the identification of the previous word of the word sequence to generate a third index, the third index is used to cover the first index in the original record, and a second hash operation is performed on the identification of the subsequent word of the word sequence to generate a fourth index, the fourth index is used to cover the second index in the original record; after removing a word sequence record from the first word statistic table each time, the word sequence record with the highest hotness score in the second word statistic table is removed from the second word statistic table to the first word statistic table, and a first hash operation is performed on the identification of the previous word of the word sequence to generate a first index, the first index is used to cover the third index in the original record, and a first hash operation is performed on the identification of the subsequent word of the word sequence to generate a second index, the second index is used to cover the fourth index in the original record.

[0064] In an example embodiment of the present disclosure, the method further comprises: word in the word dictionary is included in the first word set, and a word in the word dictionary with a frequency of access greater than or equal to a second frequency threshold and less than the first frequency threshold is included in the second word set, and a word in the word dictionary with a frequency of access less than the second frequency threshold is included in the third word set; The word embedding vector corresponding to each word in the first word set is copied k copies, and the original word embedding vector and the k copies are respectively stored in k+1 different storage bodies; wherein k is greater than 1; the word embedding vector corresponding to each word in the second word set is copied one copy, and the original word embedding vector and the one copy are respectively stored in 2 different storage bodies; the word embedding vector corresponding to each word in the third word set is stored in a storage body.

[0065] In an example embodiment of the present disclosure, when the processing unit layer includes M rows and N columns of processing units; k=M-1; The M copies of the word embedding vector corresponding to each word in the first word set are respectively stored in M different storage bodies, and different rows of processing units access different storage bodies; The 2 copies of the word embedding vector corresponding to each word in the second word set are respectively stored in 2 different storage bodies, and the first M / 2 rows of processing units access one of the storage bodies, and the last M / 2 rows of processing units access the other storage body.

[0066] Those of ordinary skill in the art will realize that the techniques taught herein can be embodied or implemented in a variety of different forms, as the software, firmware, hardware, or a combination of these different forms. In this description, the term "module" as used with reference to a functional unit or a step in the disclosed methods refers to software, firmware, hardware, or any suitable combination thereof. The term "module" as used herein refers to a piece of logic implementing the specified functionality either on a hardware or software level, and can refer to either an individual component or a combination of components, either on a hardware or software level. In hardware form, the module can be implemented by means of a hardware component such as a processor or a microprocessor, or any other hardware component, either alone or in combination with other components. In software form, the module can be implemented by means of software carried by a computer readable medium, such as a storage medium, or carried by a computer data signal, such as a bit stream or a wave signal including one or more zero or non-zero signals. The term "software" should be interpreted broadly to mean any type of instructions or set of instructions, whether referred to as software, firmware, middleware, microcode, or otherwise. In addition, the disclosure is open to the inclusion of any type of computer readable medium, either now known or in the future developed, and to a corresponding addition in the list of media in the above description.

[0067] In addition, the terms "first", "second", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of indicated technical features. Thus, the features defined with "first", "second", etc. can explicitly or implicitly include at least one of the features. In the description of the disclosure, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0068] In the description of the specification, the description using the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. means that the particular feature, structure, material or characteristic being described is contained in at least one embodiment or example of the disclosure. The illustrative representations of the above terms in the specification do not necessarily refer to the same embodiment or example. Moreover, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Furthermore, the skilled person in the art can combine and combine the features of different embodiments or examples described in the specification and the features of different embodiments or examples, without contradiction.

Claims

1. A collaborative acceleration system for accessing the embedding layer of a neural network model, characterized in that, include: Processing unit layer, control layer, routing layer, and storage layer; The processing unit layer includes at least one processing unit; the storage layer includes a cache and multiple storage units; the storage units are used to store the word embedding vector of each word in the word dictionary, and the cache includes multiple prefetch buffers, with different storage units corresponding to different prefetch buffers; The processing unit is configured to send a prefetch request to the control layer, which carries the identifier of the processing unit and the identifier of the previous word inferred by the neural network model; and, after inferring the next word using the neural network model, if a prefetch success response is received, it determines whether the next word hits a candidate word; if it does, it reads the word embedding vector of the hit candidate word from the prefetch buffer; if it does not hit, it reads the word embedding vector of the next word from the storage according to the identifier of the next word. The control layer is configured to, upon receiving a prefetch request, query the lexical statistics table based on the identifier of the previous lexical unit. If at least one preceding lexical unit of a lexical sequence is hit, a portion of the lexical sequence is selected from the hit lexical sequence according to the prefetch strategy. The subsequent lexical units in the selected lexical sequence are taken as candidate lexical units, and a first access request is sent to the routing layer, carrying the identifier of the processing unit and the identifier of the candidate lexical units. Herein, any lexical sequence includes both preceding and subsequent lexical units. The routing layer receives the first access request, reads the word embedding vector of the candidate word from the storage based on the identifier of the candidate word, saves it in the prefetch buffer corresponding to the storage, and sends a prefetch success response to the processing unit, which carries the identifier of the candidate word and the storage address of the candidate word in the prefetch buffer. The lexical statistics table is set in the cache.

2. The system according to claim 1, characterized in that: The cache also includes: a first-level cache and a second-level cache; the access speed of the first-level cache is faster than the access speed of the second-level cache; The lexical statistics table includes: a first lexical statistics table and a second lexical statistics table; the first lexical statistics table is stored in the first-level cache, and the second lexical statistics table is stored in the second-level cache.

3. The system according to claim 2, characterized in that: The first lexical statistics table includes a1 lexical sequence records; each lexical sequence record includes: the identifier of the preceding lexical, a first index generated from the identifier of the preceding lexical, the identifier of the subsequent lexical, a second index generated from the identifier of the subsequent lexical, a popularity score of the lexical sequence, and a timestamp of the lexical sequence; the timestamp is the creation time or the latest access time of the lexical sequence record; the latest access time refers to the time when the subsequent lexical in the lexical sequence was most recently read into the prefetch buffer and hit the next lexical inferred by the neural network model; the first index is the value obtained by performing a first hash operation on the identifier of the preceding lexical, and the second index is the value obtained by performing a first hash operation on the identifier of the subsequent lexical; The second lexical statistics table includes a2 lexical sequence records; each lexical sequence record includes: the identifier of the preceding lexical, a third index generated from the identifier of the preceding lexical, the identifier of the following lexical, a fourth index generated from the identifier of the following lexical, a popularity score of the lexical sequence, and a timestamp of the lexical sequence; the timestamp is the creation time or the latest access time of the lexical sequence record; the latest access time refers to the time when the following lexical in the lexical sequence was most recently read into the prefetch buffer and hit the next lexical inferred by the neural network model; The third index is a value obtained by performing a second hash operation on the identifier of the preceding word, and the fourth index is a value obtained by performing a second hash operation on the identifier of the following word. The first and second word segment statistics tables are pre-set in the cache; the first word segment statistics table includes the a1 word segments that appear most frequently during the training of the neural network model.

4. The system according to claim 3, characterized in that: Based on the identifier of the previous lexical unit, the lexical unit statistics table is queried. If at least one preceding lexical unit of a lexical unit sequence is matched, a portion of the lexical unit sequences are selected from the matched lexical unit sequences according to the prefetching strategy. The subsequent lexical units in the selected lexical unit sequences are taken as candidate lexical units, including: Perform a first hash operation on the identifier of the previous lexical unit to obtain a first target index. Then, query the first lexical unit statistics table based on the first target index. If the first target index matches the first index of m1 lexical unit sequence records and... Then, from the hit word sequence, select the n word sequence records with the highest popularity score, and use the subsequent word elements in the selected word sequence as n candidate word elements; if the first target index hits the first index of m1 word sequence records and Then, the subsequent words in the m1 word sequences are taken as m1 candidate words. A second hash operation is performed on the identifier of the previous word to obtain a second target index. The second target index is then used to query the second word statistics table. If the second target index matches the second index of the m2 word sequence records, and... Then, select the word sequence with the highest popularity score from the hit word sequence. A sequence of lexical units is recorded, and subsequent lexical units in the selected sequence are used as... There are m1 candidate words, which will match the m1 candidate words in the first word word statistics table and the candidate words in the second word word table. The candidate lexical units are grouped together as n candidate lexical units; if the second target index hits the second index of m2 lexical unit sequence records and Then, the subsequent word elements in the m2 word element sequences are taken as m2 candidate word elements, and the m1 candidate word elements that hit the first word element statistics table and the m2 candidate word elements that hit the second word element table are taken as m1+m2 candidate word elements. If the first target index does not match any word sequence record in the first word sequence statistics table, then a second hash operation is performed on the identifier of the previous word to obtain a second target index. The second target index is then used to query the second word sequence statistics table. If the second target index matches the second index of m2 word sequence records, and... Then, select the word sequence with the highest popularity score from the hit word sequence. A sequence of lexical units is recorded, and subsequent lexical units in the selected sequence are used as... There are m2 candidate lexical units; if the second target index hits the second index of m2 lexical unit sequence records and Then, the subsequent word elements in the m2 word element sequences are taken as m2 candidate word elements; Where n is a preset threshold for the number of candidate lexical units.

5. The system according to claim 3, characterized in that: The processing unit is further configured to, if a prefetch failure response is received, read the word embedding vector of the next word from the storage based on the identifier of the next word, and update the second word statistics table in the following manner: A new word sequence record is added to the second word sequence statistics table. The newly added word sequence uses the previous word inferred by the neural network model as the preceding word and the next word inferred by the neural network model as the following word. A second hash operation is performed on the identifier of the preceding word to generate a third index. A second hash operation is performed on the identifier of the following word to generate a fourth index. An initial value is set for the popularity score of the word sequence. The initial value is the maximum value of the popularity score in the current second word sequence statistics table. The timestamp of the word sequence is set to the creation time of the word sequence. The control layer is also configured to query the lexical statistics table based on the identifier of the previous lexical, and if no preceding lexical in any lexical sequence is matched, a prefetch failure response is sent to the processing unit.

6. The system according to claim 3, characterized in that: The processing unit is further configured to update the first and second word element statistics tables as follows after inferring the next word element using a neural network model: If a prefetch success response is received, the first word element statistics table and the second word element statistics table are queried according to the identifier of the previous word element and the identifier of the candidate word element. The heat score of each hit word element sequence is increased by a first step length value. If the next word element hits a candidate word element, the first word element statistics table and the second word element statistics table are queried based on the identifier of the previous word element and the identifier of the hit candidate word element. The popularity score of the hit word element sequence is increased by a second step value; wherein the second step value is greater than the first step value.

7. The system according to claim 6, characterized in that: The processing unit is also configured to periodically update the first lexical statistics table and the second lexical statistics table, and perform at least one of the following processes: Multiply the popularity score of word sequence records in the first word sequence statistics table that have a timestamp that is more than aging time threshold from the current time by a decay coefficient p. ; The word sequence records with a popularity score less than or equal to the elimination threshold in the first word sequence statistics table are removed from the first word sequence statistics table and moved to the second word sequence statistics table. A second hash operation is performed on the identifier of the preceding word of the word sequence to generate a third index, and the third index is used to overwrite the first index in the original record. A second hash operation is performed on the identifier of the subsequent word of the word sequence to generate a fourth index, and the fourth index is used to overwrite the second index in the original record. Each time a word sequence record is removed from the first word sequence statistics table, the word sequence record with the highest popularity score in the second word sequence statistics table is removed from the second word sequence statistics table and moved to the first word sequence statistics table. A first hash operation is performed on the identifier of the preceding word of the word sequence to generate a first index, and the first index is used to overwrite the third index in the original record. A first hash operation is performed on the identifier of the subsequent word of the word sequence to generate a second index, and the second index is used to overwrite the fourth index in the original record.

8. The system according to claim 1, characterized in that: The storage layer is configured to classify lexical units in the lexical dictionary whose access frequency is greater than or equal to a first frequency threshold into a first lexical set, lexical units in the lexical dictionary whose access frequency is greater than or equal to a second frequency threshold and less than the first frequency threshold into a second lexical set, and lexical units in the lexical dictionary whose access frequency is less than the second frequency threshold into a third lexical set. Make k copies of the word embedding vector corresponding to each word in the first word set, and store the original word embedding vector and the k copies in k+1 different storage entities; where k is greater than 1; make one copy of the word embedding vector corresponding to each word in the second word set, and store the original word embedding vector and one copy in two different storage entities. The word embedding vector corresponding to each word in the third word set is stored in the storage.

9. The system according to claim 8, characterized in that: When the processing unit layer includes M rows and N columns of processing units; k = M - 1; In the first lexical set, the M word embedding vectors corresponding to each lexical are stored in M ​​different storage bodies, and the processing units in different rows access different storage bodies; In the second lexical set, the two word embedding vectors corresponding to each lexical are stored in two different storage units. The processing unit of the first M / 2 rows accesses one storage unit, and the processing unit of the last M / 2 rows accesses the other storage unit.

10. A method for collaborative acceleration of embedding layer access in a neural network model, comprising: The processing unit sends a prefetch request to the control layer, which carries the identifier of the processing unit and the identifier of the previous word inferred by the neural network model; Upon receiving a prefetch request, the control layer queries the lexical statistics table based on the identifier of the previous lexical unit. If at least one preceding lexical unit of a lexical sequence is matched, a portion of the lexical sequence is selected from the matched lexical sequences according to the prefetch strategy. The subsequent lexical units in the selected lexical sequences are taken as candidate lexical units, and a first access request is sent to the routing layer, carrying the identifier of the processing unit and the identifier of the candidate lexical units. Each lexical sequence includes both preceding and subsequent lexical units. When the routing layer receives the first access request, it reads the word embedding vector of the candidate word from the storage based on the identifier of the candidate word and saves it in the prefetch buffer corresponding to the storage. It then sends a prefetch success response to the processing unit, which carries the identifier of the candidate word and the storage address of the candidate word in the prefetch buffer. After the processing unit infers the next lexical unit using the neural network model, if it receives a prefetch success response, it determines whether the next lexical unit matches a candidate lexical unit. If it matches, it reads the word embedding vector of the matched candidate lexical unit from the prefetch buffer; if it does not match, it reads the word embedding vector of the next lexical unit from the storage unit according to the identifier of the next lexical unit. The storage unit is used to store the word embedding vector of each lexical unit in the lexical dictionary. Each storage unit corresponds to a prefetch buffer. The lexical statistics table and the prefetch buffer of each storage unit are both set in the cache.

Images