Deep learning embedding data efficient processing system and method for heterogeneous memory devices

Abstract

The present invention provides a deep learning Embedding data efficient processing system and method for heterogeneous memory devices, the system includes three modules, wherein the Embedding data placement module is used to classify the Embedding data and pre-added data and place them on NVM or DRAM; the efficient index establishment module is used to establish indexes for the arranged data; the Embedding operation running module uses the established index to quickly locate the Embedding data involved in the request and perform normal Embedding operations. The present invention uses the hot and cold characteristics and the packaged appearance characteristics of deep learning Embedding data to place data on heterogeneous memory devices; and establishes a lightweight index to efficiently serve task requests. The system can maximize the use of DRAM and NVM space and improve the system's processing efficiency for Embedding data.

Description

Technical Field

[0001] The present invention relates to the field of deep learning in computer science, and in particular to an efficient processing system for deep learning embedding data. Background Art

[0002] The innovation of deep learning technology has greatly promoted the development of computer vision, natural language processing, medicine and other fields, and has received widespread attention in academia and industry. In the field of deep learning, embedding is widely used to represent data due to its powerful representation ability. At present, due to the large amount of embedding data and the large storage space required, the existing memory can no longer meet the needs of applications from the perspective of economy and device capacity. With the continuous innovation of memory technology, non-volatile memory (NVM) has brought new solutions to the problem of insufficient capacity due to its large capacity, low price, byte addressing ability and high read and write bandwidth (relative to disk). However, NVM has poor random read performance. A large amount of irregular and random embedding data access sharply reduces the operating efficiency of the NVM system (heterogeneous memory system of DRAM+NVM). Therefore, it is particularly important to design an efficient processing system for deep learning embedding data for heterogeneous memory devices to improve the operating efficiency of applications. Summary of the invention

[0003] In order to solve the performance degradation problem caused by a large amount of irregular and random Embedding data access, the present invention proposes a deep learning Embedding data efficient processing system for heterogeneous memory devices. By analyzing and deciding on data of different categories and different popularity, storing them on media with different performance, and designing an efficient index structure, the system's processing efficiency for Embedding data is improved.

[0004] The present invention includes two inventive points:

[0005] 1. Intelligent placement of Embedding data;

[0006] The present invention divides the deep learning data set into a training set and a test set, wherein the training set is known data and is used for model training, while the test set is unknown data and is used for model reasoning. Since the training set and the test set have similar data access patterns, the training set is selected for data analysis operations, and the test set is used for effect verification.

[0007] According to the obvious hot and cold distribution characteristics of Embedding data, the present invention first sorts the data according to the access frequency, and selects the top h% of the data as hot data (this h% can be customized by the user).

[0008] The present invention pre-adds and processes data for frequent item sets and stores them based on the presence of frequent items in deep learning Embedding data (multiple Embedding data are often accessed together).

[0009] 2. Establish efficient indexes for placed data;

[0010] After placing different types of Embedding data, the system needs to create a lightweight index for the data (occupying a small amount of memory space overhead) and implement a fast search function after a request arrives. Specifically:

[0011] The sorting ID is used as the index of a single data, and the stored pre-added data is indexed by the sorting ID set of the corresponding Embedding, i.e., the closed frequent item set. At the same time, the packaging relationship of each Embedding in the closed frequent item is represented according to the idea of ​​the adjacency matrix.

[0012] Specifically, the technical solution adopted by the present invention is as follows:

[0013] A method for efficiently processing deep learning embedding data for heterogeneous memory devices includes the following steps:

[0014] (1) Sort the Embedding data in the training set in descending order of access frequency and set the sorting ID. The sorted data is divided into hot data and cold data. All hot data is stored in DRAM, and all cold data is stored in NVM. Closed frequent item mining is performed on hot data to obtain closed frequent item sets. Finally, the pre-summed data of closed frequent item sets with a packing frequency greater than 3 are stored in NVM, and the pre-summed data of closed frequent item sets with a packing frequency less than or equal to 3 are stored in DRAM.

[0015] (2) According to the idea of ​​adjacency matrix, the packaging relationship of each embedding in the closed frequent itemset is represented. Each embedding will point to the embeddedness connected to it in the closed frequent itemset.

[0016] (3) Processing the request: Sort the Embedding data in the request in ascending order according to the sorting ID and determine whether each Embedding data in the request is hot or cold data, where:

[0017] For the hot data in the request, according to the packing relationship established in step (2), determine whether the embeddings belonging to the hot data have a packing relationship based on the sorting results. If the consecutive embeddings have a packing relationship and the number of packings is greater than 3, query the NVM based on the sorting IDs of the consecutive embeddings to obtain the corresponding pre-summed data. If the number of packings is less than or equal to 3, query the DRAM based on the sorting IDs of the consecutive embeddings to obtain the corresponding pre-summed data. If there is no packing relationship, directly query the DRAM to obtain the corresponding embedding data until the query is complete.

[0018] For cold data in the request, directly query NVM based on the sorting ID to obtain the corresponding Embedding data until the query is complete.

[0019] Furthermore, in the step (1), the front part of the cold data is divided into warm data, and all the Embedding data in all the warm data are packaged according to the sorting ID and the use superset and stored in the NVM, and the maximum packaging granularity M of the current superset and the sorting ID I of the corresponding maximum packaged Embedding are recorded; when processing the warm data in the request, the Embedding data with a sorting ID less than I queries M pre-added data; for the data with a sorting ID greater than I, the query is performed with a granularity of M-1.

[0020] Furthermore, the original data is further divided into the first h% of hot data, the middle w% of warm data and the last c% of cold data (where h, w and c can all be customized by the user). Preferably, the hot data accounts for 5%, the warm data accounts for 5% to 20%, and the rest is cold data.

[0021] Furthermore, the step (2) further includes the steps of sorting the closed frequent itemsets and reallocating the hot data sorting IDs according to the closed frequent itemsets, as follows:

[0022] Closed frequent items are sorted in descending order according to the number of Embedding data in them, and closed frequent items with the same number of Embedding data are sorted in descending order according to the probability of occurrence of closed frequent items.

[0023] Reassign hot data sorting IDs according to the order in which Embeddings appear in the closed frequent item set sorting.

[0024] Furthermore, in step (2), the packaging relationship of each Embedding in the closed frequent item set is represented according to the idea of ​​the adjacency matrix. Each Embedding will point to the Embedding connected to it in the closed frequent item set, specifically:

[0025] A bitmap is used to represent the packing relationship of each Embedding data, where the existence of a packing relationship is represented by 1, and the non-existence of a packing relationship is represented by 0. The packing relationship of each Embedding data only records the packing relationship between the Embedding data whose sorting ID is greater than the Embedding data and the Embedding data.

[0026] Furthermore, in step (3), if the corresponding pre-summed data is not found by querying NVM or DRAM based on the sorting IDs of the consecutive Embeddings, the largest sorting ID in the sorting IDs of the consecutive Embeddings is removed one by one and the query is repeated until the pre-summed data is obtained.

[0027] A deep learning embedding data efficient processing system for heterogeneous memory devices according to the above method comprises:

[0028] The Embedding data placement module is used to sort the Embedding data in the training set in descending order of access frequency and set the sorting ID, and divide the sorted data into hot data and cold data. All hot data is stored in DRAM, and all cold data is stored in NVM; at the same time, closed frequent items are mined for hot data to obtain closed frequent item sets, and the pre-summed data of closed frequent item sets with a packing frequency greater than 3 are stored in NVM, and the pre-summed data of closed frequent item sets with a packing frequency less than or equal to 3 are stored in DRAM.

[0029] An efficient index building module is used to represent the packaging relationship of each Embedding in a closed frequent itemset according to the idea of ​​an adjacency matrix. Each Embedding will point to the Embedding connected to it in the closed frequent itemset.

[0030] The Embedding operation running module uses the established index and packaging relationship to quickly locate the Embedding data involved in the request and perform normal Embedding operations.

[0031] The beneficial effects of the present invention are as follows: the present invention proposes an efficient processing system for deep learning Embedding data for heterogeneous memory devices, designs an intelligent data placement mode for heterogeneous memory devices, and establishes a lightweight index to efficiently serve task requests. The system can maximize the use of DRAM and NVM space and improve the system's processing efficiency for Embedding data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the system framework module diagram;

[0033] Figure 2This is a schematic diagram of the Embedding operation;

[0034] Figure 3 CDF graph of Embedding data access frequency;

[0035] Figure 4 Get a schematic diagram for closed frequent item data (pre-summed data);

[0036] Figure 5 Schematic diagram for the classification of Embedding data types;

[0037] Figure 6 A schematic diagram comparing the latency of a single NVM access and multiple DRAM accesses;

[0038] Figure 7 This is a schematic diagram of the Embedding data layout;

[0039] Figure 8 Diagram of reassigning IDs for Embedding data;

[0040] Fig. 9 is a schematic diagram of the index;

[0041] Fig.10 Schematic diagram of request processing. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with specific embodiments according to the accompanying drawings:

[0043] The present invention proposes an efficient processing method for deep learning Embedding data for heterogeneous memory devices, which utilizes the hot and cold characteristics and packaged appearance characteristics of deep learning Embedding data to place data on heterogeneous memory devices; and establishes a lightweight and efficient index structure; and designs the Embedding processing flow to improve the operational efficiency of Embedding data.

[0044] Based on the above method, the present invention also provides a deep learning Embedding data efficient processing system for heterogeneous memory devices. The system framework module diagram is as follows: Figure 1 As shown in the figure, it includes three modules, among which the Embedding data placement module is used to classify the Embedding data and the pre-added data and place them on NVM or DRAM; the efficient index establishment module is used to index the placed data; the Embedding operation running module uses the established index to quickly locate the Embedding data involved in the request and perform normal Embedding operations. Figure 2As shown, the system will randomly read multiple Embedding data in the storage device and perform addition operations.

[0045] The processing process of the present invention is specifically as follows:

[0046] (1) Data placement:

[0047] like Figure 3 As shown in the figure, since many Embedding data have hot and cold characteristics (some data are frequently accessed), and these hot data frequently appear together (frequent items), we can perform hot and cold analysis on the training Embedding data, and arrange the data reasonably in DRAM and NVM. At the same time, we can obtain these frequent item data and store them after pre-addition and operation, as follows:

[0048] Sort the Embedding data in the training set in descending order of access frequency and set the sorting ID, and divide the sorted data into hot data and cold data. Store all hot data in DRAM and all cold data in NVM.

[0049] Closed frequent item mining is performed on hot data to obtain closed frequent item sets. Finally, the pre-summed data of closed frequent item sets with a packing number greater than 3 are stored in NVM, and the pre-summed data of closed frequent item sets with a packing number less than or equal to 3 are stored in DRAM.

[0050] (2) Establish an index for the arranged data: According to the idea of ​​adjacency matrix, the packaging relationship of each embedding in the closed frequent item is represented. Each embedding will point to the embedding connected to it in the closed frequent item. And establish the index of each closed frequent item according to the embedding data sorting ID.

[0051] (3) When an inference task arrives, use the index to search for embedding data and perform embedding operations;

[0052] As a preferred solution, the step (1) can be specifically divided into the following sub-steps for acquisition:

[0053] like Figure 4 As shown in step 1, the system first sorts the data in descending order according to the access frequency, and the user selects a threshold to divide the data into hot data and cold data.

[0054] Step 2: Remove cold data from the request.

[0055] Step 3: Pass the request containing only hot data into the closed frequent item mining algorithm (MBEA algorithm or MMBEA algorithm).

[0056] Step 4: Get many closed frequent item sets, and first sort them in descending order according to the number of data in the frequent items. Closed frequent items with the same number of data are sorted in descending order according to the probability of occurrence.

[0057] Furthermore, part of the cold data can be further selected as warm data according to the threshold set by the user, such as Figure 5 As shown. In order to make full use of the space of DRAM devices, all hot data is stored in DRAM, and all warm data and cold data are stored in NVM. For the pre-summed data of closed frequent itemsets, reasonable placement is performed based on the following observations:

[0058] according to Figure 6 It can be seen that the latency of one NVM access is slightly greater than three DRAM accesses. In order to make the overall performance of the Embedding operation after using NVM greater than or equal to that of the device using full DRAM, only the pre-added data with more than 3 packing times is placed in the NVM, and the pre-added data with less than or equal to 3 packing times is placed in the DRAM. All cold data is stored in the slow NVM device. Since warm data may also be pre-added, it is chosen to use a superset to fully pack it (that is, all data from 2 to n are pre-added until the storage threshold set by the user is reached, and the current superset maximum packing granularity M and the corresponding maximum packed Embedding number I are recorded). In addition, it is also necessary to record the sorting ID range of hot data, warm data, and cold data for subsequent search. The final data placement effect is as follows. Figure 7 shown.

[0059] As a preferred solution, the index establishment process in step (2) is specifically as follows:

[0060] In order to facilitate the establishment of the index, we first need to reassign the hot data sorting ID to the Embedding data in the order in which it appears in the sorted frequent items, such as Figure 8 shown.

[0061] Then, we create an index based on the closed frequent items. The basis for this is: according to the idea of ​​the adjacency matrix, we represent the packaging relationship of each embedding in the closed frequent items. Each embedding will point to the embedding connected to it in the closed frequent items. Figure 8 As shown, take Embedding-1 data as an example. The data connected to Embedding-1 are Embedding-2, Embedding-3 and Embedding-8, so it is necessary to use an index to record their connection relationship. In addition, for the packaging of multiple data, such as 12 6 9, it is necessary to connect 1 to 2, 2 to 6, and 6 to 9. Preferably, in order to further reduce the index space overhead, a bitmap with bit granularity is used to represent it, such as Fig. 9As shown in the figure, [1] represents the sorting ID of the Embedding-1 corresponding to the record. Then the connection relationship between the Embedding data with an ID greater than 1 and Embedding-1 is recorded in order, where the corresponding bit in the bitmap is 1 for connection and 0 for disconnection.

[0062] Finally, the packaged data is pre-summed and stored using hashing. When storing, the pre-summed data uses the corresponding Embedding sorted ID set, i.e., the closed frequent itemset, as the index. For example:

[0063] 1 3 5 is a closed frequent item set. E is summed up in advance (E = Embedding-1 + Embedding-3 + Embedding-5), and then E is indexed using hash H ("1 3 5").

[0064] As a preferred solution, in step (3), when an inference task arrives, the specific process of request processing is as follows:

[0065] All embeddings involved in the new request are mapped according to the new sorting ID, and their sorting IDs are sorted from small to large.

[0066] like Fig.10 As shown in the figure, when a new request comes in, it is known that the ID in the request belongs to hot, warm or cold data. The specific search includes:

[0067] For hot data, according to the established packing relationship, the sorting results are used to determine whether the embeddings belonging to the hot data have a packing relationship. If the consecutive embeddings have a packing relationship and the number of packings is greater than 3, the NVM is queried based on the sorting IDs of the consecutive embeddings to obtain the corresponding pre-summed data. If the number of packings is less than or equal to 3, the DRAM is queried based on the sorting IDs of the consecutive embeddings to obtain the corresponding pre-summed data. If there is no packing relationship, the DRAM is directly queried to obtain the corresponding embedding data until the query is complete.

[0068] by Fig.10For example, for 1, check whether it is connected to 2, whether 2 is connected to 6, and whether 6 is connected to 9. When checking whether 9 is connected to 10, the result is that they are not connected, so the previously connected data 1 2 6 9 are hashed and searched with the sorting ID. Since the packing granularity is greater than 3 (less than or equal to 3 is searched on DRAM), the search is performed on NVM. Similarly, the rest of the hot data is searched in this way. However, if the specified data is not found in the device after hashing, then fall back to an Embedding and hash it again, and search (special low-probability situation). For example, if the hash value of 1 2 3 4 is not found, then fall back to search the hash value of 1 2 3.

[0069] For warm data, in this scenario, since all warm data are packed using a superset, when the maximum packing granularity M of the current superset stored is 2, 2 pre-addition operations are stored, so 15 22, 28 33 and 39 are searched in pairs from the NVM in turn.

[0070] Step 3: For cold data, search directly on NVM according to the sorting ID one by one until the query is complete.

[0071] The following is a specific embodiment to further illustrate the beneficial effects of the present invention:

[0072] The specific experiments are as follows:

[0073] Experimental configuration:

[0074] (1) Operating system: Ubuntu 18.04.3LTS;

[0075] (2) CPU: 8-core Intel(R) Xeon(R) Gold 6126 CPU @ 2.60 GHz, with 32 GB DRAM;

[0076] (3) Storage devices: 512GB, SK hynix SC311 SATA SSD; Western Digital WDCWD40EZRZ-75G HDD; Intel Optane NVM 256GB;

[0077] Final test results:

[0078] Comparative solution: using a random solution to store the Embedding data indiscriminately on DRAM and NVM; the solution of the present invention: dividing the data according to hot and cold characteristics and using packaged storage; the method of the present invention improves the total access performance on the shared data set MovieLens by 1.5 times compared with the comparative solution.

[0079] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A deep learning embedding data efficient processing method for heterogeneous memory devices, characterized in that: The following steps are involved: (1) Sort the Embedding data in the training set in descending order of access frequency and set a sorting ID. The sorted data is divided into hot data and cold data. All hot data is stored in DRAM, and all cold data is stored in NVM. Perform closed frequent item mining on hot data to obtain closed frequent item sets. Finally, store the pre-summed data of closed frequent item sets with a packing number greater than 3 in NVM, and store the pre-summed data of closed frequent item sets with a packing number less than or equal to 3 in DRAM. (2) According to the idea of ​​adjacency matrix, the packaging relationship of each embedding in the closed frequent item is expressed. Each embedding will point to the embedding connected to it in the closed frequent item. Specifically: A bitmap is used to represent the packaging relationship of each Embedding data, where the existence of a packaging relationship is represented by 1, and the non-existence of a packaging relationship is represented by 0. The packaging relationship of each Embedding data only records the packaging relationship between the Embedding data whose sorting ID is greater than the Embedding data and the Embedding data; (3) Processing the request: Sort the Embedding data in the request in ascending order according to the sorting ID and determine whether each Embedding data in the request is hot or cold data, where: For the hot data in the request, according to the packing relationship established in step (2), determine whether the embeddings belonging to the hot data have a packing relationship one by one according to the sorting results. If the consecutive embeddings have a packing relationship and the number of packings is greater than 3, query the NVM according to the sorting ID of the consecutive embeddings to obtain the corresponding pre-summed data. If the number of packings is less than or equal to 3, query the DRAM according to the sorting ID of the consecutive embeddings to obtain the corresponding pre-summed data. If there is no packing relationship, directly query the DRAM to obtain the corresponding embedding data until the query is complete. For cold data in the request, directly query NVM based on the sorting ID to obtain the corresponding Embedding data until the query is complete.

2. The method according to claim 1, characterized in that In the step (1), the front part of the cold data is divided into warm data, all the Embedding data in all the warm data are packaged according to the sorting ID and the use superset and stored in the NVM, and the maximum packaging granularity M of the current superset and the sorting ID I of the corresponding maximum packaged Embedding are recorded; when processing the warm data in the request, the Embedding data with a sorting ID less than I is queried for M pre-summed data; for the data with a sorting ID greater than I, the query is performed with a granularity of M-1.

3. The method according to claim 2, characterized in that The hot data accounts for 5%, the warm data accounts for 5% to 20%, and the rest is cold data.

4. The method according to claim 1, characterized in that: The step (2) also includes the steps of sorting the closed frequent itemsets and reallocating hot data sorting IDs according to the closed frequent itemsets, which are specifically as follows: Sort closed frequent items in descending order according to the number of Embedding data in them. Closed frequent items with the same number of Embedding data are sorted in descending order according to the probability of occurrence of closed frequent items. Reassign hot data sorting IDs according to the order in which Embeddings appear in the closed frequent item set sorting.

5. The method according to claim 1, characterized in that In the step (3), if the corresponding pre-summed data is not found by querying the NVM or DRAM according to the sorting IDs of the consecutive Embeddings, the largest sorting ID in the sorting IDs of the consecutive Embeddings is removed one by one and the query is repeated until the pre-summed data is obtained.

6. A deep learning embedding data efficient processing system for heterogeneous memory devices according to any one of claims 1 to 5, characterized in that: include: The Embedding data placement module is used to sort the Embedding data in the training set in descending order of access frequency and set the sorting ID, and divide the sorted data into hot data and cold data; all hot data is stored in DRAM, and all cold data is stored in NVM; at the same time, closed frequent items are mined for hot data to obtain closed frequent item sets, and the pre-summed data of closed frequent item sets with a packing frequency greater than 3 are stored in NVM, and the pre-summed data of closed frequent item sets with a packing frequency less than or equal to 3 are stored in DRAM; The efficient index building module is used to represent the packaging relationship of each Embedding in the closed frequent item according to the idea of ​​the adjacency matrix. Each Embedding will point to the Embedding connected to it in the closed frequent item. Specifically: A bitmap is used to represent the packaging relationship of each Embedding data, where the existence of a packaging relationship is represented by 1, and the non-existence of a packaging relationship is represented by 0. The packaging relationship of each Embedding data only records the packaging relationship between the Embedding data whose sorting ID is greater than the Embedding data and the Embedding data; The Embedding operation running module uses the established index and packaging relationship to quickly locate the Embedding data involved in the request and perform normal Embedding operations.

Images