Method, device, equipment and storage medium for embedding parameter cache

Abstract

The present application provides a method, apparatus, device and storage medium for embedding parameter caching. The method obtains the embedding parameters of at least one embedding table into the CPU memory and stores some hot spots of the embedding parameters into the GPU memory. Based on the coding mechanism of Huffman coding, the feature ID of the at least one embedding table is encoded to generate a global key with a unified format and global uniqueness. The global embedding cache table is used as the storage form of the GPU cache, and the global embedding cache table uses the global key as the key to index the embedding table. The unified indexing technology is used to offload the index queries in the preset number of CPU memories to the GPU memory. When reading and writing the cache, the copy operation of the embedded parameters is decoupled from the index operation of the cache. The adoption of this technical solution can solve the embedded access requirements and alleviate the performance problems during model estimation.

Description

Technical Field

[0001] The present application relates to the field of recommendation technology, and in particular to a method, apparatus, device, and storage medium for embedding parameter cache. Background Art

[0002] Recommendation systems filter information from massive amounts of content and generate customized information streams for users.

[0003] The quality of recommendations in a recommendation system is determined by two factors. From an algorithmic perspective, improved model prediction accuracy leads to more accurate predictions of user interests. Alternatively, from a systemic perspective, a system with low latency and high throughput allows the same model to examine more candidate content while maintaining the same service level agreement, thereby increasing the likelihood of finding content that is of interest to the user.

[0004] Emerging deep learning recommendation models (DLRMs) are used in nearly all current recommendation systems. DLRMs typically consist of two components: a large embedding layer, which can consume over 100GB of memory, and a fully connected layer, which can consume approximately 100MB of memory. The embedding layer contains several embedding tables that map high-dimensional categorical feature inputs (IDs) to underlying low-dimensional dense vector representations (called embedding vectors). These embedding vectors contain a number of parameters (called embedding parameters) per dimension (called embedding dimensions). It's important to note that the size and embedding dimensions of different embedding tables typically vary.

[0005] Due to the limited memory capacity of GPUs, some existing systems typically store these embedding tables in the CPU's DRAM as hash tables. However, the drawback of this approach is that the limited DRAM bandwidth on the CPU cannot meet the large number of random sparse accesses required by the embedding layer. This results in significant time overhead for the embedding layer, becoming a major performance bottleneck for recommendation model estimation.

[0006] Therefore, a method for embedding parameter cache is needed to solve the embedded access requirements and alleviate the performance issues during model estimation. Summary of the Invention

[0007] The present application provides a method, apparatus, device and storage medium for embedding parameter cache, which can solve embedded access requirements and alleviate performance issues during model estimation.

[0008] In a first aspect, the present application provides a method for embedding a parameter cache, comprising:

[0009] Obtaining embedding parameters of at least one embedding table into CPU memory, and storing some hot spots of the embedding parameters into GPU memory;

[0010] Based on a Huffman coding mechanism, encoding the feature ID of the at least one embedded table to generate a globally unique global key with a unified format;

[0011] A global embedded cache table is used as a storage form of GPU cache, wherein the global embedded cache table uses the global key as a key to index the embedded table;

[0012] Using unified indexing technology, a preset number of index queries in CPU memory are offloaded to GPU memory;

[0013] When reading and writing the cache, decouple the copying of embedded parameters from the indexing of the cache.

[0014] In a second aspect, the present application provides a device for embedding parameter cache, comprising:

[0015] an acquiring unit, configured to acquire embedding parameters of at least one embedding table into a CPU memory, and store some hot spots of the embedding parameters into a GPU memory;

[0016] a generating unit, configured to generate a globally unique global key having a unified format by encoding the feature ID of the at least one embedded table based on a Huffman coding mechanism;

[0017] An indexing unit, configured to use a global embedded cache table as a storage form of GPU cache, wherein the global embedded cache table uses the global key as a key to index the embedded table;

[0018] An offloading unit, configured to offload a preset number of index queries from the CPU memory to the GPU memory using a unified indexing technique;

[0019] The decoupling unit is used to decouple the copy operation of the embedded parameters from the cache index operation when reading and writing the cache.

[0020] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0021] The memory stores computer-executable instructions;

[0022] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0024] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the method described in the first aspect when executed by a processor.

[0025] The present application provides a method, apparatus, device and storage medium for embedding parameter caching, which obtains the embedding parameters of at least one embedding table into the CPU memory and stores some hot spots of the embedding parameters into the GPU memory;

[0026] Based on a Huffman coding mechanism, the feature ID encoding of at least one embedding table is generated into a globally unique global key with a unified format. A global embedding cache table is used as the GPU cache storage format, and the global embedding cache table uses the global key as the key to index the embedding table. A unified indexing technique is used to offload index queries from a preset number of CPU memories to GPU video memory. When reading and writing the cache, the copy operation of the embedding parameters is decoupled from the cache indexing operation. This technical solution can solve the embedded access requirements and alleviate performance issues during model estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0028] Figure 1 This is a schematic diagram of a system framework of an embedded parameter caching method provided in accordance with the first embodiment of the present application;

[0029] Figure 2 This is a flow chart of an embedded parameter caching method provided according to the first embodiment of the present application;

[0030] Figure 3 This is a flow chart of an embedded parameter caching method provided according to the second embodiment of the present application;

[0031] Figure 4 is a schematic diagram of a global key provided according to Example 2 of the present application;

[0032] Figure 5 This is a schematic diagram of a global embedded cache table provided according to the second embodiment of the present application;

[0033] Figure 6 is a schematic diagram of a global index provided according to the second embodiment of the present application;

[0034] Figure 7 is a schematic diagram of an indexing operation provided according to the second embodiment of the present application;

[0035] Figure 8This is a schematic diagram of a system executing a query according to the second embodiment of the present application;

[0036] Figure 9 is a schematic diagram of system decoupling provided according to the second embodiment of the present application;

[0037] Figure 10 is a schematic diagram of an embedded parameter cache device provided according to the third embodiment of the present application;

[0038] Figure 11 The figure is a block diagram of a terminal device according to an exemplary embodiment.

[0039] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0041] The embedded parameter caching method provided in this application is intended to solve the above technical problems in the prior art.

[0042] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0043] Figure 1 The system framework diagram of an embedded parameter caching method provided in Example 1 of the present application is shown in FIG. The system in Example 1 includes: CPU memory and GPU memory.

[0044] Specifically, Figure 2 This is a flow chart of an embedded parameter caching method provided in accordance with the first embodiment of the present application, comprising the following steps:

[0045] S201: Obtain embedding parameters of at least one embedding table into a CPU memory, and store some hot spots of the embedding parameters into a GPU memory.

[0046] In this embodiment, the CPU memory can store all embedding parameters from each embedding table in any form; the GPU memory is used to cache some hotspot embedding parameters, which are pre-trained by the recommendation model.

[0047] S202: Based on a Huffman coding mechanism, encode the feature ID of the at least one embedded table to generate a globally unique global key with a unified format.

[0048] In this embodiment, the cache system encodes the feature IDs from each embedding table into a globally unique global key with a unified format through an encoding mechanism based on Huffman coding.

[0049] S203: Using a global embedded cache table as a storage form of GPU cache, wherein the global embedded cache table uses the global key as a key to index the embedded table.

[0050] In this embodiment, the cache system adopts a global embedding cache table as a storage form of GPU cache, and the global embedding cache table uses the global key as a key to index the embedding parameter caches from all embedding tables.

[0051] S204: Unload a preset number of index queries in the CPU memory to the GPU memory using a unified indexing technology.

[0052] S205 : When reading and writing the cache, decouple the copy operation of the embedded parameters from the index operation of the cache.

[0053] In this embodiment, when the cache system reads and writes the cache, the copy operation of the embedded parameters is decoupled from the index operation of the cache; it can be understood that the embedded cache system is used to store multiple embedded tables. Figure 1 As shown in Figure 1, the embedded cache system consists of two components: CPU memory and GPU memory. The embedded cache system operates in user mode. It provides user interfaces for embedding parameter query and embedding parameter update. Assuming the number of embedding tables is n, the embedding parameter query interface inputs n feature ID lists (corresponding to n embedding tables) and outputs n embedding parameter matrices. The embedding parameter update interface inputs n ID lists and the corresponding embedding parameters and outputs whether the update was successful.

[0054] The present application provides a method, apparatus, device, and storage medium for embedding parameter caching, which obtains embedding parameters of at least one embedding table into CPU memory and stores some hot spots of the embedding parameters into GPU memory;

[0055] Based on a Huffman coding mechanism, the feature ID encoding of at least one embedding table is generated into a globally unique global key with a unified format. A global embedding cache table is used as the GPU cache storage format, and the global embedding cache table uses the global key as the key to index the embedding table. A unified indexing technique is used to offload index queries from a preset number of CPU memories to GPU video memory. When reading and writing the cache, the copy operation of the embedding parameters is decoupled from the cache indexing operation. This technical solution can solve the embedded access requirements and alleviate performance issues during model estimation.

[0056] Figure 3 This is a flow chart of an embedded parameter caching method provided in accordance with the second embodiment of the present application, comprising the following steps:

[0057] S301: Obtain embedding parameters of at least one embedding table into a CPU memory, and store some hot spots of the embedding parameters into a GPU memory.

[0058] In this embodiment, the CPU memory can store all embedding parameters from each embedding table in any form; the GPU memory is used to cache some hotspot embedding parameters, which are pre-trained by the recommendation model.

[0059] S302. For the FeatureID-th feature of the TableID-th embedded table, obtain the corresponding global key according to the following formula:

[0060]

[0061]

[0062] Among them, m represents the bit length of TableID after Huffman encoding, n represents the bit length of the global key, huffman represents Huffman coding mapping, global key (TableID, FeatureID) represents the global key encoded by the encoding mechanism, % represents the modulo operation, and hash represents the hash operation.

[0063] Specifically, during initialization, please refer to Figure 4 (a) The system uses the number of feature IDs in each input embedding table as the corresponding weight (also known as frequency) in Huffman coding, performs Huffman coding, and stores the result in a coding dictionary. This coding dictionary is implemented as a hash table that stores the mapping between embedding table IDs and corresponding extended codes, as well as the mapping between the corresponding extended code bits.

[0064] When coding, please refer to Figure 4(b) The global key is composed of the embedding table ID translated by the encoding dictionary and the hashed feature ID. It is important to note that when encoding the global key, we only need to query the encoding dictionary once for all IDs from the same embedding table, because these IDs correspond to the same embedding table. Furthermore, since there are not many embedding tables, the encoding dictionary is small and can usually be cached in the CPU cache, ensuring fast query speeds.

[0065] S303: Using a global embedded cache table as a storage form of GPU cache, wherein the global embedded cache table uses the global key as a key to index the embedded table.

[0066] In this embodiment, the cache system adopts a global embedding cache table as a storage form of GPU cache, and the global embedding cache table uses the global key as a key to index the embedding parameter caches from all embedding tables.

[0067] In one example, the global embedding cache table consists of two parts: a global index and a memory pool. The global index uses a dictionary data structure to maintain a mapping relationship between the global key and the corresponding embedding parameter storage address, where the embedding parameter storage address is a GPU memory address or a CPU memory address. The memory pool manages the allocation and release of memory space, and includes the same number of sub-pools as the embedding table, and each sub-pool contains several parameters corresponding to the dimensions of the embedding table.

[0068] Specifically, the structure of the global embedded cache table is as follows: Figure 5 As shown in the figure, the system uses global keys as an intermediate layer to logically abstract a multi-embedded table cache interface for users. Internally, only a single cache table is physically maintained to improve cache utilization. By enabling each global key to enter and exit, the caches of all embedded tables can be logically elastically scaled, capturing global hotspots and improving cache hit rates.

[0069] The global index uses a scalable GPU hash structure. Please refer to Figure 6The global index consists of multiple Slab Sets, each containing four Slabs. Each Slab contains a hash table entry equal to the number of GPU warp threads (32 in this example). Each hash table entry records a 64-bit global key and the address of the corresponding embedded parameters in GPU or CPU memory, as well as a 32-bit version number used to implement the least recent replacement (LRU) algorithm. The storage is performed by first placing 32 global keys, then 32 addresses, and finally 32 version numbers in a row, allowing GPU threads to access the involved global memory using coalesced access. Each global key is mapped to only one Slab Set, and hash conflicts are resolved using a linear exploration within the Slab Set. When accessing the global index, the system maintains a list of global keys to be queried as a work queue, with each warp fetching one global key from the work queue at a time for index access.

[0070] The memory pool inherits the existing slab allocator. The system sets the size of each slab based on the embedding dimensions predefined in the embedding table to avoid memory fragmentation. The dimensions of each embedding vector do not need to be recorded in the global index or the memory pool, because the feature ID to be queried, bound to the embedding table, implicitly includes dimension size information. To avoid the high latency of the GPU memory allocation API, the system pre-allocates a large block of memory for the memory pool during initialization, allowing for more refined management at runtime.

[0071] In one example, the global embedding cache table maintains a version number for each global key in the table. When a new embedding parameter is added to the cache or a cached embedding parameter is accessed by a user, the version number of the corresponding global key is updated.

[0072] Specifically, the system maintains a global version number that atomically increments by one upon each embedded parameter query and update operation. The global key version number is updated to the current global version number each time. During each operation, the global version number is cached in the GPU's shared memory to avoid the high latency associated with multiple global memory accesses. In particular, when the global version number is about to overflow (approximately 2^32 query and update operations), the system subtracts a preset constant from the global version number and all version numbers. If the result is negative, it is set to 0.

[0073] In one example, the version number is used to control cache eviction: the system uses the version number to implement an approximate least recently used cache eviction algorithm; when the video memory usage exceeds a preset threshold t1, the system scans the global index and continuously releases the least recently used embedded parameter cache according to the version number and returns it to the video memory pool until the video memory usage falls below another preset threshold t2, where 0 <t2<t1<1。

[0074] Specifically, the cache eviction is triggered in two situations: First, when there are no free entries in the Slab mapped to the newly inserted global key, the system finds the entry with the smallest version number from the Slab to replace it, and evicts and releases the cache corresponding to the original entry; Second, when the video memory usage exceeds the preset threshold t1, the system scans the global index and first divides the valid entries into coarse buckets according to the version number, and then releases all cache items in each bucket in ascending order of the version number until the next bucket is completely eliminated, which will make the video memory usage lower than another preset threshold t2. The system then establishes a minimum heap for the entries in the bucket and continuously releases the cache corresponding to the top of the heap until the video memory usage reaches t2. In addition, when evicting and releasing the cache, the system uses the Epoch mechanism (Epoch based reclaimation, round-based garbage collection) to avoid errors caused by the eviction and recycling of embedded parameters being read by the reading thread.

[0075] In one example, the version number is also used to implement an optimistic concurrency control protocol: before accessing the cached embedded parameters, the version number value is first read as ver1, and after reading the embedded parameters, the version number value is read again as ver2. If ver2=ver1, there is no concurrent writer; otherwise, a retry is required.

[0076] Specifically, when querying and updating embedded parameters, the system first deduplicates the ID list. This ensures that there are no multiple writers or readers of the same global key, only concurrent readers and writers. Therefore, a change in the version number when reading embedded parameters necessarily indicates the presence of concurrent writers.

[0077] S304: Unload a preset number of index queries in the CPU memory to the GPU memory using a unified indexing technology.

[0078] In this embodiment, a unified indexing technology is used to offload index queries in a preset number of CPU memories to the GPU video memory, including: first querying the global index to obtain the addresses of the embedded parameters that hit the cache; secondly, issuing a copy command, and the GPU copies the embedded parameters from these addresses. At the same time, in parallel, the CPU queries the parameters that missed in the CPU memory layer.

[0079] Specifically, inherited from the traditional key-value system accelerated by GPU, a simple coupling approach is proposed in related technologies (see Figure 7 a) It uses a kernel to copy the target embedding parameter matrix while querying the global index.

[0080] In contrast, this system decouples the copying of embedded parameters from the indexing of the cache for query cache operations (see Figure 7 b) First, an index kernel queries the global index to obtain information about whether each global key is hit or not, and the storage address if a hit occurs. Then, a copy kernel is issued to copy the storage address information returned by the index kernel. This approach preemptively queries the CPU memory layer, eliminating the need to wait for the copy kernel to complete its copy. This is because once the index kernel completes its execution, the system already knows the hit status of each global key.

[0081] In this embodiment, the unified indexing technology offloads index queries of a certain proportion of the CPU memory layer to the global index on the GPU, and uses a preset method to identify the storage address of the embedded parameters as the CPU memory layer; when the system queries the global index, it can directly obtain the address of the hit part of the embedded parameters in the CPU memory without querying the index of the slower CPU memory layer.

[0082] Specifically, the unified indexing technology uses the following identification method: for the storage address of the offload part (CPU memory layer), the lowest bit is first set to 1 and then stored in the global index.

[0083] The process of the system executing the query is as follows Figure 8 As shown, the global index is first queried. For addresses that match and are in GPU memory, the GPU copies them to the output embedding parameter matrix. For addresses that match but are in CPU memory, the CPU directly copies the obtained addresses to the output embedding parameter matrix cache in CPU memory. For global keys that do not match the global index, the CPU queries the CPU memory layer index to obtain the corresponding memory address and then copies the address. Finally, the system merges the output embedding parameter matrix cache in CPU memory with the output embedding parameter matrix on the GPU.

[0084] During cache elimination, if the entry to be replaced is found to be the CPU memory layer address unloaded by the unified index, the entry can be directly replaced without any memory recovery.

[0085] In one example, the offload ratio of the unified indexing technology is adaptively adjusted using the following algorithm: the offload ratio gradually increases from 0%, continuously reclaiming video memory originally used for the relatively cool embedding parameter cache for unified indexing; when the system reaches peak performance, the increase in the offload ratio is stopped;

[0086] When the system load hotspot situation changes, the unified index is cleared, the unloading ratio is increased again from 0%, and the above steps are repeated.

[0087] Specifically, the system continuously selects the hottest part of the uncached (CPU memory layer) embedding parameters and inserts them into the unified index, replacing the coldest embedding parameters stored in the GPU memory in the same Slab Set mapped to the global index, and returns the corresponding memory for global index expansion.

[0088] The system identifies a drop in the cache hit rate to a preset threshold as a change in the load hotspot. In this case, the unified index is cleared, the global index is scaled down, and the occupied space is returned to the video memory pool. The offload ratio is increased again from 0%.

[0089] S305 : When reading and writing the cache, decouple the copy operation of the embedded parameters from the index operation of the cache.

[0090] In this embodiment, when reading and writing the cache, the copy operation of the embedded parameters is decoupled from the cache indexing operation, including:

[0091] Each feature to be written is only allowed to enter the cache with a preset probability.

[0092] Specifically, when updating the cache, the system generates a random number between 0 and 1 for each global key to be inserted (using the curand_uniform function in the CUDA framework). If the random number is greater than the preset probability, the warp performs the insertion normally. Otherwise, the warp marks the feature update as failed in the returned result and exits directly. Based on mathematical expectation, features that appear fewer than (1 / preset probability) can bypass the cache.

[0093] Applying for space for the embedded parameters to be written into the cache through the video memory pool, and copying the parameters into the newly applied space; after the copying is completed, modifying the address mapping in the global index.

[0094] This system decouples the copying of embedded parameters from the indexing of caches for updating caches (see Figure 9b) First, sufficient embedded parameter space is allocated from the video memory pool. A separate copy kernel then pre-computes all copy operations (note that due to the Epoch mechanism, the copy kernel does not need to consider thread safety). Finally, the kernel is dispatched within the critical section to modify the address mapping information in the global index. This design shortens the critical path for modifying the global index.

[0095] In addition, the decoupled design also provides two additional opportunities to further optimize the cache system: on the one hand, the copy kernel can issue more threads for copying based on the embedded dimension information, which can make better use of the streaming multiprocessor (SM) resources on the GPU and reduce the copy latency; on the other hand, the copy kernel can be run in parallel with the kernel that modifies the global index, further reducing the time of cache update operations.

[0096] The present application provides a method, apparatus, device, and storage medium for embedding parameter caching, which obtains embedding parameters of at least one embedding table into CPU memory and stores some hot spots of the embedding parameters into GPU memory;

[0097] Based on a Huffman coding mechanism, the feature ID encoding of at least one embedding table is generated into a globally unique global key with a unified format. A global embedding cache table is used as the GPU cache storage format, and the global embedding cache table uses the global key as the key to index the embedding table. A unified indexing technique is used to offload index queries from a preset number of CPU memories to GPU video memory. When reading and writing the cache, the copy operation of the embedding parameters is decoupled from the cache indexing operation. This technical solution can solve the embedded access requirements and alleviate performance issues during model estimation.

[0098] Figure 10 1 is a schematic diagram of an embedded parameter cache device provided according to Embodiment 3 of the present application, including the device 100 as follows:

[0099] An acquiring unit 1001 is configured to acquire embedding parameters of at least one embedding table into a CPU memory, and store some hot spots of the embedding parameters into a GPU memory;

[0100] A generating unit 1002 is configured to generate a globally unique global key having a unified format by encoding the feature ID of the at least one embedded table based on a Huffman coding mechanism;

[0101] An indexing unit 1003 is configured to use a global embedded cache table as a storage form of GPU cache, wherein the global embedded cache table uses the global key as a key to index the embedded table;

[0102] An offloading unit 1004 is configured to offload a preset number of index queries in the CPU memory to the GPU memory using a unified indexing technology;

[0103] The decoupling unit 1005 is used to decouple the copy operation of the embedded parameters from the index operation of the cache when reading and writing the cache.

[0104] The generating unit 1002 is configured to obtain the corresponding global key for the FeatureID-th feature of the TableID-th embedded table according to the following formula:

[0105]

[0106]

[0107] Among them, m represents the bit length of TableID after Huffman encoding, n represents the bit length of the global key, huffman represents Huffman coding mapping, global key (TableID, FeatureID) represents the global key encoded by the encoding mechanism, % represents the modulo operation, and hash represents the hash operation.

[0108] In one example, the global embedding cache table consists of two parts: a global index and a memory pool. The global index uses a dictionary data structure to maintain a mapping relationship between the global key and the corresponding embedding parameter storage address, where the embedding parameter storage address is a GPU memory address or a CPU memory address. The memory pool manages the allocation and release of memory space, and includes the same number of sub-pools as the embedding table, and each sub-pool contains several parameters corresponding to the dimensions of the embedding table.

[0109] In one example, the global embedding cache table maintains a version number for each global key in the table. When a new embedding parameter is added to the cache or a cached embedding parameter is accessed by a user, the version number of the corresponding global key is updated.

[0110] In one example, the version number is used to control cache eviction: the system uses the version number to implement an approximate least recently used cache eviction algorithm; when the video memory usage exceeds a preset threshold t1, the system scans the global index and continuously releases the least recently used embedded parameter cache according to the version number and returns it to the video memory pool until the video memory usage falls below another preset threshold t2, where 0 <t2<t1<1。

[0111] In one example, the version number is also used to implement an optimistic concurrency control protocol: before accessing the cached embedded parameters, the version number value is first read as ver1, and after reading the embedded parameters, the version number value is read again as ver2. If ver2=ver1, there is no concurrent writer; otherwise, a retry is required.

[0112] The decoupling unit 1005 is specifically configured to allow each feature to be written into the cache only if the feature has a preset probability.

[0113] Index unit 1003 is used for the unified indexing technology to offload the index query of a certain proportion of the CPU memory layer to the global index on the GPU, and use a preset method to identify the storage address of the embedded parameters as the CPU memory layer; when the system queries the global index, it can directly obtain the address of the hit part of the embedded parameters in the CPU memory without querying the index of the slower CPU memory layer.

[0114] In one example, the unloading ratio of the unified indexing technology is adaptively adjusted using the following algorithm:

[0115] The offloading ratio gradually increases from 0%, and the video memory originally used for the relatively cool embedding parameter cache is continuously reclaimed for unified indexing;

[0116] When the system reaches a performance peak, the increase in the offload ratio is stopped;

[0117] When the system load hotspot situation changes, the unified index is cleared, the unloading ratio is increased again from 0%, and the above steps are repeated.

[0118] The unloading unit 1004 is configured to first query the global index to obtain the address of the embedded parameter hit in the cache;

[0119] Secondly, a copy command is issued, and the GPU copies the embedded parameters from these addresses. At the same time, in parallel, the CPU queries the missed parameters in the CPU memory layer.

[0120] The decoupling unit 1005 is specifically configured to apply for space for the embedded parameters to be written into the cache through the video memory pool, and copy the parameters into the newly applied space;

[0121] After the copying is completed, the address mapping in the global index is modified.

[0122] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0123] Figure 11This is a block diagram of a terminal device according to an exemplary embodiment. The device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0124] Device 1100 may include one or more of the following components: a processing component 1102 , a memory 1104 , a power component 1106 , a multimedia component 1108 , an audio component 1110 , an input / output (I / O) interface 1112 , a sensor component 1114 , and a communication component 1116 .

[0125] The processing component 1102 generally controls the overall operation of the device 1100, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1102 may include one or more modules to facilitate interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate interaction between the multimedia component 1108 and the processing component 1102.

[0126] The memory 1104 is configured to store various types of data to support the operation of the device 1100. Examples of such data include instructions for any application or method operating on the device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0127] The power supply component 1106 provides power to the various components of the device 1100. The power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 1100.

[0128] The multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 1108 includes a front-facing camera and / or a rear-facing camera. When the device 1100 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0129] The audio component 1110 is configured to output and / or input audio signals. For example, the audio component 1110 includes a microphone (MIC) that is configured to receive external audio signals when the device 1100 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the memory 1104 or transmitted via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting audio signals.

[0130] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0131] Sensor assembly 1114 includes one or more sensors for providing various aspects of the status assessment of device 1100. For example, sensor assembly 1114 can detect the open / closed state of device 1100, the relative positioning of components, such as the display and keypad of device 1100. Sensor assembly 1114 can also detect changes in the position of device 1100 or a component of device 1100, the presence or absence of user contact with device 1100, the orientation or acceleration / deceleration of device 1100, and changes in the temperature of device 1100. Sensor assembly 1114 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0132] The communication component 1116 is configured to facilitate wired or wireless communication between the apparatus 1100 and other devices. The apparatus 1100 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0133] In an exemplary embodiment, the apparatus 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0134] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions. The instructions can be executed by the processor 1120 of the apparatus 1100 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0135] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a terminal device, enables the terminal device to execute the above-mentioned embedded parameter caching method of the terminal device.

[0136] The present application also discloses a computer program product, including a computer program, which implements the method described in this embodiment when executed by a processor.

[0137] Various embodiments of the systems and techniques described above in this application can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0139] In the context of this application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as data electronics), or a computing system that includes middleware components (e.g., application electronics), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0142] A computer system may include a client and an electronic device. The client and electronic device are generally remote from each other and typically interact via a communication network. The client-electronic device relationship is established by computer programs running on the respective computers and establishing a client-electronic device relationship. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or cloud host, a host product within a cloud computing service system that addresses the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or "VPS"). The electronic device may also be an electronic device in a distributed system or an electronic device integrated with a blockchain. It should be understood that the various forms of the process shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this application may be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application are achieved. This is not intended to limit the present disclosure.

[0143] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0144] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for embedding parameter cache, characterized in that: The method comprises: Obtaining embedding parameters of at least one embedding table into CPU memory, and storing some hot spots of the embedding parameters into GPU memory; Based on a Huffman coding mechanism, encoding the feature ID of the at least one embedded table to generate a globally unique global key with a unified format; A global embedded cache table is used as a storage form of GPU cache, wherein the global embedded cache table uses the global key as a key to index the embedded table; Using unified indexing technology, a preset number of index queries in CPU memory are offloaded to GPU memory; When reading and writing cache, decouple the copy operation of embedded parameters from the cache index operation; The global embedding cache table consists of two parts: a global index and a memory pool. The global index uses a dictionary data structure to maintain the mapping relationship between the global key and the corresponding embedding parameter storage address, where the embedding parameter storage address is the GPU memory address or the CPU memory address. The memory pool manages the allocation and release of memory space and includes the same number of sub-pools as the embedding table. Each sub-pool contains several parameters corresponding to the embedding table dimensions. The unified indexing technology offloads index queries of a certain proportion of the CPU memory layer to the global index on the GPU, and uses a preset method to identify the storage address of the embedded parameters as the CPU memory layer; when the system queries the global index, it can directly obtain the address of the hit part of the embedded parameters in the CPU memory without querying the index of the slower CPU memory layer.

2. The method according to claim 1, characterized in that Based on a Huffman coding mechanism, encoding the feature ID of the at least one embedded table to generate a globally unique global key with a unified format includes: For the FeatureID-th feature of the TableID-th embedded table, the corresponding global key is obtained according to the following formula: Among them, m represents the bit length of TableID after Huffman encoding, n represents the bit length of the global key, huffman represents Huffman coding mapping, global key (TableID, FeatureID) represents the global key encoded by the encoding mechanism, % represents the modulo operation, and hash represents the hash operation.

3. The method according to claim 1, characterized in that The global embedding cache table maintains a version number for each global key in the table. When a new embedding parameter is added to the cache or a cached embedding parameter is accessed by a user, the version number of the corresponding global key is updated.

4. The method according to claim 3, characterized in that The version number is used to control cache eviction: The system uses the version number to implement an approximate least recently used cache eviction algorithm; When the video memory usage exceeds a preset threshold value t1, the system scans the global index and continuously releases the longest unused embedded parameter cache according to the version number and returns it to the video memory pool until the video memory usage falls below another preset threshold value t2, where 0 <t2<t1<1。 5. The method according to claim 3, characterized in that The version number is also used to implement an optimistic concurrency control protocol: before accessing the cached embedded parameters, the version number value is first read as ver1, and after reading the embedded parameters, the version number value is read again as ver2. If ver2=ver1, there is no concurrent writer; otherwise, a retry is required.

6. The method according to claim 1, characterized in that When reading and writing caches, decouple the copying of embedded parameters from the indexing of the cache, including: Each feature to be written is only allowed to enter the cache with a preset probability.

7. The method according to claim 1, characterized in that The unloading ratio of the unified indexing technology is adaptively adjusted using the following algorithm: The offloading ratio gradually increases from 0%, and the video memory originally used for the relatively cool embedding parameter cache is continuously reclaimed for unified indexing; When the system reaches a performance peak, the increase in the offload ratio is stopped; When the system load hotspot situation changes, the unified index is cleared, the unloading ratio is increased again from 0%, and the above steps are repeated.

8. The method according to claim 1, characterized in that Unified indexing technology is used to offload a preset amount of index queries from CPU memory to GPU memory, including: First, query the global index to obtain the address of the embedded parameter hit in the cache; Secondly, a copy command is issued, and the GPU copies the embedded parameters from these addresses. At the same time, in parallel, the CPU queries the missed parameters in the CPU memory layer.

9. The method according to claim 1, characterized in that When reading and writing caches, decouple the copying of embedded parameters from the indexing of the cache, including: Applying space for the embedded parameters to be written into the cache through the video memory pool, and copying the parameters into the newly applied space; After the copying is completed, the address mapping in the global index is modified.

10. A device for embedding parameter cache, characterized in that: The device comprises: an acquiring unit, configured to acquire embedding parameters of at least one embedding table into a CPU memory, and store some hot spots of the embedding parameters into a GPU memory; a generating unit, configured to generate a globally unique global key having a unified format by encoding the feature ID of the at least one embedded table based on a Huffman coding mechanism; An indexing unit is configured to use a global embedded cache table as a storage form for GPU cache, wherein the global embedded cache table uses the global key as a key to index the embedded table; the global embedded cache table is composed of a global index and a memory pool. The global index uses a dictionary data structure to maintain a mapping relationship between the global key and the corresponding embedded parameter storage address, wherein the embedded parameter storage address is a GPU memory address or a CPU memory address; the memory pool manages the allocation and release of memory space and includes a number of sub-pools equal to the number of embedded tables, each sub-pool containing a number of parameters corresponding to the dimensions of the embedded table. An offloading unit is configured to offload index queries from a preset number of CPU memories to GPU memory using a unified indexing technique; the unified indexing technique offloads index queries from a certain percentage of the CPU memory layer to the global index on the GPU, and uses a preset method to identify the storage address of the embedded parameters as the CPU memory layer; when querying the global index, the system can directly obtain the addresses of the hit embedded parameters in the CPU memory without querying the index of the slower CPU memory layer; The decoupling unit is used to decouple the copy operation of the embedded parameters from the cache index operation when reading and writing the cache.

11. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when the computer program is executed by a processor.

Images