Distributed system for deep learning recommendation model and data caching method
By deploying measurement and caching modules in a distributed system of a deep learning recommendation model, hot data can be analyzed and cached in real time, solving the communication and memory bottlenecks in distributed training and inference, and improving system performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU UNIV
- Filing Date
- 2026-01-20
- Publication Date
- 2026-05-08
AI Technical Summary
Distributed training and inference of deep learning recommendation models are limited by communication and memory bottlenecks on single-node devices, resulting in high communication volume and high access latency.
By employing a distributed system and data caching method, a measurement module is deployed through a central switch and a caching module is deployed on distributed nodes. This allows for real-time measurement and analysis of communication traffic, identification of hot data, and caching of it across nodes, thereby reducing all-to-all communication volume and minimizing GPU memory access requirements.
It effectively improved the cache hit rate, reduced communication volume and access latency, and solved the problems of dense data throughput and high access latency caused by excessively large model size.
Smart Images
Figure CN121996699A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a distributed system and data caching method for deep learning recommendation models, belonging to the field of communication technology for distributed deep learning systems. Background Technology
[0002] Personalized recommendation systems have become a key supporting technology in internet services, widely used in online businesses such as product recommendations, video and music recommendations, and search services. With the increasing sophistication of recommendation tasks and the continuous expansion of data scale, deep learning-based recommendation models (DLRM) have shown significant advantages in click-through rate prediction and ranking tasks, becoming the current mainstream recommendation modeling framework.
[0003] DLRM models typically consist of multilayer perceptrons and large-scale embedding operators, where the embedding layers are used to perform high-dimensional vectorization of discrete features. Compared to traditional neural networks, DLRMs have an extremely large parameter scale, and their embedding tables (EMTs) often require hundreds of gigabytes or even terabytes of storage space, far exceeding the high-bandwidth memory capacity of a single accelerator (such as a GPU). Meanwhile, the growth rate of hardware memory capacity is far slower than the expansion rate of model size, making high-performance DLRM training and inference on single-node devices infeasible.
[0004] Therefore, high-performance training and inference of DLRM typically rely on multi-node distributed systems. However, distributed training of DLRM introduces new system bottlenecks and scalability challenges, mainly in the following aspects: Communication bottleneck: Due to the large model size, traditional data parallelism cannot fully replicate model parameters on each accelerator. The industry generally adopts a hybrid strategy of "model parallelism + data parallelism", which partitions the EMT and distributes it to different nodes. Each node needs to exchange embedding results through all-to-all communication. This communication mode generates extremely high data exchange volume, becoming a key bottleneck for the overall system performance. As the model size grows exponentially, the communication volume also increases exponentially, and the communication path becomes a fundamental obstacle restricting the system's scalability. Memory bottleneck: DLRM contains trillions of parameters, placing extremely high demands on memory access bandwidth. Embedding operations involve frequent and random table accesses, and memory bandwidth struggles to support such dense data throughput, leading to increased access latency and further reducing overall training efficiency. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a distributed system and data caching method for deep learning recommendation models. This system can measure and analyze cluster communication traffic in real time, count global embedding table access hotspots, and push hotspot data to the cache of each node. This solves the problems of high data throughput and high access latency caused by the large scale of recommendation models and the frequent and random table accesses involved in embedding operations.
[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution: The present invention provides a distributed system for deep learning recommendation models, including multiple distributed nodes and a central switch for communication between the distributed nodes. The central switch is equipped with a measurement module, and the distributed nodes are equipped with distributed embedding tables and caching modules. The measurement module is used to measure the size of each query request data stream passing through the central switch in each measurement cycle, determine the popular query requests of each distributed node based on the size of the data stream, and push the cached entries corresponding to the popular query requests to the distributed nodes. The distributed nodes are used to execute query requests for each measurement period and merge the cache entries corresponding to popular query requests in the caching module with the local cache entries in the distributed embedded table after each measurement period ends.
[0007] Furthermore, the measurement module includes multiple storage buckets arranged in a matrix, with each row mapped to a hash function. The storage buckets include: The tag field is used to record the tags corresponding to the data streams in the storage bucket; The value range is used to calculate the sum of the data stream values of all query requests in the bucket. The counting field is used to indicate the count; The timestamp field is used to record the arrival time of the current query request data stream; The instantaneous frequency estimation domain is used to estimate the instantaneous access rate change of the current query request data stream; The trend priority field is used to record the priority of the current query request data stream; The path information field is used to record the forwarding path of the current query request data stream.
[0008] Furthermore, the measurement module is also used to update the storage bucket based on the matching between the tags of the current query request data stream and the tags of the recorded data stream in the tag field.
[0009] Furthermore, if the label of the current query request data stream matches the label of the record data stream in the label field, the bucket is updated using the following formula: ; ; ; in, Indicates the first The label corresponding to each data stream The hash value; Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket count field. This represents the time-weighted function. Represents a timestamp; Indicates the first The label corresponding to each data stream The measurement module in question The value of the instantaneous frequency estimation field record in the row bucket. Represents the frequency weighting parameter. Indicates the latest increment of the value range. Indicates and The corresponding timestamp variable, This indicates assignment; Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the timestamp field of the row bucket. Indicates the arrival time of the current query request data stream; ; in, This represents an exponential function with base e. This represents the time weighting parameter.
[0010] Furthermore, if the label of the current query request data stream does not match the label of the record data stream in the label field, the bucket is updated using the following formula: ; in, Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket count field. This represents the time-weighted function. Represents a timestamp. This indicates assignment; ; in, This represents an exponential function with base e. Indicates the time weighting parameter. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the timestamp field of the row bucket. Indicates the arrival time of the current query request data stream; like ,but , ; initialization and for Assign new priority, record ; in, Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket label field. Indicates the first The label corresponding to each data stream The measurement module in question The value of the instantaneous frequency estimation field record in the row bucket. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the trend priority field of the row bucket. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket path information field. Indicates the current query request data stream request node number. This indicates the target node number of the current query request data stream.
[0011] Furthermore, the measurement module is used to measure the size of each query request data stream passing through the central switch in each measurement cycle, including: The size of the data stream for each query request is calculated using the following formula: ; in Indicates the measurement module OK The column bucket records the values of the query request data stream corresponding to the data stream labels. Indicates the measurement module OK The value of the value field record in the column bucket. Indicates the measurement module OK The value recorded in the count field of the column bucket. and This indicates an adjustable weighting coefficient. This represents the activation function. Indicates the measurement module OK The values of the frequency estimation field records in the column bucket. This indicates the normalization priority value.
[0012] Furthermore, the distributed node, used to execute query requests for each measurement period, includes: The cache module includes k Cache entries and related k The counter corresponding to each cache entry; If the query request hits a cached entry in the cache module, the central switch directly obtains the feature vector corresponding to the cached entry and increments the counter value corresponding to the cached entry by 1; otherwise, the central switch continues to make query requests to the remaining nodes.
[0013] Furthermore, the distributed node is used to merge the cache entries corresponding to popular query requests in the caching module with the local cache entries in the distributed embedded table after each measurement period, including: After the measurement period ends, the central switch will receive the corresponding popular query requests pushed by the central switch. k Each cache entry and its corresponding count; Push popular query requests corresponding to the central switch k After merging cached entries with local cached entries, sort them in descending order of their count values and take the first few entries. k After each cached entry is cached, all counter values are reset to zero.
[0014] In another aspect, the present invention provides a data caching method, which is applied to a distributed system for deep learning recommendation models as described above, comprising: Obtain the tags of each query request data stream passing through the central switch in each measurement cycle; Calculate the size of the corresponding data stream based on the tags of each query request data stream; Based on the size of each data stream, determine the most popular query requests for each distributed node; The cached entries corresponding to popular query requests are pushed to distributed nodes for caching.
[0015] Furthermore, determining the most popular query requests for each distributed node based on the size of each data stream includes: Based on the size of each data stream, parse the request node number and query feature number in the corresponding tag; Construct a cross-node co-occurrence matrix based on the request node number and the query feature number; Based on the cross-node co-occurrence matrix, identify shared hotspots and node-specific hotspots; Determine the composite priority based on shared hotspots and node-specific hotspots; Generate a sorted list based on the composite priority, and take the top priority. k The query feature number of each request is used to obtain the most popular query requests for each distributed node.
[0016] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This invention acquires the tags of each query request data stream passing through the central switch in each measurement cycle, calculates the size of the corresponding data stream based on the tags of each query request data stream, and determines the popular query requests of each distributed node based on the size of each data stream. It can perform real-time measurement and analysis of cluster communication traffic, count global embedded table access hotspots, and push hotspot data to the cache of each node, effectively improving cache hit rate, reducing all-to-all communication volume, reducing the demand for GPU memory access, and solving the current problems of dense data throughput and high access latency caused by the large scale of recommendation models and frequent and random table entry accesses involved in embedding operations. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of the structure of a distributed system for deep learning recommendation models provided in an embodiment of the present invention; Figure 2 This is a flowchart of a data caching method provided in an embodiment of the present invention. Detailed Implementation
[0018] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention. Example 1
[0019] like Figure 1 As shown, a distributed system for deep learning recommendation models includes N distributed nodes and a central switch for communication between the distributed nodes. The central switch is equipped with a measurement module, and the distributed nodes are equipped with distributed embedding tables. and cache module, in which .
[0020] The measurement modules include those arranged in a matrix. d OK w The column's storage bucket, in each measurement module, the first i OK j The bucket of a column is represented as And the measurement module corresponds d Row storage bucket, mapped d A hash function, denoted as The storage bucket includes: The tag field is used to record the tags corresponding to the data streams in the bucket, represented as... ; The range is used to calculate the sum of the data stream values of all query requests in the bucket, and is represented as... ; The counting field, used to indicate the count, is represented as ; The timestamp field is used to record the arrival time of the current query request data stream, represented as... ; The instantaneous frequency estimation domain is used to estimate the instantaneous access rate change of the current query request data stream, and is represented as... ; The trend priority field records the priority of the current query request data stream, represented as... ; The path information field records the forwarding path of the current query request data stream, represented as... .
[0021] For each incoming query request data stream, extract the node number of the request. Target node number With query feature number As a label, that is Map it to d In the row storage bucket; for In the measurement module i Row bucket mapping First, update the value range of the storage bucket, as follows: , Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket value field; Based on the matching between the labels of the current query request data stream and the labels of the record data streams in the label field, update the storage bucket, specifically: If the label of the current query request data stream matches the label of the record data stream in the label field, then the bucket is updated using the following formula: ; ; ; in, Indicates the first The label corresponding to each data stream The hash value; Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket count field. This represents the time-weighted function. Represents a timestamp; Indicates the first The label corresponding to each data stream The measurement module in question The value of the instantaneous frequency estimation field record in the row bucket. Represents the frequency weighting parameter. Indicates the latest increment of the value range. Indicates and The corresponding timestamp variable, This indicates assignment; Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the timestamp field of the row bucket. Indicates the arrival time of the current query request data stream; ; in, This represents an exponential function with base e. Indicates the time weighting parameter; If the label of the current query request data stream does not match the label of the record data stream in the label field, the bucket is updated using the following formula: ; like ,but , ; initialization and for Assign new priority, record ; in, Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket label field. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the trend priority field of the row bucket. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket path information field.
[0022] The measurement module measures the size of each query request data stream passing through the central switch in each measurement cycle. Based on the data stream size, it determines the most popular query requests for each distributed node and pushes the cached entries corresponding to these popular query requests to the distributed nodes. Specifically: The size of the data stream for each query request is calculated using the following formula: ; in Indicates the measurement module OK The column bucket records the values of the query request data stream corresponding to the data stream labels. and This indicates an adjustable weighting coefficient. This represents the activation function. This indicates the normalization priority value.
[0023] Distributed nodes are used to execute query requests for each measurement period and, at the end of each measurement period, merge the cache entries corresponding to popular query requests in the caching module with the local cache entries in the distributed embedded table. Specifically: The caching module includes k 1 cache entry and with k Counters corresponding to each cache entry ; If the query request hits a cached entry in the cache module, the central switch directly obtains the feature vector corresponding to the cached entry and increments the counter value corresponding to the cached entry by 1; otherwise, the central switch continues to make query requests to the remaining nodes. After the measurement period ends, the receiving center switch pushes the corresponding popular query requests. k Each cache entry and its corresponding count; Push popular query requests corresponding to the central switch k After merging cached entries with local cached entries, sort them in descending order of their count values and take the first few entries. k After each cached entry is cached, all counter values are reset to zero.
[0024] Example 2 like Figure 2 As shown, a data caching method is applied to the distributed system in Example 1, comprising: Obtain the tags of each query request data stream passing through the central switch in each measurement cycle; Calculate the size of the corresponding data stream based on the tags of each query request data stream; Based on the size of each data stream, determine the most popular query requests for each distributed node. Specifically: Based on the size of each data stream, parse the request node number and query feature number in the corresponding tag; Construct a cross-node co-occurrence matrix M using the request node number and the query feature number, where the request node number... For matrix rows, query feature numbers For matrix columns, , , Indicates the number of nodes requested. Indicates the number of query features; Based on the cross-node co-occurrence matrix, shared hotspots and node-specific hotspots are identified, including: If M[ ][ ]=0, indicating the first The feature for the first Each node belongs to a non-hotspot; If M[ ][ ]>0 indicates the first The feature for the first Each node belongs to a hotspot, and M[ ][ The value of ] equals the number of query requests; For the cross-node co-occurrence matrix M, the first... Column, if M[ ][ ], M[ ][ ]…M[ ][ If none of them are equal to 0, then the first... The first feature is a shared hotspot; if the first feature is a shared hotspot; A column is not equal to 0 only in a certain row, for example, only M[ ][ If ] ≠ 0, then the first The first feature is the Specific hotspots of each node; Based on shared hotspots and node-specific hotspots, a composite priority is determined, including: No. The composite priority of the features can be determined by the matrix M. The result is obtained by summing the columns; Generate a sorted list based on the composite priority, and take the top priority.k The query feature number of each request is used to obtain the most popular query requests for each distributed node; The cached entries corresponding to popular query requests are pushed to distributed nodes for caching.
[0025] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage CD-ROMs, optical storage, etc.) containing computer-usable program code.
[0026] This application is described with reference to flowchart illustrations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 One or more processes or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0027] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.
[0028] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.
[0029] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.
Claims
1. A distributed system for deep learning recommendation models, characterized in that, It includes multiple distributed nodes and a central switch for communication between the distributed nodes. The central switch is equipped with a measurement module, and the distributed nodes are equipped with distributed embedded tables and cache modules. The measurement module is used to measure the size of each query request data stream passing through the central switch in each measurement cycle, determine the popular query requests of each distributed node based on the size of the data stream, and push the cached entries corresponding to the popular query requests to the distributed nodes. The distributed nodes are used to execute query requests for each measurement period and merge the cache entries corresponding to popular query requests in the caching module with the local cache entries in the distributed embedded table after each measurement period ends.
2. The distributed system for deep learning recommendation models according to claim 1, characterized in that, The measurement module includes multiple storage buckets arranged in a matrix, with each row mapped to a hash function. The storage buckets include: The tag field is used to record the tags corresponding to the data streams in the storage bucket; The value range is used to calculate the sum of the data stream values of all query requests in the bucket. The counting field is used to indicate the count; The timestamp field is used to record the arrival time of the current query request data stream; The instantaneous frequency estimation domain is used to estimate the instantaneous access rate change of the current query request data stream; The trend priority field is used to record the priority of the current query request data stream; The path information field is used to record the forwarding path of the current query request data stream.
3. The distributed system for deep learning recommendation models according to claim 2, characterized in that, The measurement module is also used to update the storage bucket based on the matching between the label of the current query request data stream and the label of the record data stream in the label field.
4. The distributed system for deep learning recommendation models according to claim 3, characterized in that, If the label of the current query request data stream matches the label of the record data stream in the label field, then the bucket is updated using the following formula: ； ； ； in, Indicates the first The label corresponding to each data stream The hash value; Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket count field. This represents the time-weighted function. Represents a timestamp; Indicates the first The label corresponding to each data stream The measurement module in question The value of the instantaneous frequency estimation field record in the row bucket. Represents the frequency weighting parameter. Indicates the latest increment of the value range. Indicates and The corresponding timestamp variable, This indicates assignment; Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the timestamp field of the row bucket. Indicates the arrival time of the current query request data stream; ； in, This represents an exponential function with base e. This represents the time weighting parameter.
5. The distributed system for deep learning recommendation models according to claim 3, characterized in that, If the label of the current query request data stream does not match the label of the record data stream in the label field, the bucket is updated using the following formula: ； in, Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket count field. This represents the time-weighted function. Represents a timestamp. This indicates assignment; ； in, This represents an exponential function with base e. Indicates the time weighting parameter. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the timestamp field of the row bucket. Indicates the arrival time of the current query request data stream; like ,but , ; initialization and for Assign new priority, record ; in, Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket label field. Indicates the first The label corresponding to each data stream The measurement module in question The value of the instantaneous frequency estimation field record in the row bucket. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the trend priority field of the row bucket. Indicates the first The label corresponding to each data stream The measurement module in question The value recorded in the row bucket path information field. Indicates the current query request data stream request node number. This indicates the target node number of the current query request data stream.
6. The distributed system for deep learning recommendation models according to claim 2, characterized in that, The measurement module is used to measure the size of each query request data stream passing through the central switch in each measurement cycle, including: The size of the data stream for each query request is calculated using the following formula: ； in Indicates the measurement module OK The column bucket records the values of the query request data stream corresponding to the data stream labels. Indicates the measurement module OK The value of the value field record in the column bucket. Indicates the measurement module OK The value recorded in the count field of the column bucket. and This indicates an adjustable weighting coefficient. This represents the activation function. Indicates the measurement module OK The values of the frequency estimation field records in the column bucket. This indicates the normalization priority value.
7. The distributed system for deep learning recommendation models according to claim 1, characterized in that, The distributed nodes are used to execute query requests for each measurement period, including: The cache module includes k Cache entries and related k The counter corresponding to each cache entry; If the query request hits a cached entry in the cache module, the central switch directly obtains the feature vector corresponding to the cached entry and increments the counter value corresponding to the cached entry by 1; otherwise, the central switch continues to make query requests to the remaining nodes.
8. The distributed system for deep learning recommendation models according to claim 7, characterized in that, The distributed node is used to merge the cache entries corresponding to popular query requests in the caching module with the local cache entries in the distributed embedded table after each measurement period, including: After the measurement period ends, the central switch will receive the corresponding popular query requests pushed by the central switch. k Each cache entry and its corresponding count; Push popular query requests corresponding to the central switch k After merging cached entries with local cached entries, sort them in descending order of their count values and take the first few entries. k After each cached entry is cached, all counter values are reset to zero.
9. A data caching method, characterized in that, The method is applied to a distributed system for deep learning recommendation models as described in any one of claims 1 to 8, comprising: Obtain the tags of each query request data stream passing through the central switch in each measurement cycle; Calculate the size of the corresponding data stream based on the tags of each query request data stream; Based on the size of each data stream, determine the most popular query requests for each distributed node; The cached entries corresponding to popular query requests are pushed to distributed nodes for caching.
10. The data caching method according to claim 9, characterized in that, The process of determining the most popular query requests for each distributed node based on the size of each data stream includes: Based on the size of each data stream, parse the request node number and query feature number in the corresponding tag; Construct a cross-node co-occurrence matrix based on the request node number and the query feature number; Based on the cross-node co-occurrence matrix, identify shared hotspots and node-specific hotspots; Determine the composite priority based on shared hotspots and node-specific hotspots; Generate a sorted list based on the composite priority, and take the top priority. k The query feature number of each request is used to obtain the most popular query requests for each distributed node.