A dual tensor parallel method for hybrid word frequency embedding
By employing a dual tensor parallel approach with hybrid word frequency embedding, the problems of uneven embedding access volume and high communication cost on the device are solved, thereby improving training efficiency and memory utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING LUCHEN TECH CO LTD
- Filing Date
- 2022-09-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies do not consider word frequency information distribution when training deep learning recommendation system models, resulting in uneven embedded access volume on devices, failing to fully utilize the advantages of tensor parallelism, and incurring high communication costs between devices.
A dual tensor parallel approach with hybrid word frequency embedding is adopted. The embedding table is uniformly cut by a greedy algorithm, the device is segmented by word frequency, and secondary cutting and algorithm compression are performed. Combined with the embedding bag operation, all-reduce and reduce operations are used to reduce communication requirements.
This achieves a uniform distribution of embedded access volume on the device, reduces communication overhead between devices, and improves training efficiency and memory utilization efficiency.
Smart Images
Figure CN115455150B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, specifically to a dual tensor parallel method with hybrid word frequency embedding. Background Technology
[0002] As the number of users and the variety of goods continue to grow, the amount of user and product information that deep learning recommendation system models need to embed is growing exponentially. In some scenarios, the model can have trillions of parameters. It is difficult to put all the embedding weights on a single card and update the model during training.
[0003] Current solutions include using tensor parallelism to distribute the embedding table across different devices by row or column for training. Specifically, the embedding table is first evenly distributed across different devices by column, then an embedding query is performed on each device, followed by an all-to-all operation to aggregate all query results on each device and then concatenate them along the initial split dimensions. Alternatively, algorithm compression can be achieved using word frequency statistics. For example, high-frequency words are assigned larger embedding dimensions, and low-frequency words are assigned smaller dimensions. Then, linear projection is used to unify the embedding dimensions and remove parameter redundancy. For instance, if the embedding table size is (|V|, D), and there are high-frequency, mid-frequency, and low-frequency words in a ratio of |V| / 3 (boundaries calculated using a specific method), then the compressed embedding dimensions could be D, D / 2, and D / 4, corresponding to linear projection sizes of 0 (no projection required), (D / 2, D), and (D / 4, D).
[0004] However, the above-mentioned tensor parallelism method does not take into account the word frequency information distribution during implementation. On some devices, there may be no embedding access, and the advantages of tensor parallelism are not utilized. At the same time, it only supports embedding instead of embedding bag operations, which will generate a large communication cost when communicating between devices. Summary of the Invention
[0005] (a) Technical problems to be solved
[0006] To address the shortcomings of existing technologies, this invention provides a dual tensor parallel method for hybrid word frequency embedding, which solves the problems of not considering word frequency information distribution, potentially having no embedding access on some devices, and failing to utilize the advantages of tensor parallelism; at the same time, it only supports embedding operations instead of embedding bags, resulting in significant communication costs when communicating between devices.
[0007] (II) Technical Solution
[0008] To achieve the above objectives, the present invention provides the following technical solution: a dual tensor parallel method for hybrid word frequency embedding, specifically comprising the following steps:
[0009] S1. First, the task allocator scans the training dataset once to count the number of times each query word ID appears. Then, using a greedy algorithm (minimax: to minimize the maximum difference in word frequency between the embedded tables after splitting), the rows of the embedded tables are evenly split into parallel devices according to the total number of word frequencies, so that the word frequency on each device is basically the same. At the same time, the input data is preprocessed, only the word IDs that can be queried on each device are kept, and the words that cannot be queried are replaced with special values.
[0010] S2. The word frequency distribution on each device follows the characteristics of a long tail. After sorting, the embedding bag is divided into embedding blocks by row according to several different quantiles (e.g., 25%, 50%, 75%). Based on the number of word frequencies in each segmented embedding block, the existing embedding dimension is compressed to a suitable value, and the corresponding linear projector is used to expand the final embedding dimension of each embedding block to a unified dimension. Since the increase in parameters required by the linear projector is generally much smaller than the reduction in parameters brought about by compressing the embedding dimension, the purpose of algorithm compression can be achieved.
[0011] S3. The parallelism of dual tensors is mainly reflected in the two partitions of the embedding table. Correspondingly, it also requires two set operations to finally collect the partitioned embedding bags. The set operation here uses (All-)reduce, whose main function is to combine the partitioned embedding table into a set according to a specific method (such as summation, taking the maximum value, or taking the average value). It takes advantage of the similarity between the set operation and the embedding bag operation (including summation, taking the maximum value, or taking the average value) to reduce the communication requirements between devices (reusing the initial representation: the original complexity of O(b*n*d) is reduced to O(b*d)), ensuring that the result obtained after the set is the same as that of single-device processing.
[0012] Preferably, in step S2, when compressing existing dimensions, the principle of higher compression intensity is followed as the word frequency is lower.
[0013] Preferably, in step S3, the embedded bag is a special embedded data structure. Compared with traditional embedding, it adds a post-processing function, supporting summation, mean and maximum value operations. For example, when the query input shape is (b, n), the result obtained by the ordinary embedded table is (b, n, d), while the embedded bag performs post-processing on the dimension dim=1, and the result obtained is (b, d). Subsequently, the embedded table and the embedded bag will be used alternately, and in this scenario, both refer to the embedded bag.
[0014] Preferably, the two segmentations of the embedded table in step S3 are as follows: the first segmentation occurs when the word frequency is evenly distributed to different devices, and the second segmentation occurs on each device according to the quantile of the word frequency.
[0015] Preferably, in step S3, during the set operation, in the first splitting scenario, since there are multiple devices, we need all-reduce to ensure that the embeddings of each device are the same after the final set; however, in the second splitting scenario, which only occurs on a single device, only a reduce operation is needed.
[0016] (III) Beneficial Effects
[0017] This invention provides a dual tensor parallel method for hybrid word frequency embedding. It has the following advantages:
[0018] This hybrid word frequency embedding dual tensor parallel method achieves uniform horizontal segmentation based on access volume during tensor parallelism by utilizing the word frequency distribution information of the embedding table, ensuring the amortization of training workload. It also supports the embedding bag operation, which effectively reduces the communication overhead between devices compared to simple embedding. On each parallel device, the embedding table is further segmented according to word frequency and compressed by the algorithm, effectively reducing the memory consumption on each device. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the system of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Please see Figure 1 This invention provides a technical solution: a dual tensor parallel method for hybrid word frequency embedding, specifically including the following steps:
[0022] S1. The task allocator first scans the training dataset to count the occurrences of each query word ID. Then, a greedy algorithm is used to evenly distribute the rows of the embedding table to parallel devices according to the total word frequency, so that the word frequency on each device is basically the same. For example, there are 5 discrete features, and the frequency information is [10, 2, 3, 4, 9]. Assuming that the number of devices implementing tensor parallelism is 2, the feature frequencies after splitting are [[10, 2, 3], [4, 9]], which minimizes the maximum difference between groups (minimax).
[0023] Simultaneously, the discrete input information is preprocessed, and the processed input is given to each device. Assuming that the discrete input of the current batch is [2,3,4,5,8] with a shape of (1,5), and there are two devices participating in tensor parallelism, and the other conditions follow the previous step, then the input given to device zero after cutting is [2,3] with a shape of (1,2), and the input given to device one is [4,5,8] with a shape of (1,3).
[0024] S2. Perform secondary segmentation on each device and compress the algorithm. Continuing the example above, take device number one as an example. The assigned feature frequencies are [3,4,9]. Then, continue to segment according to the quantiles into [[3], [4], [9]]. For the sake of the example, assume that the number of word IDs that may appear in the three features on device number one are 20, 40 and 60 respectively. The original embedding dimension is 4. The embedding dimensions after compression according to the word frequency are 1, 2 and 4 respectively. Then the linear projection dimensions required for each feature are (1,4), (2,4) and 0 respectively (the third feature is not compressed because of its relatively high access frequency). In summary, the number of parameters before compression (the size of the embedding table when the embedding dimension is 4) is (20+40+60)*4=480. The number of parameters after compression is (the sum of the parameters of the embedding table and the linear projection) (20*1+1*4+40*2+2*4+60*4)=252.
[0025] S3. Perform reduce operations within and between devices. Continuing the previous example, the embedding shape obtained from the input inside device 1 is (1, 3, 4) (corresponding to the number of input batches, the number of discrete features, and the embedding dimension, respectively). Assuming the reduce operation is a summation, the embedding shape on device 1 after reducing in the direction with dimension 1 is (1, 4). Similarly, device 0 also has an embedding shape of (1, 4). This reduce operation corresponds to the post-processing of the embedding bag. Without post-processing, the resulting embedding shape would be (1, 3, 4). When performing all-reduce between devices, the embedding table of each device is transmitted to other devices and connected together, with a shape of (1, 2, 4). Then, a summation operation is performed, and the final shape is (1, 4). The communication volume saved by the embedding bag operation is approximately (2*1*2*4-2*1*4) / 2*1*2*4*100%=50%.
[0026] S4. Each device has the same embedded query results, ready to be input into the dense layer operations behind the model.
[0027] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0028] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A dual tensor parallel method for hybrid word frequency embedding, characterized in that: Specifically, the following steps are included: S1. First, the task allocator scans the training dataset once to count the number of times each query word ID appears. Then, a greedy algorithm is used to evenly distribute the rows of the embedding table to parallel devices according to the total word frequency, so that the word frequency on each device is basically the same. At the same time, the input data is preprocessed, only the word IDs that can be queried on each device are kept, and the words that cannot be queried are replaced with special values. S2. The word frequency distribution on each device follows the characteristics of a long tail. After sorting, the embedding bag is divided into embedding blocks by row according to several different quantiles. Based on the number of word frequencies in each embedded block, the existing embedding dimension is compressed to a suitable value, and the corresponding linear projector is used to expand the final embedding dimension of each embedded block to a unified dimension. The quantiles can be 25%, 50%, or 75%. The embedding bag is an embedding data structure that supports summation, mean, and maximum value operations. S3. Use reduce and all-reduce operations to collect the cut embedding bags. For the segmentation case that occurs only on a single device, the reduce operation is performed inside the device, that is, the sum is directly taken to obtain the embedding shape. For the segmentation case that occurs on multiple devices, the all-reduce operation is performed between devices. That is, the embedding table of each device is transferred to other devices, connected, and then the sum is taken to obtain the corresponding shape.
2. The dual tensor parallel method for hybrid word frequency embedding according to claim 1, characterized in that: In step S2, the principle of higher compression intensity is followed when compressing existing dimensions, as the word frequency is lower.