A model acceleration training method, device, equipment and medium

Abstract

The application discloses a model acceleration training method and device, equipment and medium, and relates to the technical field of artificial intelligence. The method is applied to a CPU-GPU hybrid training architecture, and comprises the following steps: obtaining a to-be-trained data set; determining a hotness dimension type of an embedding vector and establishing a mapping table of index values of the embedding vector in GPU display memory to global identifiers of the embedding vector in CPU memory based on the access frequency of embedding layer parameters of the to-be-trained data set; determining operation instructions according to a model training framework, a training state and a cache state; and executing the operation instructions according to the hotness dimension type and the mapping table for each batch of model training, so as to realize model training. The space of the embedding layer parameter cache is finely divided, the memory space occupation of a single embedding layer parameter cache is reduced, more embedding layer parameters can be cached under the condition that the cache space is limited, the cache hit rate is improved, and the training throughput is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for accelerating model training. Background Technology

[0002] Current mainstream recommendation methods leverage the understanding capabilities of deep learning models to discover potential relationships between features and predict user interests. Deep learning-based recommendation models have unique structural characteristics. In addition to continuous numerical features, the training data for recommendation systems also includes a large number of discrete categorical features. Categorical features cannot be directly processed by neural networks; therefore, a large-scale embedding layer structure is typically used to transform the discrete categorical data into continuous numerical data before feeding it into the deep learning model. The size of the embedding layers in a recommendation system model far exceeds the available GPU memory on a single machine.

[0003] To address these characteristics, existing methods have been developed for cache-enhanced CPU-GPU hybrid training architectures. Under this architecture, the training of recommendation system models follows this pattern: due to the sparsity of access to the embedding layer by the recommendation system model, i.e., only a small number of parameters are accessed each time the embedding layer participates in the computation, existing technologies use caching methods to optimize training performance by creating a cache for embedding layer parameters in GPU memory, thereby reducing the number of communication operations between CPU memory and GPU memory for embedding layer parameters.

[0004] However, because the cache cannot cover enough accesses to embedding layer parameters, a large number of embedding layer parameters need to be transferred between the CPU and GPU, resulting in slow training. At this time, the training throughput is related to the cache hit rate. Cache missing will cause additional embedding layer parameter transfer overhead, affecting training performance. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for accelerating model training, thereby reducing the GPU memory usage of individual embedding layer parameter caches and improving training throughput.

[0006] According to a first aspect of the present invention, a method for accelerating model training is provided, applied to a CPU-GPU hybrid training architecture, wherein the CPU-GPU hybrid training architecture includes a CPU and a GPU, embedding layer parameters are stored in an embedding table in the CPU memory, and neural network parameters of non-embedded layers are stored in the GPU video memory, and an embedding layer parameter cache structure is set in the GPU video memory, the method comprising:

[0007] Obtain the training dataset;

[0008] Based on the access frequency of the embedding layer parameters in the training dataset, the popularity dimension type of the embedding vector is determined and a mapping table is established from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory; the popularity dimension type includes unpopular embedding vectors and popular embedding vectors.

[0009] Based on the model training framework, training state, and cache state, the operation instructions are determined, including cache loading instructions, fill instructions, and write-back instructions; the training state includes timestamps and parameter access order.

[0010] For each batch of model training, the operation instructions are executed according to the heat dimension type and the mapping table to send the embedding vectors used into the model training framework for model training in the training order of the dataset to be trained.

[0011] According to a second aspect of the present invention, a model acceleration training apparatus is provided, applied to a CPU-GPU hybrid training architecture, wherein the CPU-GPU hybrid training architecture includes a CPU and a GPU, embedding layer parameters are stored in an embedding table in the CPU memory, and non-embedded layer neural network parameters are stored in the GPU video memory, and an embedding layer parameter cache structure is set in the GPU video memory, comprising:

[0012] The data acquisition module is used to acquire the dataset to be trained.

[0013] The data preprocessing module is used to determine the popularity dimension type of the embedding vector based on the access frequency of the embedding layer parameters in the training dataset and to establish a mapping table from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory; the popularity dimension type includes unpopular embedding vectors and popular embedding vectors.

[0014] The cache management module is used to determine operation instructions based on the model training framework, training status, and cache status. The operation instructions include cache loading instructions, cache filling instructions, and cache write-back instructions. The training status includes timestamps and parameter access order.

[0015] The cache update module is used to execute the operation instructions according to the heat dimension type and the mapping table for each batch of model training, so as to send the embedding vectors used into the model training framework for model training in the training order of the dataset to be trained.

[0016] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to at least one of the processors; wherein,

[0019] The memory stores a computer program that can be executed by at least one of the processors, such that the at least one processor can execute the model acceleration training method according to any embodiment of the present invention.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the model accelerated training method according to any embodiment of the present invention.

[0021] According to a fifth aspect of the present invention, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the model acceleration training method of any embodiment of the present invention.

[0022] The technical solution of this invention further divides high-frequency embedding layer parameters and compresses their GPU memory usage, thereby supporting the caching of more embedding layer parameters and improving training throughput. It reduces the GPU memory usage of individual embedding layer parameter caches, thus caching more embedding layer parameters under limited cache space conditions, improving cache hit rate and training throughput.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a model acceleration training method provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is an example diagram of the embedded layer parameter cache structure of GPU memory in a model acceleration training method provided according to Embodiment 1 of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a model acceleration training device according to Embodiment 2 of the present invention;

[0028] Figure 4 This is a schematic flowchart of a model acceleration training device according to Embodiment 2 of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 The flowchart of a model acceleration training method provided in Embodiment 1 of the present invention is applicable to the accelerated training of recommendation system models. The method is applied to a CPU-GPU hybrid training architecture, which includes a CPU and a GPU. The parameters of the embedding layer are stored in the embedding table in the CPU memory, and the parameters of the non-embedded layer neural network are stored in the GPU memory. An embedding layer parameter cache structure is set in the GPU memory. The method can be executed by a model acceleration training device, which can be implemented in hardware and / or software and can be configured in an electronic device.

[0034] In this embodiment, the method is based on an existing cache-enhanced CPU-GPU hybrid training architecture, utilizing both CPU memory and GPU memory for training a deep learning-based recommendation system model, and includes an embedding layer parameter caching structure. During model training, embedding layer parameters are stored in CPU memory and dynamically transferred between the embedding table in CPU memory and the embedding layer parameter cache in GPU memory as needed. All neural network parameters other than embedding layer parameters are stored in GPU memory. Ultimately, all neural network parameters involved in the computation during training are stored in GPU memory.

[0035] The embedding layer parameters in the CPU memory are organized in the form of two-dimensional tensors. The index of each layer is the global ID of the embedding vector. There are a total of m tensors, and each tensor contains n floating-point numbers. Here, m is determined by the number of embedding vectors contained in the embedding layer, that is, the number of categorical objects, and n is determined by the model structure of the embedding layer.

[0036] In this embodiment, the embedding layer parameters can be understood as the core parameters in the recommender system model used to transform discrete categorical features (such as user ID, video classification, and product tags) into continuous numerical data. These parameters are fundamental to the model's understanding of the relationships between categorical features and are stored in CPU memory (due to their large size, a single-machine GPU memory cannot accommodate them). The two-dimensional tensor can be understood as a regular two-dimensional array, a structured way to store all embedding layer parameters in CPU memory. The global ID (cpu_id) of the embedding vector is a unique identifier (integer form, such as 1, 2, 3…m) for each embedding vector in the embedding layer. As the row index of the two-dimensional tensor, cpu_id directly locates the storage location of a specific embedding vector in CPU memory, serving as the core basis for parameter mapping between the CPU and GPU cache. Categorical objects can be understood as the original discrete feature carriers that generate the embedding vectors, referring to all categorical attributes (such as users, products, tags, and scenes) that need to be transformed through the embedding layer. The model structure of the embedding layer refers to the design scheme of the embedding layer in the trained deep learning recommendation model (DLRM), including the dimension (n) of the embedding vector and the connection method of the network layer, which is the core basis for fixing the dimension n of the embedding vector.

[0037] Specifically, the organization of the embedding layer parameters in CPU memory is done using a two-dimensional tensor. One layer has an index that is the global ID of the embedding vector, and there are a total of m tensors. Here, a tensor specifically refers to the storage carrier of a single embedding vector. Each tensor contains n floating-point numbers, where m is determined by the number of embedding vectors contained in the embedding layer, i.e., the number of categorical objects, and n is determined by the model structure of the embedding layer and is the input data.

[0038] The embedding layer parameter cache structure includes a popular embedding layer parameter cache area and a less popular embedding layer parameter cache area. The popular embedding layer parameter cache area is further divided into a statically stored header sub-cache area and a dynamically loaded tail sub-cache area. The ratio of the header sub-tensor to the tail sub-tensor in the popular embedding layer parameters is determined by a first hyperparameter, and the space allocation ratio between the popular embedding layer parameter cache area and the less popular embedding layer parameter cache area is determined by a second hyperparameter.

[0039] In this embodiment, a specific example can be used for demonstration. Figure 2 This is an example diagram of the GPU memory embedding layer parameter cache structure in a model acceleration training method provided in Embodiment 1 of the present invention. The embedding layer parameter cache structure can be understood as a space in GPU memory specifically used to store "highly accessed / recently needed" embedding layer parameters (essentially a temporary cache pool on the GPU side). The popular embedding layer parameter cache area can be understood as one of the sub-regions of the cache structure, specifically storing "highly accessed" embedding layer parameters (popular parameters). The unpopular embedding layer parameter cache area can be understood as another sub-region of the cache structure, specifically storing "low-frequency accessed" embedding layer parameters (unpopular parameters). The head sub-cache area (static storage) can be understood as a sub-region of the popular embedding layer parameter cache area, used to store the "head sub-tensor" of popular embedding vectors (see definition below). "Static storage" means that the parameters reside in the GPU cache for a long time after loading, without needing to be repeatedly fetched from the CPU. Its core function is to cover the high-frequency access part of popular parameters. The tail sub-buffer (dynamically loaded) can be understood as another sub-region of the popular embedding layer parameter buffer. It is used to store the tail sub-tensors of popular embedding vectors. Dynamic loading means that it is only loaded temporarily when needed in the current training batch. After use, it can be replaced by a new tail sub-tensor. Its core function is to adapt to the low-frequency access part of popular parameters, while compressing cache usage. The head sub-tensor can be understood as the first half of the popular embedding vector (complete tensor) after proportionally splitting it (e.g., if the complete vector contains 64 floating-point numbers, the head occupies 38 after proportional splitting). It is stored in the head sub-buffer and is the core part of the popular vector with the highest access frequency. The tail sub-tensor can be understood as the second half of the popular embedding vector (complete tensor) after proportionally splitting it (e.g., the tail of the above 64-dimensional vector occupies 26 floating-point numbers). It is stored in the tail sub-buffer and is a supplementary part of the popular vector with a lower access frequency. The first hyperparameter (represented by α) can be understood as a parameter used to determine the dimensionality ratio of the head sub-tensor to the tail sub-tensor in the popular embedding vector (e.g., α=0.6, then the head sub-tensor accounts for 60% of the full vector dimension, and the tail accounts for 40%). The second hyperparameter (represented by β) is used to determine the GPU memory space allocation ratio between the popular embedding layer parameter cache and the unpopular embedding layer parameter cache (e.g., β=0.7, then the popular cache accounts for 70% of the total cache space, and the unpopular cache accounts for 30%).

[0040] Specifically, the organization of embedded layer parameters in GPU memory is divided into two parts: a cache for popular embedded layer parameters and a cache for less popular embedded layer parameters. The storage space for popular embedded layer parameters can be further divided into a head and a tail, and then further subdivided into a statically stored head sub-cache and a dynamically loaded tail sub-cache. The head is statically stored, while the tail is dynamically loaded, thereby increasing the number of embedded layer parameters that can be statically cached. The two different partitions are determined by a first hyperparameter and a second hyperparameter, respectively. The first hyperparameter determines the ratio of the head sub-tensor to the tail sub-tensor in the popular embedded layer parameters, while the second hyperparameter determines the space allocation ratio between the popular and less popular embedded layer parameter caches.

[0041] One of the proposed caching strategies is suitable for datasets with a high degree of access skewness to the embedding layer. In this case, the number of cached parameters for less popular embedding layers is 2 * batchsize, and the number of cached parameters for the tail of popular embedding layers is 2 * batchsize.

[0042] like Figure 1 As shown, the method includes:

[0043] S110. Obtain the dataset to be trained.

[0044] In this embodiment, the training dataset can be understood as the original data set used to train a deep learning-based recommender system model (DLRM).

[0045] Specifically, the hybrid training architecture can obtain the training dataset specified by the user, for example, through storage media.

[0046] S120. Based on the access frequency of the embedding layer parameters in the training dataset, determine the hot dimension type of the embedding vector and establish a mapping table from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory.

[0047] In this embodiment, access frequency refers to the percentage of times each categorical feature in the training dataset calls the corresponding embedding layer parameters during model training. The popularity dimension can be understood as the classification result of the embedding vector based on "access frequency," including two categories: popular embedding vectors: high access frequency (e.g., the top 10% of parameters account for 75% of the total access), corresponding to frequently accessed categorical features; unpopular embedding vectors: low access frequency (e.g., the bottom 90% of parameters account for only 25% of the total access), corresponding to infrequently accessed categorical features. The index value (gpu_idx) of the embedding vector in GPU memory can be understood as a unique "location identifier" (integer form, such as 1, 2, 3…p+q) within the GPU memory cache after the embedding vector is loaded. The storage address of the embedding vector in the GPU cache can be directly located through the index value gpu_idx, facilitating fast reading and participation in computation. The global identifier (cpu_id) of an embedded vector in CPU memory can be understood as a unique identifier (in integer form, such as 1, 2, 3…m) for the embedded vector in the CPU memory embedding table, where m is the total number of embedded vectors (i.e., the total number of categorical objects). cpu_id is the core basis for the parameter mapping between the CPU and GPU cache. The mapping table can be understood as a structured data table recording the relationship between the index value gpu_idx and the global identifier cpu_id of the embedded vector, providing a bidirectional mapping that corresponds the index value (gpu_idx) of the embedded vector in the cache to its global identifier (cpu_id) in CPU memory.

[0048] Specifically, the training data can be preprocessed first. The frequency distribution of access to different embedding layer parameters can be determined by sampling statistics. The heat dimension type and range of the embedding vector can be divided according to the total capacity of the embedding layer cache in the GPU memory. Global identifiers can be assigned to the embedding vectors according to the heat dimension type and range. A unique index value can be assigned to each storage location in the GPU cache. A mapping table can be established from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory.

[0049] S130. Based on the model training framework, training state, and cache state, determine the operation instructions, which include cache loading instructions, fill instructions, and write-back instructions; the training state includes timestamps and parameter access order.

[0050] In this embodiment, the model training framework refers to the core software architecture / platform supporting the training of the deep learning recommendation system model (DLRM). Training state specifically refers to dynamic information about the training process related to caching decisions, which may include timestamps and parameter access order. Cache state refers to the real-time state of the embedded layer parameter cache structure in the GPU memory. Operation instructions can be understood as the instructions corresponding to the operations used for training. This can include: Cache loading instructions: The instruction cache update module retrieves "missing cache embedding layer parameters" (popular tail sub-tensors or unpopular complete tensors) from the embedding table in CPU memory, loads them into the corresponding GPU cache area, and updates the bidirectional mapping table (cpu_id↔gpu_idx) and timestamp; Filling instructions: Specifically refers to the instruction cache update module's "batch loading of high-frequency / core parameters" when the cache is initially empty (e.g., at the beginning of training) or when a certain type of cache area (e.g., the head of popular tensors) needs to be filled with parameters in batches (essentially a "batch loading instruction," distinct from single-batch sporadic loading); Write-back instructions: The instruction cache update module writes cache items in the GPU cache that are "not used in the future" or "may need to be replaced" (e.g., unpopular tensors with the oldest timestamps, popular tail tensors) back to the embedding table in CPU memory (ensuring data consistency after parameter updates), freeing up cache space for loading new parameters.

[0051] Specifically, the system can obtain the list of embedding layer parameters to be accessed in the current training batch and the current training stage from the model training framework, obtain the timestamps of loaded cached items from the training status, obtain the remaining space of the cache and cache hit results from the cache status, and determine the operation instructions, which include cache loading instructions, filling instructions and write-back instructions.

[0052] For example, a fill instruction is generated when the cache is empty (e.g., at the beginning of training); a cache load instruction is generated when there is a cache miss but the cache has remaining space; a write-back instruction combined with a cache load instruction is generated when there is a cache miss but the cache is full; a write-back instruction is generated when a cache item needs to be updated or training ends; and an advance cache load instruction is generated if it is predicted that a certain parameter will be accessed in subsequent batches.

[0053] S140. For each batch of model training, execute operation instructions according to the heat dimension type and mapping table to send the embedding vectors used into the model training framework for model training in the order of training of the dataset to be trained.

[0054] Specifically, for each batch of model training, the model training framework extracts training samples (e.g., batchsize=32) from the dataset to be trained, transforms the input categorical features from discrete class IDs into global identifiers for global embedding vectors, and searches for the corresponding index values ​​in the GPU's embedding layer parameter cache according to the popularity dimension type and range. Based on the index values ​​and mapping table, it executes operation instructions to send the used embedding vectors into the model training framework for model training in the order of training the dataset to be trained.

[0055] Independent rights have beneficial effects

[0056] Furthermore, based on the above embodiments, the steps of determining the heat dimension type of the embedding vector based on the access frequency of the embedding layer parameters of the training dataset and establishing a mapping table from the index value of the embedding vector in GPU memory to the global identifier of the embedding vector in CPU memory can be refined as follows:

[0057] By sampling and statistically analyzing the training dataset, the access frequency distribution of each embedding layer parameter in the embedding layer is determined. Based on the total capacity of the embedding layer parameter cache in the GPU memory, the ranges of popular and unpopular embedding layer parameters are divided. Based on the access frequency distribution, the ranges of popular and unpopular embedding layer parameters, global identifiers are assigned to each popular and unpopular embedding layer parameter, wherein the global identifier of the embedding vector of the popular embedding layer parameter is smaller than that of the unpopular embedding layer parameter. A mapping table is established from the index values ​​of the embedding vectors in the GPU memory to the global identifiers of the embedding vectors in the CPU memory.

[0058] In this embodiment, the total capacity of the embedding layer parameter cache in GPU memory refers to all the storage space specifically allocated in GPU memory for storing embedding layer parameters. The "popular embedding layer parameter range" refers to the set of high-frequency parameters determined by the access frequency distribution and the total cache capacity; it is a continuous interval of the embedding vector global identifier (cpu_id). The "unpopular embedding layer parameter range" refers to the set of low-frequency parameters outside the popular range; it is also a continuous interval of cpu_id.

[0059] Specifically, by sampling and statistically analyzing the training dataset, the access frequency distribution of each embedding layer parameter in the embedding layer can be determined. Based on the total capacity of the embedding layer parameter cache in GPU memory and the storage size of a single embedding vector, the number of parameters that GPU memory can cache is determined, and the range of popular and unpopular embedding layer parameters is divided based on the access frequency distribution. Global identifiers are assigned to each popular and unpopular embedding layer parameter based on the access frequency distribution, the range of popular embedding layer parameters, and the range of unpopular embedding layer parameters. For example, frequently accessed "popular" embedding layer parameters have small global embedding vector IDs (cpu_id), while infrequently accessed "unpopular" embedding vectors have large global embedding vector IDs (cpu_id). The global embedding vector identifiers of popular embedding layer parameters are smaller than those of unpopular embedding layer parameters. A mapping table is established from the index values ​​of embedding vectors in GPU memory to the global identifiers of embedding vectors in CPU memory.

[0060] Furthermore, based on the above embodiments, the steps of executing operation instructions according to the heat dimension type and mapping table to send the embedding vectors used to train the model into the model training framework according to the training order of the dataset to be trained can be refined as follows:

[0061] Extract discrete categorical features from the training dataset and convert them into corresponding target global identifiers. Based on the ranges of popular and unpopular embedding layer parameters, search for the index value corresponding to the target global identifier in the GPU's embedding layer parameter cache. For embedding layer parameters not in the GPU's embedding layer parameter cache, retrieve them from the embedding table in CPU memory to the GPU cache and update the mapping table and timestamp. If the GPU cache is full, write unused cache items back to CPU memory before performing a cache replacement operation. Feed the used embedding vectors into the model training framework for model training according to the training order of the dataset. If no cache items can be written back, wait for the current training batch to finish.

[0062] In this embodiment, the target global identifier specifically refers to the global identifier (cpu_id) of the embedded vector after the discrete categorical feature transformation in the current training batch. Unused cached items refer to cached data in the GPU cache that, based on "timestamp (TS) + parameter access order," will not be accessed in future training batches. The cache replacement operation, when the GPU cache is full and new parameters need to be loaded, performs a closed-loop operation of "removing old items, releasing space, and loading new items," which is essentially the result of the coordinated execution of "write-back instructions + load instructions."

[0063] Specifically, the model training framework extracts discrete categorical features from samples in the training dataset. Then, using a pre-generated mapping table between categorical object IDs and `cpu_id`, each discrete categorical feature is converted into a corresponding target global identifier `cpu_id`. The popularity of the target global identifier is determined based on the range of popular / unpopular parameters. For popular target `cpu_id`s, the mapping table corresponding to the "popular parameter cache" is queried (`cpu_id` → `gpu_idx`). For unpopular target `cpu_id`s, the mapping table of the "unpopular parameter cache" is queried to determine the query result. The query result can be categorized as a hit (i.e., finding the corresponding `gpu_idx`) or a miss (no corresponding `gpu_idx`). For embedding layer parameters not in the GPU's embedding layer parameter cache (i.e., the above miss query result), they can be retrieved from the embedding table in CPU memory to the GPU cache, and the mapping table and timestamp are updated. If no space is available in the GPU cache (hot tail / cold cache area), "unused cache items" (no future access scheduled) are selected. These unused cache items are first written back to the embedding table in CPU memory, and a cache replacement operation is performed. Missed target parameters are loaded into the GPU cache, and the mapping table and timestamps are updated. The used embedding vectors are then fed into the model training framework for model training according to the training order of the dataset. If the GPU cache is full and all cache items are determined to be "to be accessed in the future" (no cache items to write back), the new parameter loading instruction is paused, and the process waits for the current training batch to finish.

[0064] Based on the above embodiments, the steps of feeding the embedding vectors into the model training framework for model training according to the training order of the dataset to be trained can be refined as follows:

[0065] According to the training order of the dataset to be trained, if the embedding vector used is a popular embedding vector, the static head sub-tensor and dynamic tail sub-tensor of the popular embedding vector in the parameter cache of the popular embedding layer in the GPU are assembled into a complete tensor, and the complete tensor is sent into the model training framework for training; if the embedding vector used is a less popular embedding vector, the less popular embedding vector tensor in the parameter cache of the less popular embedding layer in the GPU is sent into the model training framework for training.

[0066] In this embodiment, a complete vector can be understood as a complete embedded vector tensor formed by concatenating and integrating the original dimensions of the embedded vectors in order to form a complete embedded vector tensor that meets the requirements of the model structure.

[0067] Specifically, following the training order of the dataset to be trained, the popularity type of the embedding vector is determined by the range of popular / unpopular parameters. If the embedding vector used is a popular embedding vector, the static head sub-tensor of the popular embedding vector is read from the GPU's head sub-cache; the dynamic tail sub-tensor of the popular embedding vector (already fetched via loading instructions or cache hit) is read from the GPU's tail sub-cache. The head and tail sub-tensors are assembled into a complete tensor according to the original dimensional order of the embedding vector (e.g., head sub-tensor corresponds to dimensions 1-38, tail sub-tensor corresponds to dimensions 39-64, n=64), ensuring that the dimensional order is consistent with the model training requirements. The complete tensor is then sent to the model training framework for training. For unpopular embedding vectors, the complete tensor of the unpopular vector (already fetched via loading instructions or cache hit) is directly read from the GPU's unpopular embedding layer parameter cache and sent to the model training framework for training. The transfer of embedding layer parameters between the CPU and GPU overlaps with the training of a batch of model parameters to hide its communication overhead.

[0068] As a first optional embodiment of this example, after executing operation instructions according to the heat dimension type and mapping table to send the embedding vectors used into the model training framework for model training according to the training order of the dataset to be trained, the method further includes:

[0069] When the training time for a set batch of models is reached, the first hyperparameter and the second hyperparameter are determined based on the training time of each batch; the embedding layer parameter cache structure is adjusted based on the first hyperparameter and the second hyperparameter.

[0070] In this embodiment, the batch setting can be understood as the number of training batches preset before training to test the training time of the model and calibrate the hyperparameters. The training time can be understood as the total time taken for a single training batch from "sample loading → feature transformation → cache query / pull → model forward / backward propagation → parameter gradient calculation".

[0071] Specifically, when the training time for a set number of batches of models is reached, the training framework can record the total time of each batch in real time. Based on the training time of each batch, it determines the first and second hyperparameters to ensure that the training time of one round covers the transmission events at the end of popular embedding layer parameters and the transmission time of less popular embedding layer parameters, or determines the values ​​of the first and second hyperparameters according to a predefined caching strategy. The embedding layer parameter caching structure is then adjusted based on the first and second hyperparameters.

[0072] The technical solution of this invention proposes a new general caching structure, which compresses the cache space occupied by a single embedding layer parameter without introducing new communication overhead, and further supports flexible allocation of GPU memory space according to model and device performance. It supports different model computation and communication loads. Furthermore, compared with existing prefetching techniques, this application proposal removes the limitation on the number of prefetching steps. This application proposal proposes a batch cache update algorithm based on the order of training data usage and timestamps, thereby achieving the loading of the embedding layer parameters required for training as early as possible, realizing more efficient transmission time of overlapping embedding layer parameters and training computation time. High-frequency embedding layer parameters are further subdivided, compressing their GPU memory usage, thereby supporting the caching of more embedding layer parameters and improving training throughput.

[0073] Example 3

[0074] Figure 3 This is a schematic diagram of a model acceleration training device provided in Embodiment 3 of the present invention. The device is applied to a CPU-GPU hybrid training architecture, which includes a CPU and a GPU. Embedded layer parameters are stored in an embedding table in the CPU memory, and non-embedded layer neural network parameters are stored in the GPU video memory. An embedded layer parameter cache structure is set in the GPU video memory, such as... Figure 3 As shown, the device includes:

[0075] Data acquisition module 31 is used to acquire the training dataset;

[0076] The data preprocessing module 32 is used to determine the popularity dimension type of the embedding vector based on the access frequency of the embedding layer parameters in the training dataset and to establish a mapping table from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory; the popularity dimension type includes unpopular embedding vectors and popular embedding vectors.

[0077] The cache management module 33 is used to determine operation instructions based on the model training framework, training status, and cache status. The operation instructions include cache loading instructions, fill instructions, and write-back instructions. The training status includes timestamps and parameter access order.

[0078] The cache update module 34 is used to execute the operation instructions according to the heat dimension type and the mapping table for each batch of model training, so as to send the embedding vectors used into the model training framework for model training in accordance with the training order of the dataset to be trained.

[0079] The embedding layer parameters in the CPU memory are organized in the form of two-dimensional tensors. The index of each layer is the global ID of the embedding vector. There are a total of m tensors, and each tensor contains n floating-point numbers. Here, m is determined by the number of embedding vectors contained in the embedding layer, that is, the number of categorical objects, and n is determined by the model structure of the embedding layer.

[0080] The embedded layer parameter cache structure includes a popular embedded layer parameter cache area and a less popular embedded layer parameter cache area. The popular embedded layer parameter cache area is further divided into a statically stored header sub-cache area and a dynamically loaded tail sub-cache area. The ratio of the header sub-tensor to the tail sub-tensor in the popular embedded layer parameters is determined by a first hyperparameter, and the space allocation ratio between the popular embedded layer parameter cache area and the less popular embedded layer parameter cache area is determined by a second hyperparameter.

[0081] Furthermore, the data preprocessing module 32 is specifically used for:

[0082] By sampling and statistically analyzing the dataset to be trained, the access frequency distribution of each embedding layer parameter in the embedding layer is determined.

[0083] Based on the total capacity of the embedded layer parameter cache in the GPU memory, the range of popular embedded layer parameters and the range of unpopular embedded layer parameters are divided.

[0084] Based on the access frequency distribution, the range of popular embedding layer parameters, and the range of unpopular embedding layer parameters, global identifiers are assigned to each of the popular embedding layer parameters and each of the unpopular embedding layer parameters, wherein the global identifier of the embedding vector of the popular embedding layer parameter is smaller than the global identifier of the embedding vector of the unpopular embedding layer parameter.

[0085] Establish a mapping table from the index values ​​of the embedded vectors in the GPU memory to the global identifiers of the embedded vectors in the CPU memory.

[0086] Furthermore, the cache update module 34 includes:

[0087] The first determining unit is used to extract discrete categorical features from the dataset used in this training and convert them into corresponding global target identifiers;

[0088] The second determining unit is used to search for the index value corresponding to the target global identifier in the embedding layer parameter cache of the GPU according to the popular embedding layer parameter range and the unpopular embedding layer parameter range;

[0089] The third determining unit is used to retrieve embedding layer parameters that are not in the embedding layer parameter cache of the GPU from the embedding table in the CPU memory to the GPU cache, and update the mapping table and timestamp.

[0090] The fourth determining unit is used to, if the GPU cache is full, first write unused cache items back to the CPU memory and then perform a cache replacement operation;

[0091] The fifth determining unit is used to send the embedding vectors used into the model training framework for model training according to the training order of the dataset to be trained;

[0092] The sixth determination unit is used to wait for the current training batch to end if there are no cached items to write back.

[0093] Specifically, the fifth determining unit is used for:

[0094] According to the training order of the dataset to be trained, if the embedding vector used is a popular embedding vector, the static head sub-tensor and dynamic tail sub-tensor of the popular embedding vector in the popular embedding layer parameter cache of the GPU are assembled into a complete tensor, and the complete tensor is sent into the model training framework for training.

[0095] If the embedding vector used is a less common embedding vector, the less common embedding vector tensor in the less common embedding layer parameter cache of the GPU is sent to the model training framework for training.

[0096] Optionally, the device may also include a hyperparameter update module.

[0097] The hyperparameter update module is specifically used for:

[0098] When the training time for a set batch of models is reached, the first hyperparameter and the second hyperparameter are determined based on the training time of each batch; and the embedding layer parameter cache structure is adjusted based on the first hyperparameter and the second hyperparameter.

[0099] For example, a specific example can be used to demonstrate the modules and steps. Figure 4 This is a flowchart illustrating a model acceleration training device provided in Embodiment 2 of the present invention. The data acquisition and preprocessing module acquires the training dataset from the dataset, classifying it into "cold" and "hot" categories. The cache management module makes decisions regarding cache filling and replacement operations based on the model training framework, training state, and cache state, determining operation instructions including cache loading, filling, and write-back instructions, and issuing these instructions. The cache update module executes the instructions issued by the cache management module, performing cache loading, write-back, concatenation, and update operations to ensure that the embedding vectors used are fed into the model training framework for model training according to the training order of the training dataset.

[0100] The model acceleration training device provided in the embodiments of the present invention can execute the model acceleration training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0101] Example 3

[0102] Figure 5 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0103] like Figure 5 As shown, the electronic device 40 includes at least one central processing unit (CPU) 41 and a graphics processing unit (GPU) 50, as well as memory such as a read-only memory (ROM) 42 and a random access memory (RAM) 43 communicatively connected to the at least one CPU 41. The memory stores computer programs executable by at least one processor. The CPU 41 can perform various appropriate actions and processes based on the computer programs stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the electronic device 40. The CPU 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0104] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0105] CPU41 and GPU50 execute the various methods and processes described above, such as methods for accelerating model training.

[0106] In some embodiments, the model acceleration training method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by CPU 41, one or more steps of the model acceleration training method described above may be performed. Alternatively, in other embodiments, CPU 41 may be configured to execute the model acceleration training method by any other suitable means (e.g., by means of firmware).

[0107] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0111] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0112] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0113] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the model acceleration training method of any embodiment of the present invention.

[0114] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for accelerating model training, characterized in that, An embedded layer parameter cache structure is applied to a CPU-GPU hybrid training architecture, wherein the CPU and GPU are included, the embedded layer parameters are stored in an embedding table in the CPU memory, and the neural network parameters of the non-embedded layers are stored in the GPU memory, and an embedded layer parameter cache structure is set in the GPU memory. The method includes: Obtain the training dataset; Based on the access frequency of the embedding layer parameters in the training dataset, the popularity dimension type of the embedding vector is determined and a mapping table is established from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory; the popularity dimension type includes unpopular embedding vectors and popular embedding vectors. Based on the model training framework, training state, and cache state, the operation instructions are determined, including cache loading instructions, fill instructions, and write-back instructions; the training state includes timestamps and parameter access order. For each batch of model training, the operation instructions are executed according to the heat dimension type and the mapping table to send the embedding vectors used into the model training framework for model training in the training order of the dataset to be trained.

2. The method according to claim 1, characterized in that, The embedding layer parameters in the CPU memory are organized in the form of two-dimensional tensors. The index of each layer is the global ID of the embedding vector. There are a total of m tensors, and each tensor contains n floating-point numbers. Here, m is determined by the number of embedding vectors contained in the embedding layer, that is, the number of categorical objects, and n is determined by the model structure of the embedding layer.

3. The method according to claim 1, characterized in that, The embedding layer parameter cache structure includes a popular embedding layer parameter cache area and a less popular embedding layer parameter cache area. The popular embedding layer parameter cache area is further divided into a statically stored header sub-cache area and a dynamically loaded tail sub-cache area. The ratio of the header sub-tensor to the tail sub-tensor in the popular embedding layer parameters is determined by a first hyperparameter, and the space allocation ratio between the popular embedding layer parameter cache area and the less popular embedding layer parameter cache area is determined by a second hyperparameter.

4. The method according to claim 1, characterized in that, The step of determining the heat dimension type of the embedding vector based on the access frequency of the embedding layer parameters in the training dataset and establishing a mapping table from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory includes: By sampling and statistically analyzing the dataset to be trained, the access frequency distribution of each embedding layer parameter in the embedding layer is determined. Based on the total capacity of the embedded layer parameter cache in the GPU memory, the range of popular embedded layer parameters and the range of unpopular embedded layer parameters are divided. Based on the access frequency distribution, the range of popular embedding layer parameters, and the range of unpopular embedding layer parameters, global identifiers are assigned to each of the popular embedding layer parameters and each of the unpopular embedding layer parameters, wherein the global identifier of the embedding vector of the popular embedding layer parameter is smaller than the global identifier of the embedding vector of the unpopular embedding layer parameter. Establish a mapping table from the index values ​​of the embedded vectors in the GPU memory to the global identifiers of the embedded vectors in the CPU memory.

5. The method according to claim 4, characterized in that, The step of executing the operation instruction according to the heat dimension type and the mapping table to send the embedding vectors used into the model training framework for model training according to the training order of the dataset to be trained includes: Extract discrete categorical features from the dataset used in this training and convert them into corresponding global identifiers for the target. Based on the range of popular embedding layer parameters and the range of unpopular embedding layer parameters, search for the index value corresponding to the target global identifier in the embedding layer parameter cache of the GPU; For embedding layer parameters that are not in the embedding layer parameter cache of the GPU, retrieve them from the embedding table in the CPU memory to the GPU cache, and update the mapping table and timestamp; If the GPU cache is full, unused cache items are first written back to the CPU memory before a cache replacement operation is performed. The embedding vectors used are fed into the model training framework for model training according to the training order of the dataset to be trained; If there are no cached entries to write back, wait for the current training batch to finish.

6. The method according to claim 5, characterized in that, The step of feeding the embedding vectors used into the model training framework for model training according to the training order of the dataset to be trained includes: According to the training order of the dataset to be trained, if the embedding vector used is a popular embedding vector, the static head sub-tensor and dynamic tail sub-tensor of the popular embedding vector in the popular embedding layer parameter cache of the GPU are assembled into a complete tensor, and the complete tensor is sent into the model training framework for training. If the embedding vector used is a less common embedding vector, the less common embedding vector tensor in the less common embedding layer parameter cache of the GPU is sent to the model training framework for training.

7. The method according to claim 3, characterized in that, After executing the operation instruction according to the heat dimension type and the mapping table to send the embedding vectors used into the model training framework for model training in the training order of the dataset to be trained, the method further includes: When the set batch of model training is reached, the first hyperparameter and the second hyperparameter are determined based on the training time of each batch. The embedding layer parameter cache structure is adjusted based on the first hyperparameter and the second hyperparameter.

8. A model acceleration training device, characterized in that, This is applied to a CPU-GPU hybrid training architecture, which includes a CPU and a GPU. Embedded layer parameters are stored in an embedding table in the CPU memory, while non-embedded layer neural network parameters are stored in the GPU video memory. An embedded layer parameter cache structure is set in the GPU video memory, including: The data acquisition module is used to acquire the dataset to be trained. The data preprocessing module is used to determine the popularity dimension type of the embedding vector based on the access frequency of the embedding layer parameters in the training dataset and to establish a mapping table from the index value of the embedding vector in the GPU memory to the global identifier of the embedding vector in the CPU memory; the popularity dimension type includes unpopular embedding vectors and popular embedding vectors. The cache management module is used to determine operation instructions based on the model training framework, training status, and cache status. The operation instructions include cache loading instructions, cache filling instructions, and cache write-back instructions. The training status includes timestamps and parameter access order. The cache update module is used to execute the operation instructions according to the heat dimension type and the mapping table for each batch of model training, so as to send the embedding vectors used into the model training framework for model training in the training order of the dataset to be trained.

9. An electronic device, characterized in that, The electronic device includes: At least one processor and at least one GPU; and a memory communicatively connected to at least one said processor and at least one said GPU; wherein... The memory stores a computer program that can be executed by at least one of the processors and at least one of the GPUs, the computer program being executed by at least one of the processors and at least one of the GPUs to enable at least one of the processors and at least one of the GPUs to perform the model acceleration training method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause the processor and GPU to execute the model acceleration training method according to any one of claims 1-6.

Images