Data processing method and equipment

Abstract

The invention discloses a data processing method and equipment, and relates to the technical field of artificial intelligence. The method is applied to a computing device deployed with a first accelerator card, and comprises the steps that the first accelerator card obtains a first sample, the first sample comprises an embedded table identifier and at least one feature identifier, the embedded table identifier is used for representing a corresponding embedded table, and each feature identifier corresponds to a feature vector in the embedded table; the first acceleration card uses a fusion operator to process the first sample, the fusion operator is used for querying feature vectors corresponding to the feature identifiers from an embedded table stored in target equipment according to the first sample, the index values are used for representing the positions of the feature vectors in the embedded table, and the target equipment is equipment for storing the embedded table; and the first accelerator card obtains a feature vector output by the fusion operator.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method and device. Background Technology

[0002] In training and inference scenarios such as recommendation models, the samples are often composed of high-dimensional, extremely sparse, categorical features. As new samples are continuously added, the number of features can reach millions or even hundreds of millions for higher accuracy, resulting in a model size of trillions of bytes. This necessitates collaborative training with multiple accelerator cards and related training servers. For example, sparse computation is performed on the central processing unit of the training server through cascaded processing of deduplication operators and feature mapping operators to obtain the corresponding feature vectors, and then dense computation is performed on the accelerator card. This often results in significant communication overhead between operators on one side and between multiple sides, leading to long model training times that fail to meet user needs. Summary of the Invention

[0003] This application provides a data processing method, apparatus, computing device, computer storage medium, and computer program product, which can improve data processing efficiency during model training.

[0004] In a first aspect, embodiments of this application provide a data processing method applied to a computing device with a first accelerator card deployed thereon. The method includes: the first accelerator card acquiring a first sample, the first sample including an embedding table identifier and at least one feature identifier, the embedding table identifier being used to characterize a corresponding embedding table, and each feature identifier corresponding to a feature vector in the embedding table; the first accelerator card processing the first sample using a fusion operator, the fusion operator being used to query the feature vectors corresponding to each feature identifier from the embedding table stored in a target device based on the first sample, the target device being a device storing the embedding table; and the first accelerator card acquiring the feature vectors output by the fusion operator.

[0005] In this embodiment, multiple accelerator cards (i.e., first accelerator cards) can be deployed on a single computing device. These accelerator cards can operate within the same model parallel architecture or parameter server architecture, collaboratively training and inferring models (such as recommendation models). When a model sample (i.e., a first sample) is input to an accelerator card, the accelerator card can use this sample as input to a fusion operator. The fusion operator then queries the feature vectors corresponding to each feature identifier from the storage device (i.e., the target device) representing the embedding table. In other words, the accelerator card can directly obtain the feature vector corresponding to the sample through a single fusion operator. Compared to multi-operator cascading and multi-side interaction to complete the mapping from sample to feature vector, this embodiment significantly reduces communication overhead and improves the data processing efficiency of the sample.

[0006] In some possible examples, the first sample includes multiple feature identifiers. After the first accelerator card acquires the first sample, the method includes: dividing the first sample into multiple sample slices, each sample slice including an embedding table identifier and at least one of multiple feature identifiers; generating a task flow for each sample slice, each task flow being used to implement the running logic of the fusion operator, each task flow including multiple tasks such as a first task and a second task in sequence; and executing a second task in a second task flow in parallel during the execution of the first task in the first task flow, wherein the first task flow and the second task flow are task flows for any two different sample slices, and the first task and the second task are both one of multiple tasks in their respective task flows.

[0007] In this way, by slicing the samples and processing them in batches, tasks in the task flow of different sample slices can be executed in parallel, thereby overlapping in time, reducing data processing time and improving processing efficiency.

[0008] In some possible examples, the first accelerator card includes a parallel processing unit (SiMT) and multiple computing cores. Each task flow includes a first task and a second task. After generating the task flow for each sample slice, the method specifically includes: distributing the tasks in each task flow to the multiple computing cores, such that the first computing core executes the first task in each task flow, and the second computing core executes the second task in each task flow; the multiple computing cores include the first computing core and the second computing core; and performing task scheduling through the parallel processing unit (SiMT) so that while the first computing core executes the first task in the first task flow, the second computing core executes the second task in the second task flow.

[0009] In this way, by binding the internal computing cores of the accelerator card to the tasks in a fine-grained manner, the parallel execution of multiple tasks between task flows can be achieved. This allows the computing resources of the accelerator to be reused, enabling the effective use of its own hardware computing power when performing sparse computation. This makes it more compatible with sparse computation, allowing the entire sparse computation to be performed on the accelerator card. It also effectively shortens the time spent processing sample data, thereby greatly reducing the time spent on model training and inference.

[0010] In some possible examples, the fusion operator includes parameter values ​​that characterize the storage strategy of the embedding table, the storage strategy including at least a model-parallel storage strategy or a parameter server storage strategy; the first accelerator card processes the first sample using the fusion operator, including: the first accelerator card, by executing the fusion operator, determines a target device from multiple accelerator cards that execute the model-parallel storage strategy on the embedding table, wherein the multiple accelerator cards include the first accelerator card and / or a second accelerator card, the second accelerator card being an accelerator card that collaborates with the first accelerator card in model training or inference; and, in the case of a parameter server storage strategy, determines the target device from a server that executes the storage strategy on the embedding table, the server being a server that collaborates with the first accelerator card in model training or inference.

[0011] In this embodiment, the constructed fusion operator includes parameter values ​​for declaring the embedded table storage strategy. When the fusion operator is executed, it can determine the target device based on the parameter values ​​under which architecture (such as model parallel architecture MP or parameter service PS) is determined, thereby facilitating the fast query of feature vectors.

[0012] In some possible examples, the embedded table is divided into multiple slices, which are distributed and stored on multiple accelerator cards. At least one feature identifier represents a feature vector belonging to a target slice. Multiple slices include the target slice. The first accelerator card processes the first sample using a fusion operator, including: the first accelerator card determines the target slice to which the feature vector corresponding to each feature identifier belongs by executing the slicing operator in the fusion operator, and identifies the accelerator card storing the target slice as the target device; the first accelerator card maps each feature identifier to a preset index value by executing the query / insertion operator in the fusion operator, and queries the feature vector corresponding to each index value from the target slice stored on the target device.

[0013] In this example, the fusion operator can incorporate multiple sub-operators, such as sharding operators and query / insert operators. Furthermore, multiple embedded table slices are stored across multiple accelerator cards; that is, the storage strategy for the embedded tables is a model-parallel storage strategy. Based on this storage strategy, the first accelerator card, by executing the sharding operator in the fusion operator, can determine the device storing the target slice from among these accelerator cards, and then perform feature mapping and other operations on that device by executing the query / insert operator, thereby quickly finding the corresponding feature vector.

[0014] In some possible examples, the first accelerator card determines the target slice to which the feature vector corresponding to each feature identifier belongs by executing the sharding operator in the fusion operator, and identifies the accelerator card storing the target slice as the target device. This includes: the first accelerator card executing the sharding operator to determine a first feature identifier and a second feature identifier from at least one feature identifier, where the feature vector corresponding to the first feature identifier belongs to the first slice stored on the first accelerator card, and the feature vector corresponding to the second feature identifier belongs to the second slice stored on the second accelerator card, and the target slice includes the first slice and the second slice; based on the first feature identifier and the second feature identifier, the first accelerator card and the second accelerator card are identified as the target device. Thus, by sharding feature identifiers whose corresponding feature identifiers are located on the same accelerator card, the target device can be determined.

[0015] In some possible examples, the first sample belongs to a sample set, which also includes a second sample processed by a second accelerator card; the embedding table is divided into multiple slices, which are distributed and stored on multiple accelerator cards, including a first accelerator card and a second accelerator card. The first accelerator card processes the first sample using a fusion operator, including: the first accelerator card swaps a second feature identifier in the first sample with a third feature identifier on the second accelerator card by executing an identifier swap operator in the fusion operator, wherein the feature vector corresponding to the second feature identifier belongs to a slice stored on the second accelerator card, the third feature identifier is determined from the second sample, and the feature vector corresponding to the third feature identifier belongs to a slice stored on the first accelerator card; the first accelerator card maps the third feature identifier to a preset index value by executing a query / insertion operator in the fusion operator, and queries the feature vector corresponding to the index value from the slice stored on the first accelerator card.

[0016] In this way, by exchanging the feature identifiers of the corresponding feature identifiers located on the same accelerator card, the accelerator cards that cooperate in model training or inference can each obtain the corresponding feature identifiers for local querying, which is conducive to quickly finding the corresponding vectors from a large number of data structures and improving query efficiency.

[0017] In some possible examples, after the first accelerator card maps the third feature identifier to a preset index value by executing the query / insertion operator in the fusion operator and queries the feature vector corresponding to the index value from the slice stored in the first accelerator card, the method includes: the first accelerator card sends the feature vector queried according to the third feature identifier to the second accelerator card by executing the vector exchange operator in the fusion operator, and obtains the feature vector queried by the second accelerator card according to the second feature identifier.

[0018] In this way, after the accelerator card completes the local query, it can exchange vectors through inter-card communication, enabling the accelerator card to obtain the feature vectors corresponding to its respective samples for model training or inference, thus ensuring the accuracy of sample data transmission, processing and use.

[0019] In some possible examples, before the first accelerator card swaps the second feature identifier in the first sample with the third feature identifier on the second accelerator card by executing the identifier swapping operator in the fusion operator, the method includes: the first accelerator card determining the number of second feature identifiers to be sent to the second accelerator card and the number of third feature identifiers to be received from the second accelerator card by executing the quantity swapping operator in the fusion operator. This can further improve the accuracy of subsequent feature identifier swapping.

[0020] In some possible examples, the embedding table is divided into multiple slices, which are distributed and stored on a server. The server is a server that collaborates with the first accelerator card to perform model training or inference. At least one feature vector representing a feature identifier belongs to a target slice. Multiple slices include the target slice. The first accelerator card processes the first sample using a fusion operator, including: the first accelerator card determines the target slice to which the feature vector corresponding to each feature identifier belongs by executing the slicing operator in the fusion operator, and determines the server storing the target slice as the target device; the first accelerator card maps each feature identifier to a preset index value by executing the query / insertion operator in the fusion operator, and queries the feature vector corresponding to each index value from the target slice stored on the target device.

[0021] In this way, under the parameter server storage strategy, the first accelerator card can determine the device storing the target slice from these accelerator cards by executing the sharding operator in the fusion operator, and then quickly find the corresponding feature vector by executing query / insertion operators to perform feature mapping and other operations in that device.

[0022] In some possible examples, the first accelerator card processes the first sample using a fusion operator, and further includes: the first accelerator card performing a deduplication operation on at least one feature identifier in the first sample by executing a deduplication operator in the fusion operator; generating a recovery matrix corresponding to at least one feature identifier, the recovery matrix being used to record at least one feature identifier; the first accelerator card mapping each deduplicated feature identifier to a preset index value by executing a query / insertion operator in the fusion operator, and querying the feature vector corresponding to the index value from the embedding table stored in the target device; and performing a recovery operation on the feature vector according to the recovery matrix to obtain each feature vector corresponding to at least one feature identifier.

[0023] In this way, the amount of data processed by deduplication can be reduced, which helps to improve data processing efficiency, and the accuracy of the data can be ensured by recovery operation.

[0024] Secondly, embodiments of this application provide a data processing apparatus applied to a computing device with a first accelerator card deployed thereon. The apparatus includes: an acquisition module for acquiring a first sample, the first sample including an embedding table identifier and at least one feature identifier, the embedding table identifier being used to characterize a corresponding embedding table, and each feature identifier corresponding to a feature vector in the embedding table; a processing module for processing the first sample using a fusion operator, the fusion operator being used to query the feature vectors corresponding to each feature identifier from an embedding table stored in a target device based on the first sample, the target device being a device storing the embedding table; and the acquisition module further for acquiring the feature vectors output by the fusion operator.

[0025] In some possible examples, the first sample includes multiple feature identifiers, and the processing module is further configured to: divide the first sample into multiple sample slices, each sample slice including an embedding table identifier and at least one of multiple feature identifiers; generate a task flow for each sample slice, each task flow being used to implement the running logic of the fusion operator, each task flow including multiple tasks in sequence; and execute a second task in a second task flow in parallel during the execution of the first task flow, wherein the first task flow and the second task flow are task flows for any two different sample slices, and the first task and the second task are both one of multiple tasks in their respective task flows.

[0026] In some possible examples, the first accelerator card includes a parallel processing unit (SiMT) and multiple computing cores. Each task flow includes a first task and a second task. The processing module is specifically used to: distribute the tasks in each task flow to the multiple computing cores, so that the first computing core executes the first task in each task flow, and the second computing core executes the second task in each task flow. The multiple computing cores include the first computing core and the second computing core. The parallel processing unit (SiMT) performs task scheduling so that while the first computing core executes the first task in the first task flow, the second computing core executes the second task in the second task flow.

[0027] In some possible examples, the fusion operator includes parameter values ​​that characterize the storage strategy of the embedding table, the storage strategy including at least a model-parallel storage strategy or a parameter server storage strategy; the processing device is specifically used to: by executing the fusion operator, in the case of a model-parallel storage strategy, determine a target device from multiple accelerator cards that execute the storage strategy on the embedding table, the multiple accelerator cards including a first accelerator card and / or a second accelerator card, the second accelerator card being an accelerator card that collaborates with the first accelerator card to perform model training or inference; in the case of a parameter server storage strategy, determine the target device from a server that executes the storage strategy on the embedding table, the server being a server that collaborates with the first accelerator card to perform model training or inference.

[0028] In some possible examples, the embedded table is divided into multiple slices, which are distributed and stored on multiple accelerator cards. At least one feature identifier represents a feature vector belonging to a target slice. Multiple slices include the target slice. The processing device is specifically used to: determine the target slice to which the feature vector corresponding to each feature identifier belongs by executing the slicing operator in the fusion operator, and determine the accelerator card storing the target slice as the target device; and map each feature identifier to a preset index value by executing the query / insertion operator in the fusion operator, and query the feature vector corresponding to each index value from the target slice stored on the target device.

[0029] In some possible examples, the processing device is specifically used to: determine a first feature identifier and a second feature identifier from at least one feature identifier by executing a slicing operator, wherein the feature vector corresponding to the first feature identifier belongs to a first slice stored on a first accelerator card, the feature vector corresponding to the second feature identifier belongs to a second slice stored on a second accelerator card, and the target slice includes the first slice and the second slice; and determine the first accelerator card and the second accelerator card as target devices based on the first feature identifier and the second feature identifier.

[0030] In some possible examples, the first sample belongs to a sample set, which also includes a second sample processed by a second accelerator card; the embedding table is divided into multiple slices, which are distributed and stored on multiple accelerator cards, including a first accelerator card and a second accelerator card. The processing device is specifically used to: exchange the second feature identifier in the first sample with the third feature identifier on the second accelerator card by executing the identifier exchange operator in the fusion operator, wherein the feature vector corresponding to the second feature identifier belongs to the slice stored on the second accelerator card, the third feature identifier is determined from the second sample, and the feature vector corresponding to the third feature identifier belongs to the slice stored on the first accelerator card; and map the third feature identifier to a preset index value by executing the query / insertion operator in the fusion operator, and query the feature vector corresponding to the index value from the slice stored on the first accelerator card.

[0031] In some possible examples, the processing device is specifically used to: send the feature vector queried according to the third feature identifier to the second accelerator card by executing the vector exchange operator in the fusion operator, and obtain the feature vector queried by the second accelerator card according to the second feature identifier.

[0032] In some possible examples, the processing device is specifically used to: determine the number of second feature identifiers to be sent to the second accelerator card and the number of third feature identifiers to be received from the second accelerator card by executing the quantity exchange operator in the fusion operator.

[0033] In some possible examples, the embedding table is divided into multiple slices, which are distributed and stored on a server. The server is a server that works with the first accelerator card to train or infer the model. At least one feature vector representing a feature identifier belongs to the target slice. Multiple slices include the target slice. The processing device is specifically used to: determine the target slice to which the feature vector corresponding to each feature identifier belongs by executing the slicing operator in the fusion operator, and determine the server storing the target slice as the target device.

[0034] By executing the query / insertion operator in the fusion operator, each feature identifier is mapped to a preset index value, and the feature vector corresponding to each index value is queried from the target slice stored in the target device.

[0035] In some possible examples, the processing device is further configured to: perform a deduplication operation on at least one feature identifier in the first sample by executing a deduplication operator in the fusion operator; generate a recovery matrix corresponding to the at least one feature identifier, the recovery matrix being used to record the at least one feature identifier; map each deduplicated feature identifier to a preset index value by executing a query / insertion operator in the fusion operator, and query the feature vector corresponding to the index value from an embedding table stored in the target device; and perform a recovery operation on the feature vector according to the recovery matrix to obtain each feature vector corresponding to the at least one feature identifier.

[0036] Thirdly, embodiments of this application provide a computing device, including: at least one memory for storing a program; at least one accelerator card for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the accelerator card is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0037] Fourthly, embodiments of this application provide a device cluster, including at least one computing device, the computing device including a memory and at least one accelerator card; the memory is used to store a computer program; the accelerator card is used to execute the computer program stored in the memory to implement the method described in the first aspect or any possible implementation of the first aspect.

[0038] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computing device, causes the computing device to perform the method described in the first aspect or any possible implementation thereof.

[0039] In a sixth aspect, embodiments of this application provide a computer program product, characterized in that, when the computer program product is run on a processor, it causes the processor to execute the method described in the first aspect or any possible implementation of the first aspect.

[0040] In a seventh aspect, embodiments of this application provide a chip, characterized in that it includes at least one processor and an interface; at least one processor obtains program instructions or data through the interface; at least one processor is used to execute program line instructions to implement the method described in the first aspect or any possible implementation of the first aspect.

[0041] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0042] Figure 1A This is a schematic diagram of the architecture of a distributed training system provided in an embodiment of this application;

[0043] Figure 1B This is a schematic diagram of the architecture of a distributed training system provided in an embodiment of this application;

[0044] Figure 2A This is a schematic diagram of the structure of a fusion operator provided in an embodiment of this application;

[0045] Figure 2B This is a schematic diagram of the structure of the fusion operator executed under the parallel model architecture in a specific embodiment of this application;

[0046] Figure 2C This is a schematic diagram of the fusion operator executed under the parameter server architecture in a specific embodiment of this application;

[0047] Figure 3A This is a schematic diagram of batch processing of samples based on the fusion operator in a specific example of this application;

[0048] Figure 3B This is a schematic diagram of task flow orchestration based on fusion operators in a specific example of this application;

[0049] Figure 4 This is a schematic diagram of the structure of an accelerator card provided in an embodiment of this application;

[0050] Figure 5 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0051] Figure 6 This is a schematic flowchart of a data processing method provided in a specific embodiment of this application;

[0052] Figure 7 This is a schematic flowchart of a data processing method provided in a specific embodiment of this application;

[0053] Figure 8This is a schematic flowchart of a data processing method provided in a specific embodiment of this application;

[0054] Figure 9 This is a schematic flowchart of a data processing method provided in a specific embodiment of this application;

[0055] Figure 10 This is a schematic flowchart of a data processing method provided in a specific embodiment of this application;

[0056] Figure 11 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0057] Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0059] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0060] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects.

[0061] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] To facilitate understanding of the technical solution of this application, some technical terms used in this document are explained below.

[0063] A recommendation model is an algorithm or system in the fields of machine learning and artificial intelligence. It aims to predict a user's rating or preference for a particular product or service and provide personalized recommendations based on these predictions. The purpose of recommendation models is to help users discover content they may be interested in, thereby improving user experience and satisfaction, and potentially increasing product sales or usage.

[0064] Sparse parameters (embedding) in recommendation models are a technique that transforms discrete words in text (such as user IDs, item IDs, etc.) into continuous vectors. During model training, the embedding layer assigns a fixed-size real-valued vector to each word, capturing the semantic and syntactic relationships between words.

[0065] An accelerator card is a processor used to handle computational tasks. The main purpose of an accelerator card is to improve the performance of a computer when performing computationally intensive tasks, especially workloads that require a large amount of parallel computing, such as deep learning, scientific computing, video encoding, and graphics rendering.

[0066] Central Processing Unit Server (CPU Server): A type of server designed for high-performance computing and data processing. It is typically equipped with a high-performance central processing unit (CPU) with a large number of cores, high clock speed, and powerful computing capabilities.

[0067] The parameter server strategy (PS) is a distributed strategy used in distributed machine learning systems to synchronously update model parameters. In this strategy, the entire model is hosted on one or more parameter servers, while multiple worker nodes are responsible for calculating gradients and updating the parameters.

[0068] Model-parallel strategy (MP) is a distributed machine learning approach that divides a large model into multiple parts, which are computed in parallel on different processors or machines. Unlike data-parallel strategy, model-parallel strategy does not divide the data into multiple parts, but rather divides the model itself so that different parts of the model can be processed simultaneously.

[0069] In computer vision and deep learning, a feature map is the output of an intermediate layer produced by a neural network after a series of operations such as convolution and pooling when processing input data. Feature maps reflect the feature representations of input data at different levels of abstraction. In natural language processing or recommender systems, hash functions can be used to map features to a fixed-size hash table, thereby reducing the number of features, speeding up computation, and reducing memory usage. In model feature mapping, `key2offset` is a data structure or function used to map a feature's key to its offset in data storage or indexing. This offset is equivalent to an index value, describing the position of the feature vector in the embedding table. This way, when a feature needs to be accessed, its position in data storage can be found directly through this mapping without traversing the entire collection.

[0070] Online training, also known as real-time training or incremental learning, refers to the continuous updating and adjustment of a model's parameters based on real-time data received during actual operation. This training method is suitable for situations requiring rapid response and adaptation to new data, such as online recommendation systems and real-time transaction analysis. Online training typically involves processing small batches of data or single data points to maintain low latency.

[0071] Offline training, also known as batch training or pre-training, refers to the comprehensive, one-time or periodic training of a model using a large historical dataset before deployment. This type of training is typically performed in environments with ample computing resources, fully utilizing all available data to optimize model performance. The results of offline training are usually saved and deployed to the production environment when needed.

[0072] Nearline training is a strategy that falls between online and offline training, combining the characteristics of both. In nearline training, the model is first initially trained in an offline environment and then deployed to a production environment. In the production environment, the model may be adjusted or fine-tuned to some extent based on real-time data, but such adjustments are usually less frequent than in online training.

[0073] An AI neural network NN acceleration chip (neural network processing unit, or NNP or NPU) is a chip (ASIC) specifically designed to accelerate deep learning and artificial intelligence computing. NNPs are specifically designed to accelerate the computation of neural network models to improve training and inference speed.

[0074] In AI training, all-to-all communication refers to the requirement that each node needs to communicate with all other nodes.

[0075] An AI computing engine (AI core) refers to a hardware or software component used to accelerate artificial intelligence (AI) computing tasks. It possesses strong parallel computing capabilities, enabling it to handle multiple computing tasks simultaneously to accelerate the processing of large-scale data, such as during the training and inference phases of neural networks. It is typically equipped with high-bandwidth memory, such as HBM (high bandwidth memory) or SRAM (Static Random-Access Memory), to reduce memory latency.

[0076] An AI central processing unit (AICPU) is a processor specifically designed for artificial intelligence applications. Unlike traditional CPUs, AICPUs are more focused on executing AI algorithms and machine learning tasks in their architecture and optimization, such as matrix operations and convolution operations in deep learning.

[0077] Training recommendation models faces challenges such as rapid updates to recommended items and extremely sparse features. With continuous updates to new data, the number of features can reach millions or even hundreds of millions, resulting in a model size exceeding trillions of bytes. This makes the model unsuitable for a single accelerator card or even the memory of a single machine, requiring multiple accelerator cards and a CPU server for collaborative training. Due to the performance limitations of accelerator cards, the CPU server often needs to perform various sparse computations during model training, such as sample deduplication and feature mapping (i.e., hash mapping), through cascading multiple operators. Meanwhile, the accelerator card needs to perform dense computations, such as obtaining embedding vectors and performing forward / backward training. However, due to the cascading of multiple operators, the frequent input and output of each operator's output as the input of the next often results in a loss of computational precision. Furthermore, significant communication overhead occurs between operators and between the two hardware sides, leading to lengthy model training times.

[0078] To shorten the training time of the model, this application provides a data processing method that, based on a large fusion operator, offloads the sparse computation of the model to the accelerator card for execution, thereby reducing the communication overhead during the model training process and accelerating the model training.

[0079] The technical solution of this application will now be described in conjunction with the accompanying drawings.

[0080] Figure 1AThis is a schematic diagram of the architecture of a distributed training system provided in an embodiment of this application. The distributed training system architecture is a model-parallel strategy (MP) architecture. The MP architecture may include one or more CPU servers 100, and multiple accelerator cards 102 may be mounted on the CPU servers 100. The CPU servers 100 may be physical machines or cloud servers (such as virtual machines or containers), and the accelerator cards 102 may be neural network processing units (NPUs) or graphics processing units (GPUs), but are not limited to these.

[0081] In the MP architecture, the model (e.g., a recommendation model) is divided into several parts (or sub-models), each of which can be processed independently on different accelerator cards 102. For example, model M is divided into two sub-models, m1 and m2. Accelerator card 102A processes m1 separately, and accelerator card 102 processes m2 separately. Each accelerator card 102 obtains the model parameters (mainly including weights w) of the part of the model it processes and independently calculates the gradient of that part of the model accordingly.

[0082] Specifically, before model training, the complete embedding table of the model can be segmented into multiple embedding table slices (slice0, slice01, ...), which are then distributed and stored on various accelerator cards 102. Accelerator cards 102 can interact with each other via all-to-all communication. During model training, each accelerator card 102 acquires its corresponding sample (shard0, shard1, ...). For example, accelerator card 102A acquires the first sample (shard0), accelerator card 102B acquires the second sample (shard1), accelerator card 102C acquires the third sample (shard3), ..., and the samples (shard0, shard1, ...) come from the same dataset.

[0083] Next, each accelerator card 102 processes its respective sample shard0, shard1, ... by obtaining the corresponding embedding vectors (also referred to as feature vectors in this paper) from the embedding table slices slice0, slice01, ... . Then, each accelerator card 102 uses the feature vectors corresponding to its samples as input to its assigned part of the model, performs forward propagation based on its model parameters to obtain the loss value, and performs backpropagation based on the loss value to obtain the gradient corresponding to its weight w. Then, the accelerator cards 102 communicate with each other through network communication to pass their gradients to other nodes. In this way, these accelerator cards 102 can calculate the gradients obtained from each other, update their assigned parts of the model, and update the model parameters of those parts of the model. Then, each accelerator card 102 obtains new samples and repeats the above process to recalculate the corresponding gradients, ..., until the model converges, resulting in a trained model.

[0084] Figure 1B This is a schematic diagram of another distributed training system architecture provided in an embodiment of this application. The distributed training system architecture is a parameter server (PS) architecture. The PS architecture includes a parameter server 200, a CPU server 100, and multiple worker nodes 201. Each worker node 201 can be a training server equipped with multiple accelerator cards 102. Worker nodes 201 can communicate with both the parameter server 200 and the CPU server 100. In the PS architecture, each accelerator card 102 obtains a complete model with the same model parameters (mainly including weights w).

[0085] Before training the model, the complete embedding table of the model can be split into multiple embedding table slices slice0, slice01, ... and distributed and stored on CPU server 100. In addition, CPU server 100 can also load the initialized model parameters to parameter server 200.

[0086] During model training, the parameter server 200 sends the initialized model parameters to each accelerator card 102. Each accelerator card 102 then acquires its corresponding sample shard0, shard1, ... Next, each accelerator card 102 processes its sample shard0, shard1, ... by retrieving the corresponding feature vector embedding from the embedding table slices slice0, slice01, ... Then, each accelerator card 102 uses its corresponding feature vector embedding as input to its assigned part of the model, performs forward propagation based on its model parameters to obtain the loss value, and performs backpropagation based on the loss value to obtain the gradient corresponding to its weight w. These gradients are then sent to the parameter server 200. Next, the parameter server 200 updates the model parameters based on these gradients and redistributes them to each accelerator card 102. Each accelerator card 102 then reacquires samples and repeats the above process to recalculate the corresponding gradients, ..., until the model converges, resulting in a trained model.

[0087] In the above Figure 1A , Figure 1BIn the distributed training system architecture shown, each accelerator card 102 acquires its own sample shard0, shard1, ... After obtaining the sample shards, since the feature identifiers (also called feature keys) in the sample shards, shards, ... are very discrete, preprocessing (such as deduplication) and feature engineering (such as feature mapping) are often required for the sample shards, shards, ... before the corresponding feature vector embedding can be found in the embedding table based on the feature identifier key for forward and backward propagation training. If the accelerator card 102 is not compatible with this type of sparse computation, the preprocessing and feature engineering operations need to be handed over to the CPU of the corresponding device (such as the CPU of CPU server 100 or the CPU of the training server) for execution. On the CPU, multiple operators are cascaded to process the data. For example, the deduplication single operator performs deduplication operation first, and the key of the deduplicated sample is then used for feature admission calculation by the feature admission single operator. Then, the feature mapping single operator performs feature mapping based on this calculation result, mapping these keys to specific feature vector embeddings, and then returns them to the accelerator card 102 for forward and backward propagation. This two-sided processing communication is time-consuming, especially in scenarios with high-volume, end-user-facing workloads, and cannot meet the daily update requirements of user model training. It should be noted that, depending on the characteristics of the hardware platform's computing resources (such as accelerator cards), for example, the number of processor cores and hardware architecture of the accelerator card 102, computing resources have different affinities for different types of computation (such as sparse and dense computation). Affinity refers to the degree to which computing resources can effectively utilize hardware computing power (abbreviated as computing power) when performing different types of computations. Computational ability is a fundamental component of mathematical ability, referring to the ability to use relevant computational knowledge to perform calculations, reason, and obtain computational results. It should be understood that the computing power of a computing resource performing any type of computation is related to the time it takes to perform that type of computation; that is, the computing power of a computing resource performing that type of computation is related to the time it takes. If the computing resource can effectively utilize its computing power when performing that type of computation, the time is shorter; conversely, the time is longer.

[0088] Therefore, in this embodiment, before training a model (hereinafter, the recommendation model is used as an example), a large fusion operator (OP) can be constructed to abstract multiple operations such as sparse computation of the model into a function execution logic, which is then executed on the accelerator card 102. This allows the accelerator card 102 to use the acquired samples as input to the fusion operator OP, and directly obtain the corresponding return value, i.e., the feature vector embedding, by performing the function execution logic of the fusion operator on the input. In other words, the fusion operator can implement the sparse computation of the model as a process similar to database query logic, exposing only one input and one output interface. Compared with the cascaded processing of multiple operators on both sides, this embodiment can avoid the accuracy loss caused by frequent input and output, the long time consumption caused by frequent memory read and write and execution of complex control statements, as well as communication time, thereby maintaining higher computational accuracy and improving data processing efficiency.

[0089] For example, the function execution logic of a fusion operator OP can include multiple sub-execution logics, each of which can be an abstract implementation of a sub-operator. In this example, please refer to... Figure 2A As shown, the fusion operator (OP) can include at least a serial partition operator and a lookupor / insert operator. Wherein:

[0090] When the partition operator is executed, it can be used to determine the target device from multiple accelerator cards 102 that stores the feature vector embedding corresponding to the feature identifier key in the current sample. For example, sample shard0 includes an embedding table identifier (table_id) and multiple feature identifier keys (the complex form of key, the same below) of {0,1,1,3,5,6,5}. Each key can represent a user's hobby, such as "0" representing football, "1" representing volleyball, etc. Each key can have a unique corresponding feature vector (embedding) in the embedding table represented by the identifier (table_id). The object storing the feature vector (such as accelerator card 120, or CPU server 100 under parameter server architecture PS) can be used as the target device.

[0091] When the lookuporinsert operator is executed, it can be used to query the feature vectors corresponding to the feature identifiers keys in the current sample from the embedding table represented by the embedding table identifier stored in each target device. For example, sample shard0 includes an embedding table identifier (table_id) and multiple feature identifier keys {0,1,1,3,5,6,5}. The feature vector embeddings corresponding to {0,1,1,3,5,6,5} are all stored in CPU server 100. Then, when accelerator card 102 executes the function execution logic of the fusion operator, it can query the feature vector embeddings corresponding to {0,1,1,3,5,6,5} from the corresponding embedding table stored in CPU server 100. If the feature vector embeddings corresponding to {0,1,1,3,5,6,5} are distributed and stored in multiple accelerator cards 102, then any accelerator card 102 can execute the function running logic of the fusion operator to query the feature vector embeddings corresponding to {0,1,1,3,5,6,5} from the corresponding embedding table slices stored in these accelerator cards 102.

[0092] In this embodiment, each accelerator card 102 can perform the same logical processing on its respective acquired sample shard0, shard1, ... by executing the same fusion operator OP.

[0093] For example, the function execution logic of the fusion operator OP can vary depending on the embedding table storage strategy of the corresponding model. The embedding table storage strategy can include, as described above... Figure 1A The storage strategy shown in the model-parallel architecture (i.e., the model-parallel storage strategy) involves dividing the model's embedding table into multiple slices (slice0, slice01, ...) and then distributing the storage across multiple accelerator cards 102. The embedding table storage strategy may also include the methods described above. Figure 1B The storage strategy shown on the parameter server architecture (i.e., the parameter server storage strategy) is to divide the model's embedding table into multiple slices slice0, slice01, ... and then distribute the storage on the CPU server 100.

[0094] In one possible implementation, the fusion operator (OP) may specifically include a serial partitioning operator, a feature identifier key exchange operator (cnt all2all), a feature identifier exchange operator (keyall2all), a lookuporinsert operator, and a feature vector exchange operator (emb all2all). This implementation can be executed when the model embedding table's storage strategy is a model parallel storage strategy. For example... Figure 2B As shown, accelerator cards 102A-102D collaboratively train and infer a recommendation model under the MP architecture. The recommendation model is divided into multiple sub-models m1-m4, and each accelerator card 102A-102D is responsible for one of these sub-models. Each of the four accelerator cards 102A-102D also stores slices (slice0-slice3) of the recommendation model's embedding table. Furthermore, the sample set (dataset) is divided into four groups (shard0-shard4) and distributed to the four accelerator cards 102A-102D. Each accelerator card 102A executes a fusion operator (OP) to process its respective sample. The processing principle within this fusion operator OP is as follows (the following example uses the fusion operator OP executed on accelerator card 102A; it should be understood that the processing within the fusion operator OP executed on other accelerator cards 102B-102D is similar):

[0095] When the fusion operator (OP) is executed by accelerator card 102A, its partitioning operator can divide the acquired sample shard0 into various feature identifiers whose corresponding feature vector embeddings (also referred to as "emb" in this paper) are located in the same target device. For example, if the keys in sample shard0 are {0,1,1,3,5,6,5}, the emb corresponding to "0" belongs to slice0 (stored in accelerator card 102A), the emb corresponding to "1" and "5" belongs to slice1 (stored in accelerator card 102B), the emb corresponding to "6" belongs to slice2 (stored in accelerator card 102C), and the emb corresponding to "3" belongs to slice3 (stored in accelerator card 102D). Optionally, since the sample shard0 contains duplicate keys, a deduplication operation can be performed. After removing the two "1"s and two "5"s, a single "1" and a single "5" are obtained, simultaneously generating a recovery matrix [0,1,1,4,3,2,2] that records the original keys {0,1,1,3,5,6,5}. Figure 2B As shown, keys (represented by numbers), counts, slices, or other data structures (such as key2offset) located within the same colored boxes indicate that they correspond to the same accelerator card.

[0096] Next, the number exchange operator (cnt all2all) of the fusion operator OP can determine the number of feature identifiers that accelerator card 102A needs to exchange with other accelerator cards 102B to 102D through all2all communication, so as to exchange the above-mentioned feature identifiers to the corresponding accelerator cards 102. For example, after shard0 is deduplicated, accelerator card 102A has two keys "1" and "5" to send to accelerator card 102B, one key "6" to send to accelerator card 102C, and one key "3" to send to accelerator card 102D. Similarly, other accelerator cards 102B to 102D execute the cnt all2all of the fusion operator OP to notify accelerator card 102A that it still needs to receive 0, 1, and 2 keys from accelerator cards 102B to 102D respectively.

[0097] Then, the feature identifier exchange operator (key all2all) of the fusion operator OP can exchange keys with other accelerator cards 102B to 102D according to the quantity determined by the quantity exchange operator (cnt all2all) through all2all communication, thereby exchanging the above-mentioned feature identifiers to the corresponding accelerator cards 102. For example, the feature identifier exchange operator sends the two keys "1" and "5" to accelerator card 102B, sends "6" to accelerator card 102C, and receives the key "4" sent by 102C, and sends "3" to accelerator card 102D and receives the two keys "4" and "8" sent by 102D. In this way, the embedding corresponding to the key on each accelerator card 102A to 102D is stored locally on the accelerator card.

[0098] Next, the lookup or insert operator of the fusion operator OP queries (lookup) the various keys currently owned on the accelerator card 102A in the local storage slice 0 (e.g., ...). Figure 2BThe embeddings corresponding to {0, 4, 4, 8} are shown below. If no corresponding embedding can be found for any key, a default embedding can be inserted into the corresponding embedding table. Specifically, the lookuporinsert operator can perform feature admission calculations on the keys to be queried based on preset admission conditions to filter features that are helpful for the current model training (e.g., stronger correlation with the predicted target value). For example, rather than limiting the conditions, the admission conditions can include that any key has a preset number (e.g., count≥2) in the current batch of samples shard0, shard1, ... before performing a lookuporinsert operation on that key, in order to ensure the quality of the sample data and the effectiveness of the model training. For example, for the recommendation system of an online video platform, when a user has more than 10 click records for a certain type of video (i.e., the number of keys describing this information in the sample set reaches 10), the lookuporinsert operator will perform a lookuporinsert operation on that key. This ensures that the model can make accurate recommendations based on sufficient behavioral data. Then, the lookuporinsert operator maps each key that meets the admission criteria to the corresponding index value in the key2offset of the embedded table. These index values ​​represent the position of the embedding vector in the embedded table. In this way, the embedding vector corresponding to the key can be found based on the mapped index value.

[0099] Next, the feature vector exchange operator (emb all2all) of the fusion operator OP exchanges the embeddings queried locally from accelerator card 102A with the embeddings queried locally from other accelerator cards 102B-102D, obtaining all the embeddings corresponding to each sample. For example, through embedding exchange, accelerator card 102A can obtain the embeddings corresponding to keys {0,1,1,3,5,6,5} in sample shard0, and accelerator card 102B can obtain the embeddings corresponding to keys {2,7,2,9,6,3,3} in sample shard1. It should be noted that if deduplication is performed on the keys before and after the partition operation, these embeddings can be recovered using the recovery matrix to reconstruct the vectors corresponding to all identifiers in each sample before deduplication. For example, after deduplication of sample shard0, the accelerator card 102A obtains the embeddings corresponding to keys{0,1,5,6,3} via emb all2all. After restoration processing using the restoration matrix, the embeddings corresponding to the deduplicated keys "1" and "5" are filled back in, resulting in the embeddings corresponding to keys{0,1,1,3,5,6,5} in sample shard0. Of course, if no deduplication operation is performed on the keys before or after the partition operation, there is no need to generate the restoration matrix, and therefore no need to perform vector restoration during the emb all2all stage.

[0100] In this way, each accelerator card 102A~102D obtains the embedding corresponding to its own sample as the input of its assigned part of the model, and then performs forward training and backward training of its assigned part of the model based on the weight w it received, until a trained model is obtained.

[0101] In this implementation, based on the all-to-all communication between accelerator cards 102, the executed fusion operator internally allows the keys of each sample to be exchanged on the accelerator card 102 storing the corresponding eMB for local query processing, which improves query efficiency. Then, through eMB exchange, the accelerator card 102 retrieves the vector corresponding to each sample for model training and inference. In other words, the computation of the entire fusion operator operation can be offloaded to the accelerator card 102. On the user side, only one input (i.e., a sample) is needed to obtain the corresponding feature vector output for subsequent model training, which is convenient and efficient.

[0102] In another possible implementation, such as Figure 2CAs shown, the fusion operator OP can specifically include a serial deduplication operator (unique), a partitioning operator (partition), and a lookupor insert operator (lookupor insert). This implementation can be executed when the storage strategy of the model embedding table is a parameter server storage strategy. For example... Figure 2C As shown, accelerator cards 102A to 102D collaborate on the training and inference of a recommendation model under the PS architecture. That is, each accelerator card 102A to 102D obtains a complete recommendation model for training and inference. The model's embedding table slices (slice0, slice1, ...) are distributed and stored on the CPU server 100 under the PS architecture. The CPU server 100 loads the model parameters (including weights w) to the parameter server 200, which then distributes them to each accelerator card 102A to 102D. During training, the dataset is divided into four groups, shard0 to shard4, and distributed to four accelerator cards 102A to 102D. Each accelerator card 102 executes a fusion operator (OP) to process its respective samples. When the fusion operator OP is executed, its function logic can be partially executed on the accelerator card 102 and partially on the CPU server 100. For example, the deduplication operator (unique) and partitioning operator (partition) are executed on the first accelerator card 102, while the lookup / insert operator (lookupor insert) is executed on the CPU server 100. It should be understood that the fusion operator OP can be pre-deployed on both the accelerator card 102 and the CPU server 100 (with a portion deployed on each side), or loaded onto both the accelerator card 102 and the CPU server 100 during sample computation (with a portion loaded on each side). Specifically, the processing principle within the fusion operator OP is as follows (the following example uses the processing within the fusion operator OP executed on accelerator card 102A; it should be understood that the processing within the fusion operator OP executed on other accelerator cards 102B to 102D is similar):

[0103] When the fusion operator OP is executed by the accelerator card 102A, the deduplication operator (unique) of the fusion operator OP deduplicates the sample shard0. For example, if the keys in sample shard0 are {0,1,1,3,5,6,5}, after deduplicating the two "1"s and two "5"s, the deduplicated keys are {0,1,5,6,3}. At the same time, a recovery matrix is ​​generated that records the original keys {0,1,1,3,5,6,5}.

[0104] Next, the partition operator of the fusion operator (OP) divides the acquired sample shard0 into various feature identifiers whose corresponding feature vector embeddings (which can also be simply referred to as "emb" in this article) are located on the same target device, thus determining the storage location of the feature vector corresponding to each key in sample shard0. These feature identifiers are then passed to the lookuporinsert operator, and the operator waits for the return value of the lookuporinsert operator.

[0105] Next, the lookuporinsert operator of the fusion operator (OP) queries (or requests the CPU server 100 to query) the embedding vectors corresponding to the current (deduplicated) keys on the accelerator card 102A. The retrieved embeddings can then be sent back to the accelerator card 102A by the CPU server 100 (sentData). Since the currently obtained embedding vectors are based on the deduplicated keys, the fusion operator (OP) restores the embedding vectors corresponding to the original keys of the sample according to the restoration matrix before outputting the vector. For example, after deduplicating the keys {0,1,5,6,3} based on sample shard0, the embedding vectors of the deduplicated {1} and {5} need to be filled back according to the restoration matrix to obtain the embedding vectors corresponding to the original keys {0,1,1,3,5,6,5}.

[0106] Then the accelerator card 102A can use the acquired embedding as input to the recommendation model, and perform forward and backward training (FP&BP) operations based on its assigned model parameters until a trained model is obtained.

[0107] In this implementation, the fusion operator is executed by the accelerator card 102. The entire function execution logic of the fusion operator is internally run separately on the accelerator card 102 and the CPU server 100. Based on the powerful computing capabilities of the CPU server 100, the corresponding embedding can be quickly queried from all embedding table slices and returned to the accelerator card 102. However, from the user's perspective, the accelerator card 102 still only exposes one input and one output of the fusion. That is, the accelerator card 102 only needs one input (i.e., sample) to obtain the corresponding feature vector emb output based on the function execution logic of the fusion operator. The operation is convenient and the processing efficiency is fast.

[0108] In some possible examples, the function execution logic of the fusion operator OP in the above implementations can be implemented internally as multiple operations on sample shard0, shard1, ... To complete data processing more quickly and accelerate model training and inference, in this example, during the processing of its own samples on accelerator card 102, the samples on each accelerator card 102 can be divided into multiple sample slices and then processed in batches. For example Figure 3A As shown, compared to Figure 2B The data processing flows shown are roughly similar, with the main difference being that... Figure 3A The example shown does not require deduplication (uniqueness) of samples shard0 to shard3. Instead, each shard0 to shard3 is divided into two slices and processed in two batches. In the first batch, accelerator cards 102A to 102D process one sample slice (shard01, shard11, shard21, shard31) respectively. In the second batch, accelerator cards 102A to 102D process the other sample slice (shard02, shard12, shard22, shard32) respectively. The processing of each sample slice in each batch is basically similar. After all batches of sample slices have been processed, each accelerator card 102A to 102D integrates the embedding vectors obtained by the fusion operator (OP) in all batches for use in model training or inference.

[0109] Furthermore, in this example, corresponding task streams can be orchestrated for each batch of sample slices to enable parallel execution of task streams across batches, thereby saving data processing time. Please refer to [reference needed]. Figure 3B As shown, the principle of task flow orchestration on the accelerator card 102 is as follows:

[0110] After the accelerator card 102 acquires samples (taking shard0 as an example), it can divide shard0 into multiple ordered sample slices. For example, the keys {0,1,1,3,5,6,5} in shard0 can be divided into three ordered sample slices: shard01, shard02, and shard03. Shard01 includes {0,1,1}, shard02 includes {3,5,6}, and shard03 includes {5}. It should be noted that "ordered" here means that the sample slices can have a corresponding processing order. For example, shard01 can be the first batch of sample slices processed on the accelerator card 102, shard02 the second batch, and shard03 the third batch. Furthermore, the embedded table identifier (table_id) in shard0 is always associated with each of the sample slices shard01, shard02, and shard03.

[0111] Next, the accelerator card 102 can generate a task stream (stream1), a task stream (stream2), and a task stream (stream3) for each sample slice (shard01, shard02, and shard03), respectively. Each task stream is used to implement the execution logic of the fusion operator (OP). Each task stream includes at least a series of tasks. In other words, each task stream of sample slices (shard01, shard02, and shard03) can include the same multiple tasks, and tasks in any task stream must be executed sequentially until one task is completed before the next task can begin.

[0112] During the execution of various task flows for sample slices shard01, shard02, and shard03 in multiple processing batches, for any two sample slice data flows, while executing one task of one task flow, one task of another task flow can be executed in parallel, so that the timing of the tasks in the former task flow overlaps with the timing of the tasks in the latter task flow.

[0113] In this embodiment, reference can be made to Figure 4As shown, the accelerator 102 may internally include a Simultaneous Multi-Threading (SiMT) unit and multiple computing cores 1, 2, ... Each computing core 1, 2, ... can be used to execute the same type of task in a task flow. For example, computing core 1 in accelerator card 102 can be bound to execute the first task in task flows stream1, stream2, and stream3, and computing core 2 in accelerator card 102 can be bound to execute the second task in task flows stream1, stream2, and stream3. By way of example and not limitation, computing cores 1, 2, ... can be an AI CPU processor for AI computing or an AI computing engine AIcore.

[0114] The parallel processing unit (SiMT) can be used for task scheduling, so that while computing core 1 is executing the first task of task stream 1, computing core 2 can execute the second task of stream 2 in parallel; while computing core 2 is executing the second task of task stream 2, computing core 1 can execute the first task of stream 3 in parallel, and so on. In this way, the task streams of each sample slice achieve task temporal overlap, thereby further saving the time spent processing sample data.

[0115] Furthermore, the accelerator 102 may also include a communication unit 1021 for external communication during task flow execution. For example, the communication unit 1021 may be a Direct Memory Access (DMA) unit, but is not limited thereto.

[0116] Please refer again to a specific example. Figure 3B As shown, the accelerator card 102 includes computing cores such as AICPU and AIcore, a communication unit 1021, and a parallel processing unit SiMT. The accelerator card 102 generates task streams stream1, stream2, and stream3 for each sample slice (shard01, shard02, and shard03) of shard0. Each task stream (stream1, stream2, and stream3) includes serial tasks such as partition, key all2all, lookuporinsert, and emball2all. The AICPU can be bound to execute the partition task of each task stream, the communication unit 1021 can be used to execute the all2all task of each task stream, and the AIcore can be used to execute the lookuporinsert operation of each task stream.

[0117] Therefore, through task scheduling by the parallel processing unit SiMT, after the AICPU completes the partition of stream1, it communicates with other accelerator cards via the communication unit 1021 to execute the key all2all operation of stream1. During the execution of the key all2all task of stream1, since the AICPU has already completed the partition of stream1, it can continue to execute the partition of stream2. Similarly, after the AIcore completes the lookupor insert operation of stream1, it hands it over to the communication unit 1021 to communicate with other accelerator cards to execute the emb all2all operation of stream1. During the execution of the emb all2all task of stream1, since the AIcore has already completed the lookupor insert of stream1, it can continue to execute the lookupor insert of stream2. Similarly, the parallel execution of stream2 and stream3 by the AICPU, AIcore, and other computing cores and the communication unit 1021 is similar. This continues until all tasks in streams 1, 2, and 3 have been completed.

[0118] In this way, the accelerator 102, based on the task scheduling of the parallel processing unit SiMT, performs concurrent execution of sparse computation and all-to-all communication operations between different task flows by computing cores 1, 2, ... in a time-overlapping manner. That is, by reusing the computing resources of the accelerator and binding tasks in a fine-grained manner, it can effectively utilize its own hardware computing power when performing sparse computation, making it more compatible with sparse computation. This not only allows the entire sparse computation to be executed on the accelerator card 102, but also effectively shortens the time spent processing sample data, thereby greatly reducing the time spent on model training and inference.

[0119] Next, a data processing method provided by an embodiment of this application will be described with reference to the accompanying drawings.

[0120] For example, Figure 5 The diagram illustrates a data processing method according to an embodiment of this application. This method can be applied to a computing device A configured with an accelerator card 102. The computing device A can be a physical machine, hereinafter the same. It should be understood that in other possible examples, this method can also be executed in a device, platform, or device cluster. Figure 5 As shown, the data processing method may include:

[0121] S510, the first accelerator card acquires a first sample, the first sample including an embedding table identifier and at least one feature identifier.

[0122] In this embodiment, before model training, the model's embedding table can be divided into multiple shards (shard0, shard1, shard...) and stored according to different storage strategies. One storage strategy is a model parallel storage strategy, where multiple shards (shard0, shard1, shard...) are distributed and stored on multiple accelerator cards 102 under an MP architecture. These accelerator cards 102 can be configured in the same computing device A or in different computing devices A and B within the same cluster. Another storage strategy is a parameter server storage strategy, where multiple shards (shard0, shard1, shard...) are distributed and stored on a CPU server 100 of a PS architecture.

[0123] For example, the accelerator card of computing device A (also referred to herein as the first accelerator card, such as accelerator card 102A) can collaboratively train the model with other accelerator cards (also referred to herein as the second accelerator cards, such as accelerator cards 102B, 102C, ... for ease of distinction) under an MP or PS architecture. It should be understood that these accelerator cards 102 can be deployed on the same computing device A, or on different computing devices A, B, ... in the same cluster. During model training, each accelerator card 102 can first acquire its own sample shard0, shard1, ... These sample shards 0, shard1, ... are different from each other and come from a training dataset. For ease of description, the sample acquired by the first accelerator card can be referred to as the first sample, the sample acquired by the second accelerator card can be referred to as the second sample, and so on.

[0124] Each sample (including the first sample) may include an embedding table identifier and at least one feature identifier. An embedding table identifier (table_id) can represent an embedding table, and a feature identifier (key) can correspond to a feature vector in the embedding table, i.e., an embedding vector.

[0125] Each accelerator card 102 can perform the same processing procedure on its respective acquired training data shard0, shard1, ... The following description uses the processing of the first sample shard0 by the first accelerator card as an example. The processing procedure of other accelerator cards 102 on their respective samples shard1, shard2, ... is similar.

[0126] S520, the first accelerator card uses a fusion operator to process the first sample. The fusion operator is used to query the feature vectors corresponding to each feature identifier from the embedding table stored in the target device based on the first sample.

[0127] In this embodiment, the entire computation graph of the model may include a fusion operator (OP). The fusion operator (OP) abstracts multiple operations on samples into a single function execution logic. The fusion operator (OP) can take a single sample (such as the first sample) as input, perform the corresponding function execution logic calculation on that sample, and then query the corresponding embedding vector from the target device storing the embedding table and output it. Thus, through the fusion operator (OP), the sparse computation on samples can be abstracted into a database-style query function. Only one input interface and one output interface of this query function need to be exposed to the accelerator card 102. Compared to the traditional technology where multiple operators are concatenated on both sides for deduplication and feature mapping, the fusion operator (OP) in this embodiment can execute independently on the accelerator card, reducing the time consumption caused by two-side communication and avoiding the accuracy loss caused by the input and output of multiple operators, thereby maintaining higher accuracy during data processing. Furthermore, since the model's computation graph only includes this one fusion operator (OP), frequent memory reads and writes or execution of control statements are not required during computation, thus improving data efficiency.

[0128] S530, the first accelerator card acquires the feature vector output by the fusion operator.

[0129] In this embodiment, the first accelerator card can directly obtain the feature vector corresponding to the first sample queried by the fusion operator from the target device, and can then train or infer the model based on these feature vectors. Using the feature vectors corresponding to the samples for model training and inference is a mature technology in the field and will not be elaborated further.

[0130] It is understood that the above steps S510 to S540 are illustrated using the example of execution on the first accelerator card configured in computing device A. Other accelerator cards 102 that cooperate with the first accelerator card to perform model training can also perform the above operations S510 to S540. The same applies to the following embodiments.

[0131] In this embodiment, the first accelerator card can be used in either an MP or PS architecture for offline, online, or near-line training of the recommendation model. Specifically, the first accelerator card can process the samples of the recommendation model through steps S510 to S540 to quickly obtain the feature vectors corresponding to the samples, and use these feature vectors to train and infer the recommendation model (or a part of the model) assigned to the first accelerator card. Since the first accelerator card processes the sample data through steps S510 to S540 in this embodiment, it can greatly shorten the time consumption. Therefore, when the training data volume of the recommendation model is large (e.g., 200G), it is beneficial to shorten the overall training time. For example, the training of the recommendation model in a browser application scenario can be completed within 0.5 hours, meeting the customer's need for daily model updates.

[0132] In some possible implementations, the first accelerator card can process the acquired sample shard0 in batches and orchestrate the corresponding task flows, allowing the task sequences between different flows to overlap, thereby saving processing time. Specifically, see [reference needed]. Figure 6 As shown, the method may specifically include:

[0133] S610, the first accelerator card acquires the first sample.

[0134] The execution principle of step S610 is similar to that of step S510 in the above embodiment. For details, please refer to the relevant description of S510, which will not be repeated here.

[0135] S620 divides the first sample into multiple ordered sample slices.

[0136] In this step, after the first accelerator card acquires sample shard0, it can divide shard0 into multiple ordered sample slices. For example, after accelerator card 102A acquires sample shard0, it divides the keys {0,1,1,3,5,6,5} in sample shard0 into three ordered sample slices: shard01, shard02, and shard03. Shard01 includes {0,1,1}, shard02 includes {3,5,6}, and shard03 includes {5}. It should be noted that "ordered" here means that the sample slices can have a corresponding processing order. For example, shard01 is the first batch of sample slices processed on accelerator card 102, shard02 is the second batch of sample slices processed on accelerator card 102, and shard03 is the third batch of sample slices processed on accelerator card 102. Furthermore, the embedded table identifier (table_id) in shard0 is always associated with each of the sample slices shard01, shard02, and shard03 of shard0.

[0137] S630 generates a task flow for each sample slice. Each task flow is used to implement the running logic of the fusion operator. Each task flow includes multiple tasks in sequence.

[0138] In this embodiment, the first accelerator card can generate task streams stream1, stream2, and stream3 for each sample slice shard01, shard02, and shard03, respectively. Each task stream of shard01, shard02, and shard03 is used to implement the function execution logic of the fusion operator OP. Each task stream includes multiple tasks executed serially. For example, when the accelerator card 102 in the MP architecture executes the fusion operator OP, each task stream may include serial partitioning tasks, feature identifier key number exchange (cet all2all) tasks, feature identifier exchange (keyall2all) tasks, lookuporinsert tasks, and feature vector exchange (emb all2all) tasks. When the accelerator card 102 in the PS architecture executes the fusion operator OP, each task stream may include synchronous serial deduplication (unique) tasks, partitioning tasks, and feature vector exchange (emb all2all) tasks. It is understood that these tasks can be implemented by the corresponding sub-operators of the fusion operator OP.

[0139] S640 distributes tasks in each task flow to multiple computing cores, so that the first computing core executes the first task in each task flow and the second computing core executes the second task in each task flow.

[0140] In this example, it can be combined Figure 4 The accelerator card structure shown can distribute tasks in various task flows to multiple computing cores 1, 2, ... in the first accelerator card. The multiple computing cores 1, 2, ... include the first computing core and the second computing core, and each computing core 1, 2, ... can be used to execute the same type of task in the task flow. For example, computing core 1 of accelerator card 102A can be bound to execute the first task in task flows stream1, stream2 and stream3, and computing core 2 of accelerator card 102A can be bound to execute the second task in task flows stream1, stream2 and stream3.

[0141] S650, while executing the first task in the first task flow, executes the second task in the second task flow in parallel.

[0142] In this embodiment, the first task flow and the second task flow are task flows for any two different sample slices, and the first task and the second task are both among multiple tasks in their respective task flows. In this way, while the first accelerator card is executing the first task in the first task flow, the second task in the second task flow can be executed in parallel, allowing the task timing of different task flows to overlap, thereby saving data processing time.

[0143] For example, the first accelerator card can (through the parallel processing unit SiMT or other software / hardware) perform parallel task scheduling between task streams and control the timing of serial tasks in a single task stream.

[0144] For example, you can refer to it again. Figure 3B In the multi-task execution process shown, after the AICPU completes the partition of stream1, it communicates with other accelerator cards via communication unit 1021 to execute the key all2all operation of stream1. During this period, since the AICPU has completed the partition of stream1, it can continue to execute the partition of stream2. Similarly, after the AIcore completes the lookupor insert operation of stream1, it hands it over to communication unit 1021 to communicate with other accelerator cards 102 to execute the emb all2all operation of stream1. During the execution of the emb all2all operation of stream1, since the AIcore has completed the lookupor insert operation of stream1, it can continue to execute the lookupor insert operation of stream2. Similarly, the parallel execution of stream2 and stream3 by the AICPU, AIcore, and other computing cores and communication unit 1021 is similar. This continues until all tasks in streams 1, 2, and 3 have been completed. By performing concurrent execution of sparse computation and all-to-all communication operations between different task flows using computation kernels 1, 2, ..., the processing time for sample data is effectively shortened, which in turn greatly reduces the time required for model training and inference.

[0145] In some possible implementations, the prototype of the fusion operator OP may specifically include an abstract description of the following fields:

[0146] .INPUT(table_id,TensorType({DT_INT32})), this field represents the table identifier embedded in the input parameter;

[0147] .INPUT(keys,TensorType({DT_INT54})), this field represents the feature identifier of the input parameter;

[0148] .OUTPUT(values,TensorType({DT_FP32})), this field represents the output values ​​(i.e., feature vectors);

[0149] The `.REQUIRED_ATTR(embedding_type, Int)` parameter, which is a required field, declares the storage strategy of the embedding table accessed by the fusion operator (OP), i.e., the model's embedding table. This storage strategy includes at least the model parallel storage strategy or the parameter server storage strategy mentioned above. Different embedding type values ​​in the fusion operator (OP) can result in different storage locations and operation methods when executed by the first accelerator card.

[0150] Specifically, in this implementation, such as Figure 7 As shown, the method may specifically include:

[0151] S710, the first accelerator card acquires the first sample.

[0152] In this example, the execution principle of step S710 is the same as that of S510 in the above embodiment. Please refer to the description of S510, which will not be repeated here.

[0153] S720, the first accelerator card uses the fusion operator to process the first sample, specifically including the following two cases: S721 and S722:

[0154] S721, the first accelerator card determines the target device from multiple accelerator cards that execute the storage strategy on the embedding table by executing the fusion operator, when the storage strategy is the model parallel storage strategy. The multiple accelerator cards include the first accelerator card and / or the second accelerator card, and the second accelerator card is an accelerator card that works with the first accelerator card to perform model training or inference.

[0155] S722, when the storage policy is a parameter server storage policy, the target device is determined from the servers that execute the storage policy on the embedded table, and the server is a server that works with the first accelerator card to train or infer the model.

[0156] In this example, the first accelerator card can obtain the parameter value (embedding_type) of the fusion operator OP, determine the storage strategy of the model's embedding table, and thereby determine the target device for storing the embedding table.

[0157] Specifically, in this example, if the storage strategy represented by the parameter value (embedding_type) is a model parallel storage strategy, that is, the model's embedding table is distributed and stored across multiple accelerator cards, then when the first accelerator card executes the fusion operator (OP) for data processing, it can determine the target device from all accelerator cards storing the embedding table slices. For example, accelerator cards 102A to 102D each store a portion of the slices, but the embedding vectors corresponding to the first sample are distributed across accelerator cards 102A to 102D respectively, then all accelerator cards 102A to 102D can be used as the target device. Each accelerator card 120 can communicate all-to-all based on its built-in network card (or DMA communication unit, etc.) to query the embedding vectors corresponding to these feature identifiers (keys) from the embedding table (or slice) stored in the target device through step S723.

[0158] For example, in this step, if the storage strategy represented by the parameter value (embedding_type) is a parameter server storage strategy, that is, the model's embedding table is distributed and stored on one or more CPU servers 100, then when the first accelerator card executes the fusion operator OP for data processing, it can determine the target device from all CPU servers 100 storing the embedding table slices. For example, if two CPU servers 100 each store a portion of the slices, but the embedding vectors corresponding to the first sample are all stored on one of the CPU servers 100, then that CPU server 100 is taken as the target device. Each accelerator card 102 can communicate with the CPU server 100 based on its own network card to query the embedding vectors corresponding to these feature identifiers (keys) from the embedding table stored by the target device in step S723.

[0159] For example, the first accelerator card may be hardware with an affinity for sparse computing (such as...). Figure 4 As shown in the accelerator card 102, when the first accelerator card is the target device, it can also support lookup or insert operations of the fusion operator OP.

[0160] S730, the first accelerator card acquires the feature vector output by the fusion operator.

[0161] In this example, the execution principle of step S730 is the same as that of S530 in the above embodiment. Please refer to the description of S530, which will not be repeated here.

[0162] In some possible embodiments, if the embedding table is divided into multiple slices slice0, slice1, ..., and the slices slice0, slice1, ... are distributed and stored on multiple accelerator cards 102 (including the first accelerator card), that is, the storage strategy of the embedding table is a model-parallel storage strategy, then the parameter value (embedding_type) of the fusion operator OP is configured with the model-parallel storage strategy. Thus, the accelerator cards 102 can process sample data through centralized communication all12all. Specifically, as... Figure 8 As shown, the method may include:

[0163] S810, the first accelerator card acquires the first sample.

[0164] In this example, the execution principle of step S810 is the same as that of S510 in the above embodiment. Please refer to the description of S510, which will not be repeated here.

[0165] The S820, the first accelerator card, uses the fusion operator to process the first sample.

[0166] In this step, the parameter value (embedding_type) of the fusion operator OP is configured as a model parallel storage strategy. Then, the fusion operator OP is executed by the first accelerator card. Based on this parameter value (embedding_type), the target device can be determined from the accelerator card 102 that stores each slice 0, slice 1, ... The target device can be the device that stores the target slice. The slice to which the feature vector corresponding to the feature identifier in the first sample belongs is the target slice. The target slice can be one or more slices among all slices 0, slice 1, ...

[0167] Next, by using the fusion operator OP, the feature vectors corresponding to each feature identifier in the first sample can be queried from the embedding table slices stored on the target device, based on the first sample.

[0168] Specifically, step S820 may include:

[0169] S821, the first accelerator card determines the target slice to which the feature vector corresponding to each feature identifier belongs by executing the slicing operator in the fusion operator, and identifies the accelerator card storing the target slice as the target device.

[0170] In this example, the first accelerator card can determine which slice (i.e., target slice) each key belongs to from the keys of the acquired sample by executing the partitioning operator in the fusion operator OP, thereby making the accelerator card storing these slices the target device.

[0171] As a concrete example, the embedded table can be pre-divided into multiple slices (slice0, slice1, ...) according to certain rules (such as range slicing, hash slicing, etc.). Each slice (slice0, slice1, ...) is stored on a different accelerator card (102). Each accelerator card (102) can maintain corresponding slicing information, which records which slice (slice0, slice1, ...) each key belongs to, and on which accelerator card (102) these slices (slice0, slice1, ...) are stored. In this way, the partition operator can determine the target slice and target device by querying the slicing information. Of course, other methods can also be used to establish the association between the key, slice, and accelerator card; these are not all listed in this example.

[0172] For example, S821 may specifically include:

[0173] S8211, the first accelerator card determines a first feature identifier and a second feature identifier from at least one feature identifier by executing a slicing operator.

[0174] In this step, the first accelerator card can classify the feature identifiers contained in the first sample by executing a slicing operator, grouping feature identifiers whose corresponding embedding vectors belong to the same slice into one category. That is, from the feature identifiers contained in the first sample, it determines the first feature identifier whose corresponding feature vector belongs to the first slice stored on the first accelerator card, and the second feature identifier whose corresponding feature vector belongs to the second slice stored on another accelerator card (i.e., the second accelerator card).

[0175] For example, with Figure 2B In the example shown, the keys in sample shard0 are {0,1,1,3,5,6,5}. The embedding vector corresponding to "0" belongs to slice0 (stored in accelerator card 102A), the embeddings corresponding to "1" and "5" belong to slice1 (stored in accelerator card 102B), the embedding corresponding to "6" belongs to slice2 (stored in accelerator card 102C), and the embedding corresponding to "3" belongs to slice3 (stored in accelerator card 102D).

[0176] S8212, Based on the first feature identifier and the second feature identifier, determine the first acceleration card and the second acceleration card as target devices.

[0177] In this step, based on the partitioning operator, it is determined which slice on the accelerator card 102 the embedding vector corresponding to each key in the first sample belongs to, and then the accelerator card 102 can be used as the target accelerator card for centralized communication.

[0178] S822, the first accelerator card swaps the second feature identifier in the first sample with the third feature identifier on the second accelerator card by executing the identifier swapping operator in the fusion operator.

[0179] In this step, if the embedding vectors corresponding to each key in the first sample belong to different target slices, and these target slices are stored in the first accelerator card and other accelerator cards (i.e., the second accelerator card), then the first accelerator card can perform key exchange between the accelerator cards 102 by executing the identifier exchange operator in the fusion operator, and exchange the keys whose corresponding embedding vectors belong to the same target slice to the corresponding accelerator card 102.

[0180] For example, a second accelerator card, which works in conjunction with the first accelerator card for model training or inference, is responsible for processing the second sample. Since the second sample belongs to the same sample set as the first sample, the second accelerator card can determine the third feature identifier from the second sample by executing the fusion operator (OP). The feature vector corresponding to the third feature identifier belongs to a slice stored on the first accelerator card. Thus, the first accelerator card sends the second feature identifier to the second accelerator card, and the second accelerator card sends the third feature identifier to the first accelerator card, completing the key exchange.

[0181] To ensure the accuracy and reliability of key exchange, before key exchange, the first accelerator card can first determine the number of second feature identifiers to be sent to the second accelerator card and the number of third feature identifiers to be received from the second accelerator card by executing the quantity exchange operator in the fusion operator.

[0182] Still with Figure 2B In the example shown, accelerator card 102A communicates centrally with other accelerator cards 102B, 102C, 102… Accelerator card 102A can determine through the quantity exchange operator that it has two keys, “1” and “5”, to send to accelerator card 102B, one key, “6”, and one key, “3”, to send to accelerator card 102D, and needs to obtain 0, 1, and 2 keys from 102B to 102D respectively.

[0183] Next, accelerator card 102A communicates centrally with other accelerator cards 102B, 102C, 102… Accelerator card 102A can perform key exchange with other accelerator cards 102B, 102C, 102… by executing the operation logic of the feature identifier exchange operator (key all2all) in the fusion operator OP, according to the quantity determined by the quantity exchange operator (cnt all2all). That is, accelerator card 102A sends two keys, “1” and “5”, to accelerator card 102B, sends one key, “6”, to accelerator card 102C, and receives the key “4” sent by 102C. The first accelerator card sends the key “3” to accelerator card 102D and receives two keys, “4” and “8”, sent by 102D. In this way, the embeddings corresponding to the keys obtained after the exchange on each accelerator card 102A to 102D are stored locally on the accelerator card.

[0184] S823, the first accelerator card maps each feature identifier to a preset index value by executing the query / insertion operator in the fusion operator, and queries the feature vector corresponding to each index value from the target slice stored in the target device.

[0185] In this step, the embedding table and the specific positions of each embedding vector within it can be described using corresponding indexes or data structures, such as the key2offset data structure. Thus, the first accelerator card can execute the lookuporinsert operator in the fusion operator (OP) to map each feature identifier to this index or data structure describing the specific position of the feature vector in the embedding table, obtaining the corresponding index values. Then, based on these index values, it can search for the embedding vector in the embedding table.

[0186] Still with Figure 2B In the example shown, the accelerator card 102A looks up the embeddings corresponding to keys such as {0, 4, 4, 8} in slice0 of local storage. It should be noted that if no corresponding embedding is found for any key, a default embedding vector can be inserted into the embedding table. This inserted embedding vector is the embedding vector corresponding to this key.

[0187] For example, before querying the corresponding embedding vector based on the key, the query / insert operator can also perform feature admission calculation on the key based on preset admission conditions to filter out keys that meet the admission conditions for query / insert operations, which will not be described in detail here.

[0188] S824, the first accelerator card sends the feature vector queried according to the third feature identifier to the second accelerator card by executing the vector exchange operator in the fusion operator, and obtains the feature vector queried by the second accelerator card according to the second feature identifier.

[0189] In this step, the first accelerator card can execute the vector exchange operator (emb all2all) logic in the fusion operator OP, thereby exchanging embedding vectors with other accelerator cards 102. Specifically, the first accelerator card can send the feature vector queried by the third feature identifier to the second accelerator card, and obtain the feature vector embedding queried by the second accelerator card from the second slice according to the second feature identifier, so that the first and second accelerator cards can train or infer their respective assigned parts of the model based on the feature vectors they obtained.

[0190] Still with Figure 2B In the example shown, accelerator card 102A looks up the embeddings corresponding to keys such as {0, 4, 4, 8} in slice0 of local storage. One of the keys, {4}, comes from accelerator card 102C, so the embedding vector corresponding to {4} is sent to accelerator card 120C in this step. The keys {4, 8} come from accelerator card 102D, so the embedding vector corresponding to {4, 8} is sent to accelerator card 120D in this step.

[0191] In this way, through vector swapping, each accelerator card 120 still obtains the embedding vector corresponding to its own sample, ensuring the correctness of the vector corresponding to the sample.

[0192] S830, the first accelerator card acquires the feature vector output by the fusion operator.

[0193] In this step, the feature vector obtained by the first accelerator card from the fusion operator is the embedding vector corresponding to the first sample, which is obtained after exchanging with other accelerator cards 102. These embedding vectors can be used for model training or inference.

[0194] In some possible implementations, the embedding table can be divided into multiple slices (slice0, slice1, ...) and stored on a CPU server 100 of a PS architecture. That is, the storage strategy for the embedding table is a parameter server storage strategy. Then, the parameter value (embedding_type) of the fusion operator OP is configured with a model-parallel storage strategy. Thus, each accelerator card 102 can communicate with the CPU server 100 to process sample data. Specifically, as... Figure 9As shown, in this implementation, since the first accelerator card is located in the parameter server architecture PS, the data processing method can be the same as described above. Figure 5 The example shown is roughly similar, except that during the process of the first accelerator card using the fusion operator OP to process the first sample, i.e., during the execution of step S920 in this implementation, it may specifically include S921 and S922:

[0195] S921, the first accelerator card determines the target slice to which the feature vector corresponding to each feature identifier belongs by executing the slicing operator of the fusion operator OP, and determines the server storing the target slice as the target device.

[0196] In this implementation, since the slices of the embedded table can be stored on one or more CPU servers 100, when the partition operator of the fusion operator OP is executed, the target device storing the target slice can be determined from all CPU servers 100 storing the slices of the embedded table based on the partition information maintained on the first accelerator card or similar information used to record the association between the key, slice, and CPU servers 100. The slice to which the feature vector corresponding to the key in the first sample belongs is the target slice.

[0197] S922, the first accelerator card maps each feature identifier to a preset index value by executing the query / insertion operator in the fusion operator OP, and queries the feature vector corresponding to each index value from the target slice stored in the target device.

[0198] In this step, since the embedded table slice is stored on the CPU server 100, to improve query efficiency, the lookupor insert operator can be deployed on the CPU server 100. The first accelerator card can only be aware of the input and output interfaces of the fusion operator. The first accelerator card inputs the first sample into the fusion operator. The partition operator of the fusion operator is executed on the first accelerator card. The partition operator passes the identifiers of the different slices it processes to the lookupor insert operator running on the CPU server 100 that stores these slices. The lookupor insert operator maps the key it receives to the preset index value and queries the embedding vector in the embedded table slice according to these index values.

[0199] In this way, the lookuporinsert operator can use the communication capabilities of the CPU server 100 to return the queried embedding vector to the first accelerator card. Then, the first accelerator card can perform forward and backward training operations on the model it is responsible for based on the obtained embedding corresponding to the first sample.

[0200] Optionally, in the above Figure 8 or Figure 9 In the illustrated embodiment, to reduce the processing of duplicate keys in the samples, a deduplication (unique) operation can be performed before the first accelerator card performs the partition operation on the first sample. Specifically, as shown... Figure 10 As shown, the data processing method further includes:

[0201] S1010, the first accelerator card performs a deduplication operation on at least one feature identifier in the first sample by executing the deduplication operator in the fusion operator.

[0202] In this step, the unique operator in the fusion operator OP can be executed through the first accelerator card to perform the deduplication operation on all feature identifiers in the first sample. For example, if the keys in sample shard0 are {0,1,1,3,5,6,5}, after deduplicating the two "1"s and two "5"s, we can get one "1" and one "5", that is, after deduplication, the keys in sample shard0 are {0,1,3,5,6}.

[0203] S1020, Generate a recovery matrix corresponding to at least one feature identifier, the recovery matrix being used to record at least one feature identifier.

[0204] In this step, while performing deduplication, a recovery matrix is ​​also generated to record the original keys in the first sample. For example, if the keys in sample shard0 are {0,1,1,3,5,6,5}, the keys obtained after deduplication are {0,1,3,5,6}. At the same time, a recovery matrix is ​​generated to record the correspondence between each original data item in {0,1,1,3,5,6,5} and each unique data item retained after deduplication. In other words, this recovery matrix can be used to recover the structure {0,1,1,3,5,6,5}.

[0205] S1030, the first accelerator card maps each deduplicated feature identifier to a preset index value by executing the query / insertion operator in the fusion operator, and queries the feature vector corresponding to the index value from the embedding table stored in the target device.

[0206] In this step, the first accelerator card maps the deduplicated feature identifiers to preset index values ​​by executing the lookuporinsert operator in the fusion operator, and then queries the feature vectors corresponding to these index values ​​from the embedding table stored in the target device.

[0207] S1040, Perform a recovery operation on the feature vector based on the recovery matrix to obtain each feature vector corresponding to at least one feature identifier.

[0208] In this step, after the deduplication operation, the first accelerator card performs the fusion operator operation based on the deduplicated keys. The resulting embedding vector is not exactly the same as the original keys in the first sample. To ensure the accuracy of training, it is necessary to recover the embedding vector that should have been obtained from the original keys in the first sample based on the recovery matrix. For example, if the keys in the first sample shard0 are {0,1,1,3,5,6,5}, after deduplication they become {0,1,3,5,6}. The first accelerator card, when performing the fusion operator operation, actually obtains the vector corresponding to {0,1,3,5,6}. Therefore, it is necessary to fill back the embedding vectors corresponding to "1" and "5" according to the recovery matrix to obtain the embedding vectors corresponding to {0,1,1,3,5,6,5}.

[0209] This application also provides a data processing apparatus, such as... Figure 11 As shown, the device 1000 can be applied to a computing device with a first accelerator card deployed, and the device 1000 may include:

[0210] The acquisition module 1001 is used to acquire a first sample, which includes an embedding table identifier and at least one feature identifier. The embedding table identifier represents the corresponding embedding table, and each feature identifier corresponds to a feature vector in the embedding table. The processing module 1002 is used to process the first sample using a fusion operator. The fusion operator is used to query the feature vectors corresponding to each feature identifier from the embedding table stored in the target device, which is the device storing the embedding table, based on the first sample. The acquisition module 1001 is also used to acquire the feature vectors output by the fusion operator.

[0211] It should be understood that the device 1000 is used to execute the methods in the above embodiments. The corresponding program modules 1001 and 1002 in the device 1000 have similar implementation principles and technical effects as described in the above methods. The working process of the device 1000 can be referred to the corresponding process in the above methods, and will not be repeated here.

[0212] This application also provides a computing device A. For example... Figure 12As shown, computing device A includes: a bus 1101, an accelerator card 102, a memory 1106, and a communication interface 1108. The accelerator card 102, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. Computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.

[0213] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1104 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1104 may include a path for transmitting information between various components of the computing device A (e.g., memory 106, processor 102, communication interface 1108).

[0214] Accelerator card 102 can be a processor for performing computational tasks, including but not limited to any one or more of the following: graphics processing unit (GPU), neural network accelerator chip (NPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC). For example, accelerator card 102 can be used to accelerate computation, that is, to improve computational speed and efficiency by utilizing its own hardware and / or software. For example, accelerator card 102 can be... Figure 4 The structures shown are examples of, but not limited to, those described.

[0215] The memory 1106 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD), but is not limited thereto.

[0216] The memory 1106 stores executable program code, and the accelerator card 102 executes the executable program code to implement the data processing method in the above embodiment.

[0217] The communication interface 1103 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between computing device A and other devices or communication networks.

[0218] This application also provides a computing device cluster. The computing device cluster includes at least one computing device A. The computing device A can be a server, such as a central server, an edge server, or a local server in a local data center. The memory 1106 of one or more computing devices A in the computing device cluster can store the same instructions for executing data processing methods.

[0219] In some possible implementations, the memory 1106 of one or more computing devices A in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices A can jointly execute the instructions for executing data processing methods.

[0220] In some possible implementations, one or more computing devices A in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. For example, two computing devices A can be connected to the network via their respective communication interfaces 1103.

[0221] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data processing method.

[0222] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data processing method.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized by, The method is applied to a computing device deployed with a first accelerator card, and the method comprises: The first accelerator card obtains a first sample, the first sample comprising an embedding table identifier and at least one feature identifier, the embedding table identifier being used to represent a corresponding embedding table, and each feature identifier corresponding to a feature vector in the embedding table; The first accelerator card processes the first sample using a fusion operator, the fusion operator being used to query, according to the first sample, a feature vector corresponding to each feature identifier from the embedding table stored in a target device, the target device being a device storing the embedding table; The first accelerator card obtains the feature vector output by the fusion operator.

2. The method of claim 1, wherein the first sample comprises a plurality of feature identifiers. After the first accelerator card obtains the first sample, the method comprises: The first sample is split into a plurality of sample slices, each sample slice comprising the embedding table identifier and at least one of the plurality of feature identifiers; A task flow for each sample slice is generated, each task flow being used to implement the running logic of the fusion operator, and each task flow comprising a plurality of tasks in series; During execution of a first task in a first task flow, a second task in a second task flow is executed in parallel, the first task flow and the second task flow being task flows of any two different sample slices, and the first task and the second task each being one of the plurality of tasks in the respective task flow.

3. The method of claim 2, wherein, The first accelerator card comprises a parallel processing unit SiMT and a plurality of computing cores, and the first task and the second task are each included in each task flow, After the generation of the task flow for each sample slice, the method specifically comprises: Tasks in each task flow are distributed to the plurality of computing cores, a first computing core is caused to execute the first task in each task flow, and a second computing core is caused to execute the second task in each task flow, the plurality of computing cores comprising the first computing core and the second computing core; Task scheduling is performed by the parallel processing unit SiMT to cause the second computing core to execute the second task in the second task flow during execution of the first task in the first task flow by the first computing core.

4. The method of any one of claims 1 to 3, wherein the fusion operator comprises a parameter value, the parameter value being used to represent a storage strategy of the embedding table, the storage strategy comprising at least a model parallel storage strategy or a parameter server storage strategy; The first accelerator card processes the first sample using a fusion operator, comprising: The first accelerator card determines, by executing the fusion operator, the target device from a plurality of accelerator cards performing the storage strategy on the embedding table in a case where the storage strategy is a model parallel storage strategy, the plurality of accelerator cards comprising the first accelerator card and / or a second accelerator card, the second accelerator card being an accelerator card that cooperates with the first accelerator card to perform model training or inference. In a case that the storage strategy is a parameter server storage strategy, the target device is determined from a server that performs the storage strategy on the embedding table, the server being a server that cooperates with the first acceleration card to perform model training or inference.

5. The method of any one of claims 1-4, wherein the embedding table is split into a plurality of slices, and the plurality of slices are stored in a plurality of acceleration cards in a distributed manner. The at least one feature identifier identifies a feature vector that belongs to a target slice, and the plurality of slices include the target slice. The first acceleration card processes the first sample using a fusion operator, including: The first acceleration card determines, by executing a slicing operator in the fusion operator, a target slice to which each feature vector corresponding to each feature identifier belongs, and determines the acceleration card that stores the target slice as the target device. The first acceleration card maps each feature identifier to a preset index value by executing a query / insert operator in the fusion operator, and queries a feature vector corresponding to each index value from the target slice stored in the target device.

6. The method of claim 5, wherein the first acceleration card determines, by executing a slicing operator in the fusion operator, a target slice to which each feature vector corresponding to each feature identifier belongs, and determines the acceleration card that stores the target slice as the target device, including: The first acceleration card determines, by executing the slicing operator, a first feature identifier and a second feature identifier from the at least one feature identifier, the feature vector corresponding to the first feature identifier belonging to a first slice stored on the first acceleration card, the feature vector corresponding to the second feature identifier belonging to a second slice stored on the second acceleration card, and the target slice including the first slice and the second slice. The first acceleration card and the second acceleration card are determined as the target device according to the first feature identifier and the second feature identifier.

7. The method of any one of claims 1-6, wherein the first sample belongs to a sample set, and the sample set further includes a second sample processed by a second acceleration card. The embedding table is split into a plurality of slices, and the plurality of slices are stored in a plurality of acceleration cards in a distributed manner, the plurality of acceleration cards including the first acceleration card and the second acceleration card. The first acceleration card processes the first sample using a fusion operator, including: The first acceleration card exchanges, by executing an identifier exchange operator in the fusion operator, a second feature identifier in the first sample with a third feature identifier on the second acceleration card, the feature vector corresponding to the second feature identifier belonging to a slice stored on the second acceleration card, and the third feature identifier being determined from the second sample and the feature vector corresponding to the third feature identifier belonging to a slice stored on the first acceleration card. The first acceleration card maps the third feature identifier to a preset index value by executing a query / insert operator in the fusion operator, and queries a feature vector corresponding to the index value from the slice stored in the first acceleration card.

8. The method of claim 6, wherein after the first accelerator card maps the third feature identifier to a preset index value by executing a query / insert operator in the fusion operators and queries a feature vector corresponding to the index value from the slice stored in the first accelerator card, the method comprises: the first accelerator card sends the feature vector queried according to the third feature identifier to the second accelerator card and acquires the feature vector queried according to the second feature identifier from the second accelerator card by executing a vector exchange operator in the fusion operators.

9. The method of claim 6, wherein before the first accelerator card exchanges the second feature identifier in the first sample with the third feature identifier on the second accelerator card by executing an identifier exchange operator in the fusion operators, the method comprises: the first accelerator card determines the number of the second feature identifiers to be sent to the second accelerator card and the number of the third feature identifiers to be received from the second accelerator card by executing a number exchange operator in the fusion operators.

10. The method of any one of claims 1 to 3, wherein the embedding table is split into a plurality of slices, and the plurality of slices are stored in servers in a distributed manner, the servers are servers for model training or inference in cooperation with the first accelerator card, the feature vector represented by the at least one feature identifier belongs to a target slice, and the plurality of slices include the target slice, the first accelerator card processes the first sample using the fusion operators, comprising: the first accelerator card determines the target slice to which the feature vector corresponding to each of the feature identifiers belongs by executing a slicing operator in the fusion operators, and determines the server storing the target slice as the target device; the first accelerator card maps each of the feature identifiers to a preset index value and queries a feature vector corresponding to each of the index values from the target slice stored in the target device by executing a query / insert operator in the fusion operators.

11. The method according to any one of claims 1 to 10, characterized in that, the first accelerator card processes the first sample using the fusion operators, further comprising: the first accelerator card performs a deduplication operation on the at least one feature identifier in the first sample by executing a deduplication operator in the fusion operators; a recovery matrix corresponding to the at least one feature identifier is generated, the recovery matrix is used to record the at least one feature identifier; the first accelerator card maps each of the deduplicated feature identifiers to a preset index value and queries a feature vector corresponding to the index value from the embedding table stored in the target device by executing a query / insert operator in the fusion operators; each feature vector corresponding to the at least one feature identifier is obtained by performing a recovery operation on the feature vector according to the recovery matrix.

12. A data processing apparatus, characterized by The device is applied to a computing device deployed with a first accelerator card, and the device comprises: An obtaining module is configured to obtain a first sample, the first sample comprising an embedding table identifier and at least one feature identifier, the embedding table identifier being used to represent a corresponding embedding table, and each feature identifier corresponding to a feature vector in the embedding table; A processing module is configured to process the first sample by using a fusion operator, the fusion operator being used to query, according to the first sample, a feature vector corresponding to each feature identifier from the embedding table stored in a target device, the target device being a device storing the embedding table; The obtaining module is further configured to obtain the feature vector output by the fusion operator.

13. The apparatus of claim 12, wherein, The first sample comprises a plurality of feature identifiers, The processing module is further configured to: divide the first sample into a plurality of sample slices, each sample slice comprising the embedding table identifier and at least one of the plurality of feature identifiers; generate a task flow for each sample slice, each task flow being used to implement a running logic of the fusion operator, and each task flow comprising a plurality of tasks in series; during execution of a first task in a first task flow, a second task in a second task flow is executed in parallel, the first task flow and the second task flow being task flows of any two different sample slices, and the first task and the second task each being one of the plurality of tasks in the respective task flow.

14. The apparatus of claim 13, wherein, The first accelerator card comprises a parallel processing unit SiMT and a plurality of computing cores, and each task flow comprises the first task and the second task, The processing module is specifically configured to: distribute tasks in each task flow to the plurality of computing cores, so that a first computing core executes the first task in each task flow, and a second computing core executes the second task in each task flow, the plurality of computing cores comprising the first computing core and the second computing core; perform task scheduling by using the parallel processing unit SiMT, so that the second computing core executes the second task in the second task flow during execution of the first task in the first task flow by the first computing core.

15. The apparatus according to any one of claims 12 to 14, characterized in that the fusion operator comprises a parameter value, the parameter value being used to represent a storage strategy of the embedding table, the storage strategy comprising at least a model parallel storage strategy or a parameter server storage strategy; The processing apparatus is specifically configured to: determine, by executing the fusion operator, the target device from a plurality of accelerator cards performing the storage strategy on the embedding table in a case where the storage strategy is the model parallel storage strategy, the plurality of accelerator cards comprising the first accelerator card and / or a second accelerator card, the second accelerator card being an accelerator card performing model training or inference in cooperation with the first accelerator card; determine, in a case where the storage strategy is the parameter server storage strategy, the target device from a server performing the storage strategy on the embedding table, the server being a server performing model training or inference in cooperation with the first accelerator card.

16. The apparatus of any one of claims 12-15, wherein the embedding table is split into a plurality of slices, and the plurality of slices are stored in a plurality of acceleration cards in a distributed manner, the at least one feature identifier indicates that a feature vector representing the feature belongs to a target slice, and the plurality of slices include the target slice, the processing apparatus is specifically configured to: determine, by executing a slicing operator in the fusion operator, a target slice to which each of the feature vectors corresponding to the feature identifiers belongs, and determine the acceleration card storing the target slice as the target device, map each of the feature identifiers to a preset index value by executing a query / insert operator in the fusion operator, and query a feature vector corresponding to each of the index values from the target slice stored in the target device.

17. The method of claim 16, wherein the processing apparatus is specifically configured to: determine, by executing the slicing operator, a first feature identifier and a second feature identifier from the at least one feature identifier, the feature vector corresponding to the first feature identifier belongs to a first slice stored in the first acceleration card, the feature vector corresponding to the second feature identifier belongs to a second slice stored in the second acceleration card, and the target slice includes the first slice and the second slice, determine the first acceleration card and the second acceleration card as the target device according to the first feature identifier and the second feature identifier.

18. The apparatus of any one of claims 12-17, wherein the first sample belongs to a sample set, and the sample set further includes a second sample processed by a second acceleration card, the embedding table is split into a plurality of slices, and the plurality of slices are stored in a plurality of acceleration cards in a distributed manner, the plurality of acceleration cards include the first acceleration card and the second acceleration card, the processing apparatus is specifically configured to: exchange, by executing an identifier exchange operator in the fusion operator, a second feature identifier in the first sample with a third feature identifier on the second acceleration card, the feature vector corresponding to the second feature identifier belongs to a slice stored in the second acceleration card, and the third feature identifier is determined from the second sample and the feature vector corresponding to the third feature identifier belongs to a slice stored in the first acceleration card, map the third feature identifier to a preset index value by executing a query / insert operator in the fusion operator, and query a feature vector corresponding to the index value from the slice stored in the first acceleration card.

19. The apparatus of claim 17, wherein the processing apparatus is specifically further configured to: send, by executing a vector exchange operator in the fusion operator, the feature vector queried according to the third feature identifier to the second acceleration card, and obtain a feature vector queried by the second acceleration card according to the second feature identifier.

20. The apparatus of claim 17, wherein the processing apparatus is specifically further configured to: By executing a number exchange operator in the fusion operator, a number of the second feature identifiers to be sent to the second accelerator card and a number of the third feature identifiers to be received from the second accelerator card are determined.

21. The apparatus of any one of claims 12-14, wherein the embedding table is split into a plurality of slices, and the plurality of slices are stored in servers that are used for model training or inference in cooperation with the first accelerator card. The feature vector represented by the at least one feature identifier belongs to a target slice, and the plurality of slices include the target slice. The processing apparatus is specifically configured to: By executing a slicing operator in the fusion operator, a target slice to which a feature vector corresponding to each feature identifier belongs is determined, and a server storing the target slice is determined as the target device; By executing a query / insert operator in the fusion operator, each feature identifier is mapped to a preset index value, and a feature vector corresponding to each index value is queried from the target slice stored in the target device.

22. The apparatus of any one of claims 12 to 21, wherein, The processing apparatus is specifically further configured to: By executing a deduplication operator in the fusion operator, a deduplication operation is performed on the at least one feature identifier in the first sample; A recovery matrix corresponding to the at least one feature identifier is generated, and the recovery matrix is used to record the at least one feature identifier; By executing a query / insert operator in the fusion operator, each feature identifier after deduplication is mapped to a preset index value, and a feature vector corresponding to the index value is queried from the embedding table stored in the target device; According to the recovery matrix, a recovery operation is performed on the feature vector to obtain each feature vector corresponding to the at least one feature identifier.

23. A computing device, comprising: The apparatus includes a memory and at least one accelerator card. The memory is configured to store a computer program. The accelerator card is configured to execute the computer program stored in the memory to implement the method of any one of claims 1-11.

24. A cluster of devices, characterized by The apparatus includes a memory and at least one accelerator card. The memory is configured to store a computer program. The accelerator card is configured to execute the computer program stored in the memory to implement the method of any one of claims 1-11.

25. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program runs on a computing device, the computing device executes the method of any one of claims 1-11.

26. A computer program product, characterised in that, When the computer program product runs on a processor, the processor executes the method of any one of claims 1-11.

Images