Data processing method, system, device, program product and storage medium

By processing sparse and dense features of sample data separately in the deep learning training architecture, the bandwidth consumption of training samples on the GPU is reduced, the resource utilization of the GPU is improved, and the training efficiency is increased.

CN113971428BActive Publication Date: 2024-11-22BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010725167.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-24
Publication Date
2024-11-22
Estimated Expiration
2040-07-24

AI Technical Summary

Technical Problem

In existing deep learning training architectures, training samples consume too much GPU input network bandwidth, resulting in low GPU resource utilization, especially insufficient resources occupied by embedding vectors, leading to resource waste.

Method used

The sparse features of the sample data are sent to the second node, which determines the embedding vector corresponding to the sparse features. The dense features and classification labels are sent directly to the third node, which performs model training. This reduces the bandwidth usage of the training samples on the third node's input network and improves the bandwidth utilization of the embedding vector.

Benefits of technology

By optimizing the data processing method, the input network bandwidth consumption of training samples on the GPU was reduced, the resource utilization of the GPU was improved, and the training efficiency was increased, with the GPU utilization rate increasing from 40% to over 65%.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113971428B_ABST
    Figure CN113971428B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a data processing method, system, device, program product and storage medium, and relates to the field of data processing technology. In the present disclosure, a first node can send sparse features of sample data to a second node, and the second node can determine the embedding vector corresponding to the sparse features of the sample data according to the sparse features of the sample data and the corresponding relationship between the stored sparse features and the embedding vector and send it to a third node, thereby greatly reducing the input network bandwidth of the third node consumed by the training sample when training the model to be trained, increasing the input network bandwidth occupied by the embedding vector, enabling the third node to make full use of resources, and training the stored model to be trained according to the dense features and classification labels of the sample data and the embedding vector corresponding to the sparse features of the sample data, thereby improving the resource utilization of the third node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, system, device, program product and storage medium. Background Art

[0002] With the development of deep learning technology, deep learning models have been widely used in various click-through rate (CTR) scenarios. For example, recommendations for advertisements, news, videos, music, and other information can be implemented based on deep learning models.

[0003] The existing deep learning training architecture generally includes worker nodes and parameter server (ParameterServer) nodes. For example, the worker node can be deployed on the graphics processing unit (GPU), and the parameter server node can be deployed on the central processing unit (CPU). The model to be trained can be set on the worker node, and the corresponding relationship between sparse features and Embedding vectors can be stored on the parameter server node. When performing deep learning training, the worker node can read the training sample, and the training sample includes sample features and classification labels, and the sample features include dense features and sparse features. After reading the training sample, the worker node can access the parameter server node to complete the Embedding Lookup according to the sparse features, and obtain the Embedding vector corresponding to the sparse feature. Then, the worker node can perform forward propagation and back propagation calculations on the model to be trained according to the dense features, classification labels, and Embedding vectors, and update the parameters of the model to be trained according to the calculation results. By continuously iterating the above process, the deep learning training of the model to be trained can be completed.

[0004] In the above deep learning training architecture, the input network bandwidth of the GPU deployed in the worker node is mainly composed of training samples and Embedding vectors. When a large number of training samples are input to the GPU, a large amount of input network bandwidth in the GPU will be consumed, resulting in fewer Embedding vectors input to the GPU. As a result, the resources actually used by the GPU for calculation based on the Embedding vectors are only a small part of the available resources of the GPU, which seriously reduces the utilization of the GPU. Summary of the invention

[0005] The present disclosure provides a data processing method, system, device, program product and storage medium, which can reduce the input network bandwidth of a GPU consumed by training samples, make full use of GPU resources, and improve the utilization rate of the GPU.

[0006] The technical solution of the embodiment of the present disclosure is as follows:

[0007] In a first aspect, an embodiment of the present disclosure provides a data processing method, which is applied to a data processing system. The data processing system includes: a first node, a second node, and a third node. The method includes: the first node obtains sample data for training a model to be trained, the sample data includes sample features and classification labels, and the sample features include sparse features and dense features. The first node sends the sparse features of the sample data to the second node, and sends the dense features and classification labels of the sample data to the third node. The second node determines the embedding vector corresponding to the sparse features of the sample data based on the sparse features of the sample data and the correspondence between the stored sparse features and the embedding vector, and sends the embedding vector corresponding to the sparse features of the sample data to the third node. The third node trains the stored model to be trained based on the dense features and classification labels of the sample data and the embedding vector corresponding to the sparse features of the sample data.

[0008] Optionally, before the first node sends the sparse features of the sample data to the second node and sends the dense features and classification labels of the sample data to the third node, the method also includes: the first node divides the sample data into multiple sample batches according to the classification labels of the sample data, each of the multiple sample batches including one or more sample data with the same classification label.

[0009] The first node sends sparse features of sample data to the second node, and sends dense features and classification labels of sample data to the third node, including: the first node sends sparse features of each sample batch in multiple sample batches to the second node, and sends dense features and classification labels of each sample batch in multiple sample batches to the third node.

[0010] Optionally, the second node determines the embedding vector corresponding to the sparse feature of the sample data based on the sparse features of the sample data and the correspondence between the stored sparse features and the embedding vector, and sends the embedding vector corresponding to the sparse feature of the sample data to the third node, including: the second node determines the embedding vector corresponding to the sparse feature of the first sample batch based on the sparse features of the first sample batch and the correspondence between the stored sparse features and the embedding vector, and sends the embedding vector corresponding to the sparse feature of the first sample batch to the third node; wherein the first sample batch is any one of the multiple sample batches.

[0011] The third node trains the stored model to be trained according to the dense features and classification labels of the sample data, and the embedding vectors corresponding to the sparse features of the sample data, including: the third node trains the stored model to be trained according to the dense features and classification labels of the first sample batch, and the embedding vectors corresponding to the sparse features of the first sample batch.

[0012] Optionally, before the second node determines the embedding vector corresponding to the sparse features of the first sample batch according to the sparse features of the first sample batch and the correspondence between the stored sparse features and the embedding vector, the method further includes: the third node obtains identification information of the first sample batch. The third node sends the identification information of the first sample batch to the second node. The second node determines the sparse features of the first sample batch from the sparse features of multiple sample batches from the first node according to the identification information of the first sample batch.

[0013] Optionally, the third node trains the stored model to be trained according to the dense features and classification labels of the sample data and the embedding vectors corresponding to the sparse features of the sample data, including: the third node determines the predicted classification label corresponding to the sample data through the model to be trained according to the dense features of the sample data and the embedding vectors corresponding to the sparse features of the sample data. The third node performs back propagation calculation on the model to be trained according to the error between the classification label of the sample data and the predicted classification label to obtain the gradient of the parameters of the model to be trained. The third node updates the parameters of the model to be trained according to the gradient.

[0014] Optionally, the method further includes: the third node sends a gradient to the second node. The second node updates the corresponding relationship between the stored sparse features and the embedded vector according to the gradient.

[0015] In a second aspect, an embodiment of the present disclosure further provides a data processing system, which includes a first node, a second node, and a third node.

[0016] The first node is used to obtain sample data for training the model to be trained, the sample data including sample features and classification labels, and the sample features including sparse features and dense features; send the sparse features of the sample data to the second node, and send the dense features and classification labels of the sample data to the third node;

[0017] The second node is used to determine the embedding vector corresponding to the sparse feature of the sample data according to the sparse feature of the sample data and the corresponding relationship between the stored sparse feature and the embedding vector, and send the embedding vector corresponding to the sparse feature of the sample data to the third node;

[0018] The third node is used to train the stored model to be trained according to the dense features and classification labels of the sample data and the embedding vectors corresponding to the sparse features of the sample data.

[0019] Optionally, the first node is specifically used to divide the sample data into multiple sample batches according to the classification labels of the sample data, each of the multiple sample batches including one or more sample data with the same classification labels; send sparse features of each of the multiple sample batches to the second node, and send dense features and classification labels of each of the multiple sample batches to the third node.

[0020] Optionally, the second node is specifically used to determine the embedding vector corresponding to the sparse features of the first sample batch according to the sparse features of the first sample batch and the correspondence between the stored sparse features and the embedding vector, and send the embedding vector corresponding to the sparse features of the first sample batch to the third node; wherein the first sample batch is any one of the multiple sample batches. The third node is specifically used to train the stored model to be trained according to the dense features and classification labels of the first sample batch and the embedding vector corresponding to the sparse features of the first sample batch.

[0021] Optionally, the third node is further used to obtain identification information of the first sample batch and send the identification information of the first sample batch to the second node. The second node is further used to determine the sparse features of the first sample batch from the sparse features of multiple sample batches from the first node according to the identification information of the first sample batch.

[0022] Optionally, the third node is specifically used to determine the predicted classification label corresponding to the sample data through the model to be trained based on the dense features of the sample data and the embedding vectors corresponding to the sparse features of the sample data; perform back propagation calculation on the model to be trained based on the error between the classification label of the sample data and the predicted classification label to obtain the gradient of the parameters of the model to be trained; and update the parameters of the model to be trained based on the gradient.

[0023] Optionally, the third node is further used to send a gradient to the second node. The second node is further used to update the corresponding relationship between the stored sparse features and the embedded vector according to the gradient.

[0024] In a third aspect, an embodiment of the present disclosure provides a data processing device, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method described in the first aspect.

[0025] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the method described in the first aspect is implemented.

[0026] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which, when executed, implements the method described in the first aspect.

[0027] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0028] In the disclosed embodiment, the first node can send the sparse features of the sample data to the second node. The second node can determine the embedding vector corresponding to the sparse features of the sample data based on the sparse features of the sample data and the correspondence between the stored sparse features and the embedding vector and send it to the third node. Therefore, when the model to be trained is trained, the input network bandwidth of the third node only includes the dense features and classification labels of the sample data from the first node and the embedding vector from the second node. The sparse features of the sample data no longer occupy the input network bandwidth of the third node. Therefore, the input network bandwidth of the third node consumed by the training samples can be greatly reduced, and the input network bandwidth occupied by the embedding vector can be increased. The third node can make full use of resources to train the stored model to be trained based on the dense features and classification labels of the sample data and the embedding vector corresponding to the sparse features of the sample data, thereby improving the resource utilization of the third node.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0031] Figure 1 A schematic diagram of the composition of a deep learning training architecture is shown;

[0032] Figure 2 A schematic diagram showing the composition of a data processing system provided by an embodiment of the present disclosure is shown;

[0033] Figure 3 A schematic diagram showing a flow chart of a data processing method provided by an embodiment of the present disclosure is shown;

[0034] Figure 4 Another schematic diagram of the data processing method provided by the embodiment of the present disclosure is shown;

[0035] Figure 5 Another schematic diagram of a data processing method provided by an embodiment of the present disclosure is shown;

[0036] Figure 6 Another schematic diagram of a data processing method provided by an embodiment of the present disclosure is shown;

[0037] Figure 7 Another flowchart of the data processing method provided by the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0040] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.

[0041] Deep learning technology has been widely used in various click-through rate (CTR) scenarios. For example, a deep learning model can be trained to predict the advertisements, news, videos, music, etc. that users like. Then, the aforementioned advertisements, news, videos, music, etc. information can be provided to users based on the prediction results of the deep learning model.

[0042] Figure 1 A schematic diagram of the composition of a deep learning training architecture is shown. Figure 1 As shown, the current common deep learning training architecture generally includes a worker node 110 and a parameter server node 120. Among them, the worker node 110 is deployed on the GPU, and the parameter server node 120 is deployed on the CPU. The worker node 110 can store the model to be trained (such as a neural network), and the parameter server node 120 can store the correspondence between sparse features and embedding vectors.

[0043] When performing deep learning training, the worker node 110 can read the training samples, which include sample features and classification labels, and the sample features include dense features and sparse features. After reading the training samples, the worker node 110 can access the Parameter Server node 120 to complete the embedding vector query (EmbeddingLookup) based on the sparse features, and obtain the Embedding vector corresponding to the sparse features. Then, the worker node 110 can perform forward propagation and back propagation calculations on the model to be trained based on the dense features, classification labels, and Embedding vectors, and update the parameters of the model to be trained based on the calculation results. By continuously iterating the above process, the deep learning training of the model to be trained can be completed.

[0044] It can be seen that in the above Figure 1 In the deep learning training architecture shown, the input network bandwidth of the GPU as the worker node 110 mainly consists of two parts: training samples and Embedding vectors. Specifically, the training samples account for about 80% of the input network bandwidth of the GPU, and the Embedding vectors account for about 20% of the input network bandwidth of the GPU.

[0045] When deep learning training is performed on a training model, a large number of training samples are often required to ensure the training effect. However, when a large number of training samples are input to the GPU, a large amount of input network bandwidth in the GPU will be consumed, resulting in fewer embedding vectors input to the GPU. Fewer embedding vectors input to the GPU will cause the GPU to perform forward propagation and back propagation calculations on the training model based on dense features, classification labels, and embedding vectors, as well as update the parameters of the training model. The resources occupied are only a small part of the GPU's available resources, which seriously reduces the utilization of the GPU. The cost of GPUs is relatively high. If GPU resources cannot be fully utilized, a large amount of GPU resources will be wasted.

[0046] Based on this, the embodiments of the present disclosure provide a data processing method, which can reduce the input network bandwidth in the GPU consumed by training samples, make full use of GPU resources, and improve the utilization rate of the GPU.

[0047] The technical solution of the embodiment of the present disclosure is exemplarily described below with reference to the accompanying drawings:

[0048] First, the application scenarios of the embodiments of the present disclosure are introduced.

[0049] Figure 2 A schematic diagram of the composition of a data processing system provided by an embodiment of the present disclosure is shown.

[0050] like Figure 2 As shown, an embodiment of the present disclosure provides a data processing system. The data processing system may include: a first node 210, a second node 220, and a third node 230. The first node 210, the second node 220, and the third node 230 may be communicatively connected to each other.

[0051] The second node 220 stores the correspondence between the sparse features and the embedding vector, and the third node 230 is provided with the model to be trained. The first node 210 can obtain sample data of the model to be trained, the sample data includes sample features and classification labels, the sample features include sparse features and dense features, and send the sparse features of the sample data to the second node 220, and send the dense features and classification labels of the sample data to the third node 230.

[0052] The second node 220 can determine the Embedding vector corresponding to the sparse feature of the sample data according to the sparse feature of the sample data and the correspondence between the stored sparse feature and the embedding vector, and send the Embedding vector corresponding to the sparse feature of the sample data to the third node.

[0053] The third node 230 can train the stored model to be trained according to the dense features and classification labels of the sample data and the Embedding vectors corresponding to the sparse features of the sample data.

[0054] Please continue to refer to Figure 2 As shown, in some embodiments, the first node 210 may be a CPU cluster 1, the second node 220 may be a CPU cluster 2, and the third node 230 may be a GPU cluster.

[0055] Alternatively, in some other embodiments, the CPU cluster 1 and the CPU cluster 2 may also be on different CPUs in the same CPU cluster. For example, the CPU cluster 1 and the CPU cluster 2 may be different nodes in the same CPU cluster.

[0056] The data processing method provided in the embodiment of the present disclosure can be applied to the above Figure 2 The data processing system shown below Figure 2 and Figure 3 , the data processing method provided by the embodiment of the present disclosure is described.

[0057] Figure 3 A flow chart of a data processing method provided by an embodiment of the present disclosure is shown.

[0058] like Figure 3 As shown, the data processing method may include:

[0059] S301. The first node obtains sample data for training a model to be trained. The sample data includes sample features and classification labels. The sample features include sparse features and dense features.

[0060] For sample data, the sample features of each sample data are composed of dense features and sparse features. Dense features can refer to the more complex data features in the sample data, such as: when the sample data is an advertisement, the dense features can be the specific content of the advertisement. Sparse features can refer to some simple features in the sample data that can be sparsely expressed, such as: the advertisement logo, advertisement number, advertisement name, etc.

[0061] The classification labels can be obtained by manually annotating the sample data. For example, when the sample data is an advertisement and the model to be trained is an advertisement classification model, the classification labels of the sample data can be different advertisement type labels such as video labels, music labels, and game labels.

[0062] Optionally, for sample data in which some dense features and sparse features are fused together, the first node may split the sample features of the sample data to obtain sparse features and dense features of the sample data.

[0063] After acquiring the sample data for training the model to be trained, the first node may execute the following S302 and S303.

[0064] S302: The first node sends sparse features of sample data to the second node.

[0065] Optionally, the first node may actively send the sparse features of all sample data to the second node.

[0066] Accordingly, the second node may receive the sparse features of the sample data from the first node.

[0067] S303: The first node sends dense features and classification labels of sample data to the third node.

[0068] In one implementation, the first node may queue up the dense features and classification labels of all sample data locally (i.e., in the first node). When the third node starts to train the model to be trained, it may first send a training request to the first node. After receiving the training request, the first node may send the dense features and classification labels of the sample data to the third node. That is, in this implementation, it can be understood that the third node reads the dense features and classification labels of the sample data from the first node.

[0069] In another implementation, the first node may also actively send the dense features and classification labels of all sample data to the third node, which is not limited here.

[0070] Accordingly, the third node may receive dense features and classification labels of the sample data from the first node.

[0071] It should be noted that Figure 3 The execution order of S302 and S303 shown in the figure is only for illustrative purposes, and the embodiment of the present disclosure does not limit the order of S302 and S303. For example, in other implementations, S303 may be executed first, and then S302; or S302 and S303 may be executed simultaneously.

[0072] S304: The second node determines the embedding vector corresponding to the sparse feature of the sample data according to the sparse feature of the sample data and the corresponding relationship between the stored sparse feature and the embedding vector.

[0073] For example, after the second node receives the sparse features of the sample data from the first node, it can query the correspondence between the stored sparse features and the embedding vector to determine the embedding vector corresponding to the sparse features of the sample data. Then, the second node can send the queried embedding vector to the third node. For example, S305 can be executed.

[0074] S305. The second node sends the embedding vector corresponding to the sparse feature of the sample data to the third node.

[0075] Accordingly, the third node may receive the embedding vector corresponding to the sparse feature of the sample data from the second node.

[0076] S306. The third node trains the stored model to be trained according to the dense features and classification labels of the sample data and the embedding vectors corresponding to the sparse features of the sample data.

[0077] After receiving the dense features and classification labels of the sample data from the first node and the embedding vectors corresponding to the sparse features of the sample data from the second node, the third node can train the stored model to be trained based on the dense features and classification labels of the sample data and the embedding vectors corresponding to the sparse features of the sample data.

[0078] In the data processing method, the first node can send the sparse features of the sample data to the second node, and the second node can determine the embedding vector corresponding to the sparse features of the sample data based on the sparse features of the sample data and the corresponding relationship between the stored sparse features and the embedding vector, and send it to the third node. Therefore, when the model to be trained is trained, the input network bandwidth of the third node only includes the dense features and classification labels of the sample data from the first node, and the embedding vector from the second node, and the sparse features of the sample data no longer occupy the input network bandwidth of the third node. Therefore, the input network bandwidth of the third node consumed by the training samples can be greatly reduced, and the input network bandwidth occupied by the embedding vector can be increased. The third node can make full use of resources, and train the stored model to be trained based on the dense features and classification labels of the sample data and the embedding vector corresponding to the sparse features of the sample data, thereby improving the resource utilization of the third node.

[0079] The following example takes the first node as a feature extraction node, the second node as a parameter server node, and the third node as a worker node. Assume that the worker node is a GPU (cluster), and the feature extraction node and the parameter server node are different CPU clusters.

[0080] Take the information recommendation model as an example of the model to be trained: the sparse features in the training samples of the information recommendation model occupy the main space. In the data processing method provided by the embodiment of the present disclosure, the feature extraction node can send the sparse features of the sample data to the Parameter Server node for queuing, and the classification labels and dense features are queued locally at the feature extraction node. The worker node reads the classification labels and dense features of the sample data from the feature extraction node, and reads the Embedding vectors corresponding to the sparse features of the sample data from the ParameterServer node, and can train the information recommendation model. The sparse features of the sample data do not occupy the input network bandwidth of the worker node, that is, they do not occupy the input network bandwidth of the GPU. Thereby, the input network bandwidth of the GPU consumed by the training samples can be greatly reduced, so that the input network bandwidth of the GPU can be used more for transmitting the Embedding vectors, thereby making full use of the resources of the GPU and improving the utilization rate of the GPU.

[0081] Optionally, the data processing method provided by the embodiment of the present disclosure has higher training efficiency than existing model training methods, and GPU utilization can be increased from at least 40% to 65%+.

[0082] Combine the following Figure 4The process of training the stored model to be trained by the third node in the embodiment of the present disclosure according to the dense features and classification labels of the sample data and the embedded vectors corresponding to the sparse features of the sample data is specifically described.

[0083] Figure 4 Another flowchart of the data processing method provided by an embodiment of the present disclosure is shown.

[0084] like Figure 4 As shown, S306 may include: S401-S403.

[0085] S401. The third node determines a predicted classification label corresponding to the sample data through the model to be trained according to the dense features of the sample data and the embedded vectors corresponding to the sparse features of the sample data.

[0086] S402: The third node performs back propagation calculation on the model to be trained according to the error between the classification label of the sample data and the predicted classification label to obtain the gradient of the parameters of the model to be trained.

[0087] S403: The third node updates the parameters of the model to be trained according to the gradient.

[0088] Taking the advertisement classification model as an example, the third node can input the dense features of the sample data and the embedding vectors corresponding to the sparse features of the sample data into the first input layer of the neural network of the advertisement classification model, and use the forward propagation algorithm to perform forward propagation calculations on them to obtain the predicted classification labels corresponding to the sample data output by the advertisement classification model. After obtaining the predicted classification labels, the third node can calculate the error between the classification labels of the sample data and the predicted classification labels, and use the back propagation algorithm to perform back propagation calculations on the advertisement classification model based on the error to obtain the gradient of the parameters of the advertisement classification model. Then, the third node can update the parameters of the advertisement classification model based on the gradient of the parameters of the advertisement classification model.

[0089] It should be noted that the forward propagation algorithm and the back propagation algorithm are relatively mature algorithms in the field of deep learning. The embodiments of the present disclosure will not be described in detail here, and the specific types of the forward propagation algorithm and the back propagation algorithm are not limited.

[0090] In addition, the embodiments of the present disclosure do not limit the specific type of the model to be trained. For example, the model to be trained may also be a news recommendation model, a video push model, etc.

[0091] In some embodiments, the third node performs back propagation calculation on the model to be trained according to the classification label of the sample data and the predicted classification label, and after obtaining the gradient of the parameters of the model to be trained, the gradient can also be sent to the second node. The second node can update the parameters of the corresponding relationship between the sparse features and the embedding vector according to the received gradient.

[0092] For example, Figure 5 FIG. 2 shows another flow chart of the data processing method provided by the embodiment of the present disclosure. Figure 5 As shown, in Figure 4 On the basis shown, the data processing method may further include:

[0093] S501: The third node sends a gradient to the second node.

[0094] S502: The second node updates the corresponding relationship between the stored sparse features and the embedded vector according to the gradient.

[0095] After each back-propagation calculation of the training model, the second node updates the parameters of the correspondence between the sparse features and the embedding vector according to the gradient sent by the third node, so that the correspondence between the sparse features and the embedding vector can be more mature.

[0096] In some possible implementations, when the third node is a GPU cluster composed of multiple GPUs, the model to be trained can be deployed on multiple GPUs in a model parallel or data parallel manner to implement model training.

[0097] Model parallelism refers to dividing the model to be trained into multiple different parts, and different parts can be deployed on different GPUs for operation. For example, different network layers of the neural network of the model to be trained can be deployed on different GPUs for operation.

[0098] Data parallelism means deploying the same model to be trained on different GPUs. Different GPUs can use different training samples to train the model to be trained. After the training of the model to be trained on all GPUs is completed, the gradients of the model to be trained on each GPU can be collected to uniformly update the parameters of the final model to be trained.

[0099] In some embodiments, the second node, i.e., the parameter server node, may be a distributed Parameter Server node. For example, a distributed Parameter Server node may be composed of multiple Parameter Server sub-nodes, and each Parameter Server sub-node may be deployed on a CPU. Among the multiple Parameter Server sub-nodes, each Parameter Server sub-node may communicate with other Parameter Server sub-nodes, and each Parameter Server sub-node is responsible for updating some parameters of the correspondence between sparse features and Embedding vectors. Multiple Parameter Server sub-nodes jointly maintain the update of all parameters of the correspondence between sparse features and Embedding vectors.

[0100] Optionally, there may be a metadata Parameter Server subnode among the multiple Parameter Server subnodes, which may be used to maintain the consistency of some metadata among the multiple Parameter Server subnodes. For example, the metadata may include: the status of each Parameter Server subnode, the allocation of parameters, etc.

[0101] Specifically, the distributed Parameter Server node can realize the correspondence between sparse features and Embedding vectors based on the distributed key-value service. After the worker node (the third node mentioned above) sends the gradient to the distributed Parameter Server node, the distributed Parameter Server node can aggregate the gradient returned from the worker node to update the parameters.

[0102] In some embodiments, in order to improve the efficiency of model training, the sample data may also be divided into multiple sample batches according to the classification labels for processing. Figure 6 Another flowchart of the data processing method provided by the embodiment of the present disclosure is shown.

[0103] like Figure 6 As shown, based on the above-mentioned embodiments, the data processing method may further include S601 before S302 and S303.

[0104] S601: The first node divides the sample data into multiple sample batches according to the classification labels of the sample data, and each of the multiple sample batches includes one or more sample data with the same classification label.

[0105] For example, if there are 100 sample data, the first node may divide the 100 sample data into n batches, where n is an integer greater than 0 and less than or equal to 100, such as 10. Among the n batches, the sample data in different batches may have the same or different classification labels, but the classification labels of all sample data in each batch are the same.

[0106] Accordingly, S302 may include S602.

[0107] S602: The first node sends the sparse features of each sample batch in multiple sample batches to the second node.

[0108] For example, the first node may send the sparse features of each batch to the second node for queuing.

[0109] S303 may include S603.

[0110] S603: The first node sends the dense features and classification labels of each sample batch in the multiple sample batches to the third node.

[0111] Similar to the aforementioned S303, in one implementation, the dense features and classification labels of each batch may be queued locally. The third node may send a training request to the first node, and when the first node receives the training request, the classification labels and dense features of each batch may be sent to the third node.

[0112] In another implementation manner, the first node may also actively send the classification label and dense features of each batch to the third node, which is not limited in the present disclosure.

[0113] S304 may include S604.

[0114] S604: The second node determines the embedding vector corresponding to the sparse features of the first sample batch according to the sparse features of the first sample batch and the stored correspondence between the sparse features and the embedding vector.

[0115] The first sample batch is any one of the multiple sample batches. The term "first sample batch" is used only for the convenience of describing the specific implementation process of this solution.

[0116] For example, the second node may determine the Embedding vector corresponding to the sparse features of the first batch according to the sparse features of the first batch and the corresponding relationship between the stored sparse features and the Embedding vector.

[0117] S305 may include S605.

[0118] S605: The second node sends the embedding vector corresponding to the sparse features of the first sample batch to the third node.

[0119] Accordingly, the third node receives the embedding vector corresponding to the sparse features of the first sample batch from the second node.

[0120] In some embodiments, the first sample batch may refer to each sample batch in the above-mentioned multiple sample batches. The second node may determine the embedding vector corresponding to the sparse feature of each sample batch and send it to the third node.

[0121] In some other embodiments, the third node may obtain identification information of the first sample batch, and request the embedding vector corresponding to the sparse features of the first sample batch from the second node according to the identification information of the first sample batch.

[0122] S306 may include S606.

[0123] S606: The third node trains the stored model to be trained according to the dense features and classification labels of the first sample batch and the embedding vectors corresponding to the sparse features of the first sample batch.

[0124] The training process of the training model in S606 is the same or similar to that described in the aforementioned S306, with the only difference being that the training model is trained in batches according to the dense features and classification labels of each batch and the embedding vectors corresponding to the sparse features of each batch, which will not be described in detail here.

[0125] In the disclosed embodiment, by dividing the sample data into multiple batches and training the model to be trained in batches, the model training efficiency can be improved.

[0126] Combine the following Figure 7 The specific process of the third node obtaining the identification information of the first sample batch and requesting the second node for the embedding vector corresponding to the sparse features of the first sample batch according to the identification information of the first sample batch is described.

[0127] Figure 7 Another flowchart of the data processing method provided by the embodiment of the present disclosure is shown.

[0128] like Figure 7 As shown in the aforementioned Figure 6 On the basis shown, before S604, the data processing method may further include: S701-S703.

[0129] S701. The third node obtains identification information of a first sample batch.

[0130] In some embodiments, certain fields in the dense features and classification labels of the first sample batch from the first node may contain identification information of the first sample batch. For example, the first node may add some fields or parameters that can represent the identity of the batch, which may also be referred to as batch identification, to the dense features and / or classification labels of each batch. Taking the first batch as an example, the first node may add a field for batch identification to the name of the classification label of the first batch. When the third node reads the classification label of the first batch, it may directly obtain the batch identification of the first batch based on the classification label of the first batch. That is, the third node obtaining the identification information of the first sample batch may refer to the third node determining the identification information of the first sample batch based on the fields in the dense features and classification labels of the first sample batch from the first node that are used to represent the identification information of the first sample batch.

[0131] In some implementations, when the first node sends the dense features and classification labels of the first sample batch to the third node, the identification information of the first sample batch may be sent at the same time, such as batch_ID of the first batch. The third node obtaining the identification information of the first sample batch may refer to the third node obtaining the identification information of the first sample batch sent by the first node, which is not limited in the present disclosure.

[0132] S702: The third node sends identification information of the first sample batch to the second node.

[0133] Correspondingly, the second node receives the identification information of the first sample batch sent by the third node.

[0134] S703: The second node determines the sparse features of the first sample batch from the sparse features of the multiple sample batches from the first node according to the identification information of the first sample batch.

[0135] For example, after receiving the batch identifier of the first batch sent by the third node, the second node can search for the sparse features of the first batch from the sparse features of multiple batches sent by the first node according to the batch identifier of the first batch.

[0136] It should be noted that the above Figure 7The figure shown is only an optional implementation method for the second node to determine the sparse features of the first sample batch in the embodiment of the present disclosure. In actual implementation, the first node may also send the batch identifier of the first batch to the second node at the same time when sending the dense features and classification labels of the first batch to the third node. After the second node receives the batch identifier of the first batch sent by the first node, it can also obtain the Embedding vector corresponding to the sparse features of the first batch according to the batch identifier of the first batch, and send it to the third node. With respect to the above Figure 7 As for the implementation mode described above, in this implementation mode, the third node does not need to actively obtain the Embedding vector from the second node. The present disclosure does not limit the specific implementation mode.

[0137] Optionally, the present disclosure also provides a data processing system. The structure of the data processing system can refer to the aforementioned Figure 2 The data processing system includes: a first node, a second node and a third node.

[0138] The first node is used to obtain sample data for training the model to be trained, the sample data includes sample features and classification labels, and the sample features include sparse features and dense features; the sparse features of the sample data are sent to the second node, and the dense features and classification labels of the sample data are sent to the third node.

[0139] The second node is used to determine the embedding vector corresponding to the sparse feature of the sample data according to the sparse feature of the sample data and the corresponding relationship between the stored sparse feature and the embedding vector, and send the embedding vector corresponding to the sparse feature of the sample data to the third node.

[0140] The third node is used to train the stored model to be trained according to the dense features and classification labels of the sample data and the embedding vectors corresponding to the sparse features of the sample data.

[0141] Optionally, the first node is specifically used to divide the sample data into multiple sample batches according to the classification labels of the sample data, each of the multiple sample batches including one or more sample data with the same classification labels; send sparse features of each of the multiple sample batches to the second node, and send dense features and classification labels of each of the multiple sample batches to the third node.

[0142] Optionally, the second node is specifically used to determine the embedding vector corresponding to the sparse features of the first sample batch according to the sparse features of the first sample batch and the correspondence between the stored sparse features and the embedding vector, and send the embedding vector corresponding to the sparse features of the first sample batch to the third node; wherein the first sample batch is any one of the multiple sample batches. The third node is specifically used to train the stored model to be trained according to the dense features and classification labels of the first sample batch and the embedding vector corresponding to the sparse features of the first sample batch.

[0143] Optionally, the third node is further used to obtain identification information of the first sample batch and send the identification information of the first sample batch to the second node. The second node is further used to determine the sparse features of the first sample batch from the sparse features of multiple sample batches from the first node according to the identification information of the first sample batch.

[0144] Optionally, the third node is specifically used to determine the predicted classification label corresponding to the sample data through the model to be trained based on the dense features of the sample data and the embedding vectors corresponding to the sparse features of the sample data; perform back propagation calculation on the model to be trained based on the error between the classification label of the sample data and the predicted classification label to obtain the gradient of the parameters of the model to be trained; and update the parameters of the model to be trained based on the gradient.

[0145] Optionally, the third node is further used to send a gradient to the second node. The second node is further used to update the corresponding relationship between the stored sparse features and the embedded vector according to the gradient.

[0146] In an exemplary embodiment, the present disclosure further provides a data processing device, comprising: a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in the aforementioned method embodiment. For example, the processor may include a GPU, a first CPU and a second CPU, the first CPU may serve as the above-mentioned first node, the second CPU may serve as the above-mentioned second node, and the GPU may serve as the above-mentioned third node.

[0147] In an exemplary embodiment, the present disclosure further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the method described in the foregoing method embodiment is implemented.

[0148] Alternatively, the computer-readable storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0149] In an exemplary embodiment, the present disclosure also provides a computer program product, which, when executed, implements: the method described in the foregoing method embodiment.

[0150] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0151] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The method is applied to a data processing system, the data processing system comprising: a first node, a second node and a third node; wherein the first node, the second node and the third node are independent nodes, and the method comprises: The first node acquires sample data for training the model to be trained, the sample data including sample features and classification labels, and the sample features including sparse features and dense features; The first node sends the sparse features of the sample data to the second node, and sends the dense features and classification labels of the sample data to the third node; The second node determines the embedding vector corresponding to the sparse feature of the sample data according to the sparse feature of the sample data and the correspondence between the stored sparse feature and the embedding vector, and sends the embedding vector corresponding to the sparse feature of the sample data to the third node; The third node trains the stored model to be trained according to the dense features and classification labels of the sample data and the embedding vectors corresponding to the sparse features of the sample data.

2. The method according to claim 1, characterized in that Before the first node sends the sparse features of the sample data to the second node and sends the dense features and classification labels of the sample data to the third node, the method further includes: The first node divides the sample data into a plurality of sample batches according to the classification label of the sample data, each of the plurality of sample batches including one or more sample data with the same classification label; The first node sends the sparse features of the sample data to the second node, and sends the dense features and classification labels of the sample data to the third node, including: The first node sends the sparse features of each sample batch in the multiple sample batches to the second node, and sends the dense features and classification labels of each sample batch in the multiple sample batches to the third node.

3. The method according to claim 2, characterized in that The second node determines the embedding vector corresponding to the sparse feature of the sample data according to the sparse feature of the sample data and the correspondence between the stored sparse feature and the embedding vector, and sends the embedding vector corresponding to the sparse feature of the sample data to the third node, including: The second node determines the embedding vector corresponding to the sparse features of the first sample batch according to the sparse features of the first sample batch and the correspondence between the stored sparse features and the embedding vector, and sends the embedding vector corresponding to the sparse features of the first sample batch to the third node; wherein the first sample batch is any one of the multiple sample batches; The third node trains the stored model to be trained according to the dense features and classification labels of the sample data and the embedded vectors corresponding to the sparse features of the sample data, including: The third node trains the stored model to be trained according to the dense features and classification labels of the first sample batch and the embedding vectors corresponding to the sparse features of the first sample batch.

4. The method according to claim 3, characterized in that Before the second node determines the embedding vector corresponding to the sparse features of the first sample batch according to the sparse features of the first sample batch and the corresponding relationship between the stored sparse features and the embedding vector, the method further includes: The third node obtains identification information of the first sample batch; The third node sends identification information of the first sample batch to the second node; The second node determines the sparse features of the first sample batch from the sparse features of the multiple sample batches from the first node according to the identification information of the first sample batch.

5. The method according to any one of claims 1 to 4, characterized in that: The third node trains the stored model to be trained according to the dense features and classification labels of the sample data and the embedded vectors corresponding to the sparse features of the sample data, including: The third node determines the predicted classification label corresponding to the sample data through the to-be-trained model according to the dense features of the sample data and the embedded vectors corresponding to the sparse features of the sample data; The third node performs back propagation calculation on the model to be trained according to the error between the classification label of the sample data and the predicted classification label to obtain the gradient of the parameters of the model to be trained; The third node updates the parameters of the model to be trained according to the gradient.

6. The method according to claim 5, characterized in that The method further comprises: The third node sends the gradient to the second node; The second node updates the corresponding relationship between the stored sparse features and the embedded vector according to the gradient.

7. A data processing system, characterized in that: The data processing system comprises a first node, a second node and a third node; wherein the first node, the second node and the third node are independent nodes; The first node is used to obtain sample data for training the model to be trained, the sample data including sample features and classification labels, the sample features including sparse features and dense features; send the sparse features of the sample data to the second node, and send the dense features and classification labels of the sample data to the third node; The second node is used to determine the embedding vector corresponding to the sparse feature of the sample data according to the sparse feature of the sample data and the corresponding relationship between the stored sparse feature and the embedding vector, and send the embedding vector corresponding to the sparse feature of the sample data to the third node; The third node is used to train the stored model to be trained according to the dense features and classification labels of the sample data and the embedded vectors corresponding to the sparse features of the sample data.

8. The data processing system according to claim 7, characterized in that: The first node is specifically used to divide the sample data into multiple sample batches according to the classification labels of the sample data, each of the multiple sample batches including one or more sample data with the same classification labels; send the sparse features of each of the multiple sample batches to the second node, and send the dense features and classification labels of each of the multiple sample batches to the third node.

9. The data processing system according to claim 8, characterized in that: The second node is specifically configured to determine the embedding vector corresponding to the sparse features of the first sample batch according to the sparse features of the first sample batch and the correspondence between the stored sparse features and the embedding vector, and send the embedding vector corresponding to the sparse features of the first sample batch to the third node; wherein the first sample batch is any one of the multiple sample batches; The third node is specifically used to train the stored model to be trained according to the dense features and classification labels of the first sample batch and the embedding vectors corresponding to the sparse features of the first sample batch.

10. The data processing system according to claim 9, characterized in that: The third node is further configured to obtain identification information of the first sample batch; and send the identification information of the first sample batch to the second node; The second node is further configured to determine the sparse features of the first sample batch from the sparse features of the multiple sample batches from the first node according to the identification information of the first sample batch.

11. The data processing system according to any one of claims 7 to 10, characterized in that: The third node is specifically used to determine the predicted classification label corresponding to the sample data through the model to be trained based on the dense features of the sample data and the embedded vectors corresponding to the sparse features of the sample data; perform back propagation calculation on the model to be trained based on the error between the classification label of the sample data and the predicted classification label to obtain the gradient of the parameters of the model to be trained; and update the parameters of the model to be trained based on the gradient.

12. The data processing system according to claim 11, characterized in that: The third node is further configured to send the gradient to the second node; The second node is further configured to update the corresponding relationship between the stored sparse features and the embedded vector according to the gradient.

13. A data processing device, characterized in that: include: A processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method according to any one of claims 1-6.

14. A computer program product, characterized in that When the computer program product is executed, the method according to any one of claims 1 to 6 is implemented.

15. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.