Method, device and electronic device for storing characteristic information of model training system

Abstract

The present disclosure provides a method, device and electronic device for storing characteristic information of a model training system, which are used to balance the load of each parameter server node storing characteristic information during the network model training process. The method includes: storing the characteristic information of the target neural network model in at least one parameter server node according to the load value of each parameter server node; superimposing the load impact value of the characteristic information with the load value of at least one parameter server node to obtain the updated load value of at least one parameter server node; the load impact value is determined based on the calculation amount of the characteristic information during the training process of the target neural network model, and the calculation amount is determined based on the feature type of the characteristic information and the network structure of the target neural network model. The method can store each characteristic information of the target network model in each parameter server node in a balanced manner, reducing the waste of CPU, bandwidth, memory and other resources of the parameter server node.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device and electronic device for storing feature information of a model training system. Background Art

[0002] In the related technology, GPU clusters and parameter server (PS) clusters are often used to perform large-scale accelerated training of neural network models. When training the neural network model, the feature information of the neural network model will be randomly stored on each PS node in the PS cluster. However, in large-scale neural network models, the data dimensions between different feature information vary greatly, and the processor CPU, bandwidth, memory, etc. consumed by the PS nodes when processing different feature information vary greatly. When the feature information is randomly stored on each PS node for storage, it will cause unbalanced loads on different PS nodes, which may easily cause some PS nodes to be abnormal, thereby wasting resources in the entire PS cluster. Therefore, how to reduce the load of the PS cluster and balance the load of each PS node in the PS cluster when storing feature information is a problem that needs to be considered. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, device and electronic device for storing feature information of a model training system, which are used to reduce the load of storing feature information of a PS cluster and balance the load of each PS node during the training of a neural network model.

[0004] In a first aspect, the present disclosure provides a method for storing feature information of a model training system, comprising:

[0005] storing characteristic information of the target neural network model in at least one parameter server node among the plurality of parameter server nodes according to a load value of each parameter server node among the plurality of parameter server nodes; and

[0006] The load impact value of the characteristic information is superimposed on the load value of the at least one parameter server node to obtain an updated load value of the at least one parameter server node; the load impact value is determined based on the amount of calculation of the characteristic information during the training process of the target neural network model, and the amount of calculation is determined based on the feature type of the characteristic information and the network structure of the target neural network model.

[0007] In a possible implementation manner, before the step of superimposing the load impact value of the characteristic information with the load value of the at least one parameter server node, the method further includes:

[0008] Determine a preset reference value corresponding to a feature type of the feature information; the preset reference value is determined based on the amount of calculation of a parameter of the feature type during the training process of the target neural network model;

[0009] The characteristic information is weighted by using the preset reference value to obtain a load impact value of the characteristic information.

[0010] In a possible implementation, the feature information includes sparse feature information and / or dense feature information.

[0011] The step of performing weighted processing on the characteristic information by using the preset reference value to obtain the load impact value of the characteristic information includes:

[0012] If the feature information is the sparse feature information, the target feature quantity and the data volume of the sparse feature information are processed using a preset first reference value to obtain a load impact value of the sparse feature information, wherein the target feature quantity is determined according to the number of target features extracted from the training samples of the target neural network model, and the target features are used to train the sparse feature information; the first reference value is determined based on the amount of calculation of the sparse feature information during the training process of the target neural network model; and / or,

[0013] If the feature information is the dense feature information, the data amount of the sharding parameter of the dense feature information is processed using a preset second reference value to obtain the load impact value of the dense feature information, and the sharding parameter is obtained by performing distributed sharding processing on the dense feature information; the second reference value is determined based on the computational amount of the dense feature information during the training process of the target neural network model.

[0014] In a possible implementation, the target feature quantity includes a first feature quantity and a second feature quantity, wherein the first feature quantity is the quantity of the extracted target features, and the second feature quantity is the quantity of the target features after deduplication processing is performed on the extracted target features;

[0015] The step of using the first reference value to process the target feature quantity and the data volume of the sparse feature information to obtain the load impact value of the sparse feature information includes:

[0016] Using the first feature weight, the second feature weight and the parameter weight, a weighted summation process is performed on the first feature quantity, the second feature quantity and the data volume of the sparse feature information to obtain a load impact reference value of the sparse feature information; the first feature weight, the second feature weight and the parameter weight are determined according to the application scenario of the target neural network model;

[0017] The product of the first reference value and the load impact reference value is determined as the load impact value of the sparse feature information.

[0018] In a possible implementation, the step of processing the data amount of the sharding parameter of the dense feature information by using the second reference value to obtain the load impact value of the dense feature information includes:

[0019] The product of the data volume of the slice parameter of the dense feature information and the second reference value is determined as the load impact value of the dense feature information.

[0020] In a possible implementation, when the feature information includes the sparse feature information and the dense feature information, the sum of the first reference value and the second reference value is a reference value threshold, and

[0021] The first reference value is not less than the product of the second reference value and a preset empirical value; the preset empirical value is determined based on the amount of calculation of the sparse feature information in one round of training of the target neural network model, and the amount of calculation of the dense feature information in one round of training of the target neural network model.

[0022] In a possible implementation, the step of storing the characteristic information of the target neural network model in at least one parameter server node among the multiple parameter server nodes according to the load value of each parameter server node among the multiple parameter server nodes includes:

[0023] Slice the feature information to obtain N slice data, where N is a positive integer;

[0024] Based on the load values ​​of the multiple parameter server nodes, the N slice data are stored in at least one parameter server node among the multiple parameter server nodes.

[0025] In a possible implementation, the step of storing the plurality of shard data in at least one parameter server node among the N parameter server nodes based on the load values ​​of the plurality of parameter server nodes includes:

[0026] Performing N rounds of placement operations on the feature information, storing the N shard data in the at least one parameter server node, wherein each round of placement operation includes: determining a target parameter server node with the smallest load value among the multiple parameter server nodes, selecting a shard data from the shard data not stored in the N shard data, and storing the selected shard data in the target parameter server node; or

[0027] The N is less than the total number of the multiple parameter server nodes, and the first N parameter server nodes with the smallest load values ​​among the multiple parameter server nodes are determined, and the N shard data are respectively stored in each of the first N parameter server nodes.

[0028] In a second aspect of the present disclosure, a device for storing feature information of a model training system is provided, comprising:

[0029] An information storage unit is configured to store characteristic information of a target neural network model in at least one parameter server node among the plurality of parameter server nodes according to a load value of each parameter server node among the plurality of parameter server nodes;

[0030] The load update unit is configured to execute superposition of the load impact value of the characteristic information and the load value of the at least one parameter server node to obtain an updated load value of the at least one parameter server node; the load impact value is determined based on the amount of calculation of the characteristic information during the training process of the target neural network model, and the amount of calculation is determined based on the feature type of the characteristic information and the network structure of the target neural network model.

[0031] In a possible implementation, the load updating unit is further configured to execute:

[0032] Before superimposing the load impact value of the feature information with the load value of the at least one parameter server node, determining a preset reference value corresponding to the feature type of the feature information; the preset reference value is determined based on the amount of calculation of the parameter of the feature type during the training process of the target neural network model;

[0033] The characteristic information is weighted by using the preset reference value to obtain a load impact value of the characteristic information.

[0034] In a possible implementation manner, the feature information includes sparse feature information and / or dense feature information, and the load updating unit is specifically configured to execute:

[0035] If the feature information is the sparse feature information, the target feature quantity and the data volume of the sparse feature information are processed using a preset first reference value to obtain a load impact value of the sparse feature information, wherein the target feature quantity is determined according to the number of target features extracted from the training samples of the target neural network model, and the target features are used to train the sparse feature information; the first reference value is determined based on the amount of calculation of the sparse feature information during the training process of the target neural network model; and / or,

[0036] If the feature information is the dense feature information, the data amount of the sharding parameter of the dense feature information is processed using a preset second reference value to obtain the load impact value of the dense feature information, and the sharding parameter is obtained by performing distributed sharding processing on the dense feature information; the second reference value is determined based on the computational amount of the dense feature information during the training process of the target neural network model.

[0037] In a possible implementation, the target feature quantity includes a first feature quantity and a second feature quantity, the first feature quantity is the quantity of the extracted target features, and the second feature quantity is the quantity of the target features after deduplication processing is performed on the extracted target features; the load update unit is specifically configured to execute:

[0038] Using the first feature weight, the second feature weight and the parameter weight, a weighted summation process is performed on the first feature quantity, the second feature quantity and the data volume of the sparse feature information to obtain a load impact reference value of the sparse feature information; the first feature weight, the second feature weight and the parameter weight are determined according to the application scenario of the target neural network model;

[0039] The product of the first reference value and the load impact reference value is determined as the load impact value of the sparse feature information.

[0040] In a possible implementation, the load updating unit is specifically configured to execute:

[0041] The product of the data volume of the slice parameter of the dense feature information and the second reference value is determined as the load impact value of the dense feature information.

[0042] In a possible implementation, when the feature information includes the sparse feature information and the dense feature information, the sum of the first reference value and the second reference value is a reference value threshold, and

[0043] The first reference value is not less than the product of the second reference value and a preset empirical value; the preset empirical value is determined based on the amount of calculation of the sparse feature information in one round of training of the target neural network model, and the amount of calculation of the dense feature information in one round of training of the target neural network model.

[0044] In a possible implementation, the information storage unit is specifically configured to execute:

[0045] Slice the feature information to obtain N slice data, where N is a positive integer;

[0046] Based on the load values ​​of the multiple parameter server nodes, the N slice data are stored in at least one parameter server node among the multiple parameter server nodes.

[0047] In a possible implementation, the information storage unit is specifically configured to execute:

[0048] Performing N rounds of placement operations on the feature information, storing the N shard data in the at least one parameter server node, wherein each round of placement operation includes: determining a target parameter server node with the smallest load value among the multiple parameter server nodes, selecting a shard data from the shard data not stored in the N shard data, and storing the selected shard data in the target parameter server node; or

[0049] The N is less than the total number of the multiple parameter server nodes, and the first N parameter server nodes with the smallest load values ​​among the multiple parameter server nodes are determined, and the N shard data are respectively stored in each of the first N parameter server nodes.

[0050] According to a third aspect of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect and any one of the possible implementation modes is implemented.

[0051] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect and any one of the possible implementation modes.

[0052] In a fifth aspect of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the method described in the first aspect and any possible implementation manner.

[0053] The solution disclosed herein brings at least the following beneficial effects:

[0054] In the present invention, characteristic information of the target network model is stored in one PS node or multiple PS nodes in the PS cluster according to the load value of each PS node in the PS cluster, and the load value of the PS node storing the characteristic information is updated according to the load impact value of the characteristic information. Based on the load condition of the PS node and the load impact of the characteristic information, each characteristic information of the target network model can be evenly stored in each PS node, thereby making full use of the resources of each PS node in the PS cluster and avoiding unnecessary waste of CPU, bandwidth, memory and other resources of the PS node. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0056] Figure 1 A structural example diagram of a model training system provided for an exemplary embodiment of the present disclosure;

[0057] Figure 2 A schematic diagram of a process of a method for storing feature information of a model training system provided by an exemplary embodiment of the present disclosure;

[0058] Figure 3 A schematic diagram of the complete process of a method for storing feature information of a model training system provided by an exemplary embodiment of the present disclosure;

[0059] Figure 4 A schematic diagram of the structure of a storage device for characteristic information of a model training system provided by an exemplary embodiment of the present disclosure;

[0060] Figure 5 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0062] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein.

[0063] In order to help those skilled in the art better understand the technical solutions of the present disclosure, the technical terms involved in the present disclosure are explained below.

[0064] GPU cluster: is a computer cluster in which each node is equipped with a graphics processing unit (GPU). By using general-purpose computing on the graphics processing unit (GPU) to utilize the computing power of modern GPUs, GPU clusters can be used to perform very fast calculations, and CPU clusters are used to perform forward calculations and back propagation of neural network models.

[0065] Parameter server cluster: a cluster composed of parameter server (PS) nodes; usually, the feature information that needs to be shared by worker nodes in the distributed training system of the Tensorflow system of the neural network model is stored on the PS node, that is, the PS node is the node that stores the feature information; the worker node is responsible for calculating the gradient of the feature information. In each iteration process, the worker node obtains the feature information from the PS node, and then returns the calculated gradient of the feature information to the PS node. The PS node aggregates the gradient returned from the worker node, then updates the feature information, and broadcasts the new feature information to the worker node. The above-mentioned worker node can be a GPU node in the above-mentioned GPU cluster.

[0066] Feature information of the neural network model: In the embodiments of the present disclosure, feature information is information that characterizes the features of the neural network model. The feature information may include, but is not limited to, model parameters, feature variables, etc. of the neural network model.

[0067] Greedy algorithm: refers to an algorithm that takes the best or optimal (i.e. most favorable) option in each step when solving a problem, hoping to lead to the best or optimal result; each greedy decision made cannot be changed, because the greedy strategy is to derive the optimal solution for the next step from the optimal solution of the previous step, and the optimal solution before the previous step is not retained.

[0068] Dynamic Programming Algorithm: Dynamic programming algorithm is usually used to solve problems with some optimal properties. In such problems, there may be many feasible solutions. Each solution corresponds to a value, and we hope to find the solution with the optimal value. The basic idea is to decompose the problem to be solved into several sub-problems, solve the sub-problems first, and then get the solution to the original problem from the solutions of these sub-problems. The global optimal solution must contain a local optimal solution, but it does not necessarily contain the previous local optimal solution, so it is necessary to record all previous optimal solutions.

[0069] The design concept of the present disclosure is described below.

[0070] With the development of Internet technology, deep neural network models are increasingly used in information recommendation, advertising, information retrieval and other fields. In order to improve the accuracy of neural network models, the model scale of neural network models is getting larger and larger, and its model scale can reach hundreds of G, TB and other levels. This brings great challenges to the training system of neural network models. At present, GPU cluster + PS cluster method is generally used to perform large-scale accelerated training of neural network models; in the above training method, the forward calculation and back propagation of the neural network model are performed in the GPU cluster, and the feature information of the neural network model is stored in the PS node in the PS cluster. At present, the feature information is often randomly stored in each PS node in the PS cluster, but in large-scale neural network models, the data dimensions between different feature information are very different. The random storage of feature information will cause large differences in the consumption of processor CPU, bandwidth, memory and other resources of different PS nodes, which will lead to unbalanced loads of different PS nodes, easily causing abnormalities of some PS nodes, and also causing serious waste of resources in the entire PS cluster.

[0071] In view of this, the present invention designs a feature information processing method, device and electronic device of a model training system, so as to reduce the resource consumption of the CPU, bandwidth, memory, etc. of the PS cluster during the training of the neural network model, and balance the load of each PS node in the PS cluster; in order to achieve this goal, in the embodiment of the present invention, the feature information is distributed to one or more PS nodes in the PS cluster for storage according to the load of each PS node in the PS cluster and the degree of influence of each feature information of the target network model on the load of the PS node; specifically, the method of the present invention includes: according to the load value of each PS node in the PS cluster, storing the feature information of the target network model in at least one PS node in the PS cluster; and updating the load value of the at least one PS node according to the load influence value of the above feature information; the load influence value is determined based on the calculation amount of the above feature information in the training process of the target neural network model, and the above calculation amount is determined based on the feature type of the above feature information and the network structure of the target neural network model.

[0072] The solution of the present disclosure is described in detail below with reference to the accompanying drawings.

[0073] See also Figure 1, a structural example diagram of a model training system is provided, wherein the model training system includes a GPU cluster 110 and a PS cluster 120, wherein the GPU cluster always includes multiple GPU nodes (such as but not limited to 110-1, 110-2, 110-3 or 110-4 as shown in the figure), and the PS cluster includes multiple PS nodes (such as but not limited to 120-1, 120-2, 120-3, 120-4 or 120-5 as shown in the figure); wherein the GPU cluster 110 and the PS cluster 120 can communicate with each other.

[0074] The GPU cluster 110 can perform forward calculation and back propagation of the target network model; the GPU node is responsible for calculating the gradient of the feature information of the target network model. In each training iteration, the GPU node obtains the feature information from the PS node, and then calculates the gradient of the feature information based on the obtained feature information, and sends the calculated gradient of the feature information to the PS node.

[0075] The PS cluster 120 can store feature information that the GPU nodes need to share, and during each training iteration, the PS node receives the gradient of the feature information sent by the GPU node, updates the stored feature information according to the received gradient, and broadcasts the updated feature information to the GPU node.

[0076] This disclosure provides a method for processing feature information for model training. Figure 2 , the method specifically comprises the following steps:

[0077] Step S201: according to the load value of each parameter server node among the plurality of parameter service nodes, the characteristic information of the target network model is stored in at least one parameter server node among the plurality of parameter server nodes.

[0078] It should be noted that the above-mentioned multiple parameter server nodes are PS nodes in the above-mentioned PS cluster 120 .

[0079] In the disclosed embodiment, various target algorithms can be flexibly used to preferentially store feature information on PS nodes with lower load values ​​in the PS cluster 120 to achieve load balancing of the PS nodes in the PS cluster 120. The target algorithms may include, but are not limited to, greedy algorithms and dynamic programming algorithms.

[0080] Specifically, in this step, the characteristic information to be stored can be sliced ​​to obtain N sliced ​​data, and based on the size of the load values ​​of the above-mentioned multiple parameter server nodes, the above-mentioned N sliced ​​data can be stored in at least one parameter server node among the above-mentioned multiple parameter server nodes; wherein N is a positive integer, and the value of N can be determined according to the specific scale of the characteristic information to be stored. If the scale of the characteristic information is small, the characteristic information can be divided into 3 or 5 slices of data, but it is not limited to; if the scale of the characteristic information is large, the characteristic information can be divided into 30 or 50 slices of data; a specific example is given here. If a characteristic information is divided into 5 slices of data, the 5 slices of data of the characteristic information can be stored in the PS node in the PS cluster 120; wherein the method for updating the load value of each PS node in the PS cluster is described in the following step S202.

[0081] It should be noted that in the embodiment of the present disclosure, there are multiple feature information of the target network model, and the number of shard data of different feature information may be the same or different, such as dividing each feature information into a first set number (such as but not limited to 5 or 10, etc.) of shard data; or dividing each feature information into a different number of shard data according to the scale of each feature information, such as setting a set data amount for each shard data, and dividing each feature information into shard data of the above set data amount. Here is an example. If the dimension of the set data amount of a shard data is (1024, 16), each feature information can be divided into shard data of size 1024×16, etc.; technicians in this field can also flexibly perform sharding processing on the feature information according to actual needs to obtain its shard data.

[0082] Further, in the embodiment of the present disclosure, the N shard data may be stored in at least one parameter server node among the multiple parameter server nodes through the following two placement strategies, but are not limited to:

[0083] The first placement strategy:

[0084] N rounds of placement operations may be performed on the feature information to store the N shard data in the at least one parameter server node, wherein each round of placement operations includes: determining a target parameter server node with the smallest load value among the multiple parameter server nodes, selecting one shard data from the shard data not stored in the N shard data, and storing the selected shard data in the target parameter server node.

[0085] Here is a specific example. If there are 10 shard data in total and the PS cluster 120 contains 100 PS nodes, 10 rounds of placement operations can be performed. In each round of placement operation, the PS node with the smallest load value among the 100 PS nodes is determined as the target parameter server node, and then an unstored shard data is selected from the 10 shard data, and the selected shard data is stored in the target parameter server node.

[0086] The second placement operation is applicable to the case where the above N is less than the total number of the above multiple parameter server nodes.

[0087] Specifically, the first N parameter server nodes with the smallest load value among the multiple parameter server nodes may be determined, and the N shard data may be stored in each of the first N parameter server nodes, respectively.

[0088] A specific example is given here. For example, if there are a total of 10 shard data and the PS cluster 120 includes 100 PS nodes, the 10 PS nodes with the smallest load values ​​among the 100 PS nodes can be determined, and then the above 10 shard data are stored in the determined 10 PS nodes; wherein each PS node can store one shard data, and can be but not limited to storing the 10 allocated data in the order of the data volume of the 10 shard data from large to small, and sequentially storing the 10 PS nodes with load values ​​from small to large in the above 10 PS nodes; each shard data of the 10 shard data can also be randomly stored in the determined 10 PS nodes, and one of the PS nodes can be but not limited to storing one shard data; technicians in this field can also flexibly adopt other methods according to actual needs to store the above N shard data in the above first N parameter server nodes respectively.

[0089] Step S202, superimpose the load impact value of the above-mentioned characteristic information with the load value of the above-mentioned at least one parameter server node to obtain an updated load value of the above-mentioned at least one parameter server node; the above-mentioned load impact value is determined based on the calculation amount of the above-mentioned characteristic information during the training process of the above-mentioned target neural network model, and the above-mentioned calculation amount is determined based on the feature type of the above-mentioned characteristic information and the network structure of the above-mentioned target neural network model.

[0090] As an embodiment, before step S202, a preset reference value corresponding to the feature type of the above-mentioned feature information can be determined, and the above-mentioned feature information can be weighted using the above-mentioned preset reference value to obtain the load impact value of the above-mentioned feature information; the above-mentioned preset reference value is determined based on the calculation amount of the parameters belonging to the above-mentioned feature type (i.e., the feature information) during the training process of the above-mentioned target neural network model.

[0091] As an embodiment, considering that the types of feature information of different network layers of a deep neural network model may be different, and different types of feature information have different degrees of influence on the load of a PS node, the feature information in the embodiment of the present disclosure may include parameters of different feature types. In order to improve the accuracy of the load influence value of feature information of different feature types, the feature information of different feature types in the embodiment of the present disclosure corresponds to different preset reference values, and the above different preset reference values ​​are determined based on the calculation amount of feature information of different feature types; further, in the embodiment of the present disclosure, when determining the load influence value of a feature information, the preset reference value corresponding to the feature type of the feature information can be used to process the feature information to obtain the load influence value of the feature information, and the preset reference value corresponding to the feature information is determined according to the calculation amount of the parameters belonging to the feature type. In this way, processing the feature information according to the preset reference value corresponding to the feature type of the feature information can improve the accuracy of the load influence value of the obtained feature information, and thus can improve the update accuracy of the load value of the PS node updated using the load update value.

[0092] As an embodiment, considering that the target network model has multiple network layers in its structure, the characteristic information of each network layer can be used as characteristic information of a characteristic type in the embodiment of the present disclosure, or the characteristic information of multiple network layers can be used as characteristic information of a characteristic type; technicians in this field can also flexibly divide the characteristic information of the target network model into characteristic categories according to other characteristics of the characteristic information (such as the data volume of the characteristic information, the data structure of the characteristic information, etc.), and no excessive restrictions are made here.

[0093] For ease of understanding, an example of dividing feature information into feature categories according to the amount of feature information data is given in the embodiment of the present disclosure. The feature information in the embodiment of the present disclosure may include feature information of at least one feature type in sparse feature information (hereinafter referred to as sparse variables or sparse parameters) and dense feature information (hereinafter referred to as dense variables or dense parameters); the sparse variables and dense variables have obvious differences in the CPU computing amount, bandwidth consumption and memory consumption of the PS node; wherein:

[0094] A sparse variable may be, but is not limited to, feature information that requires a memory space greater than a first memory threshold and a bandwidth consumption greater than a first bandwidth threshold, that is, a sparse variable may be feature information that requires a large memory space (such as GB or TB level), is suitable for storage in the form of hashes, and consumes a large amount of bandwidth. In the embodiment of the present disclosure, a sparse variable may be, but is not limited to, a model parameter of an embedding layer of an embedded network in a target network model;

[0095] A dense variable may be, but is not limited to, feature information that the memory space required for storage is less than the second memory threshold, and the bandwidth consumption is greater than the second bandwidth threshold and less than the third bandwidth threshold, that is, a dense variable may be feature information that requires a small memory space (such as may be at the MB level), is suitable for continuous storage, and has moderate bandwidth consumption. In the embodiment of the present disclosure, the dense variable may include model parameters of network layers other than the above-mentioned embedded network layer in the target network model. The above-mentioned network layers other than the embedded layer may be, but are not limited to, fully connected network layers, sigmoid network layers, and other network layers.

[0096] Correspondingly, the load influence value of the sparse variable and / or the load influence value of the dense variable may be obtained by, but is not limited to, the following first parameter processing method and / or the second parameter processing method.

[0097] The first parameter processing method is to obtain the load impact value of sparse feature information (ie, sparse variables).

[0098] In the embodiment of the present disclosure, when obtaining the load impact value of each feature information, if the feature information is sparse feature information, the target feature quantity and the data volume of the sparse feature information are processed using a preset first reference value to obtain the load impact value of the sparse feature information.

[0099] Among them, the above-mentioned target feature quantity can be determined according to the number of target features extracted from the training samples of the target network model. The above-mentioned target features are features used to train sparse feature information. If the target features are sparse features, the number of feature signatures (i.e., the number of sparse features) extracted from a training sample group (i.e., training sample batch) of the target model can be determined as the above-mentioned target feature quantity.

[0100] As an embodiment, if the sparse feature information is stored in the form of an embedding table, the spatial size of the embedding table may be determined as the data volume of the sparse feature information.

[0101] Further, in order to improve the accuracy of the load impact value of the sparse feature information, the target feature quantity in the embodiment of the present disclosure includes a first feature quantity and a second feature quantity, the first feature quantity is the number of target features extracted from the training samples of the target network model, and the second feature quantity is the number of target features after the extracted target features are deduplicated;

[0102] Furthermore, based on the following formula 1, the first feature weight, the second feature weight and the parameter weight can be used to perform weighted summation processing on the above-mentioned first feature quantity, the second feature quantity and the data amount of the sparse feature information to obtain the load impact reference value of the above-mentioned sparse feature information; and then the product of the first reference value and the above-mentioned load impact reference value can be determined as the load impact value of the sparse feature information.

[0103] Formula 1: load_of_spare_variable=(key_num×a+uniq_key_num×b+emb_size×c)×factor_sparse;

[0104] In formula 1, load_of_spare_variable is the load impact value of the sparse feature information, key_num is the number of the first features mentioned above, uniq_key_num is the number of the second features mentioned above, emb_size is the data amount of the sparse feature information mentioned above, a is the first feature weight mentioned above, b is the second feature weight mentioned above, c is the parameter weight mentioned above, (key_num×a+uniq_key_num×b+emb_size×c) is the load impact reference value of the sparse feature information mentioned above; factor_sparse is the first reference value mentioned above.

[0105] The above load_of_spare_variable comprehensively considers the impact on the CPU, bandwidth and memory consumption of the PS node. It is a comprehensive indicator, which is obtained by normalizing the first feature number, the second feature number and the data volume of the sparse feature information through three weights a, b and c (also called normalization factors), so as to facilitate horizontal comparison of different sparse feature information.

[0106] In the embodiment of the present disclosure, the three normalization factors a, b and c mentioned above need to satisfy certain constraints. The first feature weight a, the second feature weight b and the parameter weight c can be determined, but are not limited to, according to the application scenario of the target network model. For ease of understanding, some examples of specific settings are given here. For example, considering the convenience of unified measurement, the constraints satisfied by a, b and c can be set to the conditions shown in the following formula 2. Furthermore, the values ​​of a, b and c can be further set according to the association relationship between a, b and c in the actual scenario, and no excessive restrictions are made here.

[0107] Formula 2: a+b+c=1.

[0108] The second parameter processing method is to obtain the load impact value of dense feature information (dense variables).

[0109] If the feature information is dense feature information, the data amount of the sharding parameters of the dense feature information is processed using a preset second reference value to obtain the load impact value of the dense feature information. The above sharding parameters are obtained by performing distributed sharding processing on the above dense feature information, and the second reference value is determined based on the amount of calculation of the above dense feature information during the training process of the target neural network model.

[0110] Furthermore, the load impact value of the dense feature information can be determined as the product of the data volume of the slice parameter of the dense feature information and the second reference value based on, but not limited to, the following formula 3.

[0111] Formula 3: load_of_dense_variable=shard_dim_size×factor_dense;

[0112] In Formula 3, load_of_dense_variable is the load impact value of dense feature information, shard_dim_size is the data volume of the sharding parameter of the above dense feature information; factor_dense is the above second reference value.

[0113] In order to facilitate the acquisition of the load impact value of dense feature information in the embodiment of the present disclosure, the dimension sum of the sharding parameters of the dense feature information can be but is not limited to being determined as the data volume of the sharding parameters of the dense feature information. A specific example is given here. If the sharding parameter of the dense feature information is a two-dimensional (K1, K2) sharding, then the dimension sum of the sharding parameters is K1×K2. If the sharding parameter of the dense feature information is a three-dimensional (K1, K2, K3) sharding, then the dimension sum of the sharding parameters is K1×K2×K3. For example, if the dimension of a sharding parameter of dense feature information is (1024, 16), then the data volume of the sharding parameters of the dense feature information is 1024×16.

[0114] As an embodiment, when the characteristic information of the target network model contains both the above-mentioned sparse characteristic information and dense variables, since the above-mentioned sparse characteristic information and dense variables have significantly different degrees of influence on the load of the PS node, if the sparse characteristic information and the dense characteristic information are randomly stored in each PS node in the PS cluster, it is very likely that the load values ​​of each PS node in the PS cluster will be quite different, causing waste of resources of the PS cluster while affecting the training process of the target network model. Therefore, in the embodiment of the present disclosure, the sparse characteristic information and the dense characteristic information can be normalized by the first reference value and the second reference value respectively to obtain their respective load influence values, so that the load influence value of the sparse characteristic information and the load influence value of the dense characteristic information can be compared. Therefore, in the embodiment of the present disclosure, when the characteristic information contains both sparse characteristic information and dense characteristic information, the above-mentioned first preset value and the second preset value satisfy the following constraints A1 and A2.

[0115] Constraint A1: Please refer to the following formula 4, the sum of the first reference value and the above second reference value is the reference value threshold.

[0116] Formula 4: factor_sparse+factor_dense=q;

[0117] In Formula 4, factor_sparse is the first reference value, factor_dense is the second reference value, and q is the reference value threshold, wherein the reference value threshold can facilitate obtaining the load impact value of sparse feature information and dense feature information in a unified dimension, and then when the load value of each server node is updated based on the load impact value of the sparse feature information and the dense feature information, the load value of each server node can be updated in a unified dimension.

[0118] As an embodiment, in order to improve the efficiency of obtaining the load impact values ​​of each sparse feature information and dense feature information, the value of q can be set to a positive integer in the embodiment of the present application to speed up the efficiency of obtaining the load impact value of the sparse feature information based on the above formula 1, and to speed up the efficiency of obtaining the load impact value of the dense feature information based on the above formula 2, such as but not limited to setting q to 1 or 10.

[0119] Constraint A2: Please refer to the following formula 5, the first reference value is not less than the product of the second reference value and the preset experience value.

[0120] Formula 5: factor_sparse≥x×factor_dense;

[0121] In Formula 5, factor_sparse is the first reference value, factor_dense is the second reference value; x is the preset empirical value, which can be determined based on but not limited to the amount of calculation of the sparse feature information in one round of training of the target neural network model, and the amount of calculation of the dense feature information in one round of training of the target neural network model.

[0122] The preset experience value of the embodiment of the present disclosure may be automatically set by a computer or may be set by a technician based on experience. For ease of understanding, an example of determining the above-mentioned preset experience value x is given here; if in a round of training of the target network model, the average data amount of all dense feature information is m1, the average data amount of all sparse feature information is n1, and the average data amount of the sparse feature information updated in each round of training is n2, then the preset experience value x may be determined by the following formula 6;

[0123] Formula 6:

[0124] It should be noted that the method for determining the preset experience value x illustrated in the above formula 6 is only an example, and those skilled in the art can flexibly set the above preset experience value x according to actual needs.

[0125] The following is a specific example of feature information processing in a model training system.

[0126] In this example, the PS cluster includes N1 PS nodes, and the target network model includes M feature information, among which there are sparse feature information and dense feature information.

[0127] See also Figure 3 , the M feature information can be placed on N1 PS nodes in the following way.

[0128] Step S301, read N1 PS nodes in the PS cluster, and initialize the load value of each PS node to 0;

[0129] Step S302, selecting a piece of feature information from unstored feature information;

[0130] Specifically, a piece of feature information may be randomly selected from unstored feature information, or feature information may be selected in other ways.

[0131] Step S303, determining whether the selected feature information is sparse feature information, if so, proceeding to step S304, otherwise proceeding to step S305;

[0132] Step S304, using the first reference value factor_sparse to process the target feature quantity and the data volume of the sparse feature information to obtain the load impact value of the sparse feature information, and then proceeding to step S306;

[0133] Specifically, the load impact value of the sparse feature information may be calculated according to the above formula 1, which will not be repeated here.

[0134] Step S305, using the second reference value factor_dense to process the data amount of the slice parameter of the dense feature information to obtain the load impact value of the dense feature information, and then proceeding to step S306;

[0135] Specifically, the load impact value of the dense feature information may be calculated according to the above formula 3, which will not be repeated here.

[0136] It should be noted that in this example, the load impact value of each sparse feature information and the load impact value of each dense feature information can be first obtained through steps S304 and S305, and then step S302 is executed. After step S302, step S306 is executed. Technical personnel in this field can set it according to actual needs.

[0137] Step S306, performing slicing processing on the selected feature information to obtain N slicing data;

[0138] Step S307, determining the first N PS nodes with the lowest load values ​​among the N1 PS nodes;

[0139] Step S308, the obtained N slice data are stored in the first N PS nodes with the lowest load values ​​respectively;

[0140] Step S309, using the load impact value of the selected characteristic information, updating the load values ​​of the first N PS nodes;

[0141] Step S310, determine whether there is unstored feature information, if yes, proceed to step S302, otherwise end.

[0142] A code example for implementing the above process is given below. Specifically, M feature information may be placed on N1 PS nodes through, but is not limited to, the following code:

[0143] ps_list = [ps1, ps2, ..., psN1] / / Read N1 PS nodes, where ps1 to psN1 represent PS nodes numbered 1 to N1

[0144] unified variable list = [var1, var2, ..., varM] / / read M feature information, where var1 to varM represent the feature information numbered 1 to M

[0145] calc_load(unified variable list) / / Calculate the load impact value of each sparse feature information and the load impact value of each dense feature information according to the above formulas 1 and 3

[0146] init(ps_list) / / Initialize the load value of each PS node to 0

[0147] For variable in unified_variable_list: / / Perform the following steps on the read feature information

[0148] a.sorted_load_ps_list = get_sorted_ps_load (ps_list); / / According to the load value, sort the PS nodes in descending order of load value

[0149] b. Assign variable shards to top shard_num ps instance in sorted_load_ps_list / / Store the shard data of feature information on the N PS nodes with the lowest load value

[0150] / / End of allocation

[0151] Experiments have proved that the feature information processing method provided by the present invention can control the difference in the load value of the resources of each PS node in the PS cluster to within 10%, and the average resource utilization (such as but not limited to GPU, CPU, network bandwidth or memory, etc.) of the PS nodes in the PS cluster can reach more than 80%, which is about 200% higher than the traditional processing method, and the training throughput of the target network model is increased by more than 300%, which is a significant improvement.

[0152] like Figure 4 As shown, based on the same inventive concept, the embodiment of the present disclosure further provides a feature information processing device 400, including:

[0153] The information storage unit 401 is configured to store the characteristic information of the target neural network model in at least one parameter server node among the plurality of parameter server nodes according to the load value of each parameter server node among the plurality of parameter server nodes;

[0154] The load update unit 402 is configured to execute superposition of the load impact value of the above-mentioned characteristic information and the load value of the above-mentioned at least one parameter server node to obtain an updated load value of the above-mentioned at least one parameter server node; the above-mentioned load impact value is determined based on the calculation amount of the above-mentioned characteristic information during the training process of the above-mentioned target neural network model, and the above-mentioned calculation amount is determined based on the feature type of the above-mentioned characteristic information and the network structure of the above-mentioned target neural network model.

[0155] As an embodiment, the load updating unit 402 is further configured to execute:

[0156] Before superimposing the load impact value of the feature information and the load value of the at least one parameter server node, determining a preset reference value corresponding to the feature type of the feature information; the preset reference value is determined based on the amount of calculation of the parameter of the feature type during the training process of the target neural network model;

[0157] The characteristic information is weighted by using the preset reference value to obtain the load impact value of the characteristic information.

[0158] As an embodiment, the feature information includes sparse feature information and / or dense feature information, and the load updating unit 402 is specifically configured to execute:

[0159] If the feature information is the sparse feature information, the target feature quantity and the data volume of the sparse feature information are processed using a preset first reference value to obtain a load impact value of the sparse feature information, wherein the target feature quantity is determined based on the number of target features extracted from the training samples of the target neural network model, and the target features are used to train the sparse feature information; the first reference value is determined based on the amount of calculation of the sparse feature information during the training process of the target neural network model; and / or,

[0160] If the above-mentioned feature information is the above-mentioned dense feature information, the data amount of the sharding parameters of the above-mentioned dense feature information is processed using a preset second reference value to obtain the load impact value of the above-mentioned dense feature information, and the above-mentioned sharding parameters are obtained by performing distributed sharding processing on the above-mentioned dense feature information; the above-mentioned second reference value is determined based on the calculation amount of the above-mentioned dense feature information during the training process of the above-mentioned target neural network model.

[0161] As an embodiment, the target feature quantity includes a first feature quantity and a second feature quantity, the first feature quantity is the quantity of the extracted target features, and the second feature quantity is the quantity of the target features after the extracted target features are deduplicated; the load updating unit 402 is specifically configured to execute:

[0162] Using the first feature weight, the second feature weight and the parameter weight, a weighted summation process is performed on the first feature quantity, the second feature quantity and the data volume of the sparse feature information to obtain a load impact reference value of the sparse feature information; the first feature weight, the second feature weight and the parameter weight are determined according to the application scenario of the target neural network model;

[0163] The product of the first reference value and the load impact reference value is determined as the load impact value of the sparse feature information.

[0164] As an embodiment, the load updating unit 402 is specifically configured to execute:

[0165] The product of the data volume of the slice parameter of the above-mentioned dense feature information and the above-mentioned second reference value is determined as the load impact value of the above-mentioned dense feature information.

[0166] As an embodiment, when the feature information includes the sparse feature information and the dense feature information, the sum of the first reference value and the second reference value is the reference value threshold, and

[0167] The above-mentioned first reference value is not less than the product of the above-mentioned second reference value and a preset empirical value; the above-mentioned preset empirical value is determined based on the calculation amount of the above-mentioned sparse feature information in one round of training of the above-mentioned target neural network model, and the calculation amount of the above-mentioned dense feature information in one round of training of the above-mentioned target neural network model.

[0168] As an embodiment, the information storage unit 401 is specifically configured to execute:

[0169] The characteristic information is sliced ​​to obtain N sliced ​​data, where N is a positive integer;

[0170] Based on the load values ​​of the plurality of parameter server nodes, the N shard data are stored in at least one parameter server node among the plurality of parameter server nodes.

[0171] As an embodiment, the information storage unit 401 is specifically configured to execute:

[0172] Performing N rounds of placement operations on the feature information, storing the N shard data in the at least one parameter server node, wherein each round of placement operation includes: determining a target parameter server node with the smallest load value among the multiple parameter server nodes, selecting a shard data from the shard data not stored in the N shard data, and storing the selected shard data in the target parameter server node; or

[0173] The above-mentioned N is less than the total number of the above-mentioned multiple parameter server nodes, and the first N parameter server nodes with the smallest load values ​​among the above-mentioned multiple parameter server nodes are determined, and the above-mentioned N shard data are respectively stored in the above-mentioned first N parameter server nodes.

[0174] like Figure 5 As shown, the present disclosure provides an electronic device 500, including a processor 501, and a memory 502 for storing instructions executable by the processor;

[0175] Among them, the above-mentioned processor is configured to execute the process of the above-mentioned method for storing feature information of a model training system.

[0176] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of the electronic device to complete the method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0177] In an exemplary embodiment, a computer program product is also provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the method for storing feature information of the above-mentioned model training system is implemented.

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for storing feature information of a model training system, characterized in that: include: storing characteristic information of the target neural network model in at least one parameter server node among the plurality of parameter server nodes according to a load value of each parameter server node among the plurality of parameter server nodes; as well as Superimposing the load impact value of the characteristic information and the load value of the at least one parameter server node to obtain an updated load value of the at least one parameter server node; The characteristic information includes sparse characteristic information and / or dense characteristic information, and the method for determining the load impact value includes: If the feature information is the sparse feature information, a weighted summation process is performed on the target feature quantity and the data volume of the sparse feature information to obtain a load impact reference value of the sparse feature information, and then the product of a preset first reference value and the load impact reference value is determined as the load impact value of the sparse feature information, wherein the target feature quantity is determined based on the number of target features extracted from the training samples of the target neural network model, and the target features are used to train the sparse feature information; the first reference value is determined based on the computational amount of the sparse feature information in the training process of the target neural network model; and / or If the feature information is the dense feature information, the product of a preset second reference value and the data amount of a sharding parameter of the dense feature information is determined as the load impact value of the dense feature information, and the sharding parameter is obtained by performing distributed sharding processing on the dense feature information; the second reference value is determined based on the computational amount of the dense feature information during the training process of the target neural network model.

2. The method according to claim 1, characterized in that The target feature quantity includes a first feature quantity and a second feature quantity, wherein the first feature quantity is the quantity of the extracted target features, and the second feature quantity is the quantity of the target features after deduplication processing is performed on the extracted target features; The step of performing weighted sum processing on the number of target features and the data volume of the sparse feature information to obtain a load impact reference value of the sparse feature information includes: Using the first feature weight, the second feature weight and the parameter weight, the first feature quantity, the second feature quantity and the data amount of the sparse feature information are weighted and summed to obtain a load impact reference value of the sparse feature information; the first feature weight, the second feature weight and the parameter weight are determined according to the application scenario of the target neural network model.

3. The method according to claim 2, characterized in that When the feature information includes the sparse feature information and the dense feature information, the sum of the first reference value and the second reference value is the reference value threshold, and The first reference value is not less than the product of the second reference value and a preset empirical value; the preset empirical value is determined based on the amount of calculation of the sparse feature information in one round of training of the target neural network model, and the amount of calculation of the dense feature information in one round of training of the target neural network model.

4. The method according to any one of claims 1 to 3, characterized in that: The step of storing the characteristic information of the target neural network model in at least one parameter server node among the multiple parameter server nodes according to the load value of each parameter server node among the multiple parameter server nodes comprises: Slice the feature information to obtain N slice data, where N is a positive integer; Based on the load values ​​of the multiple parameter server nodes, the N slice data are stored in at least one parameter server node among the multiple parameter server nodes.

5. The method according to claim 4, characterized in that The step of storing the N shard data in at least one parameter server node among the multiple parameter server nodes based on the load values ​​of the multiple parameter server nodes comprises: Performing N rounds of placement operations on the feature information, storing the N shard data in the at least one parameter server node, wherein each round of placement operation includes: determining a target parameter server node with the smallest load value among the multiple parameter server nodes, selecting a shard data from the shard data not stored in the N shard data, and storing the selected shard data in the target parameter server node; or The N is less than the total number of the multiple parameter server nodes, and the first N parameter server nodes with the smallest load values ​​among the multiple parameter server nodes are determined, and the N shard data are respectively stored in each of the first N parameter server nodes.

6. A storage device for characteristic information of a model training system, characterized in that: include; An information storage unit is configured to store characteristic information of a target neural network model in at least one parameter server node among the plurality of parameter server nodes according to a load value of each parameter server node among the plurality of parameter server nodes; A load updating unit is configured to perform superposition of the load impact value of the characteristic information and the load value of the at least one parameter server node to obtain an updated load value of the at least one parameter server node; wherein the characteristic information includes sparse characteristic information and / or dense characteristic information, and the load updating unit is further configured to perform: If the feature information is the sparse feature information, a weighted summation process is performed on the target feature quantity and the data volume of the sparse feature information to obtain a load impact reference value of the sparse feature information, and then the product of a preset first reference value and the load impact reference value is determined as the load impact value of the sparse feature information, wherein the target feature quantity is determined based on the number of target features extracted from the training samples of the target neural network model, and the target features are used to train the sparse feature information; the first reference value is determined based on the computational amount of the sparse feature information in the training process of the target neural network model; and / or If the feature information is the dense feature information, the product of a preset second reference value and the data amount of a sharding parameter of the dense feature information is determined as the load impact value of the dense feature information, and the sharding parameter is obtained by performing distributed sharding processing on the dense feature information; the second reference value is determined based on the computational amount of the dense feature information during the training process of the target neural network model.

7. The device according to claim 6, characterized in that The target feature quantity includes a first feature quantity and a second feature quantity, wherein the first feature quantity is the quantity of the extracted target features, and the second feature quantity is the quantity of the target features after deduplication processing is performed on the extracted target features; the load update unit is specifically configured to execute: Using the first feature weight, the second feature weight and the parameter weight, the first feature quantity, the second feature quantity and the data amount of the sparse feature information are weighted and summed to obtain a load impact reference value of the sparse feature information; the first feature weight, the second feature weight and the parameter weight are determined according to the application scenario of the target neural network model.

8. The device according to claim 7, characterized in that When the feature information includes the sparse feature information and the dense feature information, the sum of the first reference value and the second reference value is the reference value threshold, and The first reference value is not less than the product of the second reference value and a preset empirical value; the preset empirical value is determined based on the amount of calculation of the sparse feature information in one round of training of the target neural network model, and the amount of calculation of the dense feature information in one round of training of the target neural network model.

9. The device according to any one of claims 6 to 8, characterized in that: The information storage unit is specifically configured to execute: Slice the feature information to obtain N slice data, where N is a positive integer; Based on the load values ​​of the multiple parameter server nodes, the N slice data are stored in at least one parameter server node among the multiple parameter server nodes.

10. The device according to claim 9, characterized in that The information storage unit is specifically configured to execute: Performing N rounds of placement operations on the feature information, storing the N shard data in the at least one parameter server node, wherein each round of placement operation includes: determining a target parameter server node with the smallest load value among the multiple parameter server nodes, selecting a shard data from the shard data not stored in the N shard data, and storing the selected shard data in the target parameter server node; or The N is less than the total number of the multiple parameter server nodes, and the first N parameter server nodes with the smallest load values ​​among the multiple parameter server nodes are determined, and the N shard data are respectively stored in each of the first N parameter server nodes.

11. An electronic device, characterized in that: comprising a processor, and a memory for storing instructions executable by the processor; The processor is configured to execute the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images