A method to accelerate the training of recommendation models on edge nodes

Abstract

The present invention discloses a method for accelerating the training of a recommendation model on an edge node, comprising: first, for each input embedding sample, respectively calculating the expected transmission cost of each embedding allocation and scheduling to each edge working node based on each edge working node, thereby obtaining a total expected transmission cost; thereafter, determining the scheduling decision of each edge working node based on the calculated total expected transmission cost, so as to obtain the embedding samples required for training the recommendation model based on the scheduling decision. The present invention can provide a scheduling decision solution that can balance solution quality and resource occupancy in response to the challenges of limited computing resources at edge working nodes and multiple tasks competing for resources. Moreover, experimental results show that, compared with the prior art, the implementation scheme provided by the present invention can achieve a training acceleration of up to 1.74 times, and can reduce the embedding transmission cost by 36.76%.

Description

Technical Field

[0001] The present invention relates to the field of intelligent technology, and in particular to a method for accelerating the training of a recommendation model on an edge node. Background Art

[0002] Recommender systems, as artificial intelligence algorithms, excel at processing high-dimensional and sparse IoT data, providing personalized recommendation services for IoT applications and enhancing the ability to retrieve information from IoT data. However, uploading IoT data to remote cloud servers for recommendation model training can lead to privacy leaks and network congestion. Therefore, training recommendation models on edge nodes near the data source has become a natural alternative.

[0003] As a key component of a recommendation system, the recommendation model typically consists of an embedding layer and a multi-layer perceptron (MLP). The embedding layer converts high-dimensional, sparse inputs into low-dimensional, dense, real-valued vectors (embeddings) and stores them in a large lookup table (i.e., the embedding table). The corresponding embedding table occupies a large amount of memory space (each embedding occupies several KB, which can reach tens of GB to TB in total). The MLP layer typically occupies a smaller space (a few MB to a few hundred MB).

[0004] Because the size of the embedding table exceeds the memory capacity of resource-limited edge workers, training recommendation models on edge workers is often implemented using a hybrid parallel and parameter server (PS) architecture. In this PS architecture, a parameter server with large memory maintains globally shared embeddings, while edge workers cache portions of the embedding table in their local memory to accelerate embedding lookups. A copy of the MLP layer is stored on each edge worker's graphics processing unit (GPU), and gradients are synchronized using the AllReduce algorithm during training.

[0005] However, in production workloads, a single training example can involve up to thousands of embeddings. Despite using hybrid parallelization and the PS architecture, existing recommendation model training methods still suffer from an embedding access bottleneck. This is because each training iteration requires embedding transfer between worker nodes and the parameter server. Experiments show that embedding transfer often accounts for over 90% of model training time, indicating that embedding transfer is extremely expensive.

[0006] Currently, in order to reduce the embedding transmission cost, the input samples assigned to each working node are scheduled. That is, when the training process is for the current iteration, the input batch samples of the next iteration can be managed, and the embedding transmission cost can be reduced by scheduling each sample to the appropriate working node.

[0007] However, existing research efforts to reduce the embedding transmission cost are mainly focused on training recommendation models in cloud data centers. This approach is only applicable to multiple worker nodes in cloud data centers that can use high-speed connections at the same rate as the parameter server, but does not take into account the network heterogeneity and resource-constrained characteristics of edge worker nodes.

[0008] In summary, existing solutions for reducing the embedding transmission cost cannot solve the problem of excessive embedding transmission cost in edge working nodes, and will further lead to the inability to ensure that each scheduling decision is completed within the training iteration during the online training of the recommendation model, resulting in a corresponding decline in training performance. Summary of the Invention

[0009] The purpose of the present invention is to provide a method for accelerating the training of a recommendation model on an edge node, so as to solve the problem of excessively high embedding transmission cost of edge working nodes in the prior art.

[0010] The purpose of the present invention is achieved through the following technical solutions:

[0011] A method for accelerating the training of a recommendation model on an edge node, comprising:

[0012] For each input embedding sample, the expected transmission cost of each embedding dispatch to each edge worker node is calculated based on each edge worker node. After completing the calculation for the input embedding sample and each edge worker node, a total expected transmission cost is obtained. The total expected transmission cost includes the expected transmission cost of each embedded sample to each edge worker node.

[0013] Determining a scheduling decision for each edge working node based on the calculated total expected transmission cost, wherein the scheduling decision is determined based on a load balancing mechanism;

[0014] Each edge working node obtains the embedded samples required for training the recommendation model based on the scheduling decision.

[0015] The processing process of calculating the expected transmission cost of each embedding deployment and scheduling to each edge working node includes: if the embedding corresponding to the current embedding ID is not on the current edge working node, then the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node is increased; if the embedding corresponding to the current embedding ID exists on other edge working nodes, then the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node is increased.

[0016] The process of increasing the expected transmission cost of scheduling the embedded sample where the current embedded ID is located to the current edge working node includes:

[0017] The cost of transmitting the embedding from the edge worker node where the embedding corresponding to the current embedding ID is located to the parameter server is added to the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge worker node.

[0018] The process of determining the scheduling decision of each edge working node includes:

[0019] Based on the expected transmission costs of each embedded sample to each edge working node included in the total expected transmission cost, obtain the two smallest expected transmission costs of each embedded sample to each edge working node, and calculate the difference between the two smallest costs;

[0020] Obtaining a predetermined number of embedded samples in sequence according to the difference values ​​from large to small;

[0021] The predetermined number of embedded samples are solved using the optimal solution algorithm, and the embedded samples of the predetermined data are scheduled to different edge working nodes according to the solution results; for the remaining embedded samples, they are scheduled to a corresponding edge working node according to the current load status of each edge working node.

[0022] The process of sequentially acquiring a predetermined number of embedded samples includes:

[0023] Based on the set solution ratio parameter, the number of embedded samples of the set solution ratio parameter is selected from the total input embedded samples as the predetermined number of embedded samples, the solution ratio parameter is set according to the expected solution quality and resource consumption, and the value α of the solution ratio parameter is 1≥α≥0.

[0024] The solution ratio parameter may take the following values: 1, 0.5, 0.25, 0.125 or 0.

[0025] The process of dispatching each edge working node to a corresponding edge working node according to its current load condition includes:

[0026] For the remaining embedded samples, each embedded sample is scheduled to the edge working node with the smallest expected transmission cost among the edge working nodes corresponding to the current embedded sample. If the edge working node has reached the maximum load value, the edge working node with the second smallest expected transmission cost is selected until the scheduling processing of all the remaining embedded samples is completed.

[0027] The method further includes:

[0028] When training the current iteration, batch samples need to be input into the scheduling decision calculation process of the next iteration to calculate the corresponding scheduling decision for the next iteration.

[0029] Compared to existing technologies, the method provided by the present invention accelerates the training of recommendation models on edge nodes by calculating the transmission cost of input samples and executing a corresponding scheduling strategy based on the transmission cost calculation results. This approach addresses the challenges of limited computing resources at edge nodes and the competition for multiple tasks, providing a scheduling decision solution that balances solution quality and resource utilization. Furthermore, the implementation provided by the present invention was applied to an experimental platform equipped with eight GPUs and evaluated using three real-world workloads. The corresponding experimental results show that compared with existing technologies, the implementation provided by the present invention can achieve up to 1.74 times the training acceleration and reduce the embedding transmission cost by 36.76%. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of the architecture for training a recommendation model on an edge node according to an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of the implementation flow of the method provided in an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of the execution timing of the scheduling decision process provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of an algorithm for calculating expected embedding transmission cost provided by an embodiment of the present invention;

[0035] Figure 5 A schematic diagram of an algorithm for making scheduling decisions based on transmission cost provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the specific content of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] First, the following terms may be used in this article:

[0038] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y includes both “X” or “Y” and “X and Y”.

[0039] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0040] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0041] Unless otherwise specified or limited, the terms "mounted," "connected," "connect," and "fixed" should be interpreted broadly. For example, they can refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in this document based on specific circumstances.

[0042] When concentration, temperature, pressure, size or other parameters are expressed in the form of a numerical range, the numerical range should be understood to specifically disclose all ranges formed by the pairing of any upper limit, lower limit, or preferred value within the numerical range, regardless of whether the range is explicitly stated. For example, if a numerical range of "2 to 8" is stated, the numerical range should be interpreted as including ranges of "2 to 7," "2 to 6," "5 to 7," "3 to 4 and 6 to 7," "3 to 5 and 7," "2 and 5 to 7," etc. Unless otherwise specified, the numerical ranges stated herein include both their endpoints and all integers and fractions within the numerical range.

[0043] The terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as a limitation to this document.

[0044] In order to address the deficiencies in the prior art, the technical personnel of the present invention conducted a detailed study and analysis of the state of the art; the study and analysis found that the connections between multiple edge working nodes and parameter servers are usually heterogeneous, that is, the cost of transmitting the same embedding between different edge working nodes and parameter servers is not the same. Moreover, in the case of online training of recommendation models, it is necessary to ensure that user interest drift and emerging trends are reflected in a timely and accurate manner. That is, unlike offline training where decision time is not restricted, online training requires that each scheduling decision be completed within a training iteration to avoid performance degradation of the recommendation system. In addition, multiple tasks often compete for limited resources on edge working nodes. If they are not managed to minimize resource consumption on edge working nodes as much as possible, the edge working nodes will not work properly. In short, the present invention needs to comprehensively consider the characteristics of online training and the challenges brought by resource consumption on the basis of the above research and analysis, and then provide an implementation solution for a method to accelerate the training of recommendation models on edge nodes.

[0045] Based on the above research and analysis results, the present invention proposes a method for scheduling input samples to reduce the time spent on training recommendation models on multiple edge devices; specifically, it provides an input batch sample scheduling mechanism and a solution method that balances solution quality and resource occupancy, in order to reduce the time required to train recommendation models on edge nodes, thereby accelerating the training of recommendation models on edge nodes. Figure 1 The architecture implementation for training the recommendation model on the edge node shown is based on the PS architecture and hybrid parallel strategy. The edge worker nodes 1, 2, and 3 of this architecture include an embedding cache and an MLP respectively, and the parameter server side includes a global embedding, which includes an ID (embedding ID) and the embedding vector itself.

[0046] The present invention provides a method for accelerating the training of recommendation models on edge nodes (i.e., edge working nodes). It first needs to calculate the expected embedding transmission cost (referred to as the expected transmission cost) of scheduling each embedded input sample to different edge working nodes (hereinafter referred to as working nodes). Then, a scheduling decision is obtained through a solution method that balances solution quality and resource occupancy.

[0047] That is to say, in the specific implementation process of the present invention, a processing process for calculating the expected embedded transmission cost and a solution processing process for making scheduling decisions based on the transmission cost may include: for each input embedded sample, the expected transmission cost of each embedded deployment to each edge working node is calculated based on each edge working node, and after completing the calculation of the input embedded sample and each edge working node, the total expected transmission cost is obtained, and the total expected transmission cost includes the expected transmission cost of each embedded sample to each edge working node respectively; the solution processing process for making scheduling decisions based on the transmission cost may include: determining the scheduling decision of each edge working node according to the calculated total expected transmission cost, and the scheduling decision can be determined based on a load balancing mechanism. After determining the scheduling decision, each edge working node can obtain the embedded samples required for training the recommendation model based on the scheduling decision to train the corresponding recommendation model.

[0048] The following describes in detail the above-mentioned process of calculating the expected embedded transmission cost and the implementation process of solving the scheduling decision based on the transmission cost, wherein:

[0049] (1) A process for calculating the expected embedding transmission cost

[0050] Specifically, during this processing, for each input embedding sample, if the embedding corresponding to the current embedding ID in a certain embedding sample is not on the current edge working node, then the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node is increased; if the embedding corresponding to the current embedding ID exists on other edge working nodes, then the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node is increased; the corresponding processing of increasing the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node means: adding the cost of transmitting the embedding from the edge working node where the embedding corresponding to the current embedding ID is located to the parameter server to the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node, so as to update the expected transmission cost of scheduling the embedding sample where the embedding ID is located to the current edge working node;

[0051] Go further, refer to Figure 2 As shown, the processing process may specifically include the following processing steps:

[0052] Step 11, initialize all elements in the expected cost matrix to 0;

[0053] That is, initializing the expected cost matrix to initialize the unit embedding transmission cost from different working nodes to the parameter server, thereby achieving the initialization setting of the expected transmission cost in the expected cost matrix;

[0054] Step 12: Calculate the expected transmission cost of each input embedded sample to be scheduled to the current working node;

[0055] Specifically, based on the situation where each input sample contains multiple embedded IDs, the following processing process can be adopted for each working node:

[0056] If the embedding corresponding to an embedding ID in the input sample is not on the current working node, increase the expected transmission cost of scheduling the input embedding sample to the current working node;

[0057] If the embedding corresponding to the current embedding ID exists on other working nodes, the expected transmission cost of scheduling the input embedding sample to the current working node also needs to be increased;

[0058] Each input sample belongs to all embedded samples used by each edge worker node to train the recommendation model, which can be referred to as a sample. Each embedded sample usually contains multiple embedding IDs, and each embedding ID corresponds to one embedding;

[0059] Step 13: Repeat step 12 for all input embedded samples and working nodes to calculate the expected transmission cost of scheduling each input embedded sample to each working node, obtain all elements in the expected cost matrix to calculate the expected transmission cost, and fill the corresponding calculation results into the expected cost matrix to obtain the updated expected cost matrix.

[0060] It can be seen that the present invention provides an implementation scheme for calculating the expected transmission cost of scheduling each embedded sample to different edge working nodes, which can take into account the different costs of transmitting the same sample between different edge nodes and parameter servers, and effectively ensure that each sample can be transmitted at a lower transmission cost, so as to improve the transmission efficiency of the sample and further ensure the efficiency of training the recommendation model.

[0061] (2) A process for solving scheduling decisions based on transmission costs

[0062] In this processing process, first, the expected transmission cost of each embedded sample to each edge working node included in the total expected transmission cost can be obtained, and the two smallest expected transmission costs of each embedded sample to each edge working node (i.e., the smallest one and the second smallest one) are obtained, and the difference between the two smallest ones is calculated; then, a predetermined number of embedded samples can be obtained in descending order according to the differences; finally, the predetermined number of embedded samples are solved using an optimal solution algorithm, and the embedded samples of the predetermined data are scheduled to different edge working nodes based on the solution results; as for the remaining embedded samples, they can be scheduled to corresponding edge working nodes based on the current load status of each edge working node;

[0063] The process of sequentially acquiring a predetermined number of embedded samples may be, but is not limited to, based on a set ratio parameter, selecting the number of embedded samples with the set ratio parameter from the total input embedded samples as the predetermined number of embedded samples.

[0064] Further, still referring to Figure 2 As shown, the processing process may specifically include the following processing steps:

[0065] Step 14: using the expected cost matrix obtained by the above method for calculating the expected embedded transmission cost as input, initializing the load value of each working node in the expected cost matrix to 0 to initialize the maximum load value;

[0066] Step 15, in the expected cost matrix, calculate the result obtained by subtracting the minimum value from the next minimum value of each row;

[0067] Step 16: re-sort each row in the expected cost matrix in descending order based on the result obtained in step 15 above;

[0068] Step 17: Solve the first several rows (i.e., a portion of the rows, for example, the first 50% of the rows) of the reordered expected cost matrix using an optimal solution algorithm. Based on the solution results, each corresponding sample can be scheduled to a different work node, and the load value of each work node needs to be increased and adjusted based on the solution results.

[0069] Step 18: For the remaining rows, the corresponding embedded samples are scheduled to the working nodes in the current row with the lowest transmission cost and which have not reached the maximum load value;

[0070] That is, for the remaining rows in the expected cost matrix, the sample is scheduled to the work node with the lowest expected transmission cost in the current row. If it is determined based on the information in the expected cost matrix that the work node has reached the maximum load value, the work node with the second lowest expected transmission cost can be selected, and so on, until the scheduling processing of the corresponding embedded samples in all the remaining rows is completed;

[0071] Through the processing of this step, it is possible to schedule each of the remaining embedded samples to the edge working node with the lowest expected transmission cost among the edge working nodes corresponding to the current embedded sample. Moreover, if the edge working node has reached the maximum load value, the edge working node with the second lowest expected transmission cost can be selected until the scheduling processing of all the remaining embedded samples is completed and the execution of the corresponding scheduling strategy is completed.

[0072] It can be seen from the above processing that compared with the existing technology, the implementation scheme provided by the present invention for accelerating the training of recommendation models on edge nodes can reduce the transmission cost of embedded samples and improve the timeliness of training recommendation models on edge working nodes by reasonably scheduling input samples.

[0073] In the process of executing the above-mentioned method for solving scheduling decisions based on transmission cost, the specific timing for calculating the corresponding scheduling decision can be referred to Figure 3 Specifically, when the GPU is processing the current iteration (Iteration) I t During training, you can set the next iteration I as needed t+1 Input batch samples of IBES t+1 The premise of adopting this scheduling decision execution method is that the data loader to be used can be used when the GPU is processing I t Provide the next iteration I when training t+1 Input batch samples of IBES t+1 This scheduling decision execution method can effectively improve the efficiency of training recommendation models on edge nodes.

[0074] It can be seen that the present invention provides a scheduling decision solution that can balance the solution quality and resource occupancy for the situation where the computing resources of edge nodes are limited and multiple tasks compete for resources. It can therefore reasonably schedule and apply limited resources based on the expected transmission cost obtained by the aforementioned method, effectively improving the efficiency of training the recommendation model.

[0075] To further validate the effectiveness of the proposed technical solution, we applied it to an experimental platform equipped with eight GPUs and conducted an evaluation using three real-world workloads. The results demonstrate that, compared to existing methods, the proposed technical solution can accelerate recommendation model training by up to 1.74 times and reduce embedding transmission costs by 36.76%.

[0076] To facilitate further understanding of the present invention, the method for accelerating the training of a recommendation model on an edge node provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0077] (1) The process of calculating the expected embedding transmission cost

[0078] Reference Figure 4 As shown, the corresponding process of calculating the expected embedding transmission cost may include:

[0079] Step 21: The cost matrix (i.e., expected cost matrix) All elements in (mn rows, n columns) are initialized to 0;

[0080] Assuming that the input batch sample ε, the total number of samples is mn, m is the maximum load value of each working node, n means there are n working nodes in total, for one of the embedded samples E i , which can consist of multiple embeddings and corresponding multiple embedding IDs (x i )composition;

[0081] Step 22, initialize the cost matrix According to its various working nodes and each sample E i Update the expected embedding transmission cost for each embedding ID;

[0082] That is, for the working node w j , for E i For each embedded ID in , the corresponding processing process may include:

[0083] If its corresponding embedding is not in the working node w j The expected transmission cost is in is the working node w j The cost of transmitting the embedding to the parameter server is the expected transmission cost of scheduling the corresponding embedding sample to the worker node Make updates;

[0084] If its corresponding embedding exists in another working node w j ′, then the expected transmission cost in is node wj ′ is the cost of transmitting the embedding to the parameter server to Make updates;

[0085] In the above process, the cost of transmitting the embedding between the corresponding working node and the parameter server can be determined in real time or pre-set based on performance parameters such as the bandwidth between the two;

[0086] Step 23, for the embedded sample E i All embedded IDs in execute the above step 22 respectively to obtain the corresponding embedded samples E i To the working node w j The expected transmission cost And fill it into the cost matrix The i row and j column in ;

[0087] Step 24, execute steps 22 and 23 for all embedded samples in the input batch sample ε, and obtain the updated cost matrix after execution.

[0088] (2) Scheduling Decision-Making Process Based on Expected Transmission Cost

[0089] Reference Figure 5 As shown, the corresponding process of making a scheduling decision based on the expected transmission cost may include:

[0090] Step 25: Based on the cost matrix obtained in step 24 above (mn rows, n columns) is used as input, the load value of each node is initialized to 0, the maximum load value of each node is m, the solution scale parameter (referred to as scale parameter) α can be set according to the desired solution quality and resource consumption. The larger α is, the better the solution quality and the more resource consumption is. α is 1 ≥ α ≥ 0. Usually, the values ​​that can be used include: 1, 0.5, 0.25, 0.125, 0, etc.

[0091] Step 26, in the cost matrix Calculate the result of subtracting the minimum value from the next minimum value in each row;

[0092] Step 27, in the cost matrix In step 26, sort the mn rows in descending order.

[0093] Step 28: sort the cost matrix 0 to The optimal solution algorithm (such as Hungarian algorithm) is used to solve the problem. The solution result is the scheduling strategy for scheduling each sample to a different working node. Before solving, copy each column to List;

[0094] Step 29: Based on the solution, increase the current load value of each working node to ;

[0095] Step 210: For each remaining row in the cost matrix, select the work node with the smallest expected transmission cost in the current row to schedule the sample. If the work node has reached the maximum load value, select the work node with the second smallest expected transmission cost and increase the corresponding work node load value, and so on, until the processing of each remaining row in the cost matrix is ​​completed.

[0096] In summary, it can be seen that the above-mentioned expected transmission cost calculation method and the corresponding scheduling decision strategy can reduce the embedded transmission cost while effectively accelerating the process of training the recommendation model on the edge node, so that the above-mentioned corresponding technical solution can well achieve the purpose of accelerating the training of the recommendation model on the edge node.

[0097] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A method for accelerating the training of a recommendation model on an edge node, characterized in that: include: For each input embedding sample, the expected transmission cost of each embedding dispatch to each edge worker node is calculated based on each edge worker node. After completing the calculation for the input embedding sample and each edge worker node, a total expected transmission cost is obtained. The total expected transmission cost includes the expected transmission cost of each embedded sample to each edge worker node. Determining a scheduling decision for each edge working node based on the calculated total expected transmission cost, wherein the scheduling decision is determined based on a load balancing mechanism; Each edge working node obtains the embedded samples required for training the recommendation model based on the scheduling decision.

2. The method according to claim 1, characterized in that The process of calculating the expected transmission cost of each embedded deployment schedule to each edge working node includes: If the embedding corresponding to the current embedding ID is not on the current edge working node, then the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node is increased. If the embedding corresponding to the current embedding ID exists on other edge working nodes, then the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge working node is increased.

3. The method according to claim 2, characterized in that The process of increasing the expected transmission cost of scheduling the embedded sample where the current embedded ID is located to the current edge working node includes: The cost of transmitting the embedding from the edge worker node where the embedding corresponding to the current embedding ID is located to the parameter server is added to the expected transmission cost of scheduling the embedding sample where the current embedding ID is located to the current edge worker node.

4. The method according to claim 1, 2 or 3, characterized in that The process of determining the scheduling decision of each edge working node includes: Based on the expected transmission costs of each embedded sample to each edge working node included in the total expected transmission cost, obtain the two smallest expected transmission costs of each embedded sample to each edge working node, and calculate the difference between the two smallest costs; Obtaining a predetermined number of embedded samples in order from large to small according to the difference values; The predetermined number of embedded samples are solved using the optimal solution algorithm, and the predetermined number of embedded samples are scheduled to different edge working nodes according to the solution results; for the remaining embedded samples, they are scheduled to a corresponding edge working node according to the current load status of each edge working node.

5. The method according to claim 4, characterized in that The process of sequentially acquiring a predetermined number of embedded samples includes: Based on the set solution ratio parameter, the number of embedded samples of the set solution ratio parameter is selected from the total input embedded samples as the predetermined number of embedded samples, the solution ratio parameter is set according to the expected solution quality and resource consumption, and the value α of the solution ratio parameter is 1≥α≥0.

6. The method according to claim 5, characterized in that The solution ratio parameter may take the following values: 1, 0.5, 0.25, 0.125 or 0.

7. The method according to claim 4, characterized in that The process of dispatching each edge working node to a corresponding edge working node according to its current load condition includes: For the remaining embedded samples, each embedded sample is scheduled to the edge working node with the smallest expected transmission cost among the edge working nodes corresponding to the current embedded sample. If the edge working node has reached the maximum load value, the edge working node with the second smallest expected transmission cost is selected until the scheduling processing of all the remaining embedded samples is completed.

8. The method according to claim 4, characterized in that The method further includes: When training the current iteration, batch samples need to be input into the scheduling decision calculation process of the next iteration to calculate the corresponding scheduling decision for the next iteration.

Images