A method and device for processing resource quota of PS node

Abstract

An embodiment of the present invention provides a method and apparatus for processing resource quotas for a PS node, wherein the method comprises: obtaining the resource utilization rate, resource quota rating, and current resource quota value of the current PS node in a PS cluster; when the resource utilization rate is less than a lower resource utilization threshold, and the resource quota rating is less than the current resource quota value, scaling down the current resource quota value; and when the resource utilization rate is greater than an upper resource utilization threshold, and the resource quota rating is greater than or equal to the current resource quota value, scaling up the current resource quota value. The embodiment of the present invention not only achieves dynamic adjustment of the resource quotas of PS nodes, but also avoids simply increasing or decreasing the number of PS nodes. Instead, it increases or decreases the resource quotas of PS nodes, thereby fully utilizing system resources.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method and device for processing resource quotas of a PS node. Background Art

[0002] With the increase in data scale and the increase in the parameter scale of machine learning models, the rapid convergence of the model is challenged. The parameter server (PS) distributed training architecture (hereinafter referred to as the PS architecture) is widely used to accelerate the convergence of the model.

[0003] In the PS architecture, model parameters are sharded and distributed to different PS nodes, which are responsible for storing and updating the sharded parameters. Training data is sharded and distributed to different Worker nodes, which perform data-parallel training, calculating the gradients used to update the model parameters and reporting them to the PS nodes. After receiving the gradients, the PS nodes update the model parameters.

[0004] Most current technical solutions use static PS resource configuration. This means that a resource configuration is selected before model training begins, and the model is trained using PS nodes according to the selected resource configuration until the training task is completed. Some technical solutions also support elastic scaling of PS nodes. Specifically, this is achieved by elastically increasing or decreasing the number of PS nodes with the primary goal of shortening training task completion time.

[0005] Neither static PS resource configuration nor horizontal elastic scaling to minimize training task completion time fully utilizes system resources, resulting in low system resource utilization. Summary of the Invention

[0006] In view of the above problems, embodiments of the present invention are proposed to provide a method and apparatus for processing resource quota of a PS node that overcomes the above problems or at least partially solves the above problems.

[0007] In order to solve the above problems, according to the first aspect of an embodiment of the present invention, a resource quota processing method for a PS node is disclosed, including: obtaining the resource utilization, resource quota rated value and resource quota current value of the current PS node in the PS cluster, wherein the current PS node is used to store and update the parameters of the model; when the resource utilization is less than a preset resource utilization lower limit threshold, and the resource quota rated value is less than the resource quota current value, scaling down the resource quota current value; when the resource utilization is greater than a preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, scaling up the resource quota current value.

[0008] Optionally, when the resource utilization is greater than a preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, the method further includes: when there are no spare resources, if the resource utilization is less than the resource utilization rated threshold, then prohibiting the expansion of the resource quota current value, and continuing to train the model.

[0009] Optionally, when the resource utilization is greater than a preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, the method further includes: when there are no spare resources, if the resource utilization is greater than or equal to the resource utilization rated threshold, saving the training progress and parameters of the model and terminating the training of the model.

[0010] Optionally, the method further includes: when the resource utilization is less than the resource utilization lower threshold, and the resource quota rated value is greater than or equal to the resource quota current value, prohibiting scaling down the resource quota current value.

[0011] Optionally, the method further includes: when the resource utilization is greater than the resource utilization upper limit threshold, and the resource quota rated value is less than or equal to the resource quota current value, prohibiting expansion of the resource quota current value.

[0012] Optionally, before obtaining the resource utilization, resource quota rated value and resource quota current value of the current PS node in the PS cluster, the method also includes: before the training task of the model is started, if there is historical training record data of the model, estimating the resource quota data of the training task of the model at the current PS node based on the historical training record data.

[0013] Optionally, obtaining the resource quota rated value includes: fitting a resource occupancy growth curve based on the resource occupancy during the training phase of the model; estimating the completion time point of the training task of the model based on the average training time of a single sample data of the model and the number of sample data; and obtaining the resource quota rated value based on the resource occupancy growth curve and the completion time point.

[0014] According to the second aspect of an embodiment of the present invention, a resource quota processing device for a PS node is also disclosed, including: an acquisition module, used to obtain the resource utilization, resource quota rated value and resource quota current value of the current PS node in the PS cluster, and the current PS node is used to store and update the parameters of the model; a shrinking module, used to shrink the current value of the resource quota when the resource utilization is less than a preset lower limit threshold of resource utilization and the resource quota rated value is less than the current value of the resource quota; an expansion module, used to expand the current value of the resource quota when the resource utilization is greater than a preset upper limit threshold of resource utilization and the resource quota rated value is greater than or equal to the current value of the resource quota.

[0015] Optionally, the device also includes: a maintenance module, which is used to prohibit expansion of the current value of the resource quota and continue training the model when the resource utilization is greater than a preset resource utilization upper limit threshold and the resource quota rated value is greater than or equal to the current value of the resource quota, if there are no spare resources and the resource utilization is less than the resource utilization rated threshold.

[0016] Optionally, the device also includes: a termination module, which is used to save the training progress and parameters of the model and terminate the training of the model when the resource utilization is greater than a preset resource utilization upper limit threshold and the resource quota rated value is greater than or equal to the resource quota current value, if there are no spare resources and the resource utilization is greater than or equal to the resource utilization rated threshold.

[0017] Optionally, the device further includes: a prohibition module, configured to prohibit scaling down the current value of the resource quota when the resource utilization is less than the lower limit threshold of the resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota.

[0018] Optionally, the prohibition module is further configured to prohibit expansion of the current value of the resource quota when the resource utilization is greater than the resource utilization upper limit threshold and the rated value of the resource quota is less than or equal to the current value of the resource quota.

[0019] Optionally, the device also includes: an estimation module, which is used to estimate the resource quota data of the training task of the model at the current PS node based on the historical training record data if there is historical training record data of the model before the acquisition module acquires the resource utilization, resource quota rated value and resource quota current value of the current PS node in the PS cluster and before the training task of the model is started.

[0020] Optionally, the acquisition module includes: a curve fitting module, used to fit a resource occupancy growth curve according to the resource occupancy during the training phase of the model; a time point calculation module, used to estimate the completion time point of the training task of the model based on the average training time of a single sample data of the model and the number of sample data; and a rated value acquisition module, used to obtain the resource quota rated value based on the resource occupancy growth curve and the completion time point.

[0021] Compared with the prior art, the technical solution provided by the embodiment of the present invention has the following advantages:

[0022] The embodiment of the present invention provides a resource quota processing solution for PS nodes, which obtains the resource utilization rate, resource quota rated value and resource quota current value of the current PS node in the PS cluster. When the resource utilization rate is less than the preset resource utilization lower limit threshold, and the resource quota rated value is less than the resource quota current value, the resource quota current value is scaled down; when the resource utilization rate is greater than the preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, the resource quota current value is scaled up. The embodiment of the present invention monitors the resource utilization rate of the PS node in real time, compares the resource utilization rate with the resource utilization lower limit threshold and the resource utilization upper limit threshold, and compares the resource quota rated value with the resource quota current value, and finally scales down or scales up the resource quota current value according to the comparison result. The embodiment of the present invention not only realizes the dynamic adjustment of the resource quota of the PS node, but also avoids simply increasing or decreasing the number of PS nodes, but increases or decreases the resource quota of the PS node, making full use of system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flowchart of a method for processing resource quotas of a PS node according to an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of a resource occupancy growth curve according to an embodiment of the present invention;

[0025] Figure 3 This is a flow chart of an automatic elastic expansion and contraction solution for PS node resource quota under a PS architecture according to an embodiment of the present invention;

[0026] Figure 4 It is a structural block diagram of a resource quota processing device for a PS node according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Reference Figure 1 , shows a flowchart of a method for processing resource quotas of a PS node according to an embodiment of the present invention. The method for processing resource quotas of a PS node may specifically include the following steps:

[0029] Step 101: Obtain resource utilization, resource quota rated value, and resource quota current value of the current PS node in the PS cluster.

[0030] In an embodiment of the present invention, the PS cluster is regarded as a resource pool, and the training task of a model occupies different resource quotas on different PS nodes of the PS cluster, and each model training task shares the resources of each PS node. The resource allocation problem of the PS node is modeled as a bin-packing problem. In the bin-packing problem, the number of bins is equal to the number N of PS nodes in the PS cluster, and the capacity of each bin is equal to the total amount of resources of the corresponding PS node. The resource quota of a model training task on N PS nodes corresponds to the volume of the N items placed in each bin. It should be noted that, unlike the traditional bin-packing problem, the volume of these N items will still change after being placed in the bin, that is, the resource quota of a model training task on N PS nodes will change dynamically with the adjustment instructions of the elastic scaling algorithm.

[0031] In the embodiment of the present invention, the resource quota rating of the current PS node is estimated in advance. How to estimate the resource quota rating of the current PS node will be described in detail later.

[0032] Step 102: When the resource utilization is less than a preset lower threshold of resource utilization and the rated value of the resource quota is less than the current value of the resource quota, the current value of the resource quota is scaled down.

[0033] In an embodiment of the present invention, a resource utilization lower threshold (low_bound) may be preset. The resource utilization is compared with the resource utilization lower threshold, and the resource quota rated value is compared with the resource quota current value. If the resource utilization is less than the resource utilization lower threshold, and the resource quota rated value is less than the resource quota current value, the current resource quota value of the current PS node is scaled down. It should be noted that the scaling down process can be used to elastically reduce the resource quota of the current PS node.

[0034] Step 103: When the resource utilization is greater than a preset upper threshold of resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota, the current value of the resource quota is expanded.

[0035] In an embodiment of the present invention, a resource utilization upper threshold (upper_bound) may be pre-set. The resource utilization is compared with the resource utilization upper threshold, and the resource quota rated value is compared with the resource quota current value. If the resource utilization is greater than the resource utilization upper threshold, and the resource quota rated value is greater than or equal to the resource quota current value, the current resource quota value of the current PS node is expanded. It should be noted that the expansion process can flexibly increase the resource quota of the current PS node.

[0036] The embodiment of the present invention provides a resource quota processing solution for PS nodes, which obtains the resource utilization rate, resource quota rated value and resource quota current value of the current PS node in the PS cluster. When the resource utilization rate is less than the preset resource utilization lower limit threshold, and the resource quota rated value is less than the resource quota current value, the resource quota current value is scaled down; when the resource utilization rate is greater than the preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, the resource quota current value is scaled up. The embodiment of the present invention monitors the resource utilization rate of the PS node in real time, compares the resource utilization rate with the resource utilization lower limit threshold and the resource utilization upper limit threshold, and compares the resource quota rated value with the resource quota current value, and finally scales down or scales up the resource quota current value according to the comparison result. The embodiment of the present invention not only realizes the dynamic adjustment of the resource quota of the PS node, but also avoids simply increasing or decreasing the number of PS nodes, but increases or decreases the resource quota of the PS node, making full use of system resources.

[0037] In a preferred embodiment of the present invention, when the resource utilization rate is greater than a preset upper limit threshold of resource utilization, and the resource quota rated value is greater than or equal to the current value of the resource quota, an implementation method for expanding the current value of the resource quota is that not only the conditions that the resource utilization rate is greater than the preset upper limit threshold of resource utilization, and the resource quota rated value is greater than or equal to the current value of the resource quota are met, but also the condition that there are free resources in the current PS node is met before the current value of the resource quota can be expanded.

[0038] In a preferred embodiment of the present invention, when the resource utilization is greater than a preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, if there are no spare resources in the current PS node and the resource utilization is less than the resource utilization rated threshold, then the expansion of the resource quota current value is prohibited, and the model training continues.

[0039] In a preferred embodiment of the present invention, when the resource utilization is greater than a preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, if there are no spare resources of the current PS node, and the resource utilization is greater than or equal to the resource utilization rated threshold, indicating that the current value of the resource quota of the current PS node is insufficient to support the continued training of the model, the training progress and parameters of the model are saved, and the training of the model is terminated.

[0040] In actual applications, the resource utilization rate threshold can be set to 100% according to actual application conditions. Moreover, the resource utilization rate threshold is greater than the resource utilization rate upper limit threshold.

[0041] In a preferred embodiment of the present invention, when the resource utilization rate is less than the lower resource utilization threshold and the rated resource quota value is greater than or equal to the current resource quota value, scaling down the current resource quota value is prohibited. This is because if the estimated rated resource quota value is greater than or equal to the current resource quota value, the model training task may trigger scaling up the current resource quota value later.

[0042] In a preferred embodiment of the present invention, when the resource utilization rate exceeds the upper resource utilization threshold and the rated resource quota value is less than or equal to the current resource quota value, expansion of the current resource quota value is prohibited. This is because the estimated rated resource quota value is less than or equal to the current resource quota value, indicating that the model training task can be successfully completed.

[0043] In a preferred embodiment of the present invention, before obtaining the resource utilization, resource quota rated value and resource quota current value of the current PS node in the PS cluster, and before starting the training task of the model, it is determined whether there is historical training record data of the model (including record data of successful training tasks and record data of failed training tasks). If there is historical training record data of the model, the resource quota data of the training task of the model at the current PS node is estimated based on the historical training record data, and the training task of the model is started based on the resource quota data. If there is no historical training record data of the model, the resource quota data required for the training task of the model to proceed stably and complete is estimated through the definition file of the model, and the training task of the model is started based on the resource quota data.

[0044] In a preferred embodiment of the present invention, an implementation method for obtaining the resource quota rated value is to fit a resource occupancy growth curve based on the resource occupancy during the training phase of the model; estimate the completion time point of the model training task based on the average training time of a single sample data of the model and the number of sample data; and obtain the resource quota rated value based on the resource occupancy growth curve and the completion time point.

[0045] In practical applications, during model training, the number of model parameters increases, and the resource usage of the PS node also increases. This resource usage increase can be divided into two phases: the loading phase and the training phase. This embodiment of the present invention primarily fits a resource usage growth curve for the resource usage during the training phase.

[0046] Reference Figure 2 , which shows a schematic diagram of a resource occupancy growth curve according to an embodiment of the present invention. Figure 2 In the example, the horizontal axis represents each time point in the model training process, and the vertical axis represents the resource usage of the PS node. In the embodiment of the present invention, resources include but are not limited to memory resources, CPU resources, GPU resources, etc.

[0047] When estimating the completion time of a model training task, you can multiply the average training time of a single sample data item by the number of sample data items to obtain the total training time of the model training task. Then, use the training start time of the first sample data item as the starting point, or use the training task start time as the starting point, and use the time after the total training time as the completion time of the training task. For example, if the average training time of a single sample data item is 3 minutes, the training task starts at 05-29 00:00, and there are 5 sample data items, the completion time is 05-29 00:15.

[0048] When obtaining the resource quota rating based on the resource usage growth curve and the completion time, the resource usage at the completion time in the resource usage growth curve can be used as the resource quota rating. For example, if the completion time is 00:15 on May 29 and the resource usage at 00:15 on May 29 is 4 GB, then the resource quota rating is 4 GB.

[0049] Based on the above description of an embodiment of a method for processing resource quotas of a PS node, the following introduces an automatic elastic scaling solution for resource quotas of PS nodes under a PS architecture.

[0050] Reference Figure 3 , shows a flow chart of an automatic elastic scaling solution for PS node resource quotas under a PS architecture according to an embodiment of the present invention. The automatic elastic scaling solution for PS node resource quotas under a PS architecture mainly includes a monitoring module, an expansion module, a contraction module, and a startup module.

[0051] The training data or sample data is sharded and distributed to different Worker nodes. Different Worker nodes perform parallel training on the training data or sample data, calculate the gradients used to update the model parameters and report them to the PS node. At the same time, the Worker node writes the model training progress to the database.

[0052] The monitoring module obtains resource utilization from the PS nodes in the PS cluster and writes it to the database. The monitoring module determines whether resource utilization exceeds the upper threshold or is lower than the lower threshold. If resource utilization exceeds the upper threshold, the expansion module is notified to perform capacity expansion. If resource utilization is lower than the lower threshold, the reduction module is notified to perform capacity reduction.

[0053] The scaling module scales down the resource quota when the estimated ideal resource quota rating after model training is less than the current resource quota value. If resource utilization is less than the lower resource utilization threshold, but the estimated ideal resource quota rating is greater than or equal to the current resource quota value, scaling down is not performed and expansion may be triggered later.

[0054] When the estimated ideal resource quota rating for model training is greater than or equal to the current resource quota value, and if there are spare resources in the PS cluster, the expansion module calculates the available resource quota based on the estimated ideal resource quota rating and the spare resources in the PS cluster, and performs expansion based on the calculated resource quota. If there are no spare resources in the PS cluster, the module determines whether resource utilization has reached 100%. If resource utilization is less than 100%, the expansion process is abandoned and model training continues. If resource utilization reaches 100%, indicating that the current resource quota value is insufficient to support continued model training, the model training progress is saved and the startup module is notified to restart the training task.

[0055] Before starting a model training task (including new tasks and restarted old tasks), the startup module first determines whether historical training data for the module exists. If so, it estimates the rated resource quota based on the historical training data and starts the task accordingly. If no historical training data exists, it estimates the initial resource quota based on the model definition file and starts the task accordingly. This reduces the probability of resource scaling after model training begins.

[0056] When a model training task is initiated, embodiments of the present invention estimate resource quota ratings based on historical training records and train the model accordingly, reducing the probability of elastic resource scaling during model training. During model training, resource utilization of PS nodes is monitored and elastic scaling is performed on the current resource quota values ​​of PS nodes, improving the stability and resource utilization of model training tasks.

[0057] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0058] Reference Figure 4 , shows a structural block diagram of a resource quota processing device for a PS node according to an embodiment of the present invention. The resource quota processing device for a PS node may specifically include the following modules:

[0059] An acquisition module 41 is configured to acquire the resource utilization, resource quota rated value, and resource quota current value of a current PS node in the PS cluster, wherein the current PS node is used to store and update model parameters;

[0060] A shrinking module 42 is configured to shrink the current value of the resource quota when the resource utilization is less than a preset lower threshold of the resource utilization and the rated value of the resource quota is less than the current value of the resource quota;

[0061] The expansion module 43 is configured to expand the current value of the resource quota when the resource utilization is greater than a preset upper threshold of the resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota.

[0062] In a preferred embodiment of the present invention, the device further comprises:

[0063] A maintenance module is used to prohibit expansion of the current value of the resource quota and continue training the model when the resource utilization is greater than a preset upper limit threshold of resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota, if there are no spare resources and the resource utilization is less than the rated threshold of resource utilization.

[0064] In a preferred embodiment of the present invention, the device further comprises:

[0065] The termination module is used to save the training progress and parameters of the model and terminate the training of the model when the resource utilization is greater than the preset resource utilization upper limit threshold and the resource quota rated value is greater than or equal to the resource quota current value, if there are no free resources and the resource utilization is greater than or equal to the resource utilization rated threshold.

[0066] In a preferred embodiment of the present invention, the device further comprises:

[0067] The prohibition module is configured to prohibit scaling down the current value of the resource quota when the resource utilization is less than the lower threshold value of the resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota.

[0068] In a preferred embodiment of the present invention, the prohibition module is further used to prohibit expansion of the current value of the resource quota when the resource utilization is greater than the resource utilization upper limit threshold and the resource quota rated value is less than or equal to the current value of the resource quota.

[0069] In a preferred embodiment of the present invention, the device further comprises:

[0070] An estimation module is used to estimate the resource quota data of the training task of the model at the current PS node based on the historical training record data if there is historical training record data of the model before the acquisition module 41 acquires the resource utilization, resource quota rated value and resource quota current value of the current PS node in the PS cluster and before the training task of the model is started.

[0071] In a preferred embodiment of the present invention, the acquisition module 41 includes:

[0072] A curve fitting module, configured to fit a resource occupancy growth curve according to the resource occupancy during the training phase of the model;

[0073] A time point calculation module is used to estimate the completion time point of the training task of the model based on the average training time of a single sample data of the model and the number of sample data;

[0074] A rated value acquisition module is used to acquire the resource quota rated value according to the resource occupancy growth curve and the completion time point.

[0075] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0076] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0077] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0081] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0082] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0083] The above is a detailed introduction to the resource quota processing method and device for a PS node provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for processing resource quotas of a PS node, characterized in that: include: Obtaining a resource utilization rate, a resource quota rated value, and a resource quota current value of a current PS node in the PS cluster, wherein the current PS node is used to store and update model parameters, wherein obtaining the resource quota rated value includes: Fitting a resource occupancy growth curve according to the resource occupancy during the training phase of the model; Estimate the completion time of the model training task based on the average training time of a single sample data of the model and the number of sample data; Obtaining the resource quota rated value according to the resource occupancy growth curve and the completion time point; When the resource utilization rate is less than a preset resource utilization rate lower threshold, and the resource quota rated value is less than the resource quota current value, scaling down the resource quota current value; When the resource utilization rate is greater than a preset resource utilization rate upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, expanding the resource quota current value; Before obtaining the resource utilization rate, the resource quota rated value, and the resource quota current value of the current PS node in the PS cluster, the method further includes: Before the training task of the model is started, if there is historical training record data of the model, the resource quota data of the training task of the model on the current PS node is estimated based on the historical training record data.

2. The method according to claim 1, characterized in that When the resource utilization is greater than the preset resource utilization upper limit threshold, and the resource quota rated value is greater than or equal to the resource quota current value, the method also includes: when there are no spare resources, if the resource utilization is less than the resource utilization rated threshold, then the expansion of the resource quota current value is prohibited, and the model training continues.

3. The method according to claim 1, characterized in that When the resource utilization is greater than the preset resource utilization upper limit threshold and the resource quota rated value is greater than or equal to the resource quota current value, the method also includes: when there are no spare resources, if the resource utilization is greater than or equal to the resource utilization rated threshold, saving the training progress and parameters of the model and terminating the training of the model.

4. The method according to claim 1, wherein The method further comprises: When the resource utilization is less than the resource utilization lower threshold, and the resource quota rated value is greater than or equal to the resource quota current value, scaling down the resource quota current value is prohibited.

5. The method according to claim 1, wherein The method further comprises: When the resource utilization is greater than the resource utilization upper limit threshold, and the resource quota rated value is less than or equal to the resource quota current value, expansion processing of the resource quota current value is prohibited.

6. A resource quota processing device for a PS node, characterized in that: include: An acquisition module is used to obtain the resource utilization, resource quota rated value, and resource quota current value of the current PS node in the PS cluster, wherein the current PS node is used to store and update the parameters of the model. The acquisition module includes: A curve fitting module, configured to fit a resource occupancy growth curve according to the resource occupancy during the training phase of the model; A time point calculation module is used to estimate the completion time point of the training task of the model based on the average training time of a single sample data of the model and the number of sample data; A rated value acquisition module, configured to acquire the resource quota rated value according to the resource occupancy growth curve and the completion time point; a shrinking module, configured to shrink the current value of the resource quota when the resource utilization is less than a preset lower threshold of the resource utilization and the rated value of the resource quota is less than the current value of the resource quota; an expansion module, configured to expand the current value of the resource quota when the resource utilization is greater than a preset upper threshold value of the resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota; An estimation module is used to estimate the resource quota data of the training task of the model at the current PS node based on the historical training record data if there is historical training record data of the model before the acquisition module acquires the resource utilization, resource quota rated value and resource quota current value of the current PS node in the PS cluster and before the training task of the model is started.

7. The device according to claim 6, characterized in that The device further comprises: A maintenance module is used to prohibit expansion of the current value of the resource quota and continue training the model when the resource utilization is greater than a preset upper limit threshold of resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota, if there are no spare resources and the resource utilization is less than the rated threshold of resource utilization.

8. The device according to claim 6, characterized in that The device further comprises: The termination module is used to save the training progress and parameters of the model and terminate the training of the model when the resource utilization is greater than the preset resource utilization upper limit threshold and the resource quota rated value is greater than or equal to the resource quota current value, if there are no free resources and the resource utilization is greater than or equal to the resource utilization rated threshold.

9. The device according to claim 6, characterized in that The device further comprises: The prohibition module is configured to prohibit scaling down the current value of the resource quota when the resource utilization is less than the lower threshold value of the resource utilization and the rated value of the resource quota is greater than or equal to the current value of the resource quota.

10. The device according to claim 9, characterized in that The prohibition module is further configured to prohibit expansion of the current value of the resource quota when the resource utilization is greater than the resource utilization upper limit threshold and the rated value of the resource quota is less than or equal to the current value of the resource quota.

Images