Model management resource pool dynamic allocation method and device, server and medium

Abstract

The application discloses a model management resource pool dynamic allocation method and device, a server and a medium, and belongs to the field of large models. The method comprises the following steps: calculating a plurality of sub-tasks of a to-be-trained model, and obtaining parameters of each sub-task; determining the time length adaptability of each preset container according to the parameters of the sub-tasks; calculating a first sequence set of container types and corresponding quantities that are adapted to each sub-task according to the time length adaptability of each preset container and the parameters of each sub-task; selecting a second sequence set that meets the training accuracy requirement from the first sequence set; calculating the use efficiency of each container in the second sequence set, wherein the use efficiency is a comprehensive parameter calculated by using hardware utilization rate and waiting time length; obtaining an optimal sorting combination according to the use efficiency of each container, and allocating training tasks to the containers according to the optimal sorting combination. The server can use as few container resources and as short time length as possible to quickly and efficiently complete model training.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a method, apparatus, server and medium for dynamically allocating model management resource pools. Background Technology

[0002] As research in big data and machine learning continues to deepen, a large amount of data is typically needed as a training set to improve the effectiveness of machine learning. This results in training data and model parameters becoming too large for a single machine to handle in large-scale machine learning training scenarios. Therefore, distributed machine learning has emerged.

[0003] Container technology not only enables rapid deployment of container clusters but also provides a lightweight solution for effectively integrating and managing bare-metal resources. It ensures consistency across different devices, provides resource isolation for application runtime environments, abstracts away the complexity of underlying hardware and node management, and supports CPU and GPU scheduling. Using container technology, distributed machine learning can be deployed across multiple containers, breaking down training tasks into smaller tasks and distributing them across multiple containers for training. Distributed machine learning not only distributes training tasks across the computing resources of multiple containers but also distributes data (including training data and intermediate results) across the storage of different containers. The effectiveness of distributed machine learning depends on container compatibility. In current technologies, when deploying distributed machine learning training on container cloud platforms, the task to be trained is often first decomposed into several sub-tasks, and then a corresponding container is created for each sub-task to ensure that the containers are matched to the sub-tasks as closely as possible, thereby improving container utilization efficiency.

[0004] However, in some small servers, the allocated containers often need to support the training of multiple models. It's impossible to recreate containers for each specific model training task. Furthermore, existing containers may also need to provide some services externally, making it impossible to create matching containers based on model training requirements. This hinders the improvement of container utilization efficiency. Summary of the Invention

[0005] This invention provides a method, apparatus, server, and medium for dynamically allocating model management resource pools to solve the technical problem of low model training efficiency caused by fixed and unadjustable containers in the prior art.

[0006] In a first aspect, embodiments of the present invention provide a method for dynamic allocation of model management resource pools, including:

[0007] Calculate multiple subtasks of the model to be trained and obtain the parameters of each subtask;

[0008] determine the time length adaptability of each preset container according to the parameters of the subtasks;

[0009] calculate a first sequence set of container types and corresponding quantities adapted to each subtask according to the time length adaptability of each preset container and the parameters of each subtask;

[0010] select a second sequence set meeting the training accuracy requirement from the first sequence set;

[0011] calculate the use efficiency of each container in the second sequence set, the use efficiency being a comprehensive parameter calculated by using the hardware use rate and the waiting time length;

[0012] obtain an optimal sorting combination according to the use efficiency of each container, and allocate training tasks to the containers according to the optimal sorting combination.

[0013] In a second aspect, an embodiment of the present application further provides a model management resource pool dynamic allocation device, comprising:

[0014] an acquisition module configured to calculate a plurality of subtasks of a model to be trained, and acquire parameters of each subtask;

[0015] a determination module configured to determine the time length adaptability of each preset container according to the parameters of the subtasks;

[0016] a calculation module configured to calculate a first sequence set of container types and corresponding quantities adapted to each subtask according to the time length adaptability of each preset container and the parameters of each subtask;

[0017] a selection module configured to select a second sequence set meeting the training accuracy requirement from the first sequence set;

[0018] a use efficiency calculation module configured to calculate the use efficiency of each container in the second sequence set, the use efficiency being a comprehensive parameter calculated by using the hardware use rate and the waiting time length;

[0019] a distribution module configured to obtain an optimal sorting combination according to the use efficiency of each container, and allocate training tasks to the containers according to the optimal sorting combination.

[0020] In a third aspect, an embodiment of the present application further provides a server, comprising:

[0021] one or more processors;

[0022] a storage device configured to store one or more programs;

[0023] when the one or more programs are executed by the one or more processors, the one or more processors implement the model management resource pool dynamic allocation method provided by the above-mentioned embodiments.

[0024] In a fourth aspect, the embodiments of the present application also provide a storage medium containing computer executable instructions for executing the model management resource pool dynamic allocation method provided by the above embodiments when executed by a computer processor.

[0025] The model management resource pool dynamic allocation method, device, server and medium provided by the embodiments of the present application obtain a plurality of sub-tasks of a to-be-trained model by calculation, and obtain parameters of each sub-task; determine the time length adaptability of each preset container according to the parameters of the sub-tasks; calculate a first sequence set of container types and corresponding quantities adapted to each sub-task according to the time length adaptability of each preset container and the parameters of each sub-task; select a second sequence set meeting the training accuracy requirement from the first sequence set;

[0026] Calculate the use efficiency of each container in the second sequence set, which is a comprehensive parameter calculated by using hardware use rate and waiting time length; obtain an optimal sorting combination according to the use efficiency of each container, and allocate training tasks to the containers according to the optimal sorting combination. By using the time length adaptability parameter, the training time length adaptability of each type of container to a specific training task can be evaluated. By using the time length adaptability, a plurality of combinations of container types and corresponding quantities used for the training task can be calculated, and by the required training accuracy requirement and use efficiency, a container combination meeting the use efficiency is selected from the plurality of combinations to jointly complete the training task. By using the above method, the use efficiency and time length of the existing model can be taken into account, and the model training task can be quickly and efficiently completed. Without affecting other containers, the server can use fewer container resources and shorter time to quickly and efficiently complete model training. BRIEF DESCRIPTION OF DRAWINGS

[0027] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0028] Figure 1 is a flowchart of the model management resource pool dynamic allocation method provided by the first embodiment of the present application;

[0029] Figure 2 is a flowchart of the model management resource pool dynamic allocation method provided by the second embodiment of the present application;

[0030] Figure 3 is a structural diagram of the model management resource pool dynamic allocation device provided by the third embodiment of the present application;

[0031] Figure 4 is a structural diagram of the server provided by the fourth embodiment of the present application. DETAILED DESCRIPTION

[0032] The application will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are intended to be merely illustrative of the application and not limiting thereof. It should also be noted that, for the purpose of clarity, only the parts of the drawings that are pertinent to the application are shown.

[0033] Embodiment One

[0034] Figure 1 is a flowchart of the model management resource pool dynamic allocation method provided by the embodiment one of the application. The embodiment can be applicable to the case of training a model by using multiple fixed containers. The method can be executed by a model management resource pool dynamic allocation device and can be integrated in a server. The method specifically includes the following steps:

[0035] In step 110, a plurality of sub-tasks of the model to be trained are calculated, and the parameters of each sub-task are obtained.

[0036] When facing a resource-limited container, a single complete training can fail due to insufficient video memory or excessive CPU and GPU load. By splitting the task into sub-tasks, the single computing load can be effectively reduced, and the resources can be used in off-peak. For example, the same container can perform different sub-tasks in succession. Moreover, the training efficiency can be improved, and the time cost can be reduced.

[0037] In the embodiment, the model training task can be split by using an inter-layer splitting method. For example, the model can be divided into multiple stages in layer order, such as Stage1: Embedding+Layer1-4, Stage2: Layer5-8. Each sub-task only calculates the activation value of the current stage, and the intermediate result is passed to the next stage in parallel. After obtaining each sub-task, the parameters of each sub-task can be calculated. Specifically, different functional layers calculate parameters in different ways. For example, for a fully connected layer, the parameter quantity can be calculated by the following method: parameter quantity = input dimension x output dimension + output dimension (if there is a bias); for a convolutional layer, the parameter quantity can be calculated by the following method: parameter quantity = convolution kernel width x convolution kernel height x input channel x output channel + output channel; and for a normalization layer, the parameter quantity can be calculated by the following method: parameter quantity = 2 x feature dimension (scaling parameter γ + offset parameter β).

[0038] In step 120, the time length adaptability of each preset container is determined according to the parameters of the sub-tasks.

[0039] In the embodiment, the time length adaptability can refer to the suitability of the container to the training task in the time dimension, i.e., the possibility of quickly and reasonably completing the training task. For example, the time length adaptability of each container to the training task is evaluated by using three types of inherent attributes of the container: operation capability, loading capability, and communication capability. For example, the following steps can be included: generating a training phase curve according to the parameters of each subtask, and calculating the area of the curve region surrounded by the training phase curve; converting the training phase curve into a corresponding horizontal line segment according to the area of the curve region; obtaining the attributes of each container, including operation capability, loading capability, and communication capability between containers; determining the average parameter amount according to the horizontal line segment, and determining the time length adaptability of each container to the training task according to the average parameter amount and the attributes of each container. As can be seen from the foregoing steps, the parameter amount of each training subtask is different, and most of them are in a multiple relationship. Therefore, a coordinate system can be established in advance, wherein the horizontal coordinate can represent the training task step label, and the vertical coordinate can represent the parameter amount of each task step. Generally, each task step can correspond to a subtask. By using the above method, a training phase curve segment can be established, and smoothing processing can be performed. The area of the region surrounded by the training phase curve and the horizontal and vertical axes of the coordinate system is calculated. The area can represent the parameter amount of the training task as a whole.

[0040] Since the training phase curve cannot directly reflect the average characteristics of the training task, which is not conducive to evaluating the time length adaptability of the container to the training task, the area formed by the training phase curve can be converted into a corresponding rectangle. The step label on the horizontal axis of the coordinate system can be used as the width of the rectangle, and the length of the rectangle can be calculated according to the area. The corresponding horizontal line segment is determined according to the length of the rectangle. Then the average parameter amount is determined. Compared with the average count, the change trend and situation are also considered. By using this method, the average parameter amount can be more accurately calculated.

[0041] The attributes of the container can represent the ability to process the task, the operation capability can include CPU and GPU computing capability, and the loading capability can be the ability to read training data into memory. The communication capability can be the ability to transfer training results out or in.

[0042] For example, the time length adaptability of each container to the training task can be calculated by the following method:

[0043]

[0044] =max( ) + , wherein = P / , = / , = / , is the duration of the container corresponding to the average parameter quantity, is the reading duration of the container corresponding to the average parameter quantity, is the communication duration of the container corresponding to the average parameter quantity, P is the average parameter quantity, is the input data quantity corresponding to the average parameter quantity, is the output data quantity corresponding to the average parameter quantity, is the computing capability of the container, is the loading capability of the container, is the communication capability of the container, and A is the duration adaptability of the container to the training task, is the dynamic reference value, is the maximum processing duration of the container corresponding to the average parameter quantity.

[0045] As can be seen from the above parameter definitions, the maximum processing duration of a type of container for the average parameter quantity can be represented, wherein, is the dynamic reference value, which can be set according to experience. It can also be set according to the characteristics of the trained model. For example: if the trained model has a higher requirement for throughput, and if the memory of the container is larger, the corresponding value can be appropriately increased. In this way, various properties of the container can be integrated into one parameter for evaluation, which can reduce the computational complexity while ensuring accuracy.

[0046] It should be noted that the properties of each of the above-mentioned preset containers can be the resources remaining after the container runs other tasks, such as providing email services, providing storage services or other services, or performing other tasks. The available resources of the container are fully utilized. Further, selection can be made on this basis, for example: the resources remaining after removing the resources for external services must meet a certain resource threshold, and only the container meeting the threshold can be regarded as a preset container and participate in the later calculation.

[0047] In step 130, the type of container adapted to each subtask and the corresponding quantity of the first sequence set are calculated according to the duration adaptability of each preset container and the parameters of each subtask.

[0048] Exemplarily, the method can comprise: calculating a deviation degree of each subtask corresponding point on the training stage curve from a horizontal line segment; determining a time length adaptability of each preset container to each subtask by using the deviation degree; selecting a container type in descending order of adaptability according to the time length adaptability of each preset container to each subtask, and calculating whether a single-type minimum container number can meet the subtask requirement; when the single-type minimum container number cannot meet the subtask requirement, establishing a minimum completion time length objective function and corresponding constraint functions, wherein the constraint functions comprise: a quantity upper limit constraint function and a resource upper limit constraint function;

[0049] The time length objective function is as follows:

[0050] ,

[0051] The constraint functions are as follows: , , which are a resource upper limit constraint function and a quantity upper limit constraint function respectively. Wherein j is a subtask, i is a container type selected in descending order of adaptability, is a quantity of the i-th container corresponding to the j-th subtask, is a parameter quantity of the j-th subtask, is a time length adaptability of the i-th container, is an operation capacity of the i-th container, is an output data quantity of the j-th subtask, is a communication capacity of the i-th container, is an input data quantity of the j-th subtask, is a maximum quantity of the i-th type of container; and a first sequence set of a container type and a corresponding quantity adapted to each subtask is obtained according to the minimum completion time length objective function and the corresponding constraint functions.

[0052] Exemplarily, the deviation degree can be used to reflect a change of each subtask relative to an average parameter quantity, and the deviation degree can be used to calculate an adaptability score of a container to a current task A i The deviation degree can be converted into a deviation ratio, and multiplied by the time length adaptability to obtain the adaptability score of the container to the current task A i .

[0053] The time length objective function can realize traversal of various types of containers for combination, and determine a minimum time length for completing all training total tasks by each combination. Based on the minimum time length, a plurality of combinations of different types of containers and corresponding quantities can be obtained to form a first sequence set. There are a plurality of sequences in the combination, and each sequence corresponds to a combination mode.

[0054] Step 140, selecting a second sequence set meeting a training accuracy requirement from the first sequence set.

[0055] There is a close interaction between training accuracy and training task decomposition, and both of them jointly affect the learning efficiency and generalization ability of the model. The task may depend on different data distribution, and if the data distribution is too fine, it will lead to insufficient data of some sub-tasks and reduce the accuracy. In this embodiment, the corresponding training container can be determined based on the sub-task, and each training container executes the grandchild task further subdivided from the sub-task. If there are too many grandchild tasks, it will affect the training accuracy, therefore, the number of containers that can meet the training accuracy requirements and the number of parameters processed by each container can be determined according to the training accuracy requirements. The container set that does not meet the requirements is removed from the first sequence set according to the above labeling, and the second sequence set that meets the training accuracy requirements is obtained.

[0056] In step 150, the use efficiency of each container in the second sequence set is calculated, the use efficiency is a comprehensive parameter calculated by using hardware utilization rate and waiting time; the optimal sorting combination is obtained according to the use efficiency of each container, and the training task is allocated to the container according to the optimal sorting combination.

[0057] The use efficiency of each container in the second sequence set is calculated respectively. For example, the hardware utilization rate of each container in the used container set for executing the training task, that is, the utilization rate of CPU, GPU and memory in executing the training task, and the waiting time for waiting for other tasks to complete data transmission. A utilization rate comprehensive parameter is obtained by using the above, and the container combination for executing the training task is evaluated by using the comprehensive parameter, and the container combination with the highest utilization rate is selected to execute the model training task.

[0058] Optionally, the following method can be used to calculate: Eff c =( U cpu+ U mem+ U gpu)×( ),wherein, Eff c The use efficiency of a container, U cpu is the utilization rate of CPU, U mem is the memory utilization rate, U gpu is the utilization rate of GPU. T wait is the queuing or ready waiting time of the task on the container, Tbusy is the actual execution time of the container. In this way, the actual hardware usage and the waiting time can be combined into a comprehensive parameter, which is convenient for later usage calculation and evaluation of training effect. The usage rates of CPU, memory and GPU in the container can be predicted using existing time series analysis or machine learning models, and can also be corrected using container monitoring tools. Alternatively, the usage rate collected during the approximate training task can be used as the predicted usage rate.

[0059] In this embodiment, a plurality of sub-tasks of the to-be-trained model are calculated to obtain parameters of each sub-task; the time length adaptability of each preset container is determined according to the parameters of the sub-tasks; a first sequence set of container types and corresponding quantities adapted to each sub-task is calculated according to the time length adaptability of each preset container and the parameters of each sub-task; and a second sequence set meeting the training accuracy requirement is selected from the first sequence set;

[0060] The usage efficiency of each container in the second sequence set is calculated, which is a comprehensive parameter calculated by the hardware usage rate and the waiting time; and the optimal sorting combination is obtained according to the usage efficiency of each container, and the training task is allocated to the container according to the optimal sorting combination. By using a parameter of time length adaptability, the training time length adaptability of each type of container for a specific training task can be evaluated. By using the time length adaptability, a plurality of combinations of container types and corresponding quantities used for the training task can be calculated, and by the required training accuracy requirement and the usage efficiency, a container combination meeting the usage efficiency is selected from the plurality of combinations to complete the training task together. By using the above method, the usage efficiency and the time length of the existing model can be considered, and the model training task can be quickly and efficiently completed. Without affecting other containers, the server can use as few container resources as possible to quickly and efficiently complete the model training.

[0061] In a preferred embodiment of the present embodiment, before the optimal combination is obtained according to the usage efficiency of each container and the training task is allocated to the container according to the optimal sorting combination, the method can further include the following steps: in the presence of a backward calculation process, the probability of container preemption / recycling is determined according to the scene type of the model application; the probability of re-routing is estimated according to the communication fluctuation; and the backward reuse coefficient is obtained according to the probability of container preemption / recycling and the probability of re-routing, and the usage efficiency of each container is optimized by using the backward reuse coefficient. In the training process with backward calculation, the same batch of containers performs both forward operation and backward operation, which is the most direct means to improve the usage efficiency. For example, a backward reuse coefficient can be added to the formula for calculating the utilization rate to correct the usage efficiency of each container, so as to further improve the accuracy of efficiency calculation.

[0062] Embodiment Two

[0063] Figure 2 is a flowchart of the model management resource pool dynamic allocation method provided by Embodiment Two of the present application. Based on the above-mentioned embodiment, the embodiment is optimized as follows: the second sequence set selected from the first sequence set and meeting the training accuracy requirement is specifically optimized as follows: the minimum data operation amount of the container is determined according to the training accuracy requirement; it is determined whether the operation capacity corresponding to the container with the lowest time length adaptability is less than the minimum data operation amount, and when it is less than the minimum data operation amount, the container with the lowest adaptability is removed, and the upper limit constraint function is adjusted according to the maximum parallel operation distribution, and the first sequence set of the container type and the corresponding number adapted to each subtask is recalculated as the second sequence set.

[0064] Referring to Figure 2 , the model management resource pool dynamic allocation method comprises:

[0065] Step 210, a plurality of subtasks of a model to be trained are calculated, and the parameters of each subtask are obtained.

[0066] Step 220, the time length adaptability of each preset container is determined according to the parameters of the subtasks.

[0067] Step 230, the first sequence set of the container type and the corresponding number adapted to each subtask is calculated according to the time length adaptability of each preset container and the parameters of each subtask.

[0068] Step 240, the minimum data operation amount is determined according to the training accuracy requirement; and it is determined whether the operation capacity corresponding to the container with the lowest adaptability is greater than the minimum data operation amount.

[0069] Step 250, when it is less than the minimum data operation amount, the container with the lowest adaptability is removed, and the upper limit constraint function is adjusted according to the maximum parallel operation distribution, and the first sequence set of the container type and the corresponding number adapted to each subtask is recalculated as the second sequence set.

[0070] The training accuracy is closely related to the number of parameters corresponding to each container. If the number of parameters is too small, the training accuracy requirement is not met, therefore, the container of this type needs to be removed, and the total number of containers corresponding to the maximum parallel operation condition needs to be adjusted, i.e. the number of N in the constraint function is reduced, which is used to adjust the container distribution condition corresponding to the parallel calculation. In this way, the container type and the corresponding number adapted to each subtask can be adjusted and optimized. A plurality of sequence sets are regenerated as the second sequence set.

[0071] In step 260, the use efficiency of each container in the second sequence set is calculated, the use efficiency being a comprehensive parameter calculated using hardware use rate and waiting time length; an optimal sorting combination is obtained according to the use efficiency of each container, and a training task is allocated to the container according to the optimal sorting combination.

[0072] The embodiment specifically optimizes by selecting a second sequence set meeting the training accuracy requirement from the first sequence set, and specifically comprises the following steps: determining the minimum data operation amount according to the training accuracy requirement; when the data operation amount is less than the minimum data operation amount, removing the container with the lowest adaptability, and adjusting the upper limit constraint function according to the maximum parallel operation distribution condition, and recalculating the first sequence set of the container type and the corresponding number adapted to each subtask as the second sequence set. The influence on the training accuracy can be judged based on the distribution condition of the training data, and the first sequence set of the container type and the corresponding number adapted to each subtask is optimized by adjusting the upper limit constraint function, so as to meet the requirement of the training accuracy.

[0073] Embodiment three

[0074] Figure 3 is a structural schematic diagram of the model management resource pool dynamic allocation device provided by the embodiment three of the application, as shown in Figure 3 The device comprises:

[0075] The acquisition module 310 is configured to calculate a plurality of subtasks of a to-be-trained model, and acquire parameters of each subtask.

[0076] The determination module 320 is configured to determine the time length adaptability of each preset container according to the parameters of the subtasks.

[0077] The calculation module 330 is configured to calculate a first sequence set of the container type and the corresponding number adapted to each subtask according to the time length adaptability of each preset container and the parameters of each subtask.

[0078] The selection module 340 is configured to select a second sequence set meeting the training accuracy requirement from the first sequence set.

[0079] The use efficiency calculation module 350 is configured to calculate the use efficiency of each container in the second sequence set, the use efficiency being a comprehensive parameter calculated using hardware use rate and waiting time length.

[0080] The allocation module 360 is configured to obtain an optimal sorting combination according to the use efficiency of each container, and allocate a training task to the container according to the optimal sorting combination.

[0081] The model management resource pool dynamic allocation device provided by the embodiment comprises: a calculation unit, configured to calculate a plurality of sub-tasks of a to-be-trained model, and obtain parameters of each sub-task; a determination unit, configured to determine time length adaptability of each preset container according to the parameters of the sub-tasks; a calculation unit, configured to calculate a first sequence set of container types and corresponding quantities that are adapted to each sub-task according to the time length adaptability of each preset container and the parameters of each sub-task; and a selection unit, configured to select a second sequence set that meets a training accuracy requirement from the first sequence set;

[0082] The use efficiency of each container in the second sequence set is calculated, the use efficiency being a comprehensive parameter calculated by using hardware use rate and waiting time length; an optimal sorting combination is obtained according to the use efficiency of each container, and a training task is allocated to the container according to the optimal sorting combination. By using the time length adaptability parameter, the training time length adaptability of each type of container to a specific training task can be evaluated. By using the time length adaptability, a plurality of combinations of container types and corresponding quantities used for the training task can be calculated, and a container combination that meets the use efficiency is selected from the plurality of combinations to jointly complete the training task according to the required training accuracy requirement and the use efficiency. By using the above method, the use efficiency and time length of the existing model can be taken into account, and the model training task can be quickly and efficiently completed. Without affecting other containers, the server can use as few container resources as possible to quickly and efficiently complete model training.

[0083] On the basis of the above embodiments, the determination module comprises:

[0084] An area calculation unit is configured to generate a training stage curve according to the parameters of each sub-task, and calculate an area of a curve region surrounded by the training stage curve;

[0085] A conversion unit is configured to convert the training stage curve into a corresponding horizontal line segment according to the area of the curve region;

[0086] An attribute acquisition unit is configured to acquire attributes of each container, the attributes including operation capability, loading capability and communication capability between containers;

[0087] A determination unit is configured to determine an average parameter quantity according to the horizontal line segment, and determine time length adaptability of each container to a training task according to the average parameter quantity and the attributes of each container.

[0088] On the basis of the above embodiments, the determination unit is implemented in the following manner:

[0089] ,

[0090] =max( ) + ,

[0091] wherein, = P / , = / , = / , is the average parameter quantity corresponding to the container calculation duration, is the average parameter quantity corresponding to the container read-in duration, is the average parameter quantity corresponding to the container communication duration, P is the average parameter quantity, is the average parameter quantity corresponding to the input data quantity, is the average parameter quantity corresponding to the output data quantity, is the operation capacity of the container, is the loading capacity of the container, is the communication capacity of the container, and A is the duration adaptability of the container to the training task, is the dynamic benchmark value, is the maximum processing duration of the average parameter quantity corresponding to the container.

[0092] On the basis of the above embodiments, the calculation module comprises:

[0093] a deviation calculation unit configured to calculate the deviation of each subtask corresponding point on the training stage curve from the horizontal line segment;

[0094] a duration adaptability determination unit configured to determine the duration adaptability of each preset container to each subtask by using the deviation;

[0095] a selection unit configured to select the container type in descending order of adaptability according to the duration adaptability of each preset container to each subtask, and to calculate whether the minimum number of containers of a single type can meet the subtask demand;

[0096] a building unit configured to build a minimum completion duration objective function and correspondingly set a constraint function when the demand cannot be met, the constraint function comprising: a quantity upper limit constraint function and a resource upper limit constraint function;

[0097] The duration objective function is as follows:

[0098] ,

[0099] The constraint function is as follows: , wherein, j is the subtask, i is the container type selected in descending order of adaptability, a quantity of the i-th container corresponding to the j-th subtask, a parameter quantity of the j-th subtask, a time length adaptability of the i-th container, a computing capacity of the i-th container, an output data quantity of the j-th subtask, a communication capacity of the i-th container, a data quantity input by the j-th subtask, a maximum quantity of the i-th type of container;

[0100] The first sequence set obtaining unit is configured to obtain, according to the minimum completion time length objective function and the corresponding constraint function, a first sequence set of the container type and the corresponding quantity that are adapted to each subtask.

[0101] On the basis of the above embodiments, the selecting module comprises:

[0102] The minimum data computing quantity determining unit is configured to determine the minimum data computing quantity of the container according to the training precision requirement.

[0103] The removing unit is configured to determine whether the computing capacity of the container with the lowest time length adaptability is less than the minimum data computing quantity, and when the computing capacity is less than the minimum data computing quantity, remove the container with the lowest adaptability, and adjust the quantity upper limit constraint function according to the maximum parallel computing distribution condition, and recompute the first sequence set of the container type and the corresponding quantity that are adapted to each subtask as a second sequence set.

[0104] On the basis of the above embodiments, the device further comprises

[0105] The probability determining module is configured to determine, in the presence of the reverse calculation process, a probability of the container being preempted / recycled according to a scene type to which the model is applied.

[0106] The estimation module is configured to estimate a probability of re-routing according to a communication fluctuation condition.

[0107] The optimization module is configured to obtain a reverse multiplexing coefficient according to the probability of the container being preempted / recycled and the probability of re-routing, and optimize the use efficiency of each container by using the reverse multiplexing coefficient.

[0108] The model management resource pool dynamic allocation device provided in the embodiments can execute the model management resource pool dynamic allocation method provided in any of the embodiments, and has the corresponding function modules and beneficial effects of the execution method.

[0109] Embodiment Four

[0110] Figure 4A structural schematic diagram of a server is provided for Embodiment Five of the present application. Figure 4 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present application is shown. Figure 4 The server 12 shown is merely one example. It should be appreciated that the present application is not limited in scope to the functionality and use of this server 12.

[0111] As Figure 4 shown, the server 12 is in the form of a general-purpose computing device. The components of server 12 can include, but are not limited to, one or more processors or processing units 16, a memory 28, and a bus 18 that couples various system components, including the memory 28 and the processing unit 16.

[0112] The bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics bus (e.g., an Accelerated Graphics Port, or AGP bus) and a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0113] The server 12 typically includes a variety of computer system readable media. Such media can be any available media that is locally and / or remotely accessible by server 12, and can include both volatile and non-volatile media, removable and non-removable media.

[0114] The memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 4 not shown, can also be used. Although Figure 4 not shown, a magnetic hard disk drive for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"); and an optical disk drive for reading from or writing to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, etc.) can also be provided. In such instances, each can be connected to the bus 18 by one or more data media interfaces. The memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the present application.

[0115] Program / utility 40 having a set of program modules 42 can be stored in memory 28 by way of example, such program modules 42 include an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which may

[0116] Server 12 can also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with server 12; and / or any devices (e.g., network card, modem, etc.) that enable server 12 to communicate with one or more other computing devices. Such communication can occur via input / output (I / O) interface(s) 22. Still yet, server 12 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of server 12 via bus 18. It should be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with server 12. Such modules include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0117] Processing unit 16 executes the various functions and processes of embodiments of the present application by running programs stored in memory 28.

[0118] Embodiment Five

[0119] Embodiment Five also provides a storage medium containing computer executable instructions, which when executed by a computer processor, perform the model management resource pool dynamic allocation method as provided by the above embodiments.

[0120] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0121] The computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which computer-readable program code is embodied. Such propagated data signals can take a wide variety of forms, including but not limited to electro-magnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium that is not a storage medium, that is, that is not a tangible medium, and that can communicate, propagate or transport programming for use by or in connection with an instruction execution system, apparatus, or device.

[0122] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the above.

[0123] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0124] Note that the above merely describes preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, modifications and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A model management resource pool dynamic allocation method, characterized in that, The application relates to a method for training a model, comprising the following steps: calculating a plurality of sub-tasks of a to-be-trained model to obtain parameters of each sub-task; determining the time length adaptability of each preset container according to the parameters of the sub-tasks; calculating a first sequence set of container types and corresponding quantities adapted to each sub-task according to the time length adaptability of each preset container and the parameters of each sub-task, wherein the time length adaptability is the suitability of the preset container to the training task in the time dimension; selecting a second sequence set meeting the training accuracy requirement from the first sequence set; calculating the use efficiency of each container in the second sequence set, wherein the use efficiency is a comprehensive parameter calculated by using the hardware use rate and the waiting time length; obtaining an optimal sorting combination according to the use efficiency of each container, and allocating the training task to the container according to the optimal sorting combination; the step of determining the time length adaptability of each preset container according to the parameters of the sub-tasks comprises the following steps: generating a training stage curve according to the parameters of each sub-task, and calculating the area of a curve region surrounded by the training stage curve; converting the training stage curve into a corresponding horizontal line segment according to the area of the curve region; obtaining the attributes of each preset container, wherein the attributes include the computing capacity, the loading capacity and the communication capacity between the preset containers; determining the average parameter quantity according to the horizontal line segment, and determining the time length adaptability of each preset container to the training task according to the average parameter quantity and the attributes of each preset container; determining the average parameter quantity according to the horizontal line segment, and determining the time length adaptability of each preset container to the training task according to the average parameter quantity and the attributes of each preset container, which is realized in the following manner: ， =max( )+ , wherein, = P / , = / , = / , is the calculation duration of the container corresponding to the average parameter quantity, is the reading duration of the container corresponding to the average parameter quantity, is the communication duration of the container corresponding to the average parameter quantity, P is the average parameter quantity, is the input data quantity corresponding to the average parameter quantity, is the output data quantity corresponding to the average parameter quantity, is the operation capability of the container, is the loading capability of the container, is the communication capability of the container, and A is the duration adaptability of the container to the training task, is the dynamic reference value, is the maximum processing duration of the container corresponding to the average parameter quantity.

2. The method of claim 1, wherein, the step of calculating a first sequence set of container types and corresponding quantities adapted to each sub-task according to the time length adaptability of each preset container and the parameters of each sub-task comprises the following steps: calculating the deviation degree of the corresponding point of each sub-task on the training stage curve from the horizontal line segment; determining the time length adaptability of each preset container to each sub-task by using the deviation degree; selecting the container type in descending order of adaptability according to the time length adaptability of each preset container to each sub-task, and calculating whether the least container number of a single type can meet the sub-task demand; when the least container number of a single type cannot meet the sub-task demand, establishing a minimum completion time length objective function and corresponding constraint functions, wherein the constraint functions include a quantity upper limit constraint function and a resource upper limit constraint function; the minimum completion time length objective function is as follows: ， The constraint function is: , wherein j is a subtask, i is a container type selected in descending order of adaptability, is the number of the ith container corresponding to the jth subtask, is the parameter quantity of the jth subtask, is the time length adaptability of the ith container to the jth subtask, is the operation capacity of the ith container, is the output data quantity of the jth subtask, is the communication capacity of the ith container, is the data quantity input by the jth subtask, is the maximum number of the ith type of container; obtaining a first sequence set of container types and corresponding quantities adapted to each sub-task according to the minimum completion time length objective function and the corresponding constraint functions.

3. The method of claim 1, wherein, the step of selecting a second sequence set meeting the training accuracy requirement from the first sequence set comprises the following steps: determining the least data operation amount of the container according to the training accuracy requirement; determining whether the operation capacity of the container with the lowest time length adaptability is less than the least data operation amount, and removing the container with the lowest time length adaptability when the operation capacity is less than the least data operation amount; and adjusting the quantity upper limit constraint function according to the maximum parallel operation distribution condition, and recalculating a first sequence set of container types and corresponding quantities adapted to each sub-task as a second sequence set.

4. The method of claim 3, wherein, Before obtaining the optimal combination according to the use efficiency of each container and performing the sub-task training task allocation to the container according to the optimal sorting combination, the method further comprises: In the reverse calculation process, according to the type of the scene where the model is applied, the probability of the container being preempted / recycled is determined; According to the communication fluctuation, the probability of re-routing is estimated; According to the probability of the container being preempted / recycled and the probability of re-routing, a reverse multiplexing coefficient is obtained, and the use efficiency of each container is optimized by using the reverse multiplexing coefficient.

5. A model management resource pool dynamic allocation apparatus, characterized in that, Comprise: The acquisition module is used to calculate a plurality of sub-tasks of a to-be-trained model, and to acquire parameters of each sub-task; The determination module is used to determine the time length adaptability of each preset container according to the parameters of the sub-tasks, wherein the time length adaptability is the suitability of the preset container to the training task in the time dimension; The calculation module is used to calculate a first sequence set of the container type and the corresponding number adapted to each sub-task according to the time length adaptability of each preset container and the parameters of each sub-task; The selection module is used to select a second sequence set meeting the training accuracy requirement from the first sequence set; The use efficiency calculation module is used to calculate the use efficiency of each container in the second sequence set, wherein the use efficiency is a comprehensive parameter calculated by using the hardware usage rate and the waiting time length; The allocation module is used to obtain the optimal sorting combination according to the use efficiency of each container, and to allocate the training task to the container according to the optimal sorting combination; The determination module comprises: The area calculation unit is used to generate a training phase curve according to the parameters of each sub-task, and to calculate the area of a curve region surrounded by the training phase curve; The conversion unit is used to convert the training phase curve into a corresponding horizontal line segment according to the area of the curve region; The attribute acquisition unit is used to acquire the attributes of each container, wherein the attributes comprise the operation ability, the loading ability and the communication ability between the containers; The determination unit is used to determine the average parameter amount according to the horizontal line segment, and to determine the time length adaptability of each container to the training task according to the average parameter amount and the attributes of each container; The determination unit is implemented in the following manner: ， = max( ) + 1 , wherein, = P / , = / , = / , is the average parameter quantity corresponding to the container calculation duration, is the average parameter quantity corresponding to the container reading duration, is the average parameter quantity corresponding to the container communication duration, P is the average parameter quantity, is the average parameter quantity corresponding to the input data quantity, is the average parameter quantity corresponding to the output data quantity, is the operation capacity of the container, is the loading capacity of the container, is the communication capacity of the container, and A is the duration adaptability of the container to the training task, is the dynamic reference value, is the maximum processing duration of the container corresponding to the average parameter quantity.

6. A server, characterized by The server comprises: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the model management resource pool dynamic allocation method according to any one of claims 1-4.

7. A storage medium containing computer executable instructions for executing the model management resource pool dynamic allocation method according to any one of claims 1-4 when executed by a computer processor.

7. A storage medium containing computer executable instructions for executing the model management resource pool dynamic allocation method according to any one of claims 1-4 when executed by a computer processor.

Images