Model management resource pool dynamic allocation method and device, server and medium

Abstract

The invention discloses a model management resource pool dynamic allocation method and device, a server and a medium, and belongs to the field of large models.The method comprises the steps that multiple subtasks of a to-be-trained model are obtained through calculation, and parameters of each subtask are obtained; determining the duration suitability of each preset container according to the parameters of the subtasks; according to the duration suitability of each preset container and the parameter of each subtask, calculating to obtain a container type matched with each subtask and a corresponding number of first sequence sets; selecting a second sequence set meeting the training precision requirement from the first sequence set; calculating the use efficiency of each container in the second sequence set, wherein the use efficiency is a comprehensive parameter obtained by calculating the hardware use rate and the waiting duration; and obtaining an optimal sorting combination according to the use efficiency of each container, and allocating training tasks to the containers according to the optimal sorting combination. Therefore, the server can quickly and efficiently complete model training by using fewer container resources and shorter duration as much as possible.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a method, device, server and medium for dynamically allocating a model management resource pool. Background Art

[0002] With the continuous advancement of big data and machine learning research, the need for larger training sets is becoming increasingly important to improve machine learning performance. However, in large-scale machine learning training scenarios, the training data and model parameters become too large for a single machine to handle. Consequently, distributed machine learning has emerged.

[0003] Container technology not only enables rapid deployment of container clusters but also provides a lightweight solution that effectively integrates and manages bare metal resources. It ensures consistent application execution across different devices, provides resource isolation for the application's runtime environment, abstracts the complexity of underlying hardware and node management, and supports CPU and GPU scheduling. Container technology enables distributed machine learning to be deployed across multiple containers, breaking down training tasks into smaller tasks and assigning them to multiple containers for training. Distributed machine learning not only distributes training tasks across the computing resources corresponding to multiple containers but also distributes data (including training data and intermediate results) across the storage of different containers. The effectiveness of distributed machine learning depends on container compatibility. Conventional technology often decomposes the training task into several subtasks when deploying distributed machine training on container cloud platforms, then creates corresponding containers for each subtask. This ensures that the containers are as closely aligned with the subtasks as possible, thereby improving container efficiency.

[0004] However, on some small servers, the allocated containers often need to support multiple model training tasks. It is not possible to create new containers for each specific model training task. At the same time, the existing containers may also need to provide some external services. Therefore, it is impossible to create matching containers based on model training requirements, which in turn cannot improve container utilization efficiency. Summary of the Invention

[0005] Embodiments of the present invention provide a method, device, server, and medium for dynamically allocating a model management resource pool to solve the technical problem in the prior art of low model training efficiency caused by the inability to adjust a fixed container.

[0006] In a first aspect, an embodiment of the present invention provides a method for dynamically allocating a model management resource pool, comprising: Calculate multiple subtasks of the model to be trained and obtain the parameters of each subtask; Determine the duration adaptability of each preset container based on the subtask parameters; Calculate a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask; Selecting a second sequence set that meets the training accuracy requirement from the first sequence set; Calculating the utilization efficiency of each container in the second sequence set, where the utilization efficiency is a comprehensive parameter calculated using hardware utilization and waiting time; The optimal sorting combination is obtained based on the utilization efficiency of each container, and training tasks are assigned to the containers based on the optimal sorting combination.

[0007] In a second aspect, an embodiment of the present invention further provides a model management resource pool dynamic allocation device, comprising: The acquisition module is used to calculate multiple subtasks of the model to be trained and obtain the parameters of each subtask; A determination module, used to determine the duration adaptability of each preset container based on the parameters of the subtask; A calculation module, configured to calculate a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask; A selection module, configured to select a second sequence set that meets the training accuracy requirement from the first sequence set; A usage efficiency calculation module is configured to calculate the usage efficiency of each container in the second sequence set, where the usage efficiency is a comprehensive parameter calculated using hardware usage and waiting time; The allocation module is used to obtain the optimal sorting combination based on the utilization efficiency of each container and allocate training tasks to the containers based on the optimal sorting combination.

[0008] In a third aspect, an embodiment of the present invention further provides a server, including: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the model management resource pool dynamic allocation method provided in the above embodiment.

[0009] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the model management resource pool dynamic allocation method provided in the above embodiment.

[0010] The model management resource pool dynamic allocation method, device, server, and medium provided in embodiments of the present invention calculate multiple subtasks of a model to be trained and obtain parameters for each subtask; determine the duration adaptability of each preset container based on the subtask parameters; calculate a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask; and select a second sequence set from the first sequence set that meets training accuracy requirements. Calculate the utilization efficiency of each container in the second sequence set, where the utilization efficiency is a comprehensive parameter calculated using hardware utilization and waiting time; obtain the optimal sorting combination based on the utilization efficiency of each container, and assign training tasks to the containers based on the optimal sorting combination. By utilizing the duration adaptability parameter, the training duration adaptability of each type of container to a specific training task can be evaluated. Using duration adaptability, multiple combinations of container types and corresponding quantities used for the training task can be calculated, and based on the required training accuracy and utilization efficiency, a container combination that meets the utilization efficiency can be selected from multiple combinations to complete the training task together. Using the above method, the utilization efficiency and duration of the existing model can be taken into account, and the model training task can be completed quickly and efficiently. Without affecting other containers. This allows the server to complete model training quickly and efficiently using as few container resources and a shorter duration as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 1 is a flow chart of a method for dynamically allocating a model management resource pool according to a first embodiment of the present invention; Figure 2 Schematic diagram of the process of the model management resource pool dynamic allocation method provided in the second embodiment of the present invention; Figure 3 Schematic diagram of the structure of the model management resource pool dynamic allocation device provided in the third embodiment of the present invention; Figure 4 This is a structural diagram of the server provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0013] Example 1 Figure 1This is a flowchart of a method for dynamically allocating a model management resource pool, provided in Example 1 of the present invention. This embodiment is applicable to situations where a model is trained using multiple fixed containers. The method can be executed by a dynamic allocation device for a model management resource pool and can be integrated into a server. The method specifically includes the following steps: Step 110: Calculate and obtain multiple subtasks of the model to be trained, and obtain the parameters of each subtask.

[0014] When working with resource-constrained containers, a single full training run may fail due to insufficient video memory or excessive CPU or GPU load. Splitting tasks into subtasks can effectively reduce the single computational load and achieve staggered resource usage. For example, the same container can execute different subtasks sequentially. This can also improve training efficiency and reduce training time.

[0015] In this embodiment, the model training task can be split using an inter-layer splitting method. For example, the model can be divided into multiple stages in layer order, such as Stage 1: Embedding + Layer 1-4, Stage 2: Layer 5-8. Each subtask only calculates the activation value of the current stage and passes the intermediate results to the next stage in parallel. After obtaining each subtask, the parameters of each subtask can be calculated. Specifically, different functional layers calculate parameters in different ways. For example: for the fully connected layer, the parameter amount can be calculated as follows: parameter amount = input dimension × output dimension + output dimension (if there is a bias); for the convolution layer, the parameter amount can be calculated as follows: parameter amount = convolution kernel width × convolution kernel height × input channel × output channel + output channel; for the normalization layer, the parameter amount can be calculated as follows: parameter amount = 2 × feature dimension (scaling parameter γ + offset parameter β).

[0016] Step 120: Determine the duration adaptability of each preset container according to the parameters of the subtask.

[0017] In this embodiment, duration adaptability may refer to the container's suitability for the training task in the temporal dimension, i.e., the likelihood of completing the training task quickly and reasonably. Exemplarily, the duration adaptability of each container to the training task is assessed using three inherent properties of the container: computing power, loading capacity, and communication capacity. Exemplarily, this may include: generating a training phase curve based on the parameters of each subtask and calculating the area of ​​the curve region enclosed by the training phase curve; converting the training phase curve into a corresponding horizontal line segment based on the area of ​​the curve region; obtaining the properties of each container, including computing power, loading capacity, and communication capacity between containers; determining the average parameter value based on the horizontal line segment, and determining the duration adaptability of each container to the training task based on the average parameter value and the properties of each container. As can be seen from the aforementioned steps, the parameter values ​​of each training subtask are different, and most are multiples of each other. Therefore, a coordinate system can be pre-established, where the horizontal axis can represent the training task step number and the vertical axis can represent the parameter value of each task step. Typically, each task step corresponds to a subtask. Using the above method, training phase curve segments can be established and smoothed. The area of ​​the region enclosed by the training phase curve and the horizontal and vertical axes of the coordinate system is calculated. This area can represent the overall parameter quantity of the training task.

[0018] Because the training phase curve cannot directly reflect the average characteristics of the training task and is not conducive to evaluating the container's adaptability to the duration of the training task, the area formed by the training phase curve can be used to convert it into a corresponding rectangle. The width of the rectangle can be calculated using the step number on the horizontal axis of the coordinate system, and the length of the rectangle can be calculated based on the area. The corresponding horizontal line segment is then determined based on the length of the rectangle. The average parameter value is then determined. Compared to the average count, this method also takes into account the trend and situation of changes. Using this method, the average parameter value can be calculated more accurately.

[0019] The container's attributes can represent its ability to process tasks. Computing capabilities can include CPU and GPU computing capabilities. Loading capabilities can include the ability to read training data into memory. Communication capabilities can include the ability to transfer training results in and out.

[0020] For example, the duration adaptability of each container to the training task can be calculated as follows:

[0021] =max( )+ ,in, = P / , = / , = / , The container calculation time corresponding to the average parameter amount, is the container reading time corresponding to the average parameter value, is the container communication duration corresponding to the average parameter value, P is the average parameter, is the amount of input data corresponding to the average parameter value, is the output data volume corresponding to the average parameter volume, The computing power of the container, is the loading capacity of the container, is the communication capability of the container, A is the adaptability of the container to the duration of the training task, is the dynamic reference value, The maximum processing time corresponding to the average parameter value of the container.

[0022] It can be seen from the above parameter definitions that It can represent the maximum processing time of a type of container for the average parameter amount, where This is a dynamic baseline value that can be set based on experience. It can also be set based on the characteristics of the trained model. For example, if the training model requires high throughput and the container has a large memory, the corresponding Using this method, multiple attributes of a container can be combined into one parameter for evaluation, which can reduce the amount of calculation while ensuring accuracy.

[0023] It should be noted that the properties of each preset container can be the remaining resources after the container runs other tasks, such as providing external mail services, storage services, or other services, or performing other tasks. This allows for full utilization of the container's available resources. Furthermore, selection can be made based on this, for example, removing resources for external services and requiring that the remaining resources meet a certain resource threshold before a container is considered a preset container and participates in subsequent calculations.

[0024] Step 130 : Calculate a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask.

[0025] Exemplarily, the method may include: calculating the deviation between the corresponding point of each subtask on the training phase curve and the horizontal line segment; using the deviation to determine the duration adaptability of each preset container to each subtask; selecting container types in descending order of adaptability based on the duration adaptability of each preset container to each subtask, and calculating whether the minimum number of containers of a single type can meet the subtask requirements; if the requirements cannot be met, establishing a minimum completion duration objective function and correspondingly setting constraint functions, wherein the constraint functions include: a quantity upper limit constraint function and a resource upper limit constraint function; The duration objective function is as follows: , The constraint function is: , , respectively, the resource upper limit constraint function and the quantity upper limit constraint function. Among them, j is the subtask, i is the container type selected in descending order of adaptability, is the number of the i-th container corresponding to the j-th subtask, is the parameter amount of the j-th subtask, is the duration adaptability of the i-th container, is the computing power of the i-th container, is the output data volume of the jth subtask, is the communication capability of the i-th container, The amount of data input for the jth subtask, is the maximum number of containers of the i-th type; according to the minimum completion time objective function and the corresponding constraint function, a first sequence set of container types and corresponding quantities adapted to each subtask is obtained.

[0026] For example, the deviation can be used to reflect the change of each subtask relative to the average parameter value, and the deviation can be used to calculate the suitability score of the container for the current task. A i The deviation can be converted into a deviation ratio and multiplied by the duration adaptability to obtain the adaptability score of this type of container for the current task. A i .

[0027] The duration objective function traverses various container types and combines them to determine the minimum duration for each combination to complete the total training task. Based on the minimum duration, multiple combinations of different container types and corresponding quantities are obtained to form a first sequence set. There are multiple sequences in the set, each corresponding to a specific combination.

[0028] Step 140: Select a second sequence set that meets the training accuracy requirement from the first sequence set.

[0029] There is a close interactive relationship between training accuracy and training task decomposition, and the two jointly affect the learning efficiency and generalization ability of the model. Tasks may rely on different data distributions. If the data distribution is too fine, it will lead to insufficient data for some subtasks and reduced accuracy. In this embodiment, the corresponding training container can be determined based on the subtask. Each training container executes a grandtask obtained by further subdividing the subtask. If there are too many grandtasks, it will affect the training accuracy. Therefore, according to the training accuracy requirements, the number of containers that can execute subtasks and the number of parameters processed by each container can be determined to meet the training accuracy requirements. According to the above annotations, the container set that does not meet the requirements is removed from the first sequence set to obtain a second sequence set that meets the training accuracy requirements.

[0030] Step 150: Calculate the utilization efficiency of each container in the second sequence set, where the utilization efficiency is a comprehensive parameter calculated using hardware utilization and waiting time; obtain an optimal sorting combination based on the utilization efficiency of each container, and assign training tasks to the containers based on the optimal sorting combination.

[0031] Calculate the utilization efficiency of each container in the second sequence set. For example, this is the hardware utilization of each container in the set of containers used to execute the training task, namely, the CPU, GPU, and memory utilization during the training task, as well as the waiting time for other tasks to complete data transmission. A comprehensive utilization parameter is obtained using this, and the comprehensive parameter is used to evaluate the container combinations executing the training task. The container combination with the highest utilization rate is selected to execute the model training task.

[0032] Optionally, Eff can be calculated as follows: c =( U cpu+ U mem+ U GPU) × ( ), among which Eff c The efficiency of a container, U cpu is the CPU usage, U mem memory usage, U gpu is the GPU usage. T Wait is the queue or ready waiting time of the task on the container. T "Busy" represents the actual duration of the container's task execution. This method combines actual hardware utilization and wait time into a single parameter, facilitating subsequent utilization calculations and evaluating training effectiveness. The container's CPU, memory, and GPU utilization can be predicted using existing time series analysis or machine learning models, and can be corrected using container monitoring tools. Alternatively, the utilization collected from previous, similar training tasks can be used as the predicted utilization.

[0033] This embodiment calculates multiple subtasks of a model to be trained and obtains parameters for each subtask; determines the duration adaptability of each preset container based on the subtask parameters; calculates a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask; and selects a second sequence set from the first sequence set that meets training accuracy requirements. Calculate the utilization efficiency of each container in the second sequence set, where the utilization efficiency is a comprehensive parameter calculated using hardware utilization and waiting time; obtain the optimal sorting combination based on the utilization efficiency of each container, and assign training tasks to the containers based on the optimal sorting combination. By using a parameter of duration adaptability, the training duration adaptability of each type of container to a specific training task can be evaluated. Duration adaptability can be used to calculate multiple combinations of container types and corresponding quantities used for the training task, and through the required training accuracy requirements and utilization efficiency, a container combination that meets the utilization efficiency is selected from multiple combinations to complete the training task together. Using the above method, the utilization efficiency and duration of the existing model can be taken into account, and the model training task can be completed quickly and efficiently. Without affecting other containers. This allows the server to complete model training quickly and efficiently using as few container resources as possible.

[0034] In a preferred implementation of this embodiment, before obtaining the optimal combination based on the utilization efficiency of each container and assigning subtask training tasks to the containers based on the optimal sorted combination, the method may further include the following steps: in the reverse calculation process, determining the probability of the container being preempted / reclaimed based on the scenario type of the model application; estimating the probability of rerouting based on the communication fluctuation; obtaining a reverse reuse coefficient based on the probability of the container being preempted / reclaimed and the probability of rerouting, and using the reverse reuse coefficient to optimize the utilization efficiency of each container. In a training process with a backward pass, having the same batch of containers perform both forward and reverse operations is the most direct means of improving utilization efficiency. For example, a reverse reuse coefficient can be added to the formula for calculating utilization to correct the utilization efficiency of each container to further improve the accuracy of efficiency calculations.

[0035] Example 2 Figure 2This is a flow chart of the dynamic allocation method for the model management resource pool provided in the second embodiment of the present invention. This embodiment is optimized based on the above embodiment, and the second sequence set that meets the training accuracy requirements is selected from the first sequence set. The specific optimization is as follows: determining the minimum data computing amount of the container according to the training accuracy requirements; determining whether the computing power corresponding to the container with the lowest time adaptability is less than the minimum data computing amount. When it is less than the minimum data computing amount, remove the container with the lowest adaptability, and adjust the quantity upper limit constraint function according to the maximum parallel computing distribution, and recalculate the first sequence set of container types and corresponding quantities that are adapted to each subtask as the second sequence set.

[0036] See also Figure 2 The model management resource pool dynamic allocation method includes: In step 210, multiple subtasks of the model to be trained are calculated and the parameters of each subtask are obtained.

[0037] Step 220: Determine the duration adaptability of each preset container according to the parameters of the subtask.

[0038] Step 230: Calculate a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask.

[0039] Step 240: Determine the minimum data computing capacity based on the training accuracy requirement; and determine whether the computing capacity corresponding to the container with the lowest adaptability is greater than the minimum data computing capacity.

[0040] Step 250: When the amount of data computation is less than the minimum amount, remove the container with the lowest adaptability, adjust the quantity upper limit constraint function based on the maximum parallel computing distribution, and recalculate the first sequence set of container types and corresponding quantities that are adapted to each subtask as the second sequence set.

[0041] Training accuracy is closely related to the number of parameters corresponding to each container. If the number of parameters is too small, the training accuracy requirement will not be met. Therefore, it is necessary to remove containers of this type and adjust the total number of containers corresponding to the maximum parallel computing condition to reduce the total number of constrained containers. For example, reducing the number N in the constraint function is used to adjust the container distribution corresponding to parallel computing. Using this method, the container type and corresponding number adapted for each subtask can be adjusted and optimized. Multiple sequence sets are regenerated as the second sequence set.

[0042] Step 260: Calculate the utilization efficiency of each container in the second sequence set, where the utilization efficiency is a comprehensive parameter calculated using hardware utilization and waiting time; obtain an optimal sorting combination based on the utilization efficiency of each container, and assign training tasks to the containers based on the optimal sorting combination.

[0043] This embodiment selects a second sequence set from the first sequence set that meets the training accuracy requirement, and specifically optimizes the process by determining a minimum data computation amount based on the training accuracy requirement; if the minimum data computation amount is less than the minimum data computation amount, removing the least suitable container, and adjusting the upper limit constraint function based on the maximum parallel operation distribution, recalculating the first sequence set with the container type and corresponding quantity suitable for each subtask as the second sequence set. Based on the distribution of the training data, the impact on training accuracy can be determined, and the upper limit constraint function can be adjusted to optimize the container type and corresponding quantity of the first sequence set suitable for each subtask to meet the training accuracy requirement.

[0044] Example 3 Figure 3 This is a schematic diagram of the structure of the model management resource pool dynamic allocation device provided in the third embodiment of the present invention. Figure 3 As shown, the device includes: An acquisition module 310 is used to calculate multiple subtasks of the model to be trained and obtain parameters of each subtask; A determination module 320 is configured to determine the duration adaptability of each preset container based on the parameters of the subtask; A calculation module 330 is configured to calculate a first sequence set of container types and corresponding quantities that are suitable for each subtask based on the duration adaptability of each preset container and the parameters of each subtask; A selection module 340 is configured to select a second sequence set that meets the training accuracy requirement from the first sequence set; A usage efficiency calculation module 350 is configured to calculate the usage efficiency of each container in the second sequence set, where the usage efficiency is a comprehensive parameter calculated using hardware usage and waiting time; The allocation module 360 ​​is configured to obtain an optimal sorting combination based on the utilization efficiency of each container and allocate training tasks to the containers based on the optimal sorting combination.

[0045] The model management resource pool dynamic allocation device provided in this embodiment calculates multiple subtasks of the model to be trained and obtains parameters for each subtask; determines the duration adaptability of each preset container based on the subtask parameters; calculates a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask; and selects a second sequence set from the first sequence set that meets training accuracy requirements. Calculate the utilization efficiency of each container in the second sequence set, where the utilization efficiency is a comprehensive parameter calculated using hardware utilization and waiting time; obtain the optimal sorting combination based on the utilization efficiency of each container, and assign training tasks to the containers based on the optimal sorting combination. By using the parameter of duration adaptability, the training duration adaptability of each type of container to a specific training task can be evaluated. Duration adaptability can be used to calculate multiple combinations of container types and corresponding quantities used for the training task, and based on the required training accuracy and utilization efficiency, a container combination that meets the utilization efficiency can be selected from multiple combinations to complete the training task together. Using the above method, the utilization efficiency and duration of the existing model can be taken into account, and the model training task can be completed quickly and efficiently. Without affecting other containers. This allows the server to complete model training quickly and efficiently using as few container resources as possible.

[0046] Based on the above embodiments, the determining module includes: an area calculation unit, configured to generate a training phase curve according to the parameters of each subtask and calculate the area of ​​a curve region enclosed by the training phase curve; a conversion unit, configured to convert the training phase curve into a corresponding horizontal line segment according to the area of ​​the curve region; An attribute acquisition unit, configured to acquire attributes of each container, including computing capability, loading capability, and communication capability between containers; A determination unit determines an average parameter value according to the horizontal line segment, and determines the duration adaptability of each container to the training task according to the average parameter value and the attributes of each container.

[0047] Based on the above embodiments, the determination unit is implemented in the following manner: , =max( )+ , in, = P / , = / , = / , The container calculation time corresponding to the average parameter amount, is the container reading time corresponding to the average parameter value, is the container communication duration corresponding to the average parameter value, P is the average parameter, is the amount of input data corresponding to the average parameter value, is the output data volume corresponding to the average parameter volume, The computing power of the container, is the loading capacity of the container, is the communication capability of the container, A is the adaptability of the container to the duration of the training task, is the dynamic reference value, The maximum processing time corresponding to the average parameter value of the container.

[0048] Based on the above embodiments, the calculation module includes: Deviation calculation unit, used to calculate the deviation between the corresponding point on the curve of each subtask in the training phase and the horizontal line segment; a duration adaptability determining unit, configured to determine the duration adaptability of each preset container to each subtask using the deviation; The selection unit is used to select container types in descending order based on the adaptability of each preset container to the duration of each subtask, and calculate whether the minimum number of containers of a single type can meet the subtask requirements; An establishment unit is used to establish a minimum completion time objective function when the requirement cannot be met, and to set a corresponding constraint function, wherein the constraint function includes: a quantity upper limit constraint function and a resource upper limit constraint function; The duration objective function is as follows: , The constraint function is: , ,in, j For subtasks, i To select container types in descending order of suitability, is the number of the i-th container corresponding to the j-th subtask, is the parameter amount of the j-th subtask, is the duration adaptability of the i-th container, is the computing power of the i-th container, is the output data volume of the jth subtask, is the communication capability of the i-th container, The amount of data input for the jth subtask, is the maximum number of containers of type i; The first sequence set obtaining unit is used to obtain a first sequence set of container types and corresponding quantities adapted to each subtask according to a minimum completion time objective function and a corresponding constraint function.

[0049] Based on the above embodiments, the selection module includes: A minimum data operation amount determination unit, configured to determine the minimum data operation amount of the container according to the training accuracy requirement; The removal unit is configured to determine whether the computing power corresponding to the container with the lowest duration adaptability is less than the minimum data computing amount. If the computing power is less than the minimum data computing amount, the container with the lowest adaptability is removed, and the quantity upper limit constraint function is adjusted according to the maximum parallel computing distribution. The first sequence set of container types and corresponding quantities adapted to each subtask is recalculated as the second sequence set.

[0050] Based on the above embodiments, the device further includes The probability determination module is used to determine the probability of a container being preempted or recycled based on the scenario type of the model application during the reverse calculation process; The estimation module is used to estimate the probability of rerouting based on communication fluctuations; The optimization module is used to obtain an inverse reuse coefficient based on the probability of a container being preempted / reclaimed and the probability of rerouting, and optimize the utilization efficiency of each container using the inverse reuse coefficient.

[0051] The model management resource pool dynamic allocation device provided in the embodiment of the present invention can execute the model management resource pool dynamic allocation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0052] Example 4 Figure 4 A schematic diagram of the structure of a server provided in Example 5 of the present invention. Figure 4 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0053] like Figure 4 As shown, server 12 is implemented as a general-purpose computing device. Components of server 12 may include, but are not limited to, one or more processors or processing units 16, memory 28, and a bus 18 that connects various system components (including memory 28 and processing unit 16).

[0054] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (I-Step A) bus, a Micro Channel Architecture (MAC) bus, an Enhanced I-Step A bus, a Video Electronics Standards Association (VE-Step A) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0055] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0056] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, usually called a "hard drive"). Although Figure 4 Although not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0057] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methodologies of the embodiments described herein.

[0058] The server 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the server 12, and / or any device that enables the server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). This communication may occur via an input / output (I / O) interface 22. Furthermore, the server 12 may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the server 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with the server 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0059] The processing unit 16 executes various functional applications and data processing by running the programs stored in the memory 28, such as implementing the model management resource pool dynamic allocation method provided in the embodiment of the present invention.

[0060] Example 5 The fifth embodiment of the present invention further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the model management resource pool dynamic allocation method provided in the above embodiment.

[0061] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0062] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0063] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0064] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Malltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0065] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for dynamically allocating a model management resource pool, characterized in that: include: Calculate multiple subtasks of the model to be trained and obtain the parameters of each subtask; Determine the duration adaptability of each preset container based on the subtask parameters; Calculate a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask; Selecting a second sequence set that meets the training accuracy requirement from the first sequence set; Calculating the utilization efficiency of each container in the second sequence set, where the utilization efficiency is a comprehensive parameter calculated using hardware utilization and waiting time; The optimal sorting combination is obtained based on the utilization efficiency of each container, and training tasks are assigned to the containers based on the optimal sorting combination.

2. The method according to claim 1, characterized in that Determining the duration adaptability of each preset container according to the subtask parameters includes: Generate a training phase curve based on the parameters of each subtask and calculate the area of ​​the curve region enclosed by the training phase curve; Converting the training phase curve into a corresponding horizontal line segment according to the area of ​​the curve region; Obtaining attributes of each container, including computing capability, loading capability, and communication capability between containers; An average parameter value is determined according to the horizontal line segment, and the duration adaptability of each container to the training task is determined according to the average parameter value and the attributes of each container.

3. The method according to claim 2, characterized in that The method of determining the average parameter value based on the horizontal line segment and determining the duration adaptability of each container to the training task based on the average parameter value and the attributes of each container is as follows: ， =max( )+ ,in, = P / , = / , = / , The container calculation time corresponding to the average parameter amount, is the container reading time corresponding to the average parameter value, is the container communication duration corresponding to the average parameter value, P is the average parameter, is the amount of input data corresponding to the average parameter value, is the output data volume corresponding to the average parameter volume, The computing power of the container, is the loading capacity of the container, is the communication capability of the container, A is the adaptability of the container to the duration of the training task, is the dynamic reference value, The maximum processing time corresponding to the average parameter value of the container.

4. The method according to claim 3, characterized in that The first sequence set of container types and corresponding quantities adapted to each subtask is calculated based on the duration adaptability of each preset container and the parameters of each subtask, including: Calculate the deviation between the corresponding point on the curve of each subtask during the training phase and the horizontal line segment; Determining the duration adaptability of each preset container to each subtask using the deviation; Based on the adaptability of each preset container to the duration of each subtask, container types are selected in descending order of adaptability, and the minimum number of containers of a single type is calculated to determine whether it can meet the subtask requirements. When the requirements cannot be met, a minimum completion time objective function is established, and corresponding constraint functions are set, including: a quantity upper limit constraint function and a resource upper limit constraint function; The duration objective function is as follows: ， The constraint function is: , , where j is the subtask, i is the container type selected in descending order of adaptability, is the number of the i-th container corresponding to the j-th subtask, is the parameter amount of the j-th subtask, is the duration adaptability of the i-th container, is the computing power of the i-th container, is the output data volume of the jth subtask, is the communication capability of the i-th container, The amount of data input for the jth subtask, is the maximum number of containers of type i; A first sequence set of container types and corresponding quantities adapted to each subtask is obtained according to the minimum completion time objective function and the corresponding constraint function.

5. The method according to claim 1, wherein The selecting a second sequence set that meets the training accuracy requirement from the first sequence set includes: Determine the minimum data computing amount of the container according to the training accuracy requirement; Determine whether the computing power corresponding to the container with the lowest duration adaptability is less than the minimum data computing amount. If it is less than the minimum data computing amount, remove the container with the lowest adaptability, adjust the quantity upper limit constraint function based on the maximum parallel computing distribution, and recalculate the first sequence set of container types and corresponding quantities that are adapted to each subtask as the second sequence set.

6. The method according to claim 5, characterized in that Before obtaining the optimal combination according to the utilization efficiency of each container and assigning subtask training tasks to the containers according to the optimal sorted combination, the method further includes: During the reverse calculation process, the probability of a container being preempted or recycled is determined based on the scenario type of the model application; Estimate the probability of rerouting based on communication fluctuations; According to the probability of a container being preempted / reclaimed and the probability of rerouting, an inverse multiplexing coefficient is obtained, and the utilization efficiency of each container is optimized using the inverse multiplexing coefficient.

7. A model management resource pool dynamic allocation device, characterized in that: include: The acquisition module is used to calculate multiple subtasks of the model to be trained and obtain the parameters of each subtask; A determination module, used to determine the duration adaptability of each preset container based on the parameters of the subtask; A calculation module, configured to calculate a first sequence set of container types and corresponding quantities that are compatible with each subtask based on the duration adaptability of each preset container and the parameters of each subtask; A selection module, configured to select a second sequence set that meets the training accuracy requirement from the first sequence set; A usage efficiency calculation module is configured to calculate the usage efficiency of each container in the second sequence set, where the usage efficiency is a comprehensive parameter calculated using hardware usage and waiting time; The allocation module is used to obtain the optimal sorting combination based on the utilization efficiency of each container and allocate training tasks to the containers based on the optimal sorting combination.

8. A server, characterized in that: The server includes: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the model management resource pool dynamic allocation method as described in any one of claims 1 to 6.

9. A storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the model management resource pool dynamic allocation method according to any one of claims 1 to 6.

Images