Deep learning environment resource elastic deployment method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a deep learning environment resource elastic deployment method and device, equipment and a medium. According to the method, the resource types possibly influencing the performance are considered, and the quantitative description of the distributed deep learning environment resources is constructed. An elastic transfer mode is defined according to a resource type, and a transfer mode can be selected according to a current resource environment decision. Aiming at a selection mode during resource scheduling, a resource allocation matrix is constructed based on a current resource type, a proper amount of services are selected according to a strategy to construct a compatible linear equation set, a method of a small norm positive value solution is provided for solving, and the transfer quantity of candidate service calculation storage network resources is obtained. State storage is carried out on the model level, the restarting efficiency is optimized, and resources are saved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for elastically deploying resources in a deep learning environment. Background Art

[0002] Distributed deep learning is a technology that uses multiple computing nodes to process data and model training in parallel. On large-scale data sets, training models on a single machine can be very time-consuming, and distributed learning can significantly improve training speed and processing power. In business scenarios, training tasks usually have clear completion time requirements. When new training tasks are added, the system needs to be able to flexibly deploy computing resources to adapt to the changing task load and priority, ensuring that all tasks can be completed before the scheduled deadline. When implementing elastic distributed learning, the key is to maintain an efficient data communication mechanism, save data in an appropriate manner for each computing node, and ensure data consistency and synchronization between computing nodes. In addition, a reasonable task scheduling strategy is also essential, which requires dynamically adjusting task queues and execution plans based on task priorities and resource availability. With these advanced features, the distributed learning framework can better adapt to rapidly changing business needs, improve resource utilization, and reduce the waste of computing resources.

[0003] The current distributed learning mainly adopts the GPU resource model and static rule-driven resource method in resource selection and scheduling. Elastic deployment often relies on the framework to perform periodic state preservation. Among them, the GPU resource model refers to the model that uses "GPU" as a resource, which is currently widely used in most laboratories. All computing nodes can uniformly manage and schedule GPU resources to efficiently perform parallel computing. This method has a more intuitive management and scheduling logic, but in the actual training process, CPU, memory, including hard disk, IO performance, etc. will affect the training efficiency of distributed learning, so the GPU resource model will less accurately represent resource requirements. Only using GPU as a resource scheduling unit also leads to a weaker response to resource changes. Static rule-driven resources allocate resources based on preset rules or strategies, which can well ensure the consistency and predictability of resource allocation, but it is difficult to adapt to rapidly changing needs and environments, and is not flexible enough to respond to dynamically changing workloads. The response to new tasks or emergencies is usually not fast enough, which ultimately affects the execution efficiency of tasks. The method of periodic state preservation saves the state of the model at the end of each epoch of deep learning for subsequent recovery or continued training. The model state at the end of each epoch training is consistent, which is convenient for tracking and comparing the performance of different training stages. However, since each epoch may take a long time in the actual training process, it may not be saved frequently in a long period of time, resulting in a large amount of progress being lost in the event of a system failure. It is not suitable for scenarios that change rapidly or require fine-grained recovery. The above three methods provide certain solutions for distributed deep learning in current projects, but the one-sidedness of resource selection makes it difficult to cope with an environment where resources change rapidly. At the same time, traditional resource scheduling solutions are difficult to cope with dynamically changing workloads and requirements, and there is a certain risk of data loss during elastic recovery.

[0004] To sum up, in the face of the above problems, building a reasonable resource selection and allocation system, realizing accurate and fast resource scheduling when distributed resources change, and improving fine-grained elastic preservation and recovery mechanisms are key technical issues that urgently need to be studied and solved. Summary of the invention

[0005] In response to the above problems, the present invention provides a method, device, equipment and medium for elastic deployment of resources in a deep learning environment, which are used to realize elastic resource selection and elastic resource scheduling in the distributed deep learning process, optimize breakpoint deployment during elastic training, and save more resources.

[0006] In a first aspect, a method for elastically deploying resources in a deep learning environment is provided, the method comprising:

[0007] Distributed deep learning environment resources are divided into computing resources C, storage resources S and network resources N;

[0008] When resources are transferred, several services are selected together with three types of resources, C, S, and N, to form a resource allocation matrix A;

[0009] According to the scheduler, the three resource combination matrices b of C, S, and N required for the newly added services are obtained, and the service proportion composition matrix x of the several services shared is set, and the resource transfer problem is converted into a process of solving the minimum norm positive value solution of the compatible linear equation system Ax=b, and the matrix x obtained when the minimum norm is obtained is used as the resource allocation scheme coefficient to obtain the resource allocation scheme;

[0010] Deploy distributed deep learning environment resources according to the resource allocation plan.

[0011] Furthermore, the computing resources include: CPU central processing unit and GPU graphics processing unit;

[0012] Storage resources include: data storage devices and media;

[0013] Network resources include: devices that support communication between devices, and links of various bandwidths.

[0014] Furthermore, a number of services and three types of resources, C, S, and N, are selected to form a resource allocation matrix, including:

[0015] Select several services based on the current service usage or select several services based on the service usage duration.

[0016] Furthermore, four businesses and three types of resources are selected to form a compatible linear equation group.

[0017] Furthermore, the resource allocation matrix includes:

[0018] In the matrix, each row represents computing resources, storage resources, and network resources from top to bottom, and each column represents a business in the data center.

[0019] Furthermore, the minimum norm is obtained by solving, including:

[0020] A least squares solution is employed to find the minimum norm solution by using the pseudo-inverse of the matrix.

[0021] Furthermore, the matrix x with the minimum norm is solved as a resource allocation solution, including:

[0022] The resource matrix of the four businesses is represented by a vector v i 、v r 、v j 、v q ;

[0023] By calculating each vector v i 、v r 、vj 、v q The norm of each vector is taken as the ratio of the norm of each vector to the sum of the norms of all vectors as the normalization coefficient β i , β r , β j , β q , with the normalization coefficient β i , β r , β j , β q Composed matrix x;

[0024] By multiplying the normalized coefficients in the matrix x with the original data of the selected four services respectively, the allocated resources of each resource of each service are obtained as the resource allocation plan.

[0025] In a second aspect, a deep learning environment resource elastic deployment device includes: a modeling unit, a solution unit, and a deployment unit;

[0026] A modeling unit, used to divide the distributed deep learning environment resources into computing resources C, storage resources S and network resources N;

[0027] The modeling unit is also used to select a number of services and three types of resources C, S, and N to form a resource allocation matrix A when resources are transferred; obtain the three types of resource combination matrix b of C, S, and N required by the new service according to the scheduler, set the service proportion composition matrix x shared by the several services, and convert the resource transfer problem into a process of solving the minimum norm positive value solution of the compatible linear equation system Ax=b;

[0028] A solving unit, used for solving the matrix x when the minimum norm is obtained as a resource allocation scheme coefficient to obtain a resource allocation scheme;

[0029] The deployment unit is used to deploy distributed deep learning environment resources according to the resource allocation plan.

[0030] In a third aspect, a distributed deep learning environment resource elasticity training method includes: using the above-mentioned deep learning environment resource elasticity deployment method to obtain a resource allocation plan;

[0031] When distributed deep learning training is abnormal or needs to be interrupted, save the model layer status reached by the current training;

[0032] When retraining, based on the resource allocation plan, the saved model state is restored to continue training.

[0033] In a fourth aspect, a distributed deep learning environment resource elastic training device includes: a resource solution acquisition unit, a training state storage unit, and a training recovery unit;

[0034] A resource scheme acquisition unit, configured to obtain a resource allocation scheme by adopting a method for elastic deployment of resources in a deep learning environment as described in any one of claims 1 to 7;

[0035] The training state saving unit is used to save the model layer state reached by the current training when the distributed deep learning training is abnormal or needs to be interrupted;

[0036] The training recovery unit is used to restore the saved model state to continue training based on the resource allocation plan during retraining.

[0037] In a fifth aspect, an electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0038] a memory storing a computer program;

[0039] The processor is used to implement the above-mentioned method for elastic deployment of resources in a deep learning environment when executing a computer program stored in the memory.

[0040] In a sixth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for elastic deployment of resources in a deep learning environment.

[0041] The present disclosure has at least the following beneficial effects:

[0042] This disclosure considers the types of resources that may affect performance and constructs a quantitative description of the resources in a distributed deep learning environment. The elastic transfer mode is defined according to the resource type, and the transfer mode can be selected according to the current resource environment decision. Regarding the selection method for resource scheduling, a resource allocation matrix is ​​constructed based on the current resource type, and a compatible linear equation group is constructed based on the appropriate amount of business selected according to the strategy. A method with a smaller norm positive solution is proposed for solving the problem, and the transfer quantity of candidate business computing storage network resources is obtained. State is saved at the model level to optimize restart efficiency and save resources.

[0043] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a schematic diagram of the elastic deployment method flow in an embodiment of the present disclosure;

[0046] Figure 2 This is a schematic diagram of the structure of the elastic deployment device according to an embodiment of the present disclosure;

[0047] Figure 3 This is a schematic diagram of the structure of the elasticity training device according to an embodiment of the present disclosure;

[0048] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure;

[0049] Figure 5 A schematic diagram for quantitatively describing resources in a distributed deep learning environment according to an embodiment of the present disclosure;

[0050] Figure 6 A schematic diagram of an optional elastic transfer mode for business changes in an embodiment of the present disclosure;

[0051] Figure 7 A schematic diagram of a method for selecting elastic transfer resources in an embodiment of the present disclosure;

[0052] Figure 8 A schematic diagram of selecting resources for solving a small norm positive value solution in an embodiment of the present disclosure;

[0053] Fig. 9 It is a flowchart of a breakpoint retraining method based on model level in an embodiment of the present disclosure;

[0054] Fig.10 It is a schematic diagram of resource changes when elastically adding services in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0056] like Figure 1As shown, a method for elastically deploying resources in a deep learning environment comprises:

[0057] S101, dividing the distributed deep learning environment resources into computing resources C, storage resources S and network resources N;

[0058] S102, when resources are transferred, a number of services are selected together with three types of resources, C, S, and N, to form a resource allocation matrix A;

[0059] S103, obtaining the C, S, and N resource combination matrix b required by the new service from the scheduler, setting the service ratio matrix x shared by the several services, converting the resource transfer problem into a process of solving the minimum norm positive value solution of the compatible linear equation system Ax=b, and solving the minimum norm matrix x as the resource allocation solution coefficient to obtain the resource allocation solution;

[0060] S104: Deploy distributed deep learning environment resources according to the resource allocation plan.

[0061] The specific implementation is as follows:

[0062] Source elastic transfer representation method:

[0063] A unified representation method for elastic resources includes the following steps:

[0064] Classify resources and construct a quantitative description of resource types; define four types of elasticity transfer modes, including elasticity increase, elasticity maintenance, elasticity reduction, and mixed.

[0065] Resources are divided into three categories, namely computing resources, storage resources, and network resources. In this disclosure, C, S, and N represent the three categories of resources. Computing resources refer to processor resources such as CPU central processing units and GPU graphics processing units. Storage resources refer to devices and media for data storage, including hard disk drives, memory, etc. Network resources refer to devices that support communication between devices, including links of various bandwidths. Based on the above resource classification, the resources of the distributed deep learning environment are quantitatively described. For distributed deep learning of various scales across domains, the values ​​of various resources are expressed as the sum of resources of different data centers. For example, for an interconnected data center built across a wide area network containing three data centers M, P, and Q, its computing resources are C=C M +C P +C Q .

[0066] The elastic transfer mode is divided into four categories: elastic increase EI, maintenance KE, elastic reduction ED, and mixed HE. Elastic increase refers to the situation where all three types of resources C, S, and N are increased, maintenance refers to the situation where all three types of resources C, S, and N remain unchanged, elastic reduction refers to the situation where all three types of resources C, S, and N are reduced, and mixed includes the situation where all three types of resources C, S, and N are not changed uniformly. By classifying the above transfer modes, it is effective to deal with resource changes encountered in the distributed learning process, including new business, completion of old business, addition of new business, and abnormal resources. For example, when a new business is added to the controller, if the remaining resources allocated to the new business are insufficient, a decision can be made. If other businesses are currently allowed to allocate resources according to the selection strategy, the resources of other businesses can be reduced by elastic reduction or elastic mixing to free up available resources for the new business; if the priority of other businesses is higher according to the current strategy, the amount of resources of other businesses will remain unchanged, and after the business is completed and the resources are released, the resource allocation will be re-decided. Through clear resource classification and elastic transfer mode, the present disclosure simplifies the complexity of resource management and makes the system easier to expand and maintain. System administrators can make decisions quickly based on predefined patterns, reducing errors and delays.

[0067] Method for selecting elastic transfer resources:

[0068] A method for selecting elastic transfer resources includes the following steps:

[0069] All businesses and resources together form a resource allocation matrix; four businesses are selected through decision-making to form a compatible linear equation group with resources; a smaller norm positive value solution method is proposed to solve the equation group.

[0070] The proposed method is used to solve the problem of resource selection and coordination of computing storage network resources during resource scheduling. In the face of the complex and changeable resource changes in distributed learning, when elastic resources are transferred, the changed resources are screened and the resource quantity is selected. This disclosure forms a resource allocation matrix with existing services and three types of resources: C, S, and N, which is formally described as:

[0071]

[0072] In the matrix, each row represents computing resources, storage resources, and network resources from top to bottom, and each column represents a business in the data center. After constructing the resource allocation matrix, since the resource adjustment process will suspend the current business, to ensure that all computing nodes can update their status synchronously, there should not be too many businesses involved in the resource adjustment. The reason for selecting four businesses instead of fewer is to make the business and the three types of resources form a compatible linear equation system, in order to convert it into a problem of finding a minimum norm positive solution. The selection strategy of the four businesses can be selected according to demand, for example, it can be selected according to the current business occupancy, or it can be selected according to the business occupancy duration. In the present disclosure, after selecting four businesses according to the occupancy of business resources, the following resource matrix is ​​obtained:

[0073]

[0074] Since the computing, storage, and network resources required for the new business can be obtained through the orchestrator, b 、s b 、n b , set the business proportion of the four businesses to be shared as x i 、x r 、x j 、x q , then the problem is transformed into the process of solving the minimum norm positive solution of the compatible linear equations Ax = b, that is:

[0075]

[0076] There is a traditional method for solving the minimum norm positive solution of the compatible linear equations, namely the least squares solution method, which uses the pseudo-inverse of the matrix to find the minimum norm solution. However, there are two problems with the usage scenario of this question. One is that negative values ​​or values ​​greater than 1 may appear, which is meaningless in engineering applications. The other is that the solution process is relatively complicated, which is not conducive to engineering implementation. The present disclosure is oriented to business applications and comprehensively solves from the perspective of the entire process of multiple businesses. It does not require the minimum norm in a strict sense, and proposes a method for selecting resources for elastic transfer of smaller norm positive solutions. First, the resource matrix of the four businesses is represented by a vector, namely:

[0077]

[0078] The above is done by calculating each vector v i 、v r 、v j 、v q The norm of each vector is taken as the ratio of the norm of each vector to the sum of the norms of all vectors as the normalization coefficient Right now:

[0079]

[0080] By multiplying these normalized coefficients with the original data of the four selected businesses, the optimal allocation of resources for each business and each resource can be obtained. The calculation process is as follows:

[0081]

[0082]

[0083] In the formula, Indicates the adjustment amount of service i to resource c, and thus calculates the transfer quantity of the four candidate services on resources C, S, and N. This method is the best solution in terms of results. By using the normalization coefficient to adjust the weight of each resource, it can ensure that the importance of each part in resource allocation is properly reflected, thereby achieving a more balanced and efficient resource utilization. While maintaining data consistency, resources are effectively allocated and optimized. In addition, this method can also flexibly adjust resource allocation in a dynamically changing environment, improve the elasticity and responsiveness of the system, and is particularly suitable for scenarios that need to quickly adapt to new conditions and requirements, such as resource management in cloud computing and network services.

[0084] Breakpoint retraining method based on model level:

[0085] A breakpoint retraining method based on model level includes the following steps:

[0086] Modify the granularity of state preservation during elastic restart so that it can be preserved at the model level. Optimize the pause and start scheme during elastic restart to make full use of resources.

[0087] The elastic restart scheme currently widely used in the market takes the training data set as the object. Each breakpoint restart requires retraining all the data sets in an epoch, which will lead to repeated data calculation, especially when the epoch is long, which will waste a lot of computing resources. The present invention proposes an elastic training method with the structural layer of the model as the restart object. When the training is abnormal or needs to be interrupted, the state of the model layer reached by the current training is saved. When retraining, the model state at this level is restored to save training resources. Taking the model layer structure of ResNet50 as an example, its hierarchical structure includes convolutional layer, batch normalization layer, activation layer (ReLU), pooling layer (maximum pooling and average pooling), and fully connected layer.

[0088] Since the ResNet50 network structure is deep and complex, and the training process is time-consuming and resource-intensive, it is very beneficial to save the state at the layer level on ResNet50. When encountering node failures, resource changes, or other situations that require adjustment, by saving the state in each residual block or specific layer, it can be quickly restored when hardware failures or other interruptions occur during training, reducing the need for repeated calculations. In addition, layer-level granular state preservation can also help debug and optimize the network, allowing researchers to observe and analyze the intermediate outputs of each layer and better understand the behavior and performance bottlenecks of the model. This approach improves the flexibility and efficiency of the training process, especially when performing large-scale training using expensive computing resources.

[0089] At the same time, when the elastic resource is powered off and saved and restarted, since the initialization of new business resources and the preservation of old business information will take time, the present disclosure proposes to overlap this part of time, effectively saving the waiting time for the elastic restart.

[0090] This disclosure defines resource types and provides a quantitative description of resources in a distributed deep learning environment, which simplifies the complexity of resource management. It also defines four types of elastic transfer modes, which simplifies the complexity of resource management and makes the system easier to expand and maintain.

[0091] This method proposes a method for selecting elastically transferred resources, which forms a resource allocation matrix with all businesses and resources; four businesses are selected through decision-making to form a compatible linear equation group with resources; at the same time, a method of smaller norm positive value solution is proposed to solve the compatible linear equation group, which can effectively handle resource competition and coordination among multiple businesses, simplify the calculation process, improve calculation efficiency, and make it easier to implement in engineering.

[0092] The present disclosure proposes a breakpoint retraining method based on the model level, which modifies the state preservation granularity during the elastic restart process so that it can save the state at the model level granularity. By saving the state in each residual block or specific layer, it can quickly recover when hardware failure or other interruptions occur during the training process, reducing the need for repeated calculations.

[0093] like Figure 2 As shown, a deep learning environment resource elastic deployment device includes: a modeling unit 201, a solution unit 202 and a deployment unit 203;

[0094] A modeling unit 201, used to divide the distributed deep learning environment resources into computing resources C, storage resources S and network resources N;

[0095] The modeling unit 201 is also used to select a number of services and three types of resources C, S, and N to form a resource allocation matrix A when resources are transferred; obtain the three types of resource combination matrix b of C, S, and N required by the newly added service according to the scheduler, set the service proportion composition matrix x shared by the several services, and convert the resource transfer problem into a process of solving the minimum norm positive value solution of the compatible linear equation system Ax=b;

[0096] A solving unit 202, used for solving the matrix x when the minimum norm is obtained as a resource allocation scheme coefficient to obtain a resource allocation scheme;

[0097] The deployment unit 203 is used to deploy distributed deep learning environment resources according to the resource allocation plan.

[0098] A distributed deep learning environment resource elastic training method, comprising: using the above-mentioned deep learning environment resource elastic deployment method to obtain a resource allocation plan;

[0099] When distributed deep learning training is abnormal or needs to be interrupted, save the model layer status reached by the current training;

[0100] When retraining, based on the resource allocation plan, the saved model state is restored to continue training.

[0101] like Figure 3 As shown, a distributed deep learning environment resource elastic training device includes: a resource solution acquisition unit 301, a training state storage unit 302 and a training recovery unit 303;

[0102] A resource scheme acquisition unit 301, configured to obtain a resource allocation scheme by adopting a method for elastically deploying resources in a deep learning environment as described in any one of claims 1 to 7;

[0103] The training state saving unit 302 is used to save the model layer state reached by the current training when the distributed deep learning training is abnormal or needs to be interrupted;

[0104] The training recovery unit 303 is used to recover to the saved model state and continue training based on the resource allocation scheme during retraining.

[0105] like Figure 4 As shown, the present disclosure provides an electronic device, including a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 communicate with each other through the communication bus 404;

[0106] Memory 403, storing computer programs;

[0107] The processor 401 is used to implement the above-mentioned method for elastic deployment of resources in a deep learning environment when executing the computer program stored in the memory 303.

[0108] The present disclosure provides a computer-readable storage medium storing a computer program, which implements the above-mentioned method for elastic deployment of resources in a deep learning environment when executed by a processor.

[0109] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0110] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0111] In order to enable those skilled in the art to better understand the present disclosure, the principles of the present disclosure are described as follows in conjunction with the accompanying drawings:

[0112] The problem to be solved by the present disclosure is: when dealing with resource management and elastic deployment in large-scale distributed systems, how to effectively handle dynamic changes in resources, quickly respond to changes in system requirements, and ensure efficient operation and data consistency of the system. The present disclosure proposes a new unified representation method for elastic resources, and provides a classification method based on resource types and elastic transfer modes, which is used to simplify the complexity of resource management and improve the scalability and maintainability of the system. In view of the multi-service competition and coordination problems in resource allocation, the present disclosure proposes a method for selecting elastic transfer resources, which optimizes resource allocation and achieves better engineering implementation by constructing a resource allocation matrix and solving a smaller norm positive solution of a compatible linear equation group. In addition, for the data loss generated when the state is saved during the resource change process, the present disclosure also proposes a breakpoint retraining method based on the model level, which can quickly restore the training process in the event of a failure by saving the state at each model level, reducing the need for repeated calculations, thereby improving the system elasticity and reliability. These innovations together constitute a comprehensive solution covering resource representation, resource selection and state preservation, providing a new methodology and technical support for resource management of large-scale distributed systems.

[0113] Embodiment 1:

[0114] In order to achieve a unified representation of multiple resources for distributed learning and an accurate definition of elastic resource allocation decisions, this paper divides resources into three categories: computing resources, storage resources, and network resources, constructs an elastic resource coordinate system, and defines four types of elastic transfer modes to meet various resource changes in engineering environments.

[0115] The following is a specific example to explain in detail the construction process of the distributed resource system. In the cross-domain network interconnected data center composed of three data centers P, Q, and M, 25 virtual machines in the cross-domain network interconnected data center are running 5 distributed services. A single domain controller is deployed in each data center, and they are connected to each other by a wide area network. The business will always reflect the current resource status to the super domain controller through the single domain controller. The computing, storage, network resource volume and total business resource volume of each business can be viewed in the super domain controller. The resource information obtained by viewing the super domain controller once is described in Table 1:

[0116] Table 1

[0117] Resource Type Business 1 Business 2 Business Three Business Four Business Five Total resources Computing resources (GFLOP / S) 627 392 518 363 223 2123 Storage resources (GB) 242 148 197 186 132 905 Network resources (Gbps) 6.5 3.8 4.2 4.6 3.2 22.3

[0118] Table 1 contains four rows. The first row represents each business and total resources. The next three rows represent computing resources, storage resources, and network resources. By representing resources in three-dimensional space, operators can monitor the resource status of the data center through the super domain controller. Figure 5As shown, the operator checks the total resource status at time t on the super domain controller and can usually decide on the resource change strategy based on the resource status and business changes at that time.

[0119] Corresponding response strategies can be set for different resource situations. This article defines four elastic modes based on the different change trends of the three types of resources. The above information is described in the elastic transfer mode definition table 2. Table 2 contains four rows. The first three rows represent the changes of the three types of resources, C, S, and N, respectively. The fourth row is the elastic mode corresponding to the changes of the three types of resources. When the business changes, you can select an elastic strategy based on the current resource status and business changes, such as Figure 6 As shown. For example, if a user wants to insert a new RESNET34 service, the orchestrator calculates that the network resources required for the new service are 667GFLOPS / s, the storage resources are 232GB hard disks, and the network resources are 7.2Gbps. However, the current total computing resources have a surplus of less than 667GFLOPS / s, which cannot provide sufficient resources for the new service. Then, three decisions can be made based on the current resources: elastic reduction, mixed elasticity, and maintenance. A certain priority can be set to make specific decisions. For example, if the priority of the currently running service is higher, the current resources will be kept unchanged, and the decision on whether to add the new service will be made again after the service is completed and the resources are released; if the priority of the new service is higher, the resources of other current services can be provided to the RESNET34 service by elastic reduction or mixed elasticity reduction.

[0120] Table 2

[0121]

[0122]

[0123] Embodiment 2:

[0124] like Figure 7 As shown, this embodiment provides a scheduling solution for services when resources change, and the overall steps are as follows:

[0125] S701, collecting new business requirements, the orchestrator completes business orchestration, and obtains computing, storage, and network resources required for the new business;

[0126] S702, obtain the running service resources from the super domain controller, construct a resource allocation matrix with the service as the column vector and the resource as the row vector, select four services from it according to the priority strategy, and form a compatible linear equation group with the allocated resource amount and the resources required for the new service;

[0127] S703, calculating the l-norm of each column vector in the resource allocation matrix, taking the ratio of the l-norm of each vector to the sum of the l-norms of all vectors as the normalization coefficient of each service, and multiplying the coefficient by the service resource to obtain a preferred allocation value of each resource for each service;

[0128] S704, according to the resource allocation value, the greedy algorithm is used to calculate the number of virtual machines actually allocated, and the single domain controllers of each data center are scheduled by the super domain controller for elastic adjustment.

[0129] Taking the case of inserting a new service of RESNET34 when running five distributed services in a cross-domain network interconnected data center, the resource selection scheme proposed in the present disclosure is described in detail:

[0130] In the above S701, the newly added RESNET34 service is firstly orchestrated by the orchestrator. The service uses the ImageNet dataset, runs 20 epochs, and is required to complete the task within 30 hours. The batch_size is 512. Through orchestration calculation, it is found that the computing resources required for the service are 667 GFLOPS / s, the storage resources are 232 GB, and the network resources are 7.2 Gbps.

[0131] In the above S702, the duration of all current services can be viewed through the super domain controller, as shown in the following table:

[0132] Table 3

[0133]

[0134] Table 3 contains two rows, which represent the model type and occupancy time of the business. In order to reduce the impact of resource changes and make the business and the three types of resources form a compatible linear equation system, four businesses are selected as the purpose of resource changes. Figure 8 As shown, this example selects services that take up a longer time to form the resource allocation matrix of the changing resources, that is, services 1, 2, 3, and 4 are selected as the resources required for the new service allocation. The resource allocation matrix D is constructed based on the computing, storage, and network resources of services 1, 2, 3, and 4. The matrix elements are as follows:

[0135]

[0136] In the above S703, after obtaining the resource allocation matrix, the ratio of the norm of each service to the sum of the norms of all services is calculated as the normalization coefficient of resource allocation. According to the smaller norm solution method proposed in the present disclosure, the following can be calculated:

[0137] ‖v1‖1=|c1|+|s1|+|n1|=875.5

[0138] ‖v2‖1=|c2|+|s2|+|n2|=543.8

[0139] ‖v3‖1=|c3|+|s3|+|n3|=719.2

[0140] ‖v4‖1=|c4|+|s4|+|n4|=553.6

[0141]

[0142]

[0143] In the above formula, the one-norm of each service is first calculated to obtain the weight of its allocated service. Then, according to the one-norm ratio, the resource share allocated to each service is 0.325, 0.201, 0.267, and 0.205. The resource allocation amount is obtained by the smaller norm positive value solution method as follows:

[0144]

[0145] According to the above formula, service one allocates 216.590 GFLOPS / s of computing resources, 75.449 GB of storage resources and 2.341 Gbps of network resources to the new service; service two allocates 134.531 GFLOPS / s of computing resources, 46.864 GB of storage resources and 1.454 Gbps of network resources to the new service; service three allocates 177.923 Gflop / s of computing resources, 61.979 GB of storage resources and 1.923 Gbps of network resources to the new service; service four allocates 136.955 Gflop / s of computing resources, 47.708 GB of storage resources and 1.481 Gbps of network resources to the new service. Since the computing resources of a standard resource configuration of a virtual machine in the laboratory are 80GFLOP / S, the storage resources are 40GB, and the network resources are 1Gbps, the greedy algorithm can be used to calculate that business one allocates 3 virtual machines to the new business, business two allocates 2 virtual machines to the new business, business three allocates 2 virtual machines to the new business, and business four allocates 2 virtual machines to the new business.

[0146] Embodiment 3:

[0147] like Fig. 9 As shown, this embodiment provides a breakpoint retraining method based on the model level in elastic deployment, and the steps of elastic restart are as follows:

[0148] S901, after obtaining the resource adjustment amount of each service to be changed through the elastic transfer resource selection method, the super domain controller notifies the single domain controllers on each data center to plan the virtual machine allocated to the new service, and determines that the virtual machine allocated to the new service will be initialized to prepare for the addition of the new service;

[0149] S902, the single domain controller notifies the unallocated virtual machine to store the optimizer status, model parameters, and training status information of the model layer operation matrix for the current business training;

[0150] S903, the single domain controller suspends all services and adjusts the resource status according to the virtual machine planned to be allocated to the new service in S901;

[0151] S904, after the resource adjustment is completed, the single domain controller starts a new service and resumes the suspended service. The super domain controller can check that the service can be restarted and run normally.

[0152] With the surge in the number of parameters of modern deep learning models and the expansion of data sets, training an epoch requires a lot of computation. Especially when dealing with high-dimensional data and complex network structures, it is difficult to complete the training process quickly even with high-performance computing resources, and elastic restart will greatly increase the risk of computing power loss and data loss. By further refining the granularity of state preservation, the model-level breakpoint retraining method can effectively address these challenges. This method allows state preservation at each independent model level. At the same time, it also allows more accurate positioning of the problem layer when an abnormal interruption occurs or debugging is required.

[0153] The following is a specific example of inserting a new ResNet34 service in the laboratory. During the insertion of the new service, the changes in virtual machine resources are as follows: Fig.10 In the figure, the original business 1 is trained by eight virtual machines VM1 to VM8, business 2 is trained by five virtual machines VM9 to VM13, business 3 is trained by seven virtual machines VM14 to VM20, and business 4 is trained by five virtual machines VM21 to VM25. After learning the amount of resource changes through the second method disclosed in the present invention, the super domain controller notifies the single domain controllers on each data center to select part of the resources of the business to be changed and allocate them to the new business. During the selection process, business training will not stop, which can save a certain amount of training resources.

[0154] The virtual machines that are determined to be allocated will be in the initialization state to prepare for the addition of new services. Unallocated virtual machines save the current training status of the model level granularity. For example, worker 1 as service 1 is running on VM1, and is currently running to the 474th batch of the 15th epoch. In the forward propagation of the current batch, the operation of the maximum pooling layer of ResNet50 is completed, and the result of the operation is a 512*64*56*56 matrix; then the current VM1 will save the training optimizer state, model parameters, and training status information down to the model layer operation matrix. Fig.10 As shown, the time for initializing the new business virtual machine overlaps with the time for saving the state of the original business virtual machine, which can save the time for restarting resources.

[0155] After all the information of unassigned virtual machines is saved, all business operations are suspended. The previously selected virtual machines to be assigned by the super domain controller will be assigned to the new business at this time. In this example, VM6-8, VM12-13, VM19-20, and VM24-25 are assigned to the new business for business 1, 2, 3, and 4, respectively. After the resource adjustment is completed, the super domain controller will notify the single domain controller to start the new business and the suspended business together. The suspended business will read the stored status and continue to run on the model layer of the original batch. For example, VM1 will resume the operation from the next layer of the maximum pooling layer in the 474th batch of the 15th epoch. The running status of the business can be viewed on the super domain controller. The information shows that the accuracy and loss value are not greatly affected before and after the business restart.

[0156] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for elastically deploying resources in a deep learning environment, characterized in that: The method comprises: Distributed deep learning environment resources are divided into computing resources C, storage resources S and network resources N; When resources are transferred, several services are selected together with three types of resources, C, S, and N, to form a resource allocation matrix A; According to the scheduler, the three resource combination matrices b of C, S, and N required for the newly added services are obtained, and the service proportion composition matrix x of the several services shared is set, and the resource transfer problem is converted into a process of solving the minimum norm positive value solution of the compatible linear equation system Ax=b, and the matrix x obtained when the minimum norm is obtained is used as the resource allocation scheme coefficient to obtain the resource allocation scheme; Deploy distributed deep learning environment resources according to the resource allocation plan.

2. A method for elastically deploying resources in a deep learning environment according to claim 1, characterized in that: Computing resources include: CPU central processing unit and GPU graphics processing unit; Storage resources include: data storage devices and media; Network resources include: devices that support communication between devices, and links of various bandwidths.

3. A method for elastically deploying deep learning environment resources according to claim 1, characterized in that: Select several businesses and three types of resources, C, S, and N, to form a resource allocation matrix, including: Select several services according to the current service usage, or select several services according to the service usage duration; Select four businesses and three types of resources to form a compatible linear equation system; Resource allocation matrix, including: In the matrix, each row represents computing resources, storage resources, and network resources from top to bottom, and each column represents a business in the data center.

4. A method for elastically deploying resources in a deep learning environment according to claim 3, characterized in that: Solving the matrix x with the minimum norm as the resource allocation solution includes: The resource matrix of the four businesses is represented by a vector v i 、v r 、v j 、v q ; By calculating each vector v i 、v r 、v j 、v q The norm of each vector is taken as the ratio of the norm of each vector to the sum of the norms of all vectors as the normalization coefficient β i , β r , β j , β q , with the normalization coefficient β i , β r , β j , β q Composed matrix x; By multiplying the normalized coefficients in the matrix x with the original data of the selected four services respectively, the allocated resources of each resource of each service are obtained as the resource allocation plan.

5. A method for elastically deploying resources in a deep learning environment according to claim 1, characterized in that: Solving for the minimum norm includes: A least squares solution is employed to find the minimum norm solution by using the pseudo-inverse of the matrix.

6. A deep learning environment resource elastic deployment device, characterized in that: include: Modeling unit, solving unit and deployment unit; A modeling unit, used to divide the distributed deep learning environment resources into computing resources C, storage resources S and network resources N; The modeling unit is also used to select a number of services and three types of resources C, S, and N to form a resource allocation matrix A when resources are transferred; obtain the three types of resource combination matrix b of C, S, and N required by the new service according to the scheduler, set the service proportion composition matrix x shared by the several services, and convert the resource transfer problem into a process of solving the minimum norm positive value solution of the compatible linear equation system Ax=b; A solving unit, used for solving the matrix x when the minimum norm is obtained as a resource allocation scheme coefficient to obtain a resource allocation scheme; The deployment unit is used to deploy distributed deep learning environment resources according to the resource allocation plan.

7. A distributed deep learning environment resource elastic training method, characterized in that: include: A resource allocation scheme is obtained by adopting a method for elastic deployment of resources in a deep learning environment as described in any one of claims 1 to 5; When distributed deep learning training is abnormal or needs to be interrupted, save the model layer status reached by the current training; When retraining, based on the resource allocation plan, the saved model state is restored to continue training.

8. A distributed deep learning environment resource elastic training device, characterized in that: include: Resource solution acquisition unit, training state preservation unit and training recovery unit; A resource scheme acquisition unit, configured to obtain a resource allocation scheme by adopting a method for elastically deploying resources in a deep learning environment as described in any one of claims 1 to 5; The training state saving unit is used to save the model layer state reached by the current training when the distributed deep learning training is abnormal or needs to be interrupted; The training recovery unit is used to restore the saved model state to continue training based on the resource allocation plan during retraining.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a memory storing a computer program; A processor, used to implement a method for elastically deploying resources in a deep learning environment as described in any one of claims 1 to 5 when executing a computer program stored in a memory.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for elastically deploying resources in a deep learning environment as described in any one of claims 1-5 is implemented.

Images