Multi-resource scheduling method and system for model training and terminal

Abstract

The invention relates to the field of artificial intelligence, and discloses a multi-resource scheduling method and system for model training and a terminal.The method comprises the steps that a training resource management server classifies model training resources, and the training resources are divided into variable training resources and supporting training resources; variable training resource request information sent by a request terminal is obtained, a corresponding support training resource is matched according to the variable training resource request information, a first support relationship between the variable training resource and the support training resource is obtained, model training information in the variable training resource request information is obtained, and a training resource support type is obtained; and the training resource management server performs corresponding scheduling on the variable training resources and the corresponding support training resource configuration to complete scheduling and management of the model training resources. According to the invention, flexible distribution and adjustment can be realized according to actual demands of training tasks. Therefore, waste and idleness of resources are effectively avoided, and the utilization rate of the resources is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a multi-resource scheduling method, system and terminal for model training. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning models have been widely used in various fields, such as image recognition, natural language processing, intelligent recommendation, etc. However, the training process of deep learning models often requires a large amount of computing resources, including computing power, storage, network bandwidth, etc. The effective scheduling and management of these resources is crucial to improving model training efficiency and reducing training costs.

[0003] In traditional model training resource scheduling methods, static resource allocation is usually adopted, that is, fixed resources are allocated to different training tasks according to pre-set resource quotas or policies. Although this method is simple and easy to implement, it has many shortcomings. On the one hand, due to the diversity and complexity of training tasks, static resource allocation often cannot meet the dynamically changing resource requirements, resulting in low resource utilization. On the other hand, when the demand for a certain resource of a training task suddenly increases, a resource bottleneck may occur, affecting the training progress and efficiency. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-resource scheduling method for model training, comprising the following steps:

[0005] Step 1: The training resource management server classifies the model training resources into variable training resources and supporting training resources;

[0006] Step 2, obtaining the variable training resource request information sent by the requesting terminal, matching the corresponding supported training resources according to the variable training resource request information, and obtaining the corresponding supported training resource status according to the corresponding supported training resources. If it is in an available state, proceed to step 3. If it is in an unavailable state, send the unavailable information to the corresponding terminal;

[0007] Step 3: Obtain the first support relationship between the variable training resource and the supporting training resource, and obtain the model training information in the variable training resource request information to obtain the training resource support type. If it is a first-class training resource support type, proceed to step 4; if it is not a first-class training resource support type, proceed to step 5.

[0008] Step 4: obtain a second support relationship associated with a type of training resource support, combine the obtained first support relationship and the second support relationship to generate a corresponding support training resource configuration, send the variable training resource and the corresponding support training resource configuration to the training resource management server, and proceed to step 6;

[0009] Step 5: Obtain a third support relationship associated with a non-class I training resource support type, generate a corresponding support training resource configuration according to the third support relationship and the first support relationship, send the variable training resource and the corresponding support training resource configuration to the training resource management server, and proceed to step 6;

[0010] Step 6: The training resource management server schedules the variable training resources and the corresponding supporting training resource configurations to complete the scheduling and management of model training resources.

[0011] Furthermore, the training resource management server classifies the model training resources into variable training resources and supporting training resources, including:

[0012] The variable training resources are dynamically divisible training resources, including computing power resources; the supporting training resources are indivisible unit training resources, including unit computing cards.

[0013] Furthermore, the obtaining of the variable training resource request information sent by the terminal, matching the corresponding training resource according to the resource request information, and obtaining the corresponding resource state according to the training resource includes:

[0014] The training resource management server determines whether the corresponding supported training is allocable according to the variable training resource demand in the variable training resource request information sent by the terminal. If it is allocable, it is in an available state, otherwise, it is in an unavailable state.

[0015] Furthermore, the step of obtaining the first supporting relationship between the variable training resource and the supporting training resource includes:

[0016] The first supporting relationship is an association relationship between variable training resources and supporting training resources, and the association relationship includes the computing power corresponding to the unit computing card.

[0017] Furthermore, the obtaining of the model training information in the variable training resource request information to obtain the training resource support type includes:

[0018] According to the model training information in the variable training resource request information, a model training test is performed to obtain the variable training resource utilization efficiency. If the utilization efficiency is greater than the set utilization efficiency threshold, it is a class I training resource support type; otherwise, it is a non-class I training resource support type.

[0019] Further, the obtaining of the second support relationship associated with the support type of a type of training resource, and generating corresponding support training resource configuration according to the second support relationship associated with the support type of a type of training resource and the second support relationship, includes:

[0020] The second support relationship associated with the one type of training resource support type is the support coefficient of the first support relationship, which is a preset value greater than 1. According to the support coefficient of the first support relationship and the first support relationship, the corresponding support training resource configuration is obtained.

[0021] Further, the obtaining of the second support relationship associated with the non-class I training resource support type, and generating the corresponding support training resource configuration according to the second support relationship associated with the non-class I training resource support type and the first support relationship, includes:

[0022] The second support relationship associated with the non-class one training resource support type is the support coefficient of the first support relationship, and the support coefficient value is 1. According to the support coefficient of the first support relationship and the first support relationship, the corresponding support training resource configuration is obtained.

[0023] A multi-resource scheduling system for model training, applying the multi-resource scheduling method for model training, comprises a training resource management server, a terminal and a communication module; the training resource management server and the terminal are respectively connected to the communication module for communication.

[0024] A multi-resource scheduling terminal for model training, applying the multi-resource scheduling method for model training, comprising a data processing module, a training resource request module and a mobile communication module;

[0025] The data processing module and the training resource request module are respectively connected to the mobile communication module.

[0026] The beneficial effect of the present invention is that the present invention classifies and manages model training resources through a training resource management server, and dynamically schedules various types of resources, so that resources can be flexibly allocated and adjusted according to the actual needs of training tasks. This effectively avoids waste and idleness of resources and improves resource utilization.

[0027] By obtaining the support relationship between the variable training resources and the supporting training resources, and judging the training resource support type according to the model training information, the present invention can generate an optimized supporting training resource configuration, which ensures that the training task can obtain the required resource support, thereby improving the training efficiency.

[0028] The present invention provides different support relationship adjustment and optimization mechanisms for different types of training resource support types (class I and non-class I), which enables the present invention to adapt to diverse training task requirements and meet different types of model training scenarios.

[0029] The training resource management server in the present invention can monitor and adjust the resource allocation strategy in real time to cope with the dynamic changes in training task requirements. This flexibility enables the system to better adapt to the ever-changing training environment and improve the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of a multi-resource scheduling method for model training;

[0031] Figure 2 This is a schematic diagram of the principle of a multi-resource scheduling system for model training;

[0032] Figure 3 A schematic diagram of the principle of a multi-resource scheduling terminal for model training. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0034] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0035] like Figure 1 As shown, a multi-resource scheduling method for model training includes the following steps:

[0036] Step 1: The training resource management server classifies the model training resources into variable training resources and supporting training resources;

[0037] Step 2, obtaining the variable training resource request information sent by the requesting terminal, matching the corresponding supported training resources according to the variable training resource request information, and obtaining the corresponding supported training resource status according to the corresponding supported training resources. If it is in an available state, proceed to step 3. If it is in an unavailable state, send the unavailable information to the corresponding terminal;

[0038] Step 3: Obtain the first support relationship between the variable training resource and the supporting training resource, and obtain the model training information in the variable training resource request information to obtain the training resource support type. If it is a first-class training resource support type, proceed to step 4; if it is not a first-class training resource support type, proceed to step 5.

[0039] Step 4: obtain a second support relationship associated with a type of training resource support, combine the obtained first support relationship and the second support relationship to generate a corresponding support training resource configuration, send the variable training resource and the corresponding support training resource configuration to the training resource management server, and proceed to step 6;

[0040] Step 5: Obtain a third support relationship associated with a non-class I training resource support type, generate a corresponding support training resource configuration according to the third support relationship and the first support relationship, send the variable training resource and the corresponding support training resource configuration to the training resource management server, and proceed to step 6;

[0041] Step 6: The training resource management server schedules the variable training resources and the corresponding supporting training resource configurations to complete the scheduling and management of model training resources.

[0042] The training resource management server classifies the model training resources into variable training resources and supporting training resources, including:

[0043] The variable training resources are dynamically divisible training resources, including computing power resources; the supporting training resources are indivisible unit training resources, including unit computing cards.

[0044] The step of acquiring the variable training resource request information sent by the terminal, matching the corresponding training resource according to the resource request information, and obtaining the corresponding resource state according to the training resource includes:

[0045] The training resource management server determines whether the corresponding supported training is allocable according to the variable training resource demand in the variable training resource request information sent by the terminal. If it is allocable, it is in an available state, otherwise, it is in an unavailable state.

[0046] The step of obtaining the first supporting relationship between the variable training resource and the supporting training resource includes:

[0047] The first supporting relationship is an association relationship between variable training resources and supporting training resources, and the association relationship includes the computing power corresponding to the unit computing card.

[0048] The obtaining of the model training information in the variable training resource request information to obtain the training resource support type includes:

[0049] According to the model training information in the variable training resource request information, a model training test is performed to obtain the variable training resource utilization efficiency. If the utilization efficiency is greater than the set utilization efficiency threshold, it is a class I training resource support type; otherwise, it is a non-class I training resource support type.

[0050] The obtaining of the second supporting relationship associated with the support type of a type of training resource, and generating corresponding supporting training resource configuration according to the second supporting relationship associated with the support type of a type of training resource and the second supporting relationship, includes:

[0051] The second support relationship associated with the one type of training resource support type is the support coefficient of the first support relationship, which is a preset value greater than 1. According to the support coefficient of the first support relationship and the first support relationship, the corresponding support training resource configuration is obtained.

[0052] The obtaining of the second support relationship associated with the non-class I training resource support type, and generating corresponding supported training resource configuration according to the second support relationship associated with the non-class I training resource support type and the first support relationship, comprises:

[0053] The second support relationship associated with the non-class one training resource support type is the support coefficient of the first support relationship, and the support coefficient value is 1. According to the support coefficient of the first support relationship and the first support relationship, the corresponding support training resource configuration is obtained.

[0054] like Figure 2 As shown, a multi-resource scheduling system for model training, applying the multi-resource scheduling method for model training, includes a training resource management server, a terminal and a communication module; the training resource management server and the terminal are respectively connected to the communication module for communication.

[0055] like Figure 3 As shown, a multi-resource scheduling terminal for model training, applying the multi-resource scheduling method for model training, includes a data processing module, a training resource request module and a mobile communication module;

[0056] The data processing module and the training resource request module are respectively connected to the mobile communication module of the age.

[0057] Specifically, the invention provides a multi-resource scheduling method for model training, the method comprising the following steps:

[0058] Step 1: The training resource management server classifies the model training resources and divides the training resources into variable training resources and supporting training resources.

[0059] In the present invention, the variable training resources are dynamically divisible training resources, such as computing power resources. Such resources can be dynamically allocated and adjusted according to the actual needs of the training tasks to improve resource utilization. The supporting training resources are indivisible unit training resources, such as unit computing cards. Such resources usually exist in fixed units and cannot be further divided, but can serve as support for variable training resources.

[0060] Step 2: Obtain the variable training resource request information sent by the requesting terminal, match the corresponding supporting training resources according to the variable training resource request information, and obtain the corresponding supporting training resource status according to the corresponding supporting training resources. If it is in an available state, proceed to step 3. If it is in an unavailable state, send the unavailable information to the corresponding terminal.

[0061] In this step, the training resource management server receives the variable training resource request information from the request terminal, which contains the requirements of the training task for the variable training resources. The server matches the corresponding supporting training resources according to the request information and checks the status of these resources. If the supporting training resources are available, the subsequent steps are continued; if not available, the unavailable information is sent to the corresponding terminal so that the terminal can take corresponding measures.

[0062] The obtaining of the variable training resource request information sent by the terminal, matching the corresponding training resource according to the resource request information, and obtaining the corresponding resource status according to the training resource specifically includes: the training resource management server determines whether the corresponding supported training is allocable according to the variable training resource demand in the variable training resource request information sent by the terminal. The judgment process may involve checking the remaining capacity, allocation status, performance parameters and other factors of the supported training resources. If it is allocable, it is in an available state; otherwise, it is in an unavailable state.

[0063] Step three, obtain the first support relationship between variable training resources and supporting training resources, and obtain the model training information in the variable training resource request information, and obtain the training resource support type. If it is a first-class training resource support type, proceed to step four; if it is not a first-class training resource support type, proceed to step five.

[0064] In this step, the training resource management server obtains the first support relationship between the variable training resource and the supporting training resource, which describes how the variable training resource depends on or utilizes the supporting training resource. At the same time, the server also extracts the model training information from the variable training resource request information and determines the training resource support type based on the information. If the support type is a type of training resource support type, execute step 4; otherwise, execute step 5.

[0065] The obtaining of the first support relationship between the variable training resources and the supporting training resources specifically includes: the first support relationship is an association relationship between the variable training resources and the supporting training resources, and the association relationship may include computing power, storage capacity, network bandwidth, etc. corresponding to the unit computing card. Through these association relationships, the server can understand how the variable training resources use the supporting training resources to meet the needs of the training task.

[0066] The method of obtaining the model training information in the variable training resource request information and obtaining the training resource support type specifically includes: performing a model training test or simulation according to the model training information (such as model type, amount of training data, training algorithm, etc.) in the variable training resource request information to obtain the variable training resource usage efficiency. If the usage efficiency is greater than a set usage efficiency threshold, it is determined to be a first-class training resource support type; otherwise, it is determined to be a non-first-class training resource support type.

[0067] Step 4, obtain a second support relationship associated with a type of training resource support, combine the obtained first support relationship and the second support relationship, generate corresponding support training resource configuration, send the variable training resources and the corresponding support training resource configuration to the training resource management server, and proceed to step 6.

[0068] In this step, for a type of training resource support, the training resource management server obtains a second support relationship associated with it. This relationship may be a set of support coefficients or rules used to adjust or optimize the resource configuration in the first support relationship. The server combines the first support relationship with the second support relationship to generate a corresponding supported training resource configuration, and sends the configuration together with the variable training resources to the training resource management server for scheduling.

[0069] The obtaining of the second support relationship associated with a type of training resource support, and generating the corresponding supported training resource configuration according to the second support relationship associated with a type of training resource support and the first support relationship, specifically includes: the second support relationship associated with a type of training resource support is the support coefficient of the first support relationship, and the coefficient is a preset value greater than 1. By applying the coefficient to the resource configuration in the first support relationship, an optimized supported training resource configuration can be obtained to meet the special needs of a type of training resource support.

[0070] Step five, obtain the third support relationship associated with the non-class one training resource support type, generate the corresponding support training resource configuration based on the third support relationship and the first support relationship, send the variable training resources and the corresponding support training resource configuration to the training resource management server, and proceed to step six.

[0071] In this step, for non-class I training resource support types, the training resource management server obtains a third support relationship associated with it. This relationship may be a set of different support coefficients or rules, which are used to adjust or optimize the resource configuration in the first support relationship to adapt to the needs of non-class I training resource support types. The server generates a corresponding supported training resource configuration based on the third support relationship and the first support relationship, and sends the configuration together with the variable training resources to the training resource management server for scheduling.

[0072] The obtaining of the third support relationship associated with the non-class I training resource support type, and generating the corresponding support training resource configuration according to the third support relationship associated with the non-class I training resource support type and the first support relationship, specifically includes: the third support relationship associated with the non-class I training resource support type is the support coefficient of the first support relationship, but the value of the coefficient is 1 (i.e., no adjustment or optimization is performed). In this case, the server directly generates the support training resource configuration according to the first support relationship to meet the basic requirements of the non-class I training resource support type.

[0073] Step 6: The training resource management server schedules the variable training resources and the corresponding supporting training resource configurations to complete the scheduling and management of model training resources.

[0074] In this step, the training resource management server performs actual resource scheduling and management based on the received variable training resources and the corresponding supporting training resource configuration information. The server may decide how to allocate and adjust resources based on factors such as resource availability, priority, and performance parameters to ensure that the training task can be carried out efficiently and smoothly.

[0075] Example 1: Resource Scheduling for Deep Learning Model Training

[0076] An AI research company needs to train a complex deep learning model, which has high requirements for computing power and storage resources. The company uses the multi-resource scheduling method for model training provided by the present invention to perform resource scheduling.

[0077] Step 1: The training resource management server classifies the available model training resources. Computing resources (such as GPU clusters) are classified as variable training resources, while unit computing cards (such as specific AI accelerator cards) are classified as supported training resources.

[0078] Step 2: The request terminal (the server of the AI ​​research company) sends a variable training resource request message, requesting a certain amount of computing power resources to train the deep learning model. After receiving the request, the training resource management server matches the corresponding supported training resources (i.e., AI accelerator cards) and checks the status of these accelerator cards. It is found that all matched AI accelerator cards are in an available state, so it proceeds to the next step.

[0079] Step 3: The training resource management server obtains the first support relationship between variable training resources (computing resources) and supporting training resources (AI accelerator cards), which describes how many AI accelerator cards are needed per unit of computing power. Extract model training information from the variable training resource request information, including model type, training data volume, and training algorithm. Perform model training tests and find that the usage efficiency is greater than the set usage efficiency threshold, so it is determined to be a type of training resource support.

[0080] Step 4: Obtain a second support relationship associated with a training resource support type, which is the support coefficient of the first support relationship and is set to 1.2 (i.e., optimizing 20% ​​of the resource configuration). Combine the first support relationship with the second support relationship to generate an optimized supported training resource configuration. Send the variable training resources and the optimized supported training resource configuration to the training resource management server.

[0081] Step 6: The training resource management server actually schedules computing resources and AI accelerator cards based on the received configuration information to ensure that deep learning model training can be carried out efficiently.

[0082] Example 2: Resource Scheduling for Natural Language Processing Model Training

[0083] A natural language processing (NLP) research team needs to train a large language model, which has high requirements on computing power and network bandwidth. The team uses the multi-resource scheduling method for model training provided by the present invention to perform resource scheduling.

[0084] Step 1: The training resource management server classifies model training resources. Computing resources (such as TPU clusters) are classified as variable training resources, while network bandwidth resources (such as high-speed network interface cards) are classified as supporting training resources.

[0085] Step 2: The request terminal (the server of the NLP research team) sends a variable training resource request message, requesting a certain amount of computing power resources to train the language model.

[0086] After receiving the request, the training resource management server matches the corresponding supported training resources (ie, high-speed network interface cards) and checks the status of these cards.

[0087] It was found that some high-speed network interface cards were unavailable, so the unavailability information was sent to the NLP research team's server. The team decided to continue using the available resources.

[0088] Step 3: The training resource management server obtains the first support relationship between variable training resources (computing power resources) and supporting training resources (network bandwidth resources), which describes how much network bandwidth support is required per unit computing power.

[0089] Model training information is extracted from the variable training resource request information, including model type, training data volume, and training algorithm. Model training simulation is performed and it is found that the usage efficiency is lower than the set usage efficiency threshold, so it is determined to be a non-class I training resource support type.

[0090] Step 5: Obtain a third support relationship associated with a non-class I training resource support type, which is the support coefficient of the first support relationship, set to 1 (i.e., no adjustment or optimization). Generate a basic support training resource configuration based on the third support relationship and the first support relationship. Send the variable training resources and the basic support training resource configuration to the training resource management server.

[0091] Step 6: The training resource management server actually schedules computing resources and network bandwidth resources based on the received configuration information to ensure that the language model training can proceed smoothly.

Claims

1. A multi-resource scheduling method for model training, characterized in that: The steps include: Step 1: The training resource management server classifies the model training resources into variable training resources and supporting training resources; Step 2, obtaining the variable training resource request information sent by the requesting terminal, matching the corresponding supported training resources according to the variable training resource request information, and obtaining the corresponding supported training resource status according to the corresponding supported training resources. If it is in an available state, proceed to step 3. If it is in an unavailable state, send the unavailable information to the corresponding terminal; Step 3: Obtain the first support relationship between the variable training resource and the supporting training resource, and obtain the model training information in the variable training resource request information to obtain the training resource support type. If it is a first-class training resource support type, proceed to step 4; if it is not a first-class training resource support type, proceed to step 5. Step 4: obtain a second support relationship associated with a type of training resource support, combine the obtained first support relationship and the second support relationship to generate a corresponding support training resource configuration, send the variable training resource and the corresponding support training resource configuration to the training resource management server, and proceed to step 6; Step 5: Obtain a third support relationship associated with a non-class I training resource support type, generate a corresponding support training resource configuration according to the third support relationship and the first support relationship, send the variable training resource and the corresponding support training resource configuration to the training resource management server, and proceed to step 6; Step 6: The training resource management server schedules the variable training resources and the corresponding supporting training resource configurations to complete the scheduling and management of model training resources.

2. A multi-resource scheduling method for model training according to claim 1, characterized in that: The training resource management server classifies the model training resources into variable training resources and supporting training resources, including: The variable training resources are dynamically divisible training resources, including computing power resources; the supporting training resources are indivisible unit training resources, including unit computing cards.

3. A multi-resource scheduling method for model training according to claim 2, characterized in that: The step of acquiring the variable training resource request information sent by the terminal, matching the corresponding training resource according to the resource request information, and obtaining the corresponding resource state according to the training resource includes: The training resource management server determines whether the corresponding supported training is allocable according to the variable training resource demand in the variable training resource request information sent by the terminal. If it is allocable, it is in an available state, otherwise, it is in an unavailable state.

4. A multi-resource scheduling method for model training according to claim 3, characterized in that: The step of obtaining the first supporting relationship between the variable training resource and the supporting training resource includes: The first supporting relationship is an association relationship between variable training resources and supporting training resources, and the association relationship includes the computing power corresponding to the unit computing card.

5. A multi-resource scheduling method for model training according to claim 4, characterized in that: The obtaining of the model training information in the variable training resource request information to obtain the training resource support type includes: According to the model training information in the variable training resource request information, a model training test is performed to obtain the variable training resource utilization efficiency. If the utilization efficiency is greater than the set utilization efficiency threshold, it is a class I training resource support type; otherwise, it is a non-class I training resource support type.

6. A multi-resource scheduling method for model training according to claim 5, characterized in that: The obtaining of the second supporting relationship associated with the support type of a type of training resource, and generating corresponding supporting training resource configuration according to the second supporting relationship associated with the support type of a type of training resource and the second supporting relationship, includes: The second support relationship associated with the one type of training resource support type is the support coefficient of the first support relationship, which is a preset value greater than 1. According to the support coefficient of the first support relationship and the first support relationship, the corresponding support training resource configuration is obtained.

7. A multi-resource scheduling method for model training according to claim 6, characterized in that: The obtaining of the second support relationship associated with the non-class I training resource support type, and generating corresponding supported training resource configuration according to the second support relationship associated with the non-class I training resource support type and the first support relationship, comprises: The second support relationship associated with the non-class one training resource support type is the support coefficient of the first support relationship, and the support coefficient value is 1. According to the support coefficient of the first support relationship and the first support relationship, the corresponding support training resource configuration is obtained.

8. A multi-resource scheduling system for model training, applying a multi-resource scheduling method for model training described in claims 1-7, comprising a training resource management server, a terminal and a communication module; the training resource management server and the terminal are respectively communicatively connected to the communication module.

9. A multi-resource scheduling terminal for model training, characterized in that: A multi-resource scheduling method for model training according to claim 1-7 is applied, comprising a data processing module, a training resource request module and a mobile communication module; The data processing module and the training resource request module are respectively connected to the mobile communication module of the age.

Images