Resource allocation method and device for training model, electronic equipment and storage medium

Abstract

The invention discloses a resource allocation method and device for a training model, electronic equipment and a storage medium. The method comprises the following steps: determining first resource configuration information of a target model in a first training stage; the first resource configuration information is determined before the target model executes the model training task; when it is determined that the training loss value of the target model in the first training stage meets a preset condition, it is determined that the target model enters a second training stage; in a second training stage of the target model, determining a loss value change trend of the target model in the current time period every first preset time; and determining second resource configuration information of the target model in the next time period according to the loss value change trend until the training loss value of the target model reaches a preset threshold value, and completing a model training task of the target model. According to the method, the resource configuration information in the model training process is adaptively adjusted, and the training time of the model is shortened.

Description

Technical Field

[0001] The present invention relates to the field of deep learning model training technology, and in particular to a resource configuration method, device, electronic device and storage medium for a training model. Background Art

[0002] With the rapid growth of deep learning model scale, reducing training time while ensuring targeted training loss has become a major challenge. Resource allocation is a major factor affecting model training time, primarily reflecting its impact on computing performance and the rate of convergence of the training loss. However, current resource allocation methods primarily consider the impact of resources on training computing performance while ignoring their impact on the rate of convergence of the training loss, resulting in longer model training times. Summary of the Invention

[0003] The present invention provides a resource configuration method, device, electronic device and storage medium for training a model, so as to adaptively adjust resource configuration information during the model training process and shorten the training time of the model.

[0004] According to one aspect of the present invention, a resource configuration method for a training model is provided, characterized in that the method comprises:

[0005] Determine first resource configuration information of a target model in a first training phase; the first resource configuration information is determined by the target model before executing a model training task; in the first training phase, the single-round training time of the target model corresponding to the first resource configuration information is the shortest;

[0006] When it is determined that the training loss value of the target model in the first training stage meets a preset condition, it is determined that the target model enters the second training stage;

[0007] In the second training phase of the target model, determining a change trend of the loss value of the target model in the current time period at intervals of a first preset time; the first preset time period is at least the sum of two single-round training times; the current time period is the time period between the current moment and the moment corresponding to the first preset time before the current moment;

[0008] The second resource configuration information of the target model in the next time period is determined according to the loss value change trend, until the training loss value of the target model reaches a preset threshold, and the model training task of the target model is completed; the loss value change trend is associated with the second resource configuration information.

[0009] According to another aspect of the present invention, a resource configuration device for a training model is provided, characterized in that the device comprises:

[0010] a resource determination module, configured to determine first resource configuration information of a target model in a first training phase; the first resource configuration information is determined by the target model before executing a model training task; and in the first training phase, the single-round training time of the target model corresponding to the first resource configuration information is the shortest;

[0011] a stage division module, configured to determine that the target model enters a second training stage when it is determined that the training loss value of the target model in the first training stage meets a preset condition;

[0012] a trend determination module, configured to determine, during the second training phase of the target model, a loss value change trend of the target model in a current time period at intervals of a first preset time; the first preset time period being at least the sum of two single-round training times; and the current time period being the time period between the current moment and a moment corresponding to the first preset time before the current moment;

[0013] A resource update module is used to determine the second resource configuration information of the target model in the next time period based on the loss value change trend, until the training loss value of the target model reaches a preset threshold, completing the model training task of the target model; the loss value change trend is associated with the second resource configuration information.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the resource configuration method for the training model described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the resource configuration method for the training model described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention determines the first resource configuration information of the target model in the first training stage; the first resource configuration information is determined by the target model before executing the model training task; in the first training stage, the single-round training time of the target model corresponding to the first resource configuration information is the shortest, ensuring that in the first training stage, the training time of the target model can be shortened while ensuring the training accuracy of the target model. When it is determined that the training loss value of the target model in the first training stage meets the preset conditions, the target model is determined to enter the second training stage; when the training loss value of the target model in the first training stage meets the preset conditions, the loss value change trend needs to be further considered during the target model training process. Therefore, the target model is determined to enter the second training stage. In the second training stage of the target model, the loss value change trend of the target model in the current time period is determined every first preset time; the first preset time is at least the sum of two single-round training times; the current time period is the time period between the current moment and the moment corresponding to the first preset time before the current moment; the second resource configuration information of the target model in the next time period is determined according to the loss value change trend, until the training loss value of the target model reaches the preset threshold and the model training task of the target model is completed; the loss value change trend is correlated with the second resource configuration information; that is, in the second training stage, the second resource configuration information is updated once according to the loss value change trend of the target model in the current time period every first preset time, so as to adaptively adjust the resource configuration information in the model training process and fully optimize the training time of the model.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flowchart of a resource configuration method for a training model provided according to an embodiment of the present invention;

[0023] Figure 2 is an architectural diagram constructed according to a first model applicable to an embodiment of the present invention;

[0024] Figure 3 is a flowchart of another resource configuration method for a training model provided according to an embodiment of the present invention;

[0025] Figure 4 is an internal architecture diagram of the second model during the second model training process applicable to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of updating resource configuration information of a target model in a second training phase according to an embodiment of the present invention;

[0027] Figure 6 1 is a schematic structural diagram of a resource configuration device for a training model provided in an embodiment of the present invention;

[0028] Figure 7 3 is a structural diagram of an electronic device for implementing a resource configuration method for a training model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", "third", "fourth", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Example 1

[0032] Figure 1This is a flowchart of a resource configuration method for a training model provided by an embodiment of the present invention. This embodiment is applicable to the case of configuring resources for the training process of a deep learning model, especially for the training process of a distributed deep learning model. The method can be executed by a resource configuration device for a training model. The resource configuration device for the training model can be implemented in the form of hardware and / or software. The resource configuration device for the training model can be configured in any electronic device with network communication function. Figure 1 As shown, the resource configuration method of the training model of the present invention includes:

[0033] S110. Determine the first resource configuration information of the target model in the first training phase; the first resource configuration information is determined by the target model before executing the model training task; in the first training phase, the single-round training time of the target model corresponding to the first resource configuration information is the shortest.

[0034] The present invention divides the training of the target model into two training phases. In the first training phase, the first resource configuration information configured for the target model remains unchanged; in the second training phase, the second resource configuration information configured for the target model is adaptively adjusted, thereby improving the flexibility of model training. The first training phase is the early training phase of the target model, and the second training phase is the late training phase of the target model.

[0035] Among them, the single-round training time can be the length of time the target model is trained once. The single-round training time of the target model corresponding to the first resource configuration information is the shortest, that is, the first resource configuration information configured for the target model in the first training stage can shorten the training time of the target model while ensuring the training accuracy of the target model.

[0036] During the target model training process of the present invention, at least one first device and at least one second device need to be configured; the first device is a device used to store and update model parameters during the target model training process; the second device is a device used to store model architecture and perform gradient calculation tasks during the target model training process; the resource configuration information is the number of first devices and the number of second devices.

[0037] The technical solution of this embodiment optionally includes determining the first resource configuration information of the target model in the first training phase, including steps A1-A3:

[0038] Step A1: Determine feature information of a target model and at least two candidate resource configuration information.

[0039] The feature information of the target model can be feature graph information that abstracts the target model into a feature graph (DAG); the feature graph is represented by the form of an adjacency matrix A and a feature matrix X, such as Figure 2 shown.

[0040] The candidate resource configuration information can be understood as different resource configuration information configured for the target model when training the target model. Specifically, the total resources that can be allocated to the target model during the training process are fixed, but the combination of different resources in the actual training process may affect the single-round training time of the target model. Therefore, the resources that can be allocated to the target model can be matched with each other to form at least two candidate resource configuration information. For example, the resources that can be configured during the training of the target model of the present invention are N first devices and M second devices; then the N first devices and the M second devices are matched with each other to form at least two candidate resource configuration information. For example, one of the candidate resource configuration information can be Na first devices and Mb second devices.

[0041] Step A2: Input the characteristic information of the target model and at least two candidate resource configuration information into the first model to determine the candidate single-round training time corresponding to each candidate resource configuration information of the target model; the first model is used to describe the correspondence between the characteristic information and resource configuration information of the target model and the single-round training time.

[0042] The candidate single-round training time can be understood as the single-round training time for the target model to complete one training based on the candidate resource configuration information.

[0043] Specifically, the characteristic information of different target models themselves will also affect the single-round training time of the model. Therefore, a first model is constructed to describe the correspondence between the characteristic information and resource configuration information of the target model and the single-round training time. The first model can quickly obtain the characteristic information of the target model and at least two candidate resource configuration information, and obtain the candidate single-round training time corresponding to each candidate resource configuration information of the target model, so as to facilitate the subsequent accurate determination of which candidate resource configuration information is more suitable for the target model based on the candidate single-round training time.

[0044] In an embodiment of the present invention, the first model may optionally include a graph convolution layer, a pooling layer, and a fully connected layer. The graph convolution layer is used to extract feature information from the target model to obtain hidden layer features; the pooling layer is used to adaptively pool the hidden layer features to generate a first feature vector; the fully connected layer is used to process the second feature vector to obtain the single-round training time of the target model; the second feature vector is a vector obtained by concatenating the first feature vector, resource configuration information, and the batch size of the target model.

[0045] For example, Figure 2As shown, the first model may include two layers of graph convolutional layers (GC), one layer of pooling (Pooling) and one layer of fully connected layer (FCNN). The feature information of the target model and at least two candidate resource configuration information are input into the first model. The two layers of graph convolutional layers of the first model extract the feature information of the target model to obtain hidden features, and input the hidden features into a layer of pooling. Then, a layer of pooling layer adaptively pools the hidden features to generate a first feature vector, and concatenates the first feature vector, a single candidate resource configuration information and the batch size of the target model to obtain a second feature vector, and then inputs the second feature vector into a layer of fully connected layer; further, a layer of fully connected layer processes the second feature vector and outputs the single-round training time of the target model corresponding to the candidate resource configuration information. If there are several single candidate resource configuration information, the first model will output the single-round training time of several target models.

[0046] Step A3: Using the candidate resource configuration information corresponding to the shortest candidate single-round training time among the candidate single-round training times as the first resource configuration information of the target model in the first training stage.

[0047] Specifically, because the first model may output a large number of candidate single-round training times, a search algorithm can be used to search the candidate single-round training times to find the shortest candidate single-round training time among the candidate single-round training times, and the candidate resource configuration information corresponding to the shortest candidate single-round training time is used as the first resource configuration information of the target model in the first training stage.

[0048] The technical solution of the embodiment of the present invention realizes accurate identification of the features of the target model itself that affect the model training time by determining the characteristic information of the target model and at least two candidate resource configuration information, and matches the resources that can be allocated to the target model to form at least two candidate resource configuration information, which are used to accurately judge which candidate resource configuration information is more suitable for the training of the first training stage of the target model. Furthermore, the characteristic information of the target model and at least two candidate resource configuration information are input into the first model to determine the candidate single-round training time corresponding to each candidate resource configuration information of the target model; the first model is used to describe the correspondence between the characteristic information and resource configuration information of the target model and the single-round training time, so as to accurately derive the candidate single-round training time corresponding to each candidate resource configuration information of the target model, that is, accurately distinguish the influence of each candidate resource configuration information on the computing performance during the training of the target model. Finally, the candidate resource configuration information corresponding to the shortest candidate single-round training time among the candidate single-round training times is used as the first resource configuration information of the target model in the first training stage, so as to accurately use the candidate resource configuration information that can make the calculation speed the fastest during the model training process as the first resource configuration information of the target model in the first training stage, and use it for the resource configuration information of the first training stage of the target model, thereby shortening the training time of the model and improving the calculation efficiency.

[0049] S120: When it is determined that the training loss value of the target model in the first training phase meets a preset condition, it is determined that the target model enters the second training phase.

[0050] The precondition can be understood as the condition that the convergence rate of the target model's training loss reaches a certain threshold. This precondition indicates that the target model needs to enter the second training phase. This means that the computational performance of model training should not be considered alone; the impact of the convergence rate of the training loss on model training must also be considered. The convergence rate of the training loss can be measured by the target model's loss trend. To minimize model training time, different loss trend variations correspond to different resource allocation information.

[0051] As an optional embodiment, determining whether the training loss value of the target model in the first training stage meets the preset conditions includes: starting from the target model training, obtaining multiple training loss value groups, and determining the loss value mean of each training loss value group; the training loss value group includes the training loss values of at least two target models; each training loss value group corresponds to a time slice; when the loss value mean corresponding to the target model at the current moment reaches a preset mean threshold, and / or when the second-order difference of the loss value mean shows a downward trend, it is determined that the training loss value of the target model in the first training stage meets the preset conditions.

[0052] Here, a time slice can be understood as the time range within which the recorded training loss values fall from the start of target model training, and the second preset time corresponding to each time slice is generally the same. The second preset time can be understood as the period between the start and end times of the time slice. For example, if model training starts at 4:00 pm and the second preset time is 10 minutes, then the first time slice can be 4:00-4:10, the second time slice can be 4:10-4:20, and so on. Multiple time slices can be used, and there will also be multiple training loss value groups in the first training phase. The single-round training time of the target model is much shorter than a time slice. The mean loss value can be the average of the training loss values within the training loss value group.

[0053] The present invention divides the training loss value groups by time slices, so that the changes in the training loss value can be reflected by the changes in the mean of the loss value, avoiding the inaccurate changes in the determined training loss value due to the inaccuracy of a single training loss value, and improving the accurate determination of the moment when the first training stage enters the second training stage.

[0054] S130. In the second training phase of the target model, the loss value change trend of the target model in the current time period is determined every first preset time; the first preset time is at least the sum of two single-round training times; the current time period is the time period between the current moment and the moment corresponding to the first preset time before the current moment.

[0055] Among them, when determining the loss value change trend of the target model in the current time period, the corresponding moment is the last moment of the current time period.

[0056] Specifically, the second training stage of the target model can be divided into multiple rounds of training, and each round of time is divided according to the first preset time, that is, when the second training stage starts, the first preset time interval is from the first moment at the beginning of the second training stage to the second moment; if the stage between the first moment and the second moment is the current time period, when the second moment is reached, the loss value change trend of the target model in the current time period can be determined based on the multiple training loss values of the recorded target model, so as to determine the second resource configuration information of the target model in the next time period according to the loss value change trend of the target model in the current time period, and when the training of the target model in the next time period starts, the next time period is used as the current time period, and when the last moment of the current time period is reached, the loss value change trend of the target model in the current time period is determined again, and so on, and the second resource configuration information of each time period in the second training stage is continuously updated.

[0057] Optionally, every first preset time, while determining the loss value change trend of the target model in the current time period, it is also necessary to suspend the training of the target model in the second training phase. When the second resource configuration information of the target model in the next time period is determined, the training of the target model in the second training phase is started again, and the target model is trained using the second resource configuration information of the target model in the next time period.

[0058] S140. Determine the second resource configuration information of the target model for the next time period based on the loss value change trend, until the training loss value of the target model reaches a preset threshold, and the model training task of the target model is completed; the loss value change trend is correlated with the second resource configuration information.

[0059] Specifically, the second resource configuration information of the target model in the reference time period corresponding to when the second training phase is just entered from the first training phase can be determined according to the change trend of the loss value in the first training phase.

[0060] The technical solution of the embodiment of the present invention determines the first resource configuration information of the target model in the first training stage; the first resource configuration information is determined by the target model before executing the model training task; in the first training stage, the single-round training time of the target model corresponding to the first resource configuration information is the shortest, ensuring that in the first training stage, the training time of the target model can be shortened while ensuring the training accuracy of the target model. When it is determined that the training loss value of the target model in the first training stage meets the preset conditions, the target model is determined to enter the second training stage; when the training loss value of the target model in the first training stage meets the preset conditions, the loss value change trend needs to be further considered during the target model training process. Therefore, the target model is determined to enter the second training stage. In the second training stage of the target model, the loss value change trend of the target model in the current time period is determined every first preset time; the first preset time is at least the sum of two single-round training times; the current time period is the time period between the current moment and the moment corresponding to the first preset time before the current moment; the second resource configuration information of the target model in the next time period is determined according to the loss value change trend, until the training loss value of the target model reaches the preset threshold and the model training task of the target model is completed; the loss value change trend is correlated with the second resource configuration information; that is, in the second training stage, the second resource configuration information is updated once according to the loss value change trend of the target model in the current time period every first preset time, so as to adaptively adjust the resource configuration information in the model training process and fully optimize the training time of the model.

[0061] Example 2

[0062] Figure 3This is a flowchart of another resource configuration method for a training model provided in an embodiment of the present invention. The technical solution of this embodiment further optimizes the process S140 in the aforementioned embodiment on the basis of the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments.

[0063] like Figure 3 As shown in the figure, the resource configuration method for training the model includes:

[0064] S210. After determining that the target model enters the second training stage from the first training stage, determine the loss value change trend of the target model in the current time period every first preset time; the first preset time is at least the sum of two single-round training times; the current time period is the time period between the current moment and the moment corresponding to the first preset time before the current moment.

[0065] S220. Obtain resource information, where the resource information includes the third resource configuration information, the CPU utilization corresponding to the third resource configuration information, the broadband utilization corresponding to the third resource configuration information, and the batch size of the target model; the third resource configuration information is the second resource configuration information of the target model in the current time period.

[0066] Specifically, a status monitor may be configured in the system to dynamically record and obtain resource information using the status monitor.

[0067] S230. Update the second resource configuration information of the target model according to the changing trend of resource information and loss value, and use the updated second resource configuration information as the second resource configuration information of the target model in the next time period, so that the target model in the next time period is trained based on the updated second resource configuration information until the training loss value of the target model reaches the preset threshold, thereby completing the model training task of the target model.

[0068] Specifically, obtain resource information and loss value change trends, determine the updated second resource configuration information corresponding to the resource information and loss value change trends based on the correlation between the resource information and loss value change trends and the resource configuration information, and use the updated second resource configuration information as the second resource configuration information of the target model for the next time period.

[0069] In this embodiment, optionally, the loss value change trend is the loss value change rate of the training loss value of the target model in the current time period. According to the resource information and the loss value change trend, the second resource configuration information of the target model is updated, including: inputting the resource information and the loss value change rate into the second model to obtain fourth resource configuration information, and using the fourth resource configuration information as the second resource configuration information of the target model in the next time period; the second model is used to describe the correspondence between the resource information and the loss value change rate and the resource configuration information.

[0070] The second model can be a Deep Q-Network (DQN) model, such as Figure 4 As shown. The DQN model collects training parameters during the training process. The training parameters include resource configuration information, CPU utilization corresponding to the resource configuration information, bandwidth utilization corresponding to the resource configuration information, and the product of the batch size and the number of second devices. The training parameters are spliced as the current state ( Figure 4 S in the example), and pass the current state state to the DQN model to be trained, and the DQN model to be trained makes a decision action ( Figure 4 The decision action is the resource configuration information after the decision, that is, the state after the decision. The current state, a, and the state after the decision are concatenated into a reference tuple. The reference tuple and the reward are then fed into the DQN training loss function for training. The model parameters of the DQN model to be trained are updated to obtain the final DQN model.

[0071] The reward is expressed as the weighted sum of the average CPU utilization of the second device per unit time and the percentage of the reduction in training loss per unit time slice to the reduction in the overall training loss, as shown in the following formula:

[0072]

[0073] Among them, r represents reward, u wk represents the average CPU utilization of the second device in the resource configuration information per unit time slice, Δloss represents the decrease in loss value per unit time slice, and Loss represents the difference between the user-provided training loss threshold and the initial loss value. W1 and W2, as adjustable parameters of the reward part, satisfy the constraint W1+W2=1. The reason is that this constraint can normalize the overall reward and avoid the situation where the rewards of different training tasks are too different, which will lead to poor DQN model training results.

[0074] For example, in the embodiment of the present invention, the training process of the target model in the second training phase can be referred to as follows: Figure 5 Flowchart shown. Figure 5The current round of training corresponds to the aforementioned current time period, and the next round of training corresponds to the next time period. Specifically, the training process of the target model in the second training phase is: when entering the second training phase from the first training phase, the initial resource information and the initial loss value change trend are obtained; wherein the initial resource information is the first resource configuration information of the first training phase, the CPU utilization corresponding to the first resource configuration information, the broadband utilization corresponding to the first resource configuration information, and the batch size of the target model; the initial loss value change trend is the loss value change rate of the first training phase. Further, the second model is used to process the initial resource information and the initial loss value change trend, and the first second resource configuration information of the target model in the second training phase is obtained, and the current round of training is started to judge in real time whether the training loss value of the target model has reached the preset threshold, and whether the training time of the target model in this round has accumulated to the first preset time. If the training loss value of the target model does not reach the preset threshold and the training time of the target model in this round has accumulated to the first preset time, the current round of training is paused, and the resource information and loss value change trend of the current round of training are obtained, and the resource information and loss value change trend of the current round of training are input into the second model to obtain the updated second resource configuration information, and the updated second resource configuration information is used as the second resource configuration information for the next round of training of the target model, and training is started, and it is continued to be judged in real time whether the training loss value of the target model reaches the preset threshold, and whether the training time of the target model in this round has accumulated to the first preset time, until the training loss value of the target model reaches the preset threshold, and the model training task of the target model is completed.

[0075] The technical solution of the embodiment of the present invention is that after the target model enters the second training stage from the first training stage, the second training stage updates the second resource configuration information of the target model according to the resource information and the loss value change trend. While considering the impact of the resource configuration information on the model training computing performance, it also fully considers the complex impact of the resource configuration information on the training loss convergence rate, thereby better shortening the model training time.

[0076] Example 3

[0077] Figure 6 This is a schematic diagram of the structure of a resource configuration device for a training model provided by an embodiment of the present invention. This embodiment is applicable to the case of configuring resources for the training process of a deep learning model, especially for the training process of a distributed deep learning model. The resource configuration device for the training model can be implemented in the form of hardware and / or software. The resource configuration device for the training model can be configured in any electronic device with network communication function. Figure 6 As shown, the resource configuration device for training the model includes:

[0078] A resource determination module 310 is configured to determine first resource configuration information of a target model in a first training phase; the first resource configuration information is determined by the target model before executing a model training task; and in the first training phase, the target model corresponding to the first resource configuration information has the shortest single-round training time;

[0079] a stage division module 320 for determining that the target model enters a second training stage when it is determined that the training loss value of the target model in the first training stage meets a preset condition;

[0080] A trend determination module 330 is configured to determine, during the second training phase of the target model, a loss value change trend of the target model in a current time period at intervals of a first preset time period, wherein the first preset time period is at least the sum of two single-round training times, and the current time period is the time period between the current moment and a moment corresponding to the first preset time period before the current moment.

[0081] The resource update module 340 is used to determine the second resource configuration information of the target model in the next time period according to the loss value change trend, until the training loss value of the target model reaches a preset threshold, and the model training task of the target model is completed; the loss value change trend is associated with the second resource configuration information.

[0082] Based on the above embodiment, optionally, a resource determination module is used to: determine the characteristic information of the target model and at least two candidate resource configuration information; input the characteristic information of the target model and at least two candidate resource configuration information into a first model to determine the candidate single-round training time corresponding to each candidate resource configuration information of the target model; the first model is used to describe the correspondence between the characteristic information and resource configuration information of the target model and the single-round training time; and use the candidate resource configuration information corresponding to the shortest candidate single-round training time among the candidate single-round training times as the first resource configuration information of the target model in the first training stage.

[0083] Based on the above embodiment, optionally, the first model includes a graph convolution layer, a pooling layer and a fully connected layer; the graph convolution layer is used to extract feature information of the target model to obtain hidden layer features; the pooling layer is used to adaptively pool the hidden layer features to generate a first feature vector; the fully connected layer is used to process the second feature vector to obtain a single-round training time of the target model; the second feature vector is a vector obtained by concatenating the first feature vector, the resource configuration information and the batch size of the target model.

[0084] Based on the above embodiment, optionally, the stage division module includes a judgment unit, which is used to: start from the training of the target model, obtain multiple training loss value groups, and determine the mean loss value of each training loss value group; the training loss value group includes at least two training loss values of the target models; each training loss value group corresponds to a time slice; when the mean loss value corresponding to the target model at the current moment reaches a preset mean threshold, and / or when the second-order difference of the loss value mean shows a downward trend, it is determined that the training loss value of the target model in the first training stage meets the preset conditions.

[0085] Based on the above embodiment, optionally, the resource update module includes a resource acquisition unit and a resource update unit; the resource acquisition unit is used to acquire resource information, wherein the resource information is the third resource configuration information, the CPU utilization corresponding to the third resource configuration information, the broadband utilization corresponding to the third resource configuration information, and the batch size of the target model; the third resource configuration information is the second resource configuration information of the target model in the current time period; the resource update unit is used to update the second resource configuration information of the target model according to the resource information and the loss value change trend, and use the updated second resource configuration information as the second resource configuration information of the target model in the next time period, so that the target model in the next time period is trained based on the updated second resource configuration information.

[0086] Based on the above embodiment, optionally, the loss value change trend is the loss value change rate of the training loss value of the target model in the current time period, and the resource update unit is used to: input the resource information and the loss value change rate into the second model to obtain fourth resource configuration information, and use the fourth resource configuration information as the second resource configuration information of the target model in the next time period; the second model is used to describe the correspondence between the resource information and the loss value change rate and the resource configuration information.

[0087] Based on the above embodiment, optionally, at least one first device and at least one second device need to be configured during the target model training process; the first device is a device used to store and update model parameters during the target model training process; the second device is a device used to store model architecture and perform gradient calculation tasks during the target model training process; the resource configuration information is the number of first devices and the number of second devices.

[0088] The resource configuration device for the training model provided in the embodiment of the present invention can execute the resource configuration method for the training model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0089] Example 4

[0090] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0091] Figure 7 A schematic diagram of the structure of an electronic device that can be used to implement the resource configuration method of the training model of an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0092] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0093] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0094] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the resource configuration method for training the model.

[0095] In some embodiments, the resource configuration method for the training model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the resource configuration method for the training model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the resource configuration method for the training model in any other appropriate manner (e.g., by means of firmware).

[0096] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0098] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0100] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0101] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0102] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0103] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A resource configuration method for a training model, characterized in that: The method comprises: Determine first resource configuration information of a target model in a first training phase; the first resource configuration information is determined by the target model before executing a model training task; in the first training phase, the single-round training time of the target model corresponding to the first resource configuration information is the shortest; When it is determined that the training loss value of the target model in the first training stage meets a preset condition, it is determined that the target model enters the second training stage; In the second training phase of the target model, determining a change trend of the loss value of the target model in a current time period at intervals of a first preset time; the first preset time period is at least the sum of two single-round training times; and the current time period is a time period between a current moment and a moment corresponding to the first preset time before the current moment; The second resource configuration information of the target model in the next time period is determined according to the loss value change trend, until the training loss value of the target model reaches a preset threshold, and the model training task of the target model is completed; the loss value change trend is associated with the second resource configuration information.

2. The method according to claim 1, characterized in that Determining first resource configuration information of the target model in the first training phase includes: Determining characteristic information of the target model and at least two candidate resource configuration information; Inputting the characteristic information of the target model and at least two candidate resource configuration information into a first model, and determining a candidate single-round training time corresponding to each candidate resource configuration information of the target model; the first model is used to describe the correspondence between the characteristic information and resource configuration information of the target model and the single-round training time; The candidate resource configuration information corresponding to the shortest candidate single-round training time among the candidate single-round training times is used as the first resource configuration information of the target model in the first training stage.

3. The method according to claim 2, characterized in that The first model includes a graph convolution layer, a pooling layer and a fully connected layer; The graph convolution layer is used to extract feature information of the target model to obtain hidden layer features; The pooling layer is used to adaptively pool the hidden layer features to generate a first feature vector; The fully connected layer is used to process the second eigenvector to obtain the single-round training time of the target model; the second eigenvector is a vector obtained by concatenating the first eigenvector, the resource configuration information and the batch size of the target model.

4. The method according to claim 1, wherein Determining whether the training loss value of the target model in the first training stage meets a preset condition includes: Starting from the training of the target model, a plurality of training loss value groups are obtained, and a loss value mean of each of the training loss value groups is determined; the training loss value groups include at least two training loss values of the target model; and each training loss value group corresponds to a time slice; When the mean of the loss values corresponding to the target model at the current moment reaches a preset mean threshold, and / or when the second-order difference of the mean of the loss values shows a downward trend, it is determined that the training loss value of the target model in the first training stage meets the preset conditions.

5. The method according to claim 1, wherein Determining second resource configuration information of the target model in the next time period according to the loss value change trend includes: Obtaining resource information, where the resource information includes third resource configuration information, CPU utilization corresponding to the third resource configuration information, bandwidth utilization corresponding to the third resource configuration information, and a batch size of the target model; the third resource configuration information is the second resource configuration information of the target model in the current time period; According to the resource information and the loss value change trend, the second resource configuration information of the target model is updated, and the updated second resource configuration information is used as the second resource configuration information of the target model in the next time period, so that the target model in the next time period is trained based on the updated second resource configuration information.

6. The method according to claim 5, characterized in that The loss value change trend is a loss value change rate of the training loss value of the target model in the current time period. Updating the second resource configuration information of the target model according to the resource information and the loss value change trend includes: The resource information and the loss value change rate are input into the second model to obtain the fourth resource configuration information, and the fourth resource configuration information is used as the second resource configuration information of the target model in the next time period; the second model is used to describe the correspondence between the resource information and the loss value change rate and the resource configuration information.

7. The method according to claim 1, characterized in that During the target model training process, at least one first device and at least one second device need to be configured; the first device is a device used to store and update model parameters during the target model training process; The second device is a device used to store the model architecture and perform gradient calculation tasks during the target model training process; the resource configuration information is the number of first devices and the number of second devices.

8. A resource configuration device for training a model, characterized in that: The device comprises: a resource determination module, configured to determine first resource configuration information of a target model in a first training phase; the first resource configuration information is determined by the target model before executing a model training task; and in the first training phase, the single-round training time of the target model corresponding to the first resource configuration information is the shortest; a stage division module, configured to determine that the target model enters a second training stage when it is determined that the training loss value of the target model in the first training stage meets a preset condition; a trend determination module, configured to determine, during the second training phase of the target model, a loss value change trend of the target model in a current time period at intervals of a first preset time; the first preset time period being at least the sum of two single-round training times; and the current time period being the time period between the current moment and a moment corresponding to the first preset time before the current moment; A resource update module is used to determine the second resource configuration information of the target model in the next time period based on the loss value change trend, until the training loss value of the target model reaches a preset threshold, completing the model training task of the target model; the loss value change trend is associated with the second resource configuration information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the resource configuration method for the training model described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the resource configuration method for the training model described in any one of claims 1 to 7 when executed.

Images