Computing resource allocation method, electronic device, storage medium, and program product

Abstract

The application discloses a kind of computing resource allocation method, electronic equipment, storage medium and program product, it is related to artificial intelligence technical field, specifically related to model training technical field.The method is used for the training of neural network model, the method comprises: obtaining the computation time of submodel on training node and the communication time of executing communication task;According to computation time and the communication time, obtain the delay step number;According to delay step number, the size of data sample and the quantity of multiple training nodes, obtain the scheduling scheme of multiple computing tasks and multiple communication tasks executed on training node;According to scheduling scheme, generate scheduling instruction sequence for multiple computing tasks and multiple communication tasks;According to scheduling instruction sequence, control training node to train submodel deployed on training node according to data sample.It can improve the training efficiency of neural network model for distributed training.

Description

Technical Field

[0001] This application generally relates to the field of artificial intelligence technology, specifically to the field of model training technology, and in particular to a method for allocating computing resources, electronic devices, storage media, and program products. Background Technology

[0002] As deep learning technology continues to develop, the number of parameters in neural network models is also gradually increasing. This increase in parameters necessitates the use of multiple computing devices for distributed model training; that is, employing distributed training techniques that utilize the storage and computing resources of multiple devices to complete the model training.

[0003] Among related technologies, the main distributed training techniques are tensor model parallelism and pipelined model parallelism. These two parallel techniques break down the entire model computation task into subtasks and distribute them to different devices for execution. Tensor model parallelism distributes individual operators across different devices, while pipelined model parallelism distributes different layers of the model across different devices. They exchange computational results through communication operators, allowing multiple devices to collaborate and complete the computational tasks required for training a large model. Tensor model parallelism requires relatively high communication bandwidth between devices; insufficient bandwidth leads to low training efficiency. While pipelined model parallelism has lower communication volume than tensor model parallelism, its communication proportion is still high, and the inability to parallelize computation and communication processes results in significant communication overhead, impacting model training efficiency. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a computing resource allocation method, electronic device, storage medium and program product that can improve the training efficiency of distributed training of neural network models.

[0005] In a first aspect, this application provides a method for allocating computing resources for training a neural network model, the neural network model comprising multiple sub-models, the multiple sub-models being deployed one-to-one on multiple training nodes, the method comprising:

[0006] Obtain the computation time and communication time of the sub-model executing computational tasks and executing communication tasks on the training node;

[0007] The delay steps are obtained based on the calculation time and the communication time;

[0008] Based on the number of delay steps, the size of the data sample, and the number of training nodes, a scheduling scheme for multiple computing tasks and multiple communication tasks executed on the training nodes is obtained, wherein the size of the data sample includes the total size of the data sample and the size of the sub-data divided from the data sample;

[0009] Generate a sequence of scheduling instructions for the plurality of computing tasks and the plurality of communication tasks according to the scheduling scheme;

[0010] The training node is controlled according to the scheduling instruction sequence to train the sub-model deployed on the training node based on the data sample.

[0011] In some examples, obtaining the delay steps based on the calculation time and the communication time includes:

[0012] Compare the computation time and the communication time;

[0013] If the communication time is less than the calculation time, the delay step number is set as the first delay step number; otherwise, the delay step number is set as the second delay step number, wherein the first delay step number is less than the second delay step number.

[0014] In some examples, the first delay step is 2 and the second delay step is 3.

[0015] In some examples, obtaining a scheduling scheme for multiple computational tasks and multiple communication tasks executed on the training nodes based on the number of delay steps, the size of the data samples, and the number of the multiple training nodes includes:

[0016] Based on the number of delay steps, the size of the data sample, and the number of training nodes, multiple computational tasks executed on the training nodes are obtained, and the multiple computational tasks are sorted.

[0017] A communication task is set for each of the plurality of computing tasks, resulting in a sorting of the plurality of computing tasks and the plurality of communication tasks;

[0018] A scheduling scheme for the multiple computing tasks and multiple communication tasks is obtained based on the order of the multiple computing tasks and multiple communication tasks.

[0019] In some examples, generating a sequence of scheduling instructions for the plurality of computing tasks and the plurality of communication tasks according to the scheduling scheme includes:

[0020] Based on the order of the multiple computing tasks and multiple communication tasks in the scheduling scheme, each of the multiple computing tasks and multiple communication tasks is converted into a scheduling instruction.

[0021] The scheduling instruction sequence is obtained based on the scheduling instructions for each task transition.

[0022] In some examples, setting a communication task for each of the plurality of computing tasks to obtain an order of the plurality of computing tasks and the plurality of communication tasks includes:

[0023] Determine whether the communication task has a dependency relationship with the computing task;

[0024] If the communication task and the computing task do not have a dependency relationship, the communication task and the computing task are arranged in parallel.

[0025] If the communication task is dependent on the computing task, the communication task and the computing task are arranged in sequence.

[0026] In some examples, determining whether the communication task has a dependency on the computing task includes:

[0027] If the data to be communicated by the communication task is generated by the execution of the computing task, or if the computing task is executed based on the data to be communicated by the communication task, then the communication task and the computing task have the dependency relationship; otherwise, the communication task and the computing task do not have the dependency relationship.

[0028] Secondly, this application provides a computing resource allocation device for training a neural network model, the neural network model including multiple sub-models, the multiple sub-models being deployed one-to-one on multiple training nodes, the computing resource allocation device comprising:

[0029] The acquisition module is used to obtain the computation time and communication time of the sub-models on the training nodes when performing computation tasks and communication tasks.

[0030] A delay step calculation module is used to obtain the delay step based on the calculation time and the communication time;

[0031] The scheduling scheme acquisition module is used to obtain a scheduling scheme for multiple computing tasks and multiple communication tasks executed on the training nodes based on the delay steps, the size of the data sample, and the number of the multiple training nodes, wherein the size of the data sample includes the total size of the data sample and the size of the sub-data divided from the data sample;

[0032] The scheduling instruction generation module is used to generate a sequence of scheduling instructions for the plurality of computing tasks and the plurality of communication tasks according to the scheduling scheme.

[0033] The scheduling instruction execution module is used to control the training node to train the sub-model deployed on the training node according to the data sample based on the scheduling instruction sequence.

[0034] In some examples, the delay step calculation module is specifically used for:

[0035] Compare the computation time and the communication time;

[0036] If the communication time is less than the calculation time, the delay step number is set as the first delay step number; otherwise, the delay step number is set as the second delay step number, wherein the first delay step number is less than the second delay step number.

[0037] In some examples, the scheduling scheme acquisition module is specifically used for:

[0038] Based on the number of delay steps, the size of the data sample, and the number of training nodes, multiple computational tasks executed on the training nodes are obtained, and the multiple computational tasks are sorted.

[0039] A communication task is set for each of the plurality of computing tasks, resulting in a sorting of the plurality of computing tasks and the plurality of communication tasks;

[0040] A scheduling scheme for the multiple computing tasks and multiple communication tasks is obtained based on the order of the multiple computing tasks and multiple communication tasks.

[0041] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the computing resource allocation method described in embodiments of this application.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the computing resource allocation method as described in embodiments of this application.

[0043] Fifthly, embodiments of this application provide a computer program product having a computer program stored thereon, which, when executed by a processor, implements the computing resource allocation method as described in embodiments of this application.

[0044] The computing resource allocation method, electronic device, storage medium, and program product proposed in this application determine different delay steps based on the difference between the computing time of the sub-models executing computing tasks and the communication time of executing communication tasks on the training nodes. Then, based on the delay steps, the size of the data samples, and the number of training nodes, a scheduling scheme for multiple computing tasks and multiple communication tasks on each training node can be determined. A scheduling instruction sequence is generated for the multiple computing tasks and multiple communication tasks according to the scheduling scheme. Finally, the training nodes are controlled to train the sub-models deployed on the training nodes according to the data samples based on the scheduling instruction sequence. Through the scheduling scheme, computing tasks and communication tasks that have no dependencies during model training can be executed in parallel. In this way, the communication overhead of distributed training of the model is reduced, and the training efficiency of distributed training of neural network models is improved.

[0045] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0046] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0047] Figure 1 A flowchart illustrating the computing resource allocation method provided in an embodiment of this application;

[0048] Figure 2 A schematic diagram of a scheduling scheme with a delay step of 2 for the computing resource allocation method provided in the embodiments of this application;

[0049] Figure 3 A schematic diagram of a scheduling scheme with a delay step of 3 for the computing resource allocation method provided in the embodiments of this application;

[0050] Figure 4 A schematic diagram of the execution of the instruction sequence of the computing resource allocation method provided in the embodiments of this application;

[0051] Figure 5 This is a schematic diagram of the structure of the computing resource allocation device provided in the embodiments of this application;

[0052] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0053] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0054] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] This application proposes a computing resource allocation method, electronic device, storage medium, and program product that enables computational and communication tasks that are independent of each other to be executed in parallel during model training, thereby reducing network communication overhead during distributed training and improving the training efficiency of neural network models in distributed training.

[0056] The implementation environment of this application embodiment can be obtained by a personal computer device, etc., for the computation time of the sub-model executing computational tasks and the communication time of executing communication tasks on the training node; a delay step is obtained based on the computation time and the communication time; a scheduling scheme for multiple computational tasks and multiple communication tasks executed on the training node is obtained based on the delay step, the size of the data sample, and the number of multiple training nodes, wherein the size of the data sample includes the total size of the data sample and the size of the sub-data divided from the data sample; a scheduling instruction sequence is generated for the multiple computational tasks and multiple communication tasks according to the scheduling scheme; and the training node is controlled to train the sub-model deployed on the training node according to the data sample based on the scheduling instruction sequence.

[0057] This application provides a computing resource allocation method for training a neural network model, wherein the neural network model includes multiple sub-models, and the multiple sub-models are deployed one-to-one on multiple training nodes.

[0058] In the following description:

[0059] Steam refers to the data structure in a parallel computing framework that executes commands sent by the host sequentially.

[0060] batch_size refers to the batch size of training data for the neural network model (i.e., the total size of the data samples);

[0061] `micro_batch_size` refers to splitting large batches of training data (i.e., data samples) into multiple smaller batches. During training, the gradient of each smaller batch with respect to the model parameters is calculated separately, and these gradients are summed or averaged. The batch size of the smaller batches is `micro_batch_size`.

[0062] tensor refers to a tensor / high-dimensional array;

[0063] forward(F) refers to the forward computation task, specifically the forward computation of a neural network model.

[0064] backward(B) refers to the reverse computation task, the reverse computation of a neural network model;

[0065] send(S) means: send task, send tensor (i.e.: the output of this training node) to other devices (i.e.: other training nodes);

[0066] receive(R) means: receive task, receive tensors from other devices;

[0067] Pipeline stage refers to the process of breaking down a complete deep neural network model into multiple sub-models that need to be executed sequentially. Each sub-model is a pipeline stage.

[0068] stage_id refers to the unique identifier of a pipeline stage;

[0069] A command queue is a container used to store a set of commands that need to be executed in sequence.

[0070] tensorstorage is a container for storing tensors using key-value pairs.

[0071] The event is a synchronization marker used to monitor the progress of device task execution;

[0072] eventstorage is a container for storing events using key-value pairs.

[0073] Computational resource allocation methods can be applied to computer equipment. To address the problem of poor training performance caused by high communication overhead in distributed training of neural network models, this invention provides a computational resource allocation method. In related technologies, pipelined parallelism offers better performance due to less data transmission when bandwidth between computing devices (i.e., training nodes) is low. However, pipelined parallel scheduling schemes in related technologies suffer from poor overall performance because adjacent computational tasks (forward and backward computational tasks) and communication tasks (send and receive tasks) are data-dependent and can only be executed serially. The computational resource allocation method of this invention effectively reduces communication overhead by rationally arranging the execution order of computational and communication tasks and utilizing the parallel execution of dependency-free computational and communication tasks. This, in turn, improves the training efficiency of distributed training of neural network models. This invention can effectively improve resource utilization and accelerate the training process even with limited communication bandwidth.

[0074] The computational resource allocation method of this invention improves the efficiency of distributed training by rationally arranging the execution intervals between dependent forward and backward computation tasks, enabling computational and communication tasks to be executed in parallel. Figure 1 As shown, the method includes the following steps:

[0075] 101. Obtain the computation time and communication time of the sub-models on the training nodes when performing computational tasks and communication tasks.

[0076] This step presupposes distributed training of the neural network, meaning the neural network model comprises multiple sub-models, each deployed one-to-one on multiple training nodes. Specifically, when the neural network model is split and pipelined in parallel distributed training using N computing devices (i.e., N training nodes), the neural network model is evenly divided into N pipeline stages (i.e., N sub-models). Each of the N sub-models is configured one-to-one on the N training nodes, and the sequence number of each pipeline stage is represented by stage_id, where id ranges from 0 to N-1. When a neural network model is executed in a pipelined manner, the forward computation task requires N pipelines with stage_ids ranging from small to large to execute the forward computation task in sequence. The computation result of each pipeline stage is sent to the next pipeline stage as input through a communication task (i.e., the send task), and the next pipeline stage receives the computation result of the previous pipeline stage through a communication task (i.e., the receive task). Similarly, when the backward computation task obtains the gradient, it needs to be executed in descending order of stage_id. Accordingly, the gradient is sent to the previous pipeline stage as input through a communication task.

[0077] Based on the above premise, the computation time of the sub-model executing computational tasks and the communication time of executing communication tasks on the training node are obtained. For example, the time spent by a single pipeline stage executing a single forward, backward, send, and receive task is measured on an actual deployed computing device (i.e., the computation time of executing computational tasks and the communication time of executing communication tasks).

[0078] 102. The delay steps are obtained based on the calculation time and the communication time.

[0079] In one embodiment of the present invention, the computation time and the communication time are compared; if the communication time is less than the computation time, the delay step is set as a first delay step; otherwise, the delay step is set as a second delay step, wherein the first delay step is less than the second delay step. In a specific example, the first delay step is, but not limited to, 2, and the second delay step is, but not limited to, 3.

[0080] Specifically, it determines whether the time (communication time) spent on a single send task and receive task is less than the computation time of a single forward task. If it is less, the delay step k is set to 2, for example; otherwise, the delay step k is set to 3.

[0081] 103. Based on the delay steps, the size of the data sample, and the number of training nodes, obtain a scheduling scheme for multiple computing tasks and multiple communication tasks executed on the training nodes, wherein the size of the data sample includes the total size of the data sample and the size of the sub-data divided from the data sample.

[0082] In a specific example, based on the delay steps, the size of the data sample, and the number of training nodes, a scheduling scheme for multiple computational tasks and multiple communication tasks executed on the training nodes is obtained, including: obtaining multiple computational tasks executed on the training nodes based on the delay steps, the size of the data sample, and the number of training nodes; sorting the multiple computational tasks; setting a communication task for each of the multiple computational tasks to obtain a sorting of the multiple computational tasks and the multiple communication tasks; and obtaining a scheduling scheme for the multiple computational tasks and the multiple communication tasks based on the sorting of the multiple computational tasks and the multiple communication tasks.

[0083] In this example, a communication task is set for each of the plurality of computing tasks to obtain the sorting of the plurality of computing tasks and the plurality of communication tasks, including: determining whether the communication task has a dependency relationship with the computing task; if the communication task has no dependency relationship with the computing task, arranging the communication task and the computing task in parallel; if the communication task has a dependency relationship with the computing task, arranging the communication task and the computing task in serial order.

[0084] The determination of whether a communication task has a dependency relationship with a computing task includes: if the data to be communicated by the communication task is data generated after the computing task is executed, or if the computing task is executed based on the data to be communicated by the communication task, then the communication task and the computing task have the dependency relationship; otherwise, the communication task and the computing task do not have the dependency relationship.

[0085] Specifically, a scheduling scheme for multiple computational tasks and multiple communication tasks executed on the training nodes is generated based on the delay step number k (i.e., delay-k), the total number of pipeline stages N (i.e., the number of training nodes), batch_size (i.e., the total size of the data samples), and micro_batch_size (i.e., the size of the sub-data divided from the data samples).

[0086] As can be seen from the model training process, in a complete training iteration (calculating the gradients of the model parameters corresponding to training data of batch size `batch_size`), each pipeline stage needs to execute `batch_size / micro_batch_size` forward computation tasks and backward computation tasks. Task scheduling needs to reasonably arrange these computation tasks and the corresponding communication tasks.

[0087] The generation of the scheduling scheme, i.e., the generation of the layout scheme, is accomplished through the following sub-steps:

[0088] Arrange the forward and backward orders. In a specific example of this invention, the forward and backward orders are arranged according to the following rules: the value of delay-k (i.e., the delay step number K, K is 2 or 3 obtained above), in the case of a total of N pipeline stages, the pipeline stage with stage_id X (X is any positive integer from 0 to N-1), the number of computation tasks between the forward and the corresponding backward (corresponding backward means that the backward depends on the forward) is (N-1-X)*K.

[0089] For each forward and backward task, corresponding send and / or receive tasks are assigned. Computation and communication are independent and can be arranged in parallel on different streams, thus generating a delay-k scheduling scheme (i.e., a scheduling scheme). The diagrams of delay-2 (i.e., delay steps of 2) and delay-3 (i.e., delay steps of 3) scheduling schemes are shown below. Figure 2 and Figure 3 As shown, in Figure 2 and Figure 3 In the diagram, the horizontal axis represents time, the vertical axis represents pipeline stages with different stage_ids, F represents forward, B represents backward, S represents send, and R represents receive. For example... Figure 2 and Figure 3 As shown, in a steady state (i.e., the phase in which forward and backward tasks are executed alternately), both delay-2 and delay-3 can hide communication overhead by executing forward and backward and send and receive tasks on different streams, thereby improving training efficiency.

[0090] 104. Generate a sequence of scheduling instructions for the multiple computing tasks and multiple communication tasks according to the scheduling scheme.

[0091] In a specific example, based on the order of the multiple computing tasks and multiple communication tasks in the scheduling scheme, each of the multiple computing tasks and multiple communication tasks is converted into a scheduling instruction, and the scheduling instruction sequence is obtained based on the scheduling instruction converted for each task.

[0092] Specifically, the delay-k scheduling scheme is transformed into a command sequence that is easy for the executor to execute. All tasks in the scheduling scheme are sorted according to their data dependency topology, and each task is converted into a corresponding command, resulting in the command sequence required by the executor. The attributes of each command and their meanings are shown in Table 1.

[0093] Table 1

[0094]

[0095] 105. Control the training node to train the sub-model deployed on the training node according to the data sample based on the scheduling instruction sequence.

[0096] That is, the training of the model is carried out according to the command sequence generated by the delay-k scheduling scheme.

[0097] A diagram illustrating the sequence of commands executed by the executor is shown below. Figure 4 As shown, the executor consists of a command queue, tensorstorage (for storing tensors generated during command execution), and eventstorage (for storing events generated during command execution). The executor first preprocesses the command sequence, marking the first and last commands to be executed in the same batch. For commands with the same `comm_group` attribute value, the first command is marked as the start of the group, and the last command as the end of the group. After preprocessing, it iterates through and executes each command in the command queue. The executor's processing flow for a single command is as follows:

[0098] Check if the command's wait attribute value is empty. If it is not empty, retrieve the event from event_storage with the wait attribute value as the key, and wait for the event's execution status to complete before continuing the subsequent process.

[0099] Determine if the command is a group start command; if so, call the group_start function to enter the batch communication operation area.

[0100] The corresponding task execution function is invoked based on the command's `type` attribute. If `type` is `forward`, `backward`, or `receive`, the tensor generated by the task execution is stored in `tensor_storage`, with the command's `tag` attribute as the storage key. If `type` is `send`, the tensor corresponding to the command's `tag` attribute is retrieved from `tensor_storage`, deleted, and then sent.

[0101] Determine if the command is a group end command; if so, call the group_end function to exit the batch communication operation area.

[0102] Check if the value of the `reader_stream` attribute of the command is empty. If it is not empty, move the tensor generated by the execution of the command to the stream where `reader_stream` is located.

[0103] Check if the command record attribute is empty. If it is not empty, create a device event and store the device event in event_storage, with the storage key being the value of the command record attribute.

[0104] According to the computing resource allocation method of this invention, different delay steps are determined based on the difference between the computing time of the sub-models executing computing tasks and the communication time of executing communication tasks on the training nodes. Then, based on the delay steps, the size of the data samples, and the number of training nodes, a scheduling scheme for multiple computing tasks and multiple communication tasks on each training node can be determined. A scheduling instruction sequence is generated for the multiple computing tasks and multiple communication tasks according to the scheduling scheme. Finally, the training nodes are controlled to train the sub-models deployed on the training nodes according to the data samples based on the scheduling instruction sequence. Through the scheduling scheme, computing tasks and communication tasks that have no dependency relationship during model training can be executed in parallel. In this way, the communication overhead of the model during distributed training is reduced, and the training efficiency of the neural network model during distributed training is improved.

[0105] Figure 5 This is a block diagram of a computing resource allocation device according to an embodiment of this application.

[0106] like Figure 5 As shown, the computing resource allocation device is used for training a neural network model. The computing resource allocation device includes: an acquisition module 510, a delay step calculation module 520, a scheduling scheme acquisition module 530, a scheduling instruction generation module 540, and a scheduling instruction execution module 550, wherein:

[0107] The acquisition module 510 is used to obtain the computation time and communication time of the sub-model executing computation tasks and executing communication tasks on the training node;

[0108] The delay step calculation module 520 is used to obtain the delay step based on the calculation time and the communication time;

[0109] The scheduling scheme acquisition module 530 is used to obtain a scheduling scheme for multiple computing tasks and multiple communication tasks executed on the training nodes based on the delay steps, the size of the data sample and the number of the multiple training nodes, wherein the size of the data sample includes the total size of the data sample and the size of the sub-data divided from the data sample.

[0110] The scheduling instruction generation module 540 is used to generate a sequence of scheduling instructions for the plurality of computing tasks and the plurality of communication tasks according to the scheduling scheme.

[0111] The scheduling instruction execution module 550 is used to control the training node to train the sub-model deployed on the training node according to the data sample based on the scheduling instruction sequence.

[0112] In one embodiment of the present invention, the delay step calculation module 520 is specifically used for:

[0113] Compare the computation time and the communication time;

[0114] If the communication time is less than the calculation time, the delay step number is set as the first delay step number; otherwise, the delay step number is set as the second delay step number, wherein the first delay step number is less than the second delay step number.

[0115] In one embodiment of the present invention, the scheduling scheme acquisition module 530 is specifically used for:

[0116] Based on the number of delay steps, the size of the data sample, and the number of training nodes, multiple computational tasks executed on the training nodes are obtained, and the multiple computational tasks are sorted.

[0117] A communication task is set for each of the plurality of computing tasks, resulting in a sorting of the plurality of computing tasks and the plurality of communication tasks;

[0118] A scheduling scheme for the multiple computing tasks and multiple communication tasks is obtained based on the order of the multiple computing tasks and multiple communication tasks.

[0119] According to the computing resource allocation device of the present invention, different delay steps are determined based on the difference between the computing time of the sub-models on the training nodes executing computing tasks and the communication time of executing communication tasks. Then, based on the delay steps, the size of the data samples, and the number of training nodes, a scheduling scheme for multiple computing tasks and multiple communication tasks on each training node can be determined. A scheduling instruction sequence is generated for the multiple computing tasks and multiple communication tasks according to the scheduling scheme. Finally, the scheduling instruction sequence is used to control the training nodes to train the sub-models deployed on the training nodes according to the data samples. Through the scheduling scheme, computing tasks and communication tasks that have no dependency relationship during model training can be executed in parallel. In this way, the communication overhead of the model during distributed training is reduced, and the training efficiency of the neural network model during distributed training is improved.

[0120] It should be understood that the units recorded in the computing resource allocation device and the reference Figure 1 The steps in the described computing resource allocation method correspond to each other. Therefore, the operations and features described above for the method also apply to the computing resource allocation device and its constituent units, and will not be repeated here. The computing resource allocation device can be pre-implemented in the browser or other security applications of a computer device, or it can be loaded into the browser or its security applications of a computer device through download or other means. The corresponding units in the computing resource allocation device can cooperate with the units in the computer device to implement the solutions of the embodiments of this application.

[0121] The division of modules or units mentioned in the detailed description above is not mandatory. In fact, according to the embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0122] It should be noted that for details not disclosed in the computing resource allocation device of this application embodiment, please refer to the details disclosed in the above embodiments of this application, which will not be repeated here.

[0123] The following is for reference. Figure 6 , Figure 6 A schematic diagram of a computer device suitable for implementing embodiments of this application is shown, such as... Figure 6As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1302 or programs loaded from storage section 1308 into random access memory (RAM) 1303. The RAM 1303 also stores various programs and data required for the system's operating instructions. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0124] The following components are connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed so that computer programs read from them can be installed into storage section 1308 as needed.

[0125] Specifically, according to embodiments of this application, the flowchart above refers to... Figure 1 The described process can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program contains program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1309, and / or installed from removable medium 1311. When the computer program is executed by central processing unit (CPU) 1301, it performs the functions defined in the system of this application.

[0126] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operational instructions of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two connected blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operational instructions, or using a combination of dedicated hardware and computer instructions.

[0128] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a first receiving module, a second receiving module, and a transmitting module. The names of these units or modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0129] In another aspect, this application also provides a computer-readable storage medium, which may be included in the computer device described in the above embodiments, or may exist independently and not assembled into the computer device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the computing resource allocation method described in this application.

[0130] In another aspect, this application also provides a computer program product, which may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device. The computer program product stores one or more programs that are executed by one or more processors according to the computing resource allocation described in this application.

[0131] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method of allocating computing resources, characterized by, The computing resource allocation method is used for training a neural network model, which includes multiple sub-models, each deployed on multiple training nodes in a one-to-one correspondence. The method includes: Obtain the computation time and communication time of the sub-model executing computational tasks and executing communication tasks on the training node; Compare the computation time and the communication time; If the communication time is less than the computation time, the delay step is set as the first delay step; otherwise, the delay step is set as the second delay step. The first delay step is less than the second delay step. The delay step is used to characterize the number of computation tasks between the forward computation task and the corresponding backward computation task of the sub-model on the training node. Based on the number of delay steps, the size of the data sample, and the number of training nodes, a scheduling scheme for multiple computing tasks and multiple communication tasks executed on the training nodes is obtained, wherein the size of the data sample includes the total size of the data sample and the size of the sub-data divided from the data sample; Generate a sequence of scheduling instructions for the plurality of computing tasks and the plurality of communication tasks according to the scheduling scheme; The training node is controlled according to the scheduling instruction sequence to train the sub-models deployed on the training node based on the data samples.

2. The computing resource allocation method according to claim 1, characterized in that, The first delay step is 2, and the second delay step is 3.

3. The computing resource allocation method according to any one of claims 1-2, characterized in that, The step of obtaining a scheduling scheme for multiple computational tasks and multiple communication tasks executed on the training nodes based on the delay steps, the size of the data samples, and the number of training nodes includes: Based on the number of delay steps, the size of the data sample, and the number of training nodes, multiple computational tasks executed on the training nodes are obtained, and the multiple computational tasks are sorted. A communication task is set for each of the plurality of computing tasks, resulting in a sorting of the plurality of computing tasks and the plurality of communication tasks; A scheduling scheme for the multiple computing tasks and multiple communication tasks is obtained based on the order of the multiple computing tasks and multiple communication tasks.

4. The computing resource allocation method according to claim 3, characterized in that, The step of generating a sequence of scheduling instructions for the plurality of computing tasks and the plurality of communication tasks according to the scheduling scheme includes: Based on the order of the multiple computing tasks and multiple communication tasks in the scheduling scheme, each of the multiple computing tasks and multiple communication tasks is converted into a scheduling instruction. The scheduling instruction sequence is obtained based on the scheduling instructions for each task transition.

5. The computing resource allocation method according to claim 3, characterized in that, The step of setting a communication task for each of the plurality of computing tasks, and obtaining the order of the plurality of computing tasks and the plurality of communication tasks, includes: Determine whether the communication task has a dependency relationship with the computing task; If the communication task and the computing task do not have a dependency relationship, the communication task and the computing task are arranged in parallel. If the communication task is dependent on the computing task, the communication task and the computing task are arranged in sequence.

6. The computing resource allocation method according to claim 5, characterized in that, The step of determining whether the communication task has a dependency relationship with the computing task includes: If the data to be communicated by the communication task is data generated after the execution of the computing task, or if the computing task is executed based on the data to be communicated by the communication task, then the communication task and the computing task have the dependency relationship; otherwise, the communication task and the computing task do not have the dependency relationship.

7. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the computing resource allocation method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, It stores a computer program that implements the computing resource allocation method according to any one of claims 1-6.

9. A computer program product, characterized in that, It stores a computer program that implements the computing resource allocation method according to any one of claims 1-6.

Images