Task scheduling method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a task scheduling method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, in particular to the technical field of distributed data processing, deep learning and large models. The method comprises the steps that a scheduling strategy used for model training is obtained, the scheduling strategy is used for distributing a plurality of calculation tasks to a plurality of calculation units based on an interlaced pipeline mechanism, and execution time intervals are distributed to the calculation tasks in sequence; a specified forward calculation task and a target reverse calculation task are determined from at least one calculation task related to the same batch of data, and the number of intervals of execution time intervals between the target reverse calculation task and the specified forward calculation task meets a preset condition; the execution time interval of the target reverse calculation task is adjusted until the number of the adjusted intervals is smaller than a preset number threshold value, and a target scheduling strategy is obtained; and scheduling the computing unit to execute the computing task based on the target scheduling strategy.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of distributed data processing, deep learning and large model technology. Background Art

[0002] In the field of deep learning, the training tasks of computing units can be scheduled based on distributed strategies to improve the training efficiency of deep learning models. Distributed strategies can include interleaved pipeline strategies, which can further optimize the task execution efficiency of multiple computing units and reduce the bubble occupancy rate during training. Summary of the invention

[0003] The present disclosure provides a task scheduling method, device, electronic device, storage medium and program product.

[0004] According to one aspect of the present disclosure, a task scheduling method is provided, including: obtaining a scheduling strategy for model training, the scheduling strategy being used to allocate multiple computing tasks to multiple computing units based on an interleaved pipeline mechanism, the multiple computing tasks being sequentially allocated execution time intervals, the computing tasks being used for at least one processing layer of the model to process batch data; determining a specified forward computing task and a target reverse computing task from at least one computing task associated with the same batch data, the specified forward computing task and the target reverse computing task being associated with a specified processing layer of the model, the number of intervals of the execution time interval between the target reverse computing task and the specified forward computing task meeting a preset condition; adjusting the execution time interval of the target reverse computing task until the adjusted number of intervals is less than a preset number threshold, thereby obtaining a target scheduling strategy; and scheduling the computing units to perform computing tasks based on the target scheduling strategy.

[0005] According to another aspect of the present disclosure, a task scheduling device is provided, including: an acquisition module, used to acquire a scheduling strategy for model training, the scheduling strategy is used to allocate multiple computing tasks to multiple computing units based on an interleaved pipeline mechanism, the multiple computing tasks are sequentially allocated with execution time intervals, and the computing tasks are used for at least one processing layer of the model to process batch data; a first determination module, used to determine a specified forward computing task and a target reverse computing task from at least one computing task related to the same batch data, the specified forward computing task and the target reverse computing task are related to the specified processing layer of the model, and the number of intervals of the execution time interval between the target reverse computing task and the specified forward computing task meets a preset condition; an adjustment module, used to adjust the execution time interval of the target reverse computing task until the adjusted number of intervals is less than a preset number threshold, thereby obtaining a target scheduling strategy; and an execution module, used to schedule the computing units to execute computing tasks based on the target scheduling strategy.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by an embodiment of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided by an embodiment of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method provided by the embodiment of the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 An exemplary system architecture to which the task scheduling method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0012] Figure 2 A flowchart of a task scheduling method according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 3 The application scenario diagram of the task scheduling method according to the embodiment of the present disclosure is schematically shown;

[0014] Figure 4A A schematic diagram schematically shows a scheduling strategy according to an embodiment of the present disclosure;

[0015] Figure 4B A schematic diagram schematically shows a target scheduling strategy according to an embodiment of the present disclosure;

[0016] Figure 5 A flowchart schematically shows a method of scheduling a computing unit to perform a computing task based on a target scheduling strategy according to an embodiment of the present disclosure;

[0017] Figure 6 A block diagram schematically shows a task scheduling device according to an embodiment of the present disclosure; and

[0018] Figure 7 A schematic block diagram of an example electronic device that can be used to implement the task scheduling method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0021] The inventors found that scheduling the training process of deep learning models based on a distributed strategy has problems such as low computing efficiency of computing units and long training time.

[0022] Embodiments of the present disclosure provide a task scheduling method, device, electronic device, storage medium and program product. The task scheduling method includes: obtaining a scheduling strategy for model training, the scheduling strategy is used to allocate multiple computing tasks to multiple computing units based on an interleaved pipeline mechanism, the multiple computing tasks are sequentially allocated with execution time intervals, and the computing tasks are used for at least one processing layer of the model to process batch data; from at least one computing task related to the same batch data, determine a specified forward computing task and a target reverse computing task, the specified forward computing task and the target reverse computing task are related to the specified processing layer of the model, and the number of intervals of the execution time interval between the target reverse computing task and the specified forward computing task meets a preset condition; adjust the execution time interval of the target reverse computing task until the adjusted number of intervals is less than a preset number threshold, and obtain a target scheduling strategy; and schedule the computing unit to perform the computing task based on the target scheduling strategy.

[0023] According to an embodiment of the present disclosure, by determining the designated forward computing task and the target reverse computing task from the scheduling strategy determined based on the staggered pipeline mechanism, it can be preliminarily determined that the factors affecting the computing efficiency of the computing unit may include that after the designated forward computing task is executed, the computing unit executing the designated forward computing task stores the computing result of the designated forward computing task in a plurality of execution time intervals that meet preset conditions, resulting in an imbalance in the storage space between the plurality of computing units and affecting the computing efficiency. By adjusting the execution time interval of the target reverse computing task so that the number of intervals between the target reverse computing task and the designated forward computing task is less than a preset number threshold, the storage time of the computing result of the designated forward computing task in the computing unit can be reduced, thereby avoiding redundant storage resource settings caused by storage imbalance between the plurality of computing units during the model training process, and avoiding the computing unit from reducing the model training efficiency due to storage resource overload, thereby improving the overall training efficiency of the distributed model training.

[0024] Figure 1 An exemplary system architecture to which the task scheduling method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0025] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the task scheduling method and apparatus can be applied may include a terminal device, but the terminal device may implement the task scheduling method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0026] like Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0027] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0028] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0029] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0030] The server 105 may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0031] It should be noted that the task scheduling method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the task scheduling device provided in the embodiment of the present disclosure can generally be set in the server 105. The task scheduling method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the task scheduling device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] Figure 2 The flowchart of the task scheduling method according to the embodiment of the present disclosure is schematically shown.

[0034] like Figure 2 As shown, the task scheduling method includes operations S210 to S240.

[0035] In operation S210, a scheduling strategy for model training is obtained.

[0036] In operation S220, a designated forward computing task and a target reverse computing task are determined from at least one computing task associated with the same batch of data.

[0037] In operation S230, the execution time interval of the target reverse computing task is adjusted until the number of intervals after the adjustment is less than a preset number threshold, thereby obtaining a target scheduling strategy.

[0038] In operation S240 , the computing unit is scheduled to perform the computing task based on the target scheduling policy.

[0039] According to an embodiment of the present disclosure, a scheduling strategy is used to allocate multiple computing tasks to multiple computing units based on an interleaved pipeline mechanism, and the multiple computing tasks are sequentially allocated execution time intervals.

[0040] According to an embodiment of the present disclosure, a computing task is used for at least one processing layer of a model to process batch data. The batch data may be obtained by dividing the sample data, and the batch data may include batch data or micro-batch data. The embodiment of the present disclosure does not limit the dimension or data volume of the batch data.

[0041] According to an embodiment of the present disclosure, the model may include a deep learning model constructed based on any algorithm.

[0042] In one example, the model may be a large model built based on a deep learning algorithm, such as a large language model (LLM), or the model may be other large models with a large parameter scale.

[0043] According to an embodiment of the present disclosure, a computing unit may include a processor for performing computing tasks, such as a graphics processing unit (GPU), a central processing unit (CPU), etc. Alternatively, the computing unit may also be a processor module of a processor, for example, the same GPU may be divided into multiple computing units. Alternatively, the computing unit may also be a collection of multiple processors or computing devices, for example, multiple GPUs may be divided into the same computing unit. The embodiments of the present disclosure do not limit the specific setting method of the computing unit.

[0044] According to an embodiment of the present disclosure, a scheduling strategy based on an interleaved pipeline mechanism can divide the model into multiple different processing layers, and distribute the multiple processing layers to multiple computing units. The sample data used to train the model can be divided into multiple batches of data, and multiple computing units can train the model by executing the computing tasks of multiple batches of data in parallel. Intermediate results can be transmitted between multiple computing units to execute the computing tasks of each processing layer. Among them, the computing tasks may include forward computing tasks and backward computing tasks. The execution time interval can be understood as the time step of the computing task, and the computing unit can execute the computing task in the execution time interval to obtain the intermediate results.

[0045] In one example, the scheduling strategy can be used to indicate the computing tasks of each batch of data assigned to each computing unit, and the computing tasks can be arranged based on the execution time interval (or time step). Based on the staggered pipeline mechanism, the computing tasks of multiple batches of data are assigned to multiple computing units, so that multiple computing units can execute computing tasks in parallel, thereby improving the training efficiency of the model.

[0046] According to an embodiment of the present disclosure, a specified forward computing task and a target reverse computing task are related to a specified processing layer of a model, and the number of intervals of execution time intervals between the target reverse computing task and the specified forward computing task meets a preset condition, for example, the number of intervals of execution time intervals between the target reverse computing task and the specified forward computing task is greater than or equal to a preset number threshold.

[0047] According to an embodiment of the present disclosure, the designated processing layer may include any one or more of the multiple processing layers of the model. The designated forward computing task may be a forward computing task for the designated processing layer to perform the i-th batch of data, and the target reverse computing task may be a reverse computing task for the designated processing layer to perform the i-th batch of data. The designated forward computing task and the target reverse computing task may be computing tasks of the same computing unit.

[0048] According to an embodiment of the present disclosure, the intermediate result obtained by the computing unit after executing the specified forward computing task can be stored in a specified storage area. For example, it can be cached in the cache area of ​​the computing unit. When the number of intervals of the execution time intervals between the target reverse computing task and the specified forward computing task meets the preset conditions, it can be indicated that the intermediate result of the specified forward computing task is stored in the cache area of ​​the computing unit for the number of intervals of execution time intervals. When the number of intervals meets the preset conditions, it can be indicated that the storage time of the intermediate result of the specified forward computing task is too long, which will result in the storage space of other intermediate results of the computing unit executing other computing tasks being limited, and may also cause the computing unit to affect computing performance due to limited cache space during the execution of other computing tasks.

[0049] According to an embodiment of the present disclosure, by adjusting the execution time interval of the target reverse computing task, the number of execution time intervals between the associated specified forward computing task and the target reverse computing task in the target scheduling strategy is made less than a preset number threshold, so that after the execution of the specified forward computing task is completed, the intermediate results can be stored in a smaller number of execution time intervals. And after a certain number of execution time intervals, the computing unit uses the intermediate results to execute the target reverse computing task, so as to release the intermediate results of the specified forward computing task in a timely manner. Reduce the storage time of the calculation result of the specified forward computing task in the computing unit, and avoid the imbalance of storage space between multiple computing units affecting the execution of computing tasks and the transmission of intermediate results between multiple computing units.

[0050] In one example, the model to be trained may be a deep learning model built based on an attention network algorithm, and the deep learning model may include multiple attention sublayers and a normalization layer connected after the last attention sublayer. The specified processing layer includes an attention network sublayer and a normalization sublayer. The normalization sublayer may output the model calculation result of the model, and the model calculation result may include any type of data such as text, image, vector, etc.

[0051] According to an embodiment of the present disclosure, the batch data includes at least one of the following: image data, audio data, and text data.

[0052] In one example, the batch data may include a batch of image data. The model calculation result may include a target detection result for the image data. Alternatively, the model calculation result may also include a generated new image.

[0053] In one example, the audio data may be encoded based on the model to obtain audio features, and the audio features may be processed through multiple processing layers to obtain a model calculation result. The model calculation result may be, for example, a text recognition result for the audio data.

[0054] In one example, text data may include any type of text, such as question text, news text, etc., and the model calculation result may include text data, such as answer text, news rewrite text, etc.

[0055] It should be noted that the above examples are only for schematically illustrating the model calculation results obtained after the model and calculation tasks involved in the embodiments of the present disclosure process different types of batch data, and are not intended to limit the actual processing capabilities of the model.

[0056] According to an embodiment of the present disclosure, the ratio between the number of batches of batch data and the number of computing units is a non-integer. The number of batches can represent the cumulative number of times of the staggered pipeline parallel mechanism, and the number of computing units can represent the staggered pipeline segmentation dimension of the staggered pipeline parallel mechanism. In the case where the ratio between the number of batches of batch data and the number of computing units is a non-integer, it can be understood that the number of batches cannot be divided by the number of computing units, resulting in a scheduling strategy based on the staggered pipeline mechanism. It is necessary to use the storage interval of the computing unit to store the intermediate results of the specified forward computing task for a longer time. By adjusting the execution time interval of the target reverse computing task in the scheduling strategy, the execution interval between the specified forward computing task and the target reverse computing task can be shortened, fully releasing the computing resources and storage space of the computing unit.

[0057] According to an embodiment of the present disclosure, adjusting the execution time interval of the target reverse computing task until the number of adjusted intervals is less than a preset number threshold, and obtaining the target scheduling strategy may include: adjusting the execution time interval of the target reverse computing task to obtain the Nth reverse execution time interval of the target reverse computing task; and adjusting the execution order of the target reverse computing task based on the Nth reverse execution time interval to obtain the target scheduling strategy.

[0058] According to an embodiment of the present disclosure, adjusting the execution order of the target reverse computing task based on the Nth reverse execution time interval can include moving the target reverse computing task in the task scheduling timing diagram represented by the scheduling strategy to a position corresponding to the Nth reverse execution time interval to obtain a task scheduling timing diagram representing the target scheduling strategy.

[0059] According to an embodiment of the present disclosure, the number of intervals between the Nth reverse execution time interval and the execution time interval of the specified forward computing task is less than a preset number threshold. Therefore, the intermediate results of the specified forward computing task in the target scheduling strategy can cache a smaller number of execution time intervals, thereby enabling the computing unit to release the storage interval of the computing unit by executing the target reverse computing task.

[0060] In one example, a specified forward computing task is related to the last processing layer among multiple processing layers, and the intermediate result of the specified forward computing task may include a model computing result. However, this is not limited to the above, and the intermediate result of the specified forward computing task may also include a loss value obtained by processing the model computing result with a loss function. The model computing result and loss value of the specified forward computing task in the target scheduling strategy may cache a smaller number of execution time intervals to avoid storage space imbalance among multiple computing units and improve the computing power resource balance attribute and storage space balance of multiple computing units.

[0061] According to an embodiment of the present disclosure, the number of execution time intervals between the forward computing task and the target reverse computing task specified in the target scheduling strategy is 0. Therefore, based on the target scheduling strategy, the Nth computing unit can start executing the target reverse computing task in the next execution time interval after executing the forward computing task. After the target reverse computing task is completed, the Nth computing unit sends the intermediate result of the target reverse computing task to the N-1th computing unit. In this way, the storage interval of the Nth computing unit can be released in time, avoiding the intermediate computing result being stored in the Nth computing unit for a long time, resulting in an imbalance in the storage space between multiple computing units.

[0062] According to an embodiment of the present disclosure, N computing units have a unit sequence attribute, and the unit sequence attribute is related to the execution order of the computing tasks of the computing units, N>1, and the computing task of the Nth computing unit for the i-th batch of data includes a target reverse computing task. i is any integer greater than or equal to 1.

[0063] Figure 3 The application scenario diagram of the task scheduling method according to an embodiment of the present disclosure is schematically shown.

[0064] like Figure 3 As shown, the application scenario includes a first computing unit 301, a second computing unit 302, a third computing unit 303, and a fourth computing unit 304 with a unit order attribute. The fourth computing unit may be associated with a specified forward computing task and a target reverse computing task of the i-th batch of data.

[0065] The target scheduling strategy 310 can control and instruct the first computing unit 301, the second computing unit 302, the third computing unit 303 and the fourth computing unit 304 to perform computing tasks. The target scheduling strategy 310 includes a designated forward task block J311 corresponding to the designated forward computing task of the sixth batch of data, and the target scheduling strategy 310 includes a target reverse task block J312 corresponding to the target reverse computing task of the sixth batch of data. The execution time interval between the designated forward task block J311 and the target reverse task block J312 is 0. By scheduling tasks for the fourth computing unit 304 based on the target scheduling strategy 310, the fourth computing unit 304 can perform the target reverse computing task of the sixth batch of data in the next execution time interval after performing the designated forward computing task of the sixth batch of data, thereby releasing the storage interval of the fourth computing unit 304 by transmitting the intermediate result of the target reverse computing task of the sixth batch of data to the third computing unit 303.

[0066] According to an embodiment of the present disclosure, adjusting the execution order of the target reverse computing tasks based on the Nth reverse execution time interval to obtain a target scheduling strategy includes: adjusting the execution order of the target reverse computing tasks based on the Nth reverse execution time interval to obtain an intermediate scheduling strategy; and adjusting the execution order of the reverse computing tasks of the n-1th computing unit based on the execution order of the target reverse computing tasks in the intermediate scheduling strategy to obtain the target scheduling strategy.

[0067] According to an embodiment of the present disclosure, the execution order of the reverse computing tasks of the N-1th to 1st computing units in the intermediate scheduling strategy is adjusted based on the staggered pipeline to obtain a target scheduling strategy. After the target reverse computing unit is executed, the N-1th to 1st computing units will execute their respective reverse computing tasks in turn in a staggered pipeline manner, thereby preventing the N-1th to 1st computing units from storing the intermediate results for too long, thereby improving the overall efficiency of model training.

[0068] According to an embodiment of the present disclosure, adjusting the execution order of the reverse computing tasks of the n-1th computing unit based on the execution order of the target reverse computing tasks in the intermediate scheduling strategy includes: adjusting the execution order of the n-1th associated reverse computing tasks of the n-1th computing unit based on the execution order of the target reverse computing tasks in the intermediate scheduling strategy, wherein, from the target reverse computing task to the 1st associated reverse computing task, multiple processing layers arranged consecutively in the representation model execute reverse computing tasks in sequence.

[0069] It should be noted that N≥n>1, and n is an integer.

[0070] Figure 4A A schematic diagram of a scheduling strategy according to an embodiment of the present disclosure is schematically shown.

[0071] Figure 4B A schematic diagram of a target scheduling strategy according to an embodiment of the present disclosure is schematically shown.

[0072] like Figure 4A As shown, the scheduling strategy 410 can allocate the computing tasks of each batch of 9 batches of batch data based on the staggered pipeline mechanism to obtain a computing task timing diagram. The computing unit used for model training may include a first computing device D1, a second computing device D2, a third computing device D3, and a fourth computing device D4 with unit sequence attributes. The first computing device D1 can perform the first forward computing task of the first batch of data in the execution time interval 1, and the second computing device D2 can perform the second forward computing task of the second batch of data in the execution time interval 2. At the same time, the fourth computing device can perform the 12th forward computing task of the sixth batch of data in the execution time interval t1. The fourth computing device can perform the 9th backward computing task of the fourth batch of data in the execution time interval t2. The fourth computing device can also perform the 1st backward computing task of the sixth batch of data in the execution time interval t6. The number of intervals of the execution time intervals between the 12th forward computing task of the sixth batch of data and the 1st backward computing task of the sixth batch of data is 4, which meets the preset conditions. Therefore, the 12th forward computing task of the 6th batch of data may be the designated forward computing task of the 6th batch of data. The 1st backward computing task of the 6th batch of data is the target backward computing task of the 6th batch of data.

[0073] Combination Figure 4A and Figure 4B As shown, when the preset number threshold indicates that the number of intervals of the execution time interval is 1, the execution time interval of the first backward computing task of the 6th batch of data can be adjusted, and the fourth reverse execution time interval is t2. The first backward computing task of the 6th batch of data is adjusted to the execution order corresponding to the execution time interval t2 by the fourth reverse execution time interval, and an intermediate scheduling strategy can be obtained. Among them, in the row corresponding to the third computing device D3, the second backward computing task of the 6th batch of data located at the execution time interval t8 is the third associated reverse computing task of the third computing device D3. In the row corresponding to the second computing device D2, the third backward computing task of the 6th batch of data located at the execution time interval t9 is the second associated reverse computing task of the second computing device D2. In the row corresponding to the first computing device D1, the fourth backward computing task of the 6th batch of data located at the execution time interval t10 is the first associated reverse computing task of the first computing device D1. The third associated reverse calculation task, the second associated reverse calculation task and the first associated reverse calculation task are all represented by white square task blocks in the first dotted line area 411 .

[0074] The execution order of the 1st backward computing task based on the 6th batch of data in the intermediate scheduling strategy is expressed based on the execution time interval t2. The positions of the white square task blocks corresponding to the 3rd associated reverse computing task, the 2nd associated reverse computing task and the 1st associated reverse computing task in the first dotted area 411 can be adjusted so that the white square task blocks corresponding to the 3rd associated reverse computing task, the 2nd associated reverse computing task and the 1st associated reverse computing task in the dotted area 411 are located at the positions corresponding to the second dotted area 421, thereby adjusting the execution order of the 3rd associated reverse computing task, the 2nd associated reverse computing task and the 1st associated reverse computing task.

[0075] Combination Figure 4A and Figure 4B As shown, by adjusting the target reverse calculation tasks of the 7th batch of data, the 8th batch of data and the 9th batch of data, and adjusting the 3rd associated reverse calculation task and the 2nd associated reverse calculation task of the target reverse calculation tasks of the 7th batch of data, the 8th batch of data and the 9th batch of data to the 1st associated reverse calculation task, the target adjustment strategy 420 can be obtained.

[0076] From the target reverse calculation task, the third associated reverse calculation task, the second associated reverse calculation task to the first associated reverse calculation task, it can be represented that multiple processing layers arranged continuously in the model perform reverse calculation tasks in sequence.

[0077] It should be understood that Figure 4A and Figure 4B The rectangular task blocks in the middle represent forward computing tasks, and the square task blocks represent reverse computing tasks.

[0078] It should be noted that Figure 4A and Figure 4B The ellipsis “…” in the table is used to indicate omitted parts in the scheduling policy and the target scheduling policy.

[0079] According to an embodiment of the present disclosure, the designated forward computing task includes an Nth forward computing subtask and a loss computing task, and the Nth forward computing subtask represents that the Nth computing unit processes the N-1th forward intermediate result sent by the N-1th computing unit using the designated processing layer.

[0080] In one example, the designated processing layer may include the last processing layer into which the model is divided. For the i-th batch of data, the designated processing layer performs the forward calculation subtask of the i-th batch of data by processing the N-1th forward intermediate result sent by the N-1th calculation unit, thereby obtaining the model calculation result of the i-th batch of data.

[0081] According to an embodiment of the present disclosure, the loss calculation task represents the loss value of the batch data calculated by the Nth computing unit according to the model calculation result output by the specified processing layer. In the target scheduling strategy, the number of execution time intervals between the specified forward computing task and the target reverse computing task is less than the preset number threshold, so that the Nth computing unit can execute the target reverse computing task in a timely manner, and send the reverse calculation result of the target reverse computing task to the N-1th computing unit, so as to improve the utilization rate of the storage interval of the Nth computing unit, avoid long-term caching of intermediate results, and improve the balance of storage space between multiple computing units.

[0082] In one example, the model calculation results of the i-th batch of data can be processed based on the loss function to obtain the loss value of the i-th batch of data.

[0083] According to an embodiment of the present disclosure, the target number of intervals between the forward computing task and the target reverse computing task specified in the target scheduling strategy is K, K≥0, and is less than a preset number threshold.

[0084] In one example, the preset number threshold can be 1, the target interval number between the forward computing task and the target reverse computing task specified in the target scheduling policy is K, and the target interval number is the number of execution time intervals between the forward computing task and the target reverse computing task specified in the target scheduling policy.

[0085] Figure 5 The flowchart of scheduling computing units to execute computing tasks based on a target scheduling strategy according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 5 As shown, scheduling computing units to perform computing tasks based on a target scheduling strategy includes operations S510 to S540.

[0087] In operation S510, based on the reference execution time interval of the specified forward computing task, the Nth computing unit calls the model computing result stored in the Nth storage area to perform the loss computing subtask and obtains the loss value stored in the Nth storage area;

[0088] In operation S520, in a reference execution time interval that is K execution time intervals away from the base execution time interval, the Nth calculation unit performs the target reverse calculation task based on the loss value to obtain an Nth reverse intermediate result stored in the Nth storage area;

[0089] In operation S530, the Nth reverse intermediate result is transmitted to the N-1th computing unit.

[0090] In operation S540, the Nth storage region is released.

[0091] According to an embodiment of the present disclosure, the benchmark execution time interval may be the execution time interval of the specified forward computing task of the i-th batch of data in the target scheduling strategy. In the benchmark execution time interval, the N-th computing unit calls the model computing result stored in the N-th storage area to execute the loss computing subtask and obtain the loss value, which may be stored in the N-th storage area.

[0092] According to an embodiment of the present disclosure, the reference execution time interval may be the execution time interval of the target reverse calculation task of the i-th batch of data. In the reference execution time interval, the N-th calculation unit may obtain the model calculation result of the i-th batch of data from the N-th storage area to obtain the loss value of the i-th batch of data. The loss value is transmitted to the N-1-th calculation unit as the N-th reverse intermediate result.

[0093] In one example, the computing unit may be a GPU unit, and the Nth storage area may be a video memory space of the GPU unit. Releasing the Nth storage area may include deleting the loss value of the Nth storage area, so as to avoid the accumulation of activation loss values ​​calculated by the loss function due to the batch number of batch data not being divisible by the number of computing units, thereby causing an imbalance in the video memory space between multiple computing units.

[0094] According to an embodiment of the present disclosure, releasing the Nth storage area includes: deleting the loss value and the Nth reverse intermediate result stored in the Nth storage area during the process of transmitting the Nth reverse intermediate result to the N-1th computing unit.

[0095] According to an embodiment of the present disclosure, by deleting the loss value and the Nth reverse intermediate result stored in the Nth storage area during the process of transmitting the Nth reverse intermediate result to the N-1th computing unit, the Nth computing unit can promptly convert the Nth storage area to a storage idle state, so that the Nth computing unit can execute the next computing task more promptly, avoiding the accumulation of activation values ​​calculated by the loss function, thereby improving the overall training efficiency of the model training.

[0096] According to an embodiment of the present disclosure, the Nth storage area may be a cache area of ​​the Nth computing unit. For example, the Nth storage area may be a video memory module of a GPU chip. The task scheduling method provided in the embodiment of the present disclosure can release the storage area of ​​the computing unit in a more timely manner, thereby improving the occupancy rate of the video memory space of the graphics card. In one example, the task scheduling method based on the embodiment of the present disclosure can save 10G video memory space.

[0097] According to an embodiment of the present disclosure, the N-1th forward intermediate result sent by the N-1th computing unit is related to the i-th batch of data.

[0098] According to an embodiment of the present disclosure, scheduling computing units to perform computing tasks based on a target scheduling strategy also includes: in the next execution time interval of the reference execution time interval, the Nth computing unit uses a designated processing layer to process the N-1th forward intermediate result related to the i+1th batch of data to obtain a model calculation result related to the i+1th batch of data.

[0099] According to an embodiment of the present disclosure, the model calculation results related to the i+1th batch of data are stored in the Nth storage area. The Nth computing unit releases the Nth storage area in the process of transmitting the Nth reverse intermediate result to the N-1th computing unit, so that the Nth computing unit can be converted to an idle state in a timely manner in the next execution time interval of the reference execution time interval, so as to use the relatively sufficient free storage interval to execute the specified forward calculation task of the i+1th batch of data and obtain the model calculation results related to the i+1th batch of data. Thereby improving the execution efficiency of the calculation tasks of different batches of data.

[0100] In one example, the Nth storage area can also perform the loss calculation subtask using the model calculation result related to the i+1th batch of data in the next execution time interval of the reference execution time interval to obtain the loss value of the i+1th batch of data. Therefore, based on the method provided in the embodiment of the present disclosure, the target reverse calculation result of the i+1th batch of data is calculated in a relatively timely manner after K interval execution time intervals, and the Nth storage area is released in a timely manner.

[0101] Figure 6 A block diagram of a task scheduling device according to an embodiment of the present disclosure is schematically shown.

[0102] like Figure 6 As shown, the task scheduling device 600 includes: an acquisition module 610 , a first determination module 620 , an adjustment module 630 and an execution module 640 .

[0103] An acquisition module 610 is used to acquire a scheduling strategy for model training, where the scheduling strategy is used to allocate multiple computing tasks to multiple computing units based on an interleaved pipeline mechanism, where the multiple computing tasks are sequentially allocated execution time intervals, and the computing tasks are used for at least one processing layer of the model to process batch data;

[0104] A first determining module 620 is used to determine a specified forward computing task and a target reverse computing task from at least one computing task associated with the same batch of data, the specified forward computing task and the target reverse computing task being associated with a specified processing layer of the model, and the number of intervals of the execution time interval between the target reverse computing task and the specified forward computing task meeting a preset condition;

[0105] An adjustment module 630, configured to adjust the execution time interval of the target reverse calculation task until the number of intervals after the adjustment is less than a preset number threshold, thereby obtaining a target scheduling strategy; and

[0106] The execution module 640 is used to schedule the computing unit to execute the computing task based on the target scheduling policy.

[0107] According to an embodiment of the present disclosure, the ratio between the batch number of batch data and the number of computing units is a non-integer.

[0108] According to an embodiment of the present disclosure, N computing units have a unit sequence attribute, the unit sequence attribute is related to the execution order of the computing tasks of the computing units, N>1, the specified forward computing task includes the Nth forward computing subtask and the loss computing task, the Nth forward computing subtask represents that the Nth computing unit processes the N-1th forward intermediate result sent by the N-1th computing unit using the specified processing layer;

[0109] The loss calculation task represents that the Nth computing unit calculates the loss value of the batch data according to the model calculation results output by the specified processing layer.

[0110] According to an embodiment of the present disclosure, the target number of intervals between the forward computing task and the target reverse computing task specified in the target scheduling strategy is K, K≥0, and is less than a preset number threshold.

[0111] According to an embodiment of the present disclosure, the execution module includes: a first execution submodule, a second execution submodule, a transmission submodule and a release submodule.

[0112] The first execution submodule is used to, based on a benchmark execution time interval of a specified forward computing task, the Nth computing unit calls the model calculation result stored in the Nth storage area to execute the loss calculation subtask and obtain the loss value stored in the Nth storage area.

[0113] The second execution submodule is used to execute the target reverse calculation task based on the loss value in the reference execution time interval which is K execution time intervals away from the benchmark execution time interval, and obtain the Nth reverse intermediate result stored in the Nth storage area.

[0114] The transmission submodule is used to transmit the Nth reverse intermediate result to the N-1th computing unit.

[0115] The release submodule is used to release the Nth storage area.

[0116] According to an embodiment of the present disclosure, the release submodule includes a release submodule.

[0117] The deleting unit is used to delete the loss value and the Nth reverse intermediate result stored in the Nth storage area during the process of transmitting the Nth reverse intermediate result to the N-1th computing unit.

[0118] According to an embodiment of the present disclosure, the N-1th forward intermediate result sent by the N-1th computing unit is related to the i-th batch of data.

[0119] According to an embodiment of the present disclosure, the execution module further includes a processing submodule.

[0120] The processing submodule is used to process the N-1th forward intermediate result related to the i+1th batch of data by the Nth computing unit using the designated processing layer in the next execution time interval of the reference execution time interval to obtain the model calculation result related to the i+1th batch of data, wherein the model calculation result related to the i+1th batch of data is stored in the Nth storage area.

[0121] According to an embodiment of the present disclosure, the number of execution time intervals between the forward computing task and the target reverse computing task specified in the target scheduling strategy is 0.

[0122] According to an embodiment of the present disclosure, the adjustment module includes: a first adjustment submodule and a second adjustment submodule.

[0123] The first adjustment submodule is used to adjust the execution time interval of the target reverse computing task to obtain the Nth reverse execution time interval of the target reverse computing task, wherein the number of intervals between the Nth reverse execution time interval and the execution time interval of the specified forward computing task is less than a preset number threshold.

[0124] The second adjustment submodule is used to adjust the execution order of the target reverse computing tasks based on the Nth reverse execution time interval to obtain a target scheduling strategy.

[0125] According to an embodiment of the present disclosure, N computing units have a unit sequence attribute, the unit sequence attribute is related to the execution order of the computing tasks of the computing units, N>1, and the computing task of the Nth computing unit for the i-th batch of data includes a target reverse computing task;

[0126] According to an embodiment of the present disclosure, the second adjustment submodule includes: a first adjustment unit and a second adjustment unit.

[0127] The first adjustment unit is used to adjust the execution order of the target reverse computing task based on the Nth reverse execution time interval to obtain an intermediate scheduling strategy.

[0128] The second adjustment unit is used to adjust the execution order of the reverse computing tasks of the n-1th computing unit based on the execution order of the target reverse computing tasks in the intermediate scheduling strategy to obtain the target scheduling strategy.

[0129] According to an embodiment of the present disclosure, the second adjustment unit includes an adjustment subunit.

[0130] The adjustment subunit is used to adjust the execution order of the n-1th associated reverse computing task of the n-1th computing unit based on the execution order of the target reverse computing task in the intermediate scheduling strategy, wherein the target reverse computing task to the first associated reverse computing task represents that multiple processing layers arranged continuously in the model execute reverse computing tasks in sequence.

[0131] According to an embodiment of the present disclosure, the designated processing layer includes an attention network sublayer and a normalization sublayer.

[0132] According to an embodiment of the present disclosure, the batch data includes at least one of the following: image data, audio data, and text data.

[0133] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0134] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0135] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0136] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0137] Figure 7 A schematic block diagram of an example electronic device that can be used to implement the task scheduling method of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0138] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0139] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the task scheduling method. For example, in some embodiments, the task scheduling method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the task scheduling method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the task scheduling method in any other appropriate manner (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A task scheduling method, comprising: Obtaining a scheduling strategy for model training, the scheduling strategy being used to allocate multiple computing tasks to multiple computing units based on an interleaved pipeline mechanism, the multiple computing tasks being sequentially allocated execution time intervals, the computing tasks being used for at least one processing layer of the model to process batch data; Determine, from at least one computing task associated with the same batch of data, a specified forward computing task and a target reverse computing task, wherein the specified forward computing task and the target reverse computing task are associated with a specified processing layer of the model, and the number of intervals of execution time intervals between the target reverse computing task and the specified forward computing task meets a preset condition; Adjusting the execution time interval of the target reverse calculation task until the number of intervals after adjustment is less than a preset number threshold, thereby obtaining a target scheduling strategy; and The computing unit is scheduled to execute computing tasks based on the target scheduling strategy.

2. The method according to claim 1, wherein: The ratio between the batch number of the batch data and the number of the computing units is a non-integer.

3. The method according to claim 1, wherein: N of the computing units have a unit sequence attribute, the unit sequence attribute is related to the execution order of the computing tasks of the computing units, N>1, the designated forward computing task includes an Nth forward computing subtask and a loss computing task, the Nth forward computing subtask represents that the Nth computing unit processes the N-1th forward intermediate result sent by the N-1th computing unit using the designated processing layer; The loss calculation task represents that the Nth calculation unit calculates the loss value of the batch data according to the model calculation result output by the designated processing layer.

4. The method according to claim 3, wherein: The target number of intervals between the specified forward computing task and the target reverse computing task in the target scheduling strategy is K, K≥0, and is less than a preset number threshold; Wherein, scheduling the computing unit to perform computing tasks based on the target scheduling strategy includes: Based on the benchmark execution time interval of the specified forward computing task, the Nth computing unit calls the model computing result stored in the Nth storage area to execute the loss computing subtask, and obtains the loss value stored in the Nth storage area; In a reference execution time interval that is K execution time intervals apart from the benchmark execution time interval, the Nth calculation unit performs the target reverse calculation task based on the loss value to obtain an Nth reverse intermediate result stored in the Nth storage area; transmitting the Nth reverse intermediate result to the N-1th computing unit; and The Nth storage area is released.

5. The method according to claim 4, wherein: The releasing the Nth storage area comprises: In the process of transmitting the Nth reverse intermediate result to the N-1th calculation unit, the loss value and the Nth reverse intermediate result stored in the Nth storage area are deleted.

6. The method according to claim 4, wherein: The N-1th forward intermediate result sent by the N-1th computing unit is related to the i-th batch of data, i>1, and i is an integer; Wherein, scheduling the computing unit to perform computing tasks based on the target scheduling strategy further includes: In the next execution time interval of the reference execution time interval, the Nth computing unit processes the N-1th forward intermediate result related to the i+1th batch of data using the designated processing layer to obtain a model calculation result related to the i+1th batch of data, wherein the model calculation result related to the i+1th batch of data is stored in the Nth storage area.

7. The method according to any one of claims 4 to 6, wherein: The number of execution time intervals between the specified forward computing task and the target reverse computing task in the target scheduling strategy is 0.

8. The method according to claim 1, wherein: The step of adjusting the execution time interval of the target reverse calculation task until the number of intervals after adjustment is less than a preset number threshold, and obtaining the target scheduling strategy includes: Adjusting the execution time interval of the target reverse computing task to obtain an Nth reverse execution time interval of the target reverse computing task, wherein the number of intervals between the Nth reverse execution time interval and the execution time interval of the specified forward computing task is less than a preset number threshold; and The execution order of the target reverse computing tasks is adjusted based on the Nth reverse execution time interval to obtain the target scheduling strategy.

9. The method according to claim 8, wherein: N of the computing units have a unit sequence attribute, the unit sequence attribute is related to the execution order of the computing tasks of the computing units, N>1, and the computing task of the Nth computing unit for the i-th batch of data includes the target reverse computing task; The step of adjusting the execution order of the target reverse computing tasks based on the Nth reverse execution time interval to obtain the target scheduling strategy includes: adjusting the execution order of the target reverse computing tasks based on the Nth reverse execution time interval to obtain an intermediate scheduling strategy; and Based on the execution order of the target reverse computing tasks in the intermediate scheduling strategy, the execution order of the reverse computing tasks of the n-1th computing unit is adjusted to obtain the target scheduling strategy, N≥n>1.

10. The method according to claim 9, wherein: The adjusting the execution order of the reverse computing tasks of the n-1th computing unit based on the execution order of the target reverse computing tasks in the intermediate scheduling strategy includes: Based on the execution order of the target reverse computing task in the intermediate scheduling strategy, the execution order of the n-1th associated reverse computing task of the n-1th computing unit is adjusted, wherein the target reverse computing task to the first associated reverse computing task represents that multiple processing layers arranged consecutively in the model execute reverse computing tasks in sequence.

11. The method according to claim 1, wherein: The designated processing layer includes an attention network sublayer and a normalization sublayer.

12. The method according to claim 1, wherein: The batch data includes at least one of the following: Image data, audio data, text data.

13. A task scheduling device, comprising: an acquisition module, configured to acquire a scheduling strategy for model training, wherein the scheduling strategy is used to allocate a plurality of computing tasks to a plurality of computing units based on an interleaved pipeline mechanism, wherein the plurality of computing tasks are sequentially allocated execution time intervals, and the computing tasks are used for at least one processing layer of the model to process batch data; A first determination module is used to determine a specified forward computing task and a target reverse computing task from at least one computing task related to the same batch of data, wherein the specified forward computing task and the target reverse computing task are related to a specified processing layer of the model, and the number of intervals of the execution time interval between the target reverse computing task and the specified forward computing task meets a preset condition; An adjustment module, used for adjusting the execution time interval of the target reverse calculation task until the number of intervals after adjustment is less than a preset number threshold, thereby obtaining a target scheduling strategy; as well as An execution module is used to schedule the computing unit to execute computing tasks based on the target scheduling strategy.

14. The device according to claim 13, wherein: The ratio between the batch number of the batch data and the number of the computing units is a non-integer.

15. The device according to claim 13, wherein: N of the computing units have a unit sequence attribute, the unit sequence attribute is related to the execution order of the computing tasks of the computing units, N>1, the designated forward computing task includes an Nth forward computing subtask and a loss computing task, the Nth forward computing subtask represents that the Nth computing unit processes the N-1th forward intermediate result sent by the N-1th computing unit using the designated processing layer; The loss calculation task represents that the Nth calculation unit calculates the loss value of the batch data according to the model calculation result output by the designated processing layer.

16. The device according to claim 15, wherein: The target number of intervals between the specified forward computing task and the target reverse computing task in the target scheduling strategy is K, K≥0, and is less than a preset number threshold; Wherein, the execution module includes: A first execution submodule, configured to, based on the benchmark execution time interval of the specified forward computing task, cause the Nth computing unit to call the model computing result stored in the Nth storage area to execute the loss computing subtask, and obtain the loss value stored in the Nth storage area; A second execution submodule is configured to execute, at a reference execution time interval that is K execution time intervals apart from the benchmark execution time interval, the target reverse calculation task based on the loss value by the Nth calculation unit, to obtain an Nth reverse intermediate result stored in the Nth storage area; a transmission submodule, configured to transmit the Nth reverse intermediate result to the N-1th computing unit; and The release submodule is used to release the Nth storage area.

17. The device according to claim 16, wherein: The release submodule includes: A deleting unit is used to delete the loss value and the Nth reverse intermediate result stored in the Nth storage area during the process of transmitting the Nth reverse intermediate result to the N-1th calculation unit.

18. The device according to claim 16, wherein: The N-1th forward intermediate result sent by the N-1th computing unit is related to the i-th batch of data; Wherein, the execution module also includes: A processing submodule is used for, in the next execution time interval of the reference execution time interval, the Nth calculation unit uses the designated processing layer to process the N-1th forward intermediate result related to the i+1th batch of data to obtain the model calculation result related to the i+1th batch of data, wherein the model calculation result related to the i+1th batch of data is stored in the Nth storage area.

19. The device according to any one of claims 16 to 18, wherein: The number of execution time intervals between the specified forward computing task and the target reverse computing task in the target scheduling strategy is 0.

20. The device according to claim 13, wherein: The adjustment module comprises: a first adjustment submodule, configured to adjust the execution time interval of the target reverse computing task to obtain an Nth reverse execution time interval of the target reverse computing task, wherein the number of intervals between the Nth reverse execution time interval and the execution time interval of the specified forward computing task is less than a preset number threshold; and The second adjustment submodule is used to adjust the execution order of the target reverse computing tasks based on the Nth reverse execution time interval to obtain the target scheduling strategy.

21. The device according to claim 20, wherein: N of the computing units have a unit sequence attribute, the unit sequence attribute is related to the execution order of the computing tasks of the computing units, N>1, and the computing task of the Nth computing unit for the i-th batch of data includes the target reverse computing task; Wherein, the second adjustment submodule includes: A first adjustment unit, configured to adjust the execution order of the target reverse computing tasks based on the Nth reverse execution time interval to obtain an intermediate scheduling strategy; and The second adjustment unit is used to adjust the execution order of the reverse computing tasks of the n-1th computing unit based on the execution order of the target reverse computing tasks in the intermediate scheduling strategy to obtain the target scheduling strategy, N≥n>1.

22. The device according to claim 21, wherein The second adjustment unit includes: An adjustment subunit is used to adjust the execution order of the n-1th associated reverse computing task of the n-1th computing unit based on the execution order of the target reverse computing task in the intermediate scheduling strategy, wherein the target reverse computing task to the first associated reverse computing task represents that multiple processing layers arranged consecutively in the model execute reverse computing tasks in sequence.

23. The device according to claim 13, wherein: The designated processing layer includes an attention network sublayer and a normalization sublayer.

24. The device according to claim 13, wherein: The batch data includes at least one of the following: Image data, audio data, text data.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 12.

27. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images