Migration method and migration device

Abstract

The invention provides a migration method and a migration device. The migration method is applied to a migration source end device and a migration destination end device, and comprises the steps that the migration source end device determines the type of an artificial intelligence (AI) calculation task, the type of the AI calculation task comprises a training task and an inference task of an AI model, a migration strategy is determined according to the type of the AI calculation task, and the AI calculation task comprises a training task and an inference task; the migration strategy is a strategy that the migration source end equipment transmits data of the AI computing task to the migration destination end equipment, and different migration strategies are adopted for different types of computing tasks when the computing tasks are migrated to the migration destination end equipment according to the migration strategy; the migration destination end device can recover execution of the computing tasks according to the data of the different types of AI computing tasks after receiving the data of the different types of AI computing tasks. According to the technical scheme, different migration strategies can be adopted for different types of AI computing tasks, and therefore efficient and reliable thermal migration can be conducted on the different types of AI computing tasks.

Description

Technical Field

[0001] The present application relates to the field of cloud computing, and more specifically, to a migration method and a migration device. Background Art

[0002] Currently, artificial intelligence (AI) accelerators in public cloud servers are generally mounted in a virtual execution environment represented by a virtual machine in the form of direct virtualization to obtain good performance and isolation. When a public cloud server is under maintenance, it is necessary to hot migrate the computing tasks in the AI ​​accelerator to another public cloud server so that the computing tasks on the AI ​​accelerator are not affected.

[0003] For example, since the AI ​​accelerator has independent memory, when the memory on the AI ​​accelerator side changes, the virtual machine manager cannot directly obtain the dirty page information in the AI ​​accelerator. In addition, due to the large throughput and migration data volume of the computing tasks in the AI ​​accelerator, the host side will generate dirty pages quickly. The source virtual machine needs to keep copying dirty pages to the destination virtual machine, resulting in a large amount of dirty page transmission data, and the source virtual machine cannot be paused.

[0004] Since the model parameters of the inference task remain unchanged, the number of dirty pages generated by the inference task during the inference task migration process is small, so the pre-copy method based on a single copy can be used for hot migration. However, there are not only inference tasks in the direct AI accelerator, but also training tasks. For training tasks, their model parameters are updated in real time during the iteration process, so the dirty pages generated are fast and large in number, and the pre-copy method based on a single copy is not suitable for hot migration. Summary of the invention

[0005] The present application provides a migration method and a migration device. In the technical solution, the migration source device can identify the type of AI computing task when migrating the AI ​​computing task, determine the corresponding migration strategy according to the type of AI computing task, and migrate the computing task to the migration destination device according to the migration strategy. Therefore, different migration strategies can be adopted for different types of AI computing tasks, so that efficient and reliable hot migration of different types of AI computing tasks can be achieved.

[0006] In a first aspect, a migration method is provided, which is applied to a migration source device, and the method includes: determining a type of artificial intelligence (AI) computing task, the type including a training task and an inference task of an AI model; determining a migration strategy according to the type of the AI ​​computing task, the migration strategy being a strategy for the migration source device to transmit data of the AI ​​computing task to the migration destination device; when migrating the AI ​​computing task to the migration destination device according to the migration strategy, adopting different migration strategies for different types of AI computing tasks.

[0007] Based on the embodiments of the present application, the migration source device can determine the type of AI computing task when performing AI computing task migration, such as determining whether an AI computing task is a training task or an inference task, and determine the strategy for transmitting the data of the AI ​​computing task of this type to the migration destination device according to the type of the AI ​​computing task. Since the data characteristics of the training task and the inference task are different, one migration strategy is adopted for the training task and another migration strategy is adopted for the inference task. The migration source device adopts different migration strategies for different types of AI computing tasks, so that a migration strategy suitable for the type of AI computing task can be selected.

[0008] In this way, when the migration source device migrates AI computing tasks to the migration destination, it can adopt different migration strategies for different types of computing tasks, so that the adopted migration strategy is adapted to the corresponding AI computing tasks, thereby achieving efficient and reliable hot migration of different types of computing tasks.

[0009] In combination with the first aspect, in an implementation of the first aspect, determining the type of the AI ​​computing task includes: obtaining dirty page information of the migration source device; and determining the type of the computing task based on the dirty page information.

[0010] Exemplarily, the migration source device may include a module for acquiring dirty page information generated by the migration source device, such as a dirty page tracking module or other modules.

[0011] Based on the embodiments of the present application, during the AI ​​computing task migration process, the migration source device can obtain the dirty page information generated by the migration source device, and determine the type of AI computing task based on the dirty page information, so that the migration source device can clearly identify the migrated AI computing task as a training task or an inference task.

[0012] This technical solution enables the migration source device to determine the type of AI computing task, which is conducive to the migration source device determining the corresponding migration strategy for different types of AI computing tasks, thereby improving the reliability of the migration AI computing tasks.

[0013] In combination with the first aspect, in an implementation method of the first aspect, determining the type of the AI ​​computing task according to the dirty page information includes: when it is determined according to the dirty page information that the memory of the model parameters of the AI ​​computing task is modified, determining that the AI ​​computing task is the training task; when it is determined according to the dirty page information that the memory of the model parameters of the AI ​​computing task is not modified, determining that the AI ​​computing task is the reasoning task.

[0014] Based on the embodiments of the present application, the migration source device can determine whether the memory of the model parameters of the AI ​​computing task is modified according to the dirty page information, so as to further determine whether the AI ​​computing task is a training task or an inference task.

[0015] In other examples, the migration source device may also directly determine the type of the AI ​​computing task based on the task instruction of the AI ​​computing task. The embodiments of the present application do not limit the specific form of how to determine the type of the computing task.

[0016] In combination with the first aspect, in an implementation method of the first aspect, migrating the AI ​​computing task to the migration destination device according to the migration strategy includes: transmitting the first memory data of the training task to the migration destination device according to the migration strategy of the training task, the first memory data including the model parameters obtained from the latest iteration of the training task, the state metadata of the execution of the training task, and the first operator queue that has been executed in the latest iteration; or, transmitting the second memory data of the reasoning task to the migration destination device according to the migration strategy of the reasoning task, the second memory data including the model parameters of the reasoning task, the state metadata of the execution of the reasoning task, and the dirty page information of the reasoning task.

[0017] Exemplarily, the state metadata of the execution of the training task may include another operator queue and memory mapping information that the training task needs to continue to execute; the state metadata of the execution of the reasoning task may include the operator queue and memory mapping information that the reasoning task needs to continue to execute.

[0018] Based on the embodiments of the present application, for a training task, the migration source device can transmit the model parameters obtained from the latest iteration of the training task, the state metadata of the training task execution, and the first operator queue that has been executed in the latest iteration to the migration destination device, so that the migration destination device can restore the execution state of the training task according to the state metadata of the training task execution, and restore the state of the latest iteration of the training task according to the model parameters of the latest iteration and the first operator queue that has been executed in the latest iteration. Afterwards, the migration destination can enable the training task to continue to execute iterations according to another operator queue that needs to be executed. In this way, for a training task, the migration source device can reduce the amount of data transmitted to the migration destination, thereby improving the reliability of the training task migration.

[0019] For inference tasks, the migration source device can transmit the model parameters of the latest iteration of the inference task, the state metadata of the inference task execution, and the dirty page information of the inference task to the migration destination device, so that the migration destination device can restore the execution state of the inference task according to the state metadata of the inference task execution, and continue to execute according to the model parameters and the operator queue that needs to continue to execute the inference task.

[0020] In this way, different migration strategies can be adopted for training tasks and reasoning tasks respectively, so that various types of computing tasks can be efficiently migrated.

[0021] In combination with the first aspect, in an implementation method of the first aspect, the first operator queue in the first memory data of the training task transmitted to the migration destination device includes: recording the start time T of the current iteration; transmitting the current model parameters of the training task to the migration destination device; caching the operators that have been completed in this iteration and the completion time of the operators in the first operator queue; clearing the operators in the operator queue before the start time T; and transmitting the first operator queue to the migration destination device.

[0022] Based on the embodiment of the present application, the migration source device can record the time T when determining the start of an iteration, transmit the current model parameters of the training task to the migration destination device, cache the operators that have been completed in this iteration and the time when the operators are completed in the first operator queue, and clear the invalid operators in the first operator queue between the time T. Since the model parameters of the training task are updated after each iteration, it is only necessary to record the operators that have been executed between two iterations, and the state of the latest iteration of the training task can be restored at the destination through the model parameters and the operators that have been executed.

[0023] In combination with the first aspect, in an implementation of the first aspect, the method further includes: determining the dirty page information of the training task; if it is determined through the dirty page information of the training task that the memory of the model parameters of the training task is completely modified, then it is determined that one iteration of the training task is completed and the next iteration begins.

[0024] Based on the embodiment of the present application, the migration source device can determine whether an iteration of the training task is completed through dirty page information. When it is determined through dirty page information that the memory of the model parameters of the training task is completely modified, it can be determined that an iteration is completed.

[0025] In other examples, the migration source device may also determine whether an iteration of the training task is completed or whether the next iteration is started through instruction information. For example, the migration source device may determine the start of an iteration through iteration start instruction information.

[0026] In a second aspect, a migration method is provided, which can be applied to a migration destination device, and the method includes: receiving data of different types of artificial intelligence (AI) computing tasks sent by a migration source device, wherein the types of AI computing tasks include training tasks and reasoning tasks of an AI model; determining a recovery strategy based on the data of the different types of AI computing tasks, wherein the recovery strategy is a strategy for the migration destination device to restore the execution of the AI ​​computing tasks; when restoring the execution of the AI ​​computing tasks according to the recovery strategy, different recovery strategies are adopted for the different types of AI computing tasks.

[0027] Based on the embodiments of the present application, the migration destination device can receive data of different types of AI computing tasks sent by the migration source device, and determine a recovery strategy for restoring the execution of the AI ​​computing tasks based on the data of different types of AI computing tasks, so that the migration destination device can adopt different recovery strategies for different types of AI computing tasks, so that different recovery strategies are suitable for different AI computing tasks.

[0028] In this way, when restoring the execution of different types of AI computing tasks on the migration destination device, different recovery strategies can be used to efficiently restore the execution of multiple types of AI computing tasks.

[0029] In combination with the second aspect, in an implementation method of the second aspect, if the AI ​​computing task is the training task, wherein the receiving data of different types of AI computing tasks sent by the migration source device includes: receiving the first memory data of the training task sent by the migration source device, the first memory data including the model parameters obtained by the latest iteration of the training task, the status metadata of the execution of the training task, and the first operator queue that has been executed in the latest iteration.

[0030] Based on the embodiments of the present application, when the AI ​​computing task is a training task, the migration destination device can receive the model parameters obtained from the latest iteration of the training task, the status metadata of the training task execution, and the first operator queue that has been executed in the latest iteration, so that the migration destination device can restore the effective execution of the training task based on the above data.

[0031] Optionally, if the AI ​​computing task is the reasoning task, wherein the receiving data of different types of AI computing tasks sent by the migration source device may also include: receiving second memory data of the reasoning task sent by the migration source device, the second memory data including model parameters of the reasoning task, state metadata of the reasoning task execution and dirty page information of the reasoning task.

[0032] In this way, the migration destination device can effectively execute the reasoning task based on the above data recovery.

[0033] In combination with the second aspect, in an implementation method of the second aspect, the restoring the execution of the AI ​​computing task according to the recovery strategy includes: restoring the execution status of the training task according to the state metadata of the execution of the training task; restoring the state of the latest iteration of the training task according to the model parameters of the latest iteration of the training task and the first operator queue; and continuing to execute the training task according to the second operator queue in the state metadata.

[0034] Based on the embodiment of the present application, the migration destination can restore the execution state of the training task according to the state metadata in the first memory data, and perform task replay according to the model parameters of the latest iteration of the training task and the first operator queue to restore the latest iteration result of the training task, and continue to execute the training task according to the second operator queue in the state metadata. In this way, the training task can transmit less data when migrating, and the migration destination can also resume the continued execution of the training task, thereby improving the migration speed and efficiency of the training task.

[0035] In a third aspect, a migration device is provided, comprising: a task context management module, used to determine the type of artificial intelligence (AI) computing task according to the instructions of the migration management backend, the type including the training task and the reasoning task of the AI ​​model; the migration management backend, used to determine the migration strategy according to the type of the AI ​​computing task, the migration strategy being the strategy for the migration device to transmit the data of the AI ​​computing task to another migration device; the migration management backend, also used to adopt different migration strategies for different types of computing tasks when migrating the computing task to the other migration device according to the migration strategy.

[0036] In combination with the third aspect, in an implementation method of the third aspect, the migration device also includes: a dirty page tracking module, used to determine the dirty page information generated by the migration device according to the instructions of the migration management backend; the task context management module is specifically used to: obtain the dirty page information from the dirty page tracking module; determine the type of the AI ​​computing task based on the dirty page information.

[0037] In combination with the third aspect, in an implementation method of the third aspect, the task context management module is specifically used to: when it is determined according to the dirty page information that the memory of the model parameters of the AI ​​computing task is modified, determine that the AI ​​computing task is a training task; when it is determined according to the dirty page information that the memory of the model parameters of the AI ​​computing task is not modified, determine that the AI ​​computing task is an inference task.

[0038] In combination with the third aspect, in an implementation method of the third aspect, the migration management backend is specifically used to: transmit the first memory data of the training task to the other migration device according to the migration strategy of the training task, the first memory data including the model parameters obtained by the latest iteration of the training task, the state metadata of the execution of the training task, and the first operator queue that has been executed in the latest iteration; or, transmit the second memory data of the reasoning task to another migration device according to the migration strategy of the training task, the second memory data including the model parameters of the reasoning task, the state metadata of the execution of the reasoning task, and the dirty page information of the reasoning task.

[0039] In combination with the third aspect, in an implementation method of the third aspect, the migration management backend is specifically used to: record the start time T of the current iteration; transmit the current model parameters of the training task to another migration device; cache the operators that have been completed in this iteration and the completion time of the operators in the first operator queue; clear the operators in the operator queue before the start time T; and transmit the first operator queue to the other migration device.

[0040] In combination with the third aspect, in an implementation method of the third aspect, the migration management backend is also used to: determine the dirty page information of the training task; if it is determined through the dirty page information of the training task that the memory of the model parameters of the training task is completely modified, then it is determined that one iteration of the training task is completed and the next iteration begins.

[0041] In a fourth aspect, a migration device is provided, comprising: a migration management backend, used to control the migration device to receive data of different types of artificial intelligence (AI) computing tasks sent by another migration device; the migration management backend is also used to determine a recovery strategy based on the data of the different types of AI computing tasks, wherein the recovery strategy is a strategy for the migration destination device to restore the execution of the AI ​​computing task; the migration management backend is also used to adopt different recovery strategies for the different types of AI computing tasks when restoring the execution of the AI ​​computing task according to the recovery strategy.

[0042] In combination with the fourth aspect, in an implementation method of the fourth aspect, if the AI ​​computing task is the training task, the migration management backend is specifically used to: control the migration device to receive the first memory data of the training task sent by the other migration device, the first memory data including the model parameters of the latest iteration of the training task, the status metadata of the execution of the training task, and the first operator queue that has been executed in the latest iteration.

[0043] In combination with the fourth aspect, in an implementation method of the fourth aspect, the migration management backend is specifically used to: restore the execution status of the training task based on the state metadata of the training task execution; restore the state of the latest iteration of the training task based on the model parameters of the latest iteration of the training task and the first operator queue; continue to execute the training task according to the second operator queue in the state metadata.

[0044] In the fifth aspect, a migration device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more programs, and when the one or more programs are executed by one or more processors, the migration method described in the first aspect and any possible implementation thereof is executed.

[0045] In the sixth aspect, a migration device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more programs, and when the one or more programs are executed by one or more processors, the migration method described in the second aspect and any possible implementation thereof is executed.

[0046] In the seventh aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the migration method described in the first aspect and any possible implementation thereof is executed.

[0047] In an eighth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the migration method described in the second aspect and any possible implementation thereof is executed.

[0048] In a ninth aspect, a readable storage medium is provided, wherein instructions are stored in the readable storage medium. When the instructions are executed on a device, the migration method as described in the first aspect and any possible implementation thereof is executed.

[0049] In a tenth aspect, a readable storage medium is provided, wherein instructions are stored in the readable storage medium. When the instructions are executed on a device, the migration method described in the second aspect and any possible implementation thereof is executed.

[0050] In an eleventh aspect, a program product is provided, the program product comprising a program code, and when the program code is run on a device, the migration method as described in the first aspect and any possible implementation thereof is executed.

[0051] In a twelfth aspect, a program product is provided, the program product comprising a program code, and when the program code runs on a device, the migration method as described in the second aspect and any possible implementation thereof is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the direct AI accelerator.

[0053] Figure 2 This is an architecture diagram of a server provided in an embodiment of the present application.

[0054] Figure 3 This is an architecture diagram of the device provided in an embodiment of the present application.

[0055] Figure 4 It is a schematic flow chart of a migration method provided in an embodiment of the present application.

[0056] Figure 5 It is a schematic flow chart of another migration method provided in an embodiment of the present application.

[0057] Figure 6 This is a flowchart of establishing a task information table entry for a new task provided in an embodiment of the present application.

[0058] Figure 7 It is a schematic flow chart of a migration method provided in an embodiment of the present application.

[0059] Figure 8It is a schematic flowchart of the migration preparation stage of a migration source end provided in an embodiment of the present application.

[0060] Fig. 9 It is a schematic flowchart of the migration preparation stage of a migration destination provided in an embodiment of the present application.

[0061] Fig.10 It is a schematic flowchart of the memory push phase of a migration source end provided in an embodiment of the present application.

[0062] Fig.11 It is a schematic flowchart of the memory push phase of a training task provided in an embodiment of the present application.

[0063] Fig.12 This is a schematic flowchart of a migration source end during shutdown copying provided by an embodiment of the present application.

[0064] Fig.13 It is a schematic flowchart of the execution of a migration destination recovery task provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0066] Before introducing the technical solution of the present application, some professional terms that may be involved in the present application are briefly introduced as follows.

[0067] AI computing tasks: computing tasks running in the AI ​​accelerator. For example, AI computing tasks can include training tasks and inference tasks.

[0068] Reasoning task: The process of inputting data into a trained AI model and calculating the final result according to the calculations defined by the AI ​​model structure. For example, it can correspond to the reasoning task in the embodiments of this application.

[0069] Training task: The process of optimizing AI model parameters by using input data through various optimization solution methods. Usually this process is a periodic iterative calculation process. For example, it can correspond to the training task in the embodiment of this application.

[0070] AI model: Currently, general artificial intelligence technology uses deep learning neural networks to accomplish AI tasks. AI model refers to the combination of a neural network structure and corresponding model parameters.

[0071] Model parameters: Configuration variables inside a neural network model.

[0072] Operator: A neural network model is composed of computing units, which are called operators. For example, a convolution layer is an operator; the weight summation process in a fully-connected layer can also be an operator.

[0073] Dirty page information: information about memory pages that have been modified within a certain period of time.

[0074] Dirty page scanning: A method of recording memory modification status. In computer systems, memory is usually organized and managed in "pages" of fixed size. Dirty page scanning is the process of recording memory pages that have been modified within a certain period of time.

[0075] System virtualization: A technology that achieves resource isolation and reuse by managing real physical hardware and presenting virtual resources to upper-level software. System virtualization generally includes central processing unit (CPU) virtualization, memory virtualization, and device virtualization.

[0076] Device pass-through virtualization: A type of device virtualization technology that improves virtualization performance by allowing certain operations on virtual devices to bypass the system virtualization layer and act directly on real hardware.

[0077] Virtual machine: A virtual execution environment built based on system virtualization technology. Applications in the virtual machine can hardly perceive the difference between the virtual machine and the real machine.

[0078] Virtual machine hot migration: A technology that migrates a running virtual machine from one physical execution platform to another physical platform. During the migration process, the application in the virtual machine is not explicitly paused and the service is not interrupted. In this application, virtual machine hot migration can be understood as migrating a running virtual machine from one cloud server to another cloud server.

[0079] AI accelerator: Specialized hardware used to accelerate the calculation of AI tasks, such as graphics processing unit (GPU), neural-network processing unit (NPU), etc.

[0080] Currently, AI accelerators in public cloud servers are generally mounted in a virtual execution environment represented by a virtual machine in the form of direct virtualization to obtain good performance and isolation. When a public cloud server is under maintenance, it is necessary to hot migrate the computing tasks in the AI ​​accelerator to another public cloud server so that the computing tasks on the AI ​​accelerator are not affected.

[0081] Current virtual machine hot migration technologies mainly include pre-copy, post-copy, and hybrid-copy. The virtual machine hot migration in public cloud servers mainly adopts the pre-copy method, which is briefly introduced as follows.

[0082] Pre-copy: After the migration process starts, the source virtual machine first copies the virtual machine's memory data to the destination virtual machine, then periodically scans dirty pages and copies the scanned dirty pages to the destination virtual machine; when the dirty page generation rate is lower than the threshold or the number of remaining dirty pages is small, the execution of the source virtual machine is paused, and the remaining dirty pages are copied to the destination virtual machine at one time; then, the destination virtual machine is started.

[0083] Figure 1 This is a schematic diagram of the direct AI accelerator.

[0084] See also Figure 1 , Task 1 and Task 2 in the virtual machine can act directly on the AI ​​accelerator without going through the virtual machine manager, so that the virtual machine manager cannot obtain the status of the AI ​​accelerator executing tasks in real time.

[0085] For example, since the AI ​​accelerator has independent memory, when the memory on the AI ​​accelerator side changes, the virtual machine manager cannot directly obtain the dirty page information in the AI ​​accelerator. In addition, due to the large throughput and migration data volume of the computing tasks in the AI ​​accelerator, the host side will generate dirty pages quickly. The source virtual machine needs to keep copying dirty pages to the destination virtual machine, resulting in a large amount of dirty page transmission data, and the source virtual machine cannot be paused.

[0086] In this case, the probability of migration failure will increase when migrating computing tasks in the AI ​​accelerator.

[0087] The above computing tasks can mainly include reasoning tasks and training tasks. The characteristics of memory usage for reasoning tasks and training tasks can be summarized as follows.

[0088] Reasoning tasks: Memory usage is good locally. On the one hand, the reasoning task uses the determined AI model parameters for calculation. The data of the AI ​​model parameters does not need to be modified during the reasoning calculation process. The data of the AI ​​model parameters can be considered as static data and does not generate dirty pages. On the other hand, although there is a dependency between layers during the AI ​​model reasoning calculation process, the data that is not dependent can be released immediately, so the memory storing the intermediate results can be reused.

[0089] Training tasks: Memory usage changes periodically, and dirty pages are generated at a high rate. The goal of the training task is to continuously optimize the model parameters, so the model parameters are updated at the end of each iterative calculation. In addition, each iteration of the training task includes two stages: forward calculation and reverse calculation. The reverse calculation depends on the intermediate results of the forward calculation. Therefore, unlike the inference task, more intermediate data can be released to save memory usage. Usually, in each training iterative calculation cycle, the memory usage will increase first and then decrease.

[0090] For inference tasks, you can use a single-copy-based pre-copy method to perform hot migration, which is briefly introduced below.

[0091] After the migration starts, the memory of the source virtual machine is copied to the destination virtual machine. Dirty pages are not copied at this stage. After pausing the execution of the inference task in the source virtual machine, a dirty page scan is performed and the obtained dirty pages are copied to the destination virtual machine at one time. After that, the destination virtual machine is started and the execution of the inference task is resumed in the destination virtual machine.

[0092] Since the model parameters of the inference task remain unchanged, the number of dirty pages generated by the inference task during the task migration process is small, so the above-mentioned pre-copy method based on a single copy can be used for hot migration. However, there are not only inference tasks in the direct AI accelerator, but also training tasks. For training tasks, their model parameters are updated in real time during the iteration process, so the dirty pages generated are fast and large in number, and hot migration based on a single copy pre-copy method is not suitable.

[0093] In view of this, an embodiment of the present application provides a migration method and a migration device. The technical solution can identify the type of computing tasks and adopt different migration strategies for different types of computing tasks, thereby enabling efficient and reliable hot migration of different types of computing tasks in the accelerator.

[0094] The following will combine Figure 2-Figure 13 Introduce the technical solution of this application.

[0095] For example, Figure 2 This is an architecture diagram of the device provided in the embodiment of the present application. Figure 2 As shown, the device 200 may include at least a virtual machine management module 210 , a virtual machine 220 , and an AI accelerator 230 .

[0096] Among them, the device 200 can be cloud server A.

[0097] In the embodiment of the present application, the virtual machine management module 210 can simulate some hardware devices for access by the virtual machine 220. When the virtual machine 220 accesses the hardware devices simulated by the virtual machine management module 210, the virtual machine management module 210 actually accesses the real hardware devices. However, in some cases, for example, for some computing tasks, the virtual machine 220 may interact directly with the AI ​​accelerator 230 without going through the virtual machine management module 210. The technical solution in the embodiment of the present application can be applied to such a scenario of accelerator direct access.

[0098] The virtual machine 220 may include at least an accelerator driver 221, and the accelerator driver 221 may be used to drive the AI ​​accelerator 230 to run. Exemplarily, the accelerator driver 221 may be an NPU driver.

[0099] The virtual machine management module 210 can be used to manage the virtual machine 220. The virtual machine management module 210 can include at least a hot migration manager 211 and a migration management front end 212. The hot migration manager 211 can be used to initiate the migration of computing tasks. The migration management front end 212 can be used to coordinate with the hot migration manager 211 to control the migration of computing tasks accordingly.

[0100] For example, the virtual machine management module 210 may be a quick emulator (QEMU).

[0101] The AI ​​accelerator 230 may include at least an execution state management system 240, a memory access management 250, a network card 260, an accelerator memory 270, and an accelerator computing resource 280. Exemplarily, the AI ​​accelerator 230 may be an NPU accelerator, a GPU accelerator, etc. in the form of a system on chip (SOC).

[0102] The execution state management system 240 may include at least a dirty page tracking module 241, a task context management module 242, an instruction parsing module 243, a task scheduling module 244 and a migration management backend 245. Exemplarily, the execution state management system 240 may be a general computing CPU of an NPU.

[0103] The dirty page tracking module 241 can be used to record and analyze dirty page information. For example, the dirty page information may include dirty pages in the accelerator memory 270 in the AI ​​accelerator 230, which can be referred to as AI accelerator dirty page information; the dirty page information may also include dirty pages generated by the AI ​​accelerator 230 accessing the device 200 through direct memory access (DMA), which can be referred to as DMA dirty page information.

[0104] The task context management module 242 may be used to determine the task type, task memory range, manage task execution status, and the like.

[0105] For example, the task type may include an inference task and a training task. The task memory range may include the memory range of the model parameters of the task and the memory range of the operator. At the migration source, the task execution status may include the suspension of the task, and at the migration destination, the task execution status may include the resumption of the task.

[0106] It should be understood that the migration source end of the present application can be understood as a migration source end device, and the migration destination end can be understood as a migration destination end device.

[0107] The instruction parsing module 243 may be configured to obtain instructions related to a task from the virtual machine 220 and parse the instructions related to the task.

[0108] The task scheduling module 244 may be used to schedule tasks.

[0109] The migration management backend 245 can receive relevant instructions from the migration management frontend 222 to promote the migration process of the task, and the migration management backend 245 has different functions at the migration source end and the migration destination end.

[0110] In the embodiment of the present application, the computing tasks in the AI ​​accelerator 230 can be migrated from the device 200 to the AI ​​accelerator of another device, and the other device can also include a virtual machine management module, a virtual machine and an AI accelerator. In this case, the device 200 can be called the migration source end, and the other device can be called the migration destination end.

[0111] For example, for the migration source, in the migration preparation stage, the migration management backend 245 can control the dirty page tracking module 241 to scan dirty pages, and control the task context management module 242 to identify the task type and determine the task memory range. In addition, the migration management backend 245 can also be used to establish a communication connection with the migration management backend of the migration destination; in the memory push stage, the migration management backend 245 can transfer the memory to the migration destination, and send the dirty page information corresponding to the DMA to the migration management front end 222; in the shutdown copy stage, the remaining dirty pages are transferred to the migration destination, and after the transfer is completed, the migration management front end 222 is notified that the migration is complete.

[0112] For the migration destination, the migration management backend 245 can control the task context management module 242 to initialize and establish a communication connection with the migration management backend of the migration source; in the memory push phase and the shutdown copy phase, the migration management backend 245 is used to receive data sent by the migration management backend of the migration source; in the recovery phase, the migration management backend 245 is used to control the operation of the recovery task.

[0113] In some examples, the memory access management 250 can be used to assist the dirty page tracking module 241 in performing dirty page scanning.

[0114] For example, the memory access management 250 has an auxiliary dirty marking capability. When a memory change occurs in the accelerator memory 270, the memory access management 250 can mark the part of memory. The marked part of memory can be understood as a dirty page.

[0115] In other examples, the dirty page tracking module 241 may also record dirty page information in the AI ​​accelerator 230 by using a dirty page marking method of a page fault exception.

[0116] For example, when the AI ​​accelerator 230 writes data to the accelerator memory 270, it needs to access a virtual address. Under normal circumstances, each virtual address can correspond to a physical address. When a dirty page is generated, the virtual address accessed by the AI ​​accelerator 230 may not have a corresponding physical address. At this time, the memory access management 250 can report the page fault exception event to the software layer, and the software layer allocates a new physical address for it. The software layer can record the virtual address as dirty page information.

[0117] The network card 260 can be used to implement data transmission between the migration source and the migration destination. For example, the network card 260 can be a remote direct memory access (RDMA) network card or a non-RDMA network card, and the specific form of the network card 260 is not limited in the embodiment of the present application.

[0118] The accelerator memory 270 is used to store content related to the accelerator. For example, the accelerator memory 270 can be the video memory of the GPU, or the high bandwidth memory (HBM) or double data rate synchronous dynamic random access memory (DDR) in the NPU. Dirty pages may also be generated in the accelerator memory 270.

[0119] The accelerator computing resources 280 may provide resources for the AI ​​accelerator 230 to perform task computing.

[0120] It is understandable that the dirty page tracking module 241, the task context management module 242 and the migration management backend 245 can be written into the programmable firmware in the form of firmware programs, or can be hardware modules with corresponding functions, which is not limited in the embodiments of the present application.

[0121] Figure 3 This is an architecture diagram of the device provided in an embodiment of the present application.

[0122] See also Figure 3 , device 200 and Figure 2 Compared with , the task context management module 242 can further mainly include an instruction information analyzer 2421, a task information table 2422, an execution controller 2423 and an operator buffer 2424.

[0123] The dirty page tracking module 241 may further mainly include a dirty page analyzer 2411 , a device dirty page recorder 2412 and a DMA dirty page recorder 2413 .

[0124] The migration management front end 212 may further mainly include a front end controller 2121 and a DMA dirty page management module 2122 .

[0125] The migration management backend 245 can further mainly include a backend controller 2451, an inference task migrator 2452 and a training task migrator 2453.

[0126] The instruction information analyzer 2421 can be used to obtain the memory range of the model parameters of the task, the operators included in the task and the memory range of the operators according to the information of the instruction parsing module 243. In other examples, the instruction information analyzer 2421 can also determine the type of the task according to the information of the instruction parsing module 243.

[0127] For example, a task A may include multiple operators and model parameters B. The memory storing the operators can be understood as the memory range of the operators, and the memory storing the model parameters B can be understood as the memory range of the model parameters.

[0128] The instruction information analyzer 2421 can also be used to receive dirty page information sent by the dirty page tracking module 241, and determine whether the model parameters of the task are modified based on the dirty page information. If it is determined that the model parameters of the task are modified, it can be determined that the task is a training task. If it is determined that the model parameters of the task are not modified, it can be determined that the task is an inference task.

[0129] The instruction information analyzer 2421 may also store the task type corresponding to the task, the memory range of the model parameters, and the memory range of the task in the task information table 2422 .

[0130] At the migration source end, the execution controller 2423 can be used to receive instructions from the migration management backend 245 to save the state metadata of the execution of the reasoning task and cache the operators that have been executed during each iteration of the training task.

[0131] For example, the state metadata of the reasoning task execution may mainly be the reasoning task execution context information, which may mainly include information such as the operator queue, the mapping table of the virtual address and the physical address corresponding to the operator's memory, and the like.

[0132] The execution controller 2423 may cache the operators that have been executed during each iteration of the training task in the operator cache 2424 .

[0133] The operator buffer 2424 can cache the operators that have been executed in each iteration of the training task according to the control of the execution controller 2423. For example, at the beginning of each iteration, the operator buffer 2424 starts to cache operators, and after each iteration, the cached operators are cleared, so that only operators of one iteration are stored in the operator buffer. In this way, during the shutdown migration stage, the migration destination can resume the execution of the training task according to the operators cached in the operator buffer 2424.

[0134] At the migration destination, for inference tasks, the execution controller 2423 can be used to restore task execution according to the state metadata of task execution. For training tasks, the execution controller 2423 can be used to restore the execution of training tasks according to the operators cached in the operator cache 2424.

[0135] In one embodiment, the device dirty page recorder 2412 can be used to actively access the accelerator memory 270 to determine the dirty page information. For example, the memory access management 250 can be used to mark the generated dirty pages and store the marking information in the accelerator memory 270. The device dirty page recorder 2412 can periodically access the marking information in the accelerator memory 270 to determine the dirty page information.

[0136] In another way, the device dirty page recorder 2412 may access the accelerator memory 270 to determine the dirty page information after receiving the message from the memory access management 250. For example, the memory access management 250 may be used to mark the generated dirty pages and store the marking information in the accelerator memory 270. The device dirty page recorder 2412 may access the marking information in the accelerator memory 270 to determine the dirty page information after receiving the message sent by the memory access management 250.

[0137] The device dirty page recorder 2412 may send the acquired dirty page information to the dirty page analyzer 2411. For example, the device dirty page recorder 2412 may periodically send the dirty page information to the dirty page analyzer 2411, so that the dirty page analyzer 2411 may determine whether the task type needs to be changed according to the dirty page information.

[0138] The DMA dirty page recorder 2413 can be used to obtain dirty page information generated by the AI ​​accelerator 230 accessing the host-side memory of the device 200 through DMA. For example,

[0139] The DMA dirty page recorder 2413 may also be used to send the obtained dirty page information to the dirty page analyzer 2411. For example, the DMA dirty page recorder 2413 may periodically send the dirty page information to the dirty page analyzer 2411.

[0140] In other examples, when the device 200 has the ability to mark dirty pages of memory on the host side of the device 200, the DMA dirty page recorder 2413 may not function.

[0141] The dirty page analyzer 2411 may receive instructions from the migration management backend 245 and record dirty page information according to tasks after the migration starts.

[0142] The dirty page analyzer 2411 can also be used to obtain the memory range of the model parameters of the task and the memory range of the task from the task context management module 242; when it is determined that the memory of the model parameters of the task is modified, it can be determined that the task is a training task, and the dirty page analyzer 2411 can change the task type of the task to a training task and send information to the task context management module 242.

[0143] For example, the dirty page analyzer 2411 can obtain the model parameter memory range of the task and the memory range of the task from the task information table 2422 in the task context management module 242, and determine whether the memory of the model parameters of the task is modified based on the obtained dirty page information. When it is determined that the memory of the model parameters of the task is modified, it can be determined that the task is a training task. The dirty page analyzer 2411 can change the task type of the task to a training task and send the information to the task context management module 242. The task context management module 242 can update the type of the task in the task information table.

[0144] The dirty page analyzer 2411 can also be used to determine whether one iteration of the training task is completed. For example, the dirty page analyzer 2411 can determine that the model parameter memory is completely updated based on the model parameter memory range and dirty page information of the task obtained from the task context management module 242, and then it can be determined that one iteration of the training task is completed. At this time, the dirty page analyzer 2411 can also send a message that one iteration of the training task is completed to the migration management backend 245, so as to facilitate the migration management backend 245 to determine whether the current iteration is completed.

[0145] The front-end controller 2121 can identify whether it is a migration source or a migration destination according to the configuration information of the virtual machine management module 210 .

[0146] In the embodiment of the present application, the hot migration of computing tasks in the accelerator can be divided into a migration preparation phase, a memory push phase, a shutdown copy phase, and a recovery task execution phase. The following will introduce in detail the functions and effects of each functional module in the device 200 in each of the above phases.

[0147] When the front-end controller 2121 identifies itself as the migration source, in the migration preparation stage, the front-end controller 2121 can establish a communication connection with the front-end controller of the destination end, for example, the communication connection can be a socket connection. The front-end controller 2121 can also be used to notify the migration management back-end 245 to start migration preparation, and to record the handle information used by the virtual machine 220.

[0148] It should be understood that the virtual machine 220 can run in the device 200, and the accelerator used is the AI ​​accelerator 230. The handle information used by the virtual machine 220 is the relevant information of the AI ​​accelerator 230 used by the virtual machine 220, such as the configuration information and status information of the accelerator.

[0149] Exemplarily, the manner in which the front-end controller 2121 notifies the migration management backend 245 to start migration preparation can be briefly described as follows:

[0150] The front-end controller 2121 may store the instruction for starting the migration preparation in a memory, and the migration management back-end may periodically access the memory to obtain the instruction for starting the migration preparation.

[0151] Similarly, after the migration preparation is completed, the migration management backend 245 can also store the message of the completion of the migration preparation in a memory, and the front-end controller 2121 can determine whether the migration management backend 245 has completed the migration preparation by accessing the memory. Alternatively, the migration management backend 245 can also notify the front-end controller 2121 that the migration preparation has been completed by interruption. When the migration management backend 245 determines that the migration preparation is completed, it can send the message to the hot migration manager 211.

[0152] In the memory push phase, the front-end controller 2121 notifies the migration management back-end 245 to start memory push and sends handle information to the migration controller at the destination. Similarly, after the memory push is completed, the migration management back-end 245 notifies the front-end controller 2121 that the memory push has been completed. After confirming that the memory push is completed, the front-end controller 2121 sends the message to the hot migration manager 211.

[0153] It should be understood that the way in which the front-end controller 2121 notifies the migration management backend 245 to start memory push can be found in the previous text, which describes the way in which the front-end controller 2121 notifies the migration management backend 245 to start migration preparation. For the sake of brevity, it will not be repeated here.

[0154] In the shutdown copy phase, the front-end controller 2121 notifies the migration management back-end 245 to start the shutdown copy.

[0155] When the front-end controller 2121 identifies itself as the migration destination, it is used to establish a communication connection with the front-end controller of the migration source in the migration preparation stage, and notify the migration management backend 245 to initialize. After the migration management backend 245 is initialized, the front-end controller 2121 can be notified that the initialization is completed, and then the front-end controller 2121 can send the message to the hot migration manager 211.

[0156] In the memory push phase, the front-end controller 2121 is used to receive data sent by the front-end controller of the migration source end. For example, the data may be handle information used by the virtual machine.

[0157] In the recovery phase, the front-end controller 2121 is used to receive instructions from the hot migration manager 211 to re-establish the handle mapping between the virtual machine and the new device; the front-end controller 2121 is also used to notify the migration management back-end 245 to resume the execution of tasks in the AI ​​accelerator 230.

[0158] In some examples, the migration management front end 212 may also include a DMA dirty page management module 2122. The DMA dirty page management module 2122 may be used to implement marking dirty page information generated when the AI ​​accelerator 230 accesses the memory on the host side of the device 200 through DMA. For example, the dirty page management module 2122 may receive dirty page information sent by the migration management back end, and send the dirty page information to the hot migration manager 211.

[0159] The back-end controller 2451 can receive instructions from the front-end controller 2121 and promote the migration process.

[0160] At the migration source, in the migration preparation stage, after obtaining the migration preparation message from the front-end controller 2121, the back-end controller 2451 can notify the task context management module 242 to identify the task type, the memory range of the task, and the memory range of the model parameters, etc., to complete the update of the task information table; the back-end controller 2451 can also notify the dirty page tracking module 241 to start dirty page scanning by task; the back-end controller 241 can also be used to establish a communication connection with the back-end controller 241 at the migration destination for subsequent memory transfer. For example, the communication connection method can be a peer-to-peer (P2P) communication connection.

[0161] During the memory push phase, the back-end controller 2451 notifies the inference task migrator 2452 to transfer the memory of the inference task to the memory of the migration destination, and notifies the training task migrator 2453 to transfer the memory of the training task to the memory of the migration destination; the back-end controller 2451 can also send the DMA dirty page information obtained from the dirty page tracking module 241 to the DMA dirty page management module 2122 in the virtual machine management module 210.

[0162] In the shutdown copy phase, the back-end controller 2451 notifies the inference task migrator 2452 and the training task migrator 2453 to start the shutdown copy respectively. After the shutdown copy is completed, the back-end controller 2451 notifies the migration management front-end 212 that the task migration in the accelerator is completed, and the migration management front-end 212 sends the task migration completion message to the hot migration manager 211, so that the hot migration manager 211 starts the recovery operation phase.

[0163] At the migration destination, in the migration preparation phase, the back-end controller 2451 is used to initialize the execution status management system 240 according to the instructions of the front-end controller 2121, for example, controlling the task context management module 242 to establish the data structure in the task information table, etc.; the back-end controller 2451 is also used to establish a communication connection with the back-end controller of the migration source end, so as to be able to receive the memory data sent by the migration source end.

[0164] In the memory push phase and the shutdown copy phase, the back-end controller 2451 is used to receive data sent by the back-end controller of the migration source end.

[0165] During the recovery phase, the back-end controller 2451 obtains the task information table from the task context management module 242 according to the instructions of the migration management front-end, and instructs the inference task migrator 2452 and the training task migrator 2453 to build a task execution environment according to the task information table, and recover the task execution respectively.

[0166] The inference task migrator 2452, the user transfers the memory data and task metadata of the inference task according to the instructions of the backend control 2451. At the migration source end, the inference task migrator 2452 can also be used to control the task context management module 242 to realize the suspension of the inference task, and at the migration destination end, the inference task migrator 2452 can also be used to resume the operation of the inference task.

[0167] The training task migrator 2453, the user transfers the memory data and task metadata of the training task according to the instructions of the backend control 2451. At the migration source end, the training task migrator 2452 can also be used to control the task context management module 242 to realize the suspension of the training task, and at the migration destination end, the training task migrator 2452 can also be used to resume the operation of the training task.

[0168] It should be understood that in some examples, if there are only reasoning tasks in the AI ​​accelerator 230, only the reasoning task migrator 2452 is working; if there are only training tasks in the AI ​​accelerator 230, only the training task migrator 2453 is working; if there are both reasoning tasks and training tasks in the AI ​​accelerator 230, both the reasoning task migrator 2452 and the training task migrator 2453 are working.

[0169] It is understandable that the above-mentioned functional modules are merely illustrative. In other examples, the migration management front end 212, the dirty page tracking module 241, the task context management module 242, and the migration management back end 245 may also include more modules or fewer modules, which is not limited in the embodiments of the present application.

[0170] The following will combine Figure 4-Figure 5 The migration methods of this application are introduced respectively.

[0171] Figure 4 is a schematic flow chart of a migration method provided in an embodiment of the present application. Figure 4 As shown, the method 900 may be applied to migrate a source device, and the method 900 may include steps 910 to 930.

[0172] 910. The migration source device determines the type of AI computing task, where the type of AI computing task includes training tasks and reasoning tasks of the AI ​​model.

[0173] It can be understood that the migration source device and the migration destination device are in a communication connection state.

[0174] When migrating an AI computing task, the migration source device may first determine the type of the AI ​​computing task to determine whether an AI computing task is a training task or an inference task.

[0175] Exemplarily, the migration source device may obtain dirty page information generated by the migration source device, and determine the type of the AI ​​computing task according to the dirty page information.

[0176] In some embodiments, when the migration source device determines that the memory of the model parameters of the AI ​​computing task is modified based on the dirty page information, the AI ​​computing task can be determined to be a training task. Since the memory of an AI computing task includes the memory of its model parameters and the memory of the operator, and for a training task, its iteration process is a process of optimizing the model parameters, its model parameters will change with each iteration. Therefore, when the migration source device determines that the memory of the model parameters of an AI computing task is modified based on the dirty page information, it can be determined that its model parameters have changed, so it can be determined that the AI ​​computing task is a training task. In this way, during the AI ​​computing task migration process, the migration source device can obtain the dirty page information generated by the migration source device, and determine the type of AI computing task based on the dirty page information, so that the migration source device can clearly identify the migrated AI computing task as a training task or an inference task.

[0177] This technical solution enables the migration source device to determine the type of AI computing task, which is conducive to the migration source device determining the corresponding migration strategy for different types of AI computing tasks, thereby improving the reliability of the migration AI computing tasks.

[0178] In some embodiments, when the migration source device determines that the memory of the model parameters of the AI ​​computing task has not been modified based on the dirty page information, the AI ​​computing task can be determined to be an inference task. For inference tasks, for each iteration, its model parameters remain fixed. Therefore, when the migration source device determines that the memory of the model parameters of an AI computing task has not been modified based on the dirty page information, it can be determined that its model parameters have not changed, so it can be determined that the AI ​​computing task is an inference task. In this way, the migration source device can determine whether the memory of the model parameters of the AI ​​computing task has been modified based on the dirty page information to further determine whether the AI ​​computing task is a training task or an inference task.

[0179] In other examples, the migration source device may also determine the type of the AI ​​computing task based on the information of the AI ​​computing task. For example, the information of the AI ​​computing task may also carry the task type information of the AI ​​computing task, which is not limited in the embodiments of the present application.

[0180] 920. The migration source device determines a migration strategy according to the type of the AI ​​computing task. The migration strategy is a strategy for transmitting data of the AI ​​computing task from the migration source device to the migration destination device.

[0181] After determining the type of the AI ​​computing task, the migration source device can determine a migration strategy corresponding to the type of the AI ​​computing task according to the type of the AI ​​computing task.

[0182] For example, for a training task, one migration strategy suitable for the training task can be adopted, and for an inference task, another strategy suitable for the inference task can be adopted.

[0183] In some examples, for training tasks, since their model parameters change with each iteration, the model parameters obtained in the latest iteration of the training task and the operators executed in this iteration of the training task can be transmitted without transmitting the intermediate data of the training task. For inference tasks, since their model parameters do not change, the model parameters and dirty page information of the inference task can be transmitted.

[0184] 930, when the migration source device migrates the computing task to the migration destination device according to the migration strategy, different migration strategies are adopted for different types of computing tasks.

[0185] When performing computing task migration, the migration source device can adopt different migration strategies for different types of AI computing tasks. For example, for training tasks, the migration strategy is determined to be a migration strategy based on task replay based on the data characteristics of the training tasks. For inference tasks, the migration strategy is determined to be a migration strategy based on a single copy of the pre-copy based on the data characteristics of the inference tasks.

[0186] It is understandable that the data characteristics of different types of computing tasks are different, so a corresponding appropriate migration strategy can be adopted. The specific migration strategy adopted for the above-mentioned reasoning tasks and training tasks in the embodiments of the present application is not limited.

[0187] In other examples, the migration source device may also adopt other migration strategies to migrate reasoning tasks and training tasks.

[0188] Based on the embodiments of the present application, the migration source device can determine the type of AI computing task when performing AI computing task migration, such as determining that an AI computing task is a training task or an inference task, and determine the strategy for transmitting the data of the AI ​​computing task of this type to the migration destination device according to the type of the AI ​​computing task. Since the data characteristics of the training task and the inference task are different, one migration strategy is adopted for the training task and another migration strategy is adopted for the inference task. The migration source device adopts different migration strategies for different types of AI computing tasks, so that a migration strategy suitable for the type of AI computing task can be selected.

[0189] In this way, when the migration source device migrates AI computing tasks to the migration destination, it can adopt different migration strategies for different types of computing tasks, so that the adopted migration strategy is adapted to the corresponding AI computing tasks, thereby achieving efficient and reliable hot migration of different types of computing tasks.

[0190] In some embodiments, the migration source device migrates the AI ​​computing task to the migration destination device according to the migration strategy, including:

[0191] Transmitting first memory data of the training task to the migration destination device according to the migration strategy of the training task, the first memory data including the model parameters obtained by the latest iteration of the training task, the state metadata of the execution of the training task, and the first operator queue that has been executed in the latest iteration; or

[0192] The second memory data of the reasoning task is transmitted to the migration destination device according to the migration strategy of the reasoning task, and the second memory data includes the model parameters of the reasoning task, the state metadata of the reasoning task execution and the dirty page information of the reasoning task.

[0193] Exemplarily, the state metadata of the training task execution may mainly include the operator queues that the training task needs to continue to execute and their memory mapping relationships.

[0194] For training tasks, the migration source device can transmit the model parameters of the latest iteration of the training task, the state metadata of the training task execution, and the first operator queue that has been executed in the latest iteration to the migration destination device, so that the migration destination device can restore the execution state of the training task according to the state metadata of the training task execution, and restore the state of the latest iteration of the training task according to the model parameters of the latest iteration and the first operator queue that has been executed in the latest iteration. Afterwards, the migration destination can enable the training task to continue to execute iterations according to another operator queue that needs to be executed. In this way, for training tasks, the migration source device can reduce the amount of data transmitted to the migration destination, and improve the reliability of training task migration.

[0195] Exemplarily, the execution state metadata of the reasoning task may mainly include the operator queue that needs to continue to be executed by the training reasoning task and its memory mapping relationship.

[0196] For inference tasks, the migration source device can transmit the model parameters of the latest iteration of the inference task, the state metadata of the inference task execution, and the dirty page information of the inference task to the migration destination device, so that the migration destination device can restore the execution state of the inference task according to the state metadata of the inference task execution, and continue to execute according to the model parameters and the operator queue that needs to continue to execute the inference task.

[0197] In this way, different migration strategies can be adopted for training tasks and reasoning tasks respectively, so that various types of computing tasks can be efficiently migrated.

[0198] In some embodiments, the migration source device transmits a first operator queue in first memory data of a training task to the migration destination device, including:

[0199] Record the start time T of the current iteration;

[0200] Transmit the current model parameters of the training task to the migration destination device;

[0201] Cache the operators that have been completed in this iteration and the time when the operators were completed in the first operator queue;

[0202] Clear the operators in the operator queue before the start time T;

[0203] The first operator queue is transmitted to the migration destination device.

[0204] It is understandable that the embodiment of the present application does not limit the specific execution order of the above steps, or several steps of the above steps can also be executed simultaneously. In other examples, the migration source device can also adopt other orders.

[0205] Based on the embodiment of the present application, the migration source device can record the time T when determining the start of an iteration, transmit the current model parameters of the training task to the migration destination device, cache the operators that have been completed in this iteration and the time when the operators are completed in the first operator queue, and clear the invalid operators in the first operator queue between the time T. Since the model parameters of the training task are updated after each iteration, it is only necessary to record the operators that have been executed between two iterations, and the state of the latest iteration of the training task can be restored at the destination through the model parameters and the operators that have been executed.

[0206] In some embodiments, the method further comprises:

[0207] Determine the dirty page information of the training task;

[0208] If it is determined through the dirty page information of the training task that the memory of the model parameters of the training task is completely modified, it is determined that one iteration of the training task is completed and the next iteration begins.

[0209] Exemplarily, the migration source device may include a module for obtaining dirty page information, such as a dirty page tracking module, so that the migration source device can obtain dirty page information of the training task through the dirty page tracking module.

[0210] Based on the embodiment of the present application, the migration source device can determine whether an iteration of the training task is completed through dirty page information. When it is determined through dirty page information that the memory of the model parameters of the training task is completely modified, it can be determined that an iteration is completed.

[0211] In other examples, the migration source device may also determine whether an iteration of the training task is completed or whether the next iteration is started through instruction information. For example, the migration source device may determine the start of an iteration through iteration start instruction information.

[0212] Figure 5 FIG. 1 is a schematic flow chart of another migration method provided in an embodiment of the present application. Figure 5 As shown, the method 1000 may be applied to migrate a destination device, and the method 1000 may include steps 1010 to 1030.

[0213] 1010. The migration destination device receives data of different types of AI computing tasks sent by the migration source device. The types of AI computing tasks include training tasks and reasoning tasks of AI models.

[0214] It can be understood that the migration source device and the migration destination device are in a communication connection state, so that the migration destination device can receive data of different types of AI computing tasks sent by the migration source device.

[0215] Exemplarily, the migration destination device receives the first memory data of the training task sent by the migration source device, and the first memory data includes the model parameters obtained by the latest iteration of the training task, the state metadata of the training task execution, and the first operator queue that has been executed in the latest iteration, so that the migration destination device can resume the effective execution of the training task according to the above data.

[0216] Optionally, for the inference task, the migration destination can also receive the second memory data of the inference task sent by the migration source device, wherein the second memory data includes the model parameters of the inference task, the state metadata of the execution of the inference task, and the dirty page information of the inference task, so that the migration destination device can restore the effective execution of the inference task based on the above data.

[0217] 1020. The migration destination device determines a recovery strategy based on data of different types of AI computing tasks. The recovery strategy is a strategy for the migration destination device to resume execution of the AI ​​computing tasks.

[0218] For example, since the migration source device adopts different migration strategies for different types of AI computing tasks, the migration source destination device also needs to adopt different recovery strategies to recover the execution of different types of AI computing tasks. Therefore, the migration destination device can determine different recovery strategies for different types of AI computing tasks.

[0219] For example, for a training task, it is determined to adopt recovery strategy A, and for a training task, it is determined to adopt recovery strategy B, and recovery strategy A is different from recovery strategy B.

[0220] 1030, when the migration destination device resumes the execution of the AI ​​computing task according to the recovery strategy, different recovery strategies are adopted for different types of AI computing tasks.

[0221] For example, for a training task, the migration destination device can use a recovery strategy based on training task replay to resume the execution of the training task. The migration destination device first recovers the latest state of the training task before migration based on the model parameters obtained from the latest iteration of the received training task and the operator queue that has been executed in the latest iteration, that is, the model parameters obtained from the latest iteration are used to execute the operators in the order in the operator queue to obtain the intermediate data of the training task and recover the latest state of the training task before migration.

[0222] For the inference task, the migration destination can resume the execution of the inference task based on the received model parameters of the inference task and the operator queue that needs to continue to be executed.

[0223] It is understandable that the migration destination device can also use other recovery strategies to restore the execution of training tasks and reasoning tasks, and the present application embodiment does not limit this. For example, the migration destination device can also use the delayed copy or hybrid copy method mentioned above to restore the execution of the reasoning task.

[0224] Based on the embodiments of the present application, the migration destination device can receive data of different types of AI computing tasks sent by the migration source device, and determine a recovery strategy for restoring the execution of the AI ​​computing tasks based on the data of different types of AI computing tasks, so that the migration destination device can adopt different recovery strategies for different types of AI computing tasks, so that different recovery strategies are suitable for different AI computing tasks.

[0225] In this way, when restoring the execution of different types of AI computing tasks on the migration destination device, different recovery strategies can be used to efficiently restore the execution of multiple types of AI computing tasks.

[0226] In some embodiments, the migration destination device resumes execution of the AI ​​computing task according to the recovery strategy, including:

[0227] Restore the execution status of the training task according to the status metadata of the training task execution;

[0228] Restore the latest iteration status of the training task according to the model parameters of the latest iteration of the training task and the first operator queue;

[0229] Continue to execute the training task according to the second operator queue in the state metadata.

[0230] Based on the embodiment of the present application, the migration destination can restore the execution state of the training task according to the state metadata in the first memory data, and perform task replay according to the model parameters of the latest iteration of the training task and the first operator queue to restore the latest iteration result of the training task, and continue to execute the training task according to the second operator queue in the state metadata. In this way, the training task can transmit less data when migrating, and the migration destination can also resume the continued execution of the training task, thereby improving the migration speed and efficiency of the training task.

[0231] The following will combine Figure 6 This section describes the process of creating task information table entries for new tasks.

[0232] For example, Figure 6 This is a flowchart of establishing a task information table entry for a new task provided by an embodiment of the present application. Figure 6 As shown, the process may include steps 310 to 360, which may be performed by the accelerator, for example, by the task context management module 242 in the accelerator.

[0233] 310, the accelerator obtains instructions.

[0234] Exemplarily, the accelerator may obtain an instruction sent by the host, and parse the instruction to determine corresponding information.

[0235] 320 , the accelerator determines whether there is a new task according to the instruction.

[0236] Exemplarily, the instruction parsing module in the accelerator can parse the instruction sent by the host, and the instruction information analyzer can analyze the information parsed by the instruction parsing module to determine whether there is a new task and the memory range of the corresponding task, the memory range of the model parameters, etc. For example, the new task can be an AI computing task, such as an inference task or a training task.

[0237] When the accelerator determines according to the instruction that there is no new task, step 350 may be executed.

[0238] 330 , when it is determined that there is a new task, the accelerator creates a task information table entry for the new task.

[0239] When the accelerator determines that there is a new task A, a corresponding table entry is created in the task information table for task A. For example, each table entry in the task information table may correspond to a task.

[0240] 340, the accelerator updates the task information table according to the instruction.

[0241] The accelerator may update the task information table according to the information of the new task in the instruction, so that the task information table includes the information of the new task. The information of the new task may include the memory range of the task, the memory range of the model parameters, the task type, etc.

[0242] At 350 , the accelerator determines whether to indicate task completion according to the instruction.

[0243] The accelerator determines whether the instruction indicates that a certain task is completed according to the instruction. When the accelerator determines that the instruction indicates that a certain task is completed according to the instruction, step 360 may be executed.

[0244] 360. When the confirmation indicates that the task is completed, the accelerator deletes the task information table entry of the completed task.

[0245] When the accelerator determines that the instruction indicates that a certain task is completed, the entry of the completed task in the task information table may be deleted, so that the task information table may be updated in a timely manner.

[0246] Based on the embodiment of the present application, when there is a new task, the accelerator can create a corresponding task information table entry and update the task information table according to the information of the new task to provide information support for the task migration phase; if the instruction indicates that the task execution is completed, the corresponding task information table entry is deleted, so that the task information table can be updated in time.

[0247] In some examples, the migration method of the present application may include multiple stages. Figure 7 Introduce the technical solution.

[0248] Figure 7 is a schematic flow chart of a migration method provided in an embodiment of the present application. Figure 7 As shown, the method 800 may include steps 810 to 840 .

[0249] In an embodiment of the present application, the computing task migration in the AI ​​accelerator may include a migration preparation phase, a memory push phase, a shutdown copy phase, and a recovery task execution phase.

[0250] 810, in the migration preparation stage, the migration source end establishes a communication connection with the migration destination end, and the migration source end determines a migration strategy according to the type of computing task.

[0251] In the migration preparation phase, the migration source end can establish a communication connection with the migration destination end to enable memory data transmission in the subsequent phase.

[0252] The migration source can also determine the corresponding migration strategy according to the type of computing task. For example, when the migration source determines that the computing task is an inference task, the migration strategy of the inference task can be determined as the aforementioned pre-copy migration strategy based on a single copy. When the migration source determines that the computing task is a training task, the migration strategy of the training task can be determined as a migration strategy based on task replay, that is, the model parameters of the latest iteration and the operators that have been executed in this iteration are transmitted to the migration destination, so that the migration destination can restore the latest iteration result of the training task based on the model parameters and the operators that have been executed in this iteration, thereby eliminating the need to transmit intermediate data during the iteration process.

[0253] It is understandable that in the embodiments of the present application, different migration strategies can be used for different computing tasks. The above is explained by taking the migration strategy based on task replay for training tasks and the pre-copy migration strategy based on single copy for inference tasks as an example. In other examples, other migration strategies can also be used for inference tasks and training tasks, which are not limited by the embodiments of the present application.

[0254] 820, in the memory push phase, the migration source end transmits the memory data A of the computing task to the migration destination end according to the migration strategy.

[0255] During the memory push phase, the migration source can migrate different computing tasks to the migration destination according to different migration strategies.

[0256] Exemplarily, for the memory push phase of the inference task, the migration source can transmit the memory data A of the inference task to the migration destination, and the memory data A can mainly include the model parameters of the inference task and the state metadata of the inference task execution. The state metadata of the inference task execution can mainly include the operator queue A1 and the memory mapping relationship that the inference task needs to continue to execute in the memory of the migration source.

[0257] During the memory push phase of the training task, the migration source can transmit the memory data A of the training task to the migration destination. Since the model parameters of the training task will change after each iteration, the memory data A of the training task can mainly include the model parameters obtained from the latest iteration of the training task and the state metadata of the training task execution. The state metadata of the training task execution can mainly include the operator queue A2 and the memory mapping relationship of the training task that needs to continue to be executed in the memory of the migration source.

[0258] 830, in the shutdown copy phase, the migration source end suspends the execution of the computing task and transmits the memory data B of the computing task to the migration destination end according to the migration strategy.

[0259] During the downtime copy phase, the migration source suspends the execution of all computing tasks. For reasoning tasks, since it adopts a pre-copy solution based on a single copy, the migration source can transmit the dirty page information generated from the memory push phase to the downtime copy phase to the migration destination. The memory data B is the dirty page data corresponding to the dirty page information.

[0260] For training tasks, since they adopt a migration solution based on task replay, the migration source can transmit the operator queue composed of operators that have been executed in the latest iteration to the migration destination, and the memory data B is the data corresponding to the operators in the operator queue.

[0261] 840, in the task execution recovery phase, the migration destination recovers the execution of the computing task according to the migration strategy.

[0262] In the task execution recovery phase, the migration destination can restore the execution of the computing task according to the migration strategy. For the inference task, the migration destination can restore the continued execution of the inference task according to its model parameters and the state metadata of the inference task execution.

[0263] For example, the model parameter of the inference task is model parameter A. In the shutdown copy phase, the inference task is executed to the fifth operator. After the execution state of the inference task is restored in the recovery task execution phase, the inference task continues to execute from the fifth operator according to the operator queue.

[0264] For training tasks, the migration destination can restore the status of the latest iteration of the training task based on the model parameters of the latest iteration and the operator queue that has been executed in the latest iteration. After that, the training task can be continued according to the operator queue A2 in the status metadata of the training task execution.

[0265] For example, in the shutdown copy stage, the fourth iteration of the training task has been completed, the fifth iteration has started and 3 operators have been executed. The model parameter of the latest iteration is the model parameter B obtained in the fourth iteration, and the operator queue that has been executed in the latest iteration is the queue composed of the three operators. The migration destination can execute the training task according to the model parameter B and the operator queue composed of the three operators. When the operator queue has been executed, the training task has been restored to the state of the latest iteration. After that, the training task can be executed normally according to the operator queue A1.

[0266] In this way, different migration strategies are adopted for different computing tasks, so that different computing tasks can be migrated efficiently.

[0267] In the following, we will combine Figure 8-13 The technical solutions for each stage of the migration process are introduced in detail.

[0268] For example, Figure 8 1 is a schematic flow chart of a migration source end migration preparation phase provided by an embodiment of the present application. Figure 8 As shown, the process may include steps 401 to 412.

[0269] In this application, the migration management front end A and the migration management back end A belong to the migration source end; the migration management front end B and the migration management back end B belong to the migration destination end.

[0270] 401, the migration management front end A sends a communication connection request to the migration management front end B.

[0271] Since the migration management front end A needs to send data to the migration management front end B during the migration process, the migration management front end A needs to establish a communication connection with the migration management front end B.

[0272] Here, the migration management front end A may first send a communication connection request to the migration management front end B. After receiving the communication connection request, the migration management front end B may establish a communication connection with the migration management front end A. For example, the communication connection may be in the form of a socket connection.

[0273] 402, the migration management front end A records the handle information of the accelerator used.

[0274] Exemplarily, the handle information may include configuration information and status information of the accelerator, etc. Migration management front end A records the handle information and transmits the handle information to migration management front end B during migration, so that handle mapping can be re-established at the migration destination according to the handle information.

[0275] 403, the migration management front end A notifies the migration management back end A to prepare for migration.

[0276] Exemplarily, the migration management front end A may store the instruction for starting the migration preparation in a memory, and the migration management back end A may periodically access the memory to obtain the instruction for starting the migration preparation and start the migration preparation according to the instruction.

[0277] 404, the migration management backend A notifies the task context management module to start task type identification.

[0278] In one example, the migration management backend A may store the instruction for starting task type identification in a memory, and the task context management module may periodically access the memory to obtain the instruction for starting task type identification and start task type identification according to the instruction.

[0279] In another example, the migration management backend A may also notify the task context management module to start task type identification through the bus protocol.

[0280] 405 , the task context management module identifies the task type.

[0281] The task context management module can determine the type of task according to whether the model parameter memory of the task is modified. For example, when the task context management module determines that the model parameter memory of task A is modified, task A can be determined to be a training task, and when the task context management module determines that the model parameter memory of task A is not modified, task B can be determined to be an inference task.

[0282] It should be understood that there may be multiple tasks in the accelerator, and for each task, its type may be determined in the above manner, and then the task context management module may store the task type in the task information table.

[0283] 406, the migration management backend A obtains the task type from the task context management module.

[0284] For example, the migration management backend A may obtain a task information table from the task context management module, and the task and the corresponding task type may be determined from the task information table.

[0285] 407, the migration management backend A notifies the dirty page tracking module to record the dirty page information.

[0286] Exemplarily, the migration management front end A may store the instruction to start task type identification in a memory, and the task context management module may periodically access the memory to obtain the instruction to start recording dirty page information, and start recording dirty page information according to the instruction.

[0287] 408, the dirty page tracking module determines the dirty page information.

[0288] The dirty page tracking module can determine the dirty page information of the accelerator side and the dirty page information of the host side according to the task. In this way, for each task, the dirty page tracking module can determine its dirty page information.

[0289] For example, the dirty page tracking module may determine the dirty page information on the accelerator side through the device dirty page recorder 2412 , and may determine the dirty page information on the host side through the DMA dirty page recorder 2413 .

[0290] 409, the dirty page tracking module sends the dirty page information to the migration management backend.

[0291] After determining the dirty page information, the dirty page tracking module may send the dirty page information to the migration management backend A, so that the migration management backend A transmits the dirty page information to the migration management backend B during the migration process.

[0292] 410 , the migration management backend A sends a communication connection request to the migration management backend B.

[0293] Since migration management backend A needs to send memory data to migration management backend B during the migration process, migration management backend A needs to establish a communication connection with migration management backend B.

[0294] Here, the migration management backend A may first send a communication connection request to the migration management backend B. After receiving the communication connection request, the migration management backend B may establish a communication connection with the migration management backend A. For example, the communication connection may be a P2P connection.

[0295] 411, the migration management backend determines the migration strategy based on the task type.

[0296] The migration management backend can obtain the task type from the task context management module and determine the migration strategy based on the task type. Different migration strategies can be adopted for different task types, so that efficient migration of various task types can be achieved.

[0297] In an embodiment of the present application, a first migration method may be used for training tasks, and a second migration method may be used for reasoning tasks. The first migration method is different from the second migration method.

[0298] For example, in the first migration mode, the migration management backend A can transmit the model parameters of the latest iteration of the training task and the operator queue that has been executed in the latest iteration to the migration management backend B. In this way, the migration management backend B can restore the latest task iteration according to the model parameters and the operator queue that has been executed in the latest iteration, so that the task can continue to be executed, avoiding the transmission of a large amount of memory data during the migration process. The second migration mode can be the pre-copy mode of the single copy described above.

[0299] It is understandable that the first migration mode and the second migration mode may also be other migration modes, which are not limited in the embodiments of the present application.

[0300] 412, the migration management backend A notifies the migration management front end A that the migration preparation is complete.

[0301] In a possible implementation, the migration management backend A may store the migration preparation completion message in a memory, and the migration management frontend A may periodically access the memory to obtain the migration preparation completion message.

[0302] In another possible implementation, the migration management backend A may notify the migration management frontend A of the completion of the migration preparation by interruption.

[0303] It should be understood that the embodiment of the present application does not limit the specific execution order of the above steps 401 to 412. In other examples, some of the steps may not be executed or may be replaced by other steps.

[0304] In an embodiment of the present application, during the migration preparation stage of the migration source, the migration management front end of the source end can establish a communication connection with the migration management front end of the destination end, and the migration management back end of the source end can establish a communication connection with the migration management back end of the destination end, so as to prepare for subsequent data transmission; the migration management back end of the source end can also determine the migration strategy according to the task type, so as to realize efficient migration of various task types.

[0305] For example, Fig. 9 1 is a schematic flow chart of a migration preparation phase of a migration destination provided by an embodiment of the present application. Fig. 9 As shown, the process may include steps 451 to 455.

[0306] 451, the migration management front end B establishes a communication connection with the migration management front end A.

[0307] Exemplarily, after receiving the communication connection request message from the migration management front end A, the migration management front end B establishes a communication connection with the migration management front end A.

[0308] 452, the migration management front end B notifies the migration management back end B to initialize the execution status management system.

[0309] It should be understood that the manner in which the migration management front end B notifies the migration management B can refer to the relevant description in the foregoing text.

[0310] 453. Migration management backend B establishes a communication connection with migration management backend A.

[0311] Exemplarily, after receiving the communication connection request message from the migration management backend A, the migration management backend B establishes a communication connection with the migration management frontend A.

[0312] 454. The migration management backend B initializes the execution status management system.

[0313] Exemplarily, the migration management backend B may control each module in the execution state management system to initialize, such as creating corresponding data structures, so that the migration management backend is ready to receive data sent by the migration management backend A.

[0314] 455. Migration management backend B notifies migration management frontend B that migration preparation is complete.

[0315] It should be understood that the step 455 can refer to the relevant description of step 412, and for the sake of brevity, it will not be repeated here.

[0316] Based on the embodiment of the present application, at the migration destination, a communication connection is established with the migration source, and initialization is performed to prepare for receiving data sent by the source.

[0317] For example, Fig.10 1 is a schematic flow chart of a migration source end memory push phase provided by an embodiment of the present application. Fig.10 As shown, the process may include steps 501 to 506.

[0318] 501, the migration management front end A notifies the migration management back end A to start memory push.

[0319] It should be understood that the manner in which the migration management front end A notifies the migration management back end A can refer to the relevant description in the foregoing text.

[0320] 502, the migration management front end A sends handle information to the migration management front end B.

[0321] The migration management front end A may send handle information to the migration management front end B, so that the migration management front end B can reconstruct the handle mapping with the accelerator according to the handle information.

[0322] 503, the migration management backend A allocates tasks to corresponding task migrators according to the task type.

[0323] For example, for an inference task, the migration management backend A may instruct the inference task migrator to transfer the memory data of the inference task.

[0324] For a training task, the migration management backend A can instruct the training task migrator to transfer the memory data of the training task.

[0325] It should be understood that during the accelerator migration process, the accelerator may include reasoning tasks or training tasks, or may include both reasoning tasks and training tasks.

[0326] 504: Migration management backend A starts memory push of the task.

[0327] For the tasks included in the accelerator, the migration management backend A can start the memory push of all tasks. Exemplarily, the migration management backend A instructs the inference task migrator to start the memory transfer of the inference task, and instructs the training task migrator to start the memory transfer of the training task.

[0328] 505, the migration management backend A determines whether the memory push is in a stable state.

[0329] For an inference task, when the memory data of the model parameters of the inference task and the state metadata of the task execution are transmitted, the migration management backend A determines that the memory push of the inference task is in a stable state.

[0330] For a training task, when the model parameters of the training task enter periodic transmission and the operators in each iteration begin to cache, the migration management backend A determines that the memory push of the training task is in a stable state. During the memory push phase, the migration source can also transmit the state metadata of the training task execution to the migration destination.

[0331] In this way, in the memory push phase, when it is determined that the memory push of the task is in a stable state, the shutdown copy phase can be entered at any time.

[0332] 506 , when it is determined that the memory push is in a stable state, the migration management backend A notifies the migration management frontend A that the memory push is in a stable state.

[0333] It should be understood that the manner in which the migration management backend A notifies the migration management frontend A can refer to the relevant description in the foregoing text.

[0334] Based on the embodiments of the present application, in the memory push stage, the migration management front end A can notify the migration management back end A to start memory push; thereafter, the migration management back end A can assign tasks to the task migrator according to the task type and start memory push; when the memory push is in a stable state, the migration management front end A can be notified that the memory push is in a stable state, so that the migration management front end A can start the next stage of the migration process.

[0335] The following will combine Fig.11 The specific process of the memory push phase for the training task in the embodiment of the present application is introduced.

[0336] For example, Fig.11 is a schematic flow chart of the memory push phase of a training task provided by an embodiment of the present application. Fig.11 As shown, the process may include steps 551 to 556.

[0337] 551. The migration management backend A sets the memory push state to the initialization state.

[0338] In this step, when the memory push state is in the initialization state, it can be understood that the migration management backend A is ready to transfer memory data and can enter the memory push phase at any time.

[0339] 552, the migration management backend A records the time T when determining the start of the current iteration of the training task.

[0340] Exemplarily, the migration management backend A can determine whether the training task iteration starts based on the dirty page information obtained from the dirty page tracking module and the task information table obtained from the task context management module.

[0341] Alternatively, the migration management backend A may also determine the start of the training task iteration according to an iteration start instruction obtained from the task context management module.

[0342] After the migration management backend A determines the start of each iteration of the training task, the start time T of the iteration can be recorded.

[0343] 553, the migration management backend A transmits the model parameters of the current iteration to the migration destination.

[0344] Since the model parameters of the training task will change after each iteration, at the beginning of each iteration, the model parameters of the current iteration are transmitted to the migration management backend B of the migration destination.

[0345] 554, the migration management backend A caches the completed operators and the completion time of the operators in the operator queue Q.

[0346] For completed operators, the migration management backend A can cache them in the operator queue Q and record the completion timestamp. In this way, since the operators in the operator queue Q all record the completion time, it is convenient for the subsequent migration management backend A to clear invalid operators.

[0347] 555, the migration management backend A controls the operator buffer to clear the operators before time T in the operator queue Q, and sets the memory push state to the ready state.

[0348] Since the start time of the current iteration is T, the operators in the operator queue Q before time T have been executed, and the previous iteration has been completed. The operators corresponding to the previous iteration are invalid operators. Therefore, the migration management backend A can control the operator cache to clear the operators in the operator queue A before time T.

[0349] For a training task, the corresponding model parameters will change with each iteration, and the intermediate data generated will be large in scale, and users are concerned about the latest iteration results. Therefore, in the embodiment of the present application, during the memory push phase, the migration source does not transmit intermediate data, but transmits model parameter data and corresponding operators that can represent training milestones. In this way, the migration destination can generate corresponding intermediate data based on the model parameter data and corresponding operators, thereby reducing the amount of data that needs to be transmitted during the memory push phase, and can restore the latest iteration results of the training task.

[0350] It is understandable that the specific execution order of the above steps 551-555 is not limited in the embodiment of the present application. In other examples, step 553 and step 554 can also be executed synchronously.

[0351] During the memory push phase, the migration management backend B at the migration destination is mainly used to receive data sent by the migration management backend A at the migration source, build task context, and restore the task execution environment.

[0352] The following will combine Fig.12 The process of migrating the source end during the shutdown copy phase in the embodiment of the present application is introduced.

[0353] For example, Fig.12 1 is a schematic flow chart of a migration source end in the shutdown copy phase provided by an embodiment of the present application. Fig.12 As shown, the process may include steps 610 to 640.

[0354] 610, the migration management front end A notifies the migration management back end A to start shutdown copying.

[0355] It should be understood that the manner in which the migration management front end A notifies the migration management back end A can refer to the relevant description in the foregoing text.

[0356] 620, the migration management backend A controls the task context management module to suspend the execution of all tasks.

[0357] Exemplarily, after receiving the message of starting shutdown copy, the migration management backend A can send a message to the task context management module to suspend the execution of all tasks. After receiving the message of suspending the execution of all tasks, the task context management module can suspend the execution of all tasks in the accelerator.

[0358] 630, the migration management backend A controls the task migrator to perform shutdown migration according to the task type.

[0359] During the shutdown copy phase, the migration management backend A can control the inference task migrator to migrate based on the pre-copy strategy of a single copy, that is, the inference task migrator can transmit the dirty page information obtained by the dirty page tracking device. The dirty page information is the dirty page generated after a memory transfer during the memory push phase.

[0360] The migration management backend A can also control the training task migrator to migrate based on the task replay strategy, that is, the training task migrator can transfer the operators in the cached operator queue Q. In this way, the migration destination can restore the latest iteration result of the training task based on the model parameter data received in the memory push phase and the operator.

[0361] 640. When the copying is completed, the migration management backend A notifies the migration management frontend A to shut down and complete the copying.

[0362] It should be understood that the manner in which the migration management backend A notifies the migration management frontend A can refer to the relevant description in the foregoing text.

[0363] It is understandable that the embodiment of the present application does not limit the specific execution order of the above steps 610-640.

[0364] Based on the embodiments of the present application, during the shutdown copy phase, the migration management backend A can execute different migration strategies according to the task type, thereby achieving efficient migration of various task types in the accelerator.

[0365] During the shutdown copy phase, the migration management backend B at the migration destination is mainly used to receive data sent by the migration management backend A at the migration source.

[0366] The following will combine Fig.13 This section describes the process of executing the recovery task on the migration destination.

[0367] Fig.13 FIG. 1 is a schematic flow chart of a migration destination recovery task execution provided by an embodiment of the present application. Fig.13 As shown, the process may include steps 710 to 740.

[0368] 710, the migration management front end B rebuilds the handle mapping according to the handle information.

[0369] It should be understood that after the shutdown copy is completed, the migration destination needs to resume task execution, and first needs to rebuild the mapping between the virtual machine and the accelerator according to the handle information.

[0370] 720 , the migration management front end B notifies the migration management back end B to resume task execution.

[0371] It should be understood that the manner in which the migration management front end B notifies the migration management back end B can refer to the above description.

[0372] 730, the migration management backend B controls the task context management module to resume task execution.

[0373] Exemplarily, the migration management backend B can instruct the task migrator to build a task execution environment according to the task information table in the received data. The reasoning task migrator can be used to build a task execution environment for the reasoning task, and the training task migrator can be used to build a task execution environment for the training task.

[0374] Specifically, the inference task migrator can control the task context management module to rebuild the inference task execution context, and use the state metadata of the inference task execution of the execution context to restore the value of the register of the execution controller to restore the execution state of the inference task (the operator queue that needs to continue to execute) and memory mapping information, and drive the execution of the inference task. It can be understood that for the inference task, since its model parameters remain unchanged, when the execution data such as the operator queue and memory mapping information that the above-mentioned inference task needs to continue to execute are ready, the execution can be directly resumed according to the model parameters and execution data.

[0375] The training task migrator can control the task context management module to rebuild the training task execution context, and use the state metadata of the training task execution in the execution context to restore the value of the register of the execution controller to restore the execution state of the training task (the operator queue that needs to continue to be executed) and memory mapping information, and drive the execution of the training task. For the training task, since its model parameters will change after each iteration, the training task migrator can restore the latest results of the current iteration of the training task based on the latest model parameter data of the training task and the operators in the operator queue that have been executed in the latest iteration. After that, the training task can continue to execute according to its model parameters, the operator queue that needs to continue to be executed, and the execution data such as memory mapping information.

[0376] 740, after all tasks are executed normally, the migration management backend B enters a dormant state and notifies the migration management frontend B.

[0377] It is understandable that after obtaining the information that the migration management backend B has entered the sleep state, the migration management front end B may also enter the sleep state and wait for the next wake-up.

[0378] It is understandable that the embodiment of the present application does not limit the specific execution order of the above steps 710-740.

[0379] Based on the embodiment of the present application, in the task execution recovery phase, the migration management backend can recover all task executions according to the task type. For inference tasks, it can directly recover execution based on the received data. For training tasks, it can recover the latest results of the current iteration of the training task based on the received model parameters and operators, and then recover normal execution. At this point, all computing tasks in the accelerator have completed the migration from the migration source to the migration destination.

[0380] An embodiment of the present application also provides a migration device, including a processor and a communication interface, the communication interface is used to receive signals and transmit the signals to the processor, the processor processes the signals so that the migration method described in any possible implementation method in the foregoing text is executed.

[0381] The device may be a chip. For example, the chip may be a chip system or an independent chip, etc. For example, the chip may be an NPU or a GPU.

[0382] An embodiment of the present application also provides a readable storage medium, in which instructions are stored. When the instructions are executed on a device, the device executes the above-mentioned related method steps to implement the migration method in the above-mentioned embodiment.

[0383] The embodiment of the present application also provides a program product. When the program product is run on a device, the device executes the above-mentioned related steps to implement the migration method in the above-mentioned embodiment.

[0384] An embodiment of the present application also provides a system, including a migration source device and a migration destination device as described above.

[0385] An embodiment of the present application also provides a migration device, including a module for implementing the migration method described in any of the above embodiments.

[0386] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0387] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0388] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0389] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0390] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0391] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0392] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A migration method, characterized in that: The method is applied to migrate a source device, and the method comprises: Determine the type of artificial intelligence (AI) computing task, which includes training tasks and reasoning tasks of the AI ​​model; Determine a migration strategy according to the type of the AI ​​computing task, where the migration strategy is a strategy for the migration source device to transmit data of the AI ​​computing task to the migration destination device; When migrating the AI ​​computing task to the migration destination device according to the migration strategy, different migration strategies are adopted for different types of AI computing tasks.

2. The method according to claim 1, characterized in that Determining the type of the AI ​​computing task includes: Obtaining dirty page information of the migration source device; The type of the AI ​​computing task is determined according to the dirty page information.

3. The method according to claim 1 or 2, characterized in that: The determining the type of the AI ​​computing task according to the dirty page information includes: When it is determined according to the dirty page information that the memory of the model parameter of the AI ​​computing task is modified, determining that the AI ​​computing task is the training task; When it is determined according to the dirty page information that the memory of the model parameter of the AI ​​computing task has not been modified, the AI ​​computing task is determined to be the inference task.

4. The method according to any one of claims 1 to 3, characterized in that The migrating the AI ​​computing task to the migration destination device according to the migration strategy includes: Transmitting the first memory data of the training task to the migration destination device according to the migration strategy of the training task, the first memory data including the model parameters obtained by the latest iteration of the training task, the state metadata of the execution of the training task and the first operator queue that has been executed in the latest iteration; or The second memory data of the reasoning task is transmitted to the migration destination device according to the migration strategy of the reasoning task, and the second memory data includes the model parameters of the reasoning task, the state metadata of the reasoning task execution and the dirty page information of the reasoning task.

5. The method according to claim 4, characterized in that The transmitting the first operator queue in the first memory data of the training task to the migration destination device includes: Record the start time T of the current iteration; Transmitting the current model parameters of the training task to the migration destination device; Cache the operators that have been completed in this iteration and the time when the operators were completed in the first operator queue; Clearing the operators in the operator queue before the start time T; The first operator queue is transmitted to the migration destination device.

6. The method according to claim 5, characterized in that The method further comprises: Determining dirty page information of the training task; If it is determined through the dirty page information of the training task that the memory of the model parameters of the training task is completely modified, it is determined that the previous iteration of the training task is completed and the next iteration begins.

7. A migration method, characterized in that: The method is applied to the migration destination device, and the method includes: Receiving data of different types of artificial intelligence (AI) computing tasks sent by a migration source device, where the types of AI computing tasks include training tasks and reasoning tasks of AI models; Determine a recovery strategy according to the data of the different types of AI computing tasks, where the recovery strategy is a strategy for the migration destination device to restore execution of the AI ​​computing task; When the execution of the AI ​​computing task is restored according to the restoration strategy, different restoration strategies are adopted for the different types of AI computing tasks.

8. The method according to claim 7, characterized in that If the AI ​​computing task is the training task, the receiving of data of different types of AI computing tasks sent by the migration source device includes: Receive the first memory data of the training task sent by the migration source device, where the first memory data includes the model parameters of the latest iteration of the training task, the state metadata of the execution of the training task, and the first operator queue that has been executed in the latest iteration.

9. The method according to claim 8, characterized in that The resuming the execution of the AI ​​computing task according to the recovery strategy includes: Restoring the execution state of the training task according to the execution state metadata of the training task; Restoring the state of the latest iteration of the training task according to the model parameters obtained from the latest iteration of the training task and the first operator queue; Continue to execute the training task according to the second operator queue in the state metadata.

10. A migration device, characterized in that: include: A task context management module, used to determine the type of artificial intelligence AI computing task according to the instructions of the migration management backend, the type including the training task and the reasoning task of the AI ​​model; The migration management backend is used to determine a migration strategy according to the type of the AI ​​computing task, where the migration strategy is a strategy for the migration device to transmit data of the AI ​​computing task to another migration device; The migration management backend is further used to adopt different migration strategies for different types of computing tasks when migrating the computing tasks to the other migration device according to the migration strategy.

11. The migration device according to claim 10, characterized in that: The migration device also includes: A dirty page tracking module, used to determine the dirty page information generated by the migration device according to the instructions of the migration management backend; The task context management module is specifically used for: Acquire the dirty page information from the dirty page tracking module; The type of the AI ​​computing task is determined according to the dirty page information.

12. The migration device according to claim 10 or 11, characterized in that: The task context management module is specifically used for: When it is determined according to the dirty page information that the memory of the model parameter of the AI ​​computing task is modified, determining that the AI ​​computing task is a training task; When it is determined according to the dirty page information that the memory of the model parameter of the AI ​​computing task has not been modified, the AI ​​computing task is determined to be an inference task.

13. The migration device according to any one of claims 10 to 12, characterized in that: The migration management backend is specifically used for: Transmitting first memory data of the training task to the other migration device according to the migration strategy of the training task, the first memory data including model parameters obtained in the latest iteration of the training task, state metadata of the execution of the training task, and a first operator queue that has been executed in the latest iteration; or The second memory data of the reasoning task is transmitted to the other migration device according to the migration strategy of the reasoning task, where the second memory data includes the model parameters of the reasoning task, the state metadata of the reasoning task execution and the dirty page information of the reasoning task.

14. The migration device according to claim 13, characterized in that: The migration management backend is specifically used for: Record the start time T of the current iteration; Transmitting the current model parameters of the training task to another migration device; Cache the operators that have been completed in this iteration and the time when the operators were completed in the first operator queue; Clearing the operators in the operator queue before the start time T; The first operator queue is transmitted to the other migration device.

15. The migration device according to claim 14, characterized in that: The migration management backend is also used to: Determining dirty page information of the training task; If it is determined through the dirty page information of the training task that the memory of the model parameters of the training task is completely modified, it is determined that the previous iteration of the training task is completed and the next iteration begins.

16. A migration device, characterized in that: include: A migration management backend, used to control the migration device to receive data of different types of artificial intelligence AI computing tasks sent by another migration device, where the types of AI computing tasks include training tasks and reasoning tasks of AI models; The migration management backend is further used to determine a recovery strategy based on the data of the different types of AI computing tasks, where the recovery strategy is a strategy for the migration destination device to restore the execution of the AI ​​computing task; The migration management backend is further used to adopt different recovery strategies for different types of AI computing tasks when resuming the execution of the AI ​​computing task according to the recovery strategy.

17. The migration device according to claim 16, characterized in that: If the AI ​​computing task is the training task, the migration management backend is specifically used for: Control the migration device to receive the first memory data of the training task sent by the other migration device, wherein the first memory data includes the model parameters obtained from the latest iteration of the training task, the state metadata of the execution of the training task, and the first operator queue that has been executed in the latest iteration.

18. The migration device according to claim 17, characterized in that: The migration management backend is specifically used for: Restoring the execution state of the training task according to the execution state metadata of the training task; Restoring the state of the latest iteration of the training task according to the model parameters of the latest iteration of the training task and the first operator queue; Continue to execute the training task according to the second operator queue in the state metadata.

19. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the migration method as described in any one of claims 1 to 6 is executed, or the migration method as described in any one of claims 7 to 9 is executed.

20. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on the device, the migration method according to any one of claims 1 to 6 is executed, or the migration method according to any one of claims 7 to 9 is executed.

Images