Task processing method and device

Abstract

The invention discloses a task processing method and device, and the method comprises the steps: obtaining the processor state data of each graphics processor and the task state data of the state of a currently processed task; if it is determined that the processing resources of the first graphics processor are insufficient according to the processor state data and the task state data and a graphics processor with idle processing resources exists, determining a to-be-migrated task in the tasks processed by the first graphics processor; migrating task context data of the to-be-migrated task from a first virtual address corresponding to the first graphics processor to a second virtual address corresponding to the second graphics processor, wherein the task context data comprises data dependent on processing the to-be-migrated task; and when the task context data of the two virtual addresses meet a migration condition, migrating the remaining to-be-processed data from the first graphics processor to the second graphics processor based on the virtual address of the remaining to-be-processed data corresponding to the to-be-migrated task so as to continue to process the to-be-migrated task on the second graphics processor.

Description

Technical Field

[0001] This application relates to the field of task processing technology, and in particular to a task processing method and apparatus. Background Technology

[0002] Electronic devices can utilize graphics processing units (GPUs) to handle computationally intensive tasks, such as data processing tasks based on neural network models. However, when a GPU handles too many tasks or the computational demands of those tasks are too high, its processing efficiency will significantly decrease due to limitations in processing resources. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] The first aspect of this application provides a task processing method, including:

[0005] Obtain processor status data and task status data for each graphics processor, wherein the task status data represents the status of the task currently being processed by the corresponding graphics processor.

[0006] If it is determined that the processing resources of the first graphics processor are insufficient based on the processor status data and the task status data, and it is determined based on the processor status data that there is a graphics processor with idle processing resources, then the task to be migrated is determined from the tasks processed by the first graphics processor based on the task status data and the processor status data.

[0007] The task context data of the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor. The task context data includes data that the processing of the task to be migrated depends on.

[0008] If the task context data of the first virtual address and the task context data of the second virtual address meet the migration conditions, the remaining unprocessed data corresponding to the task to be migrated is migrated from the first graphics processor to the second graphics processor based on the virtual address of the remaining unprocessed data, so that the task to be migrated can continue to be processed on the second graphics processor.

[0009] Optional, also includes:

[0010] Obtain the task requirement data for the newly added task;

[0011] Based on the task requirement data and the processor status data of each graphics processor, the new task is assigned to the third graphics processor for processing.

[0012] Optionally, the method for determining the second graphics processor includes:

[0013] A second graphics processor is determined from among a plurality of graphics processors other than the first graphics processor based on at least one of the hardware characteristic parameters of each graphics processor, processor status data, task status data of the task to be migrated, and task data type.

[0014] Optionally, determining the task to be migrated based on the task status data and the processor status data in the task processed by the first graphics processor includes:

[0015] Based on the task status data and the processor status data, determine the task migration cost and task migration benefit for each task processed by the first graphics processor;

[0016] Based on the task migration cost and the task migration benefit, the task to be migrated is determined among the tasks processed by the first graphics processor.

[0017] The task migration cost is related to the time taken to migrate the corresponding task, and the task migration benefit is related to the expected reduction in processing time of the corresponding task after migration.

[0018] Optionally, the task context data of the first virtual address and the task context data of the second virtual address satisfy the migration conditions, including:

[0019] The difference between the task context data of the first virtual address and the task context data of the second virtual address is less than a threshold, or the task context data of the first virtual address and the task context data of the second virtual address are the same;

[0020] The task context data includes task initialization parameters, task progress data, task cache data, and dependency data.

[0021] Optionally, migrating the remaining unprocessed data from the first graphics processor to the second graphics processor based on the virtual address of the remaining unprocessed data corresponding to the task to be migrated includes:

[0022] It is determined that the remaining unprocessed data corresponding to the task to be migrated will be migrated from the current third virtual address to the fourth virtual address;

[0023] The third physical address of the remaining data to be processed in the video memory of the first graphics processor is determined based on the third virtual address;

[0024] The fourth physical address corresponding to the fourth virtual address is determined in the video memory of the second graphics processor;

[0025] The remaining data to be processed is transferred from the third physical address to the fourth physical address.

[0026] Optional, also includes:

[0027] Regularly save task backup data for each of the graphics processors' processed tasks;

[0028] In the event that any of the graphics processors is in an abnormal state, the tasks processed by the abnormal graphics processor are assigned to the graphics processors that are not in an abnormal state, based on the task backup data.

[0029] Optionally, the processor status data includes at least one of the following: video memory usage, computing core usage, temperature, task execution efficiency, and the number of tasks being processed;

[0030] The task status data includes at least one of the following: task identifier, task processing progress, task resource requirements, task priority, task submission time, and task deadline.

[0031] Optionally, determining that the first graphics processor has insufficient processing resources based on the processor status data and the task status data includes at least one of the following:

[0032] If the difference between the video memory usage rate and the upper limit of video memory usage is less than a first threshold, it is determined that the processing resources of the first graphics processor are insufficient.

[0033] If the difference between the core utilization rate and the core utilization limit is less than the second threshold, it is determined that the processing resources of the first graphics processor are insufficient.

[0034] If the average task waiting time determined based on the task processing progress is greater than the third threshold, it is determined that the processing resources of the first image processor are insufficient.

[0035] A second aspect of this application provides an electronic device, including a memory, a central processing unit, and a plurality of graphics processors;

[0036] The memory is used to store computer programs;

[0037] The plurality of graphics processors are used to process tasks;

[0038] The central processing unit is used to execute the computer program to perform:

[0039] Obtain processor status data and task status data for each graphics processor, wherein the task status data represents the status of the task currently being processed by the corresponding graphics processor.

[0040] If it is determined that the processing resources of the first graphics processor are insufficient based on the processor status data and the task status data, and it is determined based on the processor status data that there is a graphics processor with idle processing resources, then the task to be migrated is determined from the tasks processed by the first graphics processor based on the task status data and the processor status data.

[0041] The task context data of the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor. The task context data includes data that the processing of the task to be migrated depends on.

[0042] If the task context data of the first virtual address and the task context data of the second virtual address meet the migration conditions, the remaining unprocessed data corresponding to the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor, so as to continue processing the task to be migrated on the second graphics processor. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart of a task processing method provided in an embodiment of this application;

[0045] Figure 2 This is a schematic diagram of the structure of a processing system provided in an embodiment of this application;

[0046] Figure 3 This is a flowchart of a method for determining a task to be migrated, provided in an embodiment of this application;

[0047] Figure 4 This is a flowchart of a method for migrating remaining unprocessed data provided in an embodiment of this application;

[0048] Figure 5 This is a flowchart illustrating a method for allocating newly added tasks according to an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] This embodiment provides a task processing method; please refer to [link / reference]. Figure 1 The method may include the following steps.

[0052] S101, obtain processor status data and task status data for each graphics processor. The task status data represents the status of the task currently being processed by the corresponding graphics processor.

[0053] S102, if it is determined from the processor status data and task status data that the processing resources of the first graphics processor are insufficient, and it is determined from the processor status data that there is a graphics processor with idle processing resources, the task to be migrated is determined from the tasks processed by the first graphics processor based on the task status data and processor status data.

[0054] S103, the task context data of the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor. The task context data includes the data that the processing of the task to be migrated depends on.

[0055] S104, if the task context data of the first virtual address and the task context data of the second virtual address meet the migration conditions, the remaining unprocessed data is migrated from the first graphics processor to the second graphics processor based on the virtual address of the remaining unprocessed data corresponding to the task to be migrated, so as to continue processing the task to be migrated on the second graphics processor.

[0056] The processing resources of a graphics processing unit include, but are not limited to, computing cores, video memory, and other resources required for processing tasks.

[0057] The beneficial effects of this embodiment are as follows:

[0058] On the one hand, the processor status data and task status data determine that the processing resources of the first graphics processor are insufficient, and the processor status data also determines that there are graphics processors with idle processing resources. The tasks to be migrated are identified in the tasks processed by the first graphics processor, and the tasks to be migrated are migrated to the second graphics processor for further processing. This can alleviate the problem of insufficient resources of the first graphics processor and improve the processing efficiency of the migrated tasks.

[0059] On the other hand, when migrating a task to be migrated to the second graphics processor, the task context data on which the task depends is first migrated to the second graphics processor. When the migrated task context data meets the migration conditions, the remaining data to be processed corresponding to the task to be migrated is then migrated so that the task to be migrated can continue to be processed in the second graphics processor. In this way, the first graphics processor can still process the task to be migrated while the task context data is being migrated, which shortens the time that the task to be migrated is interrupted due to the migration of data to be processed and reduces the time loss of the migration task. Furthermore, the above migration process is implemented through the virtual address of the graphics processor. In this way, the video memory of each graphics processor can be uniformly managed through the virtual address, thereby solving the problem of inconsistent video memory addresses of each graphics processor and further shortening the time for migrating data.

[0060] In summary, the task processing method of this embodiment improves task processing efficiency by migrating tasks between graphics processors when resources are insufficient and shortening the time for task migration.

[0061] The task processing method in this embodiment can be applied to... Figure 2 The processing system shown can be a server device that includes multiple graphics processors, or other computer device that includes multiple graphics processors and is capable of processing tasks based on the multiple graphics processors.

[0062] The types of tasks that each graphics processor can handle are not limited. For example, it can be inference tasks based on neural network models, optimization, encoding and decoding tasks of video and / or images, audio recognition or translation tasks, etc.

[0063] like Figure 2 As shown, the processing system may include at least one central processing unit (CPU) and multiple graphics processing units (GPUs), such as... Figure 2 The graphics processors 1 to 3 shown each have video memory for storing task-related data, which is used to process the corresponding task based on the data.

[0064] The central processing unit is used to run the scheduler and the middleware layer. The scheduler is used to allocate tasks to each graphics processor according to the method of this embodiment, and to migrate tasks between graphics processors. The middleware layer is used to manage the mapping relationship between virtual addresses and physical addresses of the video memory of each graphics processor. When the scheduler migrates data, it only accesses the virtual address of the middleware layer, and the middleware layer then accesses the corresponding physical address based on the virtual address.

[0065] The scheduler and intermediate layer are equivalent to programs running on the central processing unit. In other alternative embodiments, the above task processing method can also be implemented based on other programs with corresponding functions, and is not limited to... Figure 2The scheduler. For ease of explanation, the following description uses the scheduler as the execution body of the task processing method as an example to illustrate the implementation of the task processing method in this embodiment.

[0066] The scheduler can periodically execute step S101 for each graphics processor at certain time intervals to obtain processor status data and task status data for each graphics processor.

[0067] Processor status data may include any one or more data that can characterize the current state and load of the corresponding graphics processor. As some examples, processor status data may include any one or more of the following: memory usage, compute core usage, temperature, task execution efficiency, data throughput, and the number of tasks being processed.

[0068] Video memory utilization refers to the proportion of the graphics processor's video memory that is currently in use relative to the total storage space. Computation core utilization refers to the proportion of computing cores in the graphics processor that are actively working. Temperature refers to the current temperature of the graphics processor, which can be detected by the built-in temperature sensor. Task execution efficiency can be expressed as the average processing time of a single task, specifically equal to the average processing time of tasks completed over a past period (e.g., the past 40 minutes, the past 2 hours, etc.). The number of tasks currently being processed can be obtained from the task queue maintained by the graphics processor. The task queue contains multiple records, each corresponding to a task currently being processed by the graphics processor; therefore, the number of records in the graphics processor's task queue can be used as the number of tasks currently being processed by the graphics processor. Data throughput can be the sum of the amount of data input to the graphics processor and the amount of data output by the graphics processor per unit of time.

[0069] Optionally, the processor status data may also include task throughput, which is defined as the number of tasks processed by the graphics processor per unit time, characterizing the efficiency of the graphics processor in executing tasks.

[0070] In the task queue of a graphics processing unit (GPU), each record may include one or more of the following information: a task identifier (e.g., ID), task resource requirements, task submission time, and task deadline. The task identifier distinguishes different tasks; task resource requirements include the amount of resources needed to process the task, such as the required video memory space and the number of computing cores, and may also include requirements for the resource type, such as the need for computing cores supporting 8-bit floating-point (fp8) quantization precision; the task submission time is equivalent to the timestamp when the processing system receives the task; the task deadline can be specified by the terminal that uploaded the task when the processing system receives it, indicating that the corresponding task should be completed no later than this time.

[0071] For a graphics processing unit (GPU), the scheduler can obtain task status data for each task currently being processed by the GPU. The task status data may include any one or more data that characterize the current status of the corresponding task. For example, the task status data may include at least one of the following: task identifier, task processing progress, task resource requirements, task priority, task submission time, and task deadline.

[0072] When the graphics processor has a task queue, the above-mentioned task status data can be obtained directly from the records corresponding to each task in the task queue.

[0073] Task processing progress can be expressed as the proportion or percentage of the total amount of data to be processed for the task. For example, if a task requires image enhancement processing on 20 frames of images, and image enhancement processing has been completed on 10 frames, then the task processing progress is 50%. Alternatively, task processing progress can also be expressed as the number of iterations.

[0074] The first graphics processor refers to the graphics processor that needs to relocate some tasks. In step S102, for each graphics processor, the scheduler can determine whether the graphics processor is in a state of insufficient resources (or severe resource contention) based on the processor status data and task status data of that graphics processor. If any graphics processor is determined to be in a state of insufficient resources, then this graphics processor can be identified as the first graphics processor that needs to relocate some tasks.

[0075] The method for determining the graphics processor (GPU) is not limited. In some embodiments, the scheduler may use a pre-built and trained model for evaluating load status to process the processor state data and task state data of a GPU to obtain the load balancing status of that GPU, and determine whether resources are insufficient based on the load status. For example, the load balancing status output by the model can be any of idle, medium, busy, or overloaded. If the load balancing status of a GPU is overloaded, it is determined that the GPU is in a state of insufficient resources; if the load balancing status of a GPU is not overloaded, it is determined that the GPU is not in a state of insufficient resources.

[0076] In some embodiments, the scheduler can analyze whether the processor status data and task status data of the graphics processor meet one or more pre-defined judgment rules. If they meet the rules, the scheduler determines that the graphics processor is in a state of insufficient resources; if they do not meet the rules, the scheduler determines that the graphics processor is not in a state of insufficient resources.

[0077] For example, the scheduler can determine whether the processor status data and task status data of the graphics processor meet the following criteria: if at least one criterion is met, the system is determined to be in a resource-insufficient state; if none of the criteria are met, the system is determined not to be in a resource-insufficient state.

[0078] Rule 1: If the difference between the video memory usage rate and the video memory usage limit is less than the first threshold, it is determined that the processing resources of the first graphics processor are insufficient.

[0079] Rule 2: If the difference between the calculated core utilization rate and the core utilization limit is less than the second threshold, it is determined that the processing resources of the first graphics processor are insufficient.

[0080] Rule 3: If the average task waiting time determined based on the task processing progress is greater than the third threshold, it is determined that the processing resources of the first image processor are insufficient.

[0081] In Rule 1, different graphics processors can share a single maximum memory usage limit, or they can have their own different maximum memory usage limits. The maximum memory usage limit can be 100%, indicating that the graphics processor's memory is full, or it can be slightly less than 100%. Similarly, in Rule 2, different graphics processors can share a single maximum core usage limit, or they can have their own different maximum core usage limits. The maximum memory usage limit can be 100%, indicating that the graphics processor's core is full, or it can be slightly less than 100%.

[0082] In Rule 3, the average task waiting time can be the average of the waiting times of all tasks currently being processed by the graphics processor. The waiting time of a task can be determined based on its task processing progress. This can be done by determining that if the current task processing progress of a task is the same as or the difference is too small (e.g., only 1% progress has been made), then the task is considered to be in a waiting state during this period. Thus, the cumulative duration of a task in a waiting state can be determined as the task's waiting time.

[0083] The first to third thresholds involved in the above rules can be fixed or floating values, and the specific values ​​can be set as needed without limitation.

[0084] If it is determined that there is a first graphics processor with insufficient processing resources, it can be further determined whether there is a graphics processor with idle processing resources, that is, whether there is a graphics processor that can migrate the task. If so, S102 can be executed to determine the task to be migrated.

[0085] If it is determined that there is a first graphics processor with insufficient processing resources, and no graphics processor with idle processing resources is found, it means that the processing system is already saturated. At this time, a corresponding prompt can be output to trigger the terminal to temporarily stop uploading new tasks.

[0086] Whether a graphics processor has idle processing resources can be determined based on the aforementioned load balancing status. For example, it can be determined that graphics processors with an idle load balancing status and a medium load balancing status have idle processing resources, while graphics processors with a busy load balancing status do not have idle processing resources.

[0087] Whether a graphics processing unit (GPU) has idle processing resources can also be determined by whether the GPU's memory usage and core usage are less than the corresponding idle threshold. If they are less than the corresponding idle threshold, it means there are idle processing resources; if they are greater than or equal to the corresponding idle threshold, it means there are no idle processing resources.

[0088] In step S102, the scheduler can determine the task to be migrated and the actual task to be migrated, and then take over the processing of the task from the first graphics processor.

[0089] The scheduler can simultaneously determine the task to be migrated and the second graphics processor. It can determine the task to be migrated first and then determine the second graphics processor in combination with the task to be migrated. It can also determine the second graphics processor first and then determine a suitable task to be migrated in combination with the second graphics processor. It can also perform iterative optimization, first initially determining candidate graphics processors, then determining the task to be migrated, and then determining whether to adjust the candidate graphics processors in combination with the task to be migrated to finally determine the second graphics processor.

[0090] There are no restrictions on the method for identifying tasks to be migrated; please refer to [link / reference]. Figure 3 An optional method for determining the task to be migrated includes the following steps.

[0091] S301, based on the task status data and processor status data, determine the task migration cost and task migration benefit for each task processed by the first graphics processor.

[0092] S302, based on the task migration cost and task migration benefit, determine the task to be migrated from the various tasks processed by the first graphics processor; the task migration cost is related to the time taken to migrate the corresponding task, and the task migration benefit is related to the expected reduction in the processing time of the corresponding task after migration.

[0093] One way to determine the task migration cost of a task is to: determine the amount of task context data and the amount of remaining unprocessed data for the task; based on the sum of the two data amounts and the data transfer bandwidth between the graphics processors, determine the expected time to migrate the task context data and remaining unprocessed data between the graphics processors; and then determine the task migration cost based on this expected time. For example, this expected time can be used as the task migration cost.

[0094] The way to determine the benefits of task migration for a task is to evaluate the actual processing capacity of the first graphics processor when processing the task based on the processor status data and task status data of the first graphics processor, and determine how long it is expected to take to process the task on the first graphics processor based on the actual processing capacity and the amount of data of the remaining unprocessed data of the task. This time is recorded as the expected processing time before migration.

[0095] Based on the reference processing capacity and the amount of remaining data to be processed for this task, determine how long it is expected to take to process this task after migrating to another graphics processor. This time is recorded as the expected processing time after migration.

[0096] Finally, the difference between the expected processing time before migration and the expected processing time after migration is taken as the expected reduction in processing time. The task migration benefit is determined based on the expected reduction in processing time, for example, the expected reduction in processing time is taken as the task migration benefit.

[0097] Processing capacity when handling a task can be represented by the amount of data that can be processed per unit of time. Reference processing capacity can be determined by analyzing the actual processing capacity of various graphics processors with available processing resources over a period of time; for example, it can be equal to the average of these actual processing capacities.

[0098] In S302, the task migration cost and task migration benefit of each task in the first graphics processor can be compared, and at least one task to be migrated can be determined according to the rule of minimizing task migration cost and maximizing task migration benefit.

[0099] For example, you can filter all tasks whose migration cost is less than a certain threshold, and then select the one or several tasks with the greatest migration benefit from the filtered tasks as the tasks to be migrated. Alternatively, you can first filter by migration benefit and then determine by migration cost. Or you can weight and merge the migration benefit and migration cost of each task, and determine the tasks to be migrated based on the fusion result.

[0100] The advantage of determining the task to be migrated using the method of this embodiment is that:

[0101] By determining the tasks to be migrated based on the migration costs and benefits, the processing time of the tasks to be migrated after migration can be shortened as much as possible. This reduces the transmission resources between graphics processors and the migration time required for the migration of the tasks to be migrated, thereby improving the overall processing performance of the processing system.

[0102] Optionally, tasks to be migrated can be determined based on the characteristics of different tasks. For example, assess the dependencies of tasks. Tasks with strong dependencies require more stringent maintenance of intermediate states, making migration more difficult and prone to errors after migration. Therefore, tasks with weak dependencies are prioritized for migration, while tasks with strong dependencies are not. Assess whether a task belongs to a shared resource task. Tasks that do not belong to a shared resource task are prioritized for migration.

[0103] A task with strong dependencies can be understood as one that requires the output of one or more other tasks as its input, and / or the output of this task as the input of one or more other tasks. If neither of these conditions applies, the task can be considered to have weak dependencies.

[0104] A task belonging to the shared resource task can be understood as a task that can share some data with one or more other tasks. For example, in a data processing task based on a neural network model, if two tasks share the model weights of the same model, they belong to the shared resource task. The advantage of prioritizing the migration of non-shared resource tasks is that it ensures that multiple shared resource tasks are processed on the same graphics processor as much as possible, thereby saving the processing resources occupied by multiple tasks sharing resources.

[0105] Optionally, if a second graphics processor (or candidate graphics processor) has been determined first, the tasks to be migrated can be determined by considering the resource utilization characteristics of the tasks. These characteristics characterize the strength of the resource requirements for different types of resources when processing the task. For example, tasks based on Large Language Models (LLM) have a strong demand for video memory resources but a weak demand for computing core resources. When determining the tasks to be migrated, resources with strong resource utilization characteristics and those with more spare resources in the second graphics processor can be prioritized for migration. For instance, if the second graphics processor has a lot of spare video memory, LLM-based tasks can be prioritized; if it has a lot of spare computing cores, tasks with strong computing core requirements can be prioritized.

[0106] The resource utilization characteristics of a task can be determined by periodically measuring and analyzing the performance and resource usage of the task and / or the corresponding program. For specific determination methods, please refer to relevant technologies, which will not be elaborated here.

[0107] Optionally, if a second graphics processor (or a candidate graphics processor) is determined first, the actual processing power of the second graphics processor can be replaced with the aforementioned reference processing power when determining the task migration benefit, so as to more accurately assess the expected processing time of the task after migration.

[0108] Optionally, the method for determining the second graphics processor includes:

[0109] Based on at least one of the following: hardware characteristic parameters of each graphics processor, processor status data, task status data of the task to be migrated, and task data type, a second graphics processor is determined from among multiple graphics processors other than the first graphics processor.

[0110] In the above determination method, the multiple graphics processors other than the first graphics processor can also be limited to the aforementioned graphics processors with idle processing resources other than the first graphics processor.

[0111] The hardware characteristics of a graphics processing unit (GPU) include, but are not limited to, any one or more of the following: the number of GPU cores, the total memory capacity of the GPU, the bandwidth of the GPU, and the quantization precision supported by the GPU.

[0112] Here are some examples of methods for determining a second graphics processor.

[0113] The second graphics processor (GPU) can be determined based on the data type of the task to be migrated and the processor status data of the graphics processor. Different data types of tasks consume different amounts of processing resources; some data types consume more computing cores, while others consume more video memory. Therefore, the type of processing resource that the task to be migrated uses more can be determined based on its data type. The GPU with the most available video memory of that type can be selected as the second GPU. For example, if the task to be migrated uses a lot of video memory, the GPU with the most free video memory can be selected as the second GPU.

[0114] The second graphics processor can be determined based on hardware characteristic parameters and the data type of the task to be migrated. Different data types of tasks require different levels of precision, so the graphics processor whose quantization precision in the hardware characteristic parameters matches the precision corresponding to the data type of the task can be selected as the second graphics processor.

[0115] The second graphics processor can be determined based on the task status data of the task to be migrated and the processor status data of the graphics processor. From the task resource requirements contained in the task status data of the task to be migrated, the processing resource most needed by the task to be migrated is determined; the idle processing resources of each graphics processor are obtained, and the graphics processor whose idle processing resources are the same as the processing resource most needed by the task to be migrated is determined as the second graphics processor.

[0116] If the second graphics processor is determined before the task to be migrated, it can be determined solely based on processor status data. For example, based on the processor status data of each graphics processor, the graphics processor with the most idle processing resources can be selected as the second graphics processor.

[0117] Optionally, the scheduler can invoke a neural network model with data processing and decision-making capabilities to process relevant information in order to determine the second image processor and the task to be transferred in accordance with the above method, or the above method for determining the second image processor and the task to be transferred can be implemented through a pre-set program and decision rules.

[0118] Once the task to be migrated and the second graphics processor are determined, step S103 can be executed. During the execution of S103, the first virtual address corresponding to the first graphics processor and the second virtual address corresponding to the second graphics processor can be determined.

[0119] The first virtual address refers to the virtual address corresponding to the first physical address, which is the physical address in the video memory of the first graphics processor where the task context data of the task to be migrated is stored. The second virtual address refers to the virtual address in the video memory of the second graphics processor, which is also used to store the task context data of the task to be migrated.

[0120] In step S103, the scheduler can determine the second virtual address for migrating in task context data as follows: based on the amount of task context data of the task to be migrated, and provide this amount of data to the intermediate layer, the intermediate layer allocates a storage space in the video memory of the second graphics processor to store the task context data based on this amount of data, and uses the virtual address corresponding to this storage space as the second virtual address.

[0121] It should be noted that in S103, migrating the task context data from the first virtual address to the second virtual address means that the task context data of the first virtual address is synchronized to the second virtual address in real time. That is, on the one hand, the task context data is written to the second virtual address, and on the other hand, the task context data of the first virtual address remains unchanged. In this way, the first graphics processor can continue to process the remaining data to be processed corresponding to the task to be migrated during the migration of task context data, without having to stop the task to be migrated.

[0122] By migrating task context data to the second graphics processor, the necessary processing resources can be pre-allocated on the second graphics processor before migrating the remaining data to be processed. This means pre-allocating the required video memory and computing cores, and initializing the necessary runtime environment. In this way, after migrating the remaining data to the second graphics processor, the second graphics processor can directly continue to process the remaining data based on the allocated video memory and computing cores, thereby minimizing the time of task interruption caused by migrating the remaining data.

[0123] Task context data may include one or more of the following: task initialization parameters, task progress data, task cache data, and dependency data. Task initialization parameters may include, for example, the model structure parameters and hyperparameters of the model required by the task to be migrated; task cache data may include, for example, intermediate computation results from the first graphics processor processing the task to be migrated, and cached data generated during the processing of the task by the first graphics processor, such as key-value cache (KV cache) when the task to be migrated is processed based on a large language model; dependency data represents the dependencies of the task to be migrated, such as indicating which tasks the task to be migrated depends on.

[0124] When executing S103, the scheduler can send a request to the intermediate layer to synchronize data from the first virtual address to the second virtual address. Then, the intermediate layer can synchronize the task context data stored at the first physical address of the first graphics processor's video memory to the second physical address of the second graphics processor's video memory based on the mapping relationship between the virtual addresses and physical addresses it manages.

[0125] In some embodiments, the scheduler may also obtain the task context data of the task to be migrated from the first virtual address, package the obtained task context data into one or more data packets, send these data packets to the second virtual address, and then the intermediate layer forwards the data packets sent to the second virtual address to the second physical address of the video memory of the second graphics processor.

[0126] After starting to migrate task context data to the second virtual address, the scheduler can periodically check whether the task context data of the first virtual address and the task context data of the second virtual address meet the migration conditions. If the migration conditions are not met, S103 is executed; if the migration conditions are met, S104 is executed.

[0127] Among them, the task context data of the first virtual address and the task context data of the second virtual address satisfy the migration conditions, including:

[0128] Condition 1: The difference between the task context data of the first virtual address and the task context data of the second virtual address is less than a threshold.

[0129] Alternatively, under condition two, the task context data of the first virtual address and the task context data of the second virtual address are the same.

[0130] In other words, as long as either condition one or condition two is met, the scheduler determines that the migration condition is met and executes S104.

[0131] The degree of difference in condition one can be represented by the amount of data that is different between the task context data of the two virtual addresses. For example, if the task context data of the second virtual address contains 50 megabytes (MB) of data that is different from the task context data of the first virtual address, then the degree of difference is 50MB.

[0132] The threshold for condition one can be set as needed and is not limited. For example, it can be based on the data transmission bandwidth and set as the amount of data that can be transmitted between two graphics processors in one unit of time. For example, if the data transmission bandwidth is 100MB per second, then the threshold for condition one can be 100MB.

[0133] To confirm whether the migration conditions are met, the scheduler can mark the task context data of the task to be migrated. Specifically, it can mark which data in the task context data has been synchronized to the second virtual address, which data has not been synchronized to the second virtual address, and which data has been synchronized to the second virtual address and then updated. Thus, if all the task context data of the task to be migrated is marked as having been synchronized to the second virtual address, then condition two can be confirmed to be met. If the total amount of data that has not been synchronized to the second virtual address and data that has been synchronized to the second virtual address and then updated (i.e., the aforementioned degree of difference) in the total task context data of the task to be migrated is less than the threshold, then condition one can be confirmed to be met.

[0134] When S104 is executed, the scheduler can control the first graphics processor to stop executing the task to be migrated, and then start migrating the remaining unprocessed data corresponding to the task to be migrated from the video memory of the first graphics processor to the video memory of the second graphics processor. After the migration is completed, the scheduler can control the second graphics processor to start processing the remaining unprocessed data corresponding to the task to be migrated, so as to continue executing the task to be migrated.

[0135] The remaining unprocessed data of the task to be migrated may include the data of the task to be migrated that has not yet been processed and is stored in the video memory of the first graphics processor when the first graphics processor stops executing, and / or intermediate result data during the processing.

[0136] Migrating the remaining data to be processed from the first graphics processor to the second graphics processor still requires going through the aforementioned intermediate layer, and is performed based on virtual addresses. Figure 4 As shown, the process of migrating the remaining data to be processed may include the following steps.

[0137] S401, determine to migrate the remaining unprocessed data corresponding to the task to be migrated from the current third virtual address to the fourth virtual address.

[0138] S402, determine the third physical address of the remaining unprocessed data in the video memory of the first graphics processor based on the third virtual address.

[0139] S403 determines the fourth physical address corresponding to the fourth virtual address in the video memory of the second graphics processor.

[0140] S404, transfer the remaining data to be processed from the third physical address to the fourth physical address.

[0141] In step S401, the scheduler may, in response to the fulfillment of the migration condition, obtain the third virtual address where the remaining data to be processed is currently located, and determine the fourth virtual address for migrating the remaining data to be processed from the virtual address corresponding to the video memory of the second graphics processor. The method for determining the fourth virtual address can refer to the method for determining the second virtual address in the foregoing embodiment, and will not be repeated here.

[0142] Once the fourth virtual address is determined, the scheduler can send a migration instruction to the intermediate layer to move the remaining unprocessed data from the third virtual address to the fourth virtual address.

[0143] In response to receiving the migration instruction, the middle layer determines the third physical address corresponding to the third virtual address and the fourth physical address corresponding to the fourth virtual address based on the mapping relationship between the virtual addresses and physical addresses it manages. Then, it can execute S404 to read out the remaining unprocessed data from the third physical address and write it to the fourth physical address to complete the migration of the remaining unprocessed data.

[0144] Furthermore, after migrating the remaining data to be processed to the second graphics processor, any subsequent requests to execute the migration task can be directly forwarded to the second graphics processor, which will then execute the migration task.

[0145] As can be seen from the above migration process, on the one hand, before starting to migrate the remaining unprocessed data, the second graphics processor has already obtained the task context data of the task to be migrated. Based on this task context data, the environment and dependencies required to execute the task to be processed are configured. Therefore, after obtaining the remaining unprocessed data, the second graphics processor can immediately start to execute the task to be migrated, realizing the seamless migration of the task to be migrated. On the other hand, during the migration of task context data, the first graphics processor can continue to execute the task to be migrated, and only stop executing the task to be migrated when the migration of the remaining unprocessed data begins, shortening the time when the execution of the task to be migrated is interrupted due to task migration.

[0146] Optionally, when a new task requiring processing is received, the scheduler can determine a suitable third graphics processor for that new task and assign the new task to the third graphics processor for processing. See also... Figure 5 The method for assigning new tasks may include the following steps.

[0147] S501, obtain the task requirement data for the newly added task.

[0148] S502 assigns the new task to the third graphics processor for processing based on the task requirement data and the processor status data of each graphics processor.

[0149] New tasks can be uploaded to the processing system from any terminal device that is connected to the processing system. When uploading, the terminal device can provide the task requirement data of the new task at the same time, or the scheduler can determine the task requirement data of the new task by analyzing the requirements of similar tasks received in the past.

[0150] The task requirements data for a new task can include resource requirements data and computational requirements data. Resource requirements data can include one or more of the following: video memory bandwidth requirements, computational core requirements, data transfer bandwidth requirements, and other hardware support requirements. Other hardware support requirements can include, for example, requirements for computational precision, such as requiring 16-bit floating-point precision FP16 or 18-bit floating-point precision FP18, or requirements for specific types of dedicated hardware units, such as requiring ray tracing units.

[0151] Computational requirements data may include processing speed requirements, i.e., the number of floating-point operations per second (FLOPS), parallelism requirements, i.e. the number of parallel computing threads required by the task, latency sensitivity requirements, representing the task's real-time requirements, and other data describing the computational performance requirements of new tasks.

[0152] The third graphics processor can be any one of multiple graphics processors. An optional implementation of S502 may involve obtaining the load balancing state of each graphics processor determined based on processor state data, identifying graphics processors with idle and moderate load balancing states; then, based on task demand data, among the idle and moderate graphics processors, determining the graphics processor with the highest matching degree between its currently available resources and computing performance and the task demand data as the third graphics processor. For example, a graphics processor whose available resources are greater than the resource demand data of the new task, and whose current computing performance is greater than the computing demand data, may be determined as the third graphics processor.

[0153] The advantage of allocating new tasks using the above method is that by determining the most suitable third graphics processor to handle the new task and assigning it to the third graphics processor for execution, the additional overhead caused by frequently migrating tasks between different graphics processors in load balancing can be avoided, thereby improving the overall performance of the processing system.

[0154] Optionally, the method in this embodiment may further include:

[0155] Regularly save backup data of the tasks processed by each graphics processor;

[0156] If any graphics processor is in an abnormal state, the tasks that the abnormal graphics processor is handling are reassigned to the graphics processors that are not in an abnormal state, based on the task backup data.

[0157] Methods for obtaining task backup data include:

[0158] Obtain task status data and video memory status data. The video memory status data of a task may include the current video memory usage of the task, the video memory allocation of the graphics processor where the task is located (i.e., the distribution of video memory occupied by each task), the degree of memory fragmentation of the video memory where the task is located, whether there is locked or unreleased video memory in the video memory occupied by the task, and may also include various related data stored in the video memory when the graphics processor executes the task, such as model weight data loaded when executing the task, temporary calculation result data, etc.

[0159] Then, for each task, the scheduler can use a snapshot creation method from relevant technologies to package the task's task state data and memory state data into a corresponding task state snapshot, and then store this task snapshot in... Figure 2 The backup database shown in the image contains a snapshot of a task. Previously stored snapshots of the same task can be deleted from the backup database to save storage space.

[0160] Simultaneously, the scheduler can periodically check whether each graphics processor is in an abnormal state. An abnormal state for a graphics processor includes, but is not limited to, a graphics processor crashing, a hardware, software, or firmware failure, or a graphics processor being subjected to a network attack. In other words, an abnormal state can include any state that prevents the graphics processor from continuing to process tasks.

[0161] For a graphics processor that is determined to be in an abnormal state, the scheduler can immediately control the abnormal graphics processor to stop processing all tasks, and then query the backup database for a task snapshot of each task processed by the abnormal graphics processor.

[0162] Next, for each task processed by the faulty graphics processor, the scheduler, following the method described in the previous embodiment for determining a second graphics processor based on the task to be migrated, determines a substitute graphics processor to continue executing the task in place of the faulty graphics processor. A task snapshot of this task is then sent to the video memory of the substitute graphics processor, enabling the substitute graphics processor to continue executing the task based on the task snapshot. For different tasks, the corresponding substitute graphics processors may be the same or different.

[0163] Optionally, the scheduler sends a task snapshot to the corresponding substitute graphics processor and obtains execution status information from the substitute graphics processor to confirm whether the substitute graphics processor can handle the corresponding task normally. The execution status information may include the processor status data of the substitute graphics processor itself after receiving the task snapshot and executing the corresponding task, as well as the task execution status corresponding to the task snapshot; the task execution status indicates whether the task corresponding to the task snapshot has started execution.

[0164] Upon receiving a substitute graphics processor execution status information, if the load balancing status of the substitute graphics processor is determined to be overloaded based on the processor status data, or the task execution status indicates that the corresponding task has not started execution, then it can be determined that this substitute graphics processor cannot properly process the task corresponding to the task snapshot; if the load balancing status of the substitute graphics processor is not overloaded based on the processor status data, and the task execution status indicates that the corresponding task has started execution, then it can be determined that this substitute graphics processor can properly process the task corresponding to the task snapshot.

[0165] If, after a task snapshot is sent to the corresponding alternative graphics processor, it is confirmed based on the execution status information that the alternative graphics processor cannot process the corresponding task normally, the scheduler can re-determine a new alternative graphics processor for this task snapshot in the aforementioned manner; if it is confirmed based on the execution status information that the alternative graphics processor cannot process the corresponding task normally, then the alternative graphics processor will continue to process this task.

[0166] Optionally, to improve the reliability of the processing system, the scheduler and middleware layer themselves can also implement a disaster recovery mechanism. This can be achieved by deploying the scheduler and middleware layer in a distributed manner when the processing system includes multiple nodes. That is, they can be distributed across the central processing units (CPUs) of multiple nodes simultaneously. A distributed protocol (including but not limited to the Raft protocol) is used to ensure the consistency of the state of each scheduler and middleware layer. When any scheduler or middleware layer is in an abnormal state and cannot schedule the GPU tasks as described above, the scheduler and middleware layer on other nodes can take over, thereby improving the reliability of the processing system.

[0167] This embodiment also provides an electronic device; please refer to [link / reference]. Figure 6 It includes a memory 601, multiple graphics processors 602 and a central processing unit 603;

[0168] Memory 601 is used to store computer programs;

[0169] Multiple graphics processors 603 are used to process tasks;

[0170] The central processing unit 603 is used to execute computer programs to perform:

[0171] Obtain processor status data and task status data for each graphics processor. The task status data represents the status of the task currently being processed by the corresponding graphics processor.

[0172] If it is determined from the processor status data and task status data that the processing resources of the first graphics processor are insufficient, and it is determined from the processor status data that there is a graphics processor with idle processing resources, the task to be migrated is determined from the tasks processed by the first graphics processor based on the task status data and processor status data.

[0173] The task context data of the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor. The task context data includes the data that the processing of the task to be migrated depends on.

[0174] If the task context data at the first virtual address and the task context data at the second virtual address meet the migration conditions, the remaining unprocessed data corresponding to the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor, so that the task to be migrated can continue to be processed on the second graphics processor.

[0175] The working principle of the electronic device in this embodiment can be found in the relevant steps of the aforementioned task processing method, and will not be repeated here.

[0176] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0177] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0178] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0179] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A task processing method, comprising: Obtain processor status data and task status data for each graphics processor, wherein the task status data represents the status of the task currently being processed by the corresponding graphics processor. If it is determined that the processing resources of the first graphics processor are insufficient based on the processor status data and the task status data, and it is determined based on the processor status data that there is a graphics processor with idle processing resources, then the task to be migrated is determined from the tasks processed by the first graphics processor based on the task status data and the processor status data. The task context data of the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor. The task context data includes data that the processing of the task to be migrated depends on. If the task context data of the first virtual address and the task context data of the second virtual address meet the migration conditions, the remaining unprocessed data corresponding to the task to be migrated is migrated from the first graphics processor to the second graphics processor based on the virtual address of the remaining unprocessed data, so that the task to be migrated can continue to be processed on the second graphics processor.

2. The method according to claim 1, further comprising: Obtain the task requirement data for the newly added task; Based on the task requirement data and the processor status data of each graphics processor, the new task is assigned to the third graphics processor for processing.

3. The method according to claim 1, wherein the method for determining the second graphics processor comprises: A second graphics processor is determined from among a plurality of graphics processors other than the first graphics processor based on at least one of the hardware characteristic parameters of each graphics processor, processor status data, task status data of the task to be migrated, and task data type.

4. The method according to claim 1, wherein determining the task to be migrated based on the task status data and the processor status data in the task processed by the first graphics processor includes: Based on the task status data and the processor status data, determine the task migration cost and task migration benefit for each task processed by the first graphics processor; Based on the task migration cost and the task migration benefit, the task to be migrated is determined among the tasks processed by the first graphics processor. The task migration cost is related to the time taken to migrate the corresponding task, and the task migration benefit is related to the expected reduction in processing time of the corresponding task after migration.

5. The method according to claim 1, wherein the task context data of the first virtual address and the task context data of the second virtual address satisfy the migration condition, including: The difference between the task context data of the first virtual address and the task context data of the second virtual address is less than a threshold, or the task context data of the first virtual address and the task context data of the second virtual address are the same; The task context data includes task initialization parameters, task progress data, task cache data, and dependency data.

6. The method according to claim 1, wherein migrating the remaining unprocessed data from the first graphics processor to the second graphics processor based on the virtual address of the remaining unprocessed data corresponding to the task to be migrated comprises: It is determined that the remaining unprocessed data corresponding to the task to be migrated will be migrated from the current third virtual address to the fourth virtual address; The third physical address of the remaining data to be processed in the video memory of the first graphics processor is determined based on the third virtual address; The fourth physical address corresponding to the fourth virtual address is determined in the video memory of the second graphics processor; The remaining data to be processed is transferred from the third physical address to the fourth physical address.

7. The method according to claim 1, further comprising: Regularly save task backup data for each of the graphics processors' processed tasks; In the event that any of the graphics processors is in an abnormal state, the tasks processed by the abnormal graphics processor are assigned to the graphics processors that are not in an abnormal state, based on the task backup data.

8. The method according to claim 1, wherein the processor status data includes at least one of the following: video memory usage, computing core usage, temperature, task execution efficiency, and the number of tasks being processed; The task status data includes at least one of the following: task identifier, task processing progress, task resource requirements, task priority, task submission time, and task deadline.

9. The method according to claim 8, wherein determining that the first graphics processor has insufficient processing resources based on the processor state data and the task state data includes at least one of the following: If the difference between the video memory usage rate and the upper limit of video memory usage is less than a first threshold, it is determined that the processing resources of the first graphics processor are insufficient. If the difference between the core utilization rate and the core utilization limit is less than the second threshold, it is determined that the processing resources of the first graphics processor are insufficient. If the average task waiting time determined based on the task processing progress is greater than the third threshold, it is determined that the processing resources of the first image processor are insufficient.

10. An electronic device comprising a memory, a central processing unit, and a plurality of graphics processors; The memory is used to store computer programs; The plurality of graphics processors are used to process tasks; The central processing unit is used to execute the computer program to perform: Obtain processor status data and task status data for each graphics processor, wherein the task status data represents the status of the task currently being processed by the corresponding graphics processor. If it is determined that the processing resources of the first graphics processor are insufficient based on the processor status data and the task status data, and it is determined based on the processor status data that there is a graphics processor with idle processing resources, then the task to be migrated is determined from the tasks processed by the first graphics processor based on the task status data and the processor status data. The task context data of the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor. The task context data includes data that the processing of the task to be migrated depends on. If the task context data of the first virtual address and the task context data of the second virtual address meet the migration conditions, the remaining unprocessed data corresponding to the task to be migrated is migrated from the first virtual address corresponding to the first graphics processor to the second virtual address corresponding to the second graphics processor, so as to continue processing the task to be migrated on the second graphics processor.

Images