A heterogeneous computing power dynamic scheduling method for scientific computing tasks

Abstract

The present application relates to the technical field of heterogeneous computing resource scheduling, and discloses a heterogeneous computing power dynamic scheduling method for scientific computing tasks. A task submission file, a CPU running queue state, a GPU running queue state, a device memory state and a device-to-device data transmission state are read, and a data object write pass, an actual resident device, a write closure state and an unfinished reading relationship are registered; continuous reuse resident data is identified according to the unfinished reading relationship and a continuous execution segment is formed; after a to-be-executed subtask reaches a migration observation starting boundary and a target device meets an admission condition, a preset copy is established on the target device with the write pass; a migration switching boundary is formed according to the unfinished reading relationship and the preset copy closure state, the source device is withdrawn, the target device is submitted, and the execution device is switched, and the source data residence is released according to the subsequent reading state. The present application is used for reducing repeated cross-device data transmission and reloading caused by dynamic migration of scientific computing tasks.

Description

Technical Field

[0001] This invention relates to the field of heterogeneous computing resource scheduling technology, specifically a method for dynamic scheduling of heterogeneous computing power for scientific computing tasks. Background Technology

[0002] Scientific computing tasks typically involve computational processes with continuous data dependencies, such as matrix operations, grid computing, and iterative solutions. Different subtasks of the same task have different requirements for the computing power, storage space, and data access bandwidth of computing units such as CPUs and GPUs. Existing heterogeneous computing platforms usually allocate the subtasks to be executed to the corresponding computing units based on the current load of the computing units, the historical execution time of the tasks, and the availability of resources, and reschedule the tasks when the resource load changes.

[0003] In iterative scientific computing, intermediate computational data generated by completed subtasks typically resides in the original computing unit's memory or GPU memory and is continuously read by subsequent dependent subtasks. Existing dynamic scheduling methods, when reallocating subsequent subtasks, do not adequately consider the location of this intermediate computational data and its subsequent reuse relationships. This can easily lead to related, sequentially executed subtasks being assigned to different computing units, resulting in redundant data movement and reloading. When the time spent on additional data transfer exceeds the computational waiting time reduced by rescheduling, it can actually prolong the actual completion time of the scientific computing task. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for dynamic scheduling of heterogeneous computing power for scientific computing tasks, thereby solving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for dynamic scheduling of heterogeneous computing power for scientific computing tasks, comprising the following steps:

[0006] S1. Read the task submission file, CPU run queue status, GPU run queue status, device memory status, and inter-device data transfer status of the scientific computing task. Based on the subtask input data object, output data object, and device execution completion signal, register the data object write generation, actual resident device, write closure status, and incomplete read relationship.

[0007] S2. Based on the data object that was written and closed and its incomplete reading relationship, the data object that is still continuously read by multiple subsequent subtasks is registered as continuously reused resident data, and a continuous execution segment is formed from the position where the continuously reused resident data is first read to the position where the last incomplete reading is completed, and the resident device corresponding to the continuous execution segment is registered.

[0008] S3. When the actual waiting time of the subtask to be executed in the resident device reaches the migration observation start boundary corresponding to the migration admission table, and the target device continuously meets the conditions of execution entry available, device memory acceptable and device state stable, a pre-set copy of the continuously multiplexed resident data that has been written closed in the continuous execution segment is established in the target device with the same writing generation, and the original data object in the source device is kept valid.

[0009] S4. Based on the incomplete reading relationship of continuously reused resident data and the closed state of the same write generation preset copy in the target device, a migration switching boundary is formed. After the migration switching boundary is established, the source device queue status and the target device status are reread. After the source device returns the successful withdrawal receipt of the sub-task to be executed, the corresponding sub-task and the same write generation input data object are submitted to the target device.

[0010] S5. Start execution receipt registration and switch execution devices with the target device returning subtask. Keep the source device data objects with incomplete read relationships still resident, and release the source device data objects with incomplete read relationships that have been closed. When the data object write generation changes, cancel the valid state of the corresponding target preset copy. When the target device has not generated an execution receipt, restore the execution entry of the source device's pending subtask.

[0011] Furthermore, S1 includes:

[0012] Establish the reading relationship between data objects and subsequent subtasks based on the input data object number and output data object number of the subtask in the task submission file;

[0013] When a data object enters a new write process, a corresponding write generation is added;

[0014] After the device completes the execution signal, if the actual byte length written to the output data object does not exceed the output reserved byte length, the device exception code is empty, and the corresponding device memory object is effectively allocated, the current write generation is registered as a write closed state, and the device number holding the corresponding device memory object is registered as the actual resident device.

[0015] If a subsequent subtask has not yet completed reading the data of the current write generation, the corresponding read relationship is registered as an incomplete read relationship.

[0016] Furthermore, S2 includes:

[0017] Read the incomplete read relationships of the data object in the current write generation;

[0018] When the number of incomplete read relationships is greater than one, the corresponding data object will be registered as continuously reused resident data;

[0019] The start position of the continuous execution segment is the position where the continuously reused resident data is first read by the subsequent subtask, and the end position of the continuous execution segment is the position where the last unfinished reading of the continuously reused resident data is completed.

[0020] When the read positions of multiple consecutively reused resident data overlap, the earliest initial read position to the latest incomplete read position form the same continuous execution segment, and the device where the consecutively reused resident data resides in the segment is registered as a resident device.

[0021] Furthermore, the migration admission table in S3 is established according to the source device type, target device type, data size range, and subtask execution entry category;

[0022] Perform a full migration test on the same entry and record the duration from when the source device submits the data to when the target device begins executing the verification subtask.

[0023] The longest duration in a complete migration experiment under the same table entry is registered as the migration observation start boundary.

[0024] Furthermore, the specific conditions for the target device in S3 to continuously meet the requirements of available execution entry point, acceptable device memory, and stable device state are as follows:

[0025] The executable device type registered in the task submission file is consistent with the target device type, and the target device driver has been loaded into the current subtask execution entry point;

[0026] The target device’s currently allocable memory, after deducting the device’s reserved space, is greater than the memory requirement formed by the actual byte length of continuously reused resident data and the reserved byte length of the current subtask output.

[0027] The target device has no device error codes or device reset signals in three consecutive status acquisition windows, and the device memory remains acceptable in all three status acquisition windows.

[0028] Furthermore, the specific steps for establishing a pre-built copy in S3 for the same write generation are as follows:

[0029] Read the data object number, current write generation, source device memory object, and actual byte length of continuously multiplexed resident data that is in the write closed state within the continuous execution segment;

[0030] A pre-set memory object is created on the target device, and the corresponding data object is written into the pre-set memory object through the inter-device data transmission interface;

[0031] After the transmission is completed, when the number of bytes transmitted is equal to the actual byte length of the data object, the transmission error code is empty, and the write generation of the source data object remains unchanged, the corresponding data in the target device is registered as a preset closed copy with the same write generation.

[0032] The original data objects in the source device remain valid during the preset period.

[0033] Furthermore, the migration and switching boundaries in S4 are formed according to the following rules:

[0034] When the continuous execution segment reaches the last unfinished reading position of the continuously reused resident data, the current subtask position is registered as the migration switching boundary;

[0035] When the continuous execution segment has not yet reached the last incomplete read position, the continuously reused resident data with incomplete read relationships after the current position is verified item by item. When the corresponding data objects have all formed a pre-set closed copy of the same write generation on the target device and there is no incomplete write action of the corresponding write generation on the source device, the current subtask position is registered as the migration switching boundary.

[0036] Furthermore, S4 also includes:

[0037] After the migration and switching boundary is formed, the source device's run queue is read, and if it is confirmed that the current subtask is still in the pending execution state, a request to withdraw the license is submitted to the source device.

[0038] After the source device returns a successful withdrawal receipt, verify the write generation and readable status of all input data objects of the current subtask in the target device;

[0039] After all input data objects are compiled into readable data consistent with the current valid write generation, the current subtask, the target device memory object corresponding to the input data object, and the output data object are submitted to the target device execution queue.

[0040] Furthermore, S5 includes:

[0041] After the target device returns the start execution receipt corresponding to the current subtask, the device switchover effective time is recorded.

[0042] The source device remains resident when there are still incomplete read relationships for the current write generation of the source device data object. When the incomplete read relationship is closed and there are no pending subtasks referencing the current write generation in the source device run queue, the corresponding source device data object is released from resident status.

[0043] When the write generation of the source data object changes after the pre-set copy of the same write generation is formed, the valid state of the corresponding pre-set copy of the same write generation is revoked.

[0044] If the target device does not return a confirmation receipt for the start of the current subtask within the start confirmation boundary, and provided that the source device remains normal and the data object write generation of the source device remains unchanged, the pending entry point of the current subtask in the source device is restored.

[0045] Compared with existing technologies, this invention provides a method for dynamic scheduling of heterogeneous computing power for scientific computing tasks, which has the following beneficial effects:

[0046] 1. By reading the task submission file, CPU run queue status, GPU run queue status, device memory status, and inter-device data transfer status of scientific computing tasks, and based on the data object write generation, actual resident device, write closure status, and incomplete read relationships, continuously reused resident data is identified and formed into continuous execution segments. After the subtask to be executed reaches the migration observation start boundary and the target device meets the acceptance conditions, a pre-set copy with the same write generation is first established on the target device. Then, a migration switching boundary is formed based on the incomplete read relationships and the pre-set copy closure status. The execution device switch is completed with the source device withdrawal success receipt and the target device start execution receipt. This avoids disrupting the resident continuity of intermediate computing data by changing the subtask execution position solely based on the CPU and GPU real-time load. It reduces the repeated cross-device data transfer and reloading after continuous subtask migration, ensuring that the task execution position adjustment is consistent with the subsequent reuse relationship of intermediate data. This reduces the situation where the actual completion time of scientific computing tasks increases due to the migration of new data transfer time exceeding the reduced queuing waiting time.

[0047] 2. During device switching, the original data objects in the source device remain valid, and the retention or release of the source device is determined based on the incomplete reading relationship of the data objects. When the write generation of the data object changes, the valid state of the corresponding preset copy in the target device is revoked. When the target device has not generated a subtask start execution receipt, the execution entry of the source device is restored. This makes data preset, task withdrawal, target execution, source data release and abnormal rollback form a continuous device execution chain, reducing the number of times old write generation data is called by subsequent subtasks, the premature release of source data when the target device has not actually taken over the task, and the round-trip migration caused by short-term load changes, thus improving the data continuity and execution stability of the task scheduling process in heterogeneous computing nodes. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the process of a heterogeneous computing power dynamic scheduling method for scientific computing tasks according to the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Example: Figure 1A method for dynamic scheduling of heterogeneous computing power for scientific computing tasks is presented, including the following steps:

[0051] S1. Read the task submission file, CPU run queue status, GPU run queue status, device memory status, and inter-device data transfer status of the scientific computing task. Based on the subtask input data object, output data object, and device execution completion signal, register the data object write generation, actual resident device, write closure status, and incomplete read relationship.

[0052] S2. Based on the data object that was written and closed and its incomplete reading relationship, the data object that is still continuously read by multiple subsequent subtasks is registered as continuously reused resident data, and a continuous execution segment is formed from the position where the continuously reused resident data is first read to the position where the last incomplete reading is completed, and the resident device corresponding to the continuous execution segment is registered.

[0053] S3. When the actual waiting time of the subtask to be executed in the resident device reaches the migration observation start boundary corresponding to the migration admission table, and the target device continuously meets the conditions of execution entry available, device memory acceptable and device state stable, a pre-set copy of the continuously multiplexed resident data that has been written closed in the continuous execution segment is established in the target device with the same writing generation, and the original data object in the source device is kept valid.

[0054] S4. Based on the incomplete reading relationship of continuously reused resident data and the closed state of the same write generation preset copy in the target device, a migration switching boundary is formed. After the migration switching boundary is established, the source device queue status and the target device status are reread. After the source device returns the successful withdrawal receipt of the sub-task to be executed, the corresponding sub-task and the same write generation input data object are submitted to the target device.

[0055] S5. Start execution receipt registration and switch execution devices with the target device returning subtask. Keep the source device data objects with incomplete read relationships still resident, and release the source device data objects with incomplete read relationships that have been closed. When the data object write generation changes, cancel the valid state of the corresponding target preset copy. When the target device has not generated an execution receipt, restore the execution entry of the source device's pending subtask.

[0056] S1 is specifically implemented as follows:

[0057] After a scientific computing task enters the current heterogeneous computing node, the task submission file provides the task number, subtask number, subtask execution entry point, input data object number, output data object number, preceding subtask number, subsequent subtask number, executable device type, actual byte length of the input data object, and reserved byte length of the output data object. The data object number is established when the scientific computing task forms matrix blocks, grid blocks, iterative state blocks, and boundary data blocks. The same data object retains the same number throughout the current task's lifecycle. When the same data object is written again, the data object number is not changed; instead, the writing generation is increased by one unit under the original data object number to distinguish the data content formed by the two writes.

[0058] After the task submission file is loaded, an incomplete read relationship is established based on whether the output data object number of the current subtask matches the input data object number of the subsequent subtask. When the subsequent subtask has not yet obtained the device start execution receipt, the corresponding read relationship remains in an incomplete state. After the subsequent subtask completes the data reading of the current write generation and obtains the device execution completion signal, the corresponding read relationship is closed.

[0059] The CPU run queue status is directly read from the operating system task queue interface, including the currently running subtask number, the subtask number to be executed, and the queue entry time registered by the scheduling control terminal for each subtask to be executed. The GPU run queue status is directly read from the GPU driver interface, including the currently running subtask number, the subtask number to be executed, the task submission time, the device completion interrupt, and the device exception code. The device memory status is directly read from the CPU memory allocation object and the GPU video memory allocation object, including the device number, memory object handle, actual allocated byte length, and currently allocable byte length. The inter-device data transfer status is directly read from the inter-device direct storage access interface, including the source device number, target device number, data object number, number of bytes submitted for transfer, number of bytes completed for transfer, transfer completion interrupt, and transfer exception code.

[0060] After the CPU run queue interface, GPU driver interface, and device direct storage access interface return their statuses, the scheduling control terminal registers the return time of each interface status according to the same monotonically increasing clock reference. The interface status whose return time falls within the current status acquisition window is written into the current acquisition result, and the interface status whose return time reaches the start time of the next status acquisition window is written into the next acquisition result, so that different interface statuses are ordered according to the same clock reference of the scheduling control terminal.

[0061] In one engineering embodiment, the status acquisition period is 20 milliseconds. This period comes from the continuous trial run of the operating system run queue refresh interval, GPU driver queue refresh interval, and direct memory access interface refresh interval before the target heterogeneous computing power node is deployed. The longest stable refresh interval among the three interfaces is less than 10 milliseconds. At the same time, the execution duration of the shortest portable subtask in the target task is more than 120 milliseconds. Therefore, using 20 milliseconds can obtain multiple complete status acquisition results during the normal execution of a portable subtask.

[0062] Other hardware nodes do not use a fixed 20 milliseconds interval, but instead form node scheduling configuration values ​​according to the same trial operation criteria before deployment. When a data object begins to be written, the corresponding write generation is marked as write in progress; after the device completes the interrupt, the device currently executing subtask number, output data object number, actual written byte length, and device exception code are checked. If the device exception code is empty, the actual written byte length is not greater than the output reserved byte length, and the corresponding CPU memory object or GPU video memory object still maintains valid allocation, the current write generation is marked as write closed, and the actual CPU number or GPU number holding the valid memory object is registered as the actual resident device.

[0063] When a device completes an interrupt but the actual written byte length is zero, the device memory object is invalid, or the device exception code is not empty, a write closure state is not formed. Subsequent subtasks that depend on the current data object continue to wait, without entering subsequent migration processing. If a collection window fails to obtain the complete state from the CPU queue, GPU queue, and inter-device transmission interface, that collection window will not trigger a new migration action. If there are gaps in the interface state for three consecutive collection windows, the current task will stop adding new migrations, and subtasks that are already in execution will continue to execute on the current device until the three interfaces return three complete collection windows consecutively. This forms the data foundation for S2 with a clear write generation, actual resident device, and incomplete read relationships.

[0064] S2 is specifically implemented as follows:

[0065] Only data objects in S1 that have already formed a write closure state are processed. Data objects in the write in progress state, output exception state, or whose corresponding device memory objects have expired are not included in the continuous reuse relationship determination. For each data object with a write closure, all incomplete read relationships corresponding to the current write generation are read, and subsequent read subtasks are arranged according to the subtask dependency order registered in the task submission file.

[0066] When the current write generation retains only one incomplete read relationship, the data object is treated as single-read resident data and no continuous reuse resident data state is established; when the current write generation has two or more incomplete read relationships, the data object is registered as continuously reuse resident data. The number of two or more is determined based on the number of incomplete read relationships corresponding to the current write generation in the task submission file.

[0067] The position where continuously reused resident data is first read by a subsequent subtask is taken as the start position of the continuous reuse interval of the data object, and the position of the subtask corresponding to the last incomplete read relationship of the current write generation is taken as the end position of the continuous reuse interval.

[0068] After a subsequent subtask completes its read, it closes the corresponding incomplete read relationship. The end position of the continuous reuse interval is updated synchronously forward along with the remaining incomplete read relationships. The corresponding continuous reuse interval ends after all read relationships of the current write generation are closed. When a subtask reads multiple continuously reused resident data simultaneously, it obtains the continuous reuse interval for each data object. When multiple continuous reuse intervals overlap in task dependency order, they extend from the earliest start position to the last end position of all overlapping continuous reuse intervals to form a continuous execution segment.

[0069] When new continuously reused resident data begins to be read before the end of the current continuous execution segment, the end position of the continuous execution segment is extended to the last incomplete read position of the newly added continuously reused resident data. When the first read position of a new data object is after the end position of an existing continuous execution segment, a new continuous execution segment is formed. Through this process, continuous execution segments are formed based on the data object write closure state and incomplete read relationships, and are updated synchronously as the incomplete read relationships change.

[0070] The resident device corresponding to a consecutive execution segment is determined based on the actual resident device of the consecutively multiplexed resident data within the segment. When all consecutively multiplexed resident data within a segment resides on the same CPU or the same GPU, the actual device number is registered as the resident device corresponding to the consecutive execution segment. When the consecutively multiplexed resident data within a segment is distributed across different devices, the current consecutive execution segment is registered as a distributed resident state, and the actual resident device, actual byte length, and remaining unread count for each data object are retained.

[0071] Before proceeding to subsequent processing in the distributed resident state, check all input data objects of the currently pending subtask, including the correspondence between devices that already have valid copies of the same write generation and data objects that need to be read by multiple subsequent subtasks. Only when a device simultaneously holds the continuously reused resident data required by the currently pending subtask, and the current write generation of this data is valid, is the device registered as the resident device of the current continuous execution segment.

[0072] If no device meets the condition, the current continuous execution segment remains dispersed and does not perform the normal dynamic migration in S3. It will be re-checked after the subsequent subtask read relationship changes or the data copy status changes. After the continuous execution segment is established, each segment saves the start subtask number, end subtask number, corresponding continuous reuse resident data number, current write generation of each data object, remaining unread quantity of each data object, and corresponding resident device number.

[0073] When the read relationship is closed, the end position of the continuous execution segment is updated synchronously; when a new write occurs to the data object, the continuous reuse relationship corresponding to the original write generation will no longer continue to the new write generation. The continuous reuse relationship is re-established after the new write generation re-forms the write closure in S1.

[0074] In an engineering scenario, after GPU0 completes the mesh region update, it forms the eighth write generation of data object D240. The task submission file indicates that the subsequent three subtasks will consecutively read the eighth write generation of D240. Therefore, D240 is registered as continuously reused resident data. The first read position to the third incomplete read position in the three subsequent subtasks form a continuous execution segment, and GPU0 is registered as a resident device. After the second read subtask is completed, the remaining incomplete read relationship becomes one, and the end position of the continuous execution segment is updated to the position of the last read subtask. Through the above rules, S3 is provided with a well-defined continuous execution segment, resident device, and continuously reused resident data set.

[0075] S3 is specifically implemented as follows:

[0076] After the subtask to be executed is written into the waiting queue of the resident device registered by S2, the time when the subtask enters the waiting queue is registered by the scheduling control terminal as the waiting start time. The actual waiting duration is formed by subtracting the waiting start time from the current status acquisition window reading time. When the resident device returns the subtask start execution receipt, the waiting timer stops immediately and the normal migration check is canceled.

[0077] The migration observation starting boundary is determined by the device migration test before the current heterogeneous computing power node is deployed and written into the migration admission table. The migration test is conducted according to the source device type, target device type, data size range, and subtask execution entry category. The data size range is formed based on the actual byte length distribution of intermediate data objects during the trial operation of the target scientific computing task.

[0078] A project node divides the data size into five intervals: no more than 64 megabytes, more than 64 megabytes but no more than 256 megabytes, more than 256 megabytes but no more than 1 gigabyte, more than 1 gigabyte but no more than 4 gigabytes, and more than 4 gigabytes. This interval only corresponds to the data distribution of the current node's trial operation and is not used as a fixed parameter for other nodes. Ten complete migration tests are continuously executed for each device's transmission path and data size interval. The source device's transmission submission time, the target device's completion of reception time, the target memory object creation completion time, and the target device's start time for executing the verification subtask are recorded. The longest complete test among the ten normal tests is recorded as the migration observation start boundary of the current entry.

[0079] The ten times are derived from the coverage results of short-term fluctuations in the transmission between devices during the deployment verification of the current node. The first five times still have newly added samples with longer durations. After reaching ten times, the longest duration of the corresponding interval remains stable. Other nodes form their own number of tests and boundary values ​​according to the same verification criteria.

[0080] During operation, the executable device type corresponding to the subtask to be executed in the task submission file is read, and the device running queue status, device memory status and device status of the device under the device type other than the currently resident device are read; devices whose execution entry has been loaded and whose current allocable memory meets the memory requirements of this operation are registered as candidate target devices, and the candidate target devices are continuously verified by the status acquisition window.

[0081] If only one candidate target device continuously meets the device state stability condition, that device is registered as the target device. If multiple candidate target devices continuously meet the device state stability condition, the entry time of the earliest-entered subtask in each candidate target device's run queue is compared, and the candidate target device with the latest entry time is registered as the target device. If the entry times are the same, the target device is determined in ascending order of device number. After the target device is determined, the corresponding migration admission entry is found based on the actual total number of bytes of data to be migrated in the current continuous execution segment, the source device number, the target device number, and the execution entry category.

[0082] The target device execution entry is available when the executable device type registered in the task submission file is consistent with the target device type, and the target device driver has returned a successful loading status for the corresponding execution entry. The device memory is acceptable when the target device's currently allocable memory, after deducting the device reserved space in the node scheduling configuration table, is still greater than the current memory requirement formed by the actual total byte length of the continuously reused resident data to be preset and the reserved byte length of the current subtask output.

[0083] The reserved space for the device is derived from the measured maximum values ​​of the space occupied by the driver context and execution context under no-load, single-task, and concurrent task states before the target device is deployed, and then registered upwards according to a memory allocation granularity. The stability of the target device is confirmed using continuous status acquisition windows. In one engineering implementation, three consecutive complete status acquisition windows are used. Only when there are no device error codes, device reset signals, or transmission channel blockages within any of the three windows, and the allocatable memory meets the current memory requirements, does the target device enter the stable state condition. If any window fails to meet the requirements, the continuous count is reset to zero. The three windows are derived from the verification results of the instantaneous queue occupancy duration during the current node's trial operation being concentrated within two acquisition cycles.

[0084] Once all conditions are met, the continuously multiplexed resident data that is still in the write-closed state is read from the continuous execution segment. For each data object, the data object number, current write generation, source device memory object handle, and actual byte length are read. An independent pre-configured memory object is created on the target device, and the corresponding data is then transferred through the inter-device direct storage access channel. After the pre-configuration begins, the original data objects in the source device are not released; the original memory objects in the source device remain effectively allocated, ensuring that the resident device still has the conditions to continue execution before the current migration has officially switched over.

[0085] After the target device returns a transmission completion interrupt, it checks the number of bytes transferred, the transmission error code, and the current write generation of the source data object. If the number of bytes transferred equals the actual byte length registered at the start of the preset, the transmission error code is empty, and the write generation of the source data object remains unchanged, the target device copy is marked as a preset closed copy of the same write generation. During the preset period, if the source data object enters a new write generation, the current target copy immediately loses its valid state and will not enter the preset closure even if a subsequent transmission completion interrupt occurs.

[0086] In the event of a transmission failure, the target copy's valid state is revoked, while the original data objects and atomic task queue states of the source device remain unchanged. Thus, before entering S4, the source device maintains the executable conditions for the continuous execution segment, and the target device only generates valid pre-set data that has undergone write generation verification.

[0087] S4 is implemented as follows:

[0088] The migration and switching boundary only applies to the continuous execution segments formed by the target device and the pre-set copy of the write generation that have been established in S3. During task execution, the incomplete read relationships of continuously reused resident data in the continuous execution segment are read item by item, and the write generation and pre-set closure status of each corresponding pre-set copy of the target device are read at the same time.

[0089] When the current continuous execution segment has reached the last incomplete read position of its continuously reused resident data, the continuous reuse relationship is closed at the current position, forming a migration switching boundary; when the continuous execution segment has not yet reached the last read position, the target device copy is verified item by item for the continuously reused resident data that still maintains an incomplete read relationship after the current position. If the data objects that still need to be read have the same write generation as the current valid data of the source device and are in a preset closed state, and the source device does not have any write actions for these write generations that have not yet been completed, the current position is also registered as a migration switching boundary.

[0090] When a migration / switching boundary is formed, the subtask number corresponding to the boundary, the related continuously reused resident data number, the valid write generation of each data object, the target device's preset memory object handle, and the boundary formation clock time are registered synchronously. If the write generation of any related data object changes, the target preset copy becomes invalid, or the target device returns an exception code, the migration / switching boundary that has been formed but has not yet been switched over is immediately revoked and returns to the S3 state for verification.

[0091] The source device running queue is a queue of tasks to be executed maintained by the heterogeneous computing power scheduler before the device execution entry point. After a subtask is formally submitted to the corresponding device execution interface, it exits the queue of tasks to be executed. Subtasks that have already entered the device execution stage do not enter the withdrawal process.

[0092] After the migration / switching boundary is established, the subtask is not immediately submitted to the target device. Instead, the source device's run queue and the target device's status are read again. If the source device's run queue returns that the currently pending subtask has entered the execution state, the migration / switching is canceled, and the current subtask continues to execute on the source device. If the source device still returns to the pending execution state, a permission withdrawal request is submitted to the source device's queue interface.

[0093] The permission withdrawal request carries the task number, subtask number, and current queue entry time. The source device's queue interface only returns a successful withdrawal receipt if the subtask has not yet entered the device execution phase. If the source device returns that execution has started, withdrawal failed, or the corresponding subtask does not exist in the queue, the current subtask will not be resubmitted to the target device. In cases where the subtask does not exist in the queue and there is no device start execution receipt, the queue status will be written to pending verification and re-verified in the next complete status acquisition window.

[0094] After the source device returns a successful withdrawal receipt, the current subtask loses its pending execution permission from the source device, but the original input data objects in the source device remain validly allocated. At this point, all input data objects of the current subtask are re-verified. Input data that has formed a pre-defined closed copy of the same write generation directly references the pre-defined memory object on the target device. Other input data that has not participated in continuous reuse of the pre-defined copy but has been written closed are transmitted to the target device according to the current valid write generation of the data object. After the target device returns readable confirmation for all input data objects of the same write generation, the current subtask is submitted to the target device queue.

[0095] The submission uses the existing subtask execution entry point from the task submission file, and associates it with the current subtask number, target device number, input data object number, target memory object handle for each input data object, write generation, output data object number, and output reserved byte length, without changing the subtask dependency order of the scientific computing task itself. The target device memory condition is checked again before formal submission. If the currently allocatable memory decreases and falls below the actual input data and output reserved space requirements, target device submission is stopped, unused target output reserved space is released, and the current subtask's execution entry point is restored to the source device queue if the source device is healthy and the source data objects are still valid. If an exception occurs during inter-device transmission, target device execution is also not submitted.

[0096] In an engineering scenario, the continuous execution segment on GPU0 corresponds to the eighth write generation of D240. GPU1 has already formed a pre-closed copy of the eighth write generation of D240. After subsequent continuous readings of data from D240 have all formed readable objects of the same generation on GPU1, the current position forms a migration switching boundary. The system reads the GPU0 queue again to confirm that the current subtask is still in the pending execution position. After the GPU0 driver returns a successful withdrawal receipt, it submits the current subtask and the input objects of the same write generation in GPU1 to GPU1. Thus, the execution device switching in S4 is based on the joint establishment of the data reuse boundary and the device queue receipt.

[0097] S5 is implemented as follows:

[0098] After S4 submits the current subtask to the target device, the current migration first enters the handover confirmation state. It cannot directly register the device handover execution based on the successful reception of the target device queue. After the target device driver returns a start execution receipt that matches the current task number and subtask number, the start execution receipt time registered by the scheduling control terminal corresponding to the start execution receipt is used as the effective time of the device handover execution, and the actual execution device of the current subtask is updated to the target device.

[0099] Before the target device starts executing the confirmation receipt arrives, the original input data objects in the source device remain effectively allocated to prevent the loss of data required for rollback before the target device task has actually started. The target device start confirmation boundary is derived from the device acceptance test completed synchronously when S3 establishes the migration admission table. In one engineering embodiment, the longest duration from receiving the target queue to the device starting execution in the ten normal task submissions at the corresponding execution entry point is increased by one status acquisition cycle as the target device start confirmation boundary; the additional acquisition cycle is used to cover the case where the driver start execution confirmation receipt falls within an adjacent status window.

[0100] If the target device returns a start execution receipt within this boundary, the handover will proceed. If no start execution receipt is returned within this boundary, the target device queue status, device error code, and device reset signal will be read. If the target queue still retains the current subtask and the device is not abnormal, a complete status acquisition window will continue to be observed. If the current subtask disappears from the target queue and no start execution receipt is received, the target device returns an error code, or the target device resets, the current migration will fail and the handover will proceed.

[0101] After a failed switchover, the health status of the source device and the write generation of the source data object are verified. If the source device has no device abnormalities and the current write generation of the original input data object is consistent with that when S4 was withdrawn, the current subtask is resubmitted to the source device queue, the pending execution entry of the source device is restored, and the unused output space and invalid preset copy corresponding to the current subtask in the target device are released from occupation after obtaining the device driver release receipt.

[0102] If the source device experiences a device malfunction, the source device's pending entry point recovery will not be executed. Instead, the current migration will be registered as an execution failure, and the existing device fault recovery process for scientific computing tasks will be initiated. Once the target device begins execution, the source device's data objects will not be uniformly released.

[0103] If the incomplete read relationships of the current write generation of the corresponding data object are read one by one, and there are still relationships that need to be read by subsequent subtasks of the source device, the original data object of the source device remains resident in the actual memory or video memory, and the target device copy and the source device copy are registered as dual-resident data of the same write generation; when all incomplete read relationships of the current write generation are closed on the source device side, and there are no pending subtasks referencing the write generation in the source device queue, a release request is submitted to the source device memory management interface, and the source device resident is released after obtaining the memory object release completion acknowledgment.

[0104] When either device writes new data during dual residency, the write generation of the newly written data object increases by one unit, and the copy of the old write generation still held by the other device is no longer used as input data for the new generation. When subsequent subtasks require reading the new write generation, they are only allowed to reference data from the device holding the currently valid write generation; if execution needs to continue on the other device, the data transfer and closure verification for the same write generation must be re-executed.

[0105] Before the formal switchover, if a change in the write generation of the source data object is detected in the target pre-built copy formed by S3, its valid status is immediately revoked. After the formal switchover, when a new write occurs, a new residency relationship is established according to the new write generation, and the old copy is not marked as a valid synchronous copy. To avoid short-term load fluctuations causing the same batch of continuously reused residency data to be moved back and forth between the CPU and GPU, after the formal migration takes effect, the continuously reused residency data brought into the target device during this migration that has not yet completed its final read is used as the execution device maintenance boundary. During the period when these data still have incomplete read relationships, the corresponding continuous execution segments will not be migrated back to the source device due to normal queue fluctuations. When the target device experiences a device malfunction, this restriction is lifted and fault recovery begins.

[0106] After all the continuously reused resident data brought in during the migration has been read for the last time, the execution device retain boundary is released, and the next continuous execution segment re-enters the scheduling process according to the S1 to S4 states. A dynamic scheduling loop is completed by the target device start execution receipt, the source device memory release receipt, the actual write generation of the data object, and the incomplete read relationship, so that the device execution state and data resident state of the previous migration action can serve as direct inputs for the processing of the next continuous execution segment.

[0107] In this embodiment, the scheduling control process can be executed by the scheduler in the heterogeneous computing power node. The scheduler completes the above-mentioned status reading, data transmission, task submission, task cancellation and data object release operations through the operating system task queue interface, device driver interface, device memory management interface and inter-device data transmission interface.

[0108] The above content is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamic scheduling of heterogeneous computing power for scientific computing tasks, characterized in that, Includes the following steps: S1. Read the task submission file, CPU run queue status, GPU run queue status, device memory status, and inter-device data transfer status of the scientific computing task. Based on the subtask input data object, output data object, and device execution completion signal, register the data object write generation, actual resident device, write closure status, and incomplete read relationship. S2. Based on the data object that was written and closed and its incomplete reading relationship, the data object that is still continuously read by multiple subsequent subtasks is registered as continuously reused resident data, and a continuous execution segment is formed from the position where the continuously reused resident data is first read to the position where the last incomplete reading is completed, and the resident device corresponding to the continuous execution segment is registered. S3. When the actual waiting time of the subtask to be executed in the resident device reaches the migration observation start boundary corresponding to the migration admission table, and the target device continuously meets the conditions of execution entry available, device memory acceptable and device state stable, a pre-set copy of the continuously multiplexed resident data that has been written closed in the continuous execution segment is established in the target device with the same writing generation, and the original data object in the source device is kept valid. S4. Based on the incomplete reading relationship of continuously reused resident data and the closed state of the same write generation preset copy in the target device, a migration switching boundary is formed. After the migration switching boundary is established, the source device queue status and the target device status are reread. After the source device returns the successful withdrawal receipt of the sub-task to be executed, the corresponding sub-task and the same write generation input data object are submitted to the target device. S5. Start execution receipt registration and switch execution devices with the target device returning subtask. Keep the source device data objects with incomplete read relationships still resident, and release the source device data objects with incomplete read relationships that have been closed. When the data object write generation changes, cancel the valid state of the corresponding target preset copy. When the target device has not generated an execution receipt, restore the execution entry of the source device's pending subtask.

2. The method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 1, characterized in that, S1 includes: Establish the reading relationship between data objects and subsequent subtasks based on the input data object number and output data object number of the subtask in the task submission file; When a data object enters a new write process, a corresponding write generation is added; After the device completes the execution signal, if the actual byte length written to the output data object does not exceed the output reserved byte length, the device exception code is empty, and the corresponding device memory object is effectively allocated, the current write generation is registered as a write closed state, and the device number holding the corresponding device memory object is registered as the actual resident device. If a subsequent subtask has not yet completed reading the data of the current write generation, the corresponding read relationship is registered as an incomplete read relationship.

3. The method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 1, characterized in that, S2 include: Read the incomplete read relationships of the data object in the current write generation; When the number of incomplete read relationships is greater than one, the corresponding data object will be registered as continuously reused resident data; The start position of the continuous execution segment is the position where the continuously reused resident data is first read by the subsequent subtask, and the end position of the continuous execution segment is the position where the last unfinished reading of the continuously reused resident data is completed. When the read positions of multiple consecutively reused resident data overlap, the earliest initial read position to the latest incomplete read position form the same continuous execution segment, and the device where the consecutively reused resident data resides in the segment is registered as a resident device.

4. The method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 1, characterized in that, The migration admission table in S3 is established according to the source device type, target device type, data size range, and subtask execution entry category; Perform a full migration test on the same entry and record the duration from when the source device submits the data to when the target device begins executing the verification subtask. The longest duration in a complete migration experiment under the same table entry is registered as the migration observation start boundary.

5. A method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 4, characterized in that, The specific conditions for the target device in S3 to continuously meet the requirements of execution entry point availability, device memory capacity, and device state stability are as follows: The executable device type registered in the task submission file is consistent with the target device type, and the target device driver has been loaded into the current subtask execution entry point; The target device’s currently allocable memory, after deducting the device’s reserved space, is greater than the memory requirement formed by the actual byte length of continuously reused resident data and the reserved byte length of the current subtask output. The target device has no device error codes or device reset signals in three consecutive status acquisition windows, and the device memory remains acceptable in all three status acquisition windows.

6. The method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 5, characterized in that, The specific steps for establishing a pre-built copy of the same generation in S3 are as follows: Read the data object number, current write generation, source device memory object, and actual byte length of continuously multiplexed resident data that is in the write closed state within the continuous execution segment; A pre-set memory object is created on the target device, and the corresponding data object is written into the pre-set memory object through the inter-device data transmission interface; After the transmission is completed, when the number of bytes transmitted is equal to the actual byte length of the data object, the transmission error code is empty, and the write generation of the source data object remains unchanged, the corresponding data in the target device is registered as a preset closed copy with the same write generation. The original data objects in the source device remain valid during the preset period.

7. A method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 6, characterized in that, The migration and switching boundaries in S4 are formed according to the following rules: When the continuous execution segment reaches the last unfinished reading position of the continuously reused resident data, the current subtask position is registered as the migration switching boundary; When the continuous execution segment has not yet reached the last incomplete read position, the continuously reused resident data with incomplete read relationships after the current position is verified item by item. When the corresponding data objects have all formed a pre-set closed copy of the same write generation on the target device and there is no incomplete write action of the corresponding write generation on the source device, the current subtask position is registered as the migration switching boundary.

8. A method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 7, characterized in that, S4 also includes: After the migration and switching boundary is formed, the source device's run queue is read, and if it is confirmed that the current subtask is still in the pending execution state, a request to withdraw the license is submitted to the source device. After the source device returns a successful withdrawal receipt, verify the write generation and readable status of all input data objects of the current subtask in the target device; After all input data objects are compiled into readable data consistent with the current valid write generation, the current subtask, the target device memory object corresponding to the input data object, and the output data object are submitted to the target device execution queue.

9. A method for dynamic scheduling of heterogeneous computing power for scientific computing tasks according to claim 8, characterized in that, S5 include: After the target device returns the start execution receipt corresponding to the current subtask, the device switchover effective time is recorded. The source device remains resident when there are still incomplete read relationships for the current write generation of the source device data object. When the incomplete read relationship is closed and there are no pending subtasks referencing the current write generation in the source device run queue, the corresponding source device data object is released from resident status. When the write generation of the source data object changes after the pre-set copy of the same write generation is formed, the valid state of the corresponding pre-set copy of the same write generation is revoked. If the target device does not return a confirmation receipt for the start of the current subtask within the start confirmation boundary, and provided that the source device remains normal and the data object write generation of the source device remains unchanged, the pending entry point of the current subtask in the source device is restored.

Images