A computer multi-GPU cooperative scheduling-based video memory fragmentation integration method

Abstract

The application relates to the technical field of cooperative scheduling, and particularly discloses a graphics memory fragmentation integration method based on computer multi-GPU cooperative scheduling. The method comprises the following steps: extracting graphics memory allocation records of the multi-GPU, evaluating physical continuity when the next micro-batch arrives, and obtaining a blockage deterioration area; extracting data moving time consumption required for combining discrete graphics memory blocks in the blockage deterioration area, judging whether there is a fragmentation arrangement time gap in the target GPU based on difference analysis; if there is, locating an upstream sending node; generating a back pressure adjustment instruction of the micro-batch based on the arrangement time gap; generating an anti-overflow interleaving segmentation strategy based on the back pressure adjustment instruction, and dividing micro-batch data to be sent by the upstream into multiple asynchronous transmission sub-blocks; and cross-receiving the asynchronous transmission sub-blocks while driving the target GPU to execute segmented fragmentation combination. The application quantizes the space-time contradiction and applies back pressure, which is beneficial to reducing the spatial fault of graphics memory allocation.

Description

Technical Field

[0001] This invention relates to the field of collaborative scheduling technology, and specifically to a method for consolidating fragmented video memory based on multi-GPU collaborative scheduling in computers. Background Technology

[0002] With the development of deep learning models, multi-GPU collaborative pipelined parallelism has become the mainstream architecture for distributed training. In pipelined parallelism, in order to hide communication latency and improve the throughput of computing units, the global training batch is usually divided into multiple micro-batches and executed interleaved between different nodes.

[0003] However, in such high-concurrency scheduling scenarios that heavily rely on multiple micro-batches, pipeline parallelism has strict computational timing spans. The activation values ​​generated during the forward computation phase of micro-batches are long-lived tensors that must remain resident in GPU memory to await consumption during the backpropagation phase several ticks later. Meanwhile, the temporary workspace required for various temporary operators during this period consists of extremely short-lived data. The frequent alternation of long-lived and short-lived GPU memory blocks in physical addresses cuts the originally contiguous GPU memory space into numerous discrete physical holes, forming a lifecycle-degrading zone.

[0004] Because pipeline parallelism has microsecond-level clock synchronization requirements, the physical movement of large-capacity memory blocks not only occupies device bus bandwidth, but its transmission time often far exceeds the idle bubble time naturally generated in pipeline scheduling. If the target GPU forcibly initiates physical fragmentation consolidation, the excessively long movement time will inevitably break through the bubble boundary, blocking the normal reception of the next micro-batch and triggering an avalanche-like wait across the entire link.

[0005] Therefore, this invention provides a method for consolidating fragmented video memory based on multi-GPU collaborative scheduling in a computer. Summary of the Invention

[0006] The purpose of this invention is to provide a method for consolidating fragmented video memory based on multi-GPU collaborative scheduling in a computer, so as to solve the aforementioned background problems.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A method for consolidating fragmented video memory based on multi-GPU collaborative scheduling in a computer includes the following steps:

[0009] Extract memory allocation records for forward and backward computation of each micro-batch in the multi-GPU pipeline, and analyze the differences in the lifecycle of memory blocks to obtain the residence time of fragments; evaluate the physical continuity when the next micro-batch arrives based on the residence time to obtain the stagnation and degradation zone of each GPU lifecycle.

[0010] Extract the data transfer time required for merging discrete memory blocks within the stagnant degradation zone, and combine it with the current pipeline scheduling cycle to obtain the natural idle time of the target GPU. Perform difference analysis based on the transfer time and the natural idle time, and determine whether there is a time gap for defragmentation in the target GPU.

[0011] If there is a defragmentation time gap, trace the upstream data dependency link that causes the target GPU memory usage and locate the upstream sending node of the micro-batch; set the sending delay of the upstream sending node based on the defragmentation time gap, and generate backpressure adjustment instructions for the micro-batch.

[0012] Based on the micro-batch backpressure adjustment command, the accumulated memory increment of the upstream node due to delayed transmission is extracted, and an anti-overflow interleaving splitting strategy is generated based on the fragment topology descriptor of the life cycle hindrance deterioration zone.

[0013] Furthermore, the method for obtaining the aforementioned hindered degradation region is as follows:

[0014] Extract the maximum single tensor memory allocation size;

[0015] By comparing and analyzing the calculated physical continuity length with the maximum single tensor memory allocation, it was determined that the current GPU exhibits physical fragmentation due to insufficient overall capacity but continuous spatial discontinuities.

[0016] If physical fragmentation occurs, obtain the specific physical address segment that is interleaved by forward-active value memory blocks and causes large tensor allocation failure, extract the base address and offset range, and lock it as the current GPU's stifling and degraded area.

[0017] Furthermore, the process of performing the aforementioned difference analysis is as follows:

[0018] The time taken for data migration and the natural idle time are obtained, and the time difference is calculated by algebraic subtraction to obtain the timing deviation.

[0019] Logical determination of the target GPU's sorting window state based on timing deviation;

[0020] If the timing deviation is greater than zero, it is determined that there is a spatiotemporal misalignment in the target GPU. The absolute value of the timing deviation is then extracted and converted into a quantitative time compensation index, which is defined as the fragmentation cleanup time gap.

[0021] Furthermore, the process of obtaining the time consumed by the data migration is as follows:

[0022] Read the total amount of data that needs to be physically rearranged and the real-time throughput load rate within the preset sliding time window;

[0023] Based on real-time throughput load rate, construct a bandwidth attenuation weighting coefficient;

[0024] Obtain the theoretical read / write bandwidth, multiply the peak theoretical read / write bandwidth by the bandwidth attenuation weighting coefficient, and calculate the effective available bandwidth.

[0025] The dynamic transmission time under the current bus environment is obtained by calculating the ratio of the total data transfer volume to the effective available bandwidth.

[0026] Call the runtime performance analysis tool to extract the statistical average of the kernel function scheduling latency for starting memory copy instructions in the previous scheduling cycle;

[0027] The data migration time is obtained by algebraically superimposing the dynamic transmission time with the statistical average.

[0028] Furthermore, the method for generating the back pressure regulation command is as follows:

[0029] Extract the physical starting address and byte length of each discrete memory block within the target GPU's lifecycle stagnation and degradation zone;

[0030] Based on the distribution relationship between physical starting address and byte length, a fragment topology descriptor for the target GPU physical memory hole size sequence is constructed;

[0031] An out-of-band control channel is established based on the physical device network address of the upstream sending node;

[0032] Obtain the sending delay, encode and concatenate the globally unique sequence number of the next micro-batch to be executed, the sending delay, and the fragment topology descriptor to construct a data packet for cross-node handshake;

[0033] The data packets from the cross-node handshake are encapsulated into standard remote procedure call requests and sent to the upstream sending node as micro-batch backpressure adjustment instructions.

[0034] Furthermore, the process of locating the upstream sending node is as follows:

[0035] Read the physical device's network address and pipeline stage identifier to determine whether the source device is strictly located in the previous level sequence of the target GPU in the logical computation flow;

[0036] If it is located in the previous sequence, the physical device network address of the source device and its corresponding communication port are identified as the upstream sending node.

[0037] Furthermore, the method for generating the anti-overflow interleaving splitting strategy is as follows:

[0038] Read the incremental video memory and compare it with the remaining video memory threshold. If it is determined that there is a risk of local memory overflow, parse the fragment topology descriptor and extract the byte ladder of discrete video memory blocks that the target GPU can release in a single physical relocation operation.

[0039] Based on the byte ladder, a time-series model of the available space increment of the target GPU in the future reconstruction process is established;

[0040] The complete micro-batch data to be sent from upstream is non-equally divided and mapped according to the release capacity of each step in the available space incremental time series model, generating a boundary truncation index that corresponds one-to-one with the single-transfer capability of the target GPU.

[0041] The boundary truncation index and the corresponding asynchronous communication state machine are encapsulated to generate an anti-overflow interleaving splitting strategy for controlling the overlapping execution of multiple streams.

[0042] Furthermore, the method for obtaining the incremental video memory is as follows:

[0043] Analyze the backpressure adjustment command of the micro-batch and extract the set transmission delay and fragment topology descriptor;

[0044] Based on the time window corresponding to the set transmission delay, extract the calculation operator queue pre-scheduled by the upstream transmission node within that time window;

[0045] Traverse the pre-scheduled computation operator queue, discretely accumulate the temporary workspace required for each computation operator's execution and the estimated physical byte occupancy of the forward activation value, and obtain the accumulated memory increment due to delayed transmission.

[0046] Furthermore, based on the anti-overflow interleaving segmentation strategy, the upstream micro-batch data to be sent is divided into multiple asynchronous transmission sub-blocks. While driving the target GPU to perform unidirectional sliding fragment merging, the asynchronous transmission sub-blocks are continuously and linearly spliced ​​together for reception.

[0047] Furthermore, the method of receiving asynchronous transmission sub-blocks by continuous linear splicing is as follows:

[0048] Control the first asynchronous transfer sub-block to be strictly aligned and fall into the available memory base address that is slid out, and record the physical offset pointer of the end of the current sub-block;

[0049] The task of shifting and receiving asynchronous transmission sub-blocks at the high address end is executed in a loop. The subsequent arriving asynchronous transmission sub-blocks are linearly appended along the physical offset pointer until the complete micro-batch data is continuously physically reconstructed at the low address end.

[0050] Send a command to the upstream node to release the backpressure state, and complete the consolidation of video memory fragments under multi-GPU collaborative scheduling.

[0051] The beneficial effects of this invention are as follows:

[0052] 1. By extracting memory allocation records from forward and backward computations in a multi-GPU pipeline and analyzing lifecycle differences, the residence time of fragments can be calculated. This helps transform static memory monitoring into dynamic space prediction, reducing interference from transient idle illusions caused by frequent allocation and release of short-lived operators. Assessing the physical continuity of the next micro-batch arrival based on residence time helps identify lifecycle stagnation and degradation zones, thus helping to identify spatiotemporal fragmentation clusters constrained by pipeline cycle time and reducing the risk of wasted computing power due to blind global shutdowns for cleanup.

[0053] 2. For the data migration time required to merge discrete memory blocks in the congested and degraded region, the natural idle time of the target GPU is calculated by combining the current pipeline scheduling cycle. The bubble time and migration time are then algebraically subtracted to determine the cleanup time gap. This feature transforms the spatial dimension contradiction of memory allocation into a timing indicator of multi-node collaboration, effectively mitigating the negative impact of pipeline bubble breakdown and congestion caused by large memory block copies. If a time gap exists, the upstream sending node of the micro-batch is located by analyzing the distributed computing graph and cross-device tensor transfer events, which helps reduce the invalid synchronization overhead caused by broadcasting to the entire cluster. Variance analysis of the network heartbeat packet round-trip time series is used to extract the link jitter margin, which is then superimposed with the cleanup time gap to generate backpressure adjustment instructions for the micro-batch. This incorporates the uncertainty of the network physical state into the latency setting, improving the stability of control instructions in complex data center environments.

[0054] 3. Parse the backpressure adjustment instruction and obtain the tensor generation rate of the upstream sending node in the current pipeline parallel stage to extract the memory increment. Compare the memory increment with the reserved safety boundary capacity after deducting context overhead and the remaining memory threshold. When facing overflow risk, generate a non-equal overflow prevention interleaving strategy according to the target GPU's stepped capacity release strategy. This helps protect upstream nodes from the impact of their own memory overflow crashes and improves the stability of global system scheduling.

[0055] 4. Based on an anti-overflow interleaving segmentation strategy, asynchronous transfer sub-blocks are partitioned in upstream memory using zero-copy view technology. This operation only logically partitions continuous tensors, reducing the possibility of secondary physical fragmentation caused by data copying at the source. Simultaneously, a multi-stream concurrent control framework is initiated in the target GPU. A physical move instruction for the first discrete memory block is issued in one device compute stream to release local contiguous space. By listening to the completion event of this instruction, the first asynchronous transfer sub-block is triggered and controlled to fall into the newly released space in the concurrent device communication stream. The linkage mechanism of multi-stream overlap and segmented interleaving helps to reduce the contradiction that receiving new data often disrupts continuity, enabling discrete memory blocks to complete physical address convergence and continuous reconstruction. Attached Figure Description

[0056] The invention will now be further described with reference to the accompanying drawings.

[0057] Figure 1 This is a flowchart of a method for consolidating fragmented video memory based on multi-GPU collaborative scheduling in a computer, according to the present invention.

[0058] Figure 2 This is a flowchart for determining whether a target GPU has a spatiotemporal misalignment contradiction in this invention. Detailed Implementation

[0059] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0060] Example 1:

[0061] like Figure 1 As shown, a method for consolidating fragmented video memory based on multi-GPU collaborative scheduling includes the following steps:

[0062] S1. Extract the memory allocation records of each micro-batch in the multi-GPU pipeline for forward and backward computation, and analyze the differences in the life cycle of memory blocks to obtain the residence time of fragments; evaluate the physical continuity when the next micro-batch arrives based on the residence time to obtain the hindrance and degradation zone of each GPU life cycle.

[0063] The method for extracting memory allocation records from the forward and reverse computation of each micro-batch in the multi-GPU pipeline and analyzing the differences in memory block lifecycles to obtain fragment residence time is as follows:

[0064] In some embodiments, GPU memory allocation and release events are captured in real time by hooking the underlying memory allocator interface of the deep learning framework.

[0065] Extract the video memory allocation record corresponding to each allocation event;

[0066] It should be noted that the video memory allocation record includes: physical base address, allocation capacity, micro-batch index, and the identifier of the computation stage to which it belongs. The computation stage identifier is used to distinguish whether the current allocation behavior belongs to the forward computation stage or the reverse computation stage.

[0067] Obtain the start timestamp of the memory block corresponding to the allocated capacity when it was created, and combine it with the corresponding computation stage identifier to query the expected trigger time of the release instruction corresponding to the memory block in the preset pipeline scheduling timing diagram;

[0068] The theoretical lifespan of the memory block in the GPU memory space is calculated by subtracting the expected trigger time from the start timestamp, and is defined as the residence time of the fragment.

[0069] As those skilled in the art will understand, due to the parallel nature of the GPU pipeline, the memory blocks of activated values ​​generated during the forward computation phase must wait for the corresponding reverse computation phase to be completed before they are released, so their residence time is significantly longer than that of memory blocks used only for temporary operators.

[0070] The method for determining the stalled degradation zone of each GPU's lifecycle by assessing the physical continuity of the next micro-batch arrival based on dwell time is as follows:

[0071] Obtain the current GPU global memory address mapping table, scan the contiguous address range that is currently physically free based on the global memory address mapping table, and mark them as candidate available space;

[0072] Obtain the execution cycle of the current micro-batch and the expected arrival time of the next micro-batch based on the pipeline global clock;

[0073] Traverse the adjacent allocated memory blocks that constitute the physical boundary of each candidate available space, and extract the residence time of the fragments corresponding to the adjacent allocated memory blocks.

[0074] The absolute release time is calculated by summing the residence time of the fragment with the fragment's start timestamp, and it is then determined whether the absolute release time is later than the expected arrival time of the next micro-batch.

[0075] If it is later than the arrival time, it means that during the next micro-batch execution, the adjacent allocated memory blocks that constitute the boundary are still occupied and cannot be physically relinquished.

[0076] Based on the above judgment status, the physical contiguous length of the candidate available space when the next micro-batch arrives is calculated, that is, the size of the largest contiguous physical memory block that the current GPU can directly provide without triggering forced synchronization cleanup is evaluated.

[0077] The method for determining the lifecycle stagnation and degradation zone of each GPU is as follows:

[0078] In some embodiments, the memory allocation metadata of operators that have been completed in the current training round is captured by the monitoring hook at the bottom layer of the framework, and an operator feature history index table is established.

[0079] Based on the operator feature history index table, by querying the historical allocation records that have the same operator label as the next micro-batch to be executed, the corresponding historical memory peak value is extracted as the estimated value, which is defined as the maximum single tensor memory request amount of the operator to be executed in the next micro-batch.

[0080] The calculated physical continuity length is compared and analyzed with the maximum single tensor memory allocation.

[0081] If the physical contiguous length of all candidate available space is less than the maximum single tensor memory allocation, and the current total remaining available memory capacity of the GPU is greater than or equal to the maximum single tensor memory allocation, it is determined that the current GPU has a physical fragmentation phenomenon with sufficient overall capacity but contiguous space discontinuity.

[0082] If physical fragmentation occurs, the base address and offset range of the specific physical address segment that is interleaved by the forward active value memory block with a long residence time and causes the large tensor allocation failure will be extracted and locked as the current GPU life cycle stagnation and degradation area.

[0083] It is understandable that the physical behavior of a specific physical address segment caused by the interleaving of forward activation value memory blocks with long residence time and resulting in the failure of large tensor allocation is as follows: In the pipelined parallel scheduling mode, the activation value tensors generated by the forward computation of micro-batch must reside in physical memory and wait for the reverse computation stage to be completed after several ticks before being released. This type of tensor is the forward activation value memory block with long residence time.

[0084] During the aforementioned forward activation value memory block resides, the short-lifetime temporary workspace allocated by the system for other micro-batch interleaving is quickly released after the operator is executed, leaving multiple discrete free holes in the physical memory.

[0085] For example, when the next micro-batch needs to request physical contiguous video memory for a large tensor of size 2GB, even if the total available capacity of the above-mentioned scattered free holes reaches 5GB, the long-resident forward active value video memory block hard blocks on the physical address, resulting in no single hole having a continuous length exceeding 2GB, thus triggering a video memory allocation failure error. The above-mentioned address range with the total available capacity meeting the standard, but whose physical contiguousness is fragmented by the long-lived tensor, is the specific physical address segment.

[0086] The life cycle stagnation and degradation zone is not a static physical damage zone, but a high-dimensional spatiotemporal fragment aggregation region formed by the alternating placement of long and short life cycle tensors, constrained by pipeline scheduling timing.

[0087] Understandably, the purpose of identifying the lifecycle stagnation and degradation zones for each GPU is to:

[0088] Function 1: Transform static memory monitoring into dynamic timing prediction: By analyzing the differences in the forward and reverse lifecycles of memory blocks, the current physical fragmentation location can be identified, and the true availability of memory space under future pipeline cycles can be predicted, which helps to reduce being misled by the illusion of microsecond-level transient idle time.

[0089] Secondly, it provides a benchmark for subsequent counter-pressure strategies: lifecycle stagnation and degradation zones identify obstacles that hinder the smooth execution of the next micro-batch, which helps reduce the waste of computing power caused by blind global cleanup.

[0090] S2. Extract the data transfer time required for merging discrete memory blocks in the stagnant degradation zone, and obtain the natural idle time of the target GPU in combination with the current pipeline scheduling cycle. Perform difference analysis based on the transfer time and the natural idle time, and determine whether there is a time gap for defragmentation in the target GPU.

[0091] The method for determining the time consumption of data migration required for merging discrete memory blocks within the stagnant and degraded region is as follows:

[0092] In some embodiments, the effective load memory blocks scattered within the lifecycle stagnation degradation zone of the target GPU, whose absolute release time is later than the expected arrival time of the next micro-batch, are extracted as discrete memory blocks to be merged.

[0093] Read the physical byte length corresponding to the discrete video memory block, and sum the physical byte lengths of all discrete video memory blocks to obtain the total amount of data to be moved that needs to be physically rearranged.

[0094] Through the GPU's underlying hardware management interface, the theoretical peak read / write bandwidth of the current memory controller is read, and the real-time throughput load of the device communication bus within a preset sliding time window is collected simultaneously.

[0095] Preferably, the method for calculating the time required to move data to merge discrete memory blocks is as follows:

[0096] Based on real-time throughput load rate, a bandwidth attenuation weighted coefficient is constructed to characterize the degree of bus contention and congestion;

[0097] For example, the method for constructing the bandwidth attenuation weighting coefficient is as follows: periodically collect the average used transmission bandwidth of the device communication bus within a preset sliding time window (the sliding time window is preset to 500 microseconds), and calculate the ratio of the average used transmission bandwidth to the theoretical peak read / write bandwidth to obtain the real-time throughput load rate;

[0098] Extract the real-time throughput load rate and input it into a pre-set multi-segment degradation mapping table for interval matching. The multi-segment degradation mapping table is built based on the bus arbitration physical characteristics of the underlying hardware and divides the load state into light load, medium load and heavy load intervals (the threshold for the light load interval is 0 to 30%, the threshold for the medium load interval is 30% to 80%, and the threshold for the heavy load interval is 80% to 100%).

[0099] If the signal is matched to a lightly loaded region, the preset minimum constant (the preset minimum constant is 0.95) is directly extracted as the bandwidth attenuation weighting coefficient.

[0100] If the match is in the medium load range, the preset bus arbitration penalty factor (the bus arbitration penalty factor is preset to 0.5) is extracted, and the effective loss rate is calculated by multiplying the real-time throughput load rate with the bus arbitration penalty factor; the effective loss rate is subtracted from the normalized baseline value (the normalized baseline value is set to 1.0) to obtain the current bandwidth attenuation weighting coefficient.

[0101] If a match is found in an overloaded region, a preset penalty truncation constant (the preset penalty truncation constant is 0.3) is extracted as a bandwidth attenuation weighting coefficient.

[0102] The effective available bandwidth after eliminating hardware contention loss is calculated by multiplying the theoretical peak read / write bandwidth with the bandwidth attenuation weighting coefficient.

[0103] The dynamic transmission time under the current bus environment is obtained by calculating the ratio of the total data transfer volume to the effective available bandwidth.

[0104] Call the runtime performance analysis tool to extract the statistical average of the kernel function scheduling latency for starting memory copy instructions in the previous scheduling cycle;

[0105] The statistical average of dynamic transmission time and kernel function scheduling delay is algebraically superimposed to obtain the data transfer time, which includes software instruction issuance overhead and physical transmission overhead.

[0106] The method for obtaining the natural idle time of the target GPU by combining the current pipeline scheduling cycle is as follows:

[0107] Extract the end timestamp of the current micro-batch operator being executed by the target GPU under the current pipeline scheduling cycle;

[0108] Obtain the model slice parameters assigned to the next micro-batch by the upstream node of the pipeline. Based on the floating-point operation volume of the model slice parameters and the measured peak computing power of the upstream node, the system combines the historical computing power time mapping table to obtain the pre-computation time of the upstream node.

[0109] Read the network topology matrix between the target GPU and the upstream node, and find the static transmission latency of the current physical link based on the network topology matrix;

[0110] Based on the start timestamp of the upstream node starting to execute the next micro-batch, the system calculates the estimated arrival timestamp of the next micro-batch data to the target GPU receive buffer by accumulating the pre-calculation time and static transmission delay.

[0111] Preferably, the method for obtaining the natural idle time of the target GPU is as follows:

[0112] The theoretical pipeline bubble width of the target GPU between two intensive computing tasks is obtained by taking the difference between the expected arrival timestamp and the end timestamp of the current micro-batch operator.

[0113] Read the safety redundancy duration used to maintain video memory context switching from the operating system kernel parameters;

[0114] Subtracting the safety redundancy time from the theoretical pipeline bubble width, we obtain an absolutely safe and purely memory-free time window, which is defined as the natural idle time.

[0115] The method for determining whether the target GPU has a time gap for defragmentation is based on the difference analysis between the relocation time and the natural idle time:

[0116] The time series deviation is obtained by performing an algebraic difference operation between the calculated data transfer time and the natural idle time.

[0117] like Figure 2 As shown, the logical determination of the target GPU's sorting window state is based on the timing deviation.

[0118] If the timing deviation is less than or equal to zero, it indicates that the natural idle time is sufficient to cover the physical merging process. It is determined that the target GPU can complete the memory closed-loop reorganization without interfering with the pipeline cycle and is marked as a gap-free state.

[0119] If the timing deviation is greater than zero, it indicates that the existing pipeline bubble width is insufficient to accommodate the required physical data transfer process. If a memory copy is forcibly initiated, the bubble will inevitably be broken and block the next micro-batch, indicating that there is a spatiotemporal misalignment contradiction in the target GPU.

[0120] If a spatiotemporal misalignment is determined to exist, the absolute value of the time sequence deviation is extracted and converted into a quantitative time compensation index, which is defined as the time gap for fragment organization.

[0121] Example 2:

[0122] Please see Figure 1 As shown, a method for consolidating fragmented video memory based on multi-GPU collaborative scheduling includes the following steps:

[0123] S3. If there is a fragmentation cleanup time gap, trace the upstream data dependency link that causes the target GPU memory usage and locate the upstream sending node of the micro-batch; set the sending delay of the upstream sending node based on the cleanup time gap, and generate the back pressure adjustment command for the micro-batch.

[0124] The method for tracing the upstream data dependency links that cause the target GPU's memory usage and locating the upstream sending node of the micro-batch is as follows:

[0125] Preferably, the distributed computing graph at the bottom layer of the deep learning framework is extracted, and the cross-device tensor transfer events associated with the next micro-batch to be executed in the target GPU are parsed.

[0126] From the cross-device tensor transfer event, read the global communication rank of the source device responsible for providing the forward activation value and the backward gradient tensor;

[0127] The global communication rank is input into the preset pipeline parallel topology configuration table for hash matching query to obtain the physical device network address bound to the global communication rank and the pipeline stage identifier.

[0128] The preferred method for locating the upstream sending node of a micro-batch is as follows:

[0129] Based on the pipeline stage identifier, determine whether the source device is strictly located in the previous level sequence of the target GPU in the logical computation flow;

[0130] If it is located in the previous sequence, the physical device network address of the source device and its corresponding communication port are identified as the upstream sending node that has a direct data feeding relationship with the target GPU;

[0131] If it is not located, no action will be taken;

[0132] The method for generating micro-batch backpressure adjustment commands, based on setting the transmission delay of the upstream transmitting node according to the consolidation time gap, is as follows:

[0133] Preferably, the method for setting the transmission delay of the upstream sending node based on the consolidation time gap is as follows:

[0134] Extract the round-trip time series of network heartbeat packets between the upstream sending node and the target GPU within a preset historical window;

[0135] Statistical variance analysis was performed on the round-trip time series of network heartbeat packets to calculate the link jitter margin, which characterizes the fluctuation range of the current network state.

[0136] The fragmentation cleanup time gap determined in step S2 is algebraically superimposed with the link jitter margin to calculate the transmission delay that absorbs the network uncertainty risk.

[0137] Preferably, the method for generating micro-batch back pressure regulation commands is as follows:

[0138] Extract the physical starting address and byte length of each discrete memory block within the lifecycle stagnation and degradation zone of the target GPU. Based on the distribution relationship between the physical starting address and byte length, construct a fragment topology descriptor that describes the sequence of physical memory hole sizes of the target GPU.

[0139] An out-of-band control channel is established based on the physical device network address of the upstream sending node;

[0140] It should be noted that the out-of-band control channel avoids the in-band main communication link used for transmitting tensor data;

[0141] The globally unique sequence number of the next micro-batch to be executed, the calculated sending delay, and the fragment topology descriptor are encoded and concatenated to construct a data packet for cross-node handshake.

[0142] The data packets from the cross-node handshake are encapsulated into standard remote procedure call requests and sent to the upstream sending node as micro-batch backpressure adjustment instructions.

[0143] S4. Based on the micro-batch back pressure adjustment command, extract the memory increment accumulated by the upstream node due to delayed transmission, and generate an anti-overflow interleaving splitting strategy based on the fragment topology descriptor of the life cycle blocking degradation zone.

[0144] The method for extracting the accumulated memory increment of upstream nodes due to delayed transmission based on micro-batch backpressure adjustment commands is as follows:

[0145] Preferably, the method for extracting the accumulated memory increment due to delayed transmission from upstream nodes is as follows:

[0146] Analyze the backpressure adjustment command of the micro-batch and extract the set transmission delay and fragment topology descriptor;

[0147] Based on the time window corresponding to the set transmission delay, extract the calculation operator queue pre-scheduled by the upstream transmission node within that time window;

[0148] Traverse the pre-scheduled computation operator queue, discretely accumulate the estimated physical byte occupancy of the temporary workspace and forward activation value required when each computation operator is executed, and obtain the total memory tension generated within the time window.

[0149] The calculated total memory tension is taken as the absolute resource demand that the upstream node will inevitably increase due to the execution of the backpressure instruction. The absolute resource demand is defined as the cumulative increase in video memory due to delayed transmission.

[0150] The method for generating an anti-overflow interleaving splitting strategy based on the fragment topology descriptor of the lifecycle-degraded region is as follows:

[0151] Poll the underlying memory manager of the upstream sending node to obtain the current absolute physical free memory capacity that has not been allocated, and deduct the memory context overhead necessary to maintain the concurrent kernel operation to obtain the remaining memory threshold of the upstream sending node.

[0152] The calculated video memory increment is compared with the remaining video memory threshold. If the video memory increment is greater than or equal to the remaining video memory threshold, it is determined that the upstream sending node faces the risk of local memory overflow due to the execution of backpressure instructions.

[0153] If a local memory overflow risk is identified, the fragment topology descriptor is parsed to extract the byte ladder of discrete memory blocks that the target GPU can release in a single physical relocation operation.

[0154] If it is determined that there is no risk of local memory overflow, the upstream node is directly driven to send regular tensors in complete micro-batch format.

[0155] Based on the byte ladder, a time-series model of the available space increment of the target GPU in the future reconstruction process is established;

[0156] Preferably, the method for constructing the available space incremental time series model is as follows: extract the physical byte length of the discrete memory block that the target GPU can release in a single physical relocation operation, and obtain the data transfer time of the single block corresponding to the relocation;

[0157] The discrete video memory blocks are sorted into queues according to their physical starting addresses from low to high.

[0158] Along the sorted queue, the data transmission time of a single block is calculated in a step-by-step manner to generate a predicted sequence of future time nodes; at the same time, the physical byte length is calculated in a synchronous step-by-step manner to obtain the cumulative release capacity sequence corresponding to the future time node sequence.

[0159] Based on the future time node sequence and the cumulative released capacity sequence, a binary discrete mapping matrix is ​​constructed, and this binary discrete mapping matrix is ​​used as the available space incremental time series model.

[0160] The complete micro-batch data to be sent from upstream is non-equally divided and mapped according to the release capacity of each step in the available space incremental time series model, generating a boundary truncation index that corresponds one-to-one with the single-transfer capability of the target GPU.

[0161] The boundary truncation index and the corresponding asynchronous communication state machine are encapsulated to generate an anti-overflow interleaving splitting strategy for controlling the overlapping execution of multiple streams.

[0162] S5. Based on the anti-overflow interleaving segmentation strategy, the upstream micro-batch data to be sent is divided into multiple asynchronous transmission sub-blocks. While driving the target GPU to perform unidirectional sliding fragment merging, the asynchronous transmission sub-blocks are continuously and linearly spliced ​​and received.

[0163] The method for continuously linearly splicing and receiving asynchronously transmitted sub-blocks while driving the target GPU to perform unidirectional sliding fragment merging is as follows:

[0164] Preferably, the method for performing unidirectional sliding fragment merging and continuous linear splicing reception is as follows:

[0165] Based on the boundary truncation index in the anti-overflow interleaving splitting strategy, the continuous complete micro-batch tensor logic is divided into multiple asynchronous transmission sub-blocks in the video memory of the upstream sending node using zero-copy view technology.

[0166] Initiate a multi-stream concurrent control framework in the target GPU and issue unidirectional sliding instructions to the DMA (Direct Memory Access) engine in a single device compute stream;

[0167] Based on the unidirectional sliding shift instruction, the discrete video memory blocks in the life cycle stagnation and degradation zone are uniformly shifted towards the physical high address end of the video memory space, and then slid outward at the physical low address end to make up continuous available memory base addresses.

[0168] By monitoring the phased completion event of the unidirectional sliding move instruction through hardware synchronization primitives, after confirming that the continuous available memory capacity slid out at the low address end meets the reception requirements of the first asynchronous transmission sub-block, the upstream node is immediately triggered to send the first asynchronous transmission sub-block in the concurrent device communication stream.

[0169] Control the first asynchronous transfer sub-block to be strictly aligned and fall into the available memory base address that is slid out, and record the physical offset pointer of the end of the current sub-block;

[0170] The task of shifting and receiving asynchronous transmission sub-blocks at the high address end is executed repeatedly, and the subsequent arriving asynchronous transmission sub-blocks are strictly appended to the physical offset pointer in a linear manner until the complete micro-batch data is continuously physically reconstructed at the low address end.

[0171] Based on the reconstructed continuous large tensor space and the surviving tensor blocks at high addresses, the global memory addressing mapping table of the target GPU is updated, and a node feedback instruction to release the backpressure state is sent upstream to complete the memory fragmentation consolidation under multi-GPU collaborative scheduling.

[0172] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A method for consolidating fragmented video memory based on multi-GPU collaborative scheduling in a computer, characterized in that, Includes the following steps: Extract memory allocation records for forward and backward computation of each micro-batch in the multi-GPU pipeline, and analyze the differences in the lifecycle of memory blocks to obtain the residence time of fragments. Based on the dwell time, the physical continuity when the next micro-batch arrives is evaluated, and the stagnation and degradation zone of each GPU's lifecycle is obtained. Extract the data transfer time required for merging discrete memory blocks within the stagnant degradation zone, and combine it with the current pipeline scheduling cycle to obtain the natural idle time of the target GPU. Perform difference analysis based on the transfer time and the natural idle time, and determine whether there is a time gap for defragmentation in the target GPU. If there is a defragmentation time gap, trace the upstream data dependency link that causes the target GPU memory usage and locate the upstream sending node of the micro-batch; set the sending delay of the upstream sending node based on the defragmentation time gap, and generate backpressure adjustment instructions for the micro-batch. Based on the micro-batch backpressure adjustment command, the accumulated memory increment of the upstream node due to delayed transmission is extracted, and an anti-overflow interleaving splitting strategy is generated based on the fragment topology descriptor of the life cycle hindrance deterioration zone.

2. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling as described in claim 1, characterized in that: The method for obtaining the aforementioned hindered degradation region is as follows: Extract the maximum single tensor memory allocation size; By comparing and analyzing the calculated physical continuity length with the maximum single tensor memory allocation, it was determined that the current GPU exhibits physical fragmentation due to insufficient overall capacity but continuous spatial discontinuities. If physical fragmentation occurs, obtain the specific physical address segment that is interleaved by forward-active value memory blocks and causes large tensor allocation failure, extract the base address and offset range, and lock it as the current GPU's stifling and degraded area.

3. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 1, characterized in that: The process of performing the difference analysis is as follows: The time taken for data migration and the natural idle time are obtained, and the time difference is calculated by algebraic subtraction to obtain the timing deviation. Logical determination of the target GPU's sorting window state based on timing deviation; If the timing deviation is greater than zero, it is determined that there is a spatiotemporal misalignment in the target GPU. The absolute value of the timing deviation is then extracted and converted into a quantitative time compensation index, which is defined as the fragmentation cleanup time gap.

4. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 3, characterized in that: The process of obtaining the time taken for the data migration is as follows: Read the total amount of data that needs to be physically rearranged and the real-time throughput load rate within the preset sliding time window; Based on real-time throughput load rate, construct a bandwidth attenuation weighting coefficient; Obtain the theoretical read / write bandwidth, multiply the peak theoretical read / write bandwidth by the bandwidth attenuation weighting coefficient, and calculate the effective available bandwidth. The dynamic transmission time under the current bus environment is obtained by calculating the ratio of the total data transfer volume to the effective available bandwidth. Call the runtime performance analysis tool to extract the statistical average of the kernel function scheduling latency for starting memory copy instructions in the previous scheduling cycle; The data migration time is obtained by algebraically superimposing the dynamic transmission time with the statistical average.

5. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 1, characterized in that: The method for generating the back pressure regulation command is as follows: Extract the physical starting address and byte length of each discrete memory block within the target GPU's lifecycle stagnation and degradation zone; Based on the distribution relationship between physical starting address and byte length, a fragment topology descriptor for the target GPU physical memory hole size sequence is constructed; An out-of-band control channel is established based on the physical device network address of the upstream sending node; Obtain the sending delay, encode and concatenate the globally unique sequence number of the next micro-batch to be executed, the sending delay, and the fragment topology descriptor to construct a data packet for cross-node handshake; The data packets from the cross-node handshake are encapsulated into standard remote procedure call requests and sent to the upstream sending node as micro-batch backpressure adjustment instructions.

6. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 4, characterized in that: The process of locating the upstream sending node is as follows: Read the physical device's network address and pipeline stage identifier to determine whether the source device is strictly located in the previous level sequence of the target GPU in the logical computation flow; If it is located in the previous sequence, the physical device network address of the source device and its corresponding communication port are identified as the upstream sending node.

7. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 1, characterized in that: The method for generating the anti-overflow interleaving splitting strategy is as follows: Read the incremental video memory and compare it with the remaining video memory threshold. If it is determined that there is a risk of local memory overflow, parse the fragment topology descriptor and extract the byte ladder of discrete video memory blocks that the target GPU can release in a single physical relocation operation. Based on the byte ladder, a time-series model of the available space increment of the target GPU in the future reconstruction process is established; The complete micro-batch data to be sent from upstream is non-equally divided and mapped according to the release capacity of each step in the available space incremental time series model, generating a boundary truncation index that corresponds one-to-one with the single-transfer capability of the target GPU. The boundary truncation index and the corresponding asynchronous communication state machine are encapsulated to generate an anti-overflow interleaving splitting strategy for controlling the overlapping execution of multiple streams.

8. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 7, characterized in that: The method for obtaining the incremental video memory is as follows: Analyze the backpressure adjustment command of the micro-batch and extract the set transmission delay and fragment topology descriptor; Based on the time window corresponding to the set transmission delay, extract the calculation operator queue pre-scheduled by the upstream transmission node within that time window; Traverse the pre-scheduled computation operator queue, discretely accumulate the temporary workspace required for each computation operator's execution and the estimated physical byte occupancy of the forward activation value, and obtain the accumulated memory increment due to delayed transmission.

9. The method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 1, characterized in that: Based on the anti-overflow interleaving segmentation strategy, the upstream micro-batch data to be sent is divided into multiple asynchronous transmission sub-blocks. While driving the target GPU to perform unidirectional sliding fragment merging, the asynchronous transmission sub-blocks are continuously and linearly spliced ​​together for reception.

10. A method for consolidating fragmented video memory based on multi-GPU collaborative scheduling according to claim 9, characterized in that: The method of receiving asynchronous transmission sub-blocks by continuous linear splicing is as follows: Control the first asynchronous transfer sub-block to be strictly aligned and fall into the available memory base address that is slid out, and record the physical offset pointer of the end of the current sub-block; The task of shifting and receiving asynchronous transmission sub-blocks at the high address end is executed in a loop. The subsequent arriving asynchronous transmission sub-blocks are linearly appended along the physical offset pointer until the complete micro-batch data is continuously physically reconstructed at the low address end. Send a command to the upstream node to release the backpressure state, and complete the consolidation of video memory fragments under multi-GPU collaborative scheduling.

Images