Method and apparatus for managing display memory blocks

Abstract

This application discloses a method and apparatus for managing video memory blocks, belonging to the field of data processing technology. The method includes: acquiring the occupancy status and access activity curves of video memory blocks within multiple consecutive address segments; for each consecutive address segment, performing frequency domain transformation and encoding based on the occupancy status and access activity curves of the video memory blocks within the segment to generate a video memory convolution spectrum characterizing the fragmentation degree and temporal access activity of that segment of video memory; determining candidate migration regions based on the video memory convolution spectra of each consecutive address segment; determining the differential tensors of candidate video memory blocks within the candidate migration regions; performing a comprehensive evaluation based on the differential tensors of all candidate video memory blocks under preset bandwidth and quality of service constraints to generate a video memory migration plan; and calling the underlying interface to execute the batch migration of candidate video memory blocks according to the video memory migration plan.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, specifically relating to a method and apparatus for managing video memory blocks. Background Technology

[0002] With the widespread deployment of large-scale deep learning training and online inference in data centers, GPU clusters have become a core computing infrastructure. In this environment, multi-GPU and multi-tenant hybrid deployments on single machines have become the norm, with a single GPU often simultaneously handling training, inference, and data preprocessing tasks for multiple models. These tasks dynamically allocate and release intermediate tensors, caches, and model weights during operation, resulting in high-frequency, fine-grained, and highly dynamic characteristics in GPU memory allocation and reclamation.

[0003] GPU memory management primarily relies on mechanisms such as allocators, unified memory, and heterogeneous memory management within the driver and runtime, alleviating local memory shortages through page merging or on-demand page migration. However, from a scheduling and operations perspective, existing solutions struggle to perceive the actual memory utilization structure within a single GPU, such as the maximum contiguous free block size, fragmentation distribution, and access activity across different memory regions. While the driver layer possesses some page migration and merging capabilities, these are largely implemented using kernel-adaptive strategies, remaining invisible to upper-layer scheduling. Consequently, they cannot effectively support memory fragmentation management and fine-grained scheduling optimization for multi-tasking and multi-tenant environments.

[0004] Currently, it is not possible to achieve fine-grained management of video memory blocks. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for managing video memory blocks, which can solve the problem that it is currently impossible to achieve fine-grained management of video memory blocks.

[0006] In a first aspect, embodiments of this application provide a method for managing video memory blocks, the method comprising: Obtain the occupancy status and access activity curves of memory blocks within multiple consecutive address segments of GPU memory; For each consecutive address segment, frequency domain transformation and encoding are performed based on the occupancy status and access activity curve of the memory blocks within the consecutive address segment to generate a memory convolution spectrum that characterizes the fragmentation degree and temporal access activity of the memory space in that segment. Candidate migration regions are determined based on the memory convolution spectrum of each consecutive address segment; Determine the differential tensor of candidate memory blocks within the candidate migration region; the differential tensor is used to characterize the changes caused by migrating candidate memory blocks to the target free address; Based on the differential tensors of all candidate memory blocks, a comprehensive evaluation is performed under preset bandwidth and quality of service constraints to generate a memory migration plan; the memory migration plan includes source address, target address and migration length information; Call the underlying interface to perform batch migration of candidate memory blocks according to the memory migration plan; After the migration is completed, the memory occupancy status of the relevant contiguous address segments is updated, and the status information of the corresponding logical memory slices is reported to the cluster scheduling system.

[0007] Secondly, embodiments of this application provide a memory block management device, the device comprising: The acquisition module is used to obtain the occupancy status and access activity curve of memory blocks within multiple consecutive address segments of GPU memory. The generation module is used to perform frequency domain transformation and encoding on each continuous address segment based on the occupancy status and access activity curve of the video memory blocks within the continuous address segment, and generate a video memory convolution spectrum to characterize the fragmentation degree and temporal access activity of the video memory space in that segment. The first determining module is used to determine candidate migration regions based on the memory convolution spectrum of each consecutive address segment; The second determining module is used to determine the differential tensor of candidate memory blocks within the candidate migration region; the differential tensor is used to characterize the changes caused by migrating the candidate memory blocks to the target free address; The evaluation module is used to perform a comprehensive evaluation based on the differential tensors of all candidate memory blocks under preset bandwidth and quality of service constraints, and generate a memory migration plan; the memory migration plan includes source address, target address and migration length information; The migration module is used to call the underlying interface and perform batch migration of candidate memory blocks according to the memory migration plan; The update module is used to update the memory occupancy status of the relevant contiguous address segments after the migration is completed, and to report the status message of the corresponding logical memory slice to the cluster scheduling system.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In the embodiments of this application, the occupancy status and access activity curves of memory blocks within multiple consecutive address segments of GPU memory are obtained. For each consecutive address segment, frequency domain transformation and encoding are performed based on the occupancy status and access activity curves of memory blocks within the consecutive address segment to generate a memory convolution spectrum that characterizes the fragmentation degree and temporal access activity of the memory space in that segment. The high-dimensional, raw occupancy and activity data are compressed and encoded into unified mathematical features, realizing a joint quantitative characterization of the two key attributes of memory fragmentation degree and data hot / coldness. This allows the subsequent system to evaluate and compare the consolidation value of different regions based on clear numerical indicators. Based on the memory convolution spectrum of each consecutive address segment, candidate migration regions are determined, concentrating computational resources and operational risks on the most effective target. The differential tensor of candidate memory blocks within the candidate migration region is determined. The differential tensor is used to characterize the changes caused by migrating candidate memory blocks to the target free address. The differential tensor encapsulates the expected benefits, operational costs, and business risks of a single migration, making the migration value of each candidate block calculable and comparable. Based on the differential tensors of all candidate memory blocks, a comprehensive evaluation is performed under preset bandwidth and quality of service constraints to generate a memory migration plan. The memory migration plan includes source address, target address, and migration length information. It can generate a specific operation sequence that maximizes the overall benefit within the current period, transforming defragmentation from a kernel adaptive behavior that may lead to unstable consequences into a predictable and controllable online optimization service. The underlying interface is called to execute the batch migration of candidate memory blocks according to the memory migration plan. Through planned batch migration of data blocks, the memory layout is physically rearranged as expected, thereby actually merging small fragments and forming larger contiguous free areas, directly achieving the ultimate goal of fragmentation convergence. After the migration is completed, the memory occupancy status of the relevant contiguous address segments is updated, and the status information of the corresponding logical memory slices is reported to the cluster scheduling system, improving resource utilization and task stability at the system level. Attached Figure Description

[0013] Figure 1 This is a flowchart of a video memory block management method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a video memory block management architecture provided in an embodiment of this application; Figure 3 This is a structural diagram of a video memory block management device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0015] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0016] The technical terms used in the embodiments of this application will be introduced below.

[0017] Dynamic Elastic Convergence (DEC) refers to the process by which a system can dynamically and online detect fragmentation within the GPU memory and, without interrupting business operations and adapting flexibly to load changes, continuously converge the memory layout towards a state with less fragmentation and larger contiguous free blocks through planned data migration.

[0018] MemBlock: The smallest statistical and operational unit for managing GPU memory. It is obtained by dividing the linear address space of memory into fixed-size blocks and is the basic granularity for collecting occupancy status, access records, and executing migrations.

[0019] A contiguous memory segment is a logical unit comprised of multiple contiguous memory blocks. It serves as the basic spatial range for constructing memory convolutional spectra and calculating fragmentation and activity scores, thus balancing analytical accuracy and computational overhead.

[0020] Memory convolution spectrum: A feature vector of a contiguous address segment. It is generated by fusing the occupancy status and access activity curves of all memory blocks within this segment, and then extracting the main feature coefficients after frequency domain transformation and encoding. It compactly represents the degree of spatial fragmentation and temporal access activity of this segment of memory.

[0021] Differential tensor (Δ-Tensor): A multi-dimensional evaluation vector constructed for a candidate memory block to be migrated. Its components quantify the expected benefits, costs, and potential risks of migrating the block, and are used to uniformly and quantitatively evaluate the value of a single migration.

[0022] Memory migration plan: A structured list of operation instructions. It contains specific parameters for a series of migration operations to be executed in the current cycle, such as the source memory address, the target memory address, and the migration data length.

[0023] Logical memory slices: From a resource management and scheduling perspective, these are abstractions of physical GPU memory resources. A slice represents a piece of memory resource with specific capacity and attributes, and is the basic unit for task placement and resource allocation by the cluster scheduler. Its status can be described by metrics such as maximum contiguous free block size and average fragmentation score.

[0024] DEC Node Agent: A software module or service deployed on each GPU compute node. It is responsible for executing core DEC logic such as memory state acquisition, memory convolution spectrum construction, differential tensor evaluation, migration plan generation and execution, and is the core executor of the embodiments in this application.

[0025] In response to the problems in related technologies, this application provides a method and apparatus for managing video memory blocks, which can solve the problem that it is currently impossible to achieve fine-grained management of video memory blocks in related technologies.

[0026] The method for managing video memory blocks provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0027] Figure 1 A flowchart illustrating a method for managing video memory blocks provided in an embodiment of this application.

[0028] like Figure 1 As shown, the method for managing video memory blocks may include steps 110-170. This method is applied to a video memory block management device, as detailed below: Step 110: Obtain the occupancy status and access activity curves of memory blocks within multiple consecutive address segments of GPU memory; Step 120: For each consecutive address segment, perform frequency domain transformation and encoding based on the occupancy status and access activity curve of the memory blocks within the consecutive address segment to generate a memory convolution spectrum that characterizes the fragmentation degree and temporal access activity of the memory space in that segment. Step 130: Determine candidate migration regions based on the memory convolution spectrum of each consecutive address segment; Step 140: Determine the differential tensor of the candidate memory blocks within the candidate migration region; the differential tensor is used to characterize the changes caused by migrating the candidate memory blocks to the target free address. Step 150: Based on the differential tensors of all candidate memory blocks, a comprehensive evaluation is performed under preset bandwidth and quality of service constraints to generate a memory migration plan; the memory migration plan includes source address, target address and migration length information. Step 160: Call the underlying interface to perform batch migration of candidate memory blocks according to the memory migration plan; Step 170: After the migration is completed, update the memory occupancy status of the relevant continuous address segments and report the status information of the corresponding logical memory slices to the cluster scheduling system.

[0029] The status information includes the maximum contiguous free block size, fragmentation score, and activity score, which are determined based on the memory convolution spectrum.

[0030] In this context, a memory block refers to the smallest management unit obtained by dividing the linear address space of GPU memory into fixed-size blocks. Its size can be configured according to the granularity of the underlying hardware interface, such as 64KB or 1MB. A contiguous address segment is formed by aggregating multiple memory blocks with adjacent addresses, serving as a logical unit for unified analysis and feature extraction; for example, 32 consecutive memory blocks can be grouped into one segment. The access activity curve is a continuous curve reflecting the recent access frequency of each memory block, generated by using a timing smoothing method to transform the discrete access counts of each memory block over multiple scheduling cycles.

[0031] The memory convolution spectrum is a low-dimensional, dense feature vector generated by jointly analyzing and frequency-domain encoding the occupancy status and activity curves of all memory blocks within a contiguous address segment. This vector compactly represents the spatial fragmentation degree and temporal access activity pattern of that memory segment. The differential tensor is a multi-dimensional evaluation vector constructed for each candidate memory block to be migrated. Its components quantify the expected benefits, required overhead, and potential impact on business performance of migrating the data block. The memory migration plan is a set of instructions containing a series of specific migration operations, generated based on the comprehensive evaluation results. Logical memory slicing is an abstraction of physical memory resources from a scheduling perspective; it represents a contiguous memory address space as a resource unit that can be independently monitored and scheduled.

[0032] Step 110 establishes the ability to observe the micro-state of video memory. This requires continuously collecting two types of information, using video memory blocks as the basic unit: their static allocation status and their dynamic access history. Occupancy status can be obtained by querying the GPU driver or the runtime memory allocation map. Access records can be collected through hardware performance counters or by lightweight instrumentation in the memory management path. To extract stable trends from noisy access data, the raw access counts need to be time-series smoothed, for example, using an exponentially weighted moving average algorithm to convert instantaneous access frequencies into continuous activity values, thus forming activity curves that reflect hot, warm, and cold data areas. One example is that the agent module collects the access count of each video memory block once per second and updates the activity value of each block using an exponential smoothing algorithm with a decay factor of 0.8. After continuous collection and recording for a period of time, the access activity curve of each video memory block is obtained.

[0033] Step 120 directly processes the raw state and curve data, which are high-dimensional and difficult to use directly for evaluating fragmentation and heat. Therefore, a feature extraction method is needed to compress the spatial distribution information and time series information within a continuous address segment into a representative feature vector. Frequency domain transformation and encoding are effective means to achieve this goal. By fusing the spatial sequence and time series and converting it to the frequency domain, the low-frequency components in the frequency domain reflect the macroscopic trend and overall activity level of the state within the segment, while the high-frequency components correspond to local, fragmented abrupt changes. By retaining the main low-frequency coefficients, a memory convolution spectrum that reflects both the overall fragmentation pattern and activity status and is sufficiently compact can be generated. For example, for a segment containing 16 memory blocks, a fused sequence of length 16 is constructed, where the value of each element is determined by the occupancy flag of the corresponding memory block and the normalized activity value; a discrete cosine transform is performed on the sequence, and the first 8 coefficients after the transform are taken to form an 8-dimensional vector, which is the memory convolution spectrum of the segment.

[0034] Step 130 requires selecting regions that are severely fragmented but currently infrequently accessed as priority consolidation targets to achieve high returns and low risks. The memory convolutional spectrum, as the feature vector of a segment, can be used to calculate two key metrics: fragmentation score and activity score. The fragmentation score measures the scattered distribution of free blocks within a segment; a higher score indicates more severe fragmentation. The activity score measures the average access frequency of data within a segment. By setting thresholds, segments with high fragmentation scores (e.g., above 0.7) and low activity scores (e.g., below 0.3) can be selected from all segments; these segments are marked as candidate migration regions. For example, the system calculates the fragmentation and activity scores for each segment based on the convolutional spectrum, then automatically selects all segments that meet the criteria of fragmentation score > 0.7 and activity score < 0.3, identifying the address ranges contained in these segments as candidate migration regions requiring focused attention in the current cycle.

[0035] Step 140: Within the candidate region, the value of migrating each specific memory block needs to be quantitatively evaluated. Differential tensors provide a unified, multi-dimensional evaluation framework. It typically contains three core components: the first component measures the activity and locality gains brought by the migration, such as the positive benefits of moving cold data out of hot areas or moving frequently accessed data to a closer location; the second component measures the migration overhead, i.e., the bandwidth and time costs consumed in moving the data block itself; and the third component measures the quality of service impact, i.e., the negative disturbances that the migration may cause to the latency, throughput, etc., of the application. By estimating these three components for each memory block within the candidate region, a three-dimensional differential tensor can be formed, comprehensively characterizing the advantages and disadvantages of migrating the block. For example, for a candidate memory block in an idle state, its migration overhead component depends on the block size and the current bus bandwidth, while its quality of service impact component may be zero because it is not used by any task.

[0036] Step 150 requires selecting a batch of blocks from numerous candidate blocks for actual migration within the current cycle, from a globally optimal perspective. This first requires synthesizing the differential tensors of each block into a scalar evaluation value, for example, by combining the benefits, costs, and impacts into a single score through weighted summation. Then, under given constraints, the batch of memory blocks with the highest comprehensive scores is selected to ensure maximum consolidation benefits within controllable costs and risks. Finally, specific target free addresses are planned for these selected blocks, and a detailed migration plan containing source addresses, target addresses, and lengths is generated. For example, the system sets the bandwidth budget for this cycle to 800MB. It first calculates the score for each candidate block, then adds the highest-scoring block sequentially to the migration list, accumulating the total data volume of the selected blocks until the total data volume approaches but does not exceed the 800MB budget. Subsequently, it searches for suitable target addresses in the free region for each block in the list and outputs the final migration plan.

[0037] Step 160 translates the decision into action. Based on the entries in the migration plan, the node agent efficiently moves data from the source address to the target address in batches by calling the GPU driver or runtime-provided memory copy interface. During execution, task synchronization or error handling mechanisms may need to be considered to ensure the atomicity and correctness of the data migration and avoid data inconsistencies or access conflicts during the migration process. For example, the agent module submits a plan containing 10 migration operations to the execution engine, which then sequentially calls the underlying API to perform 10 memory data copies, completing the entire batch migration process.

[0038] Step 170: After the migration is complete, the physical layout of the video memory has changed. Local management metadata must be updated to reflect the latest status, and the changes in resource health must be notified to the upper-layer scheduler. The agent module needs to rescan the contiguous address segments whose address ranges have changed and update the occupancy status flags of the video memory blocks within them. More importantly, based on the updated information, it recalculates the key health metrics for each logical video memory slice, including the size of its current largest contiguous free block, the average fragmentation score calculated based on the latest convolutional spectrum, and the average activity score. This status information constitutes a quantitative description of the video memory slice resource quality. After being reported to the cluster scheduling system, it can be used to guide future task scheduling decisions. For example, after the migration is complete, the agent detects that several previously scattered small free blocks within a logical slice have been merged into a single 2GB large free block. It then updates the maximum contiguous free block size of that slice to 2GB, recalculates its fragmentation score, and finally sends this updated status information to the cluster scheduler.

[0039] In one possible embodiment, step 110 may specifically include the following steps: The linear address space of the GPU memory is divided into multiple smallest statistical units of a fixed size, which are called memory blocks; Multiple contiguous memory blocks are aggregated into a single contiguous address segment to obtain multiple contiguous address segments; Collect the occupancy status and historical access records of video memory blocks within each consecutive address segment at a preset granularity; Based on historical access records, determine the access activity curve for each memory block.

[0040] The entire linear address space of GPU memory is divided into units of a fixed size, such as 64KB per unit. This results in a large number of uniform, minimal statistical units, each called a memory block. This fixed size can be configured based on the underlying hardware's memory page size or management granularity.

[0041] To perform region-level analysis with reasonable overhead, multiple contiguous memory blocks need to be further aggregated into a larger logical unit, namely a contiguous address segment. For example, aggregating 32 contiguous memory blocks into a segment allows each segment to manage a 2MB address space. This two-level partitioning preserves fine-grained information at the memory block level while enabling efficient data aggregation and feature calculation at the segment level. Subsequently, the system, at a preset time interval (e.g., once per second), queries the internal state of the GPU driver or intercepts memory allocation calls to collect the current occupancy status of each memory block—whether it is idle or allocated—and records the task or process identifier to which it belongs.

[0042] Simultaneously, historical records of the number of times each memory block was accessed within the same period are collected through hardware performance counters or runtime instrumentation. Finally, based on these time-series recorded historical access records, the access frequency of each memory block is processed by time-series smoothing, for example, by using an exponential decay algorithm to transform the discrete and fluctuating access counts into a smooth and continuous curve. This curve is the access activity curve of the memory block, which intuitively reflects the popularity of the data block over time.

[0043] The step of determining the access activity curve of each memory block based on historical access records may specifically include the following steps: Within each scheduling cycle, the number of times each video memory block is accessed is counted based on historical access records; Based on the number of accesses in the current period and the historical activity value in the previous period, the current activity value of each memory block is updated using an exponential decay smoothing algorithm. The access activity curve for each memory block is determined based on the current activity value updated over multiple scheduling cycles.

[0044] In the specific implementation process of determining the access activity curves of each memory block, the goal is to transform the original, discrete access event sequence into a continuous quantitative indicator that can stably reflect the recent access trend of the data block. The scheduling cycle refers to the fixed time interval at which the system periodically performs memory status collection and updates, for example, once per second; the historical activity value represents the activity evaluation result calculated for each memory block at the end of the previous scheduling cycle, serving as the baseline for this calculation; the exponential decay smoothing algorithm is a time-series data processing method that smooths short-term fluctuations by weighted averaging of historical data, where recent data is given higher weight, and the weight of historical data decays exponentially over time.

[0045] Access count is the most direct and raw signal for measuring the activity level of a data block. The system obtains these raw counts through performance counters or monitoring interfaces, providing basic input for subsequent quantitative analysis. For example, a memory block used to store model weights may be read hundreds or thousands of times during one cycle of forward inference, and the system will record this specific access count.

[0046] Relying solely on access counts from a single cycle can lead to significant fluctuations due to the randomness of task scheduling, failing to distinguish between consistently hot data and occasionally accessed cold data. Exponential decay smoothing introduces a decay factor to balance old and new information. The update formula can be expressed as: New activity value = Decay factor × Old activity value + (1 - Decay factor) × Normalized access count for the current cycle. This mechanism makes the activity value a continuous variable with memory; recent high-frequency accesses will rapidly increase the value, while once access stops, the value will gradually decay over time. For example, setting the decay factor to 0.9, if the historical activity value of a memory block in the previous cycle was 0.5, and high-frequency access is detected in the current cycle with a normalized access contribution of 0.8, then its new current activity value is updated to 0.9 * 0.5 + 0.1 * 0.8 = 0.53. This reflects the upward trend in activity more accurately than simply using data from the current cycle.

[0047] The above statistical and update steps are repeated over multiple scheduling cycles. Based on this series of current activity values ​​generated in chronological order, an access activity curve for each memory block can be formed. By concatenating discrete sampling points along the time dimension, a continuous characterization of the memory block's access pattern over time is created. This curve reveals whether a data block is consistently active, experiences short-term bursts, or remains continuously idle. For example, a memory block whose activity value remains above 0.7 for ten consecutive cycles exhibits a high-level stable curve, clearly identifying it as hot data; while another memory block whose activity value rapidly declines from 0.6 to 0.1 shows a decreasing curve indicating that it is gradually cooling down.

[0048] Exponential smoothing effectively filters out noise from instantaneous access peaks or troughs, generating an activity signal that better represents the true trend and improving the robustness of state perception. The resulting access activity curve provides an accurate and direct basis for distinguishing between hot and cold data and identifying low-activity data blocks that can be safely migrated. It is an important data foundation for the entire memory defragmentation strategy to achieve a balance between benefits and risks.

[0049] For example, within each scheduling cycle Δt, the DEC node agent can maintain an access counter for each MemBlock by cooperating with the GPU driver or runtime. This counts the number of times the MemBlock is read, written, or hit within a given period. Access events can originate from the video memory access statistics interface, unified memory migration callbacks, or lightweight instrumentation of framework layer allocation / release calls.

[0050] To suppress short-term noise in the access sequence and distinguish between "recently accessed frequently" and "accessed once a long time ago," embodiments of this application employ an exponential decay method for temporal smoothing mapping of access counts. For the i-th MemBlock, its activity value at time t... It can be updated using the following recursive method:

[0051] in, Let λ be the activity value of the previous period, λ be the decay coefficient with a value range of (0,1), and norm(·) be the function that normalizes or scales the access count of the current period. By adjusting the value of λ, the influence of historical accesses on the current activity can be controlled. A larger λ emphasizes long-term trends, while a smaller λ is more sensitive to recent changes.

[0052] As the scheduling cycle progresses, A discrete activity curve is generated for the access behavior of each MemBlock over a recent period. Compared to directly using the original access count, the activity curves of the embodiments of this application have the following characteristics: a) It can smooth out occasional single-time access spikes and highlight areas with consistently high access. b) It can distinguish between "newly active" and "historically active but now cooled down" MemBlocks through a decay mechanism; c) Facilitates subsequent aggregation and frequency domain coding of multiple activity curves at the Segment level.

[0053] The embodiments of this application can coarsely classify MemBlocks according to their activity value ranges, and... Blocks above the high threshold are marked as highly active blocks, those below the low threshold and remaining low over multiple periods are marked as inactive blocks or candidate cold blocks, and the rest are considered moderately active blocks. This classification result can directly provide a reference for estimating the activity gain in subsequent differential tensors, and can also be used to limit the activity range of migrated blocks during the candidate region screening stage, avoiding the misselection of currently frequently accessed hot data.

[0054] Through the aforementioned temporal smoothing mapping based on access records, the embodiments of this application transform discrete and noisy memory access events into activity curves that can be continuously updated, easily aggregated, and analyzed, providing a fundamental temporal representation for the construction of segment-level memory convolution spectra and the calculation of activity gain terms in differential tensors.

[0055] In one possible embodiment, step 120 may specifically include the following steps: Map the occupancy status of each video memory block within a contiguous address segment to a space occupancy bitmap; The frequency domain feature coefficients are extracted by performing discrete cosine transform or wavelet transform on the access activity curves of each memory block within a continuous address segment. The memory convolution spectrum is obtained based on the space occupancy bitmap and frequency domain feature coefficients.

[0056] A memory space occupancy bitmap is a binary encoded representation of the spatial distribution of memory blocks within an address segment. It is typically implemented using a sequence, where each bit corresponds to a memory block within the segment; for example, 0 indicates the block is free, and 1 indicates the block is occupied. Frequency domain characteristic coefficients are a set of new values ​​obtained by mathematically transforming data representing temporal activity. These coefficients no longer directly represent activity at a specific point in time, but rather reveal the energy distribution characteristics of the data in the frequency dimension, such as which changes are gradual low-frequency components and which are dramatic high-frequency components.

[0057] To efficiently analyze and calculate the degree of fragmentation in memory space, the discrete occupancy states need to be transformed into a structured form that is easy to perform pattern recognition and statistical operations. A bitmap provides such a concise representation, intuitively reflecting the alternating distribution pattern of free and occupied blocks in the address space. For example, for a segment containing 8 memory blocks, its occupancy state sequence is [occupied, free, occupied, occupied, free, free, occupied, free]. The resulting space occupancy bitmap is an 8-bit binary sequence 10110010, where the order of each bit corresponds to the physical address order of the memory blocks.

[0058] As a time series, the access activity curve contains patterns in the access patterns of the data block. Directly comparing multiple original curves is computationally complex and unintuitive. By transforming to the frequency domain, the shape characteristics of the curve can be converted into a series of coefficients. The low-frequency coefficients capture the overall trend and average activity level of the curve, while the high-frequency coefficients reflect the local fluctuations and abrupt changes. This transformation achieves dimensionality reduction and feature extraction of time-dimensional information.

[0059] For example, performing a discrete cosine transform on a curve consisting of 32 periodic activity values ​​yields 32 transform coefficients. In practice, only the first 8 low-frequency coefficients, representing the main trend, can be retained as the frequency domain characteristics of the activity curve. As another example, analyzing the same activity curve using wavelet transform allows for the acquisition of a set of coefficients characterizing activity fluctuations at different time scales by selecting specific wavelet bases and decomposition levels.

[0060] The memory convolution spectrum needs to simultaneously carry spatial fragmentation information and temporal activity patterns. Therefore, it is necessary to fuse the bitmap features representing spatial information and the frequency domain feature coefficients representing temporal information. A direct approach is to concatenate several statistical features calculated from the bitmap with the frequency domain feature coefficients extracted from each activity curve into a longer one-dimensional vector, and then normalize this vector to standardize its numerical range. This fused and standardized multi-dimensional vector is the memory convolution spectrum for that continuous address segment.

[0061] For example, for a segment containing 16 memory blocks, two spatial statistical features are extracted from its bitmap, and two main frequency domain coefficients are extracted from each of the 16 activity curves in the segment. By concatenating them in order, a 2+16*2=34-dimensional feature vector can be generated, which serves as the unique memory convolution spectrum of the segment.

[0062] For example, each segment consists of a set of consecutive MemBlocks, and embodiments of this application construct a sequence of free / occupied bits within the segment in address order. ∈{0,1}, where j is the index of MemBlock within the Segment, 0 indicates free, and 1 indicates occupied. Based on this, the fragmentation index of the Segment can be defined by statistically analyzing the number and length of consecutive free intervals. Optional definition methods include: a) The ratio of the number of free intervals to the total number of free MemBlocks; b) Average free interval length and maximum free interval length; c) The degree of dispersion of the length distribution of the free interval.

[0063] The above statistical results will be used as one of the input features in the subsequent FragScore calculation.

[0064] To provide a quantifiable fragmentation metric, embodiments of this application define the fragmentation rate of a segment as: Let the number of free MemBlocks within that segment be... The longest consecutive free interval contains 100 MemBlocks. Then the fragmentation rate of this segment is...

[0065] When free blocks are mainly concentrated in a contiguous region near The fragmentation rate approaches 0; when free blocks are cut into a large number of scattered small pieces, much smaller The fragmentation rate is close to 1.

[0066] To simultaneously consider the activity curve in the time dimension and the fragmentation distribution in the spatial dimension, embodiments of this application combine the activity value and idle / occupied bits of each MemBlock within a Segment to form a one-dimensional sequence for frequency domain coding. For the j-th MemBlock within a Segment, the input sequence elements are constructed as follows:

[0067] in, Let be the activity value at time t. For the free / occupied flag of the block, f(·) can be a simple weighted sum or segmented mapping, such as amplifying low activity for free blocks to highlight potentially migrated areas that are "free and low-activity". In one implementation, it can be:

[0068] in, This is the activity value normalized to the range of 0 to 1. When MemBlock is in an idle state... =0, thus obtaining = That is, the lower the activity level, the larger the value of that position in the input sequence, thus highlighting the potential transferable region as "idle and inactive"; when MemBlock is in an occupied state, =1, thus obtaining =0, this position does not participate in the emphasis of idle features in subsequent frequency domain coding.

[0069] For the above input sequence The embodiments of this application employ frequency domain analysis methods such as Discrete Cosine Transform (DCT) or Wavelet Transform to map the state correlation of spatially adjacent MemBlocks to the frequency domain. In one embodiment, a DCT-II transform is performed on a sequence of length M.

[0070] A frequency domain coefficient sequence of the same length M is obtained. Then, only the first K low-frequency coefficients (K is a positive integer much smaller than M, such as 16 to 32) are retained to form the memory convolution spectral vector of this segment:

[0071] Low-frequency coefficients primarily reflect the smooth trend of overall idle / occupied and activity distribution within the segment, while high-frequency coefficients correspond to localized sharp changes and fragmented data. Embodiments of this application achieve a dense, compressed representation of the memory state by retaining the low-frequency components.

[0072] To enhance the comparability between different segments, embodiments of this application can normalize the memory convolution spectral vectors, for example, by standardizing the vectors according to their magnitude or by scaling each coefficient within a range, so that the spectral vectors of different segments are within a uniform numerical range. If necessary, coefficient truncation or smoothing strategies can be used to suppress abnormal peaks and improve robustness in the presence of noise and short-term jitter.

[0073] Through the above fragmentation characterization and frequency domain dense coding, the embodiments of this application compress the original high-dimensional memory state at the MemBlock level into a low-dimensional memory convolution spectrum at the Segment level, providing an efficient feature foundation for subsequent extraction of HotScore / FragScore indicators and rapid screening of candidate regions based on spectral vectors.

[0074] By compressing high-dimensional, heterogeneous raw state data into a relatively low-dimensional, homogeneous feature vector, the computational complexity of subsequent global analysis and comparison is greatly reduced. Second, this feature vector simultaneously embeds information on spatial fragmentation patterns and temporal access patterns, enabling a comprehensive assessment of the health status of a segment of memory by analyzing only this spectral vector. This provides a solid and efficient data foundation for the subsequent accurate localization of candidate migration regions.

[0075] In one possible embodiment, step 130 may specifically include the following steps: Obtain the space occupancy bitmap for each contiguous address segment; Based on the memory convolution spectrum and space occupancy bitmap of each consecutive address segment, calculate the fragmentation score and activity score of the consecutive address segment. A contiguous address range with a fragmentation score higher than the first threshold and an activity score lower than the second threshold is identified as a candidate migration region.

[0076] Fragmentation score is a quantitative metric used to measure the dispersion of free memory blocks within a contiguous address range. A higher score indicates more severe fragmentation; for example, a large number of small free blocks scattered throughout the memory will receive a high score. Activity score is another quantitative metric used to characterize the average access frequency of the data stored in the range. A higher score indicates that the data in the range is more frequently accessed and read / written.

[0077] The memory occupancy bitmap is the most direct basis for calculating fragmentation scores. This bitmap, in the form of a binary sequence, precisely records whether each memory block within a segment is occupied or free, thus providing raw data for analyzing the continuous distribution of free blocks. For example, for a segment consisting of 16 memory blocks, its memory occupancy bitmap might be 1100110011110011, where '0' represents free. This sequence visually demonstrates the intervals and distribution of free blocks.

[0078] A comprehensive judgment requires integrating information from both spatial and temporal perspectives. The fragmentation score primarily relies on the space occupancy bitmap, analyzed by examining the distribution of the number and length of consecutive '0' (free) sequences. A common approach is to first count the total number of free blocks, then identify the longest consecutive free block. The fragmentation score can be designed to be positively correlated with the total number of free blocks and negatively correlated with the longest consecutive free block length, thus awarding a high score even with a large number of small fragments. The activity score, on the other hand, mainly utilizes the memory convolution spectrum, as it encodes the frequency domain characteristics of the activity curves of each memory block within that segment, particularly the low-frequency components which are correlated with the overall energy level.

[0079] By extracting coefficients representing the overall energy from the convolutional spectrum, and then performing weighted summation and normalization, it can be mapped to an activity score between 0 and 1. For example, when calculating the fragmentation score of a segment, the system analyzes its bitmap 1100110011110011 and finds that it has 6 free blocks, but they are divided into 3 groups, with the longest consecutive free block being only 2 blocks. Therefore, a high fragmentation score of 0.8 is calculated. At the same time, energy features are extracted from the memory convolutional spectrum of this segment, and their values ​​are found to be low, thus a low activity score of 0.2 is calculated.

[0080] A dual-threshold filtering strategy is employed to achieve precise selection. The first threshold sets a threshold for the severity of fragmentation, ensuring that only areas truly requiring remediation are considered. The second threshold sets an upper limit for activity, ensuring that frequently accessed hotspot data areas are avoided, preventing migration operations from interfering with high-performance services. By combining these two conditions with logic, the system can automatically identify ideal remediation targets that have significant issues but minimal impact. For example, setting the first threshold to 0.7 and the second threshold to 0.4. The system iterates through all segments, marking segments with a fragmentation score of 0.8 (above 0.7) and an activity score of 0.2 (below 0.4) as candidate migration areas. Segments with a fragmentation score of 0.6 (below 0.7) or an activity score of 0.5 (above 0.4) are excluded.

[0081] For example, for the memory convolution spectral vector of each segment The activity score of this application can be estimated in the following ways: a) Select several coefficients in the spectral vector that are related to the overall energy (e.g., DC component). (and several low-order frequency components), calculate their weighted sum or energy value, which serves as the original activity index of the segment; b) Combine the activity curves of each MemBlock mentioned above to normalize the original metrics and ensure that the HotScore value falls within a preset range (e.g., 0 to 1), where a larger value indicates more frequent overall access within the segment.

[0082] In one implementation, HotScore can be defined as:

[0083] in, , For adjustable weights, The number of low-order frequencies involved in the calculation is denoted by norm(·), which is the normalization function.

[0084] The fragmentation score comprehensively considers the statistical characteristics of the free / occupied bitmaps within a segment and the components reflecting local fluctuations in the memory convolution spectrum. The fragmentation rate of this segment is denoted as [fragment rate]. Based on this, embodiments of this application combine the fragmentation rate with frequency components in the memory convolution spectrum that are related to local variations to construct a fragmentation score, FragScore, for example:

[0085] in, These are the weighting coefficients. This is the frequency component index range used to reflect local fluctuations. Through normalization, the FragScore is made to fall within the range of 0 to 1. The higher the score, the more severe the fragmentation within the segment and the greater the potential for cleanup.

[0086] After obtaining the HotScore and FragScore of all segments, embodiments of this application use a combination of thresholding and ranking to filter candidate regions for the current period. Typically, the following rules can be set: a) Only consider segments with FragScore higher than the preset fragmentation threshold, and filter out areas with relatively light fragmentation and limited sorting benefits; b) For segments that meet the fragmentation criteria, further limit the activity range based on the HotScore. For example, select segments with low to medium activity levels and avoid selecting hotspot areas with extremely high activity levels to reduce the impact of migration on service QoS. c) Among the segments that meet the above conditions, sort them in descending order by FragScore or by a combination of HotScore and FragScore, and select the top few as candidate regions for priority processing in this cycle.

[0087] For the selected candidate segments, embodiments of this application further select MemBlocks that meet certain conditions within the segment to form a candidate MemBlock set. For example, idle or low-activity MemBlocks are given priority, as well as MemBlocks located at the edge of idle intervals that, after being sorted, are conducive to forming larger continuous idle areas; if necessary, a small number of moderately active MemBlocks but with low QoS sensitivity can also be included in the candidates to complete the splicing of continuous idle areas within a local range.

[0088] Through the HotScore / FragScore calculation and candidate region selection steps described above, the embodiments of this application quickly locate severely fragmented but moderately active local regions from a large number of segments within the global memory address space. Within these regions, a candidate MemBlock set of appropriate size is constructed, providing a high-yield, low-risk set of objects to be migrated for subsequent differential tensor evaluation and migration plan generation. This effectively reduces the complexity of subsequent optimization calculations and improves memory fragmentation convergence efficiency. Thus, automated and index-based focusing from the global memory space to local problem areas is achieved, avoiding manual judgment or blind selection and improving operational efficiency. By introducing activity scoring as a key screening dimension, business access frequency is incorporated into the decision-making system, enabling defragmentation operations to proactively avoid performance-sensitive areas. This fundamentally reduces the risk of business performance fluctuations due to data migration, laying the foundation for subsequent safe and effective online defragmentation.

[0089] In one possible embodiment, step 140 may specifically include the following steps: Based on the space occupancy bitmap within the continuous address segment corresponding to the candidate migration region, select video memory blocks that are in an idle state or whose activity value is lower than the third threshold from the candidate migration region as candidate video memory blocks. For candidate memory blocks, assess the changes in access activity, data migration overhead, and impact on service quality involved in their migration to determine the differential tensor.

[0090] The third threshold, used as a specific operational parameter for filtering low-activity blocks, is a value between 0 and 1, used to distinguish the hotness or coldness of data, for example, 0.3.

[0091] Based on the existing space occupancy bitmap within the continuous address segment corresponding to the candidate migration region, two types of video memory blocks are selected from this region as candidate video memory blocks: the first type is blocks that are currently in an idle state; the second type is blocks that are occupied, but whose current activity value is lower than a preset third threshold.

[0092] Migration operations prioritize data blocks that will free up space or reconfigure the layout after migration, minimizing the impact on currently running tasks. Free blocks contain no valid data, so migrating them incurs minimal data transfer overhead and business impact. For occupied blocks, it's crucial to ensure their data is "cold," meaning it's accessed infrequently, minimizing the potential performance disruption from migrating such data. For example, the system sets a third threshold of 0.3. In the bitmap of a candidate migration region, the system identifies all free blocks marked '0' and simultaneously queries the latest activity value of each occupied block within that region, filtering out occupied blocks with activity values ​​below 0.3. These two types of blocks together constitute the candidate memory block set for this round of fine-grained operations.

[0093] For each candidate memory block selected, it is necessary to comprehensively evaluate the three key dimensions involved in its migration: changes in access activity, data migration overhead, and impact on service quality, and determine the corresponding differential tensor for that block.

[0094] The difference tensor needs to serve as a unified mathematical representation to encompass the expected benefits, inherent costs, and potential risks of this migration operation. Assessing changes in access activity aims to predict improvements in access latency or locality of reference after the migration, as the data is placed closer to visitors or in a more suitable location—an estimate of positive benefits. Assessing data migration overhead involves calculating the memory bandwidth and time consumed by performing the data transfer itself—a necessary cost. Assessing the impact on service quality analyzes the potential negative disturbances the migration operation may cause to key metrics such as response time and throughput of the application—a risk that needs to be mitigated.

[0095] For example, for a selected free block, its access activity change and service quality impact can be assessed as zero, but its data migration overhead is not zero, as it may require memory mapping changes or minor metadata updates. For another inactive occupied block, the system predicts the potential small locality benefits of migrating it from the current fragmented region to a newly consolidated contiguous region, while accurately calculating the bandwidth overhead based on its data size and transmission path, and assessing a low service quality impact level based on its task type.

[0096] By combining spatial bitmaps and activity values ​​for dual screening, a balance between security and effectiveness is ensured in the candidate memory block set, providing high-quality raw materials for subsequent decision-making. A structured difference tensor is constructed for each candidate block by systematically evaluating three core dimensions: activity changes, migration overhead, and service quality impact. This allows the subsequent decision-making process to evolve from a simple "whether to migrate" to a multi-objective trade-off, providing precise and quantitative input data for generating the optimal migration plan under global constraints.

[0097] In one possible embodiment, the step described above, which involves evaluating the changes in access activity, data migration overhead, and impact on quality of service related to the migration of candidate memory blocks in order to determine the differential tensor, may specifically include the following steps: For candidate memory blocks within the candidate migration region, evaluate the changes in access activity, data migration overhead, and the impact on the quality of service of the associated computing tasks resulting from migrating them to the target free address; The change in access activity, the inverse of data migration cost, and the penalty coefficient derived from the service quality impact level are encapsulated as a differential tensor.

[0098] The change in access activity measures the expected improvement or decrease in access efficiency of the data block after migration, and its value may be positive, negative or zero; data migration overhead refers to the comprehensive measure of the transmission bandwidth and time required to complete the data transfer operation itself, which is usually inversely proportional to the data block size and transmission path performance; the service quality impact level is a scale that characterizes the potential interference of migration on the stability of business operations based on task characteristics and data popularity, and is divided into three levels: high, medium and low.

[0099] For each candidate memory block within the candidate migration region, it is necessary to evaluate the specific values ​​generated in the three dimensions mentioned above when migrating it to a selected target free address. The access efficiency of a data block is closely related to its physical location. Migration may move data closer to the computing unit or to a less contentious storage area, thereby improving access locality and bringing positive changes.

[0100] The evaluation needs to comprehensively consider factors such as the block's historical access patterns, the relative positions of the source and target addresses within the GPU memory architecture, and the current access pressure on the target region. For example, migrating a frequently accessed memory block from one end of the memory space to the other end, closer to the currently active area of ​​the compute core, may significantly reduce access latency, thus yielding a large positive change. Conversely, migrating a long-idle block may result in a zero change in access activity.

[0101] Any data movement consumes bandwidth within or across video memory and takes time. The amount of overhead depends primarily on the size of the candidate memory block and the available bandwidth of the data transfer path from the source address to the destination address. The system can estimate this based on the known hardware topology and real-time monitored bandwidth utilization. For example, migrating a 64KB memory block on a high-bandwidth storage channel within the GPU incurs significantly less overhead than migrating a block of the same size across the GPU via the PCIe bus.

[0102] Migration operations may interrupt or delay task access to data, the extent of which depends on the task's Service Level Agreement (SLA) and the criticality of the data block to the task. Evaluation requires consideration of the priority of the task to which the memory block belongs, the task type (e.g., online inference or offline training), and the current activity level of the block.

[0103] For example, migrating a memory block belonging to a high-priority online inference service that is frequently accessed may cause a noticeable increase in latency, and is therefore assessed as having a high impact level. Conversely, migrating a memory block belonging to a background batch training task that has not been accessed for a long time can be assessed as having a low impact level.

[0104] To enable unified comparison and comprehensive decision-making across blocks, three indicators with different dimensions and meanings need to be mapped onto a unified scale and combined into a multidimensional vector. Specifically, the change in access activity is usually used directly; since data migration cost is a cost item, its reciprocal is often used to convert it into a benefit item, so that the larger the cost, the smaller the conversion value; the impact level of service quality is converted into a penalty coefficient through a preset mapping table, with a larger penalty coefficient for a higher impact level. For example, for a candidate block that is evaluated to have a positive change in access activity of 0.3, an estimated data migration cost of 200 microseconds, and a low service quality impact level, its data migration cost is converted to the reciprocal of 0.005 per microsecond, and finally encapsulated into a three-dimensional difference tensor [0.3, 0.005, 0.1].

[0105] For example, embodiments of this application abstract the changes in key metrics caused by each candidate MemBlock before and after migration into a three-component difference tensor (Δ-Tensor), used to uniformly characterize the benefits and costs brought about by migration. For the i-th candidate MemBlock, its difference tensor is denoted as:

[0106] in: The activity and locality gains brought about by migration; The bandwidth and time overhead of the migration itself; : The penalty amount for the impact of migration on service QoS (Quality of Service).

[0107] (1) Activity and local gain component ΔH ΔH primarily reflects the degree of improvement in overall access locality and remote access ratio after migrating a MemBlock from its source location to its target location. In one embodiment, the embodiments of this application can estimate ΔH from the following aspects: a) Based on the aforementioned HotScore and activity curve, compare the changes in HotScore of the segment where the MemBlock is located before and after migration. When the hot data is more concentrated near the memory slice where the actual task is executed, ΔH is considered to be a positive gain. b) If the MemBlock is currently frequently accessed by tasks on a certain GPU or a certain memory slice, and migration can reduce remote access across GPUs or across NUMA paths, then further increase the value of ΔH. c) For MemBlocks that have been in a low-activity state for a long time, moving them to a cold area far away from the hot area does not significantly improve locality, so ΔH can take a small value close to zero.

[0108] In one embodiment, the activity and locality gain of the i-th candidate MemBlock can be defined as follows:

[0109] in, , These represent the activity scores of the segment where the MemBlock was located before migration and the target segment after migration, respectively. , These represent the values ​​of the remote access ratio associated with this MemBlock before and after the migration. To adjust the parameters of the locality improvement weights, norm(·) is a monotonic mapping function that normalizes the results to a preset interval. After migration, if the MemBlock falls into a more active region closer to the task execution and the remote access ratio decreases, then... Take a positive value and the value should be larger.

[0110] The migration cost component ΔC quantifies the resources required to perform a single MemBlock migration, including memory bandwidth, PCIe / NVLink bandwidth, and migration duration. ΔC can be estimated as follows: a) Estimate the theoretical transmission time required to complete one migration based on the MemBlock size (e.g., 64KB to 1MB) and the bandwidth parameters of the underlying link; b) If the source and destination locations are on different GPUs or different NUMA nodes, additional cross-device transfer overhead is introduced; c) Optionally, the current system load status (high or low) can be taken into account, and a higher cost weight can be given to the same number of bytes migration when the system is under high load.

[0111] In one embodiment, embodiments of this application can estimate the migration cost based on the MemBlock size and link bandwidth, defining the migration cost of the i-th candidate MemBlock as:

[0112] in, The number of bytes in this MemBlock. This refers to the effective transmission bandwidth between the source and destination locations (such as the memory bandwidth on the same GPU, the NVLink / PCIe bandwidth across GPUs, etc.). To reflect the current system load state, the coefficient is set to a larger value when the overall system load is high, thus reflecting the higher cost of migrating the same number of bytes. `norm(·)` is the normalization function. This results in... It is proportional to the combined cost of migration time and bandwidth usage, and is used to suppress excessively costly migration operations in subsequent evaluations.

[0113] The QoS penalty component ΔQ reflects the potential impact of migration operations on service QoS, primarily considering the degree of interference to latency-sensitive or high-priority tasks. In one embodiment, the embodiments of this application can estimate ΔQ using the task / context identifier in the aforementioned MemBlock state table, as well as task priority, SLA, and other information provided by the service side. a) If a MemBlock currently belongs mainly to a latency-sensitive online inference service or a high-priority task, migrating it may temporarily lengthen the access path or reduce the cache hit rate. In the embodiments of this application, a higher penalty is imposed on such MemBlocks when calculating ΔQ. b) If MemBlock is only used for offline training or batch processing tasks, and the task is not sensitive to short-term latency jitter, then the corresponding ΔQ can be a smaller value. c) MemBlocks whose activity has significantly decreased in recent cycles and which do not belong to critical business contexts can be considered to have low QoS sensitivity, and their ΔQ can be close to zero.

[0114] In one embodiment, embodiments of this application can combine task priority, QoS sensitivity level, and current activity level to estimate the potential impact of migration on services, and define the QoS penalty of the i-th candidate MemBlock as:

[0115] in, This indicates the priority factor of the task or service to which the MemBlock belongs (e.g., a smaller value for offline tasks and a larger value for online inference services). This indicates the sensitivity level of this type of task to QoS indicators such as latency (for example, a larger value is used for tasks with strict P99 latency requirements). Let be the activity value at time t, and norm(·) be the normalization function. For a MemBlock that is currently in a highly active state and belongs to a high-priority, strongly QoS-constrained service, its A larger value will result in explicit penalty in subsequent comprehensive evaluation, thus preventing migration at inappropriate times.

[0116] To facilitate comparisons between different candidate MemBlocks, embodiments of this application can... Each component is normalized to fall within a preset numerical range (e.g., 0 to 1), and the scale and sensitivity of each component are adjusted according to the application scenario. Furthermore, the three components of the difference tensor are not unique; those skilled in the art can add or remove components as needed, such as adding energy-related components or cache hit rate components. As long as the basic concept of "constructing a multi-dimensional difference vector for evaluating the migration value of a single MemBlock from three dimensions—activity gain, migration cost, and QoS impact"—is still followed, it falls within the protection scope of the embodiments of this application.

[0117] Through the above definition of differential tensors, the embodiments of this application establish a unified, multi-dimensional migration value description for each candidate MemBlock, laying the foundation for the construction of a comprehensive evaluation function and the selection of migration sets under constraints.

[0118] By decomposing the complex multi-factors upon which migration decisions depend into independent, precisely calculable, or evaluable indicators, the decision-making process is transformed from experience-driven to data-driven. Secondly, by standardizing operations such as taking the reciprocal of costs and mapping impact levels to penalty coefficients, the mathematical consistency of the components within the difference tensor is ensured. This lays a solid foundation for subsequent tensor-based comprehensive scoring and optimization ranking, enabling the system to quantitatively weigh local benefits, operational costs, and business risks.

[0119] In one possible embodiment, step 150 may specifically include the following steps: For each candidate memory block, a comprehensive evaluation value is calculated based on the three components of its differential tensor and the preset weighting coefficients. Sort all candidate memory blocks according to their comprehensive evaluation value; Under the premise of meeting the preset bandwidth and quality of service constraints, the target set of memory blocks to be migrated is selected from high to low according to the sorting. A memory migration plan is generated based on the source address of each memory block in the target memory block set and the selected target free address.

[0120] The overall evaluation score is a single score obtained by fusing information from various dimensions of the differential tensor using a preset formula. It is used to measure the net benefit of migrating this block. Weighting coefficients are configurable parameters used to adjust the relative importance of the three components—activity change, migration overhead, and service impact—in the overall evaluation score. For example, they can be set to a set of [0.5, -0.3, -0.2]. Bandwidth constraints refer to the upper limit of the total data transfer volume set for a single migration cycle, such as not exceeding 1GB, to prevent migration operations from exhausting memory bandwidth and affecting normal business operations. Service quality constraints refer to the upper limit of the cumulative service quality impact set for a single migration cycle, used to control the total risk of systemic interference with business operations. The target migration memory block set is the final list of memory blocks determined after all screening criteria and that will be actually migrated in the current cycle.

[0121] To achieve a comparable priority ranking of numerous candidate blocks, it is necessary to project multidimensional difference tensors (rewards, costs, risks) onto a one-dimensional numerical axis. This is typically achieved through a linearly weighted summation, which can be expressed as: The overall evaluation score is calculated as follows: α * change in access activity + β * reciprocal of data migration cost + γ * (negative service quality penalty coefficient). The weights α, β, and γ correspond to the system's trade-off preferences regarding performance gains, operational costs, and business risks, respectively. For example, for a candidate block with a difference tensor of [0.3, 0.005, 0.1], given weight coefficients α = 0.6, β = -50, and γ = -0.8 (assigning negative weights to cost and risk items), its overall evaluation score is calculated as: 0.6 * 0.3 + (-50) * 0.005 + (-0.8) * 0.1 = 0.18 - 0.25 - 0.08 = -0.15. Another candidate block with a difference tensor of [0.1, 0.02, 0.05], using the same weights, has a score of 0.06 - 1.0 - 0.04 = -0.98. Through this calculation, each block obtains a comparable overall score.

[0122] Given limited resources, priority should be given to migrating the blocks with the highest overall benefit, i.e., the highest evaluation value. The sorting operation places all candidate blocks in a unified priority queue, providing a clear order for subsequent constraint filtering. For example, after the system calculates the scores of all blocks, it generates a list with the highest-scoring block (e.g., 0.5) at the top and the lowest-scoring block (e.g., -1.2) at the bottom.

[0123] Globally optimal decisions must be made within limited resource budgets and risk capacities. The system sets two cumulative budgets: a total migration data volume budget and a total service quality impact budget. Then, starting with the highest-scoring block, it checks each block sequentially. If adding a block to the set does not cause any cumulative budget to exceed its limit, it is included; otherwise, it is skipped, and the next block is checked. This greedy strategy can quickly obtain a near-optimal feasible solution under constraints. For example, the total migration data volume can be capped at 1GB, and the total service quality impact at 5.0.

[0124] Starting from the top of the sorting list, the data size and service quality impact value of candidate blocks are accumulated sequentially. When a block of 200MB with an impact value of 0.8 is added, the cumulative data size reaches 900MB, and the cumulative impact value reaches 4.2, both within the limits, so the block is selected. The next block to be checked is 300MB in size. If added, the total data volume will reach 1.2GB, exceeding the 1GB limit, so this block is skipped.

[0125] The decision results are transformed into a list of precise operation instructions that can be executed by the underlying driver. The migration plan is structured data where each entry explicitly specifies the source start address, target start address, and contiguous data length for a migration operation. For example, if the final selected target set contains three memory blocks, the system finds suitable free target addresses for each, and then generates a migration plan containing three entries. Each entry records detailed information such as source address: 0x7F000, target address: 0xA5000, and length: 64KB.

[0126] In the embodiments of this application, the comprehensive migration benefit of the i-th candidate MemBlock is defined as:

[0127] in, These are adjustable non-negative weighting coefficients used to balance the importance of three factors: activity and locality gain, migration overhead, and QoS penalty. Generally, The bigger, the better. and The larger the value, the more unfavorable it is; therefore, in the comprehensive evaluation, the former is counted with a positive sign, while the latter two are counted with a negative sign. This is achieved through adjustments. The value of can make the system more inclined to "more actively converge fragments" or "more conservatively control migration costs and QoS risks" at different stages of operation.

[0128] In another embodiment, to ensure that the comprehensive evaluation value falls within a finite numerical range, embodiments of this application may further add a normalization or nonlinear mapping step after the above linear combination, for example:

[0129] in, The mapping function is monotonically increasing and can be a linear interval scaling function, a sigmoid function, or other normalization methods commonly used in the art. The embodiments of this application are not limited to a specific function form, as long as the relative magnitude of the comprehensive evaluation values ​​among the candidate MemBlocks is maintained.

[0130] Weighting coefficients in the embodiments of this application This can be achieved through operational configuration or offline tuning. For example, when memory fragmentation is severe and the overall cluster load is low, improvements can be made. The value of should be appropriately reduced. This encourages more proactive migration and consolidation; during peak business periods or when there are many delay-sensitive transactions, the impact can be increased. This suppresses high-cost migration operations that have a significant potential impact on QoS.

[0131] The embodiments of this application can also perform simple adaptive adjustments to the weighting coefficient based on the global fragmentation rate, the maximum continuous idle block size, and the service alarm status over the most recent few periods. For example, the weighting coefficient can be automatically increased when the global fragmentation rate is consistently higher than a certain threshold and QoS alarms are few. The value is set to accelerate fragment convergence; the value is automatically increased when delay alarms occur frequently. The value of is chosen to minimize interference with critical business operations.

[0132] After obtaining all candidate MemBlocks After setting the weighting coefficients, the DEC node agent can calculate a comprehensive evaluation value for each candidate MemBlock. or its normalized form The candidate set is then ranked accordingly. Generally, MemBlocks with higher overall evaluation values ​​are considered to have higher migration benefits and more acceptable costs and risks, and are given priority in subsequent migration set selection processes; MemBlocks with negative or near-zero overall evaluation values ​​can be directly eliminated in the current cycle and retained for reassessment in subsequent cycles based on environmental changes.

[0133] It should be noted that the aforementioned comprehensive evaluation function is essentially an implementation of transforming the three objectives of "improving locality and reducing fragmentation," "controlling migration costs," and "protecting service QoS" into a single-objective optimization problem through a linear combination of weights. Those skilled in the art, understanding the basic concept of the embodiments in this application, can achieve similar effects using other multi-objective synthesis methods, such as hierarchical sorting, first by... Filter objects with excessively high QoS risk, then press and Ranking, nonlinear combination, or scoring methods based on utility functions, etc., do not depart from the spirit of the embodiments of this application.

[0134] By using the aforementioned comprehensive evaluation function and migration benefit quantification method, the embodiments of this application uniformly convert the benefits and costs represented by each component in the difference tensor into comparable single scores, so that when selecting migration sets under bandwidth budget and QoS constraints, a systematic decision can be made based on the principle of prioritizing objects with high benefits, low costs, and low risks.

[0135] By introducing weighting coefficients and comprehensive evaluation values, the multi-objective optimization problem is transformed into an operable single-objective ranking problem, thus automating and quantifying the decision-making process. Strict adherence to both bandwidth and quality-of-service constraints ensures that migration operations are always conducted within controllable resource consumption and business risk thresholds, avoiding system overload or service degradation caused by fragmentation. The resulting memory migration plan is clear in structure and complete in information, providing a direct basis for the efficient and accurate execution of step 160.

[0136] In one possible embodiment, the step of selecting the target migration memory block set from high to low according to the sorting under the conditions of satisfying preset bandwidth constraints and quality of service constraints may specifically include the following steps: Set the upper limit for the total amount of migration data and the upper limit for the total service quality impact penalty in the current period; The candidate memory blocks are traversed and sorted in descending order of comprehensive evaluation value. For the memory block currently being traversed, it is determined whether the cumulative amount of migration data exceeds the upper limit of the total amount of migration data after adding it to the target migration memory block set, and whether the cumulative service quality impact penalty exceeds the upper limit of the total service quality impact penalty. If none of them exceed the limit, the current memory block is added to the target migration memory block set; if any one of them exceeds the limit, the current memory block is skipped. Continue traversing the next candidate memory block until the traversal is complete or the filtering termination condition is met.

[0137] The total migration data volume limit is set to protect system memory bandwidth and represents the maximum amount of data allowed to be migrated within a single scheduling cycle, for example, 1GB. It is pre-configured by the system administrator based on the total GPU bandwidth and workload. The total service quality impact penalty limit is set to control business risks and represents the cumulative upper limit of service quality impact penalties allowed from migration operations within a single cycle, for example, 5.0. It reflects the system's overall tolerance for business disruptions. The filtering termination condition is a rule used to prematurely end the traversal process, such as terminating when the cumulative migration data volume has reached 90% of the limit and the subsequent candidate blocks are all large, to simplify calculations.

[0138] Any resource optimization operation must be conducted within limited and controlled resource consumption and risk exposure to prevent the optimization process itself from overloading the system or causing unacceptable disruption to critical business operations. These two upper limits serve as hard constraints, jointly defining the decision boundaries for this migration operation. For example, based on the judgment that the current business is in a low-peak period, the system configures the total migration data volume limit for this cycle to be 800MB, and the total service quality impact penalty limit to be 4.0.

[0139] For the currently traversed memory block, the system will make a key judgment: whether the cumulative amount of migrated data in the new set exceeds the upper limit of the total amount of migrated data after the simulation adds it to the target migrated memory block set, and whether the cumulative service quality impact penalty of the new set exceeds the upper limit of the total service quality impact penalty.

[0140] Specifically, an incremental, simulated addition strategy can be used to ensure that global constraints are always met. The cumulative data size is calculated by adding the data sizes of the selected blocks and the current block; the cumulative penalty is calculated by adding the service quality impact penalty coefficients corresponding to the selected blocks and the current block. For example, suppose the cumulative data size of the selected set is 600MB, and the cumulative penalty is 2.5. The currently traversed candidate block has a data size of 100MB and a service quality impact penalty coefficient of 0.8. After simulated addition, the new cumulative data size will be 700MB, and the new cumulative penalty will be 3.3. The system compares this simulation result with preset upper limits (800MB and 4.0).

[0141] If, after the simulated addition, the cumulative migrated data volume and the cumulative service quality impact penalty do not exceed their respective limits, it indicates that migrating the block is safe under the current constraints, and the system officially adds it to the target set of migrated memory blocks. If either constraint exceeds the limit, it means that migrating the block would cause the system to exceed the predetermined resource or risk budget, therefore the candidate block is skipped and not selected. A greedy selection algorithm is implemented within the constraint boundaries, prioritizing migration operations that satisfy high value and do not violate constraints. Continuing the previous example, since 700MB is less than 800MB and 3.3 is less than 4.0 after the simulated addition, neither exceeds the limit, so the candidate block is successfully added to the target set.

[0142] Continue iterating through the next candidate memory block, repeating the above judgment and decision-making process, until the entire candidate list has been traversed, or the preset screening termination condition is reached. The screening termination condition aims to balance the completeness of the screening with execution efficiency. For example, when the accumulated data volume is very close to the upper limit, or when the number of remaining candidate blocks is very small and the scores are very low, the traversal can be terminated early to save computing resources. Ensure that the screening process is completed within an acceptable time, avoiding unnecessary checks on low-scoring or large-volume candidate blocks that are clearly unlikely to be selected.

[0143] By precisely implementing macro-level resource and risk budgeting into the micro-level, block-by-block selection logic, the final migration set inevitably meets the requirement of global controllability. The mechanism of sorting by value and checking constraints sequentially can quickly find a feasible solution that is approximately optimal under given constraints within polynomial time complexity. That is, it migrates the memory blocks with the highest overall benefits as much as possible within the allowable cost, achieving a good balance between decision-making efficiency and effectiveness.

[0144] The step of generating a memory migration plan based on the source address of each memory block in the target memory block set and the selected target free address may specifically include: The embodiments of this application configure a set of migration constraints at the node level for each scheduling cycle to control the resource consumption and business risks of a single migration operation, typically including: a) Bandwidth and data volume constraints: Limit the total number of bytes migrated within this period to no more than a preset upper limit. And the total migration time does not exceed the given time budget; b) QoS Risk Constraints: Limit the cumulative QoS penalty for the selected MemBlock to no more than a threshold. ,For example

[0145] in The set of migrations selected for this period; c) Single task impact constraint (optional): Limit the number of migration MemBlocks or total bytes involved in the same high-priority task to avoid applying excessive migration disturbances to a critical business.

[0146] The above constraint parameters can be configured by operations and maintenance personnel according to the business load, or they can be adaptively adjusted based on historical operating data.

[0147] In one embodiment, the embodiments of this application employ an iterative screening strategy of "sorting + constraint cumulative checking" to select MemBlocks from the candidate set that satisfy the constraints and have a high comprehensive evaluation value. Specifically, this includes: a) Ranking candidate MemBlocks according to their overall evaluation score Sort the data from largest to smallest to obtain an ordered sequence; b) Attempt to add the migration set sequentially from the head of the sequence. For each MemBlock i to be added, temporarily calculate the cumulative number of migrated bytes and the cumulative QoS penalty amount after the addition. c) If all constraints are still satisfied after adding MemBlock i, then i is fixed and added to the set. If a constraint is triggered (e.g., the cumulative number of migrated bytes exceeds...), or the cumulative QoS penalty amount exceeds If the condition is met, skip that MemBlock and try the next one. d) When the traversal ends, or even adding a highly rated MemBlock would cause a significant constraint overrun, end the filtering process and return the current set to its original state. This is the target migration set for this cycle.

[0148] In another embodiment, the embodiments of this application may also employ a constrained iterative optimization strategy: based on the current comprehensive evaluation value, the process of "selecting according to gradient or scoring direction—projecting or pruning according to constraints—updating available budget" is repeatedly executed for a batch of candidate MemBlocks, approximating the migration set with the maximum comprehensive benefit under constraints within a finite number of iterations. The embodiments of this application do not limit the specific optimization algorithm, as long as the final migration set satisfies the preset constraints and has a superior total benefit in the sense of the comprehensive evaluation value.

[0149] In obtaining the target migration set Subsequently, embodiments of this application construct corresponding migration entries for each MemBlock to be migrated, and aggregate them to form a structured migration plan (MigrationPlan). Typically, each migration entry may contain the following fields: Source memory location: includes source GPU identifier, source memory slice identifier, and source start address; Target memory location: includes target GPU identifier, target memory slice identifier, and target starting address; Migration length: The number of bytes in the corresponding MemBlock or the total number of bytes in several consecutive MemBlocks; Priority or batch flags: used to arrange the migration order during the execution phase, such as prioritizing migration entries that have less impact on QoS and higher benefits; Associated task / context information: Used to track and analyze the migration of specific tasks when necessary.

[0150] In one embodiment, the present application can merge adjacent MemBlock migration entries within the same source / target region, combining multiple small migrations into a larger continuous migration, thereby reducing the number of interface calls and improving migration execution efficiency.

[0151] After generating the MigrationPlan, the DEC node agent can submit it to the execution module responsible for calling the underlying memory migration interface. Before submission, the system can make simple adjustments to the plan based on the current node's bandwidth usage and business window, such as postponing some low-priority migration items to later cycles, or grouping migration items by batch, in order to better coordinate with business traffic scheduling.

[0152] Through the above-mentioned migration set screening and migration plan generation under constraints, the embodiments of this application prioritize the migration of MemBlocks with higher overall benefits and acceptable costs and risks, while ensuring the overall controllability of the migration operation. This provides a clear operation list and execution order for subsequent memory migration execution and memory slice resource pool interaction.

[0153] The embodiments of this application do not require intrusive modifications to the internal implementation of the GPU driver. Instead, they utilize existing memory management capabilities on the node side through the DEC node proxy to perform data migration between the source and target memory regions specified in the MigrationPlan. The "memory management capabilities" referred to herein can be provided by the GPU runtime library, driver, virtualization middleware, or memory management modules built into the business framework. The embodiments of this application do not limit their specific sources.

[0154] Before the start of each scheduling cycle, the DEC node agent first organizes and groups the migration entries in the MigrationPlan to improve migration execution efficiency and reduce interface call overhead. Typically, grouping can be done according to the following principles: a) Group migration entries between the same source and the same target by source GPU / target GPU or source slice / target slice; b) For multiple small migrations with adjacent source and destination addresses and consecutive MemBlocks, merge them into a larger consecutive migration without changing the logical meaning. c) Group by priority or batch, and schedule migration entries with less impact on QoS and higher overall benefits to be executed in the previous batch.

[0155] The grouping and merging operations described above can effectively reduce the number of migration calls within a single cycle.

[0156] In one embodiment, the present application employs a "target-side reallocation + data copy + source-side release" method to migrate MemBlocks. For each migration entry, the DEC node agent can drive the memory management subsystem according to the following steps: a) Allocate a contiguous memory region in the target memory slice that is the same size as or compatible with the size of the MemBlock to be migrated, and obtain the target starting address; b) Use the in-device or cross-device video memory copy interface provided by the video memory management subsystem to copy the specified length of data from the source start address to the target start address; c) After confirming that the copy is complete and the upper-level task has switched to the new address, release the source video memory area or mark it as free at an appropriate time so that it can be reorganized into a larger contiguous free block later; d) Update the MemBlock state table and address mapping maintained internally by the DEC node agent, and migrate the location of the MemBlock from the source slice to the target slice.

[0157] The above steps rely only on the basic "allocation / release" and "copying between memory regions" capabilities provided by the memory management subsystem, and do not depend on the page migration implementation details of a specific vendor. Therefore, they can be adapted to different GPU platforms and virtualization environments.

[0158] To avoid excessive disruption to normal business access during migration operations, embodiments of this application introduce bandwidth and rhythm control mechanisms when performing migrations in batches: a) Within the same set of migration entries, the DEC node agent can utilize the asynchronous copy or batch commit capabilities provided by the memory management subsystem to queue multiple migration requests for execution, while limiting the number of bytes in transit based on the bandwidth budget for this cycle. b) When the current migration queue length or link occupancy is detected to be close to the preset limit, the submission of new migration entries is paused, and the remaining entries are postponed to the next cycle for execution; c) In scenarios where business load has peaks and troughs, large-scale migrations after merging can be performed first during low-volume periods, while only a small number of necessary migrations can be performed during high-volume periods.

[0159] In one implementation, after submitting a migration request, the DEC node agent determines whether the migration is complete through the return status or callback mechanism provided by the memory management subsystem. For migration entries that are confirmed to be complete, the MemBlock status table is updated immediately; for failed or timed-out migration entries, the reason for failure is recorded, and retry, skip, or hold for re-evaluation in a later period is selected as appropriate. If a persistent error is detected, the DEC node agent can pause the remaining migrations in the current period and report the anomaly to the upper-level scheduling or operation and maintenance system.

[0160] After the above migration is completed, the DEC node agent can proceed to the memory layout and fragmentation index update step described in 3.4.2 based on the actual successfully migrated MemBlock set, providing accurate basic data for refreshing and reporting the memory slice status.

[0161] In one possible implementation, GPU memory is abstracted into multiple logical memory slices; Based on the updated memory usage status and memory convolution spectrum, a status record is generated for each logical memory slice. The status record includes: total slice capacity, current maximum contiguous free memory block size, average fragmentation score and average activity score. The status record is reported to the cluster scheduling system.

[0162] Logical memory slicing is a resource management abstraction that represents a contiguous range of physical memory addresses, or an isolated instance in a virtualization scenario, as an independent resource unit that can be uniformly perceived and scheduled by the upper-layer system.

[0163] Directly managing the physical address space is too complex for cluster schedulers and does not match business needs. By introducing a logical layer called slicing, heterogeneous and fragmented physical resources can be transformed into regular resource objects with clear attribute descriptions, thereby greatly simplifying scheduling decisions. For example, a physical GPU with 40GB of video memory can be abstracted into four logical video memory slices, each managing 10GB of address space and independently carrying different computing tasks.

[0164] Based on the updated memory occupancy status after the migration and the recalculated memory convolution spectrum of each contiguous address segment, a structured state record is generated for each logical memory slice. The migration operation changes the memory layout, therefore, key indicators reflecting the current health and availability of each logical slice must be calculated based on the latest state. The total slice capacity in the state record indicates the total amount of memory managed by that slice; the current maximum contiguous free memory block size is the maximum contiguous free address range found after scanning all free blocks within the slice, directly determining whether the slice can accommodate a new large-scale tensor; the average fragmentation score is the average of the fragmentation scores of all contiguous address segments within the slice, reflecting the degree of fragmentation within the slice's internal space; and the average activity score is the average of the activity scores of all contiguous address segments within the slice, characterizing the overall access frequency of data within the slice.

[0165] These metrics together constitute a comprehensive profile of the quality of logical slice resources. For example, for a logical slice, the system scans after migration and finds that its largest contiguous free block is 2GB, calculates the average fragmentation score of its eight contiguous address ranges to be 0.2, and the average activity score to be 0.6, and records them together with its total capacity of 10GB.

[0166] Only by synchronizing the fine-grained internal state of the resource provider (GPU nodes) with the resource manager, i.e., the cluster scheduler, can collaborative global optimization be achieved. The reported state records provide the scheduler with deeper insights beyond traditional remaining capacity, enabling intelligent scheduling based on continuous idle capacity, fragmentation levels, and data temperature. For example, after receiving state records from multiple GPU nodes, when the cluster scheduler needs to select a placement location for a new task requiring 1.5GB of contiguous video memory, it can prioritize logical slices whose state records show a maximum contiguous free block size greater than 1.5GB and a low average fragmentation score, thus avoiding the risk of allocation failure due to memory fragmentation at task startup.

[0167] By generating and reporting status records containing multi-dimensional health indicators, the implicit states of GPU memory, such as fragmentation and heat, are transformed into explicit parameters that are visible and understandable to the cluster scheduling system, thus solving the problem of invisible resource status. Using these refined status records, the cluster scheduling system can implement resource-quality-aware scheduling strategies. For example, it can direct high-capacity tasks to slices with large, continuous periods of free space, or direct latency-sensitive tasks to slices with low fragmentation and moderate heat, thereby improving the overall resource utilization efficiency of the cluster and the success rate and stability of task execution.

[0168] Specifically, the set of MemBlocks that actually migrated successfully within a scheduling cycle is denoted as . The DEC node agent can perform incremental updates to the source and target segments containing these MemBlocks based on the MigrationPlan and migration execution feedback. Specifically, this includes: a) For each successfully migrated MemBlock i, update its GPU identifier, memory slice identifier, and location information in the address space in the MemBlock status table, remove it from the source segment and add it to the target segment; b) For the source segment involved, update the occupied flag of the MemBlock from "occupied" to free, and merge adjacent free blocks if necessary; c) For the target segment involved, mark the newly allocated target area as "occupied" and record its corresponding task / context identifier and the activity statistics unit to which it belongs.

[0169] By performing the above operations, only a small number of segments affected by the migration need to be locally modified, without having to fully scan the entire card's memory in each cycle, which greatly reduces the overhead of state maintenance.

[0170] After completing the MemBlock state table update, the embodiments of this application reconstruct the free / occupied bitmap within the relevant source and target segments, and recalculate the segment-level fragmentation rate defined in the embodiments of this application based on the updated bitmap. The typical steps are as follows: a) Traverse the free / occupied flags of MemBlocks within the Segment in address order to obtain the number of free intervals, their length distribution, and the maximum free interval length. ; b) Retain the total number of idle MemBlocks in the statistics window. Update the fragmentation rate accordingly.

[0171] c) Use the latest fragmentation rate and related statistics as input to update the derived metrics such as FragScore corresponding to the Segment.

[0172] For segments not affected by the current cycle migration, their fragment bitmap and fragmentation rate can remain unchanged until a migration operation falls into them in a subsequent cycle, at which point they will be updated.

[0173] The migration operation itself does not directly generate additional computational access, but it changes the segment affiliation of the MemBlock and its position in the address space. To maintain the coherence of the memory convolution spectrum, the embodiments of this application synchronously adjust the activity and convolution spectrum of the relevant segments after updating the segment layout: a) For each migration, MemBlock i has an activity value. To maintain continuity, only update the segment to which it belongs, its position index in the new segment, and the input sequence that participated in that segment. Location; b) For the affected source and target segments, based on the updated input sequences Re-execute DCT-II or equivalent frequency domain coding to obtain a new memory convolution spectral vector. Based on this, update spectrum-related indicators such as HotScore and FragScore; c) For the remaining unaffected segments, there is no need to recalculate; simply retain the activity and spectral representation from the previous cycle.

[0174] Through the above incremental recalculation, the embodiments of this application, while ensuring feature consistency, strictly limit the update range of the memory convolution spectrum to the local region related to migration, thus avoiding excessive overhead on performance.

[0175] After completing the segment-level metric update, embodiments of this application further aggregate key information at the slice and whole-card levels for subsequent slice status reporting and performance evaluation. Typically, the following metrics can be maintained: a) For each video memory slice, calculate its total capacity, remaining free capacity, and maximum contiguous free block size. And update the slice-level global fragmentation rate accordingly.

[0176] in This represents the total number of free bytes within the slice. b) Based on the HotScore and FragScore of each Segment in the slice, calculate the average activity score and average fragmentation score at the slice level. If necessary, a weighted average method can be used to give greater weight to high-capacity or high-activity Segments. c) Within the scope of the entire card or node, further aggregate the above indicators to obtain global indicators such as the overall card fragmentation rate and the maximum continuous idle block of the entire card, providing data support for operation and maintenance optimization and effect comparison.

[0177] In one embodiment, the embodiments of this application may also maintain a finite-length historical record for key indicators (such as slice-level fragmentation rate, maximum contiguous free blocks, average FragScore, etc.), and perform simple time smoothing or trend analysis on them to evaluate the convergence effect of the current migration strategy in the subsequent optimization stage. It should be noted that the historical record and trend analysis are not core steps of the embodiments of this application, but are mainly used for parameter adjustment and effect monitoring. The specific implementation can be flexibly selected by those skilled in the art according to actual needs.

[0178] Through the incremental updates of the memory layout and fragmentation metrics after the migration described above, the embodiments of this application can obtain a set of segment-level and slice-level metrics consistent with the current actual state of the memory at the end of each scheduling cycle. In one embodiment, the embodiments of this application are deployed on a single computing node, which is configured with four GPUs to support mixed workloads such as training, inference, and batch processing. The memory of each GPU is divided into several memory slices with a fixed capacity, and each slice serves as a memory resource unit with a scheduling granularity.

[0179] On the node side, a DEC node agent process is deployed to periodically collect MemBlock usage and access statistics from the memory management subsystem on each GPU, construct a MemBlock / Segment status table, and generate the memory convolution spectrum, HotScore, and FragScore for each segment. Based on this, the node agent calculates the difference tensor Δ-Tensor for candidate MemBlocks, generates a migration plan (MigrationPlan) that meets the constraints, and executes the migration within the same GPU or between different memory slices through the memory management subsystem.

[0180] At the end of each scheduling cycle, the node agent incrementally updates the Segment layout, fragmentation rate, and memory convolution spectrum based on the actual migration results, and aggregates them to form the SliceStatus for each memory slice. When allocating memory resources for new tasks, the node's local task scheduling module can directly query the local node's memory slice resource pool, prioritizing memory slices with lower fragmentation rates and larger maximum contiguous free blocks for placement. For running loads, a small number of memory slice-level migrations are triggered within the business idle window in conjunction with the SliceStatus to further improve the local node's memory utilization and task stability.

[0181] In this embodiment, all components are centrally deployed within a single node, eliminating the need for additional cluster coordination components. This facilitates the rapid implementation of the memory fragmentation convergence and task placement optimization scheme of this application in a single-machine development environment or on a small-scale inference node.

[0182] In another embodiment, the embodiments of this application are applied to a GPU cluster environment containing multiple compute nodes. Each compute node in the cluster deploys a DEC node agent, and performs memory status acquisition, memory convolution spectrum construction, differential tensor evaluation, and memory migration execution within the node, generating a memory slice resource pool view for that node.

[0183] The cluster scheduling center maintains a global task queue and node resource view, where the video memory resource portion comes from the SliceStatus summaries periodically reported by each node. When selecting target nodes and target video memory slices for new tasks, the scheduling center first filters out nodes and their video memory slices that meet the basic conditions globally based on the task's video memory capacity requirements and QoS requirements. Subsequently, combining indicators such as the fragmentation rate, maximum contiguous free block size, and average HotScore reported by each candidate slice, it prioritizes slices with lower fragmentation and sufficient contiguous free space for task placement.

[0184] For running tasks, when memory fragmentation on a node continues to worsen or local resource hotspots appear, the scheduling center can combine the global fragmentation indicators and SliceStatus reported by the node to select some low-priority, low-QoS-sensitive tasks and perform task-level migration, provided that the cross-node migration cost allows, to migrate them from severely fragmented or high-load nodes to nodes with lighter fragmentation and more balanced load. The internal memory layout adjustments for these tasks within the node are specifically executed by the DEC node agent in this embodiment of the application through a MemBlock-level migration plan.

[0185] Through the above-described multi-node cluster deployment method, the embodiments of this application enable the cluster scheduling center to make cross-node task placement and migration decisions based on a unified view of the memory slice resource pool while maintaining the gradual convergence of memory fragmentation within the nodes, thereby achieving collaborative optimization of memory fragmentation governance and cluster-level resource scheduling.

[0186] like Figure 2 As shown, the cluster scheduling and video memory resource pool layer includes: Cluster scheduler: As the brain of the cluster, it makes task placement / migration decisions based on task requirements and resource status.

[0187] Memory slicing: This is the core resource abstraction. The system divides the physical GPU's memory into multiple logical resource units (slices). The scheduler perceives and manages these slices, not the direct physical hardware. Slices create / update their state from lower layers and report it to the scheduler.

[0188] The middle layer (GPU nodes) includes: DEC node agent: This is the core service running on each GPU server. It receives decisions from the upper layer and drives the underlying hardware to perform specific memory management operations. It is the main body responsible for implementing the intelligent logic for memory fragmentation convergence.

[0189] The underlying layer includes the physical GPU, providing basic hardware capabilities such as memory allocation / release and page migration interfaces. The DEC agent calls these interfaces to actually perform data movement and layout adjustments.

[0190] The cluster scheduler makes a decision and issues instructions to the corresponding DEC agent on the node. The DEC agent ultimately calls the physical GPU's interface to complete the task. The physical GPU's state information and high-level metrics such as memory convolution spectrum and fragmentation calculated by the DEC agent are aggregated and abstracted into the state of memory slices, and reported to the cluster scheduler for better decision-making. By introducing DEC node agents on top of the hardware driver for intelligent decision-making and execution, and by introducing memory slice resource abstraction at the cluster level, the aim is to solve the problem of memory fragmentation in multi-tasking environments.

[0191] In the embodiments of this application, the occupancy status and access activity curves of memory blocks within multiple consecutive address segments of GPU memory are obtained. For each consecutive address segment, frequency domain transformation and encoding are performed based on the occupancy status and access activity curves of memory blocks within the consecutive address segment to generate a memory convolution spectrum that characterizes the fragmentation degree and temporal access activity of the memory space in that segment. The high-dimensional, raw occupancy and activity data are compressed and encoded into unified mathematical features, realizing a joint quantitative characterization of the two key attributes of memory fragmentation degree and data hot / coldness. This allows the subsequent system to evaluate and compare the consolidation value of different regions based on clear numerical indicators. Based on the memory convolution spectrum of each consecutive address segment, candidate migration regions are determined, concentrating computational resources and operational risks on the most effective target. The differential tensor of candidate memory blocks within the candidate migration region is determined. The differential tensor is used to characterize the changes caused by migrating candidate memory blocks to the target free address. The differential tensor encapsulates the expected benefits, operational costs, and business risks of a single migration, making the migration value of each candidate block calculable and comparable. Based on the differential tensors of all candidate memory blocks, a comprehensive evaluation is performed under preset bandwidth and quality of service constraints to generate a memory migration plan. The memory migration plan includes source address, target address, and migration length information. It can generate a specific operation sequence that maximizes the overall benefit within the current period, transforming defragmentation from a kernel adaptive behavior that may lead to unstable consequences into a predictable and controllable online optimization service. The underlying interface is called to execute the batch migration of candidate memory blocks according to the memory migration plan. Through planned batch migration of data blocks, the memory layout is physically rearranged as expected, thereby actually merging small fragments and forming larger contiguous free areas, directly achieving the ultimate goal of fragmentation convergence. After the migration is completed, the memory occupancy status of the relevant contiguous address segments is updated, and the status information of the corresponding logical memory slices is reported to the cluster scheduling system, improving resource utilization and task stability at the system level.

[0192] The memory block management method provided in this application can be executed by a memory block management device. This application uses the execution of the memory block management method by a memory block management device as an example to illustrate the memory block management device provided in this application.

[0193] Figure 3 This is a block diagram of a video memory block management device provided in an embodiment of this application. The device 300 includes: The acquisition module 310 is used to acquire the occupancy status and access activity curve of memory blocks within multiple consecutive address segments of GPU memory. The generation module 320 is used to perform frequency domain transformation and encoding on each continuous address segment based on the occupancy status and access activity curve of the video memory blocks within the continuous address segment, and generate a video memory convolution spectrum to characterize the fragmentation degree and temporal access activity of the video memory space in that segment. The first determining module 330 is used to determine candidate migration regions based on the memory convolution spectrum of each consecutive address segment; The second determining module 340 is used to determine the differential tensor of the candidate memory blocks in the candidate migration region; the differential tensor is used to characterize the changes caused by migrating the candidate memory blocks to the target free address; The evaluation module 350 is used to perform a comprehensive evaluation based on the differential tensors of all candidate memory blocks under preset bandwidth and quality of service constraints, and generate a memory migration plan; the memory migration plan includes source address, target address and migration length information; The migration module 360 ​​is used to call the underlying interface and perform batch migration of candidate memory blocks according to the memory migration plan; The update module 370 is used to update the memory occupancy status of the relevant continuous address segments after the migration is completed, and to report the status information of the corresponding logical memory slices to the cluster scheduling system.

[0194] In one possible embodiment, the generation module 320 is specifically used for: Map the occupancy status of each video memory block within a contiguous address segment to a space occupancy bitmap; The frequency domain feature coefficients are extracted by performing discrete cosine transform or wavelet transform on the access activity curves of each memory block within a continuous address segment. The memory convolution spectrum is obtained based on the space occupancy bitmap and frequency domain feature coefficients.

[0195] In one possible embodiment, the acquisition module 310 is specifically used for: The linear address space of the GPU memory is divided into multiple smallest statistical units of a fixed size, which are called memory blocks; Multiple contiguous memory blocks are aggregated into a single contiguous address segment to obtain multiple contiguous address segments; Collect the occupancy status and historical access records of video memory blocks within each consecutive address segment at a preset granularity; Based on historical access records, determine the access activity curve for each memory block.

[0196] In one possible embodiment, the acquisition module 310 is specifically used for: Within each scheduling cycle, the number of times each video memory block is accessed is counted based on historical access records; Based on the number of accesses in the current period and the historical activity value in the previous period, the current activity value of each memory block is updated using an exponential decay smoothing algorithm. The access activity curve for each memory block is determined based on the current activity value updated over multiple scheduling cycles.

[0197] In one possible embodiment, the first determining module 330 is specifically used for: Obtain the space occupancy bitmap for each contiguous address segment; Based on the memory convolution spectrum and space occupancy bitmap of each consecutive address segment, calculate the fragmentation score and activity score of the consecutive address segment. A contiguous address range with a fragmentation score higher than the first threshold and an activity score lower than the second threshold is identified as a candidate migration region.

[0198] In one possible embodiment, the second determining module 340 is specifically used for: Based on the space occupancy bitmap within the continuous address segment corresponding to the candidate migration region, select video memory blocks that are in an idle state or whose activity value is lower than the third threshold from the candidate migration region as candidate video memory blocks. For candidate memory blocks, assess the changes in access activity, data migration overhead, and impact on service quality involved in their migration to determine the differential tensor.

[0199] In one possible embodiment, the second determining module 340 is specifically used for: For candidate memory blocks within the candidate migration region, evaluate the changes in access activity, data migration overhead, and the impact on the quality of service of the associated computing tasks resulting from migrating them to the target free address; The change in access activity, the inverse of data migration cost, and the penalty coefficient derived from the service quality impact level are encapsulated as a differential tensor.

[0200] In one possible embodiment, the evaluation module 350 is specifically used for: For each candidate memory block, a comprehensive evaluation value is calculated based on the three components of its differential tensor and the preset weighting coefficients. Sort all candidate memory blocks according to their comprehensive evaluation value; Under the premise of meeting the preset bandwidth and quality of service constraints, the target set of memory blocks to be migrated is selected from high to low according to the sorting. A memory migration plan is generated based on the source address of each memory block in the target memory block set and the selected target free address.

[0201] In one possible embodiment, the evaluation module 350 is specifically used for: Set the upper limit for the total amount of migration data and the upper limit for the total service quality impact penalty in the current period; The candidate memory blocks are traversed and sorted in descending order of comprehensive evaluation value. For the memory block currently being traversed, it is determined whether the cumulative amount of migration data exceeds the upper limit of the total amount of migration data after adding it to the target migration memory block set, and whether the cumulative service quality impact penalty exceeds the upper limit of the total service quality impact penalty. If none of them exceed the limit, the current memory block is added to the target migration memory block set; if any one of them exceeds the limit, the current memory block is skipped. Continue traversing the next candidate memory block until the traversal is complete or the filtering termination condition is met.

[0202] In one possible embodiment, the device 300 may further include: The reporting module is specifically used for: Abstract the GPU memory into multiple logical memory slices; Based on the updated memory usage status and memory convolution spectrum, a status record is generated for each logical memory slice. The status record includes: total slice capacity, current maximum contiguous free memory block size, average fragmentation score and average activity score. The status record is reported to the cluster scheduling system.

[0203] In the embodiments of this application, the occupancy status and access activity curves of memory blocks within multiple consecutive address segments of GPU memory are obtained. For each consecutive address segment, frequency domain transformation and encoding are performed based on the occupancy status and access activity curves of memory blocks within the consecutive address segment to generate a memory convolution spectrum that characterizes the fragmentation degree and temporal access activity of the memory space in that segment. The high-dimensional, raw occupancy and activity data are compressed and encoded into unified mathematical features, realizing a joint quantitative characterization of the two key attributes of memory fragmentation degree and data hot / coldness. This allows the subsequent system to evaluate and compare the consolidation value of different regions based on clear numerical indicators. Based on the memory convolution spectrum of each consecutive address segment, candidate migration regions are determined, concentrating computational resources and operational risks on the most effective target. The differential tensor of candidate memory blocks within the candidate migration region is determined. The differential tensor is used to characterize the changes caused by migrating candidate memory blocks to the target free address. The differential tensor encapsulates the expected benefits, operational costs, and business risks of a single migration, making the migration value of each candidate block calculable and comparable. Based on the differential tensors of all candidate memory blocks, a comprehensive evaluation is performed under preset bandwidth and quality of service constraints to generate a memory migration plan. The memory migration plan includes source address, target address, and migration length information. It can generate a specific operation sequence that maximizes the overall benefit within the current period, transforming defragmentation from a kernel adaptive behavior that may lead to unstable consequences into a predictable and controllable online optimization service. The underlying interface is called to execute the batch migration of candidate memory blocks according to the memory migration plan. Through planned batch migration of data blocks, the memory layout is physically rearranged as expected, thereby actually merging small fragments and forming larger contiguous free areas, directly achieving the ultimate goal of fragmentation convergence. After the migration is completed, the memory occupancy status of the relevant contiguous address segments is updated, and the status information of the corresponding logical memory slices is reported to the cluster scheduling system, improving resource utilization and task stability at the system level.

[0204] The memory block management device provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be repeated here to avoid repetition.

[0205] Optionally, Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown.

[0206] An electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0207] Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0208] Memory 402 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is a non-volatile solid-state memory. In a particular embodiment, memory 402 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0209] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the video memory block management methods in the embodiment shown in the figure.

[0210] In one example, the electronic device may also include a communication interface 404 and a bus 410. For example, Figure 4 As shown, the processor 401, memory 402, and communication interface 404 are connected through bus 410 and complete communication with each other.

[0211] Communication interface 404 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0212] Bus 410 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0213] This electronic device can execute the memory block management method in the embodiments of this application, thereby achieving the combination of Figure 2 The method for managing video memory blocks is described.

[0214] Furthermore, in conjunction with the memory block management method in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; these computer program instructions are implemented when executed by a processor. Figure 1 The method for managing video memory blocks.

[0215] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0216] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0217] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0218] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for managing video memory blocks, characterized in that, The method includes: Obtain the occupancy status and access activity curves of memory blocks within multiple consecutive address segments of GPU memory; For each of the continuous address segments, frequency domain transformation and encoding are performed based on the occupancy status of the memory blocks within the continuous address segment and the access activity curve to generate a memory convolution spectrum that characterizes the fragmentation degree and temporal access activity of the memory space in that segment. Candidate migration regions are determined based on the memory convolution spectrum of each of the aforementioned consecutive address segments; Determine the differential tensor of candidate memory blocks within the candidate migration region; the differential tensor is used to characterize the changes caused by migrating the candidate memory blocks to the target free address; Based on the differential tensors of all candidate memory blocks, a comprehensive evaluation is performed under preset bandwidth and quality of service constraints to generate a memory migration plan; the memory migration plan includes source address, target address and migration length information; Call the underlying interface to perform batch migration of the candidate memory blocks according to the memory migration plan; After the migration is completed, the memory occupancy status of the relevant contiguous address segments is updated, and the status information of the corresponding logical memory slices is reported to the cluster scheduling system.

2. The method according to claim 1, characterized in that, The step of performing frequency domain transformation and encoding based on the occupancy status of memory blocks within the continuous address segment and the access activity curve to generate a memory convolution spectrum characterizing the fragmentation degree and temporal access activity of that memory segment includes: The occupancy status of each video memory block within the continuous address segment is mapped to a space occupancy bitmap; The frequency domain feature coefficients are extracted by performing discrete cosine transform or wavelet transform on the access activity curves of each memory block within the continuous address segment. The memory convolution spectrum is obtained based on the space occupancy bitmap and the frequency domain feature coefficients.

3. The method according to claim 1, characterized in that, The process of obtaining the occupancy status and access activity curves of memory blocks within multiple consecutive address segments of GPU memory includes: The linear address space of the GPU memory is divided into multiple smallest statistical units of a fixed size, which are called memory blocks; Multiple consecutive memory blocks are aggregated into a single consecutive address segment to obtain multiple consecutive address segments; Collect the occupancy status and historical access records of the video memory blocks within each consecutive address segment at a preset granularity; Based on the historical access records, the access activity curve of each memory block is determined.

4. The method according to claim 3, characterized in that, The step of determining the access activity curve of each memory block based on the historical access records includes: Within each scheduling cycle, the number of times each memory block is accessed is counted based on the historical access records; Based on the number of accesses in the current period and the historical activity value in the previous period, the current activity value of each memory block is updated using an exponential decay smoothing algorithm. The access activity curve for each memory block is determined based on the current activity value updated over multiple scheduling cycles.

5. The method according to claim 1, characterized in that, The determination of candidate migration regions based on the memory convolution spectrum of each of the consecutive address segments includes: Obtain the space occupancy bitmap for each of the contiguous address segments; Based on the memory convolution spectrum and the space occupancy bitmap of each consecutive address segment, calculate the fragmentation score and activity score of the consecutive address segment; A contiguous address range with a fragmentation score higher than the first threshold and an activity score lower than the second threshold is identified as the candidate migration region.

6. The method according to claim 1, characterized in that, The step of determining the differential tensor of candidate memory blocks within the candidate migration region includes: Based on the space occupancy bitmap within the continuous address segment corresponding to the candidate migration region, select memory blocks that are in an idle state or whose activity value is lower than the third threshold from the candidate migration region as candidate memory blocks. For the candidate memory blocks, the changes in access activity, data migration overhead, and impact on service quality involved in their migration are evaluated to determine the differential tensor.

7. The method according to claim 6, characterized in that, The process of evaluating the changes in access activity, data migration overhead, and impact on service quality involved in the migration of the candidate memory blocks to determine the differential tensor includes: For candidate memory blocks within the candidate migration region, evaluate the changes in access activity, data migration overhead, and the impact on the quality of service of the associated computing tasks resulting from migrating them to the target free address; The change in access activity, the reciprocal of the data migration cost, and the penalty coefficient derived from the impact level of the service quality are encapsulated into the differential tensor.

8. The method according to claim 1, characterized in that, The process of generating a memory migration plan based on the differential tensors of all candidate memory blocks, under preset bandwidth and quality of service constraints, includes: For each candidate memory block, a comprehensive evaluation value is calculated based on the three components of its differential tensor and the preset weighting coefficients. Sort all candidate memory blocks according to their comprehensive evaluation value; Under the condition of satisfying the preset bandwidth constraints and quality of service constraints, the target migration memory block set is selected from high to low according to the sorting. The memory migration plan is generated based on the source address of each memory block in the target migration memory block set and the selected target free address.

9. The method according to claim 8, characterized in that, The step of selecting the target migration memory block set from high to low according to the sorting, under the condition of satisfying preset bandwidth constraints and quality of service constraints, includes: Set the upper limit for the total amount of migration data and the upper limit for the total service quality impact penalty in the current period; The candidate memory blocks are traversed and sorted in descending order of comprehensive evaluation value. For the memory block currently being traversed, it is determined whether the cumulative amount of migration data exceeds the upper limit of the total amount of migration data after adding it to the target migration memory block set, and whether the cumulative service quality impact penalty exceeds the upper limit of the total service quality impact penalty. If none of them exceed the limit, the current memory block is added to the target migration memory block set; if any one of them exceeds the limit, the current memory block is skipped. Continue traversing the next candidate memory block until the traversal is complete or the filtering termination condition is met.

10. The method according to claim 1, characterized in that, After the migration is completed, the memory occupancy status of the relevant contiguous address segments is updated, and the status information of the corresponding logical memory slice is reported to the cluster scheduling system, including: Abstract the GPU memory into multiple logical memory slices; Based on the updated memory usage status and memory convolution spectrum, a status record is generated for each logical memory slice. The status record includes: total slice capacity, current maximum contiguous free memory block size, average fragmentation score and average activity score. The status record is reported to the cluster scheduling system.

11. A device for managing video memory blocks, characterized in that, The device includes: The acquisition module is used to obtain the occupancy status and access activity curve of memory blocks within multiple consecutive address segments of GPU memory. The generation module is used to perform frequency domain transformation and encoding on each of the continuous address segments based on the occupancy status of the video memory blocks within the continuous address segments and the access activity curve, to generate a video memory convolution spectrum that characterizes the fragmentation degree and temporal access activity of the video memory space in that segment. The first determining module is used to determine candidate migration regions based on the memory convolution spectrum of each of the consecutive address segments; The second determining module is used to determine the differential tensor of the candidate memory blocks in the candidate migration region; the differential tensor is used to characterize the changes caused by migrating the candidate memory blocks to the target free address; The evaluation module is used to perform a comprehensive evaluation based on the differential tensors of all candidate memory blocks under preset bandwidth and quality of service constraints, and generate a memory migration plan; the memory migration plan includes source address, target address and migration length information; The migration module is used to call the underlying interface and perform batch migration of the candidate memory blocks according to the memory migration plan; The update module is used to update the memory occupancy status of the relevant contiguous address segments after the migration is completed, and to report the status information of the corresponding logical memory slices to the cluster scheduling system.

Images