Distributed memory pool memory batch allocation method and system
By dividing the Slab allocator into logical slices and utilizing atomic addition operations, the latency and memory utilization issues caused by lock contention in high-concurrency scenarios are resolved, achieving high-concurrency, low-latency batch memory allocation and meeting the memory pool requirements of distributed inference for large language models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNICOM INTERNET OF THINGS CO LTD
- Filing Date
- 2026-05-21
- Publication Date
- 2026-07-21
Smart Images
Figure CN122261845B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed memory pools, specifically to a method and system for batch allocation of memory in distributed memory pools. Background Technology
[0002] In a distributed memory pool system, the Slab allocator manages the physical memory accessible via Remote Direct Memory Access (RDMA). Batch allocation of contiguous slots is a prerequisite for achieving RDMA scattering, aggregation, and merging. Existing lock-based Slab batch allocation methods require bitmap scanning and marking operations to be completed within a lock-protected critical section. In high-concurrency scenarios, lock contention is severe, and allocation latency increases linearly with the number of requested slots. Existing lock-free batch allocation methods use a per-CPU dedicated Slab architecture, where the Slab memory (memory blocks) corresponding to different CPU cores are physically isolated. Idle slots cannot be reused across CPU cores, resulting in a significant decrease in memory utilization. None of the existing solutions can meet the high-concurrency, low-latency KV-Cache access requirements in distributed inference scenarios for large language models. Summary of the Invention
[0003] In response to at least one of the problems in the prior art, this application provides a method and system for batch allocation of memory in a distributed memory pool, which ensures stable and controllable allocation latency in high-concurrency scenarios and avoids the problem of decreased memory utilization.
[0004] The distributed memory pool batch allocation method proposed in this application pre-divides a single Slab into multiple logical slices corresponding to CPU cores. Each logical slice independently maintains an atomic slice tail pointer to record the offset of the highest allocated bit within the slice. Upon receiving a batch allocation request carrying the number of slots, the CPU core identifier of the currently running thread is first obtained. Based on this identifier, an atomic addition operation is performed on the slice tail pointer of the corresponding logical slice to lock the continuous slot range. Subsequently, only the bit state of the corresponding slot range in the bitmap of the single allocator needs to be updated to return the continuous physical address corresponding to the slot. The entire allocation process does not require the intervention of software locks and also supports cross-core reuse of idle slots.
[0005] The first aspect of this application provides a method for batch allocation of memory in a distributed memory pool, including: Receive a batch allocation request containing the number of slots, and obtain the CPU core identifier of the currently running thread; According to the CPU core identifier, an atomic addition operation is performed on the fragment tail pointer of the corresponding logical fragment in the single allocator to lock the continuous slot range. The single allocator is divided into multiple logical fragments, each logical fragment corresponds to a CPU core, and each logical fragment maintains an independent fragment tail pointer. The fragment tail pointer is an atomic variable that records the offset of the highest allocated bit in the corresponding logical fragment. Update the bit state of the corresponding slot interval in the bitmap of the single allocator, complete the batch continuous slot allocation, and return the continuous physical address corresponding to the slot interval.
[0006] Optionally, receive a batch allocation request containing the number of slots, and obtain the CPU core identifier of the currently running thread, including: Read the global free slot count of the single allocator and determine whether the global free slot count is greater than or equal to the number of slots; If the global free slot count is greater than or equal to the number of slots, obtain the CPU core identifier of the currently running thread; Read the end pointer of the logical segment corresponding to the CPU core identifier, and calculate the number of remaining slots of the logical segment.
[0007] Optionally, based on the CPU core identifier, an atomic addition operation is performed on the end pointer of the corresponding logical slice within a single allocator to lock a continuous slot range, including: If the number of remaining slots in the logical shard is greater than or equal to the number of slots, perform an atomic addition operation on the shard tail pointer to obtain the old tail pointer returned by the atomic addition operation; Determine whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical fragment; If the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical segment, lock a continuous slot interval starting from the old tail pointer and with a length equal to the number of slots.
[0008] Optional, also includes: If the number of remaining slots in the logical partition is less than the number of slots, traverse the other logical partitions in order of CPU core identifier; Perform an atomic addition operation on the end pointer of the logical slice that has been traversed to obtain the corresponding old end pointer; Determine whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical segment; If the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical segment, lock a continuous slot interval starting from the old tail pointer and with a length equal to the number of slots.
[0009] Optionally, update the bit state of the corresponding slot interval in the bitmap of the single allocator, and return the continuous physical address corresponding to the slot interval, including: In the non-critical unlocked state, the bit in the corresponding slot interval of the bit map is set to the occupied state; Calculate the consecutive physical addresses corresponding to the slot interval based on the starting offset of the slot interval and the physical base address of the single allocator; Return the consecutive physical addresses.
[0010] Optionally, in the unlocked state of the non-critical section, setting the bit of the corresponding slot interval in the bitmap to the occupied state includes: Obtain the cycle number of the current allocation cycle, where the cycle number is an incrementing integer; Write the cycle number into the reserved bit of the corresponding slot interval in the bitmap, and set the occupied bit of the corresponding slot interval in the bitmap to the occupied state.
[0011] Optional, also includes: Check whether the end pointer of the logical fragment has reached the upper limit offset of the corresponding logical fragment; If the end pointer of the segment reaches the upper limit offset of the corresponding logical segment, read the lowest free bit offset within the logical segment; The fragment tail pointer is reset to the lowest free bit offset by an atomic comparison swap operation.
[0012] Optional, also includes: Receive a slot release request, the slot release request containing the offset of the slot to be released; Set the occupied bit at the corresponding offset in the bitmap to an idle state; Atom increases the global free slot count of the single allocator.
[0013] Optional, also includes: If the number of slots is greater than the number of remaining slots in a single logical shard and greater than the number of remaining slots in a single allocator, traverse multiple consecutive single allocators in physical address order. Perform an atomic addition operation on the end pointer of the corresponding logical slice in each traversed single allocator to lock a continuous slot range across multiple single allocators; Update the bit state of the corresponding slot interval in the bitmap of each single allocator, and return the continuous physical address across multiple single allocators.
[0014] A second aspect of this application provides a distributed memory pool batch allocation system, the system comprising: The request processing module is used to receive batch allocation requests containing the number of slots and obtain the CPU core identifier of the currently running thread. The Slab memory management module is used to perform atomic addition operations on the tail pointer of the corresponding logical slice within a single allocator according to the CPU core identifier, thereby locking a continuous slot range. The single allocator is divided into multiple logical slices, each logical slice corresponds to a CPU core, and each logical slice maintains an independent tail pointer. The tail pointer is an atomic variable that records the offset of the highest allocated bit within the corresponding logical slice. The bitmap update module is used to update the bit state of the corresponding slot interval in the bitmap of the single allocator, complete the batch continuous slot allocation, and return the continuous physical address corresponding to the slot interval.
[0015] As can be seen from the above technical solution, this application first matches logical shards with CPU cores, and reduces the probability of conflict in allocation operations between different cores by leveraging CPU core affinity. Then, it uses the atomic addition operation natively supported by the CPU hardware to update the shard tail pointer, which can ensure thread safety of multi-threaded operations without software locks and eliminate the overhead caused by lock contention. At the same time, the unidirectional incrementing characteristic of the shard tail pointer can naturally ensure the continuity of the slot range locked by each atomic operation, which meets the memory continuity requirements of RDMA scattering, gathering and merging. The logical sharding is only a logical division of address ranges rather than a physical isolation design, which allows all free slots to be reused by allocation requests from any CPU core. This ensures stable and controllable allocation latency in high-concurrency scenarios and avoids the problem of decreased memory utilization. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a flowchart illustrating a distributed memory pool batch allocation method in an embodiment of this application.
[0018] Figure 2 This is a flowchart illustrating one of the possible implementation methods in the embodiments of this application.
[0019] Figure 3 This is a schematic diagram of the process for locking a continuous slot range in an embodiment of this application.
[0020] Figure 4 This is a second schematic flowchart of an implementable method in the embodiments of this application.
[0021] Figure 5This is a schematic diagram of the process of returning the continuous physical address corresponding to the slot range in the embodiments of this application.
[0022] Figure 6 This is a schematic diagram of the process of setting the bit of the corresponding slot interval in the bit map to an occupied state in a non-critical, unlocked state in an embodiment of this application.
[0023] Figure 7 This is a flowchart of one possible implementation method in the embodiments of this application.
[0024] Figure 8 This is a flowchart illustrating one possible implementation method in the embodiments of this application.
[0025] Figure 9 This is the fifth flowchart illustrating one possible implementation method in the embodiments of this application.
[0026] Figure 10 This is a schematic diagram of the structure of a distributed memory pool memory batch allocation system in an embodiment of this application. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0028] Taking the KV cache allocation scenario of distributed inference for large language models as an example, existing lock-based batch allocation methods require each thread to compete for a global spin lock when handling batch allocation requests from 100 concurrent threads. Within the lock, the thread needs to scan a bitmap to find consecutive free segments and mark the corresponding bits. When allocating 100 slots in a batch, the lock operation requires scanning hundreds of bitmaps, resulting in a critical section length of hundreds of CPU cycles. Under high concurrency, multiple threads competing for the lock simultaneously leads to a large number of threads entering a waiting state, increasing context switching overhead and raising allocation latency from microseconds to milliseconds. Existing lock-free batch allocation methods allocate a dedicated memory block for each CPU core. When the dedicated memory block of a CPU core is exhausted while other cores have a large number of free slots, the free slots cannot be reused across cores, resulting in memory utilization typically below 70%. In scenarios with limited memory resources, memory shortage errors may occur.
[0029] Based on this, this application provides an implementation method for a distributed memory pool batch memory allocation method, such as... Figure 1 As shown, the method includes: S101, receive a batch allocation request containing the number of slots, and obtain the CPU core identifier of the currently running thread; S102, according to the CPU core identifier, perform an atomic addition operation on the shard tail pointer of the corresponding logical shard in the single allocator to lock the continuous slot range. The single allocator is divided into multiple logical shards, each logical shard corresponds to a CPU core, and each logical shard maintains an independent shard tail pointer. The shard tail pointer is an atomic variable that records the offset of the highest allocated bit in the corresponding logical shard. S103, update the bit state of the corresponding slot interval in the bitmap of the single allocator, complete the batch continuous slot allocation and return the continuous physical address corresponding to the slot interval.
[0030] In this embodiment, a single memory block refers to a contiguous physical memory block of fixed size, which is the basic management unit of the memory allocator. Logical sharding refers to dividing a single memory block into multiple address ranges of equal length according to the number of CPU cores, with each address range corresponding to one CPU core. Logical sharding is merely a logical division of address ranges and does not involve physical memory isolation; all logical shards share the same physical memory and global bitmap of the same memory block. The shard tail pointer is an atomic variable independently maintained by each logical shard, used to record the highest bit offset of the allocated slots within that logical shard, with its initial value being the starting offset of the corresponding logical shard. Atomic addition operations are atomic instructions provided by the CPU hardware, ensuring that when multiple threads simultaneously perform addition operations on the same variable, no data race occurs, and the operation result is atomic.
[0031] It's important to note that the entity receiving the batch allocation request is the request processing unit of the memory allocator. The batch allocation request is initiated by the upper-layer application and contains the number of consecutive slots to be allocated. Obtaining the CPU core identifier of the currently running thread can be achieved by reading specific CPU registers. Different CPU architectures have corresponding registers to store the currently running core number. The process of finding the corresponding logical slice based on the CPU core identifier is implemented through a pre-established mapping table between CPU core identifiers and logical slice indices. This mapping table is created during memory block initialization, and each CPU core identifier corresponds to a unique logical slice index.
[0032] For example, when an upper-layer application needs to allocate N consecutive memory slots in batches, it sends a batch allocation request containing N to the memory allocator. Upon receiving the request, the memory allocator first obtains the CPU core number of the thread currently handling the request, then locates the corresponding logical slice based on that number, and reads the slice's tail pointer. An atomic addition operation is performed on the slice tail pointer, incrementing it by N. The atomic addition operation returns the old tail pointer value before the addition. The interval of length N, starting from the old tail pointer, is the consecutive slot interval locked for this allocation. The bit state of the corresponding interval in the bitmap is then updated to mark it as occupied. Finally, the consecutive physical addresses are calculated based on the old tail pointer and the physical base address of the memory block and returned to the upper-layer application.
[0033] Specifically, the unidirectional incrementing nature of the fragment tail pointer ensures that the slot range locked by each atomic addition operation is continuous. Because the fragment tail pointer only increases and never decreases, the ranges between the old tail pointers returned by two atomic addition operations will not overlap or have gaps. This guarantees that the slots obtained by each batch allocation are contiguous in physical address, satisfying the slot continuity requirement of remote direct memory access scattering-gathering-merging.
[0034] Furthermore, atomic addition operations are thread-safe thanks to CPU hardware. When multiple threads simultaneously perform atomic addition operations on the same segment tail pointer, the CPU serializes these operations, ensuring that each thread obtains the correct old tail pointer value. This prevents two threads from locking the same slot range. Therefore, the entire allocation process does not require any software lock protection, eliminating the overhead of lock contention.
[0035] Those skilled in the art will understand that the Remote Direct Memory Access (RDA) scatter-gather-merge mechanism requires that the accessed memory blocks be contiguous in physical address space. This allows accesses to multiple memory blocks to be merged into a single RDA work request, reducing the number of work requests and improving RDA transmission performance. In this embodiment, the atomic addition operation of the fragment tail pointer ensures that the slots obtained in each batch allocation are contiguous.
[0036] In one embodiment that can be implemented in this application, such as Figure 2 As shown, it receives a batch allocation request containing the number of slots and obtains the CPU core identifier of the currently running thread, including: S201, Read the global free slot count of the single allocator and determine whether the global free slot count is greater than or equal to the number of slots; S202, if the global free slot count is greater than or equal to the number of slots, obtain the CPU core identifier of the currently running thread; S203, read the end pointer of the logical segment corresponding to the CPU core identifier, and calculate the number of remaining slots of the logical segment.
[0037] In this embodiment, the global free slot count is an atomic variable maintained by the memory allocator to record the total number of remaining free slots within the entire memory block. Each time a slot is allocated, the global free slot count atomically decreases by the corresponding amount; each time a slot is released, the global free slot count atomically increases by the corresponding amount. Reading the global free slot count is a lock-free operation because the read operation of an atomic variable is inherently atomic, and no data race will occur.
[0038] It's important to note that first checking if the global free slot count is greater than or equal to the requested number of slots is to quickly filter out unfulfillable allocation requests and avoid subsequent invalid atomic operations. If the global free slot count is less than the requested number of slots, it means there are not enough free slots in the entire memory block, and the allocation fails immediately, switching to the next memory block for allocation. This reduces unnecessary computational overhead and improves the overall performance of the allocator.
[0039] For example, when an upper-layer application requests a batch allocation of 10 memory slots, the memory allocator first reads the global free slot count. If the global free slot count is 5, which is less than 10, the allocation fails immediately. If the global free slot count is 20, which is greater than or equal to 10, the subsequent steps are executed. The CPU core identifier of the currently running thread is obtained as 2, and then the tail pointer of the logical slice corresponding to CPU core 2 is read as 1000. The upper limit offset of this logical slice is 2000, so the remaining number of slots is 2000 minus 1000, which equals 1000, which is greater than or equal to 10, satisfying the allocation requirement.
[0040] Specifically, the formula for calculating the remaining slots of a logical slice is: the remaining slots equal to the upper limit offset of the logical slice minus the current slice tail pointer value. The upper limit offset of a logical slice is the offset of the last slot of that logical slice plus 1, equal to the logical slice index plus 1 multiplied by the slice length. The slice length equals the total number of memory slots divided by the number of CPU cores, rounded down.
[0041] Furthermore, obtaining the CPU core identifier of the currently running thread is a lightweight operation, requiring only a read of a single CPU register value, taking only a few CPU cycles. Therefore, the overhead of the pre-verification step is minimal and will not significantly impact the overall performance of the allocator.
[0042] In one embodiment that can be implemented in this application, such as Figure 3 As shown, based on the CPU core identifier, an atomic addition operation is performed on the tail pointer of the corresponding logical slice within a single allocator to lock a continuous slot range, including: S301, if the number of remaining slots in the logical segment is greater than or equal to the number of slots, perform an atomic addition operation on the tail pointer of the segment to obtain the old tail pointer returned by the atomic addition operation; S302, determine whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical segment; S303, if the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical segment, lock a continuous slot interval starting from the old tail pointer and with a length equal to the number of slots.
[0043] In this embodiment, the old tail pointer returned by the atomic addition operation is the value of the fragment tail pointer before the addition operation. This value is the starting offset of the continuous slot range allocated this time. Determining whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical fragment is to ensure that the slot range allocated this time does not exceed the range of the current logical fragment. If it does exceed the limit, it means that although the atomic addition operation was successfully executed, the locked slot range crossed the boundary of the logical fragment, and it is necessary to roll back this atomic addition operation and try other logical fragments again.
[0044] It's important to note that although the remaining slot count of the logical shard was calculated before the atomic addition operation, and it was confirmed that the remaining slot count was greater than or equal to the requested slot count, other threads might have also performed atomic addition operations on the same shard tail pointer between the calculation of the remaining slot count and the execution of the atomic addition operation. This could cause the shard tail pointer value to increase, resulting in a decrease in the remaining slot count. Therefore, after performing the atomic addition operation, it is necessary to verify again whether the locked slot range is within the logical shard's range.
[0045] For example, the current value of the shard tail pointer of the logical shard corresponding to CPU core 2 is 1995. The upper-layer application requests batch allocation of 10 slots. The remaining number of slots is calculated as 2000 minus 1995, which equals 5, less than 10. Therefore, the cross-shard switching process will be initiated. If the upper-layer application requests batch allocation of 5 slots, the remaining number of slots is calculated as 5, which equals 5. An atomic addition operation is performed, incrementing the shard tail pointer by 5, and returning the old tail pointer of 1995. It is determined that 1995 plus 5 equals 2000, which is equal to the upper limit offset of the logical shard of 2000. The verification is successful, and the slot range of 1995 to 2000 is locked. The specific values in the above example of this application can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the purpose of this application is achieved. This application does not limit this.
[0046] Specifically, if the verification fails, it means that between calculating the remaining slot count and performing the atomic addition operation, another thread allocated some slots for this logical shard, causing the allocated slot range to exceed the logical shard's range. In this case, the requested slot count needs to be atomically subtracted from the shard's tail pointer, the atomic addition operation needs to be rolled back, and then the cross-shard switching process needs to be initiated to try other logical shards.
[0047] Furthermore, locking a contiguous slot range does not require modifying any other data structures; it only requires updating the value of the end-of-slice pointer through an atomic addition operation. This is because the unidirectional incrementing nature of the end-of-slice pointer ensures that the range will not be allocated by other threads, thus the locking operation is atomic and requires no additional synchronization mechanisms.
[0048] In one embodiment that can be implemented in this application, such as Figure 4 As shown, it also includes: S401, if the number of remaining slots in the logical segment is less than the number of slots, traverse the other logical segments in the order of CPU core identifiers. S402, perform an atomic addition operation on the end pointer of the logical slice that has been traversed to obtain the corresponding old end pointer; S403, determine whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical segment; S404, if the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical segment, lock the continuous slot interval starting from the old tail pointer and with a length equal to the number of slots.
[0049] In this embodiment, traversing other logical slices in CPU core identifier order means starting from the next core after the current CPU core identifier and sequentially traversing all logical slices corresponding to all CPU cores until a logical slice with enough remaining slots is found, or all logical slices have been traversed. This traversal order ensures fair allocation and avoids some logical slices being idle for a long time while others are overused.
[0050] It should be noted that when traversing other logical partitions, the operations performed on each logical partition are the same as those performed on the logical partition corresponding to the current CPU core. First, the partition's tail pointer is read, and the number of remaining slots is calculated. If the number of remaining slots is greater than or equal to the requested number of slots, an atomic addition operation is performed. Then, it is verified whether the locked slot range is within the range of this logical partition. If the verification passes, the slot range is locked, and the traversal ends; if the verification fails, the traversal continues to the next logical partition.
[0051] For example, the current CPU core identifier is 2, and the remaining number of slots in the corresponding logical slice is 3. The upper-layer application requests batch allocation of 5 slots. The logical slice corresponding to CPU core 3 is traversed sequentially. Its slice tail pointer is read as 3000, the upper limit offset is 4000, and the remaining number of slots is 1000, which is greater than or equal to 5. An atomic addition operation is performed, returning the old tail pointer 3000. It is determined that 3000 plus 5 equals 3005, which is less than or equal to 4000. The verification passes, the slot range 3000 to 3005 is locked, and the traversal ends. The specific values in the above example can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the inventive purpose of this application is achieved. This application does not limit these values.
[0052] Specifically, if not enough remaining slots are found after traversing all regular logical slices, a global spare slice will be used for allocation. A global spare slice consists of slots less than one slice remaining after dividing the total number of memory slots by the number of CPU cores; it serves as a fallback allocation when all logical slices are exhausted. The global spare slice also maintains its own tail pointer, and the allocation process is the same as for regular logical slices.
[0053] Furthermore, all operations during cross-shard switching are lock-free, with thread safety ensured only by atomic addition operations and verification steps. When multiple threads perform cross-shard switching simultaneously, no data races occur because the tail pointer of each logical shard is an independent atomic variable, and the atomic addition operation is thread-safely guaranteed by the CPU hardware.
[0054] In one embodiment that can be implemented in this application, such as Figure 5 As shown, the bit state of the corresponding slot interval in the bitmap of the single allocator is updated, and the continuous physical address corresponding to the slot interval is returned, including: S501, in the non-critical unlocked state, set the bit of the corresponding slot interval in the bit map to the occupied state; S502, calculate the continuous physical address corresponding to the slot interval based on the starting offset of the slot interval and the physical base address of the single allocator; S503, return the continuous physical address.
[0055] In this embodiment, the lock-free state of the non-critical section means that the operation does not need to hold any locks and will not cause data competition with the operations of other threads. This is because the locked slot range has been exclusively allocated to the current thread through an atomic addition operation of the fragment tail pointer, and the fragment tail pointer of any other thread will not point to this range, so no other thread will modify the map bit corresponding to this range.
[0056] It should be noted that moving bitmap updates out of the critical section is a key design feature of this application. In existing technologies, bitmap updates must be performed within a lock, because if performed outside the lock, other threads might simultaneously modify the same bitmap, leading to data races. However, in this application, since the slot range is exclusively locked via an atomic addition operation of the fragment tail pointer, bitmap updates can be safely performed outside the critical section without data races. This shortens the length of the critical section and improves the allocator's concurrency performance.
[0057] For example, after successfully locking slot range 1000 to 1010, in a non-critical, unlocked state, bits 1000 to 1009 of the bitmap are set to 1 sequentially. Then, based on the physical base address of the memory block (0x10000000) and the slot size (64 bytes), the contiguous physical address is calculated as 0x10000000 plus 1000 multiplied by 64, which equals 0x1000F000. Finally, this contiguous physical address is returned to the upper-layer application.
[0058] Specifically, the formula for calculating contiguous physical addresses is: contiguous physical address equals the physical base address of the memory block plus the starting offset multiplied by the slot size. The slot size is determined during memory block initialization and is consistent with the size class corresponding to that memory block. Each size class corresponds to a fixed slot size, usually a power of 2, such as 64 bytes, 128 bytes, 256 bytes, etc.
[0059] Furthermore, the returned contiguous physical addresses are accessible via Remote Direct Memory Access (RDMI), allowing upper-layer applications to directly use these addresses for RDMI read and write operations. Because the slots are contiguous, upper-layer applications can merge accesses to multiple contiguous slots into a single RDMI job request, using a scatter-gathering mechanism for transmission, thus improving RDMI transmission performance.
[0060] In one embodiment that can be implemented in this application, such as Figure 6 As shown, in the unlocked state of the non-critical section, setting the bit of the corresponding slot interval in the bitmap to the occupied state includes: S601, Obtain the epoch number of the current allocation period, wherein the epoch number is an incrementing integer; S602, write the epoch number into the reserved bit of the corresponding slot interval in the bitmap; S603, set the occupied bit of the corresponding slot interval in the bitmap to the occupied state.
[0061] In this embodiment, the cycle number is a globally incrementing integer, increasing by 1 with each allocation cycle. The allocation cycle refers to the entire time period from memory block initialization to memory block release, or the time period during which defragmentation is performed after the memory block is completely filled. Reserved bits in the bitmap refer to the unused higher bits in each slot of the bitmap, excluding the least significant bit used to mark the occupancy status. For example, each slot corresponds to a 64-bit bitmap word, with the least significant bit used to mark the occupancy status and the higher 63 bits reserved.
[0062] It's important to note that the periodicity is introduced to address timing consistency issues in extreme scenarios. In some extreme cases, a slot might be incorrectly marked as free before the bitmap update is complete. For example, thread A locks slots 1000 to 1010 and is updating the corresponding bits in the bitmap. At this point, thread B releases slot 1000, clearing the 1000th bit of the bitmap. If thread A's bitmap update is later than thread B's release operation, the already released slot will be marked as occupied again, leading to a memory leak.
[0063] For example, the current allocation cycle number is 10. After successfully locking slot range 1000 to 1010, in the unlocked state of the non-critical section, the cycle number 10 is first written into the reserved bits corresponding to bits 1000 to 1009 in the bitmap, and then the least significant bit of these bits is set to 1. When releasing slot 1000, it is first checked whether the cycle number in the reserved bits of that slot is consistent with the cycle number of the current allocation cycle. If they are consistent, it means that the slot was allocated in the current allocation cycle and can be safely released; if they are inconsistent, it means that the slot is residual data from the previous allocation cycle and does not need to be processed.
[0064] Specifically, the check steps when releasing a slot are as follows: read the entire word at the corresponding offset in the bitmap, and extract the cycle number and the occupancy status of the least significant bit from the reserved bits. If the occupancy status is occupied and the cycle number matches the cycle number of the currently allocated cycle, set the least significant bit to the idle state and atomically increment the global idle slot count. Otherwise, no operation is performed.
[0065] Furthermore, the increment operation of the period number is atomic and maintained by a global atomic variable. At the beginning of each allocated period, the atomic variable increments the period number. This ensures that period numbers from different allocated periods will not be repeated, avoiding misjudgments.
[0066] In one embodiment that can be implemented in this application, it further includes: S701, detect whether the end pointer of the logical segment has reached the upper limit offset of the corresponding logical segment; S702, if the end pointer of the segment reaches the upper limit offset of the corresponding logical segment, read the lowest free bit offset in the logical segment; S703, the fragment tail pointer is reset to the lowest free bit offset through an atomic comparison swap operation.
[0067] In this embodiment, detecting whether the end pointer of a logical shard has reached the upper limit offset of the corresponding logical shard is performed after each allocation operation or periodically in the background. If the end pointer reaches the upper limit offset, it means that all slots in the logical shard have been allocated and need to be reset so that the free slots that have been released in the logical shard can be reused.
[0068] It's important to note that the dynamic reset mechanism of the fragment tail pointer is crucial for ensuring high memory utilization. Without this mechanism, once the fragment tail pointer reaches its upper offset, even if there are many free slots within that logical fragment, they cannot be reallocated, leading to decreased memory utilization. By resetting the fragment tail pointer to the lowest free slot offset within that logical fragment, these free slots can be reused, improving memory utilization.
[0069] For example, the tail pointer of the logical slice corresponding to CPU core 2 is 2000, reaching the upper limit offset of 2000 for that logical slice. At this point, the bitmap within the logical slice is scanned, and the lowest free bit offset is found to be 500. An atomic compare-swap operation is used to change the value of the tail pointer from 2000 to 500. If the atomic compare-swap operation succeeds, subsequent allocation operations will start from offset 500, reusing the free slots already released within that logical slice.
[0070] Specifically, reading the lowest free bit offset within a logical slice can be achieved by scanning the bitmap range corresponding to that logical slice. Starting from the beginning offset of the logical slice, scan each bit of the bitmap sequentially, finding the first bit that is 0; its offset is the lowest free bit offset. To reduce scanning overhead, a cache of the lowest free bit offset for each logical slice can be maintained, and this cache can be updated each time a slot is released.
[0071] Furthermore, atomic comparison-swap operations ensure that resetting the fragment tail pointer is thread-safe. The fragment tail pointer is only modified to the lowest free bit offset of the new value if its current value equals the upper limit offset of the expected value. If another thread modifies the value of the fragment tail pointer during the reset process, the atomic comparison-swap operation will fail, and the state of the fragment tail pointer needs to be rechecked.
[0072] In one embodiment that can be implemented in this application, such as Figure 8 As shown, it also includes: S801, Receive a slot release request, the slot release request including the offset of the slot to be released; S802, set the occupied bit at the corresponding offset in the bitmap to an idle state; S803, atomically increases the global free slot count of the single allocator.
[0073] In this embodiment, the slot release request is initiated by the upper-layer application and includes the starting offset and length of the slot to be released. After receiving the release request, the memory allocator determines the slot range to be released based on the starting offset and length, then sets the corresponding bits in the bitmap to the free state, and simultaneously increments the global free slot count.
[0074] It should be noted that the release logic in this application is fully compatible with the release logic of existing memory allocators and requires no modification. This is because the allocation logic in this application only changes the locking method of the slots, without altering the structure and meaning of the bitmap. Each bit in the bitmap still corresponds to the occupancy status of a slot, with 1 indicating that it is occupied and 0 indicating that it is free. Therefore, the release logic of existing systems can be directly reused without modifying the code of the upper-layer application.
[0075] For example, the upper-layer application releases slot range 1000 to 1010. Upon receiving the release request, the memory allocator sets bits 1000 through 1009 in the bitmap to 0 sequentially. Then, it atomically increments the global free slot count by 10. After the release is complete, these slots can be reused by subsequent allocation requests.
[0076] Specifically, if the released slot range is contiguous and falls within the range of a logical shard, these released slots will be reused when the shard tail pointer of that logical shard reaches its upper limit offset. If the released slot range spans multiple logical shards, each logical shard will process the released slots within its own range.
[0077] Furthermore, the release operation is lock-free, updating the global free slot count only through atomic operations. When multiple threads release slots simultaneously, no data race occurs because the bit corresponding to each slot is only modified by the thread that allocated the slot and the thread that released the slot; no other thread can modify the same bit at the same time.
[0078] In one embodiment that can be implemented in this application, such as Figure 9 As shown, it also includes: S901, if the number of slots is greater than the number of remaining slots in a single logical segment and greater than the number of remaining slots in a single allocator, traverse multiple consecutive single allocators in physical address order. S902 performs an atomic addition operation on the end pointer of the corresponding logical slice in each traversed single allocator to lock a continuous slot range across multiple single allocators; S903, update the bit state of the corresponding slot interval in the bitmap of each single allocator, and return the continuous physical address across multiple single allocators.
[0079] In this embodiment, multiple consecutive single memory blocks refer to multiple memory blocks with contiguous physical addresses. During initialization, the memory allocator divides a large contiguous physical memory segment registered for remote direct memory access into multiple consecutive memory blocks of fixed size. Therefore, the physical addresses of these memory blocks are contiguous, and the end address of the previous memory block is equal to the start address of the next memory block.
[0080] It should be noted that when the number of slots allocated in a batch exceeds the capacity of a single memory block, allocation needs to be performed across multiple memory blocks. In existing technologies, slots allocated across memory blocks are not contiguous in physical address, making remote direct memory access (RDA) scattering, aggregation, and merging impossible. However, in this application, since the physical addresses of multiple memory blocks are contiguous, by traversing the contiguous memory blocks in physical address order and locking contiguous slot intervals within each memory block, a contiguous physical address interval across multiple memory blocks can be obtained, satisfying the requirements for RDA scattering, aggregation, and merging.
[0081] For example, a single memory block is 16MB in size, with a slot size of 64 bytes, resulting in a total of 262,144 slots. An upper-layer application requests a batch allocation of 500,000 slots, exceeding the capacity of a single memory block. The system traverses consecutive memory blocks 1, 2, and 3 in physical address order. It locks 262,144 consecutive slots in memory block 1 and 237,856 consecutive slots in memory block 2. Since the physical addresses of memory blocks 1 and 2 are contiguous, these two intervals are also contiguous, forming a contiguous physical address interval of 500,000 slots.
[0082] Specifically, when allocating across multiple memory blocks, the allocation process within each memory block is the same as that within a single memory block. The corresponding logical slice is located based on the current CPU core identifier, an atomic addition operation is performed to lock the contiguous slot range, the bitmap is updated, and the physical address is returned. Concatenating the contiguous physical address ranges within multiple memory blocks yields the contiguous physical addresses spanning those memory blocks.
[0083] Furthermore, the release logic for slots allocated across memory blocks is the same as that for slots within a single memory block; each memory block handles the release of slots within its own scope. After release, these slots can be reused by subsequent allocation requests.
[0084] The second aspect of this application proposes a distributed memory pool batch allocation system, such as... Figure 10 As shown, it includes: Request processing module 1001 is used to receive a batch allocation request containing the number of slots and obtain the CPU core identifier of the currently running thread; Slab memory management module 1002 is used to perform atomic addition operation on the shard tail pointer of the corresponding logical shard in the single allocator according to the CPU core identifier, and lock the continuous slot range. The single allocator is divided into multiple logical shards, each logical shard corresponds to a CPU core, and each logical shard maintains an independent shard tail pointer. The shard tail pointer is an atomic variable that records the offset of the highest allocated bit in the corresponding logical shard. Bitmap update module 1003 is used to update the bit state of the corresponding slot interval in the bitmap of the single allocator, complete the batch continuous slot allocation and return the continuous physical address corresponding to the slot interval.
[0085] In this embodiment, taking an example of a distributed inference KV cache allocation scenario for a large language model on an 8-core CPU server, the complete execution process of this solution is described in detail. A single memory block is 16MB in size, with a size class of 64 bytes and a total of 262,144 slots. It is divided into 8 logical shards based on the 8 CPU cores, each shard having 32,768 slots. The initial value of the shard tail pointer for each logical shard is cpu_id multiplied by 32,768. The initial value of the global free slot count is 262,144, and the bitmap is all 0s. The current allocation cycle number is 10.
[0086] Inference thread 1, running on CPU core 0, initiates a request to allocate 4 slots in batches. Upon receiving the request, the request processing module reads the global free slot count as 262144, which is greater than or equal to 4. It obtains the CPU core identifier of the current thread as 0. It reads the tail pointer of the logical slice corresponding to CPU core 0 as 0, calculates the remaining slot count as 32768 minus 0, which equals 32768, greater than or equal to 4. It calls the atomic locking module to perform an atomic addition operation on the tail pointer, incrementing it by 4, and returns the old tail pointer 0. It checks if 0 plus 4 equals 4, which is less than or equal to 32768; if the verification passes, it locks the slot range 0 to 4. It calls the bitmap update module, and in a non-critical, unlocked state, writes the cycle number 10 into the reserved bits of bits 0 to 3 of the bitmap, then sets the least significant bit of these bits to 1. It calls the address return module, calculates the continuous physical address 0x10000000 based on the physical base address of the memory block (0x10000000) and the starting offset (0), and returns it to thread 1.
[0087] Simultaneously, inference thread 2 running on CPU core 1 initiates a request to allocate 8 slots in batches. The request processing module reads the global free slot count as 262140, which is greater than or equal to 8. It obtains the CPU core identifier as 1. It reads the fragment tail pointer as 32768, indicating that the remaining slot count is 32768, which is greater than or equal to 8. The atomic locking module performs an atomic addition operation, incrementing by 8, and returns the old tail pointer 32768. It verifies that 32768 plus 8 equals 32776, which is less than or equal to 65536, and locks the slot range from 32768 to 32776. The bitmap update module updates the corresponding bit status, and the address return module calculates the physical address as 0x10000000, adds 32768 multiplied by 64, which equals 0x10080000, and returns it to thread 2.
[0088] Thread 1 continuously initiates batch allocation requests until the tail pointer of the logical slice corresponding to CPU core 0 increases to 32768, reaching the upper limit offset. At this point, Thread 1 initiates another batch allocation request for 2 slots. The request processing module reads the global free slot count as 262144 - 4 - 8 - 32756 = 262076, which is greater than or equal to 2. The CPU core identifier is obtained as 0. The tail pointer of the slice is read as 32768, and the number of remaining slots is 0, which is less than 2. The atomic locking module traverses the logical slice corresponding to CPU core 1 in sequence, reads the tail pointer of the slice as 32776, and the number of remaining slots is 65536 - 32776 = 32760, which is greater than or equal to 2. An atomic addition operation is performed to increment by 2, and the old tail pointer 32776 is returned. It is verified that 32776 plus 2 equals 32778, which is less than or equal to 65536, thus locking the slot range from 32776 to 32778. The bitmap update module updates the corresponding bit status, and the address return module calculates the physical address as 0x10000000 plus 32776 multiplied by 64, which equals 0x10080800, and returns it to thread 1.
[0089] After a period of time, thread 1 releases the four previously allocated slots, ranging from 0 to 4. Upon receiving the release request, the request processing module sets the least significant bit of bits 0 to 3 of the bitmap to 0 and atomically increments the global free slot count by 4, making it 262080. At this point, the memory management module detects that the tail pointer of the logical slice corresponding to CPU core 0 is 32768, reaching the upper limit offset. It scans the bitmap of this logical slice and finds the lowest free bit offset at 0. Through an atomic comparison and swap operation, it modifies the tail pointer from 32768 to 0.
[0090] Inference thread 3, running on CPU core 2, initiates a request to allocate 3 slots in batches. The request processing module reads the global free slot count as 262080, which is greater than or equal to 3. It obtains the CPU core identifier as 2. It reads the fragment tail pointer as 65536, and the remaining slot count is 32768, which is greater than or equal to 3. At this time, the memory management module detects that the fragment tail pointer of the logical fragment corresponding to CPU core 0 has been reset to 0, and there are consecutive free segments 0 to 4 with a length greater than or equal to 3. The atomic locking module prioritizes using the logical fragment corresponding to CPU core 0, performs an atomic addition operation, increments by 3, and returns the old tail pointer 0. It verifies that 0 plus 3 equals 3, which is less than or equal to 32768, and locks the slot range 0 to 3. The bitmap update module updates the corresponding bit state, and the address return module calculates the physical address as 0x10000000 and returns it to thread 3.
[0091] When the upper-layer application needs to allocate 500,000 slots in batches, it exceeds the capacity of a single memory block. The memory management module provides a list of memory blocks 1, 2, and 3 with contiguous physical addresses. The atomic locking module first performs an atomic addition operation on the logical slice corresponding to CPU core 0 of memory block 1, locking 262,144 slots, ranging from 0 to 262,144. Then, it performs an atomic addition operation on the logical slice corresponding to CPU core 0 of memory block 2, locking 237,856 slots, ranging from 0 to 237,856. Since the physical base address of memory block 1 is 0x10000000 and the physical base address of memory block 2 is 0x11000000, the two physical address ranges are contiguous, with a total length of 500,000 slots. The address return module concatenates the two physical addresses into a contiguous physical address range from 0x10000000 to 0x11E84800 and returns it to the upper-layer application. The specific values in the examples above can be flexibly set by those skilled in the art according to actual needs in practical applications, as long as the inventive purpose of this application can be achieved. This application does not limit them in this regard.
[0092] Those skilled in the art will understand that this solution achieves lock-free batch contiguous allocation through logical partitioning and atomic addition of partition tail pointers, eliminating the overhead caused by lock contention. Simultaneously, global reuse of logical partitions and dynamic resetting of partition tail pointers ensure consistency between memory utilization and global lock allocation. This solution is fully compatible with the release logic of existing memory allocators and can be integrated into existing systems without intrusion, meeting the high-concurrency, low-latency KV cache access requirements in distributed inference scenarios for large language models.
[0093] It should be noted that the logical partitioning method described in the embodiments of this application is only one example, and this application is not limited thereto. In other embodiments, the length of the logical partitions may be unequal, or the size of the logical partitions may be dynamically adjusted according to the CPU core load. In addition, the cycle numbering mechanism described in the embodiments of this application can also be adjusted according to actual needs, for example, a shorter cycle number can be used, or other methods can be adopted to solve timing consistency problems.
[0094] Furthermore, the cross-memory block allocation method described in the embodiments of this application is only one example, and this application is not limited thereto. In other embodiments, other methods can be used to allocate contiguous physical addresses across multiple memory blocks, such as pre-allocating large contiguous segments of physical memory and then dynamically dividing them into multiple memory blocks as needed. These will not be elaborated upon here.
Claims
1. A method for batch allocation of memory in a distributed memory pool, characterized in that, include: Receive a batch allocation request containing the number of slots, and obtain the CPU core identifier of the currently running thread; According to the CPU core identifier, an atomic addition operation is performed on the fragment tail pointer of the corresponding logical fragment in the single allocator to lock the continuous slot range. The single allocator is divided into multiple logical fragments, each logical fragment corresponds to a CPU core, and each logical fragment maintains an independent fragment tail pointer. The fragment tail pointer is an atomic variable that records the offset of the highest allocated bit in the corresponding logical fragment. Update the bit state of the corresponding slot interval in the bitmap of the single allocator, complete the batch continuous slot allocation, and return the continuous physical address corresponding to the slot interval; Based on the CPU core identifier, perform an atomic addition operation on the end pointer of the corresponding logical slice within the single allocator to lock a continuous slot range, including: If the number of remaining slots in the logical shard is greater than or equal to the number of slots, perform an atomic addition operation on the shard tail pointer to obtain the old tail pointer returned by the atomic addition operation; Determine whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical fragment; If the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical segment, lock a continuous slot interval starting from the old tail pointer and with a length equal to the number of slots; The method further includes: If the number of remaining slots in the logical partition is less than the number of slots, traverse the other logical partitions in order of CPU core identifier; Perform an atomic addition operation on the end pointer of the logical slice that has been traversed to obtain the corresponding old end pointer; Determine whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical segment; If the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical segment, lock the continuous slot interval starting from the old tail pointer and with a length equal to the number of slots; Update the bit state of the corresponding slot interval in the bitmap of the single allocator, and return the continuous physical address corresponding to the slot interval, including: In the non-critical unlocked state, the bit in the corresponding slot interval of the bit map is set to the occupied state; Calculate the consecutive physical addresses corresponding to the slot interval based on the starting offset of the slot interval and the physical base address of the single allocator; Return the consecutive physical addresses.
2. The method according to claim 1, characterized in that, Receive a batch allocation request containing the number of slots, and obtain the CPU core identifier of the currently running thread, including: Read the global free slot count of the single allocator and determine whether the global free slot count is greater than or equal to the number of slots; If the global free slot count is greater than or equal to the number of slots, obtain the CPU core identifier of the currently running thread; Read the end pointer of the logical segment corresponding to the CPU core identifier, and calculate the number of remaining slots of the logical segment.
3. The method according to claim 1, characterized in that, In the unlocked state of the non-critical region, setting the bit of the corresponding slot interval in the bitmap to the occupied state includes: Obtain the cycle number of the current allocation cycle, where the cycle number is an incrementing integer; Write the cycle number into the reserved bit of the corresponding slot interval in the bitmap, and set the occupied bit of the corresponding slot interval in the bitmap to the occupied state.
4. The method according to claim 1, characterized in that, Also includes: Check whether the end pointer of the logical fragment has reached the upper limit offset of the corresponding logical fragment; If the end pointer of the segment reaches the upper limit offset of the corresponding logical segment, read the lowest free bit offset within the logical segment; The fragment tail pointer is reset to the lowest free bit offset by an atomic comparison swap operation.
5. The method according to claim 1, characterized in that, Also includes: Receive a slot release request, the slot release request containing the offset of the slot to be released; Set the occupied bit at the corresponding offset in the bitmap to an idle state; Atom increases the global free slot count of the single allocator.
6. The method according to claim 1, characterized in that, Also includes: If the number of slots is greater than the number of remaining slots in a single logical shard and greater than the number of remaining slots in a single allocator, traverse multiple consecutive single allocators in physical address order. Perform an atomic addition operation on the end pointer of the corresponding logical slice in each traversed single allocator to lock a continuous slot range across multiple single allocators; Update the bit state of the corresponding slot interval in the bitmap of each single allocator, and return the continuous physical address across multiple single allocators.
7. A distributed memory pool memory batch allocation system, characterized in that, include: The request processing module is used to receive batch allocation requests containing the number of slots and obtain the CPU core identifier of the currently running thread. The Slab memory management module is used to perform atomic addition operations on the tail pointer of the corresponding logical slice within a single allocator based on the CPU core identifier, thereby locking a continuous slot interval. The single allocator is divided into multiple logical slices, each corresponding to a CPU core. Each logical slice maintains an independent tail pointer, which is an atomic variable recording the offset of the highest allocated bit within the corresponding logical slice. The process of performing atomic addition operations on the tail pointer of the corresponding logical slice within the single allocator based on the CPU core identifier to lock a continuous slot interval includes: if the remaining number of slots in the logical slice is greater than or equal to the number of slots, performing atomic addition operations on the tail pointer to obtain the old tail pointer returned by the atomic addition operation; determining whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical slice; and if the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the logical slice, locking a continuous slot interval starting from the old tail pointer and with a length equal to the number of slots. The bitmap update module is used to update the bit state of the corresponding slot interval in the bitmap of the single allocator, complete the batch continuous slot allocation, and return the continuous physical address corresponding to the slot interval. In the non-critical section unlocked state, the bits of the corresponding slot interval in the bitmap are set to the occupied state; the continuous physical address corresponding to the slot interval is calculated according to the starting offset of the slot interval and the physical base address of the single allocator; and the continuous physical address is returned. The system is further configured to: if the number of remaining slots in the logical partition is less than the number of slots, traverse other logical partitions in the order of CPU core identifiers; perform atomic addition on the partition tail pointer of the traversed logical partition to obtain the corresponding old tail pointer; determine whether the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical partition; if the sum of the old tail pointer and the number of slots is less than or equal to the upper limit offset of the corresponding logical partition, lock a continuous slot interval starting from the old tail pointer and with a length equal to the number of slots.