A data sharing method of a memory engine and related products

Abstract

The application provides a data sharing method of a memory engine and related products. The method comprises the following steps: creating a shared memory pool when the memory engine is initialized; constructing a hierarchical free memory record structure of the shared memory pool; in response to receiving a storage request of a data row, obtaining a row width of the data row to be stored, determining a target level from the hierarchical free memory record structure according to the row width; determining a target free memory block corresponding to the target level, allocating a storage space from the target free memory block, and writing the data row to be stored into the allocated storage space. According to the scheme, the low-delay and high-throughput storage and access of the data row under the multi-process architecture are realized without the premise of transferring data in the operating system kernel buffer.

Description

Technical Field

[0001] This invention relates to database technology, and in particular to a data sharing method and related products for a memory engine. Background Technology

[0002] Database systems typically employ a multi-process architecture (such as a main process, worker processes, and background service processes), requiring frequent data exchange between processes, including data rows, indexes, and cache pages. However, communication methods such as pipes, message queues, sockets, and local files all suffer from multiple data copying issues: data needs to be repeatedly copied between user space and kernel space, resulting in high CPU overhead and latency, reducing the response speed of database transaction processing and data queries, and failing to meet the requirements of high-frequency, low-latency data read and write operations. Summary of the Invention

[0003] One objective of this invention is to improve the data read and write performance of database systems under a multi-process architecture.

[0004] A further objective of this invention is to reduce the number of data copies during inter-process data interaction, thereby reducing CPU overhead and latency.

[0005] Specifically, the present invention provides a data sharing method for a memory engine, comprising: A shared memory pool is created during memory engine initialization. The shared memory pool is used for multiple processes to share data. Each process accesses the shared memory pool by mapping the shared memory pool to a contiguous range of virtual addresses in its own virtual address space. A hierarchical free memory record structure is constructed for the shared memory pool. The hierarchical free memory record structure includes multiple levels, each level corresponding to a free memory block of a preset capacity. The hierarchical free memory record structure records the starting address and capacity of each free memory block, which is used to locate available free memory blocks when allocating storage space. The capacity of the free memory block of level i+1 is twice the capacity of the free memory block of level i. The capacity of the free memory block of level 1 is the minimum allocation granularity of the shared memory pool, and i is a positive integer greater than or equal to 1. In response to receiving a storage request for a data row, the row width of the data row to be stored is obtained, and a target level is determined from the hierarchical free memory record structure based on the row width, wherein the free memory block capacity corresponding to the target level is greater than or equal to the row width; Determine the target free memory block corresponding to the target level, allocate storage space from the target free memory block, and write the data row to be stored into the allocated storage space.

[0006] Optionally, creating a shared memory pool includes: Obtain preset shared memory pool configuration parameters, including initial capacity, capacity limit, and minimum allocation granularity; Based on the shared memory pool configuration parameters, determine the physical memory corresponding to the continuous virtual address range, and initialize the physical memory as the storage space of the shared memory pool.

[0007] Optionally, each process accesses the shared memory pool by mapping it to a contiguous range of virtual addresses within its own process's virtual address space, including: Create or obtain a shared memory object; The shared memory object is mapped to a contiguous range of virtual addresses in the virtual address space of each process, and the association between the contiguous virtual address range and the physical memory of the shared memory pool is established. The starting address of the contiguous virtual address range satisfies a preset alignment requirement, which includes at least one of page alignment, minimum allocation granularity alignment, and cache line alignment.

[0008] Optionally, constructing the hierarchical free memory record structure of the shared memory pool includes: The number of levels in the hierarchical free memory record structure is determined based on the total capacity of the shared memory pool and the minimum allocation granularity.

[0009] Optionally, after constructing the hierarchical free memory record structure of the shared memory pool, the method further includes: In response to allocating storage space from a target level and the target level having no available free memory blocks, starting from the target level, the system searches level by level in ascending order of level number until a level with free memory blocks is found. The free memory blocks are then split into two free memory blocks with capacities corresponding to the target level. When releasing a free memory block, determine whether there is another free memory block with the same address and capacity as the free memory block. If so, merge the two free memory blocks into a single free memory block with double the capacity and one level larger than the current level.

[0010] Optionally, determining the target free memory block corresponding to the target level includes: Each level corresponds to a free memory block linked list, which is used to record the address information of the free memory blocks at that level; If a free memory block exists in the free memory block list at the target level, a free memory block is obtained from the free memory block list as the target free memory block, and the target free memory block is removed from the free memory block list. Otherwise, starting from the target level, search level by level in ascending order of level number until a level with free memory blocks is found. The found free memory blocks are used as candidate free memory blocks. The candidate free memory blocks are then split into free memory blocks with capacities corresponding to the target level, and the resulting free memory block is used as the target free memory block.

[0011] Optionally, the candidate free memory block is progressively split into free memory blocks with capacities corresponding to the target level, including: The candidate free memory block is split into two sub-blocks of equal size; If the capacity of the sub-block is equal to the capacity of the free memory block corresponding to the target level, one of the sub-blocks is designated as the target free memory block, and the other sub-block is added to the free memory block linked list of the target level. If the capacity of the sub-block is greater than the capacity of the free memory block corresponding to the target level, one of the sub-blocks is added to the free memory block list of the level with half the capacity, and the splitting operation is recursively performed on the other sub-block until a free memory block with a capacity corresponding to the target level is obtained.

[0012] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the data sharing method of any of the above-described memory engines.

[0013] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of the data sharing method of any of the above-described memory engines.

[0014] According to another aspect of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a data sharing method of any memory engine.

[0015] The data sharing method of the memory engine of this invention creates a shared memory pool that can be shared by multiple processes. Each process maps the shared memory pool to its own virtual address space, allowing multiple processes to directly access the same physical memory for data reading and writing without the need for data transfer through the operating system kernel buffer. This avoids multiple copies of data between user mode and kernel mode, significantly reducing CPU overhead and data transmission latency. By constructing a hierarchical free memory recording structure, each level corresponds to a free memory block of a preset capacity, with the capacity increasing in a doubling relationship. When a data row storage request is received, the target level is determined based on the row width of the data row to be stored, and the smallest level memory block with a capacity greater than or equal to the row width is selected for allocation. This avoids memory waste caused by fixed-size memory blocks and improves the space utilization of the shared memory pool. The hierarchical free memory recording structure records the starting address and capacity of each free memory block, enabling quick location of available free memory blocks when allocating storage space without traversing the entire memory pool, thus improving memory allocation efficiency.

[0016] Furthermore, the data sharing method of the memory engine of the present invention establishes an association between virtual addresses and the physical memory of the shared memory pool by creating or acquiring shared memory objects and mapping these shared memory objects to the virtual address spaces of each process. Each process directly accesses the same physical memory through its own virtual address space without going through the operating system kernel buffer, avoiding multiple data copies between user mode and kernel mode in inter-process communication, significantly reducing CPU overhead and data transmission latency. The starting addresses of a continuous virtual address range are aligned according to preset alignment requirements, including at least one of page alignment, minimum allocation granularity alignment, and cache line alignment. The aligned starting addresses can match the memory management granularity of the operating system, the cache line size of the CPU, or the minimum allocation unit of the memory pool, reducing additional memory access overhead caused by address misalignment and improving data read and write efficiency.

[0017] Furthermore, the data sharing method of the memory engine of the present invention, when there are no available free memory blocks at the target level, searches for and obtains free memory blocks at higher levels with larger capacities, and then splits them into memory blocks of the required size. Since the splitting operation is only performed on demand during allocation, it avoids memory fragmentation and management overhead caused by pre-splitting all large blocks into smaller blocks, thereby improving memory utilization. When releasing a free memory block, it is determined whether there are adjacent free memory blocks with contiguous addresses and the same capacity. If so, they are merged into a free memory block of the next higher level with double the capacity. Since the merging operation can be performed recursively, by merging small blocks into large blocks in a timely manner, memory fragmentation is effectively reduced, ensuring that subsequent large-capacity allocation requests can be satisfied.

[0018] Furthermore, the data sharing method of the memory engine of this invention corresponds to a free memory block linked list for each level, used to record the address information of free memory blocks at that level. When a free memory block exists in the free memory block linked list of the target level, a free memory block is directly obtained from the linked list as the target free memory block, making the time complexity of the acquisition operation O(1), eliminating the need for traversal or searching, and greatly improving the memory allocation speed. Fast allocation when there is a free memory block at the target level, and ensuring successful allocation through step-by-step searching and splitting when there is no free memory block at the target level, ensures allocation efficiency in common scenarios and allocation success rate in extreme scenarios, improving the overall utilization of the shared memory pool.

[0019] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0020] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic flowchart of a data sharing method for a memory engine according to an embodiment of the present invention; Figure 2 This is a schematic flowchart illustrating the creation of a shared memory pool in a data sharing method of a memory engine according to an embodiment of the present invention; Figure 3 This is a schematic flowchart illustrating access to a shared memory pool in a data sharing method of a memory engine according to an embodiment of the present invention; Figure 4 This is a schematic flowchart illustrating the determination of a target free memory block in a data sharing method of a memory engine according to an embodiment of the present invention; Figure 5 This is a schematic flowchart illustrating the step-by-step splitting of a data sharing method for a memory engine according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] To ensure service stability and achieve parallel task processing and fault isolation, database systems commonly adopt a multi-process architecture. A multi-process architecture typically includes a main process responsible for global scheduling, worker processes responsible for executing business read and write requests, and background service processes handling tasks such as log flushing, data persistence, index maintenance, and cache eviction. During database operation, these processes do not work independently but need to continuously and frequently perform data interaction and state synchronization. The interaction content covers core business data rows, index data that retrieval depends on, cache pages for accelerated access, and critical data such as transaction status and metadata. The efficiency of data interaction between processes directly determines the overall read / write performance and transaction processing capabilities of the database.

[0022] However, existing database systems typically rely on conventional inter-process communication methods such as pipes, message queues, socket communication, and local file I / O for data exchange. These methods suffer from the drawback of multiple data copies, making them difficult to adapt to the high-frequency interaction characteristics of database business. All of these communication methods depend on the operating system kernel buffer for data transfer. The entire data transmission process involves multiple data copies between user space and kernel space: the sending process copies data from user space memory to the kernel buffer, and then the kernel buffer copies it to the receiving process's user space memory. The complete data exchange process involves two or more data copying operations.

[0023] Multiple data copying operations generate significant overhead in CPU computation and memory bandwidth, substantially increasing data transmission latency. For database systems, whether it's high-frequency user data queries, additions, modifications, and deletions, or continuous background transaction commits, log synchronization, and cache updates, frequent inter-process data interactions are required, amplifying the accumulated copy latency and resource overhead. This leads to increased processing time for individual database transactions, decreased throughput of batch data writes, and slow response times for queries on hot data. Performance bottlenecks become increasingly pronounced, especially in high-concurrency, high-volume business scenarios.

[0024] This embodiment provides a shared memory pool management scheme based on a hierarchical free memory record structure. It achieves low-latency, high-throughput storage and access of data rows in a multi-process architecture without requiring data to be transferred through the operating system kernel buffer.

[0025] Figure 1 This is a schematic flowchart of a data sharing method for a memory engine according to an embodiment of the present invention, such as... Figure 1 As shown, the data sharing method of this memory engine can generally include: Step S101: A shared memory pool is created during memory engine initialization. The shared memory pool is used for multiple processes to share data. Each process accesses the shared memory pool by mapping the shared memory pool to a continuous range of virtual addresses in its own virtual address space. Step S102: Construct a hierarchical free memory record structure for the shared memory pool. The hierarchical free memory record structure includes multiple levels, each level corresponding to a free memory block of a preset capacity. The hierarchical free memory record structure records the starting address and capacity of each free memory block, which is used to locate available free memory blocks when allocating storage space. The capacity of the free memory block at level i+1 is twice the capacity of the free memory block at level i. The capacity of the free memory block at level 1 is the minimum allocation granularity of the shared memory pool, and i is a positive integer greater than or equal to 1. Step S103: In response to receiving a storage request for a data row, obtain the row width of the data row to be stored, and determine the target level from the hierarchical free memory record structure based on the row width, wherein the free memory block capacity corresponding to the target level is greater than or equal to the row width. Step S104: Determine the target free memory block corresponding to the target level, allocate storage space from the target free memory block, and write the data row to be stored into the allocated storage space.

[0026] In this embodiment, during the memory engine initialization phase, the database system requests a contiguous physical memory region from the operating system to construct a shared memory pool. This shared memory pool is independent of the private address space of any single process and serves as a unified shared storage carrier for multi-process data interaction, thereby eliminating the user-space copy overhead of inter-process data transfer. Based on the operating system's virtual memory mapping mechanism, each business process maps the shared memory pool to a contiguous virtual address range in its own virtual address space through page tables. Relying on the address translation capabilities of the Memory Management Unit (MMU), concurrent access to the same physical memory resource by multiple processes is achieved. While ensuring global data consistency, the virtual address space isolation mechanism guarantees the independence and security of process access.

[0027] In this embodiment, after the shared memory pool is created, the database system initializes a hierarchical free memory record structure to achieve hierarchical management of free memory resources. This hierarchical free memory record structure employs a power-of-2 hierarchical mechanism, with each level corresponding to a free memory block of a preset capacity: the capacity of the first-level free memory block is the smallest allocation granularity of the memory pool, representing the smallest unit of memory allocation in the database system; for any positive integer i, the capacity of the (i+1)th level free memory block is twice that of the i-th level free memory block. This hierarchical free memory record structure maintains a free memory block index, recording the starting offset address and effective capacity of all free memory blocks in the shared memory pool, providing fast retrieval for memory allocation, achieving standardized and structured management of free resources, and balancing memory allocation efficiency with memory fragmentation control capabilities.

[0028] In this embodiment, in response to a received storage request for a data row, the byte size of the data row to be stored is parsed to determine the row width. Based on the row width, the hierarchical free memory record structure is traversed from the first level in ascending order of capacity. The target level is selected according to the minimum fit principle, that is, the level corresponding to the smallest memory block with a capacity greater than or equal to the data row width is selected to accurately fit the data storage requirements, thereby avoiding internal fragmentation caused by allocating small data to large free memory blocks and effectively improving the space utilization of the shared memory pool.

[0029] In this embodiment, after locking the target level, the database system retrieves available free memory blocks at that target level and allocates storage space of corresponding capacity based on the data row width. If there is remaining space after allocating the target memory block, the remaining space is assigned to the free memory block index of the corresponding level according to its capacity. Subsequently, the structured data rows are written to the allocated physical memory space. After verifying the integrity of the data write, the virtual address and capacity information of the storage space segment are returned to the business layer, completing a complete data row shared storage process.

[0030] Based on the above steps, by creating a shared memory pool that can be shared by multiple processes, each process maps the shared memory pool to its own virtual address space. This allows multiple processes to directly access the same physical memory for data reading and writing without needing to transfer data through the operating system kernel buffer, avoiding multiple copies of data between user mode and kernel mode, and significantly reducing CPU overhead and data transfer latency. By constructing a hierarchical free memory record structure, each level corresponds to a free memory block of a preset capacity, with the capacity increasing in a doubling relationship. When a data row storage request is received, the target level is determined based on the row width of the data row to be stored, and the smallest level memory block with a capacity greater than or equal to the row width is selected for allocation. This avoids memory waste caused by fixed-size memory blocks and improves the space utilization of the shared memory pool. The hierarchical free memory record structure records the starting address and capacity of each free memory block, enabling quick location of available free memory blocks when allocating storage space without traversing the entire memory pool, thus improving memory allocation efficiency.

[0031] Figure 2 This is a schematic flowchart illustrating the creation of a shared memory pool in a data sharing method of a memory engine according to an embodiment of the present invention, as shown below. Figure 2 As shown, creating a shared memory pool includes: Step S201: Obtain preset shared memory pool configuration parameters, including initial capacity, capacity limit and minimum allocation granularity; Step S202: Determine the physical memory corresponding to the continuous virtual address range according to the shared memory pool configuration parameters, and initialize the physical memory as the storage space of the shared memory pool.

[0032] In this embodiment, when the database system creates a shared memory pool, it first reads predefined global configuration items to load and parse the shared memory pool configuration parameters. These parameters include: initial pool capacity, maximum capacity limit, and minimum allocation granularity. The initial capacity determines the amount of physical memory requested at once during the initialization phase, ensuring sufficient read / write throughput for basic business operations. The maximum capacity is the threshold for dynamic expansion of the memory pool, limiting maximum physical memory usage and preventing memory overflow and resource exhaustion. The minimum allocation granularity is the smallest memory allocation unit supported by the memory pool, determining the precision of fine-grained memory partitioning and serving as the baseline granularity for subsequent hierarchical memory management. After completing parameter validation, null value detection, and compliance checks, the database system solidifies the shared memory pool configuration parameters into the memory pool control structure, serving as the basis for subsequent physical memory allocation, space initialization, and memory allocation management.

[0033] In this embodiment, based on the operating system's memory management mechanism, corresponding physical memory resources are matched for a preset continuous virtual address range, completing the page table binding mapping between virtual addresses and physical addresses. After the address mapping relationship is established, a standardized initialization operation is performed on the entire requested physical memory region, clearing dirty data in memory, unifying memory status identifiers, initializing memory control header information, and characterizing this segment of physical memory as a dedicated storage space for the shared memory pool.

[0034] Based on the above steps, the specifications of the shared memory pool can be flexibly adjusted according to business needs by using preset shared memory pool configuration parameters (initial capacity, maximum capacity, minimum allocation granularity). Determining the physical memory corresponding to a continuous range of virtual addresses based on the configuration parameters ensures the physical continuity of the shared memory pool, which is beneficial for improving CPU cache hit rate and memory access efficiency.

[0035] Figure 3 This is a schematic flowchart illustrating access to a shared memory pool in a data sharing method of a memory engine according to an embodiment of the present invention, as shown below. Figure 3 As shown, each process accesses the shared memory pool by mapping it to a contiguous range of virtual addresses within its own virtual address space, including: Step S301: Create or obtain a shared memory object; Step S302: Map the shared memory object to a contiguous range of virtual addresses in the virtual address space of each process, and establish the association between the contiguous virtual address range and the physical memory of the shared memory pool. The starting address of the contiguous virtual address range meets the preset alignment requirements, which include at least one of page alignment, minimum allocation granularity alignment, and cache line alignment.

[0036] In this embodiment, in response to a process's access request to the shared memory pool, a kernel-mode shared memory object handle is created or obtained. If the current process is the first process in the database system initialization process, the memory management interface provided by the operating system kernel is called based on the memory pool configuration parameters to create a shared memory core object bound to the physical memory storage space and generate a globally unique identifier handle to complete kernel-mode resource registration and permission initialization. If the current process is not the initial process, the kernel-mode shared memory resource is retrieved through the globally unique identifier, and the already created shared memory object handle is directly obtained without needing to request physical memory again, thereby reusing the same shared memory resource and ensuring the global consistency of the underlying physical memory data.

[0037] In this embodiment, based on the acquired shared memory object, the process performs user-mode virtual address mapping, allocates a contiguous virtual address range in its private virtual address space, establishes a mapping relationship between this virtual address range and the physical memory of the shared memory pool, and completes real-time conversion between virtual and physical addresses through the MMU. To ensure memory access efficiency, hardware cache hit rate, and memory allocation compliance, the starting address of the mapped contiguous virtual address range must meet a preset alignment specification, which includes at least one of page alignment, minimum allocation granularity alignment, and cache line alignment. Through standardized address alignment constraints, cross-page access, cache false sharing, and memory allocation out-of-bounds errors are avoided, enabling efficient, secure, and compliant read and write access to shared memory by multiple processes.

[0038] Based on the above steps, by creating or acquiring shared memory objects and mapping these objects to the virtual address spaces of each process, an association is established between virtual addresses and the physical memory of the shared memory pool. Each process directly accesses the same physical memory through its own virtual address space, without needing to go through the operating system kernel buffer, avoiding multiple data copies between user mode and kernel mode in inter-process communication, significantly reducing CPU overhead and data transfer latency. The starting addresses of a continuous virtual address range are aligned according to preset alignment requirements, including at least one of page alignment, minimum allocation granularity alignment, and cache line alignment. The aligned starting addresses can match the operating system's memory management granularity, the CPU's cache line size, or the minimum allocation unit of the memory pool, reducing additional memory access overhead caused by address misalignment and improving data read / write efficiency.

[0039] In the implementation of step S102, constructing the hierarchical free memory record structure of the shared memory pool includes: determining the number of levels of the hierarchical free memory record structure based on the total capacity of the shared memory pool and the minimum allocation granularity.

[0040] In this embodiment, during the initialization phase of the hierarchical free memory record structure, the database system uses the total capacity of the shared memory pool and the preset minimum allocation granularity as core input parameters to determine the total number of hierarchical levels. Based on the power-law hierarchical mechanism of shared memory pool 2, the minimum allocation granularity is used as the base capacity of the first-level memory block, increasing by a factor of 2 for each subsequent level. Through logarithmic operations and rounding up, the minimum number of levels that can completely cover the total capacity of the shared memory pool is calculated. This number of levels ensures that the capacity of the highest-level memory block is not less than the overall capacity of the memory pool, thus achieving power-law hierarchical coverage of all free memory resources while avoiding increased management overhead due to excessive level redundancy, achieving a balance between memory hierarchical management accuracy and scheduling performance.

[0041] In the implementation of step S102, after constructing the hierarchical free memory record structure of the shared memory pool, the method further includes: In response to the allocation of storage space from the target level and the absence of available free memory blocks at the target level, the search proceeds level by level from the target level in ascending order of level number until a level with free memory blocks is found. The free memory blocks are then split into two free memory blocks with capacities corresponding to the target level. When releasing a free memory block, check if there is another free memory block with the same address and capacity as the free memory block. If so, merge the two free memory blocks into a single free memory block with double the capacity and one level larger than the current level.

[0042] In this implementation, within the hierarchical allocation logic of the shared memory pool, once a business request determines the target level, if there are no available free memory resources at that target level, a progressive upward search is performed. Starting from the current target level, the memory lists of higher levels are traversed in ascending order of level number until a higher level with a free memory block is found. Based on the power-law hierarchical nature of memory pool 2, the found free memory block at a higher level is recursively split: the memory block is divided into two sub-blocks with half the capacity at each level until a free memory block matching the target level is obtained. During the splitting process, non-target sub-blocks generated at each level are added to the free memory block list of the corresponding level; the final sub-block matching the target level's capacity is used for this memory allocation. This achieves the sinking and reuse of free resources, solves the allocation failure problem caused by memory exhaustion at the current level, and improves the memory allocation success rate.

[0043] In this implementation, when memory space is released and free memory blocks are reclaimed to the hierarchical free memory record structure, the database system performs adjacent block merging verification. For the free memory block to be reclaimed, the same-level record structure is searched to determine if there are paired free memory blocks with contiguous physical addresses and the same capacity. If adjacent free blocks exist, the two free memory blocks with the same capacity and contiguous addresses are merged to generate a free memory block with doubled capacity and a higher-level hierarchical position. At the same time, the two old free block records at the original level are deleted, and the merged memory block record is added in the higher-level hierarchical structure to achieve memory fragmentation consolidation.

[0044] Based on the above implementation, when there are no available free memory blocks at the target level, free memory blocks are searched and obtained level by level from the larger capacity level. These blocks are then split into memory blocks of the required size level by level. Since the splitting operation is only performed on demand during allocation, memory fragmentation and management overhead caused by pre-splitting all large blocks into smaller blocks are avoided, thus improving memory utilization. When releasing a free memory block, it is determined whether there are adjacent free memory blocks with contiguous addresses and the same capacity. If so, they are merged into a free memory block of the next higher level with double the capacity. Since the merging operation can be performed recursively, memory fragmentation is effectively reduced by merging small blocks into large blocks in a timely manner, ensuring that subsequent large-capacity allocation requests can be satisfied.

[0045] Figure 4 This is a schematic flowchart illustrating the determination of a target free memory block in a data sharing method of a memory engine according to an embodiment of the present invention, as shown below. Figure 4 As shown, the target free memory blocks corresponding to the target level include: Step S401: Each level corresponds to a free memory block linked list, which is used to record the address information of the free memory blocks at each level. Step S402: If there is a free memory block in the target level free memory block list, obtain a free memory block from the free memory block list as the target free memory block, and remove the target free memory block from the free memory block list; Step S403: Otherwise, starting from the target level, search level by level in ascending order of level number until a level with free memory blocks is found. The found free memory blocks are taken as candidate free memory blocks. The candidate free memory blocks are split into free memory blocks with capacities corresponding to the target level, and the free memory block obtained after splitting is taken as the target free memory block.

[0046] In this embodiment, for each level of the hierarchical shared memory pool structure, the database system maintains a linked list of free memory blocks to manage all free memory resources at the corresponding level using a linked data structure. Each level's linked list node stores the starting offset address and block size of the corresponding free memory block, supporting fast traversal, querying, and removal of free memory blocks at that level. This hierarchical linked list mechanism physically isolates and manages free memory resources of different granularities hierarchically, serving as the data foundation for subsequent constant-time free block retrieval and allocation, avoiding the performance overhead of full memory traversal.

[0047] In this embodiment, after determining the target level, the database system accesses the free memory block list corresponding to the target level. If the free memory block list is not empty, it indicates that there are free memory resources of the appropriate specifications at the current level. An available free memory block is retrieved from the free memory block list as the target free memory block for this allocation. At the same time, the memory block node is removed from the corresponding level's free list, the free memory block list status is updated, and the memory block is marked as pending allocation to avoid duplicate allocation conflicts.

[0048] In this embodiment, if the target level's free memory list is empty and there are no available resources, an upward allocation process is executed. Starting from the target level, the free memory lists of higher levels are checked sequentially along the increasing level number until the first level with a free memory block is found, and this large-capacity memory block is obtained as a candidate free memory block. Based on the power-law hierarchical characteristic of memory pool 2, the candidate free block is split into two blocks at each level, generating two free memory blocks of the current level's standard capacity each time. This splitting continues until a free memory block matching the target level's capacity is obtained. One memory block with the matching capacity is selected as the target free memory block for this storage allocation, and the remaining free blocks generated by the splitting are sequentially added to the corresponding free memory lists at each level, completing the hierarchical resource completion.

[0049] Based on the above steps, each level corresponds to a linked list of free memory blocks, used to record the address information of free memory blocks at that level. When a free memory block exists in the linked list of the target level, a free memory block is directly retrieved from the list as the target free memory block, making the time complexity of the retrieval operation O(1), eliminating the need for traversal or searching, and greatly improving the memory allocation speed. Fast allocation when there is a free memory block at the target level, and ensuring successful allocation through step-by-step searching and splitting when there is no free memory block at the target level, guarantees allocation efficiency in common scenarios and allocation success rate in extreme scenarios, improving the overall utilization of the shared memory pool.

[0050] Figure 5 This is a schematic flowchart illustrating the step-by-step splitting of a data sharing method for a memory engine according to an embodiment of the present invention, as shown below. Figure 5 As shown, splitting candidate free memory blocks into free memory blocks of corresponding capacity to the target level includes: Step S501: Split the candidate free memory block into two sub-blocks of equal size; Step S502: If the capacity of the sub-block is equal to the capacity of the free memory block corresponding to the target level, one of the sub-blocks is designated as the target free memory block, and the other sub-block is added to the free memory block linked list of the target level. Step S503: If the capacity of a sub-block is greater than the capacity of the free memory block corresponding to the target level, one of the sub-blocks is added to the free memory block list of the level with half the capacity, and the splitting operation is recursively performed on the other sub-block until a free memory block with a capacity corresponding to the target level is obtained.

[0051] In this embodiment, for the high-level candidate free memory blocks retrieved, based on the power-order hierarchical characteristics of the shared memory pool 2, the candidate free memory blocks are split into two sub-blocks with contiguous physical addresses and equal capacity, each sub-block being half the capacity of the original candidate free memory block.

[0052] In this embodiment, after the split is completed, if the capacity of the sub-block is equal to the capacity of the free memory block corresponding to the target level, then the current split granularity has met the business allocation accuracy requirements. One of the sub-blocks is selected as the target free memory block for this allocation and used for data row storage space allocation; another sub-block of the same specification is added to the free memory block linked list corresponding to the target level to complete the free resource index of this level for reuse in subsequent allocations of memory of the same specification.

[0053] In this embodiment, if the capacity of the sub-block obtained by splitting is still greater than the capacity of the free memory block corresponding to the target level, it indicates that the memory granularity has not met the allocation requirements. One of the sub-blocks is classified into the free memory block linked list of the current corresponding halving level to complete resource archiving, so as to maintain the integrity of the hierarchical linked list resources. The splitting operation is recursively performed on the other sub-block, repeating the closed-loop process of capacity comparison, hierarchical archiving, and recursive splitting, until the capacity of the split sub-block completely matches the target level specification, thereby outputting a compliant target free memory block to complete the step-by-step sinking adaptation of large-granularity memory blocks.

[0054] Based on the above steps, a recursive splitting mechanism is used to obtain free memory blocks from higher levels and split them level by level only when no free memory blocks are available at the target level, rather than pre-splitting all free memory blocks into smaller sizes. This avoids free memory fragmentation and management overhead caused by premature splitting, improving the overall utilization of the shared memory pool. During the splitting process, free memory blocks that are not further split at each level are added to the free list of the corresponding level, allowing them to be directly used by subsequent allocation requests of the same level without having to be split again from higher levels. This distributes the cost of the splitting operation and improves the efficiency of subsequent allocations.

[0055] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0056] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0057] This embodiment also provides a computer program product 10, a computer-readable storage medium 20, and a computer device 30. Figure 6 This is a schematic diagram of a computer program product 10 according to an embodiment of the present invention. Figure 7 This is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. Figure 8 This is a schematic diagram of a computer device 30 according to an embodiment of the present invention. Figure 6 As shown, the computer program product 10 includes a computer program 11, which, when executed by the processor 32, implements the steps of the data sharing method of any of the above-described memory engines. Figure 7 As shown, a computer-readable storage medium 20 stores the aforementioned computer program 11, which, when executed by the processor 32, implements the steps of the data sharing method of the memory engine in any of the above embodiments. Figure 8 As shown, the computer device 30 may include a memory 31, a processor 32, and a computer program 11 stored on the memory 31 and running on the processor 32.

[0058] The computer program 11 used to perform the operations of this invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program 11 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer. In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing status information of the computer-readable program instructions.

[0059] For the purposes of this embodiment, computer program product 10 is a related product that includes computer program 11.

[0060] For the purposes of this embodiment, computer-readable storage medium 20 is a tangible device capable of holding and storing a computer program 11. It can be any device capable of containing, storing, communicating, propagating, or transmitting the program 11 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 20 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0061] Computer device 30 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 30 can be a cloud computing node. Computer device 30 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 30 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0062] Computer device 30 may include a processor 32 adapted to execute stored instructions and a memory 31 that provides temporary storage space for the operation of said instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0063] Computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows external devices that can be connected to the computer device to input and output data. The network adapter / interface provides communication between the computer device and a network, typically represented as a communication network.

[0064] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A data sharing method for a memory engine, characterized in that, include: A shared memory pool is created during memory engine initialization. The shared memory pool is used for multiple processes to share data. Each process accesses the shared memory pool by mapping the shared memory pool to a contiguous range of virtual addresses in its own virtual address space. A hierarchical free memory record structure is constructed for the shared memory pool. The hierarchical free memory record structure includes multiple levels, each level corresponding to a free memory block of a preset capacity. The hierarchical free memory record structure records the starting address and capacity of each free memory block, which is used to locate available free memory blocks when allocating storage space. The capacity of the free memory block of level i+1 is twice the capacity of the free memory block of level i. The capacity of the free memory block of level 1 is the minimum allocation granularity of the shared memory pool, and i is a positive integer greater than or equal to 1. In response to receiving a storage request for a data row, the row width of the data row to be stored is obtained, and a target level is determined from the hierarchical free memory record structure based on the row width, wherein the free memory block capacity corresponding to the target level is greater than or equal to the row width; Determine the target free memory block corresponding to the target level, allocate storage space from the target free memory block, and write the data row to be stored into the allocated storage space.

2. The data sharing method for a memory engine according to claim 1, characterized in that, Creating a shared memory pool includes: Obtain preset shared memory pool configuration parameters, including initial capacity, capacity limit, and minimum allocation granularity; Based on the shared memory pool configuration parameters, determine the physical memory corresponding to the continuous virtual address range, and initialize the physical memory as the storage space of the shared memory pool.

3. The data sharing method for a memory engine according to claim 1, characterized in that, Each process accesses the shared memory pool by mapping it to a contiguous range of virtual addresses within its own virtual address space, including: Create or obtain a shared memory object; The shared memory object is mapped to a contiguous range of virtual addresses in the virtual address space of each process, and the association between the contiguous virtual address range and the physical memory of the shared memory pool is established. The starting address of the contiguous virtual address range satisfies a preset alignment requirement, which includes at least one of page alignment, minimum allocation granularity alignment, and cache line alignment.

4. The data sharing method for a memory engine according to claim 1, characterized in that, The hierarchical free memory record structure for constructing the shared memory pool includes: The number of levels in the hierarchical free memory record structure is determined based on the total capacity of the shared memory pool and the minimum allocation granularity.

5. The data sharing method for a memory engine according to claim 1, characterized in that, After constructing the hierarchical free memory record structure of the shared memory pool, the following is also included: In response to allocating storage space from a target level and the target level having no available free memory blocks, starting from the target level, the system searches level by level in ascending order of level number until a level with free memory blocks is found. The free memory blocks are then split into two free memory blocks with capacities corresponding to the target level. When releasing a free memory block, determine whether there is another free memory block with the same address and capacity as the free memory block. If so, merge the two free memory blocks into a single free memory block with double the capacity and one level larger than the current level.

6. The data sharing method for a memory engine according to claim 1, characterized in that, Determining the target free memory block corresponding to the target level includes: Each level corresponds to a free memory block linked list, which is used to record the address information of the free memory blocks at that level; If a free memory block exists in the free memory block list at the target level, a free memory block is obtained from the free memory block list as the target free memory block, and the target free memory block is removed from the free memory block list. Otherwise, starting from the target level, search level by level in ascending order of level number until a level with free memory blocks is found. The found free memory blocks are used as candidate free memory blocks. The candidate free memory blocks are then split into free memory blocks with capacities corresponding to the target level, and the resulting free memory block is used as the target free memory block.

7. The data sharing method for a memory engine according to claim 6, characterized in that, The process of splitting the candidate free memory blocks into free memory blocks with capacities corresponding to the target level includes: The candidate free memory block is split into two sub-blocks of equal size; If the capacity of the sub-block is equal to the capacity of the free memory block corresponding to the target level, one of the sub-blocks is designated as the target free memory block, and the other sub-block is added to the free memory block linked list of the target level. If the capacity of the sub-block is greater than the capacity of the free memory block corresponding to the target level, one of the sub-blocks is added to the free memory block list of the level with half the capacity, and the splitting operation is recursively performed on the other sub-block until a free memory block with a capacity corresponding to the target level is obtained.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data sharing method of the memory engine according to any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data sharing method of the memory engine as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The system includes a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the data sharing method of the memory engine according to any one of claims 1 to 7.

Images