A block sharing based unikernel memory deduplication method and system
By constructing a global dependency graph and generating position-independent code using binary orchestration techniques, combined with a hypervisor-level shared block cache pool and direct mapping mechanism, the problems of Unikernel memory redundancy and performance overhead are solved, achieving efficient memory sharing and optimized startup performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-02-09
- Publication Date
- 2026-06-26
AI Technical Summary
The existing Unikernel memory deduplication technology suffers from severe memory redundancy, high performance overhead, long latency, and memory fragmentation in multi-instance deployments, and cannot effectively utilize resources, especially in cloud computing scenarios.
By constructing a global dependency graph to identify shared block boundaries, using binary orchestration technology to generate position-independent code, and combining a hypervisor-level shared block cache pool and a direct mapping mechanism, zero-latency memory sharing is achieved.
It significantly reduces memory resource consumption, improves memory utilization, optimizes startup performance, eliminates runtime overhead, and resolves the deduplication mechanism failure issue caused by Unikernel static linking.
Smart Images
Figure CN122285255A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cloud computing and microservice architecture, and in particular to a Unikernel memory deduplication method and system based on block sharing. Background Technology
[0002] With the evolution of cloud computing technology, microservice architecture and serverless computing are becoming increasingly popular, placing extremely high demands on the startup speed, resource density, and security of the underlying execution environment. Unikernel, a lightweight operating system architecture designed specifically for cloud environments, constructs a virtual machine image in a single address space by statically linking application code with operating system kernel functional components (such as the file system and network protocol stack). This architecture eliminates the overhead of switching between user mode and kernel mode in traditional operating systems and strips away redundant system services, thus possessing significant advantages such as millisecond-level startup, extremely low memory consumption, and high performance, making it a crucial technology for building high-density cloud infrastructure.
[0003] In real-world deployments, to maximize hardware resource utilization, physical servers typically need to run hundreds or thousands of Unikernel instances simultaneously. Since these instances often rely on the same build system or depend on the same underlying libraries, they contain a large amount of duplicate binary code and read-only data, leading to significant memory redundancy. A large amount of valuable memory resources are occupied by redundant data, severely offsetting the low-memory advantage of Unikernel itself and contradicting the core goal of cloud computing: pursuing ultimate resource efficiency. Memory deduplication techniques are commonly used to address this redundancy problem. Among these, the most representative solutions are memory deduplication based on page content scanning and auxiliary deduplication techniques that optimize Unikernel layout.
[0004] Dynamic deduplication techniques based on page content scanning are typically exemplified by the Kernel-Single Page Merging (KSM) mechanism in the Linux kernel. This approach runs a background daemon in the host operating system that periodically scans anonymous pages in physical memory. The system identifies identical physical pages by calculating the hash value of their content and comparing it byte-by-byte. Once duplicate pages are found, the system merges them, maps them to the same physical page frame, and marks the page as copy-on-write, thereby freeing up redundant physical memory.
[0005] A representative deduplication technique for layout optimization in Unikernel is Spacer. This approach addresses the issue of inconsistent code segment locations caused by static linking in Unikernel by intervening during the linking phase. Spacer mandates that every library file in the Unikernel image must be aligned to the physical page boundary (typically 4KB) during memory loading. Through this forced alignment, Spacer aims to ensure that the same library occupies the same page offset across different images, thus assisting the underlying KSM mechanism in more easily scanning and identifying identical memory pages.
[0006] However, existing technologies have the following main drawbacks:
[0007] (1) Due to the reuse of the Unikernel library, there is severe physical memory redundancy among multiple instances, and existing deduplication techniques fail for statically linked Unikernels. Dynamic deduplication techniques based on page content scanning will only merge and map these pages onto the same physical page frame when they find that the binary content of two or more physical pages is completely identical. Therefore, the premise for this mechanism to be effective is that the memory pages to be shared must be completely identical at the bit level. However, Unikernel is statically linked as a whole, which means that any slight inconsistency in content can lead to the variation of the originally identical content at the binary level. This is specifically reflected in the following two aspects:
[0008] Page offset inconsistency: Because different Unikernel instances may contain different library combinations or use different linking orders, even identical code libraries may have inconsistent starting offsets in the image file and memory segments. Since KSM strictly compares based on fixed-size (4KB) physical page boundaries, misalignment of library code within a page can completely alter the binary arrangement of the entire page, making it unrecognizable as the same page.
[0009] Instruction-level address dependency: The static linking process resolves the jump targets of function calls and the access addresses of global data in the code into specific numerical values (absolute addresses or relative offsets). Because the memory layout of libraries differs across different Unix kernel images, the hard-coded address values in the code instructions vary. This means that even functions with identical logical functionality will ultimately generate different binary machine code, thus completely blocking content-based deduplication paths.
[0010] (2) Existing deduplication mechanisms introduce unacceptable runtime performance overhead.
[0011] Scan-based deduplication techniques rely on a background daemon to continuously scan, calculate hashes, and compare memory pages. This process continuously consumes host CPU resources, directly interfering with the performance of foreground applications (potentially causing a 10% performance drop in high-density scenarios). In other words, traditional memory deduplication solutions introduce additional CPU overhead and performance fluctuations during runtime, undermining the inherent high-performance advantages of Unikernel.
[0012] (3) The existing deduplication mechanism has a delay of minutes in taking effect.
[0013] Scan-based memory deduplication schemes are essentially "post-hoc." The discovery and merging of shared memory occurs after the instance starts running, relying on slow, periodic scans by kernel threads. This process typically takes several minutes to reach a stable shared state (convergence). However, modern cloud-native applications such as Serverless functions have extremely short lifecycles (only seconds or even hundreds of milliseconds), often resulting in instance destruction before the deduplication mechanism takes effect. This "delayed" deduplication mechanism fails to function effectively in short-task scenarios, leading to a continuous waste of memory resources.
[0014] (4) Existing deduplication mechanisms suffer from a large amount of memory fragmentation.
[0015] To address the memory deduplication failure issue caused by static linking in Unikernel, Spacer employs a strategy of performing page alignment on all libraries. While this method forces the starting address of each library to fall on a page boundary, thus creating identical physical pages across different Unikernel instances, it also introduces significant memory fragmentation. This is because a considerable portion of Unikernel libraries are smaller than a single page (4KB). For a 3KB library, to align to a 4KB boundary, 1KB of invalid bytes must be padded after it, meaning that the alignment operation for just this one library wastes 25% of the space on its page. Summary of the Invention
[0016] To address the contradiction between low memory resource utilization and high system performance requirements when deploying Unikernel applications at high density in cloud computing scenarios, this invention proposes a block-sharing-based Unikernel memory deduplication method and system (BMS).
[0017] In a first aspect, the present invention provides a Unikernel memory deduplication method based on block sharing, comprising the following steps:
[0018] Query and feedback phase: BMS maintains a global dependency graph through a knowledge base. When building Unikernel X, it executes query and feedback logic. Through differential analysis, it maps the library set required by Unikernel to the global dependency graph, analyzes the cohesion of the library set to determine the boundary of shared blocks, and feeds back the new dependency pattern to the knowledge base to achieve incremental updates of the dependency graph.
[0019] Binary orchestration stage: After determining the boundaries of the shared block, the orchestrator merges the read-only code segment and the read-only data segment through a compact aggregation strategy, converts them into position-independent code through the address-independent code feature of the PIC compiler, and marks them as shared regions; at the same time, a private global offset table is introduced, which allows the shared code to access fixed offsets, while the private global offset table is located in the private data segment, thereby decoupling the shared code from the Unikernel memory layout;
[0020] Image packaging stage: The build toolchain links the application's private code with the private data segment to generate a Unikernel image;
[0021] Load-time sharing phase: Zero-latency memory sharing is achieved through a Hypervisor-level shared block cache pool and direct mapping mechanism, optimizing startup performance and memory usage.
[0022] Secondly, the present invention provides a Unikernel memory deduplication system based on block sharing, comprising:
[0023] The query and feedback module is used by BMS to maintain a global dependency graph through a knowledge base. When building Unikernel X, it executes query and feedback logic, maps the library set required by Unikernel to the global dependency graph through differential analysis, analyzes the cohesion of the library set to determine the boundary of shared blocks, and feeds back the new dependency pattern to the knowledge base to realize incremental updates of the dependency graph.
[0024] The binary orchestration module, after determining the boundaries of the shared block, uses a compact aggregation strategy to merge read-only code segments and read-only data segments, converts them into position-independent code using the address-independent code characteristics of the PIC compiler, and marks them as shared regions; at the same time, a private global offset table is introduced, which allows shared code to access fixed offsets, while the private global offset table is located in the private data segment, thereby decoupling shared code from Unikernel memory layout;
[0025] The image encapsulation module is used to build a toolchain that links the application's private code with the private data segment to generate a Unikernel image;
[0026] The module is shared during loading, and zero-latency memory sharing is achieved through a Hypervisor-level shared block cache pool and direct mapping mechanism, optimizing startup performance and memory usage.
[0027] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described block-shared Unikernel memory deduplication method.
[0028] The beneficial effects of this invention are as follows: This invention significantly reduces memory resource consumption for Unikernel on the same physical server without compromising application performance, and achieves extremely high memory deduplication efficiency. By using binary object orchestration to eliminate page offsets and address dependencies, this invention constructs standardized, reusable shared components from the source, effectively solving the problem of deduplication mechanism failure and non-shareability caused by Unikernel's static linking characteristics; it uses a load-time direct mapping mechanism to replace runtime scanning and copying, achieving zero latency and zero runtime overhead during memory deduplication; and it adopts block-based sharing, rather than finer-grained library-based sharing, effectively reducing memory fragmentation and further improving memory utilization. Attached Figure Description
[0029] To illustrate this application more clearly, the accompanying drawings used in this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0030] Figure 1 Flowchart of the memory deduplication method provided by the present invention;
[0031] Figure 2 This invention provides a flowchart for the identification of shared blocks.
[0032] Figure 3 The binary object arrangement diagram provided for this invention;
[0033] Figure 4 This is a diagram illustrating the zero-latency memory sharing architecture provided by the present invention. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0035] like Figure 1As shown, this invention proposes a Unikernel memory deduplication method based on block sharing, consisting of four closely connected stages, realizing a closed loop from static dependency analysis to dynamic, latency-free sharing. Specifically, it includes the following steps:
[0036] S1: Knowledge Base-Based Shared Block Recognition
[0037] Optionally, the workflow begins with component analysis of the target Unikernel. To maximize memory sharing opportunities between different Unikernel sets, such as... Figure 1 As shown in ①, the build system BMS introduces a knowledge base to identify shareable components. This approach goes beyond simple library matching, focusing instead on identifying functionally cohesive and reusable software build units. The core of the knowledge base maintains a global dependency graph reflecting the dependencies of all libraries within the entire software ecosystem (the Unikraft ecosystem). This graph is built offline through static analysis of build system files (Makefiles and KConfig used to build the Unikernel image).
[0038] like Figure 2 As shown, this knowledge base captures the intricate dependencies between all libraries, from the upper-level application library (lib-nginx) all the way down to the core system library (libc). When a new Unikernel build request is initiated, the BMS performs differential analysis and mapping of its required libraries against this global graph. This knowledge-driven process not only accurately identifies which components are reusable but also reveals their connections to the broader ecosystem. Furthermore, the knowledge base is designed to be incrementally updated; any new libraries or dependency patterns discovered during the analysis are fed back into the knowledge base, enabling it to continuously refine its understanding of shared opportunities.
[0039] Optionally, based on the results of dependency analysis, BMS employs a cohesion-based aggregation strategy to form standardized, versioned "atomic shared blocks." This strategy no longer treats each library as an independent unit, but rather combines a core library providing a public interface with all its private, internal dependencies into a single, indivisible block, ensuring that functionally tightly coupled components are always managed and shared uniformly. The output of this stage is a logical mapping that defines which requirements of a Unikernel can be satisfied by these standard shared blocks, laying the foundation for subsequent build and linking processes.
[0040] Optionally, when building Unikernel X, the system executes a "query and feedback" logic: First, it maps the set of libraries required by the Unikernel to the global dependency graph through differential analysis, accurately identifying which component combinations have high cohesion, thereby determining the boundaries of reusable shared blocks; Second, any new dependency patterns discovered during the building process are fed back to the knowledge base, enabling incremental updates to the dependency graph, allowing the system's understanding of sharing opportunities to continuously evolve with the development of the ecosystem.
[0041] S2: Binary Object Arrangement
[0042] Alternatively, once the shareable components are identified, the fundamental challenge remains: how to transform these logically related libraries into a physically shareable binary artifact. This requires addressing two core issues stemming from static linking: inconsistent page offsets and instruction-level address dependencies. BMS addresses this problem through a dedicated binary orchestration stage that employs customized compilation and linking techniques to generate self-contained, position-independent shared blocks.
[0043] Therefore, after determining the logical boundaries of the shared block, such as Figure 1 As shown in ②, the system enters the core binary orchestration stage. The orchestrator performs instruction-level refactoring on the identified shared blocks. For example... Figure 3 As shown in the shared block memory layout of the A-level module, the orchestrator employs a compact aggregation strategy. The linker script merges segments of the same type from all component libraries into a single, monolithic segment for that block. For example, all independent .text segments are concatenated into a single .text_shared segment. Only the final aggregated read-only segments (.text_shared, .rodata_shared) are designated for sharing and page-aligned. Writable segments like .data, although also aggregated, are ultimately placed in a private memory region for each Unikernel, rather than for sharing.
[0044] A more critical challenge is resolving instruction-level address dependencies. To achieve this, all code within a shared block is compiled into position-independent code using the PIC compiler flag. This involves merging the read-only code segments (.text) and read-only data segments (.rodata) scattered across various library files, then using the compiler's Address-Independent Code (PIC) feature to convert them into position-independent code and mark them as "shared regions."
[0045] Furthermore, to completely decouple shared code from the final memory layout of any particular Unikernel, the BMS redirects all external references to an instance-specific Global Offset Table (GOT). For example... Figure 3As shown in B, any instruction within a shared block that accesses an external symbol is translated into an indirect access via a GOT entry. The shared code itself remains consistent across all instances, always accessing the same offset in the GOT. However, the GOT table is allocated in a private, writable data segment for each Unikernel instance. During load-time sharing, the loader populates this private GOT with the correct absolute addresses within that specific instance's memory layout. This essentially strips all absolute address references to external symbols to the "private region." This process generates standardized shared block artifacts, enabling them to be safely reused across different Unikernel instances. This approach decouples shared, position-independent code from instance-specific, absolute addresses, thus allowing shared blocks to be universally shared.
[0046] S3: Component-based mirror building
[0047] Optionally, such as Figure 1 As shown in ③, the image packaging stage begins. The build toolchain links the application-specific private code with the generated private data segments to produce the final Unikernel image. Unlike traditional monolithic images, the image generated at this stage is a lightweight "skeleton file." It no longer contains the physical binary data of shared blocks, but instead includes a specially generated metadata segment. This segment records the shared block indexes and their version hashes that the current instance depends on. This component-based image model significantly reduces disk usage, providing a foundation for large-scale concurrent deployments.
[0048] S4: Shared on Load
[0049] Optionally, to address the high latency and runtime overhead issues of traditional deduplication techniques, BMS introduces a novel architecture for achieving zero-latency memory sharing. This architecture is built upon two core design principles: a shared block caching strategy to balance memory footprint and startup performance, and a load-time direct mapping mechanism to eliminate runtime overhead and deduplication latency.
[0050] Optionally, such as Figure 1 As shown in ④, the runtime loading and sharing phase begins. When multiple Unikernel instances are scheduled simultaneously on a physical machine, the enhanced loader parses the metadata in each image and sends a mapping request to the Hypervisor. For example... Figure 1As shown in the physical memory view, although the two instances logically have independent address spaces, and Unikernel 2 requires an additional shared block 2, the Hypervisor maps the existing copy of the shared block (shared block 0) in physical memory to the address spaces of both instances simultaneously by manipulating the extended page table. This mechanism achieves zero-copy sharing at the physical layer, and because the mapping is completed instantaneously at startup, the entire sharing process is completely transparent to upper-layer applications and has no runtime latency.
[0051] Optionally, the BMS manages shared blocks in a Hypervisor-level shared block cache pool by dividing the cache into two distinct regions: the base region and the normal region.
[0052] Optionally, the base block contains critical, frequently used libraries (e.g., core libc components) identified through dependency analysis, which are aggressively preloaded into the base area during system boot to minimize startup latency for most Unikernels.
[0053] Optionally, regular blocks, typically composed of less frequently used, application-specific libraries, are loaded into the regular area on demand. The first Unikernel instance to need a particular regular block triggers its loading from storage, allowing subsequent instances to reuse it. Blocks in this area are managed using a Least Recently Used (LRU) eviction policy to optimize memory footprint. This tiered caching strategy strikes a crucial trade-off between rapid instance startup and efficient use of physical memory.
[0054] Optionally, the efficiency of the BMS depends on its direct mapping mechanism, which is executed when the instance is loaded. This process utilizes a shared block registry within the Hypervisor to track the state of the cache pool. Figure 4 As shown, its workflow is as follows: When a Unikernel starts, the loader first parses its metadata to identify the required set of shared block IDs. For each block ID, it sends a mapping request to the Hypervisor via an ioctl call. The Hypervisor then performs a cache lookup in the registry.
[0055] Alternatively, in the case of a normal block cache hit, which means that the block is already residing in the pool (whether as a preloaded base block or a normal block loaded on demand), the Hypervisor will immediately manipulate the extended page table to map the existing physical frame into the address space of the new Unikernel.
[0056] Optionally, in the event of a miss in the normal block cache, the Hypervisor first loads the block from storage, updates the registry, and then performs EPT mapping. Once all necessary blocks have been mapped, the loader populates the instance's private GOT, and then the Unikernel continues its boot process.
[0057] In one exemplary embodiment, a block-sharing-based Unikernel memory deduplication system is provided, comprising:
[0058] The query and feedback module is used by BMS to maintain a global dependency graph through a knowledge base. When building Unikernel X, BMS executes query and feedback logic, maps the library set required by Unikernel to the global dependency graph through differential analysis, analyzes the cohesion of the library set to determine the boundary of shared blocks, and feeds back the new dependency pattern to the knowledge base to realize incremental updates of the dependency graph.
[0059] The binary orchestration module, after determining the boundaries of the shared block, uses a compact aggregation strategy to merge read-only code segments and read-only data segments, converts them into position-independent code using the address-independent code characteristics of the PIC compiler, and marks them as shared regions; at the same time, a private global offset table is introduced, which allows shared code to access fixed offsets, while the private global offset table is located in the private data segment, thereby decoupling shared code from Unikernel memory layout;
[0060] The image encapsulation module is used to build a toolchain that links the application's private code with the private data segment to generate a Unikernel image;
[0061] The module is shared during loading, and zero-latency memory sharing is achieved through a Hypervisor-level shared block cache pool and direct mapping mechanism, optimizing startup performance and memory usage.
[0062] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above-described block-shared Unikernel memory deduplication method.
[0063] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.
Claims
1. A Unikernel memory deduplication method based on block sharing, characterized in that, Includes the following steps: Query and feedback phase: BMS maintains a global dependency graph through a knowledge base. When building Unikernel X, it executes query and feedback logic. Through differential analysis, it maps the library set required by Unikernel to the global dependency graph, analyzes the cohesion of the library set to determine the boundary of shared blocks, and feeds back the new dependency pattern to the knowledge base to achieve incremental updates of the dependency graph. Binary orchestration stage: After determining the boundaries of the shared block, the orchestrator merges the read-only code segment and the read-only data segment through a compact aggregation strategy, converts them into position-independent code through the address-independent code feature of the PIC compiler, and marks them as shared regions; at the same time, a private global offset table is introduced, which allows the shared code to access fixed offsets, while the private global offset table is located in the private data segment, thereby decoupling the shared code from the Unikernel memory layout; Image packaging stage: The build toolchain links the application's private code with the private data segment to generate a Unikernel image; Load-time sharing phase: Zero-latency memory sharing is achieved through a Hypervisor-level shared block cache pool and direct mapping mechanism, optimizing startup performance and memory usage.
2. The method according to claim 1, characterized in that, The Unikernel image is a lightweight skeleton file that does not contain the physical binary data of the shared blocks. It only contains a specially generated metadata segment, which records the indexes of the shared blocks that the Unikernel instance depends on and their version hashes.
3. The method according to claim 1 or 2, characterized in that, The binary orchestration stage employs customized compilation and linking technology to generate self-contained and position-independent shared blocks.
4. The method according to claim 1, characterized in that, BMS implements the compact aggregation strategy by merging similar segments: only read-only segments are placed into the shared block and page-aligned, while writable segments, although aggregated, remain in the private memory area of each Unikernel.
5. The method according to claim 1 or 4, characterized in that, The shared block cache pool manages shared blocks, which are divided into a basic area and a normal area to balance performance and memory efficiency. The direct mapping mechanism uses the Hypervisor's internal shared block registry to track cache status and maps shared block copies directly to the instance address space through extended page tables, achieving zero-copy transparent sharing with no runtime latency.
6. The method according to claim 5, characterized in that, The base block contains critical, high-frequency libraries that are preloaded into the base area during BMS boot to reduce startup latency for most Unikernels. Ordinary blocks consist of infrequently used and application-specific libraries, which are loaded into the ordinary area as needed; The first Unikernel instance to request the block triggers its loading from storage for subsequent reuse, and is managed by an LRU policy to optimize memory usage.
7. The method according to claim 5, characterized in that, The direct mapping mechanism utilizes the Hypervisor's internal shared block registry to track cache state, including the following steps: When Unikernel starts, the loader determines the required set of shared block IDs by parsing metadata; Request the mapped shared block ID from the Hypervisor via an ioctl call; The hypervisor performs cache lookups in the registry.
8. The method according to claim 6, characterized in that, When the basic block cache is hit, the Hypervisor maps the physical frame to the new Unikernel address space by expanding the page table; When a regular block is not cached and a hit occurs, the Hypervisor loads the block from storage, updates the registry, and performs EPT mapping. Once all necessary blocks are mapped, the loader fills the private GOT, and the Unikernel continues the boot process.
9. A Unikernel memory deduplication system based on block sharing, characterized in that, include: The query and feedback module is used by BMS to maintain a global dependency graph through a knowledge base. When building Unikernel X, it executes query and feedback logic, maps the library set required by Unikernel to the global dependency graph through differential analysis, analyzes the cohesion of the library set to determine the boundary of shared blocks, and feeds back the new dependency pattern to the knowledge base to realize incremental updates of the dependency graph. The binary orchestration module, after determining the boundaries of the shared block, uses a compact aggregation strategy to merge read-only code segments and read-only data segments, converts them into position-independent code using the address-independent code characteristics of the PIC compiler, and marks them as shared regions; at the same time, a private global offset table is introduced, which allows shared code to access fixed offsets, while the private global offset table is located in the private data segment, thereby decoupling shared code from Unikernel memory layout; The image encapsulation module is used to build a toolchain that links the application's private code with the private data segment to generate a Unikernel image; The module is shared during loading, and zero-latency memory sharing is achieved through a Hypervisor-level shared block cache pool and direct mapping mechanism, optimizing startup performance and memory usage.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.