A method for efficient cross-virtual machine memory sharing based on bidirectional shadow memory object mapping
By establishing a master mapping table and a shadow mapping table in the virtualization environment, pre-allocating a three-level memory pool, and batch transmitting instructions, the low efficiency and consistency issues of GPU memory sharing in the virtualization environment are solved, rendering performance is improved, and a multi-guest synchronization mechanism is supported, achieving efficient memory sharing and consistency.
Patent Information
- Application Number
- CN202611610980.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from problems such as low memory sharing efficiency, difficulty in ensuring memory consistency, severe memory fragmentation, high instruction transmission overhead, and conflicts among multiple guests when sharing GPU memory in virtualized environments. In particular, the lack of the VK_KHR_external_memory_fd extension on the host side leads to decreased rendering performance and data corruption.
By establishing a main mapping table and a shadow mapping table on the Host and Guest sides respectively, pre-allocating a three-level video memory pool, and converting video memory operations into video memory object identifiers and offsets, and then batch transmitting instructions after merging and encoding, unified mapping of video memory and efficient transmission of instruction data are achieved. A hybrid storage of hash table and red-black tree is used to store the main mapping table, supporting multi-Guest concurrent consistency, and using Fence to confirm the completion of rendering.
It achieves efficient video memory sharing in the absence of VK_KHR_external_memory_fd extension, improves rendering performance, ensures memory consistency, reduces video memory fragmentation, and supports fine-grained permission control and synchronization mechanism for multiple guests, thereby improving the efficiency of virtualized graphics rendering.
Smart Images

Figure CN123118346A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer software development technology, specifically relating to a method for efficient cross-virtual machine memory sharing based on bidirectional shadow memory object mapping. Background Technology
[0002] Virtio-GPU Venus is a Vulkan command serialization protocol designed for virtualization environments within the Mesa open-source project. Its core logic involves the guest capturing and serializing Vulkan commands, then transmitting them to the host machine via the Virtio interface. The host parses the commands, invokes its local Vulkan driver to execute rendering, and finally sends the rendering results back to the guest. In this process, the VK_KHR_external_memory_fd extension is crucial for achieving zero-copy sharing of GPU memory between the guest and host. This extension allows the host to pass GPU memory objects to the guest as file descriptors (FDs). The guest can then directly access the host's physical GPU memory through the mapped FDs, avoiding the performance overhead of data copying.
[0003] However, some older GPU drivers or Vulkan drivers on niche hardware platforms do not implement the VK_KHR_external_memory_fd extension. Existing solutions suffer from the following problems: First, memory sharing is inefficient, requiring four levels of data copying between Host GPU memory, Host system memory, Guest system memory, and Guest virtual GPU memory for memory interaction, increasing rendering latency and significantly reducing frame rates in high-resolution scenes. Second, memory consistency is difficult to guarantee; the lack of a standardized synchronization mechanism for Guest and Host memory operations easily leads to data corruption issues such as Guest data not being synchronized to the Host GPU and Host rendering results not being promptly returned to the Guest, resulting in screen tearing and texture loss. Third, memory fragmentation is severe; pre-allocated large blocks of memory cannot be dynamically adjusted, and free memory blocks are scattered. Fourth, instruction transmission overhead is high; Guest-side Vulkan instructions are transmitted one at a time, and frequent interactions in the Virtio queue increase CPU utilization, further reducing rendering performance. Fifth, multi-Guest sharing conflicts occur when multiple Guest virtual machines share the same Host. When using GPU memory, the lack of fine-grained access control and synchronization mechanisms can easily lead to resource contention or data leaks.
[0004] In existing technologies, some solutions attempt to replace VK_KHR_external_memory_fd with vendor-specific extensions, but this has the problem of poor cross-vendor compatibility; other solutions use system shared memory as a transfer, but this cannot solve the core contradiction between performance and consistency, and does not cover multi-guest shared scenarios. Summary of the Invention
[0005] In view of this, the present invention provides a method for efficient cross-virtual machine memory sharing based on bidirectional shadow memory object mapping. By establishing a master mapping table and a shadow mapping table on the Host and Guest sides respectively and pre-allocating a three-level memory pool, converting memory operations into memory object identifiers and offsets on the Guest side and transmitting them in batches after instruction merging and encoding, verifying and reversing the physical address on the Host side and restoring it into Vulkan instructions, submitting it to the GPU through locking arbitration and confirming completion with Fence, the method achieves unified mapping between Guest virtual memory and Host physical memory, efficient batch transmission of instruction data, and consistency of multi-Guest concurrent operations under the condition of no VK_KHR_external_memory_fd extension, thereby supporting near-native virtualized graphics rendering.
[0006] The present invention provides a method for efficient cross-virtual machine memory sharing based on bidirectional shadow memory object mapping, which specifically includes the following steps:
[0007] When the host Vulkan driver does not support the VK_KHR_external_memory_fd extension, a main mapping table is created on the host and a shadow mapping table is created on the guest, using the memory object identifier as the key. The host pre-allocates the memory pool and obtains the memory handles of each memory block. It assigns a globally unique memory object identifier to each memory block, configures a default sharing policy, writes the memory object identifier, physical memory address, memory handle, and default sharing policy into the main mapping table, and sends an initialization synchronization packet to the guest via the Virtio control queue. The guest fills in the shadow mapping table accordingly, sets the shared role to non-shared, and returns an acknowledgment to the host via the Virtio control queue.
[0008] When a client application performs a controlled operation, the client obtains the target virtual address of the target memory block for that operation. Based on this, it looks up the corresponding memory object identifier in the shadow mapping table and converts the target virtual address into a combination of the memory object identifier and the instruction offset address. The instruction offset address is the difference between the target virtual address and the virtual memory start address corresponding to the memory object identifier. A semantic tag representing the Vulkan semantics of the operation instruction is generated. According to the operation instruction type, it is assigned to the corresponding instruction merging window. The instruction merging window maintains the memory object identifier, offset address, cumulative data length, and instruction list to form an instruction unit. The instruction unit is encoded into a binary instruction that sequentially includes an instruction header, semantic information, data payload, and shared control fields. This instruction is written as a transmission instruction into the client's transmission area. When the trigger condition is met, the transmission instruction is packaged into a batch packet and sent to the Virtio data queue and the host is notified. The transmission instruction is then transferred to the transmission area.
[0009] The host retrieves batch packets from the Virtio data queue. After verification, it queries the main mapping table to convert the video memory object identifier and instruction offset address into physical video memory address. Based on the semantic tags, the binary instructions are restored to Vulkan native instructions and submitted to the GPU queue for execution. The execution result is returned to the client via the Virtio data queue. The client updates the shadow mapping table accordingly and transfers the executed instructions to the completed area.
[0010] Furthermore, when assigning an operation instruction to the corresponding instruction merging window according to its type, if the operation instruction and the operation instruction in the instruction merging window belong to the same video memory object identifier, the corresponding instruction offset addresses are consecutive, and the cumulative length after merging is not greater than the maximum transmission unit, then the operation instruction is merged into the window; otherwise, the accumulated operation instructions in the instruction merging window are merged into an instruction unit and output. After clearing the window, the operation instruction is added again, and each instruction merging window is periodically forced to output instruction units according to a set number.
[0011] Furthermore, the master client of the video memory block is the first client to request and obtain the video memory block, possess full read and write permissions to it, and is responsible for initiating and controlling the sharing authorization, modification, and reclamation of the video memory block. When the master client shares the video memory block with other clients, it sends an authorization request carrying the target client's identifier and sharing permissions to the host. After the host verifies the request, it adds the target client to the sharing list, writes the sharing permissions, increments the sharing version, and then sends a sharing invitation to the target client. The target client establishes a shadow mapping table entry based on the invitation and sets it as a sharing client.
[0012] Furthermore, during system operation, when a client requests a memory block, the client driver sends a memory object identifier request to the host. The request carries the memory type, required size, access permissions, and a sharing permission identifier. The sharing permission identifier is a boolean field indicating whether the memory block is allowed to be accessed by other clients. The host-side Virtio-GPU kernel driver selects a free memory block from the memory pool, obtains the main mapping table entry for the memory block that was established during pre-allocation and has a reference count of 0, updates the reference count from 0 to 1, sets the identifier of the main client, and initializes the shared client list and the shared permission matrix to empty. Then, based on the value of the sharing permission identifier in the request, the sharing mode is set. If the identifier is not shared, the sharing mode is set to exclusive mode; if the identifier is allowed to share, it is set to a shareable pending authorization state. The host returns the memory object identifier and the client's virtual address offset to the main client. The main client driver updates the corresponding initialized entry in the shadow mapping table accordingly and sets the sharing role to the main client.
[0013] When releasing video memory blocks, for non-shared video memory blocks, when a client application releases video memory, the client driver sends a video memory release request to the host. The host updates the corresponding entry in the main mapping table, decrements the reference count by 1, and when the reference count reaches 0, the video memory block is marked as free and added to the video memory pool for reuse. Then, a release confirmation is sent to the client, and the host driver deletes the corresponding entry from the shadow mapping table. For shared video memory blocks, when a host client application releases video memory, the host client driver sends a sharing termination notification to all shared clients. After all shared clients confirm the release, the host client sends a video memory release request to the host. The host clears the shared client list and the shared permission matrix, and decrements the reference count by 1.
[0014] Furthermore, the host client sends a sharing authorization request to the host, which includes the video memory object identifier, the target client identifier, and sharing permissions. The host first verifies that the sharing mode of the entry corresponding to the video memory object identifier in the main mapping table is shareable. After the verification is successful, the host adds the target client identifier to the shared client list, writes the granted sharing permissions item by item into the sharing permission matrix, and increments the sharing version number. Then, the host sends a sharing invitation to the target client, carrying the video memory object identifier, video memory block size, client virtual address range, sharing permissions, and sharing version number. After receiving the sharing invitation, the target client creates a corresponding entry in its own shadow mapping table according to the invitation information and sets its sharing role to shared client, thus completing the sharing authorization.
[0015] Furthermore, the host-pre-allocated video memory pool is divided into three levels according to the access frequency of the stored data: the first-level pool is used to store frame buffers and data of commonly used texture classes with an access frequency greater than the maximum threshold, and supports read-only sharing among multiple clients; the second-level pool is used to store temporary rendering targets and shader resource classes with an access frequency no greater than the maximum threshold but greater than the minimum threshold, and supports read-write sharing among multiple clients but requires authorization from the host client; the third-level pool is used to store static models and configuration data classes with an access frequency no greater than the minimum threshold, and supports read-only sharing among multiple clients.
[0016] When a client requests a memory block, the host selects a free memory block from the corresponding memory pool based on the expected access frequency of the data stored in that memory block, allocates it to the client, and updates the master mapping table. During operation, the host re-determines the frequency range of each memory block according to the measured access frequency of each memory block according to a set period, migrates the memory block to the memory pool at the level corresponding to that frequency range, and synchronously updates the master mapping table and the shadow mapping table of the corresponding client.
[0017] Furthermore, for the instruction unit obtained from the instruction merging window, compression judgment is performed based on its data payload. Data compression is triggered only when the data payload length is not less than the threshold and its data type is determined to be texture data or vertex data by semantic tagging. If the data payload is structured data, the LZ4 algorithm is used for compression; if the data is unstructured data, the ZSTD algorithm is used for compression. After compression, the compression identifier field in the instruction header marks the compression algorithm used. At the same time, a checksum is calculated for the compressed data, and the original checksum of the instruction is replaced with the checksum. Then, the decompressed metadata containing the original length and the compressed length is embedded in the header of the data payload.
[0018] Furthermore, this includes hardware-level cache synchronization: For architectures supporting CPU and GPU cache consistency, after each memory write instruction is executed, the host sends a memory range containing the memory object handle, address, and data length to the Vulkan driver, triggering GPU cache refresh and writing the data in the cache back to physical video memory; when the client needs to read a block of video memory, it sends a cache invalidation request carrying the video memory object identifier and read range to the host via the Virtio control queue. The host invalidates the mapped cache for that range and calls the corresponding CPU cache invalidation instruction according to the CPU architecture. After invalidation is completed and confirmation is returned, the client executes the read; for rendering commit instructions, the host constructs commit information during restoration, sets the waiting phase to the transfer phase so that memory writes are executed before rendering, and sets the completion semaphore and fence. After committing, it waits for rendering to complete before triggering cache refresh to write the rendering result back to physical video memory; for architectures that do not support CPU and GPU cache consistency, the host exports the memory object as a DMA-BUF descriptor and hands it over to the host-side Virtio-GPU kernel driver for mapping. The kernel DMA subsystem automatically triggers GPU cache refresh and invalidation before and after CPU access.
[0019] Furthermore, it also includes semantic barrier processing at the driver layer: when a client application inserts a memory barrier, the client side captures and converts it into host-side barrier parameters, constructs synchronization instructions and writes them into the instruction buffer. The synchronization instructions are transmitted in the highest priority through the synchronization instruction sub-queue in the Virtio control queue. When the number of synchronization instructions captured within a set time exceeds a set number, they are merged into a batch synchronization instruction carrying multiple barrier parameters. After parsing by the host, a corresponding number of memory barriers are generated, and the host-side memory barriers are inserted before executing the corresponding rendering instructions according to the converted source and target access masks. When the host Vulkan driver does not support a certain source access mask, barrier degradation is triggered, and it is replaced with a general memory write mask.
[0020] Furthermore, it also includes application-layer locking control: Before the client writes to the video memory block, it sends a lock request to the host carrying the video memory object identifier and the request timeout time. The host queries the lock flag bit in the main mapping table. If it is not locked, it sets the flag, records the locking client identifier and timestamp, and returns a lock success. If it is already locked, it determines whether the locker is the current client. If yes, it returns a success due to reentrancy; otherwise, it returns a lock busy. After receiving a lock busy, the client spins and waits. The initial waiting time is set, and then it increases by multiples but does not exceed the request timeout time. If the lock is not acquired after the request timeout time, it returns a lock timeout and retryes the instruction. It can retry a maximum of the set number of times. If the retry still fails, it notifies the application layer. After the client completes the write, it sends a lock release request. The host confirms that the lock flag bit is set and that the locker is the current client. After that, it resets the flag bit, increments the synchronization version number, and automatically grants the lock to the first instruction in the waiting queue when there is a waiting instruction for the video memory block.
[0021] Beneficial effects:
[0022] When the VK_KHR_external_memory_fd extension is missing in the Host-side Vulkan driver, this invention, after Virtio-GPU Venus starts, first establishes a main mapping table in the Host-side Virtio-GPU kernel driver, using memory object identifiers as indexes and employing a hybrid storage of hash tables and red-black trees. In the Guest-side virtual GPU driver, a shadow mapping table with a hash structure, retaining only core and shared state fields, is established. Simultaneously, memory pools are pre-allocated according to a three-level strategy of high, medium, and low frequency. A unique memory object identifier is assigned to each memory block, a default sharing strategy is configured, and this information is written to the main mapping table. Then, an initialization synchronization packet is sent via the Virtio control queue, the Guest fills in the shadow mapping table, and confirmation is obtained through the response channel. During runtime, the Guest first verifies the parameters for controlled operations, converts the target virtual address into a memory object identifier and block offset in the shadow mapping table, marks it as shared, and then merges them using the instruction merging window. The output instruction unit is encoded into a format containing an instruction header and a statement. The binary instructions containing semantic information, data payload, and shared control fields are written to the circular buffer waiting area for transmission. When the water level is reached, rendering is captured and submitted, or a timeout occurs, the data is packaged into a batch packet, sent to the Virtio data queue, and transferred to the transmission area. The Host retrieves the batch packet and verifies it layer by layer. Based on the memory object identifier, it queries the master mapping table, reverse-calculates the Host's physical memory address, and completes out-of-bounds and shared write permission checks. Then, according to the semantic tags, the instructions are restored to Vulkan native instructions. After priority sorting and locking arbitration, the instructions are submitted to the GPU. The Fence or completion semaphore confirms the completion of the execution. The Guest classifies and generates and sends back the execution result packet. The Guest updates the dual version number accordingly, transfers successful instructions to the completed area, and retryes failed instructions a set number of times. If the retry still fails, the application layer is notified, thus realizing cross-virtual machine memory sharing. Attached Figure Description
[0023] Figure 1 This is a flowchart illustrating the efficient cross-virtual machine memory sharing method based on bidirectional shadow memory object mapping provided by the present invention. Detailed Implementation
[0024] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.
[0025] The core idea of this invention, which provides a cross-virtual machine efficient video memory sharing method based on bidirectional shadow memory object mapping, is as follows: When the VK_KHR_external_memory_fd extension is missing in the Vulkan driver on the host side, after Virtio-GPU Venus starts, a main mapping table is first established in the Virtio-GPU kernel driver on the host side, using video memory object identifiers as indexes and employing a hybrid storage of hash tables and red-black trees. A shadow mapping table with a hash structure, retaining only core and shared state fields, is established in the virtual GPU driver on the guest side. Simultaneously, a video memory pool is pre-allocated according to a three-level strategy of high, medium, and low frequency. A unique video memory object identifier is assigned to each video memory block, a default sharing strategy is configured, and this information is written to the main mapping table. Then, an initialization synchronization packet is sent via the Virtio control queue, the guest fills in the shadow mapping table, and confirmation is obtained through the response channel. During runtime, the guest first verifies the parameters of the controlled operation, converts the target virtual address into a video memory object identifier and an offset within the block in the shadow mapping table, marks it as shared, and then merges it using the instruction merging window. The output instruction unit is encoded into a format containing an instruction header and a statement. The binary instructions containing semantic information, data payload, and shared control fields are written to the circular buffer waiting area for transmission. When the water level is reached, rendering is captured and submitted, or a timeout occurs, the data is packaged into a batch packet, sent to the Virtio data queue, and transferred to the transmission area. The Host retrieves the batch packet and verifies it layer by layer. Based on the memory object identifier, it queries the master mapping table, reverse-calculates the Host's physical memory address, and completes out-of-bounds and shared write permission checks. Then, according to the semantic tags, the instructions are restored to Vulkan native instructions. After priority sorting and locking arbitration, the instructions are submitted to the GPU. The Fence or completion semaphore confirms the completion of the execution. The Guest classifies and generates and sends back the execution result packet. The Guest updates the dual version number accordingly, transfers successful instructions to the completed area, and retryes failed instructions a set number of times. If the retry still fails, the application layer is notified, thus realizing cross-virtual machine memory sharing.
[0026] The present invention provides a method for efficient cross-virtual machine video memory sharing based on bidirectional shadow video memory object mapping, the processing flow of which is as follows: Figure 1 As shown, the specific steps include:
[0027] Step 1: When the VK_KHR_external_memory_fd is missing in the Vulkan driver on the Host side, after Virtio-GPUVenus starts, a main mapping table is created in the kernel space of the Virtio-GPU kernel driver on the host side. It adopts a hybrid storage structure combining hash table and red-black tree, with the memory object identifier as the index, and contains basic information, permission information, status information, association information, statistical information, and shared information among multiple guests, serving as a global record for all memory objects. A shadow mapping table is created in the user space of the virtual GPU driver on the Guest side. It adopts a hash table structure, with the memory object identifier as the key, retaining only the core fields required for Guest operations and supplementing the shared status fields, serving as a lightweight mapping copy of the main mapping table on the Guest side.
[0028] The Host calls the Vulkan driver's memory allocation interface to pre-allocate three levels of GPU memory pools according to a three-level hierarchical strategy. The first level pool is used to store high-frequency access memory data such as frame buffers and commonly used textures. The second level pool is used to store mid-frequency access memory data such as temporary rendering targets and shader resources. The third level pool is used to store low-frequency access memory data such as static models and configuration data. It returns the memory handles of the pre-allocated memory blocks in each level pool, assigns a unique memory object identifier to each memory block, configures a default sharing strategy for each memory block, writes the memory object identifier, physical memory address, memory handle, and default sharing strategy to the main mapping table, and then sends a mapping table initialization synchronization packet to the Guest through the Virtio control queue.
[0029] After receiving the mapping table initialization synchronization packet, the guest-side virtual GPU driver fills the information in the shadow mapping table, sets the shared role to non-shared, and completes the initialization of the shadow table; then it returns a synchronization completion confirmation signal to the host through the Virtio response queue.
[0030] The basic information in the main mapping table includes the video memory object identifier, physical video memory starting address, video memory block size, and video memory type. The permission information includes the access permission identifier and sharing mode. The status information includes the application count, lock flag, synchronization version number, and health status. The associated information includes the Host Vulkan memory object handle, Virtio data queue ID, host Guest virtual machine ID, and Guest virtual address offset. The statistical information includes access frequency, last access timestamp, number of writes, number of reads, and fragmentation level. The multi-Guest sharing information includes the shared Guest list, shared permission matrix, shared lock flag, and shared version number.
[0031] The mapping table initialization synchronization package contains core information such as video memory object identifier, video memory block size, guest virtual address range, and default sharing policy.
[0032] The shadow mapping table contains the video memory object identifier, the starting address of the Guest virtual video memory, the video memory block size, local access permissions, synchronization version number, lock status, shared role, number of shared Guests, and shared version number.
[0033] Furthermore, during system operation, when a Guest requests a memory block, the Guest driver sends a memory object identifier request to the Host. This request includes the memory type, required size, access permissions, and a sharing permission flag. The sharing permission flag is a Boolean field indicating whether the memory block is allowed to be accessed by other Guests. The Host-side Virtio-GPU kernel driver selects a free memory block from the corresponding memory pool, retrieves the main mapping table entry for that memory block (which was created during pre-allocation and has a reference count of 0), updates the reference count from 0 to 1, sets the main Guest identifier to the identifier of the Guest currently sending the request, and sets the sharing permission flag. The Guest list and shared permission matrix are initialized to empty; then the sharing mode is set according to the value of the sharing permission flag in the request. When the flag is not shared, the sharing mode is set to exclusive mode; when the flag is allowed to share, it is set to a shareable pending authorization state; the Host returns the video memory object flag and the Guest virtual address offset to the Guest that sent the request. The Guest driver updates the corresponding initialized table entry in the shadow mapping table accordingly and sets the sharing role to the main Guest; if the requested video memory block exceeds the pre-allocated and is newly allocated on demand, the Host first supplements the table entry information to the Guest, and the Guest then creates a new shadow mapping table entry accordingly.
[0034] When releasing video memory blocks, for non-shared video memory blocks, when the Guest application releases video memory, the Guest driver sends a video memory release request to the Host. The Host updates the corresponding entry in the main mapping table, decrements the reference count by 1, and when the reference count reaches 0, the video memory block is marked as free and included in the corresponding level video memory pool for reuse. Then, a release confirmation is sent to the Guest, and the Guest driver deletes the corresponding entry from the shadow mapping table. For shared video memory blocks, when the main Guest application releases video memory, the main Guest driver sends a sharing termination notification to all shared Guests. After all shared Guests confirm the release, the main Guest then sends a video memory release request to the Host. The Host clears the shared Guest list and the shared permission matrix, and decrements the reference count by 1.
[0035] Furthermore, the primary Guest sends a sharing authorization request to the Host, which includes the memory object identifier, the target Guest identifier, and sharing permissions. The Host first verifies that the sharing mode of the corresponding entry in the primary mapping table for the memory object identifier is shareable. After the verification is successful, the Host adds the target Guest identifier to the shared Guest list, writes the granted sharing permissions item by item into the shared permission matrix, and increments the sharing version number. Then, the Host sends a sharing invitation to the target Guest, carrying the memory object identifier, memory block size, Guest virtual address range, sharing permissions, and sharing version number. After receiving the sharing invitation, the target Guest creates a corresponding entry in its own shadow mapping table according to the invitation information and sets its sharing role to shared Guest, thus completing the sharing authorization.
[0036] Furthermore, when the main Guest modifies access permissions or memory type, it first confirms that no shared Guest is operating by using the shared lock flag. Then, the Host updates the permissions and type fields of the main mapping table, synchronously updates the shared permission matrix, and synchronizes the changes to the shadow mapping tables of all related Guests, ensuring that the main mapping table and the shadow mapping tables of each Guest remain consistent throughout the entire lifecycle.
[0037] Step 2: When the Guest application performs a controlled operation, the Guest obtains the operation instruction and the target virtual address in the Guest is recorded as the target virtual address. The validity of the instruction parameters is verified. If the verification fails, a Vulkan standard error code is returned. If the verification succeeds, the Guest searches the shadow mapping table for an entry whose address range contains the target virtual address. If the entry does not exist, it is determined to be an illegal video memory address and a corresponding error is returned. If the entry exists, the video memory object identifier of the entry is read as the instruction video memory object identifier, and the target virtual address is converted into a combination of the video memory object identifier and the instruction offset address. The instruction offset address is the difference between the target virtual address and the starting address of the Guest virtual video memory corresponding to the entry. If the corresponding video memory block is determined to be a shared video memory block based on the shared role of the entry, and the current operation type is write, then a shared write identifier is marked in the semantic information.
[0038] Based on the type of the operation instruction, it is assigned to the corresponding instruction merging window. The instruction merging window maintains four status variables: the current memory object identifier, the current offset address, the cumulative data length, and the instruction list. If the instruction merging window is empty, the operation instruction is added and the status variables are initialized, that is, the current memory object identifier is updated to the instruction memory object identifier, the current offset address is updated to the instruction offset address, the cumulative data length is updated to the data length of the operation instruction, and the operation instruction is added to the instruction list. If the instruction merging window is not empty, and the operation instruction and the instructions in the window belong to the same memory object identifier, satisfy the condition that the sum of the current offset address and the cumulative data length equals the instruction offset address, and the sum of the cumulative data length and the instruction data length is not greater than the maximum transfer single... If the maximum transfer unit (MTU) is reached, the operation instruction is added to the window, keeping the current memory object identifier and current offset address unchanged. The accumulated data length is updated to the sum of the original accumulated data length and the instruction data length, and the operation instruction is added to the instruction list. Otherwise, if the memory object identifiers are different, the offset addresses are not contiguous, or the accumulated length after merging is greater than the maximum transfer unit, the multiple instructions already accumulated in the window are first merged into one instruction unit and output. Then, the instruction merging window is cleared, and the current operation instruction is added as the first instruction in the window. The four state variables are initialized in the same way as when the window is empty. In addition, when a set number of operation instructions are captured, the instructions already accumulated in all non-empty instruction merging windows are merged into instruction units and output, and the corresponding windows are cleared.
[0039] Each time the instruction merging window outputs an instruction unit, it encodes the instruction unit into a binary instruction, which sequentially includes four parts: instruction header, semantic information, data payload, and shared control field. The instruction header includes the magic number, total length, instruction type, data payload length, compression flag, and priority. The semantic information includes the memory object identifier, offset address, semantic tag, synchronization flag, and checksum. The data payload is the payload data. The shared control field is only written when the instruction is a shared instruction, including the initiating Guest identifier, the target Guest identifier, and sharing permissions. The obtained binary instruction is written as a transmission instruction to the transmission area in the circular instruction buffer on the Guest side. When the remaining space in the transmission area is less than the threshold, or when a rendering submission instruction is captured, or when the longest waiting time for a transmission instruction exceeds the threshold, the transmission instructions are packaged into a batch packet. The batch packet header includes the packet magic number, packet length, and instruction count. The batch packet is sent to the Virtio data queue on the Guest side and the Host is notified. The transmitted batch packet is then removed from the transmission area and saved to the transmission middle area of the circular instruction buffer, and a transmission timestamp is recorded.
[0040] The controlled operations include those related to memory lifecycle management, recording data transfer commands, and submitting data to the GPU for execution.
[0041] Furthermore, compression is performed on the data payload of the merged instruction. Data compression is triggered only when the data payload length is not less than a threshold and its data type is determined to be texture data or vertex data by semantic tagging. After compression is triggered, a compression algorithm is selected based on the data's structure level. If the data is structured, the LZ4 algorithm is used for compression; if the data is unstructured, the ZSTD algorithm is used. After compression, the compression identifier field in the instruction header marks the compression algorithm used. At the same time, a CRC32 checksum is calculated for the compressed data, and this checksum replaces the original checksum of the instruction. In addition, to support host-side decompression, decompression metadata containing the original length and the compressed length is embedded in the data payload header. This data payload consists of one of four parts: binary instruction header, semantic information, data payload, and shared control field. It carries the actual data to be transmitted and written or copied to the target video memory block, such as texture data and vertex data. Its byte count is recorded in the data payload length field of the instruction header; a value of 0 in this field indicates that the instruction has no data payload.
[0042] Furthermore, when sending batch packets to the Virtio data queue on the Guest side, if the free space in the Virtio data queue is not less than the length of the batch packets, then the batch packets are written into the Virtio data queue; otherwise, the batch packets are split into multiple sub-packets with a length not greater than the free space, and then written into the Virtio data queue sequentially.
[0043] The semantic marker is an enumeration code used to describe the Vulkan operation primitive corresponding to the instruction.
[0044] Step 3: The Host verifies the batch packets. If the verification is successful, it queries the main mapping table based on the memory object identifier in the instruction to obtain the corresponding Host physical memory address, Vulkan memory object handle, sharing mode, and shared Guest list, and converts the memory object identifier and offset address into the Host physical memory address. If the memory block is a shared memory block and the instruction type is write, the Host queries the shared permissions of the initiating Guest from the shared permission matrix. If there is no write permission, the instruction is discarded and insufficient permissions are returned.
[0045] Based on the semantic tags of the instructions, the binary instructions are restored to the host-side Vulkan native instructions, and then sorted according to the instruction priority from high to low. The target memory block is locked and arbitrated, and then the instructions are submitted to the host-side GPU queue for execution. The host waits for the instructions to complete on the GPU through a fence or completion semaphore. Then, an execution result packet is generated for each instruction based on the actual completion status. The execution result packet contains the memory object identifier, instruction type, execution status, error code, updated synchronization version number, and updated shared version number of shared instructions. Each execution result packet is classified according to the Guest identifier and written in batches to the corresponding Virtio response queue. After receiving the execution result packet, the Guest side updates the synchronization version number and shared version number of the shadow mapping table, and transfers the corresponding successful instructions in the transmission area to the completed area of the circular instruction buffer on the Guest side to release buffer space. Failed instructions are retried up to a maximum of a set number of times. If the retry still fails, the application layer is notified.
[0046] The process involves the Host retrieving batch packets from the Virtio data queue and performing layer-by-layer checks: The first layer checks the packet magic number in the batch packet header; if it doesn't match, the entire packet is discarded and a packet format error is returned to the Guest. The second layer checks if the packet length matches the actual received length; if they don't, the packet is considered truncated, discarded, and an error is returned. The third layer splits the batch packets into individual instructions according to the binary instruction format, checking the magic number and checksum of each instruction; if the check fails, only that instruction is discarded, an error log is recorded, and the remaining instructions are parsed. The fourth layer, for shared instructions, checks if the initiating Guest identifier matches the main Guest identifier recorded in the main mapping table; if not, insufficient permissions are determined, the instruction is discarded, and an error is returned.
[0047] Furthermore, for commands that pass verification, the decision to decompress is made based on the compression flag in the command header: for commands whose compression flag indicates no compression, decompression is skipped and their original payload is used directly; for commands whose compression flag indicates compression, the decompression metadata in the data payload header, including the original length and the compressed length, is extracted first. If the compression flag is LZ4, the LZ4 decompression function is called; if it is ZSTD, the ZSTD decompression function is called. The compressed data is decompressed to a temporary buffer, and the decompressed length is checked to see if it matches the original length. If decompression fails or the length is inconsistent, the command is discarded and a data corruption message is returned to the Guest. If decompression succeeds, the original data is used to replace the data payload.
[0048] Further, based on the memory object identifier in the instruction, the main mapping table is queried to obtain the corresponding Host physical memory start address, Vulkan memory object handle, sharing mode, and shared Guest list information; the memory object identifier and offset address are converted into the Host physical memory address, which is equal to the sum of the physical memory start address and the instruction offset address in the main mapping table; then, an out-of-bounds validity check is performed to determine whether the sum of the physical address and the data payload length is greater than the sum of the physical memory start address and the memory block size. If it is greater, it is determined to be an address out of bounds, the instruction is discarded and an error is returned; if it is not greater, the address is considered valid; if the memory block is a shared memory block and the instruction type is write, the shared permissions of the initiating Guest are also queried from the shared permission matrix. If there is no write permission, the instruction is discarded and insufficient permissions are returned.
[0049] Specifically, the Vulkan semantic markers of the instructions are used for restoration. For example: if the semantic marker is buffer update, it is restored to vkCmdUpdateBuffer, a command buffer is constructed, and the Vulkan memory object handle, physical address, and data payload from the main mapping table are passed in; if the semantic marker is image copy, it is restored to vkCmdCopyBufferToImage, a buffer object and an image object bound to a physical address are constructed, and a copy region containing offset address and data length is set; if the semantic marker is queue commit, it is restored to vkQueueSubmit, commit information is constructed, the restored command buffer is associated, and a waiting semaphore is set; if the semantic marker is memory barrier, it is restored to vkCmdPipelineBarrier, the source access mask and target access mask in the instruction are parsed, and a memory barrier is constructed; if the semantic marker is shared authorization, and there is no corresponding Vulkan native instruction, the Host-side Virtio-GPU kernel driver interface is called to update the shared Guest list and shared permission matrix in the main mapping table.
[0050] Furthermore, if the host-side Vulkan driver does not support a certain native command during the restoration process, command downgrading is triggered, replacing it with an equivalent combination of vkMapMemory, memory copy, and vkUnmapMemory operations.
[0051] Before instruction execution, the target video memory block is locked and arbitrated. Specifically, the lock flag of the master mapping table is checked, and the shared lock flag is also checked for shared video memory blocks. If it is in an unlocked state, the lock flag is set and the lock Guest identifier and lock time are recorded. If it has been locked by another Guest and is not from the same Guest, the instruction is added to the waiting queue, and the lock status is polled at set intervals. If the waiting time exceeds the set time, the resource busy is returned. After the instruction is executed, the lock is released and the lock is granted to the next instruction in the waiting queue.
[0052] Furthermore, locked and pending instructions are sorted by priority from high to low, so that high-priority instructions such as framebuffer writing are submitted to the host-side GPU queue first. When the number of instructions in the waiting queue reaches a set number or the waiting time exceeds a threshold, the instructions in the queue are submitted to the GPU queue in batches, so that the queue submission is reduced from once per instruction to once per set number of instructions. After submission, the execution status of the instructions is monitored. If the execution is successful, the execution result is fed back. If the execution fails, the error code is recorded and the abnormal recovery process such as retry and memory block replacement is triggered.
[0053] Furthermore, this invention establishes a cross-domain memory consistency processing procedure, specifically as follows:
[0054] At the hardware layer, for architectures supporting CPU-GPU cache consistency, after each memory write instruction is executed on the host side, a memory range containing the memory object handle, physical address, and data length is passed in to trigger GPU cache refresh and write the cached data back to physical video memory. After refresh, the master mapping table synchronization version number is incremented by 1. When the guest side sends a cache invalidation request carrying the video memory object identifier and read range via the Virtio control queue, the host side invalidates the old data in the GPU cache and calls the corresponding CPU cache invalidation instruction. After invalidation is completed and confirmation is returned, the guest side performs the read. For rendering commit instructions, the host side constructs commit information when restoring the instruction, sets the waiting phase to the transmission phase to ensure that memory writes are executed before rendering, and sets the completion semaphore and fence. After committing, it waits for rendering to complete, and after completion, it triggers cache refresh to write the rendering result back to physical video memory. For older architectures that do not support CPU-GPU cache consistency, DMA-BUF is used for forced synchronization.
[0055] At the driver layer, when the Guest application inserts a memory barrier, the Guest side captures and parses the source access mask, target access mask, and source and target queue family indices. It then converts these into Host-side barrier parameters according to the Guest-Host semantic mapping table, constructs CMCP synchronization instructions, and writes them to the instruction buffer. Synchronization instructions are transmitted with the highest priority via a separate synchronization instruction sub-queue in the Virtio control queue, isolated from ordinary data instructions to avoid blocking. When more than a set number of synchronization instructions are captured within a set time, they are merged into a single batch synchronization instruction carrying multiple barrier parameters. The Host side parses this instruction and generates the corresponding number of memory barriers. Before executing the corresponding rendering instruction, the Host side constructs and inserts a Host-side memory barrier based on the converted source and target access masks. If the Host-side Vulkan driver does not support a certain source access mask, barrier downgrading is triggered, replacing it with a generic memory write mask to ensure the barrier takes effect.
[0056] At the application layer, before writing to the video memory block, the Guest sends a lock request carrying the video memory object identifier and the request timeout period. The Host queries the lock flag in the main mapping table. If it is unlocked, the flag is set, the Guest identifier and lock timestamp are recorded, and a lock success is returned. If it is already locked, it checks if the locker is the current Guest. If yes, it rejoins and returns success; otherwise, it returns lock busy. After receiving lock busy, the Guest spins and waits, initially for a set time, which increments by a multiple but not exceeding the request timeout period. If the lock is not acquired within the timeout period, a lock timeout is returned, and the instruction is retried. It retries a maximum of a set threshold number of times. If it still fails, the application layer is notified. After the Guest completes writing, it sends a lock release request. The Host confirms that the flag is set and that the locker is the current Guest, then resets the lock flag, increments the synchronization version number by 1, and waits for the video memory block. When an instruction is given, the lock is automatically granted to the first instruction in the waiting queue. The version number is synchronized incrementally. After each modification of the video memory block data or shared permissions, the host increments the synchronized version number or shared version number by 1. Every set time interval or when the version number changes more than a set number of times, a version number synchronization packet carrying the video memory object identifier, synchronized version number and shared version number is sent to each relevant guest via the Virtio control queue. The guest updates its local version number. When the difference between the local version number and the host version number exceeds a threshold, a full synchronization is triggered and the complete table entry is reacquired. Before reading, the guest checks whether the local synchronized version number is consistent with the host. If they are inconsistent, it waits for the synchronization to complete. Before executing a shared instruction, the host checks whether the shared version number of the guest that initiated the instruction is consistent with the main mapping table. If they are inconsistent, it refuses to execute and requires the guest to complete the version synchronization first.
[0057] For multiple guests concurrently sharing the same video memory block, a multi-guest shared consistency layer handles conflict detection and arbitration. When multiple guests simultaneously request to write to the same video memory block, the host sets the shared lock flag in the main mapping table to 1 to mark a write conflict. The shared permission arbitrator then sorts the conflict requests according to instruction type priority, guest priority, and request timestamp priority, generates an execution queue, and allocates locks sequentially according to the queue order. The next guest is only allowed to execute after the previous guest has completed writing and released the lock. During execution, the shared version number is synchronized to ensure that each guest obtains the latest data.
[0058] The DMA-BUF forced synchronization method works as follows: the host side exports the memory object as a DMA-BUF descriptor through the vendor's private interface, which is then mapped by the host side Virtio-GPU kernel driver. The kernel DMA subsystem automatically triggers the refresh and invalidation of the GPU cache before and after CPU access.
[0059] Furthermore, this invention calculates the heat value of each memory block according to the access frequency, last access time and read / write ratio recorded in the master mapping table, based on the access frequency, time decay factor and read / write weight, and divides them into three levels: hot, warm and cold, and adds them to the first, second and third level pools respectively. The grading threshold is dynamically adjusted according to the usage rate of each pool.
[0060] Furthermore, for scenarios where multiple guests share the same host GPU memory block, this invention constructs a guest memory sharing permission control mechanism to ensure secure and controllable cross-guest sharing, specifically as follows:
[0061] Shared permissions are divided into three types: read-only, read-write, and execute, and are represented by a permission bitmap. Bits 0, 1, and 2 of the bitmap correspond to read, write, and execute permissions, respectively. During authorization, the host Guest sends a shared authorization request to the host, carrying the target Guest identifier, the video memory object identifier, and the permission bitmap. The host first verifies that the host Guest has read and write permissions for the video memory block. If the verification is successful, the host adds the target Guest identifier to the shared Guest list in the main mapping table, writes the permission bitmap item by item into the shared permission matrix, and makes the two correspond one-to-one according to the index. Then, the host sends a shared invitation to the target Guest, carrying the video memory object identifier, video memory block size, Guest virtual address range, permission bitmap, and shared version number. After the target Guest accepts the invitation, it creates a corresponding entry in its own shadow mapping table according to the invitation information, writes the local access permissions, and sets the shared role to shared Guest, thus completing the authorization.
[0062] During recycling, the primary Guest sends a shared recycling request to the Host, carrying the target Guest identifier and the memory object identifier. The Host removes the record of the target Guest from the shared Guest list and shared permission matrix in the primary mapping table, increments the shared version number, and then sends a shared recycling notification to the target Guest. After receiving the notification, the target Guest deletes the corresponding entry in the shadow mapping table, releases the relevant resources, and completes the recycling.
[0063] Example:
[0064] This embodiment employs the efficient cross-virtual machine memory sharing method based on bidirectional shadow memory object mapping provided by this invention. When the VK_KHR_external_memory_fd extension is missing in the host-side Vulkan driver, it achieves efficient cross-domain sharing of GPU memory between the Guest virtual machine and the Host host, low-latency transmission of Vulkan instructions, and ensures the consistency of memory operations. It is suitable for virtualization scenarios relying on Virtio-GPU Venus, such as cloud computing, desktop virtualization, and game streaming. The specific implementation process includes:
[0065] S1. Construct a bidirectional shadow memory object mapping to unify the memory address mapping between Guest and Host. The bidirectional shadow memory object mapping is the core foundation for replacing VK_KHR_external_memory_fd. This is achieved by constructing a main mapping table H-MainDSVMO on the Host side and a shadow mapping table G-ShadowDSVMO on the Guest side. A bidirectional real-time synchronization and multi-Guest access control method is designed to realize the mapping between Guest virtual memory addresses and Host physical memory addresses, while also supporting secure sharing among multiple Guests.
[0066] S1.1 The structure of the main mapping table H-MainDSVMO.
[0067] H-MainDSVMO is created in the kernel space of the Virtio-GPU kernel driver on the host side. It adopts a hybrid storage structure of hash table and red-black tree. The hash table is used to quickly query the VMOID of the video memory object, and the red-black tree is used to sort and manage by video memory address. Each table entry contains 26 sub-fields in six categories: basic information, permission information, status information, association information, statistical information, and multi-Guest shared information. The specific definitions are shown in the table below.
[0068] Table 1. Field Information Table of the Main Mapping Table H-MainDSVMO
[0069] S1.2, The structure of the shadow mapping table G-ShadowDSVMO.
[0070] In the user space of the virtual GPU driver on the Guest side, create G-ShadowDSVMO, using VMOID as the key and a hash table storage structure. Only retain the core fields related to Guest side operations, and supplement the shared state fields for multiple Guests, as shown in the table below.
[0071] Table 2 Field Information Table of G-ShadowDSVMO
[0072] S1.3. Complete the initialization of DSVMO when Virtio-GPU Venus starts.
[0073] Host-side memory pool pre-allocation and sharing configuration. The host-side Virtio-GPU kernel driver calls the vkAllocateMemory interface of the host Vulkan driver to pre-allocate the GPU memory pool according to a three-level hierarchical strategy, including: Level 1 pool, i.e., high-frequency pool, allocates 40% of the total memory, consisting of a 2GB contiguous memory block, used to store high-frequency access resources such as frame buffers and commonly used textures, and supports multi-guest read-only sharing by default; Level 2 pool, i.e., mid-frequency pool, allocates 30% of the total memory, consisting of a 512MB contiguous memory block, used to store mid-frequency access resources such as temporary rendering targets and shader resources, and supports multi-guest read-write sharing, requiring authorization from the primary guest; Level 3 pool, i.e., low-frequency pool, allocates 30% of the total memory, consisting of a 64MB memory block, used to store low-frequency access resources such as static models and configuration data, and supports multi-guest read-only sharing by default.
[0074] After pre-allocation is completed, the Host Vulkan driver returns the VkDeviceMemory handle for each memory block, and the HostVirtio-GPU kernel driver configures the default sharing strategy for each level of memory block, such as read-only sharing for the first-level pool and a maximum of 8 guests, and writes the multi-Guest sharing information field of H-MainDSVMO.
[0075] VMOID generation and main table initialization: The host-side Virtio-GPU kernel driver generates a unique VMOID for each pre-allocated memory block, and fills the corresponding table entry of H-MainDSVMO with information such as VMOID, physical memory address, VkDeviceMemory handle, and default sharing policy. The reference count is initialized to 0, the synchronization version number is 0, the lock flag is 0, and the sharing version number is 0.
[0076] Guest-side shadow table synchronization: The Host side sends a DSVMO initialization synchronization packet to the Guest through the virtio control queue. The packet contains core information such as VMOID, memory block size, Guest virtual address range, and default sharing policy. After receiving the packet, the Guest-side virtual GPU driver fills the information into G-ShadowDSVMO, sets the sharing role to non-shared, completes the initialization of the shadow table, and returns a synchronization completion confirmation signal to the Host through the virtio response queue.
[0077] During the operation of Virtio-GPU Venus, when the Guest side needs to allocate, release, or modify video memory blocks, or initiate multi-Guest sharing, a runtime dynamic update of DSVMO is triggered. The specific process is as follows:
[0078] When requesting video memory blocks, the Guest application calls the vkAllocateMemory interface. The Guest virtual GPU driver sends a VMOID request to the Host. The request packet includes the video memory type, required size, access permissions, and a sharing permission flag, which indicates whether the memory is open to other Guests. The Host-side Virtio-GPU kernel driver selects the most suitable free video memory block from the corresponding level of the video memory pool based on the request, and updates the reference count, sharing mode, and primary Guest virtual machine ID fields of H-MainDSVMO. If the request includes a sharing permission flag, the shared Guest list and the shared permission matrix are set to empty. Then, the VMOID and Guest virtual address offset are returned to the Guest. The Guest driver updates G-ShadowDSVMO, sets the sharing role to primary Guest, and completes the video memory block allocation.
[0079] When releasing a video memory block, if it is a non-shared video memory block, when the Guest application calls the vkFreeMemory interface, the Guest driver sends a VMOID release request to the Host. The Host updates the reference count of H-MainDSVMO. When the reference count is 0, the video memory block is marked as free and included in the corresponding level of video memory pool for reuse. At the same time, the Host sends a release confirmation to the Guest, and the Guest driver deletes the corresponding entry from G-ShadowDSVMO. If it is a shared video memory block, when the primary Guest releases it, it must first send a shared termination notification to all shared Guests. After the shared Guests confirm the release, the primary Guest then sends a VMOID release request. The Host clears the shared Guest list and the shared permission matrix, updates the reference count, and the subsequent process is the same as for non-shared blocks.
[0080] For multi-guest shared authorization, when the primary guest needs to share a block of video memory with other guests, it sends a VMOID shared authorization request to the host, including the target guest ID and sharing permissions such as read or write. The host checks the sharing mode of H-MainDSVMO, adds the target guest ID to the shared guest list, writes the corresponding sharing permissions to the shared permission matrix, and updates the sharing version number. Then, the host sends a sharing invitation to the target guest, and the target guest driver updates G-ShadowDSVMO, sets the sharing role to shared guest, and completes the shared authorization.
[0081] When the main Guest needs to modify the access permissions or type of a video memory block, it must first confirm that no shared Guest is currently operating. This can be determined by checking the shared lock flag. Then, it sends a VMOID modification request to the Host. The Host updates the permission information or video memory type field of H-MainDSVMO. If it is a shared video memory block, the shared permission matrix is updated synchronously. Then, the data is synchronized to the G-ShadowDSVMO of all relevant Guests to ensure the consistency of the mapping tables between the two sides.
[0082] S2. By semantically capturing, pre-compiling, batch transmitting, and compressing the Vulkan commands on the Guest side, the memory data transfer logic of VK_KHR_external_memory_fd is replaced, reducing the number of interactions in the virtio queue and avoiding data copying, thus achieving efficient transmission of commands and data.
[0083] The S2.1 and VSPBT instruction sets feature semantic design, supporting compressed identifiers and shared fields among multiple guests.
[0084] We designed a pre-compiled instruction set based on Vulkan semantics, encapsulating Guest-side Vulkan memory operation instructions such as vkCmdUpdateBuffer, vkCmdCopyBufferToImage, and vkQueueSubmit into semantic intermediate code. The intermediate code consists of four parts: instruction header, semantic information, data payload, and shared control fields. The specific format is shown in the table below.
[0085] Table 3. Format Information of Semantic Intermediate Code
[0086] Meanwhile, an instruction rearrangement and merging algorithm was designed, with the following specific logic:
[0087] Command classification involves categorizing captured commands according to their type, such as write, copy, commit, and synchronization, with commands of the same type grouped into the same queue. A sliding window merging algorithm with a window size of 16 is used to merge consecutive commands of the same type within the window; for example, 16 `vkCmdUpdateBuffer` commands are merged into one `VSPBT` batch write command. The merging rules are: they must meet the conditions of the same VMOID and consecutive offset addresses; data payloads are merged into a continuous byte stream with a total length not exceeding the MTU; semantic tags are used for intersection, such as 0x0001 for 16 commands that are all buffer updates; and the merged command queue is sorted in descending order of priority to ensure that high-priority commands, such as frame buffer writes, are transmitted first.
[0088] S2.2. Guest-side Vulkan command capture and pre-compilation: The Guest-side Venus driver intercepts Vulkan API calls through the API hook layer and performs pre-compilation.
[0089] Command capture. A Vulkan API hook layer is implanted in the Guest-side virtual GPU driver, setting hook functions for core APIs such as vkAllocateMemory, vkFreeMemory, vkCmdUpdateBuffer, vkCmdCopyBufferToImage, and vkQueueSubmit. When the Guest application calls the target API, the hook function intercepts the call parameters, such as dstBuffer, dstOffset, dataSize, and pData of vkCmdUpdateBuffer, and records the call context, such as the current queue and command buffer handle. The parameters are validated for validity, such as whether dataSize exceeds the buffer size and whether dstOffset is aligned. If validation fails, a Vulkan standard error code such as VK_ERROR_INVALID_OFFSET is returned; if validation succeeds, the process proceeds to the next step.
[0090] Semantic mapping. The VMOID corresponding to the current operation is queried from G-ShadowDSVMO. Taking vkCmdUpdateBuffer as an example, the VMOID is obtained by matching the virtual memory address corresponding to dstBuffer within the range of the Guest virtual memory start address + memory block size in G-ShadowDSVMO. The Guest virtual address is then converted to VMOID + offset address: offset address = virtual address of the operation - Guest virtual memory start address in G-ShadowDSVMO. If it is a shared memory block, the shared roles and the number of shared Guests in G-ShadowDSVMO are queried simultaneously. If it is a shared Guest and the operation type is write, a shared write identifier is marked in the semantic information for subsequent conflict control.
[0091] Instruction merging. Sixteen instruction merging windows are initialized, corresponding to 16 instruction type combinations. Each window maintains four state variables: current VMOID, current offset address, cumulative data length, and instruction list. For captured instructions that have completed semantic mapping, they are categorized into their corresponding windows according to type. If a window is empty, the instruction is directly added to the window, and the state variables are updated: current VMOID becomes the instruction VMOID, current offset address becomes the instruction offset address, and cumulative data length becomes the instruction data length. If a window is not empty and satisfies the following conditions: the sum of the same VMOID, current offset address, and cumulative data length equals the instruction offset address, and the sum of the cumulative data length and instruction data length is not greater than the MTU, then the instruction is added to the window, and the cumulative data length and instruction list are updated. If the above conditions are not met, the merging of instructions within the window is triggered, generating one batch instruction. After clearing the window, the current instruction is added. Every 1024 captured instructions, the merging of all non-empty windows is forcibly triggered to avoid window accumulation.
[0092] Data payload compression. A compression check is performed on the data payload of the merged instruction. If the data payload length is not less than 1KB and the data type is texture or vertex data, compression is triggered. A compression algorithm is selected: if the data is structured data such as vertex coordinates, LZ4 compression is used; if it is unstructured data such as texture maps, ZSTD compression is used. After compression, the corresponding algorithm is marked in the compression identifier field of the instruction header. Simultaneously, the compressed CRC32 checksum is calculated and replaced with the original checksum. Decompression metadata, such as the compressed length and original length, is prepared for decompression on the host side and embedded in the header of the data payload.
[0093] Pre-compiled encoding. The merged instructions are binary encoded according to the VSPBT instruction format, specifically: Instruction header, the magic number, total length, instruction type, data payload length, compression flag, and priority are written in the following field order. The total length is calculated based on the total number of bytes of all fields, and the data payload length is the compressed length; Semantic information, including VMOID, offset address, semantic tags, synchronization flag, and checksum; Data payload, including the compressed payload data, containing decompression metadata; Shared control fields, if it is a shared instruction, including the initiating Guest ID, the target Guest ID, and sharing permissions.
[0094] After encoding, the binary instructions are written to the instruction buffer on the Guest side. This buffer is 64KB in size and has a circular buffer structure. It is divided into three areas: the area to be transmitted, the area in transmission, and the area that has been completed. Each area has a water level. For example, the water level of the area to be transmitted is 80% of the capacity, which is 51.2KB.
[0095] Batch triggering. Triggering conditions include the total length of instructions in the area to be transmitted reaching the watermark, capturing a rendering submission instruction such as vkQueueSubmit with a semantic flag of 0x0003, and the longest wait time for instructions in the area to be transmitted exceeding 1ms. When any of the above triggering conditions are met, all instructions in the area to be transmitted are packaged into a VSPBT batch packet. The batch packet header contains three fields: packet magic number, packet length, and number of instructions. The packet magic number is 0x5642504B and VBPK. The free space of the virtio data queue is checked. If the free space is not less than the batch packet length, the batch packet is directly written to the queue. If it is insufficient, the batch packet is split into multiple sub-packets, where the size of the sub-packet is equal to the free space in the queue, and then written in order. The transmitted instructions are moved from the area to be transmitted to the transmission phase, and the transmission timestamp is recorded to update the instruction buffer status.
[0096] Exception handling. If writing to the virtio queue fails due to a full queue, the instruction is rolled back to the waiting area and retried after 100us. If it still fails after 3 retries, VK_ERROR_DEVICE_LOST is returned. If the host side returns a verification failure response, the corresponding instruction is retrieved from the transmission area, recompressed and re-encoded, and then retransmitted.
[0097] The host-side pre-compiled instructions are parsed and restored. The host-side virglrenderer receives VSPBT batch packets through the VSPBT parsing module, performing packet verification, decompression, semantic parsing, instruction restoration, batch execution, and result feedback. The batch packet verification involves a multi-layered verification logic: the first layer verifies the magic number (0x5642504B) in the batch packet header; if it doesn't match, the packet is discarded, and a packet format error is returned to the Guest. The second layer verifies whether the packet length matches the actual received length; if they don't match, the packet is considered truncated, discarded, and an error is returned. The third layer splits the batch packets into individual instructions according to the VSPBT instruction format, verifying the magic number (0x56535042) and checksum of each instruction; if the verification fails, the instruction is discarded, an error log is recorded, and other instructions are parsed. The fourth layer verifies whether the initiating Guest ID for the shared instruction (0x05) is the same as the primary Guest ID in H-MainDSVMO; if not, insufficient permissions are determined, the instruction is discarded, and an error is returned.
[0098] Data payload decompression involves decompressing the corresponding compression algorithm. For instructions with a non-zero compression flag in the instruction header, the decompression metadata in the data payload header is extracted, including the original length and the compressed length. If the compression flag is 1 (LZ4), the LZ4 decompression function LZ4_decompress_safe is called to decompress the compressed data to a temporary buffer, and the decompressed length is checked to ensure it matches the original length. If the compression flag is 2 (ZSTD), the ZSTD decompression function ZSTD_decompress is called, and the decompression length is checked similarly. If decompression fails, the instruction is discarded, and data corruption is returned to the Guest. If decompression succeeds, the data payload is replaced with the original data.
[0099] Semantic parsing, or reverse address mapping, involves querying H-MainDSVMO based on the VMOID in the instruction to obtain information such as the corresponding Host physical memory address, VkDeviceMemory handle, sharing mode, and shared Guest list. The VMOID + offset address is then converted to the Host physical memory address, i.e., physical address = physical memory start address in H-MainDSVMO + instruction offset address. The validity of the physical address is verified by checking if the sum of the physical address and the data payload length exceeds the sum of the physical memory start address and the memory block size. If it does, it is considered an address out of bounds error, the instruction is discarded, and an error is returned. If it is a shared memory block and the instruction type is write, the sharing permissions of the initiating Guest are verified. If write permissions are not available, the instruction is discarded, and insufficient permissions are returned.
[0100] Vulkan native command mapping enables command restoration. Based on the Vulkan semantic tags of the commands, the VSPBT intermediate code is restored to the host-side Vulkan native commands. The specific mapping rules are shown in the table below.
[0101] Table 4 Vulkan Native Command Mapping Rules
[0102] During the restoration process, if the Host Vulkan driver does not support a certain native command, such as older drivers not supporting certain formats of vkCmdCopyBufferToImage, then command downgrading is triggered: for example, vkCmdCopyBufferToImage is downgraded to a combination of vkMapMemory+memcpy+vkUnmapMemory operations.
[0103] Batch execution process. The scheduling strategy is to sort the restored Vulkan instructions according to their priority from high to low. High-priority instructions, such as framebuffer writes, are submitted to the Host GPU queue first. Before executing an instruction, the locking flag of H-MainDSVMO is checked. If it is 0, the flag is set to lock the video memory block. If it is 1, the instruction is added to the waiting queue. The locking status is checked every 50us. If the timeout occurs, the resource busy status is returned. When the number of instructions in the waiting queue reaches 32 or the waiting time exceeds 200us, the instructions in the waiting queue are submitted to the GPU queue in batches to reduce the number of vkQueueSubmit calls. The instruction execution status is monitored by vkGetFenceStatus. If the execution fails, the error code is recorded and the exception recovery process is triggered, such as retrying or replacing the video memory block.
[0104] For each instruction, an execution result packet is generated, containing VMOID, instruction type, execution status, error code, synchronization version number, and shared version number. In the execution status, 0 indicates success and 1 indicates failure. The shared version number is the updated value when the instruction was shared, the synchronization version number is the updated version number, and the shared version number is the updated value when the instruction was shared. The execution result packets are categorized by Guest ID and written to the virtio response queue in batches. For example, every 128 result packets are merged into one response packet. After receiving the response packet, the Guest side updates the synchronization version number and shared version number of G-ShadowDSVMO, triggers a retry for failed instructions, up to 3 times. If the retry fails, the application layer is notified.
[0105] S3. Construct the Cross-Domain Memory Consistency Protocol (CMCP). The CMCP protocol adopts a four-layer architecture. The core components, responsibilities, and interaction methods of each layer are shown in the table below.
[0106] Table 5 CMCP Protocol Architecture Information Table
[0107] S3.1 Hardware-layer assisted synchronization: The hardware layer provides differentiated synchronization solutions for different CPU or GPU architectures to ensure physical-level memory consistency.
[0108] For architectures that support CPU-GPU cache coherency, such as Intel Xe, AMD RDNA 3, and ARM Mali-G720, after the VSPBT memory write instruction is executed on the host side, the host Vulkan driver calls the corresponding cache refresh instruction according to the CPU architecture, specifically:
[0109] For Intel architecture, the _mm_clflushopt instruction is called for optimized cache flushing with a latency of approximately 10ns, flushing the corresponding cache line in the CPU cache for that memory block; for AMD architecture, the _mm_sfence instruction is called for a storage barrier, ensuring that all storage operations are completed before executing subsequent instructions; for ARM architecture, the dsb st instruction is called for a data synchronization barrier, ensuring that storage operations are committed to memory.
[0110] Simultaneously, the Vulkan vkFlushMappedMemoryRanges interface is called, passing in the VkMappedMemoryRange containing the VkDeviceMemory handle, physical address, and data length, triggering GPU cache refresh and writing the data in the GPU cache back to the physical video memory; vkGetFenceStatus is used to wait for the cache refresh to complete, with a timeout of 100us, and after completion, the synchronization version number of H-MainDSVMO is updated.
[0111] Cache invalidation notification: When the Guest needs to read a block of video memory, the Guest virtual GPU driver sends a cache invalidation request to the Host through the virtio control queue, including the VMOID and the read range. The Host calls Vulkan's vkInvalidateMappedMemoryRanges interface, passing in the corresponding VkMappedMemoryRange, to invalidate the old data in the video memory block in the GPU cache, ensuring that the latest data is read. For the CPU cache, the corresponding invalidation instruction is called, such as Intel's _mm_clflush, to prevent the CPU from reading old data in the cache. After the invalidation is completed, the Host returns an invalidation completion confirmation to the Guest, and the Guest then performs the read operation.
[0112] GPU synchronization primitives work in conjunction with the following: When the VSPBT rendering submission command is semantically marked as 0x0003 and restored to vkQueueSubmit on the host side, a VkSubmitInfo is constructed and synchronization primitives are set. The waiting semaphore pWaitDstStageMask is set to VK_PIPELINE_STAGE_TRANSFER_BIT to ensure that the memory write operation is completed before rendering is executed; the semaphore pSignalSemaphores is set to a custom VkSemaphore to mark the completion of rendering; the Fence, i.e. pFence, is set to VkFence to monitor the rendering progress; after calling vkQueueSubmit, vkWaitForFences is used to wait for the rendering to complete, with a waiting timeout of 1ms. After completion, the cache is refreshed, and the rendering result is synchronized to the physical video memory.
[0113] For architectures that do not support CPU-GPU cache coherency, such as older NVIDIA Kepler and Intel Gen9, DMA-BUF is forcibly synchronized. On the host side, VkDeviceMemory is exported as a DMA-BUF file descriptor through vendor-specific interfaces such as NVIDIA's vkGetMemoryFdNV and Intel's vkExportMemoryFdEXT, without requiring VK_KHR_external_memory_fd and relying on vendor-specific extensions; the DMA-BUF FD is then passed to the host. The Virtio-GPU kernel driver maps DMA-BUFs through the `dma_buf_map_attachment` interface, utilizing the DMA-BUF's cache synchronization mechanism, guaranteed by the kernel DMA subsystem, to achieve data consistency. When synchronization is needed, `dma_buf_begin_cpu_access` is called before CPU access, or `dma_buf_end_cpu_access` is called after CPU access, with the kernel automatically triggering GPU cache refresh or invalidation. A double-buffering mechanism allocates two DMA-BUF blocks, A and B, for frequently accessed memory blocks such as the framebuffer, employing an alternating double-buffering strategy. When a guest writes, it first writes to block A, and when the host renders, it uses block B. After rendering, the roles of A and B are swapped to avoid synchronization waits between writing and rendering. The `dma_buf_sync` interface marks the DMA-BUF access type, such as read or write, ensuring the kernel correctly handles cache synchronization.
[0114] S3.2, Driver Layer Semantic Synchronization. The driver layer achieves command synchronization between the Guest and Host through Vulkan semantic barrier conversion and synchronization command priority transmission.
[0115] Guest-side barrier capture and conversion:
[0116] When the Guest application calls vkCmdPipelineBarrier to insert a memory barrier, the Guest Venus driver's hook function captures the instruction and parses the parameters of VkMemoryBarrier, including: the source access mask srcAccessMask, such as VK_ACCESS_TRANSFER_WRITE_BIT; the destination access mask dstAccessMask, such as VK_ACCESS_SHADER_READ_BIT; and the queue family index srcQueueFamilyIndex / dstQueueFamilyIndex.
[0117] Based on the Guest-Host semantic mapping table, the barrier parameters on the Guest side are converted into CMCP synchronization command parameters. The specific information is shown in the table below.
[0118] Table 6 Guest-Host Semantic Mapping Table
[0119] Construct a CMCP synchronization instruction, such as VSPBT instruction type 0x04, set the synchronization flag to 1, and write it to the instruction buffer.
[0120] Priority transmission of synchronization commands: Synchronization commands are set to the highest priority, higher than frame buffer commands, ensuring that synchronization commands are transmitted to the Host first; a separate channel is used for transmission, that is, a sub-queue for synchronization commands is opened in the virtio control queue, isolated from ordinary data commands, to avoid being blocked by ordinary commands; transmission frequency control, that is, when more than 8 synchronization commands are captured on the Guest side within 100us, they are merged into 1 batch synchronization command to reduce the number of transmissions; the batch synchronization command contains multiple barrier parameters, and the Host side generates a corresponding number of VkMemoryBarriers after parsing.
[0121] Host-side barrier insertion and execution: After parsing the CMCP synchronization instructions, the host side constructs a VkMemoryBarrier based on the converted srcAccessMask and dstAccessMask. Before executing the corresponding rendering instructions, the vkCmdPipelineBarrier is called to insert the barrier, ensuring that the memory access order conforms to the semantic requirements of the Guest. If the host Vulkan driver does not support a certain srcAccessMask, such as older drivers not supporting VK_ACCESS_SHADER_WRITE_BIT, then barrier downgrading is triggered: the unsupported mask is replaced with the generic memory write mask VK_ACCESS_MEMORY_WRITE_BIT, ensuring that the barrier function is effective.
[0122] S3.3 Lightweight Locking and Version Control at the Application Layer. The application layer employs optimistic locking, exponential backoff waiting, and real-time version number synchronization mechanisms to resolve concurrent access conflicts between a single Guest and Host instance.
[0123] Fine-grained implementation of optimistic locking: When requesting a lock, if the Guest needs to write to a video memory block, it first sends a lock request to the Host, including the VMOID and the request timeout. The Host queries the locking flag of H-MainDSVMO. If it is 0, it means that the device is not locked. The flag is set to 1, the Guest ID of the locker and the locking timestamp are recorded, and the lock is returned as successful. If it is 1, it means that the device is locked. The Host checks whether the locker is the current Guest and writes again. If it is, the lock is returned as successful; otherwise, the lock is busy.
[0124] To optimize lock waiting, after the Guest receives a busy lock request, it enters a spin wait with an initial waiting time of 10us. If the lock is not acquired after 10us, the waiting time is doubled, such as to 20us, 40us, 80us, etc. The maximum waiting time does not exceed the request timeout. If the lock is not acquired within the timeout period, a lock timeout is returned, and the Guest triggers a retry command, up to 3 times. If the retry fails, the application layer is notified.
[0125] After the Guest completes the write operation, it sends a lock release request to the Host, including the VMOID. The Host checks whether the lock flag of H-MainDSVMO is 1 and whether the lockr is the current Guest. If so, it resets the flag to 0 and updates the synchronization version number, i.e., the version number is incremented by 1. If there is an instruction with the VMOID in the Host's waiting queue, it automatically allocates the lock to the first instruction in the queue to avoid the lock being idle.
[0126] Real-time version number synchronization: Version number incremental synchronization occurs. Each time the host side modifies memory block data (e.g., executes a write command, updates shared permissions), the synchronization version number or shared version number of H-MainDSVMO increments by 1. Every 100µs or if the version number changes more than 8 times, the host side sends a version number synchronization packet, containing the VMOID, synchronization version number, and shared version number, to all relevant guests, including the main guest and shared guests, via the virtio control queue. Upon receiving the packet, the guest side updates the corresponding version number of G-ShadowDSVMO. If the difference between the local version number and the host version number exceeds 3, a full synchronization is triggered, i.e., the complete entry of H-MainDSVMO is re-acquired. Version number verification occurs at the following times: Before the guest side performs a read operation, it checks whether the local synchronization version number is consistent with the host version number; if not, it waits for synchronization to complete. Before the host side executes a shared command, it checks whether the shared version number of the initiating guest is consistent with the shared version number of H-MainDSVMO; if not, execution is rejected, and the guest is required to synchronize its version number first.
[0127] Conflict resolution strategy: Conflict detection is performed by using the shared lock flag of H-MainDSVMO to detect multi-Guest write conflicts. When multiple shared guests request to write simultaneously, the shared lock flag is set to 1. Priority determination criteria are as follows: for instruction type priority, framebuffer write, texture write, and static data write instructions have high priority in descending order; guest priority is that of the primary guest, which has higher priority than the shared guest; timestamp priority is that the guest that initiates the request first has higher priority than the guest that initiates the request later; conflict resolution is performed by the host-side shared permission arbitrator, which sorts conflict requests by priority and generates an execution queue; locks are allocated according to the queue order, and the next guest is allowed to execute only after the previous guest completes the write and releases the lock; during execution, version number synchronization ensures that all guests can obtain the latest data.
[0128] S3.4 Detection and recovery of consistency anomalies.
[0129] A multi-layered anomaly detection and tiered recovery mechanism is designed to handle memory consistency anomalies. Anomaly detection methods include periodic verification, event-triggered verification, and multi-Guest cross-verification. Periodic verification involves scanning the H-MainDSVMO every 200µs to compare the increments of the synchronized version number and the shared version number. A normal increment is a change of 1 for each modification; if the increment jumps, the version number is considered abnormal. Event-triggered verification triggers immediate verification when an instruction execution fails (e.g., vkQueueSubmit returns VK_ERROR_OUT_OF_DATE_KHR) or when the data checksum does not match. Multi-Guest cross-verification involves the primary Guest periodically sending a data verification request to the shared Guest. The shared Guest returns the CRC32 value of the data at a specified offset address. The primary Guest compares this value with its local data; if they do not match, a cross-Guest data anomaly is identified.
[0130] Exception Types and Recovery. For version number anomalies, a full DSVMO synchronization is triggered. The Host sends a full synchronization request to all relevant Guests, including the complete entries in the H-MainDSVMO. Upon receiving this request, the Guest clears the G-ShadowDSVMO and reinitializes it to ensure version number consistency. After synchronization, the command before the anomaly is executed again. For data verification failures, if the data is corrupted during transmission (e.g., checksum mismatch), the Host sends a data retransmission request to the Guest, requesting the Guest to retransmit the data payload of the command. If retransmission fails after three attempts, the data source is considered corrupted. The Host restores the data from a backup, such as a copy in system memory. If no backup exists, a data loss report is returned, and the application layer is notified. For hardware-level memory errors, when the Host Vulkan driver returns VK_ERROR_DEVICE_LOST or the kernel detects a GPU memory error, it is determined to be a hardware-level exception. The Host side marks the memory block as corrupted, such as when the H-MainDSVMO health status is 2, and removes it from the memory pool. A new memory block is allocated to the relevant Guest, the valid data in the old block is copied to the new block, and the DSVMO mapping is updated. The Guest is notified that the application-layer memory block has been replaced, ensuring that the application is unaware of the switch.
[0131] S4. Implements intelligent dynamic scheduling and fragmentation management of the video memory pool. Through refined heat modeling, improved scheduling algorithms, and real-time fragmentation management, it optimizes the utilization of video memory resources and supports efficient memory sharing.
[0132] S4.1, Thermal modeling and grading of intelligent memory pools.
[0133] Based on statistical information from H-MainDSVMO, such as access frequency, last access timestamp, and read / write ratio, a multi-dimensional heat model is established for each memory block. This includes heat parameter calculations: Access frequency (TPS) is the number of accesses per 100ms, including reads and writes, calculated as: TPS = (reads + writes) / 0.1s; Time decay factor is calculated based on the last access timestamp, calculated as: Decay factor = 1 - (current time - last access timestamp) / (30s), where 30s is the decay period, and the decay factor is 0 if there is no access for more than 30s; Read / write weight is calculated as: Write operations contribute more to heat than read operations, calculated as: Read / write weight = (writes * 2 + reads) / (reads + writes), with a weight of 1 when there are no writes; Finally, the heat value is calculated as: Heat value = TPS * Decay factor * Read / write weight, with a value range from 0 to 100.
[0134] Popularity grading criteria: Hot blocks, with a popularity value of not less than 70, correspond to the first-level pool (high-frequency pool), storing high-frequency read / write resources such as frame buffers and real-time textures; Warm blocks, with a popularity value of not less than 30 and less than 70, correspond to the second-level pool (medium-frequency pool), storing mid-frequency access resources such as temporary render targets and shader code; Cold blocks, with a popularity value less than 30, correspond to the third-level pool (low-frequency pool), storing low-frequency access resources such as static models and configuration files. Threshold dynamic adjustment: When the utilization rate of a certain level pool exceeds 90%, the threshold is automatically adjusted. For example, if the utilization rate of the first-level pool is too high, the hot block threshold is increased from 70 to 75, and some low-population hot blocks are downgraded to warm blocks.
[0135] Dynamic adjustment of heat level: The heat level values of all memory blocks are recalculated every 100ms to ensure real-time heat level accuracy. Migration trigger conditions include: migration from a hot block to a warm block requires a heat level value of less than 70 for three consecutive calculations; migration from a warm block to a hot block requires a heat level value of not less than 70 for three consecutive calculations; migration from a warm block to a cold block requires a heat level value of less than 30 for five consecutive calculations; migration from a cold block to a warm block requires a heat level value of not less than 30 for five consecutive calculations. The migration process is as follows: after determining the migration direction, the host side checks the target... If there is enough free space in the pool, the memory block is migrated directly; otherwise, the cold block in the target pool is released first. During migration, the memory block data is copied from the original address to the new address in the target pool using vkCopyMemory, and the physical memory start address, memory type, and other fields of H-MainDSVMO are updated. A migration notification is sent to the relevant Guest, and the virtual address mapping of G-ShadowDSVMO is updated to ensure that the Guest is unaware of the migration. After the migration is completed, the memory block at the original address is marked as free and included in the original pool for reuse.
[0136] S4.2 Intelligent scheduling of video memory objects. An improved LRU-K algorithm (K=2) and a Markov chain prediction model are used to realize the dynamic scheduling of video memory blocks, while supporting fine-grained management of sparse video memory.
[0137] The improved LRU-K algorithm (K=2) determines the access frequency of each memory block by recording the timestamps of the first two accesses and prioritizes retaining blocks with higher access frequencies. Each time a memory block is accessed, the last access timestamp is updated, and the original last access timestamp is stored in the previous access timestamp field. The previous access timestamp field is also added to the statistics of H-MainDSVMO. The eviction priority is calculated as current time - previous access timestamp, with a higher value indicating a higher eviction priority. When the memory pool is insufficient, memory blocks are released in descending order of eviction priority.
[0138] The preloading strategy is implemented based on the Markov chain prediction model: collect historical access data, such as the VMOID access sequence within 1 hour, and construct a state transition matrix. The matrix element P(i,j) represents the probability of transitioning from memory block i to memory block j after accessing memory block i. Monitor the currently accessed memory block in real time. For example, if the memory block with VMOID A is accessed, query the state transition matrix to obtain the top 3 memory blocks with the highest probability of transitioning from A, such as B, C, and D. Check whether these 3 memory blocks are in the cold block pool and have not been loaded into the high-frequency pool. If so, preloading is triggered, that is, they are moved to the first-level pool and pre-allocated to the GPU cache. The state transition matrix is retrained every 30 minutes to adapt to changes in access patterns, such as changes in access sequences caused by game scene switching.
[0139] A pre-release strategy is implemented through tiered release and data backup: Release priority is ordered as follows: the first priority is unreferenced memory blocks in the cold block pool (i.e., reference count of 0); the second priority is low-referenced memory blocks in the cold block pool (e.g., reference count of 1 and no access within 2 minutes); the third priority is unreferenced memory blocks in the warm block pool. A data backup mechanism checks the data type of the memory block before release. If it is static data such as models or configurations, it is backed up to the host system memory, using the hugetelbfs file system to reduce page table overhead. If it is dynamic data such as temporary rendering targets, it is released directly without backup. After backup, the backup path and data length are recorded in H-MainDSVMO for quick recovery when needed.
[0140] Dynamic management of sparse video memory: Sparse block allocation: When a Guest requests sparse video memory (i.e., video memory type 2), the Host does not pre-allocate a complete video memory block, but dynamically allocates it in 256KB sparse blocks. When the Guest writes to a sparse block, the Host checks whether the block has been allocated. If not, it calls vkAllocateMemory to allocate a 256KB block, then updates the sparse block list of H-MainDSVMO, adding a field to store the offset address of the allocated sparse block; Sparse block release: When a sparse block is not accessed within 3 minutes, it is automatically released. If it contains important data, the data is backed up to system memory; Sparse block merging: When adjacent sparse blocks are allocated and accessed consecutively, they are merged into 512KB blocks to reduce the number of blocks and lower management overhead.
[0141] S4.3 Real-time Fragmentation Detection and Intelligent Merging. A multi-dimensional fragmentation detection index and a phased merging algorithm are constructed to reduce the fragmentation rate of video memory.
[0142] Fragmentation detection metrics include core metrics and auxiliary metrics. The core metric is the fragmentation rate, calculated as: Fragmentation Rate = Total number of free discrete blocks / Total number of memory blocks * 100%. Auxiliary metrics include the maximum consecutive free block length, the standard deviation of free block size, the gap ratio, and the detection frequency. The maximum consecutive free block length is the maximum length of all consecutive free blocks, reflecting the maximum allocatable block size. The standard deviation of free block size reflects the uniformity of free block size distribution; the larger the standard deviation, the more severe the fragmentation. The gap ratio is calculated as: Total length of free blocks / Total memory length * 100%. Merging is triggered if the gap ratio is greater than 20% and the fragmentation rate is greater than 10%. The detection frequency is calculated by scanning H-MainDSVMO every 500ms. If the fragmentation rate is greater than 10% and the maximum consecutive free block length is less than the current maximum requested block size, merging is triggered.
[0143] Phased merging algorithm: Phase 1: Region scanning and marking. The host GPU memory is divided into multiple management regions of 1GB each, and each region is scanned independently. For each region, free blocks are traversed in ascending order of physical address, and consecutive free blocks (i.e., consecutive addresses and all free blocks) are marked. Consecutive free blocks with a length of not less than 256MB are directly marked as blocks to be merged. For blocks with a length less than 256MB, their address and length are recorded, and the process proceeds to Phase 2. Phase 2: Greedy merging. Free blocks with a length less than 256MB are sorted in ascending order of physical address. Starting with the block with the smallest address, attempts are made to merge it with subsequent blocks: if the current block address + current block length = the address of the next block, then they are merged into one block, and the length is updated, until the length of the merged block is not less than 64MB or there are no subsequent blocks to merge. The merged block is marked as a block to be merged, and unmerged small blocks are retained. The third stage involves merging and updating the mapping. For blocks to be merged, the vkMergeMemory interface of the Host Vulkan driver can be called, or the merging can be achieved by allocating new blocks, copying data, and releasing old blocks. The physical memory start address, memory block size, fragmentation level, and other fields of H-MainDSVMO are updated, and the old block table entries to be merged are deleted. A merge synchronization notification is sent to the relevant Guest, and the virtual address offset of G-ShadowDSVMO is updated to ensure mapping consistency.
[0144] Further optimize the timing of merging: obtain the busy / idle status of the GPU queue through vkGetQueueFamilyProperties, and only perform merging when the queue idle rate is greater than 50% to avoid peak GPU rendering times; merge in order of decreasing priority from low-frequency pool to mid-frequency pool to high-frequency pool to reduce interference with high-frequency access resources.
[0145] S4.4 Overload Protection and Degradation Strategy for Video Memory Resources. When the host GPU's video memory utilization exceeds the threshold, dynamic overload protection is triggered, reducing video memory usage through a three-level strategy of resource release, rendering degradation, and load migration.
[0146] Overload detection and dynamic threshold adjustment:
[0147] Memory utilization: Memory utilization = allocated memory length / total memory length * 100%; Dynamic thresholds include normal threshold, warning threshold, and emergency threshold. Normal threshold: no overload protection if utilization is less than 85%. Warning threshold: mild protection (pre-release of cold blocks) is triggered when utilization is not less than 85% but less than 95%. Emergency threshold: severe protection (rendering degradation and load migration) is triggered when utilization is not less than 95%. Threshold dynamic adjustment is based on factors such as lowering the warning threshold to 80% when GPU temperature exceeds 80℃, and lowering the emergency threshold to 90% when rendering frame rate is less than 30fps, to avoid hardware overload.
[0148] Three-level overload protection strategy:
[0149] The pre-release of cold blocks provides mild protection by releasing unreferenced memory blocks in the cold block pool. The release order is to release data with low backup cost first, followed by data with high backup cost. Preloaded cold blocks, such as blocks that are predicted to be loaded, are released in advance if they are not accessed, thus canceling the preloading. The protection goal is to reduce memory utilization to below 85%.
[0150] The rendering degradation is moderately protected by gradually reducing the rendering resolution to 1080P, 720P, and 480P, with each degradation reducing VRAM usage by 25%. Special effects are disabled by priority, with the order being: first, MSAA anti-aliasing, then lighting effects, texture filtering, and dynamic shadows, each disabling reducing VRAM usage by 5% to 15%. Uncompressed textures such as RGBA8888 are converted to compressed formats such as ETC2 and ASTC to reduce texture VRAM usage. The protection goal is to reduce VRAM utilization to below 80%.
[0151] Heavy protection for load migration: Identify guests with high VRAM usage. If a guest's VRAM usage exceeds 20% of the total VRAM, migrate some of its non-real-time tasks, such as texture pre-compilation and model loading, to the host CPU for execution. If the host has multiple GPUs, migrate some guests to idle GPUs to balance the load. For guests that cannot be migrated, limit their VRAM allocation, such as a maximum allocation of 15% of the total VRAM, to avoid monopolizing resources. The protection goal is to reduce VRAM utilization to below 75%.
[0152] Overload recovery strategy: When the video memory utilization rate is below the normal threshold such as 85% for 10 consecutive seconds, overload recovery is triggered, and the resolution is restored in a gradient of 480P, 720P, and 1080P; the special effects are restored in the order of dynamic shadows, texture filtering, lighting effects, and anti-aliasing; tasks migrated to the CPU or GPU are migrated back to the original GPU or Guest; during the recovery process, version number synchronization is used to ensure data consistency and avoid screen flickering.
[0153] S5. For scenarios where multiple guests share the host GPU's video memory, a video memory block-level permission control and data isolation mechanism is designed to ensure sharing security.
[0154] S5.1 Fine-grained management of shared permissions.
[0155] Permission types include three categories: read-only, read-write, and execute. Permission combinations are implemented through bitmaps, such as bit 0 corresponding to read, bit 1 corresponding to write, and bit 2 corresponding to execute.
[0156] The permission granting method is as follows: The main Guest sends a shared authorization request to the Host, including the target Guest ID, VMOID, and permission bitmap; the Host checks the main Guest's permissions, which must be read and write permissions. If valid, the Host updates the shared Guest list and shared permission matrix of H-MainDSVMO; a shared invitation is sent to the target Guest, including the VMOID and permission bitmap; after the target Guest accepts the invitation, the Host updates the shared role and local access permissions of G-ShadowDSVMO, completing the authorization.
[0157] The permission revocation method is as follows: The main Guest sends a shared revocation request to the Host, including the target Guest ID and VMOID; the Host deletes the target Guest's record from the shared Guest list and permission matrix of H-MainDSVMO; sends a shared revocation notification to the target Guest; the target Guest updates G-ShadowDSVMO, deletes the corresponding table entry, and releases the relevant resources.
[0158] S5.2 Data isolation mechanism.
[0159] Address isolation: Each Guest's virtual memory address space is independent. The virtual address offset of the Guest is used by G-ShadowDSVMO to ensure that the virtual addresses do not overlap when different Guests access the same physical memory block; Access control: When the Host executes instructions for a shared Guest, its sharing permissions are verified. If a read-only Guest performs a write operation, the operation is rejected and an insufficient permission is returned. If a Guest without execution permissions executes a shader instruction, the operation is rejected; Data encryption: Sensitive data, such as user-input textures, is encrypted with AES-128 and stored in the memory block. When sharing, the decryption key is only sent to authorized Guests to ensure data isolation.
[0160] S5.3 Shared Auditing and Monitoring.
[0161] Audit Logs: Record all shared operations, including authorization, revocation, and access. The logs include the operation timestamp, initiating Guest ID, target Guest ID, VMOID, permission type, and operation result. Real-time Monitoring: Count the number of accesses to the shared Guest every 100ms. If the access frequency of a shared Guest is abnormal, such as more than 1,000 accesses per second, it is marked as suspicious access, and an alarm is sent to the main Guest. Anomaly Blocking: If unauthorized access to a shared Guest is detected, such as a read-only Guest attempting to write, its access is temporarily blocked, and an alarm log is recorded.
[0162] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for efficient cross-virtual machine video memory sharing based on bidirectional shadow memory object mapping, characterized in that, Specifically, the following steps are included: When the host Vulkan driver does not support the VK_KHR_external_memory_fd extension, a main mapping table is created on the host and a shadow mapping table is created on the guest, using the memory object identifier as the key. The host pre-allocates the memory pool and obtains the memory handles of each memory block. It assigns a globally unique memory object identifier to each memory block, configures a default sharing policy, writes the memory object identifier, physical memory address, memory handle, and default sharing policy into the main mapping table, and sends an initialization synchronization packet to the guest via the Virtio control queue. The client fills in the shadow mapping table, sets the shared role to non-shared, and returns an acknowledgment to the host via the Virtio control queue; When a client application performs a controlled operation, the client obtains the target virtual address of the target memory block for that operation. Based on this, it looks up the corresponding memory object identifier in the shadow mapping table and converts the target virtual address into a combination of the memory object identifier and the instruction offset address. The instruction offset address is the difference between the target virtual address and the virtual memory start address corresponding to the memory object identifier. A semantic tag representing the Vulkan semantics of the operation instruction is generated. According to the operation instruction type, it is assigned to the corresponding instruction merging window. The instruction merging window maintains the memory object identifier, offset address, cumulative data length, and instruction list to form an instruction unit. The instruction unit is encoded into a binary instruction that sequentially includes an instruction header, semantic information, data payload, and shared control fields. This instruction is written as a transmission instruction into the client's transmission area. When the trigger condition is met, the transmission instruction is packaged into a batch packet and sent to the Virtio data queue and the host is notified. The transmission instruction is then transferred to the transmission area. The host retrieves batch packets from the Virtio data queue. After verification, it queries the main mapping table to convert the video memory object identifier and instruction offset address into physical video memory address. Based on semantic tags, the binary instructions are restored to Vulkan native instructions and submitted to the GPU queue for execution. The execution result is returned to the client via the Virtio data queue. The client updates the shadow mapping table accordingly and transfers the executed instructions to the completed area.
2. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, When assigning an operation instruction to the corresponding instruction merging window based on its type, if the operation instruction and the operation instruction in the instruction merging window belong to the same video memory object identifier, have consecutive instruction offset addresses, and the cumulative length after merging is not greater than the maximum transfer unit, then the operation instruction is merged into the window. Otherwise, the accumulated operation instructions in the instruction merging window are merged into an instruction unit and output. After clearing the window, the operation instruction is added again, and each instruction merging window is periodically forced to output instruction units according to a set number.
3. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, The primary client is the first client to request and obtain the video memory block, possessing full read and write permissions to it, and responsible for initiating and controlling the sharing authorization, modification, and reclamation of the video memory block. When the primary client shares the video memory block with other clients, it sends an authorization request carrying the target client's identifier and sharing permissions to the host. After the host verifies the request, it adds the target client to the sharing list, writes the sharing permissions, increments the sharing version, and then sends a sharing invitation to the target client. The target client creates a shadow mapping table entry based on the invitation and sets it as a sharing client.
4. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, During system operation, when a client requests a video memory block, the client driver sends a video memory object identifier request to the host. The request carries the video memory type, required size, access permissions, and sharing permission identifier. The sharing permission identifier is a boolean field indicating whether the video memory block is allowed to be accessed by other clients. The host-side Virtio-GPU kernel driver selects a free memory block from the memory pool, obtains the main mapping table entry with a reference count of 0 that was established during the pre-allocation of the memory block, updates the reference count from 0 to 1, sets the identifier of the host client, and initializes the shared client list and shared permission matrix to empty. Then, set the sharing mode according to the value of the sharing permission flag in the request. When the flag is not to share, set the sharing mode to exclusive mode. When the flag is to allow sharing, set it to a shareable pending authorization state. The host returns the video memory object identifier and the client virtual address offset to the host client. The host client driver updates the corresponding initialized entry in the shadow mapping table accordingly and sets the shared role to the host client. When releasing a video memory block, for non-shared video memory blocks, when the client application releases video memory, the client driver sends a video memory release request to the host. The host updates the corresponding entry in the main mapping table, decrements the reference count by 1, and when the reference count reaches 0, the video memory block is marked as free and added to the video memory pool for reuse. Then, a release confirmation is sent to the client, and the host driver deletes the corresponding entry from the shadow mapping table. For shared video memory blocks, when the host client application releases video memory, the host client driver sends a shared termination notification to all shared clients. After all shared clients confirm the release, the host client then sends a video memory release request to the host. The host clears the shared client list and shared permission matrix, and decrements the reference count by 1.
5. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, The host client sends a sharing authorization request to the host, which includes the video memory object identifier, the target client identifier, and sharing permissions. The host first verifies that the sharing mode of the entry corresponding to the video memory object identifier in the main mapping table is shareable. After the verification is successful, the host adds the target client identifier to the shared client list, writes the granted sharing permissions item by item into the sharing permission matrix, and increments the sharing version number. Then, the host sends a sharing invitation to the target client, carrying the video memory object identifier, video memory block size, client virtual address range, sharing permissions, and sharing version number. After receiving the sharing invitation, the target client creates a corresponding entry in its own shadow mapping table according to the invitation information and sets its sharing role to shared client, thus completing the sharing authorization.
6. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, The host-pre-allocated video memory pool is divided into three levels according to the access frequency of the stored data: the first level pool is used to store frame buffers and data of commonly used texture classes whose access frequency is greater than the maximum threshold, and supports read-only sharing among multiple clients; The secondary pool is used to store temporary rendering targets and data whose access frequency is no greater than the maximum threshold and greater than the minimum threshold. It supports read and write sharing by multiple clients and requires authorization from the host client. The third-level pool is used to store static models and configuration data whose access frequency does not exceed the minimum threshold, and supports read-only sharing among multiple clients. When a client requests a memory block, the host selects a free memory block from the corresponding memory pool based on the expected access frequency of the data stored in that memory block, allocates it to the client, and updates the master mapping table. During operation, the host re-determines the frequency range of each memory block according to the measured access frequency of each memory block according to a set period, migrates the memory block to the memory pool at the level corresponding to that frequency range, and synchronously updates the master mapping table and the shadow mapping table of the corresponding client.
7. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, For instruction units obtained from the instruction merging window, compression is performed based on their data payload. Data compression is triggered only when the data payload length is not less than a threshold and its data type is determined to be texture data or vertex data by semantic tagging. If the data payload is structured data, the LZ4 algorithm is used for compression; if the data is unstructured data, the ZSTD algorithm is used for compression. After compression, the compression identifier field in the instruction header marks the compression algorithm used. At the same time, a checksum is calculated for the compressed data, and the original checksum of the instruction is replaced with the checksum. Then, decompressed metadata containing the original length and the compressed length is embedded in the header of the data payload.
8. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, This also includes hardware-level cache synchronization: For architectures that support CPU and GPU cache consistency, after each memory write instruction is executed, the host sends a memory range containing the memory object handle, address, and data length to the Vulkan driver, triggering GPU cache refresh and writing the data in the cache back to physical video memory; when the client needs to read a block of video memory, it sends a cache invalidation request carrying the video memory object identifier and the read range to the host via the Virtio control queue. The host invalidates the mapped cache of that range and calls the corresponding CPU cache invalidation instruction according to the CPU architecture. After the invalidation is completed and an acknowledgment is returned, the client executes the read; for rendering commit instructions, the host constructs commit information during restoration, sets the waiting phase to the transfer phase so that memory writes are executed before rendering, and sets the completion semaphore and fence. After committing, it waits for rendering to complete before triggering cache refresh to write the rendering result back to physical video memory; for architectures that do not support CPU and GPU cache consistency, the host exports the memory object as a DMA-BUF descriptor and hands it over to the host-side Virtio-GPU kernel driver for mapping. The kernel DMA subsystem automatically triggers GPU cache refresh and invalidation before and after CPU access.
9. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, It also includes semantic barrier processing at the driver layer: when a client application inserts a memory barrier, the client side captures and converts it into host-side barrier parameters, constructs synchronization instructions and writes them into the instruction buffer. The synchronization instructions are transmitted in the highest priority through the synchronization instruction sub-queue in the Virtio control queue. When the number of synchronization instructions captured within a set time exceeds a set number, they are merged into a batch synchronization instruction carrying multiple barrier parameters. After parsing by the host, a corresponding number of memory barriers are generated, and the host-side memory barriers are inserted before executing the corresponding rendering instructions according to the converted source and target access masks. When the host Vulkan driver does not support a certain source access mask, barrier degradation is triggered, and it is replaced with a general memory write mask.
10. The method for efficient cross-virtual machine video memory sharing according to claim 1, characterized in that, It also includes application-layer locking control: Before the client writes to the video memory block, it sends a lock request to the host, carrying the video memory object identifier and the request timeout time. The host queries the lock flag bit in the main mapping table. If it is not locked, it sets the flag, records the locking client identifier and timestamp, and returns a lock success. If it is already locked, it determines whether the locker is the current client. If yes, it returns a success due to reentrancy; otherwise, it returns a lock busy. After receiving a lock busy, the client spins and waits. The initial waiting time is set, and then it increases by multiples but does not exceed the request timeout time. If the lock is not acquired after the request timeout time, it returns a lock timeout and retryes the instruction. It can retry a maximum of the set number of times. If the retry still fails, it notifies the application layer. After the client completes the write, it sends a lock release request. The host confirms that the lock flag bit is set and that the locker is the current client. It then resets the flag bit, increments the synchronization version number, and automatically grants the lock to the first instruction in the waiting queue when there is a waiting instruction for the video memory block.
