A hardware graphics interface-based cross-running environment thumbnail efficient acquisition method

Abstract

The application discloses a cross-running environment thumbnail efficient acquisition method based on a hardware graphical interface, and establishes a time slice scheduler in a host computer to realize scheduling of thumbnail capture related logical commands, preallocates a plurality of video memory buffer groups and a synchronization object pool in a container, parses logical commands to build physical commands, adopts the physical commands to multiplex the video memory buffer groups to realize copying of the exchange chain images in a frame buffer, multiplexes synchronization objects to control synchronous execution of the physical commands, and finally realizes acquisition of the container application thumbnail by the host computer based on cross-running environment sharing of the video memory buffer groups, so that the capture efficiency of the cross-running environment thumbnail is effectively improved, system resource consumption is greatly reduced, and the method can be widely applied to various running environments.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics processing technology, specifically relating to a method for efficient acquisition of thumbnails across operating environments based on a hardware graphics interface. Background Technology

[0002] When running an application from another operating system on a host operating system using container technology or virtualization technology, in order to achieve seamless switching between the host operating system application and these other operating system applications, it is necessary to simultaneously display runtime thumbnails of all applications in different operating systems during the switching process. Real-time acquisition of running application thumbnails becomes a fundamental and critical function necessary for seamless application integration and switching.

[0003] However, existing technologies have the following limitations: performance bottlenecks. Existing methods rely on the operating system's screenshot APIs, such as View.draw / Window.PixelCopy / MediaProjectionManager on Android, BitBlt on Windows, and X11 screenshot on Linux. These APIs are usually abstracted in multiple layers, resulting in low performance and making it difficult to achieve stable high-frequency captures of more than 30 times per second. High resource consumption. Existing solutions generally require transferring image data from GPU memory to CPU memory for scaling and other processing, which not only consumes a lot of memory bandwidth but also consumes excessive CPU resources. High latency. The round-trip data transfer between the GPU and CPU inevitably leads to latency, making it difficult to meet the needs of scenarios with high real-time requirements, such as game live streaming previews. Synchronization issues. Existing methods struggle to accurately synchronize the capture operation with the application's rendering process, often resulting in screen tearing or incompleteness. Poor scalability. When a large number of applications are running simultaneously, the performance of existing methods drops sharply, making it difficult to maintain a stable capture frequency.

[0004] In summary, how to efficiently capture thumbnails of multiple running applications across different runtime environments using low-level graphics interfaces such as Vulkan or OpenGL ES has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a method for efficient acquisition of thumbnails across runtime environments based on a hardware graphics interface, which realizes efficient capture of container application thumbnails by the host machine with low transmission latency.

[0006] This invention provides a method for efficient acquisition of thumbnails across operating environments based on a hardware graphics interface, specifically including the following steps:

[0007] Step 1: Start the time-slice scheduler and performance monitor on the host machine. When the system load is higher than the threshold, send a system resource-limited message to the container. Start the container, create a physical command queue containing physical commands for thumbnail capture, pre-allocate a shared video memory buffer group, and build a synchronization object pool containing synchronization objects.

[0008] Step 2: The host machine determines the first priority of the container and constructs a logical command, i.e., the first scheduling command, to capture the application thumbnail of the container. The time slice scheduler adds the first scheduling command with an earlier generation time and higher first priority to a time slice with an available time slot and an earlier execution time. The first scheduling command in the time slice is sent to the corresponding container according to the schedule.

[0009] Step 3: The container obtains the second priority of the container application according to the first scheduling command, converts the first scheduling command into a second physical command, reuses idle synchronization objects and idle video memory buffer groups, adds an application identifier to the video memory buffer group, and adds the second physical command to the physical command queue according to the generation time and the second priority.

[0010] If a resource-constrained message is received, the physical command queue is executed sequentially, the synchronization object controls the synchronous execution of the second physical command, and then the scaling quality is reduced to copy the swap chain image from the frame buffer to the video memory buffer group; otherwise, it is copied at the original scaling quality.

[0011] Step 4: The host machine reads the thumbnail to be captured from the video memory buffer group according to the application identifier.

[0012] Furthermore, the length of the time slice is the shortest total time required for all applications to be captured to update the thumbnail once.

[0013] Furthermore, when the system load recovers to below a threshold, a system resource recovery message is sent to the container. Upon receiving the message, the container restores the swap chain image to its original scaling quality.

[0014] Furthermore, the video memory buffer group contains multiple buffers that support different resolutions.

[0015] Furthermore, a command buffer is established in the container. When the container receives multiple concurrent first scheduling commands, it records multiple second physical commands converted from the concurrent first scheduling commands into the command buffer, and then merges all commands in the command buffer and adds them to the physical command queue.

[0016] Furthermore, after sorting the concurrent second physical commands according to the priority of the container application, they are sequentially recorded into the command buffer.

[0017] Furthermore, when copying the swap chain image from the frame buffer to the video memory buffer group, the GPU's hardware texture sampling synchronization is used to convert the resolution of the swap chain image to a uniform pixel format.

[0018] Furthermore, multiple reference buffers are allocated when the container starts up to store a thumbnail of the previous frame of the container application as a reference frame. When the display interface of the container application is static or the changing area is less than a threshold, the difference between the current frame of the container application and the reference frame is compared to generate a difference mask. The difference mask is then copied to the video memory buffer group, and the reference frame is updated at the same time.

[0019] Furthermore, when copying the swap chain image from the frame buffer to the video memory buffer group, the scaling quality of the swap chain image is dynamically adjusted. Bilinear or bicubic filtering is used for high-priority container applications, nearest-neighbor filtering is used for low-priority container applications, and motion-compensated scaling is enabled for video container applications.

[0020] Furthermore, a mapping relationship is established between the container application and the graphics layer. The application's frame buffer information is queried, and a mapping relationship including window handle, process ID, frame buffer handle, layer ID, position coordinates and size is established. A hash table structure is used to optimize the query of the mapping relationship. Beneficial effects

[0021] This invention achieves the scheduling of thumbnail capture-related logical commands by establishing a time-slice scheduler in the host machine, pre-allocating multiple video memory buffer groups and a synchronization object pool in the container, parsing the logical commands to construct physical commands, using the physical commands to reuse the video memory buffer groups to copy the swap chain image in the frame buffer, and using the synchronization objects to control the synchronous execution of the physical commands. Finally, based on the cross-runtime environment sharing of the video memory buffer groups, the host machine can obtain the container application thumbnail, which effectively improves the cross-runtime environment thumbnail capture efficiency and significantly reduces system resource consumption, and can be widely used in a variety of runtime environments. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a method for efficiently acquiring thumbnails across operating environments based on a hardware graphics interface, as provided by the present invention. Detailed Implementation

[0023] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0024] This invention provides an efficient method for obtaining thumbnails across runtime environments based on a hardware graphics interface. The core idea is as follows: A time-slice scheduler is established in the host machine to schedule logical commands related to thumbnail capture; multiple video memory buffer groups and a synchronization object pool are pre-allocated in the container; logical commands are parsed to construct physical commands; physical commands are reused in the video memory buffer groups to copy the swap chain image in the frame buffer; synchronization objects are reused to control the synchronous execution of physical commands; and finally, the host machine obtains thumbnails for container applications based on cross-runtime environment sharing of the video memory buffer groups.

[0025] This invention provides a method for efficient acquisition of thumbnails across operating environments based on a hardware graphics interface, the process of which is as follows: Figure 1 As shown, the specific steps include:

[0026] Step 1: Start the time-slice scheduler on the host machine, limit the time slice, which is the length of the processing cycle, and schedule the application to the corresponding time slice. The time slice contains multiple time slots that correspond one-to-one with the application. The length of the time slice is the length of the processing cycle, which is the shortest total time required for all the thumbnails of the applications to be captured to complete one update. Start the performance monitor to monitor the system load in real time, including GPU load, capture performance, etc. When the system load is higher than the threshold, send a system resource limited message to the container, and send a system resource recovery message to the container when the system load recovers to below the threshold.

[0027] The time-slice scheduler schedules the processing cycles of application-related capture operations based on the container's priority and the time when the capture operation occurs, ensuring that the capture frequency requirement of 30 times per second is met while avoiding screen tearing.

[0028] Step 2: Start the container, initialize the graphics interface context within the container to enable the container application to access the system-level frame buffer, create a physical command queue containing physical commands related to thumbnail capture, pre-allocate multiple memory buffer groups in GPU memory, and build a synchronization object pool. The physical command queue has higher priority than the application's rendering command queue. The memory buffer groups can be shared across runtime environments and are used to store captured thumbnails of the container application. The synchronization object pool contains multiple synchronization objects used to control the synchronous execution of commands in the physical command queue.

[0029] Existing methods typically share a command queue with the application, making thumbnail capture operations susceptible to being blocked by the application's rendering tasks. Therefore, this invention employs a dedicated queue, namely a physical command queue, to perform thumbnail capture operations, avoiding interference with the application's normal rendering tasks and ensuring that capture commands are executed with priority, maintaining a capture frequency of 30 FPS even under high system load.

[0030] The number of pre-allocated video memory buffer groups can be set to the maximum number of concurrent applications supported by the host system, such as 64.

[0031] To further improve the generalization capability of the pre-allocated video memory buffer group, the buffers in the pre-allocated video memory buffer group of this invention correspond to different resolutions, so that images of different resolutions can be stored. For example, the video memory buffer group contains buffers of five resolutions, so that when the host system switches between landscape and portrait modes, it can adapt to application thumbnails of different resolutions.

[0032] In addition, to further optimize the access speed of the video memory buffer group, all buffers in the video memory buffer group in this invention adopt a linear layout and support copy target and sampling access.

[0033] The sharing of video memory buffer groups across runtime environments is achieved through shared memory or virtual PCI devices, allowing the host operating system to obtain thumbnails of container applications through this shared video memory buffer group.

[0034] The synchronization object pool contains multiple synchronization objects, corresponding to a set of semaphore and fence objects. The number of synchronization objects is 2 to 3 times the number of potentially concurrent capture tasks, such as 128 semaphores and 64 fences. These synchronization objects are used to precisely control the execution order of multiple capture operations, avoiding resource contention and operational conflicts. Existing methods dynamically create synchronization objects for each capture task, incurring creation and destruction overhead. The synchronization object pool established in this invention, however, can disseminate the creation overhead of synchronization objects to the initialization phase through a reuse mechanism, improving runtime performance.

[0035] Step 3: The host machine determines the first priority for the started containers. When a request to capture the application thumbnail is generated in the host machine, the host machine parses the request to obtain the container application to be captured and its container. The container application is recorded as the current application, the first scheduling command is constructed and added to the logical command queue. The time slice scheduler queries the current logical command queue and adds the first scheduling command with an earlier generation time and a higher first priority to the time slice with an idle time slot and an earlier execution time. When the time slice scheduler rotates to a certain time slice, the first scheduling command in that time slice will be sent to the container where the current application is located.

[0036] The first scheduling command is a logical command built by the host machine to perform thumbnail capture operations, which includes the source and priority of the application to be captured, as well as the command generation timestamp.

[0037] Step 4: The container parses the received first scheduling command, obtains the priority of the current application in the container related to the first scheduling command, and records it as the second priority; converts the first scheduling command into a second physical command, and adds the second physical command to the physical command queue according to the generation time and the second priority; searches for an idle synchronization object in the synchronization object pool, records it as the current synchronization object, and reuses the current synchronization object for the second physical command; selects an idle video memory buffer group to store the thumbnail of the current application, and adds the identifier of the current application to the video memory buffer group.

[0038] When a container receives multiple concurrent first scheduling commands, to reduce the command overhead of the container application, this invention establishes a command buffer within the container. Multiple second physical commands, converted from the concurrent first scheduling commands, are recorded in the command buffer, and then all commands in the command buffer are merged and added to the physical command queue. Furthermore, to further improve command execution efficiency, this invention sorts the concurrent second physical commands according to the container application's priority and records them sequentially in the command buffer.

[0039] Step 5: If the container does not receive a resource-constrained message, the current synchronization object controls the synchronous execution of the second physical command. That is, after the current application finishes rendering, the swap chain image of the current application is read from the frame buffer and copied from the frame buffer to the video memory buffer group; otherwise, when copying the swap chain image from the frame buffer to the video memory buffer group, the scaling quality of the swap chain image is reduced, and after receiving the system resource recovery message, the swap chain image is restored to the original scaling quality.

[0040] Furthermore, to ensure that the process of saving the swap chain image to the video memory buffer group is completed in the GPU and improve the real-time performance of the operation, this invention uses GPU hardware texture sampling to synchronously complete the resolution conversion of the swap chain image during the copying process, converting the pixel format of the swap chain image into a unified pixel format.

[0041] To further reduce GPU computation and improve thumbnail acquisition efficiency, this invention allocates multiple reference buffers when the container starts up to store the thumbnail of the previous frame of the container application as a reference frame. When the display interface of the container application is static or the changing area is less than a threshold, the GPU computing shader is used to compare the difference between the current frame of the container application and the reference frame to generate a difference mask. The difference mask is then copied to the video memory buffer group, and the reference frame is updated at the same time.

[0042] To reduce overall resource consumption and achieve a balance between performance and quality, this invention dynamically adjusts the scaling quality of the swap chain image when copying it from the frame buffer to the video memory buffer group, based on the priority and type of the container application. Specifically, for high-priority container applications, bilinear or bicubic filtering is used to ensure image quality; for low-priority container applications, nearest-neighbor filtering is used to improve processing speed; and for video container applications, motion-compensated scaling is enabled to reduce blurring of dynamic images.

[0043] Furthermore, to improve system resource utilization, when a resource-constrained message is received, this invention adjusts the thumbnail resolution according to the priority of the container application. Specifically, for high-priority container applications, the resolution is reduced by one level; for low-priority container applications, the resolution is reduced by two levels. In addition, to avoid frequent fluctuations, the original resolution is gradually restored when the load returns to normal.

[0044] When a resource-constrained message is received, the present invention can also adjust the resolution of the thumbnail according to the activity level of the container application.

[0045] To improve the copying efficiency of images in the swap chain, this invention establishes a mapping relationship between the container application and the graphics layer. Specifically, it queries the application's framebuffer information through the underlying graphics interface to establish a mapping relationship including window handle, process ID, framebuffer handle, layer ID, position coordinates, and size. A hash table structure is used to optimize the query speed of the mapping relationship. Furthermore, the active state of the container application can be dynamically updated based on window events and periodic checks, and the thumbnail capture frequency can be adjusted according to the active state.

[0046] Step 6: The host machine determines the video memory buffer group based on the identifier of the current application and reads the thumbnail of the current application from the video memory buffer group. Example

[0047] This embodiment takes the underlying graphics interface Vulkan or OpenGL ES as an example, and adopts the efficient thumbnail acquisition method based on hardware graphics interface provided by this invention, which achieves a capture frequency of at least 30 times per second across different runtime environments. The specific process includes:

[0048] S1. Container Initialization and Resource Configuration. This section establishes the basic runtime environment for the thumbnail capture process, ensuring the efficiency of subsequent operations through pre-allocated resources and optimized configuration.

[0049] S1.1 Initialize the underlying graphics interface context: When creating a Vulkan instance or OpenGL ES context, specifically specify support for multi-application framebuffer access. For Vulkan, extensions such as VK_EXT_acquire_drm_display need to be enabled to obtain system-level framebuffer access permissions; for OpenGL ES, the EGL_EXT_platform_base extension needs to be configured to support cross-platform window system integration.

[0050] S1.2 Create a dedicated command queue: Create a high-priority dedicated command queue on the graphics device. Its priority is higher than the rendering queue of ordinary applications. For example, in Vulkan, set the queue priority to 1.0, and execute commands asynchronously. This queue is specifically used to handle thumbnail capture-related commands, avoiding interference with the application's normal rendering tasks.

[0051] Existing technologies often share command queues with applications, causing thumbnail capture to be blocked by the application's rendering tasks. In this embodiment, a dedicated queue ensures that capture commands are executed first, maintaining a capture frequency of 30 FPS even under high system load.

[0052] S1.3, Pre-allocate multi-level resolution video memory buffer.

[0053] Existing methods typically allocate a single resolution video memory buffer for each application and scale it as needed. This embodiment pre-allocates multiple resolutions, allowing users to directly select the appropriate resolution based on requirements, avoiding real-time scaling operations. For example, the task switcher can directly use a 256×144 resolution, while the preview window can use a 512×288 resolution.

[0054] Specifically, based on the maximum number of concurrent applications supported by the system, such as 64, a corresponding number of buffer groups are pre-allocated in the GPU memory. Each buffer group contains 3 to 5 images of different resolutions, such as 128×72, 256×144, 512×288, etc. All buffers use a linear layout to optimize access speed. For Vulkan or OpenGL ES, GL_TEXTURE_2D is set as follows:

[0055] VK_IMAGE_USAGE_TRANSFER_DST_BIT|VK_IMAGE_USAGE_SAMPLED_BIT

[0056] To support target replication and sampling access.

[0057] Subsequently, by using shared memory or virtual PCI devices, the host machine and the container can share the video memory buffer, thereby enabling the host operating system to obtain thumbnails through the video memory buffer.

[0058] S1.3, Construct a synchronization object pool.

[0059] Key code examples (Vulkan):

[0060] / / Create a high-priority command queue

[0061] VkDeviceQueueCreateInfoqueueInfo{};

[0062] queueInfo.sType=VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO;

[0063] queueInfo.queueFamilyIndex=transferQueueFamily;

[0064] queueInfo.queueCount = 1;

[0065] float priority = 1.0f; / / Highest priority

[0066] queueInfo.pQueuePriorities=&priority;

[0067] / / Pre-allocate multi-level resolution video memory buffer

[0068] for(uint32_ti=0;i <MAX_APPS;i++){

[0069] for(auto&res:resolutions){ / / resolutions contains various resolution levels

[0070] createImage(res.width,res.height,

[0071] VK_FORMAT_R8G8B8A8_UNORM,

[0072] VK_IMAGE_USAGE_TRANSFER_DST_BIT|VK_IMAGE_USAGE_SAMPLED_BIT,

[0073] &appThumbnails[i].levels[res.level]);

[0074] }

[0075] }

[0076] S2, Application Layer Monitoring and Tracking: By monitoring running applications in real time, it accurately tracks their window status and graphics layer information, providing a data foundation for efficient capture.

[0077] S2.1 Register system window event callbacks. Through the window management interfaces provided by the operating system, such as Windows' WndProc and Linux's XEvent, register callback functions to receive real-time events such as the creation, destruction, movement, resizing, minimization, and maximization of application windows. For modern compositors like Wayland, window information is obtained through the zwlr_foreign_toplevel_manager_v1 protocol.

[0078] S2.2 Establish the mapping relationship between the application and the graphics layer: For each window, query its corresponding framebuffer or swapchain information through the underlying graphics interface. In Vulkan, the application's swapchain image is obtained through vkGetSwapchainImagesKHR; in OpenGL ES, the window and framebuffer are associated through eglQuerySurface and glBindFramebuffer. A mapping table containing window handles, process IDs, framebuffer handles, layer IDs, position coordinates, and sizes is established, and a hash table structure is used to optimize query speed.

[0079] By directly mapping window information to graphics layer handles, the overhead of indirectly obtaining image data through operating system APIs, as in traditional methods, is avoided, reducing capture preparation time by more than 60%.

[0080] S2.3 Maintain an active application status list: Based on window events and periodic checks, the active status of applications is dynamically updated every 100ms to make resource allocation more reasonable. For example, for applications playing videos in the background, which are characterized by low interaction but high rendering frequency, a medium capture frequency is maintained; for static windows, such as document editors, the capture frequency is reduced.

[0081] The activity level is represented by activity level, which is assessed based on user input, such as recent clicks, keyboard focus, and CPU or GPU usage.

[0082] By monitoring the rendering frequency of the switching chain, the actual frame rate of the application is recorded, such as 60FPS or 30FPS, as the rendering frequency.

[0083] S2.4 Dynamically Adjust the Capture Target Set: Based on application priority, the list of applications requiring thumbnail capture is dynamically adjusted. Unlike fixed-frequency capture methods that adjust the capture frequency in real-time based on application status, this reduces unnecessary resource consumption while ensuring user experience. Tests show that this strategy can reduce GPU resource usage by 30-50%.

[0084] S3, an efficient frame buffer capture mechanism, enables efficient data transfer from the application frame buffer to the thumbnail buffer. The core of this mechanism is to complete all copying operations within the GPU, avoiding data flow to the CPU.

[0085] S3.1 Obtaining Read-Only Access to the Frame Buffer: This involves acquiring read-only access to the application's frame buffer through a special mechanism in the underlying graphics interface. In Vulkan, this is achieved through the VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL layout transformation, converting the application's swap-chain image into a transportable source. In OpenGL ES, the application's frame buffer is bound using glBindFramebuffer, and an extended form of glReadPixels, such as GL_ARB_copy_image, is used for internal GPU reading. This process does not involve data transfer; it only modifies the image's access status.

[0086] S3.2 Direct Copying Within the GPU: An image copy command is constructed to directly copy a specified area of ​​the application framebuffer to a pre-allocated thumbnail buffer. In Vulkan, the `vkCmdCopyImage` command is used; in OpenGL ES, the `glCopyImageSubData` function is used. Both operations are completed within the GPU, eliminating the need for data to pass through CPU memory, resulting in a 5-10 times improvement in transmission efficiency compared to traditional methods. Existing methods require transferring image data from the GPU to CPU memory, such as the default behavior of `glReadPixels`. This embodiment achieves direct data migration within the GPU through the image copy command of the underlying interface, completely avoiding PCIe bus transmission and reducing capture latency from tens of milliseconds to less than one millisecond.

[0087] S3.3 Command Batch Processing and Merging: Capture commands from multiple applications are recorded sequentially into the same command buffer and submitted for batch execution. For example, copy commands from 10 applications can be merged into one batch, reducing the number of command submissions. Simultaneously, commands are sorted, prioritizing capture commands from high-priority applications to ensure more timely thumbnail updates for important applications. Existing methods submit capture commands separately for each application, with each submission incurring fixed overhead (such as context switching). This application merges multiple commands, reducing the command overhead per application by over 80%, with significant advantages, especially in multi-application scenarios.

[0088] Key code examples (Vulkan):

[0089] / / Start recording command buffer

[0090] vkBeginCommandBuffer(cmdBuffer,&beginInfo);

[0091] / / Add copy command to each application

[0092] for(auto&app:captureList){

[0093] / / Convert the application image to the source layout

[0094] transitionImageLayout(cmdBuffer,app.swapchainImage,

[0095] VK_IMAGE_LAYOUT_PRESENT_SRC_KHR,

[0096] VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL);

[0097] / / Define the region to be copied (visible only)

[0098] VkImageCopycopyRegion{};

[0099] copyRegion.srcSubresource.aspectMask=VK_IMAGE_ASPECT_COLOR_BIT;

[0100] copyRegion.srcOffset={app.visibleRect.x,app.visibleRect.y,0};

[0101] copyRegion.srcExtent={app.visibleRect.width,app.visibleRect.height,1};

[0102] copyRegion.dstSubresource.aspectMask=VK_IMAGE_ASPECT_COLOR_BIT;

[0103] copyRegion.dstOffset={0,0,0};

[0104] copyRegion.dstExtent={app.thumbWidth,app.thumbHeight,1};

[0105] / / Execute the copy command

[0106] vkCmdCopyImage(cmdBuffer,

[0107] app.swapchainImage,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,

[0108] app.thumbnailImage,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,

[0109] 1,©Region);

[0110] }

[0111] / / Submit batch command

[0112] vkEndCommandBuffer(cmdBuffer);

[0113] submitCommandBuffer(cmdBuffer); / / Handles multiple applications in a single submission

[0114] S4. Real-time resolution conversion and format optimization. Thumbnail resolution adjustment and format conversion are performed directly on the GPU, avoiding CPU involvement in image processing and ensuring real-time performance.

[0115] S4.1 Hardware-Accelerated Downsampling: Utilizing the GPU's hardware texture sampling capabilities, resolution conversion is performed simultaneously during the copying process. In Vulkan, by setting the `dstExtent` parameter of `VkImageCopy` to the target thumbnail size, the GPU automatically performs downsampling. For higher-quality scaling, a dedicated image sampler can be used, configured with an appropriate downsampling filter, such as `VK_FILTER_LINEAR`. In OpenGL ES, scaling is achieved through differences in the size parameters of `glCopyImageSubData`, or high-quality scaling can be performed using fragment shaders.

[0116] S4.2 Unified Format Conversion: Various pixel formats used by different applications, such as RGBA8, BGRA8, and RGB10_A2, are uniformly converted to a standard format, such as RGBA8, for easier subsequent use. In Vulkan, this is achieved through the automatic format conversion function of vkCmdCopyImage; for incompatible formats, a ComputeShader is used for format conversion, fully utilizing the parallel computing capabilities of the GPU.

[0117] Existing methods typically copy the entire image to CPU memory first, then use the CPU for scaling and format conversion. This invention performs the copying, scaling, and format conversion all at once on the GPU in the first two steps, reducing processing steps and improving efficiency. Tests show that this method is 10-20 times faster than the CPU method in image processing.

[0118] S4.3 Incremental Update Strategy: For static or minimally changing application screens, only the changed areas are updated, not the entire thumbnail. This is achieved through the following mechanism:

[0119] Maintain thumbnail data of the previous frame for each application, stored in a dedicated reference buffer;

[0120] The GPU compute shader is used to compare the differences between the current frame and the reference frame to generate a difference mask;

[0121] Only copy the difference region to the target buffer, and update the reference buffer.

[0122] By using GPU-accelerated difference detection to process only the changed areas, applications primarily focused on static content, such as document editing and web browsing, can significantly reduce GPU computation. In practical applications, this strategy reduces average GPU utilization by more than 40%.

[0123] S4.4 Differentiated quality control: dynamically adjust scaling quality based on application priority and type.

[0124] For the currently active window, use bilinear or bicubic filtering; for background applications, use nearest-neighbor filtering; for video container applications, enable motion-compensated scaling.

[0125] Unlike a fixed-quality processing approach, this embodiment dynamically adjusts the processing quality based on the importance of the application. While ensuring the quality of the application screen that users care about, it reduces the overall resource consumption and achieves a balance between performance and quality.

[0126] S5, multi-application capture scheduling and synchronization, enables efficient scheduling and precise synchronization of multiple application capture tasks, ensuring that the capture frequency requirement of 30 times per second is met, while avoiding screen tearing.

[0127] S5.1 Time-Slice Scheduler. A time-slice-based scheduling algorithm is designed, dividing one second into 30 time slices, each approximately 33ms. Within each time slice, a thumbnail update is performed for all applications requiring capture. The scheduler maintains a priority queue; high-priority applications, such as the currently focused window, receive processing time first. When the number of applications is large, low-priority applications are automatically allocated to different time slices, ensuring that the capture frequency of high-priority applications is not less than 30 FPS.

[0128] Existing methods employ a simple polling approach, which struggles to guarantee a stable frequency of 30 FPS. In this embodiment, the time-slice scheduler strictly controls the processing time of each cycle, ensuring a stable capture frequency even when the number of applications changes through priority sorting and load balancing.

[0129] S5.2, Precise Synchronization Mechanism: Utilizing the synchronization primitives of the underlying graphics interface, precise synchronization between capture operations and application rendering is achieved. In Vulkan, VkSemaphore is used to wait for the VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT phase of the application exchange chain to complete, ensuring that a complete rendered frame is captured. VkFence is used to track the execution status of capture commands, avoiding duplicate submissions or resource conflicts. For OpenGL ES, EGLSync objects are used to synchronize capture operations and application rendering.

[0130] S6. Efficient access to thumbnails in video memory: Provides an efficient way to access thumbnail data in GPU video memory, allowing other modules to directly use the data in video memory and avoiding unnecessary data transfer.

[0131] S6.1 Direct Access Mechanism within Video Memory: The thumbnail buffer is wrapped as a standard texture object, such as Vulkan's VkImageView and VkSampler, or OpenGL ES's GL_TEXTURE_2D. Other GPU tasks (such as UI rendering) can directly access this data through texture sampling without transferring the data to CPU memory. For example, the Task Manager's UI rendering can directly sample the thumbnail textures in video memory and draw them onto the interface.

[0132] Existing methods require thumbnail data to be transferred to CPU memory before it can be used by UI and other modules. This invention allows GPU tasks to directly access texture data in video memory, completely avoiding data transfer and improving UI rendering performance by 3 to 5 times.

[0133] S6.2 Indexed Query Interface: Implements an indexed query mechanism based on application ID or window handle. It establishes a mapping from application identifiers to the video memory buffer using a hash table, achieving a time complexity of O(1) for the query operation. The following interface functions are provided:

[0134] getThumbnailTexture(appId, level): Retrieves the texture object for the specified application and resolution level;

[0135] getThumbnailSize(appId,level): Gets the width and height of the thumbnail;

[0136] isThumbnailUpdated(appId): Checks if the thumbnail has been updated since the last query.

[0137] Through an efficient indexing mechanism and on-demand mapping strategy, only necessary data is transferred to the CPU, reducing unnecessary data transfer by more than 90%. For example, Task Manager only queries textures when displaying data and only maps data to the CPU when the user takes a screenshot.

[0138] S6.3 On-Demand CPU Mapping: Thumbnail data in video memory is transferred to CPU memory only when necessary, such as when saving to a file or performing CPU processing. This is achieved through the following optimization steps:

[0139] Use non-blocking mapping operations, such as Vulkan's vkMapMemory in conjunction with VK_MEMORY_PROPERTY_HOST_COHERENT_BIT; support partial region mapping, transferring only the required regions; automatically release the mapping after the transfer is complete to avoid occupying GPU resources for a long time.

[0140] S7 features dynamic performance monitoring and adaptive adjustment, which monitors system performance status in real time and dynamically adjusts the capture strategy according to the load to ensure optimal performance under different hardware conditions and system loads.

[0141] S7.1 Multi-dimensional Performance Monitoring: The following performance metrics are collected in real-time through underlying interfaces and system APIs. Existing methods use fixed capture parameters, making it difficult to adapt to different system loads. This invention combines multi-dimensional data such as GPU load and application activity to dynamically adjust strategies, ensuring system smoothness under high load and improving user experience under low load.

[0142] GPU load is obtained through Vulkan's VK_EXT_device_performance_query or OpenGL ES extensions, including GPU utilization and core frequency; capture performance is obtained by recording the time taken for each capture, actual frame rate, and command queue latency; system load includes CPU utilization, memory usage, and PCIe bandwidth usage.

[0143] The monitoring frequency can be once every 100ms to ensure timely detection of performance changes.

[0144] S7.2 Adaptive Resolution Adjustment: When the system load exceeds a threshold, such as GPU utilization > 85%, the thumbnail resolution is automatically reduced to decrease resource consumption. Specifically, this includes:

[0145] For high-priority applications, reduce the resolution from 512×288 to 256×144; for low-priority applications, reduce the resolution by two levels or pause capture; gradually restore the original resolution as the load returns to normal.

[0146] S7.3 Activity-based resource allocation, which dynamically adjusts resource allocation based on application activity:

[0147] The currently focused window is guaranteed 30 FPS with high-quality scaling; recently used windows are maintained at 15 to 20 FPS to balance quality and performance; and windows that have not been used for a long time are reduced to 5 to 10 FPS with fast scaling.

[0148] Activity assessment is based on factors such as user input frequency and window switching frequency, and is updated at a fixed interval, such as every 2 seconds.

[0149] In addition, a more accurate way to determine activity is to intercept the application's graphics APIs that perform on-screen operations, such as OpenGL ES's eglSwapBuffers or Vulkan's vkQueuePresentKHR, and monitor the execution frequency of these APIs to determine the frequency of thumbnail updates for the application.

[0150] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hardware graphics interface based cross-runtime environment thumbnail efficient acquisition method, characterized in that, Specifically comprising the following steps: Step 1, starting a time slice scheduler and a performance monitor in a host computer, and sending a system resource limited message to a container when the system load is higher than a threshold value; Starting the container, creating a physical command queue containing thumbnail capturing physical commands, pre-allocating a shared video memory buffer group, and constructing a synchronization object pool containing synchronization objects; the thumbnail is a runtime thumbnail of all applications in different operating systems simultaneously displayed when the host operating system application and the application in the other operating system are switched in the application running in the other operating system on the operating system through the container technology or the virtualization technology; the length of the time slice is the shortest total time required for all applications to be captured to update the thumbnail once; Step 2, the host computer determines the first priority of the container, and constructs a logical command for capturing the thumbnail of the container application, i.e. a first scheduling command; The time slice scheduler adds the first scheduling command with an earlier generation time and a higher first priority to a time slice with an idle time slot and an earlier execution time; According to the scheduling, the first scheduling command in the time slice is sent to the corresponding container; Step 3, the container obtains the second priority of the container application according to the first scheduling command, converts the first scheduling command into a second physical command, multiplexes the idle synchronization object and the idle video memory buffer group, and adds an application identifier in the video memory buffer group; the second physical command is added to the physical command queue according to the generation time and the second priority; If the resource limited message is received, the physical command queue is executed in sequence, the synchronization object controls the synchronous execution of the second physical command, and the exchange chain image is copied from the frame buffer to the video memory buffer group with reduced scaling quality; otherwise, it is copied with the original scaling quality; Step 4, the host computer reads the thumbnail to be captured from the video memory buffer group according to the application identifier.

2. The cross-runtime environment thumbnail efficient acquisition method of claim 1, wherein, When the system load returns to less than the threshold value, a system resource recovery message is sent to the container, and the exchange chain image is restored to the original scaling quality after the container receives the message.

3. The cross-runtime environment thumbnail efficient acquisition method of claim 1, wherein, The video memory buffer group contains a plurality of buffers supporting different resolutions.

4. The cross-runtime environment thumbnail efficient acquisition method of claim 1, wherein, A command buffer is established in the container, and when the container receives a plurality of concurrent first scheduling commands, a plurality of second physical commands converted from the concurrent first scheduling commands are recorded in the command buffer, and all commands in the command buffer are merged and added to the physical command queue.

5. The cross-runtime environment thumbnail efficient acquisition method of claim 4, wherein, After the concurrent second physical commands are sorted according to the priority of the container application, they are recorded in the command buffer in sequence.

6. The cross-runtime environment thumbnail efficient acquisition method of claim 1, wherein, When the exchange chain image is copied from the frame buffer to the video memory buffer group, the resolution of the exchange chain image is converted to a uniform pixel format by using the hardware texture sampling synchronization of the GPU.

7. The cross-runtime environment thumbnail efficient acquisition method of claim 1, wherein, A plurality of reference buffers are allocated when the container is started, which are used to save the thumbnail of the previous frame of the container application as a reference frame; when the display interface of the container application is static or the change area is less than a threshold value, a difference mask is generated by comparing the difference between the current frame and the reference frame of the container application, and the difference mask is copied to the video memory buffer group, and the reference frame is updated.

8. The cross-runtime environment thumbnail efficient acquisition method of claim 1, wherein, When copying the swap chain image from the frame buffer to the video memory buffer group, the scaling quality of the swap chain image is dynamically adjusted, bilinear or bicubic filtering is used for the container application with high priority, nearest-neighbor filtering is used for the container application with low priority, and motion compensation scaling is enabled for the video container application.

9. The cross-runtime environment thumbnail efficient retrieval method of claim 1, wherein, A mapping relationship between the container application and the graphics layer is established, frame buffer information of the application is queried, and a mapping relationship containing a window handle, a process ID, a frame buffer handle, a layer ID, position coordinates and size is established. A hash table structure is used to optimize the query of the mapping relationship.

Images