System and method for optimizing graphical display processing of user interface software

Abstract

An integrated compositing block architecture is configured to generate a single graphics plane from graphics components of different graphics processes. The integrated compositing block architecture intercepts function calls from a set of frame buffers to a hardware abstraction layer. The function calls from each frame buffer can be associated with a graphics component to be rendered. The integrated compositing block architecture composites the graphics components of one or more frame buffers of the set of frame buffers into a single graphics plane. The integrated compositing block architecture transmits the single graphics plane to the hardware abstraction layer for rendering in a display of a display device.

Description

[0001] Cross-references to related applications

[0002] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 539,633, filed September 21, 2023, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure relates generally to graphics display processing, and more specifically to a graphics management subsystem configured to optimize graphics display processing by coordinating rendering and display functions into a single management context. Background Technology

[0004] Rendering user interfaces on display devices that support a mix of native and third-party applications—such as TVs, monitors, and mobile devices like smartphones or tablets—can be challenging. The combination of text, graphics, and video elements presents challenges to supporting graphics systems, potentially overloading storage and computing resources. Current methods for rendering complex mixtures of text, graphics, and video elements often result in noticeable delays in the overall user experience.

[0005] For example, a current user application might expect its own graphical output to take precedence over other applications and native processes by allowing the user application to execute processes that typically display rich graphics to the user interacting with that application. Simultaneously, the native user interface of the display device or set-top box might also expect to take precedence over applications executing through the display device or set-top box, including the user application, to ensure the timely display of all information and related graphics. When the graphical output of the user application and the graphical output of the display device or set-top box's native processes compete to present graphical output simultaneously, the current approach causes visual delays, resulting in the user application, the display device or set-top box's native processes, or both needing to wait while the display device or set-top box processes their respective graphical outputs in a predefined order.

[0006] The systems and methods disclosed herein significantly improve the efficiency of managing complex interactive user interface experiences, thereby enhancing timeliness. Summary of the Invention

[0007] This document describes methods and systems for optimizing graphics display processing in a display device. The method may include intercepting function calls from a set of framebuffers to a hardware abstraction layer (HAL), wherein the function call from each of the set of framebuffers is associated with a graphics component to be displayed in a graphical user interface; compositing the graphics components of one or more framebuffers into a single graphics plane, wherein the function calls of the one or more framebuffers are intercepted within a common time interval; and transmitting the single graphics plane to the hardware abstraction layer for rendering on the display of the display device.

[0008] The system described herein is used to optimize graphics display processing in a display device. The system may include one or more processors and a non-transitory computer-readable medium for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods described above.

[0009] The non-transitory computer-readable medium described herein may store instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods described above.

[0010] The illustrative examples cited are not intended to limit or define this disclosure, but rather to aid in understanding it. Additional implementation methods are discussed and further described in the detailed description. Attached Figure Description

[0011] The features, implementation methods and advantages of this disclosure can be better understood by reading the following detailed description in conjunction with the accompanying drawings.

[0012] Figure 1 A block diagram of an example user interface graphics processing block architecture according to aspects of this disclosure is shown.

[0013] Figure 2 A block diagram of an example graphical management subsystem including an integrated synthesis module architecture, according to aspects of this disclosure, is shown.

[0014] Figure 3 A block diagram of an exemplary graphics compositing system using the Wayland integrator instead of DirectFB, according to aspects of this disclosure, is shown.

[0015] Figure 4A An example graphical representation of a time allocation trace generated using the current architecture according to aspects of this disclosure is shown.

[0016] Figure 4BAn example graphical representation of a time allocation trace generated using a graphical management subsystem according to aspects of this disclosure is shown.

[0017] Figure 5 A flowchart illustrating an example process for detecting the boundary of a linear program according to aspects of this disclosure is shown.

[0018] Figure 6 An example computing device architecture is shown that enables the implementation of various technologies described herein, according to aspects of this disclosure. Detailed Implementation

[0019] This document discloses systems and methods for optimizing graphics display processing in display devices and / or set-top boxes. In some examples, a graphics management subsystem can be provided to coordinate critical graphics rendering and display functions into a single managed pipeline. The graphics management subsystem can intercept function calls from processes with graphics output, thereby enabling it to optimize graphics display processing without any additional modifications to the processes of the display device with graphics input. The graphics management subsystem can utilize a dedicated compositor for the first graphics output protocol (GOP) layer, which reduces memory usage and allows the graphics management subsystem to render multiple graphical user interfaces (and / or windows) (referred to herein as contexts) in a single graphics display process. The graphics management subsystem can allow for more efficient handling of complex user interfaces in display devices and set-top boxes, where third-party applications can generate graphics for screen display concurrently with the native display process of the display device or set-top box. Furthermore, the graphics management subsystem can achieve consistent high-definition display in display devices and set-top boxes using central processing units (CPUs) of arbitrary performance (e.g., high-performance CPUs, medium-performance CPUs, or low-performance CPUs, etc.).

[0020] Currently, the graphics processor of a display device or set-top box may generate a new buffer each time an application and the native processes of the display device or set-top box attempt to output graphics (e.g., a new graphical user interface, an updated graphical user interface, etc.). The graphics processor may then render (also referred to as compositing) all the graphical user interfaces (and / or windows) to be displayed each time a new buffer is generated. Rendering all windows each time a new buffer is generated can increase memory usage, reduce frame rates when multiple application animations are running, cause significant memory bandwidth consumption, and increase rendering latency.

[0021] For example, Figure 1A block diagram of an exemplary graphics processing block architecture according to aspects of this disclosure is shown. The display device may execute multiple processes, each capable of outputting to the display device. For example, processes 104, 108, and 112 shown are three example processes that can simultaneously render graphics output. Each of processes 104, 108, and 112 may contain a GPU context (e.g., GPU context 116 in process 104) that can output graphics information to a screen buffer (e.g., screen buffer 120 in process 104). The screen buffer can pass graphics information to a direct framebuffer process (e.g., referred to as DirectFB, such as DirectFB 124 in process 104). The DirectFB process can be instantiated from a DirectFB library, which may be an open-source component of the display device's operating system (e.g., a Linux-based operating system, etc.). The DirectFB library provides graphics rendering services to applications with graphics output, including native applications, operating system applications or processes, third-party applications, etc.

[0022] DirectFB (short for Direct Framebuffer, for example) is a library that provides a hardware-accelerated graphics environment for embedded systems and other low-level platforms. The DirectFB library allows applications to directly access and manipulate graphics hardware without using a traditional windowing system, which is a fast and efficient way to render graphics on resource-constrained systems.

[0023] DirectFB windows can be created and managed by the DirectFB window manager, which handles window creation, resizing, and positioning. DirectFB windows can be created in various sizes and shapes and can be assigned different attributes such as transparency, borders, and decorations. In summary, DirectFB windows provide a flexible and efficient way to create graphical user interfaces in DirectFB display systems, and are an important component of many embedded and low-level systems.

[0024] DirectFB windows can support hardware acceleration features, such as, but not limited to: bit-block transfer (e.g., "block image transfer" or bit-block transfer, which is a highly parallel memory transfer and logic operation unit that supports multiple operation modes, including copying memory blocks, filling memory blocks with polygon fills, line drawing, etc.), alpha blending (e.g., merging graphics layers using an alpha (transparency) channel that determines the opacity of each pixel in its associated framebuffer layer), drawing primitives (achieving smoother graphics performance on supported devices), multiple layers (e.g., associating graphics components with layers in the scene to improve the visual effects and corresponding processing of rendering media), window systems (e.g., DirectFB can include a built-in window system that supports basic window handling, input events, and event handling), input devices (e.g., DirectFB can support various input devices, such as keyboards, mice, and touchscreens that allow users to interact with graphics applications), image and font support (e.g., DirectFB can support loading and rendering common image formats such as JPEG, PNG, and GI, and can include built-in support for specific font families), and combinations thereof.

[0025] DirectFB can be used in systems that do not require a desktop environment or windowing system (such as X11 or Wayland) or cannot use such an environment due to resource constraints. DirectFB can be used to create lightweight graphics applications, self-service terminal systems, and user interfaces on Linux-based embedded devices with limited processing power and memory. For 3D graphics or more complex 2D graphics needs, other APIs such as OpenGL, Vulkan, etc., can be used.

[0026] DirectFB can include one or more DirectFB processes to meet the need for multi-process processing. For example, if process 104 is outputting graphics intended to be overlaid on the graphics of process 108, the corresponding DirectFB process may need to recognize other executing DirectFB processes. Screen buffer 120 can call DirectFB 128 of process 108 and DirectFB 132 of process 112. Similarly, screen buffer 136 of process 108 can call DirectFB 124 and DirectFB 132 of process 112, while screen buffer 140 of process 112 can call DirectFB 124 and DirectFB 128 of process 112.

[0027] Each DirectFB process can make calls to the Graphics Processing Unit (GPU) rendering process, which can be accessed through the Hardware Adaptation Layer (HAL) 148. HAL 148 is an abstraction layer configured to enable hardware access to the executing application. Calls to the GPU rendering process may instantiate one or more temporary memory buffers 144. Since each call to the GPU rendering process may instantiate one or more temporary memory buffers when the corresponding DirectFB is ready to send a job to the HAL for rendering and display, even a small number of graphics processes can instantiate a large number of temporary memory buffers. This can generate significant memory and processor load, resulting in a significant decrease in the responsiveness of the user interface or other graphical content.

[0028] The display device may include additional processes besides processes 104, 108, and 112. In some examples, these processes may represent different applications of the smart TV. For example, process 104 may represent the Conjure process, process 108 may represent a system-level user interface (e.g., menus that control the operation of the smart TV and allow selection of various media streams), and process 112 may represent digital TV services (e.g., broadcast media, streaming media, etc.). Because process 108 (e.g., the system-level user interface) runs in parallel with other processes on the display device, process 108 may compete for process resources and time when rendering new or modified graphics output. In some cases, competition between processes may cause visual artifacts and rendering delays. For example, the rendering of the smart TV's user interface may be delayed when a user interacts with the smart TV while the smart TV is rendering the graphics output of process 104 and / or process 112. Furthermore, the response to user interaction with the user interface may be delayed, thus hindering the effective use of the smart TV.

[0029] The systems and methods described herein include an integrated composition block architecture configured to directly write to a frame buffer located at / dev / fb0, which may correspond to the first frame buffer device in the operating system of the display device (e.g., a Linux-based operating system, etc.). The frame buffer can be an intermediate stage between the graphics hardware and the final output on the display (e.g., a television, monitor, etc.). The frame buffer may correspond to a region in random access memory (or other memory containing a bitmap that drives the display). This bitmap may represent each pixel of the display. The RAM region can be written to or read from (like any other accessible memory). The operating system can reference the device as a file, allowing access to the frame buffer through a special file located in the / dev directory. The name "fb0" may refer to the first (or main) frame buffer device available on the system. Multiple frame buffer devices may be provided, numbered sequentially (e.g., / dev / fb1, / dev / fb2). Software processes can directly access and manipulate the frame buffer by reading and writing pixel data to / dev / fb0. This approach provides a low-level method for managing graphics output without the need for more complex graphics libraries such as OpenGL.

[0030] The / dev directory can store operating system libraries for software interfaces, including frame buffers (such as fb0, fb1, etc.), audio libraries (such as drivers), network libraries (such as Ethernet drivers), and hardware libraries (such as Universal Serial Bus drivers).

[0031] In some examples, the integrated composite block architecture can write directly to / dev / fb0 without going through the Direct Framebuffer (DirectFB). DirectFB is a software library that provides graphics acceleration, input device handling, and an abstraction layer, as well as an integrated windowing system that supports translucent windows and multiple display layers on top of the Linux framebuffer, without any kernel modifications.

[0032] Figure 2 A block diagram of an exemplary graphics management subsystem including an integrated composition block architecture according to aspects of this disclosure is shown. The integrated composition block 204 is configured to intercept calls executed by the DirectFB process and integrate graphics data into a single composition process. This single graphics output can then be output to the HAL 148 via a temporary memory buffer 144. Since only a single call to the HAL 148 is required for one or more processes providing graphics output, fewer temporary memory buffers can be instantiated, significantly reducing the load and latency on the processing hardware when providing graphics output, thereby improving the responsiveness of graphics applications running on display devices such as user interfaces.

[0033] The integrated composition block 204 may include DirectFB interceptor process 208 and DirectFB interceptor process 212. In some cases, the integrated composition block 204 may include any number of DirectFB interceptor processes. In some cases, a single DirectFB interceptor process may be configured to intercept calls from any number of DirectFB processes. DirectFB interceptor process 208 and DirectFB interceptor process 212 may be configured to intercept calls from both DirectFB processes and GPU rendering processes.

[0034] In some cases, the DirectFB library can be modified to invoke the DirectFB interceptor process. In other cases, the DirectFB library may not be modifiable. In these cases, the integration composition block 204 can capture the call before it causes the GOP buffer to be instantiated.

[0035] Integration composition block 204 may include GPU context 220, which is configured to internalize the graphics output of system-level processes (e.g., user interfaces associated with the operating system, native processes of the display device, etc.) and / or first-party applications or processes (e.g., native applications and / or processes of the display device, or applications and / or processes specified as native to the display device via user input, etc.). Therefore, processes and / or first-party applications or processes providing system-level graphics output may not execute as external processes (e.g., processes 104, 108, 112, etc.). Integration composition block 204 can perform integration using the same context used for rendering system-level graphics output and / or first-party applications or processes, without providing an additional context. By internalizing system-level processes and / or first-party applications or processes into the integration composition block 204, the graphics output of system-level processes or first-party applications or processes can be facilitated (and / or prioritized) even when other processes are requesting composition. This can optimize resource utilization for rendering graphics output and reduce latency in updating system-level processes or first-party applications or processes (e.g., when a user interacts with a system-level user interface, etc.).

[0036] The compositing block 204 can generate a single context (e.g., compositing 216) to handle calls to the GPU rendering process intercepted from multiple DirectFB processes. Compositing 216 can process the graphics data contained in the intercepted calls by compositing multiple graphics data layers into a single graphics plane 220. For example, the single graphics plane 220 can be defined based on the graphics data intercepted from the DirectFB processes and on the determination of what to render, the layer identifier of each part of the graphics data, the identifier of potentially occluded parts of the graphics data (e.g., potentially invisible parts of a window when another window is rendered above it), etc. The compositing block 204 can execute a call to the HAL to render the single graphics plane 220. The single graphics plane 220 can be passed to a Graphics Output Protocol (GOP) buffer (e.g., temporary memory buffer 144) to continue through the aforementioned graphics pipeline.

[0037] In some examples, such as when implementing integrated composition block 204 in a Linux-based operating system, a framebuffer (fb) can be used to supplement or replace DirectFB. A framebuffer can correspond to a region in random access memory (or other memory containing a bitmap that drives the display device). This bitmap can represent each pixel of the display. The RAM region can be written to or read from (like any other accessible memory). The operating system can expose the RAM region using the / dev / fb0 reference. Dev corresponds to the location of a special device file, and fb0 represents framebuffer 0. / dev / fb0 can be configured before instantiating integrated composition block 204. Configuring / dev / fb0 can include: selecting a resolution between 4K and 1080p at startup; configuring the framebuffer to use ARM Framebuffer Compression (AFBC); and optionally achieving parallelism by coordinating with / dev / fb0 independently of GPU threads. Implementing integrated composition block 204 can include implementing double buffering, triple buffering, or n-buffering.

[0038] In some examples, the integrated composition block 204 can be operated by directly writing to / dev / fb0, in which case the unified composition process can render the written graphics data without using the DirectFB process. In this example, / dev / fb0 can be configured via an initialization process when the display device is powered on or when the display device's operating system starts. In other examples, the display device may include a system-on-a-chip (SoC) that may include modifications or configurations to DirectFB.

[0039] In either example, the integrated composition block 204 can be configured before compositing graphics data. This configuration may include: selecting a display resolution at startup; determining whether to configure the framebuffer to use AFBC; determining whether to activate parallel processing (e.g., making the integrated composition block 204 independent of threads waiting for the GPU, waiting (synchronizing) / dev / fb0), etc. Activating parallel processing can be achieved by directly controlling / dev / fb0 or by using one or more DirectFB application programming interfaces (APIs).

[0040] Visual artifacts (e.g., screen tearing) can occur when processing buffer data during screen refreshes. The merging composition block 204 can prevent these artifacts by processing buffer data between screen refreshes. For example, the merging composition block 204 can send a vertical sync (VSync) signal to synchronize buffer data processing with screen refreshes (or the time interval between screen refreshes). The graphics process can sleep (e.g., wait) until it receives the VSync signal to process one or more buffers instantiated since the last processing of buffer data. If a new buffer is detected, the reference to the previous buffer can be dropped, and processing can continue with the new buffer. The graphics process can then perform a call to the HAL interface to compose the buffers.

[0041] Windows can be imported into an integrated composition block architecture by using buffers from other applications as GPU textures. In the context of Linux application development, GPU textures may be a dedicated data structure used to store and represent image data on the GPU. Textures can provide visual details that constitute the appearance of 3D objects and scenes in graphics applications. Textures can represent various graphics attributes such as color, transparency, and reflection.

[0042] Textures can be two-dimensional (2D) pixel arrays, where each pixel contains specific information such as color, transparency, or other attributes. Textures can have various formats, such as 1D, 2D, 3D, cubemaps (which represent six 2D textures organized in a cube-like structure), and so on. Each of these formats can serve different purposes and be used in different scenarios. For GPU rendering, textures can be managed through the OpenGL ES API. These APIs provide a set of functions and commands for creating, manipulating, and rendering textures. In some cases, managing textures may involve: 1) loading texture data (e.g., from an image file) into memory; 2) creating and binding texture objects on the GPU; 3) specifying texture parameters, such as filtering, wrapping mode, and mipmapping techniques; 4) uploading texture data to the GPU, which stores it in dedicated memory; 5) binding the texture to specific texture units during rendering; and 6) sampling the texture using a shader program and applying it to 3D objects in the scene.

[0043] Texture binding refers to associating a texture object with a specific texture unit, effectively making it the active texture for subsequent operations. The process of texture binding establishes a connection between the texture object and the GPU's texture processing pipeline, enabling the GPU to access and manipulate texture data. Texture binding identifies the texture object used for subsequent texture-related operations (such as setting texture parameters, rendering using shaders, etc.). For example, in OpenGL ES, a texture can be bound using `glBindTexture:glBindTexture(GL_TEXTURE_2D, textureId)`, where `GL_TEXTURE_2D` represents the texture target (a 2D texture in this example), and `textureId` can refer to a unique identifier of the texture object previously created using `glGenTextures`.

[0044] Once a texture is bound to a texture unit, various operations can be performed on that texture, such as setting its filtering and wrapping modes, uploading texture data, and specifying multi-level mipmapping. During rendering, the shader program can access the active texture by sampling it using texture coordinates, allowing the texture to be applied to 3D objects in the scene. Binding a texture is a stateful operation, meaning the binding remains active until it is explicitly changed by binding to another texture or unbinding the current texture. To unbind a texture, you can call glBindTexture using the texture identifier: 0:glBindTexture(GL_TEXTURE_2D, 0).

[0045] A texture unit can be a component within the GPU used to handle texture processing and mapping during the rendering process. A texture unit manages textures and applies them to 3D geometry in the scene. A GPU can include multiple texture units, enabling the processing of multiple textures in parallel (e.g., almost simultaneously) for more complex rendering effects. A texture unit can be part of the GPU rendering pipeline. When rendering a scene, the GPU processes each fragment (or pixel) of geometry and determines which textures should be applied to it. A texture unit can acquire texture data, perform filtering, and combine the textures with the fragment's color data to generate the final output.

[0046] For example, in OpenGL ES, different textures can be bound to different texture units using the following steps: First, activate the texture unit using `glActiveTexture`, then bind the texture: `glActiveTexture(GL_TEXTURE0 + textureUnitIndex); glBindTexture(GL_TEXTURE_2D, textureId);` where `GL_TEXTURE0` represents the first texture unit, and `textureUnitIndex` represents an integer value used to select the specific texture unit to activate (e.g., 0 for the first unit, 1 for the second unit, and so on). After activating the texture unit, `glBindTexture` can be used to bind the texture to that texture unit, as described above. In implementations including shader routines, textures bound to different texture units can be accessed by declaring uniform sampler variables in the shader code. Uniform sampler variables can be used to obtain texture data based on texture coordinates. For example, in a fragment shader, a sampler can be declared like this: `uniform sampler2D myTexture`. Its uniform value can be matched with the texture unit index used when binding the texture: GLint myTextureLocation = glGetUniformLocation(shaderProgramId, "myTexture"); glUniform1i(myTextureLocation,textureUnitIndex).

[0047] The integrated compositional architecture can use buffers from other applications as GPU textures by acquiring buffers, creating texture objects, copying buffer data, and using the texture in rendering. Buffers can be acquired based on the integrated compositional architecture's permissions. The integrated compositional architecture can acquire access to the buffer directly from the application or process associated with the buffer (e.g., the application or process outputting the graphics), or it can acquire system-level permissions. Once the buffer is acquired, the application can create a texture object that stores image data by, for example, allocating GPU storage, defining the texture format, and setting other attributes (e.g., filtering modes). The application can then copy the data from the acquired buffer into the texture object. In some cases, the application can access specialized APIs such as glTexSubImage2D or glCopyTexSubImage2D to transfer data from the buffer to the texture. Once the data is copied to the texture, that texture can be used as a standard texture in rendering. For example, rendering may include binding the texture to texture units, setting its sampling parameters, and using the texture in vertex and fragment shaders to render the scene.

[0048] The process of using other applications' buffers as GPU textures can vary depending on the platform and API used. For example, an application can use the SurfaceTexture class to obtain a buffer from another application and use it as a texture. Similarly, the DirectX® API can provide mechanisms for sharing textures between applications. Regardless of the platform or API, a single application may not attempt to composite buffers itself after rendering a frame; the cohesive compositional architecture can be notified when buffers are updated, and the cohesive compositional architecture can be configured to lock / unlock and read window buffers.

[0049] The SoC of a display device can implement the following example APIs:

[0050] 1 struct Window {

[0051] 2 / / The PID of the process with this window

[0052] 3 pid_t pid; 4

[0054] 5 / / Handle to the EGLImage buffer

[0055] 6 EGLImage buffers[3]; 7

[0057] 8 / / Index of the 'buffer' that should be displayed at the moment

[0058] 9 int current_buffer; 10

[0060] 11 / / Width and height of the buffer, in pixels

[0061] 12 int buffers_width, buffers_height; 13

[0063] 14 / / Source rectangle to be read from the buffer

[0064] 15 int src_left, src_top, src_width, src_height; 16

[0066] 17 / / Target rectangle to be written to in the buffer

[0067] 18 int dst_left, dst_top, dst_width, dst_height; 19

[0069] 20 / / Window opacity, ranging from 0 to 255

[0070] 21 uint8_t opacity; twenty two

[0072] 23 / / Window's DirectFB layer

[0073] 24 int layer;

[0074] 25}; 26

[0076] 27 / / Register a callback that is invoked when the window has a new buffer that needs to be composited or when the window's opacity changes.

[0077] 29 void set_window_change_callback(void(*window_change_callback)()); 30

[0079] 31 / / Returns a list of windows to be composited and locks their windows.

[0080] 32 / / To prevent the current buffer from being modified.

[0081] 33 void lock_windows(int max_windows, struct Window * windows, int *

[0082] window_count); 34

[0084] 35 / / Allows previously locked windows to be modified again.

[0085] 36 void unlock_windows();

[0086] The example API can be customized to operate using DirectFB. In some implementations, the integrated composition block architecture can serve as a Wayland server. Wayland is a display protocol configured to be more efficient and flexible than older protocols such as the X Window System (X11). By acting as a Wayland server, the integrated composition block architecture can be configured to provide improved graphics performance, more efficient resource utilization, and greater flexibility and compatibility with other software components.

[0087] While the integrated composition block architecture can include access to Layer 0 windows for compositing, it can also include access to both Layer 1 and Layer 0 in its API to enable additional functionality, such as, but not limited to: the ability to create transition animations for Layer 1 applications that utilize GPU temporary compositing; the ability to generate combined Layer 1 and Layer 0 screenshots for test automation; and the ability to inspect Layer 1 windows for GPU-based blank screen detection. By including both layers in the API, the exposed system can provide a more comprehensive graphics rendering solution, addressing a wider range of use cases and scenarios.

[0088] In some examples, an integrated composition block architecture can be implemented without using the DirectFB library. For example, Figure 3A block diagram of an example graphics compositing system utilizing the Wayland integrator instead of DirectFB, according to aspects of this disclosure, is shown. The Wayland integrator operates in a manner similar to a DirectFB process. Wayland client 304 and Wayland client 308 represent applications or processes executing in the operating system capable of rendering graphics. Each Wayland client can instantiate a shared buffer, which can be accessed by both the Wayland client and the Wayland integrator 312. When a Wayland client generates new graphics output, it can instantiate the shared buffer and invoke the Wayland integrator 312. The Wayland integrator 312 can then facilitate the rendering of the graphics output via the kernel or a self-contained library (KMS316).

[0089] KMS 316 can send events to Wayland Composer 316, such as when receiving user input that may modify the graphical output of the Wayland client (e.g., selecting a window from one or more displayed windows, selecting an object or button, executing a new application, etc.). Wayland Composer 316 can then execute a call to the Wayland client that may be affected by the event, causing the Wayland client to update the graphical output. The Wayland client can then generate the modified graphical output based on the event and instantiate a new buffer. The Wayland client can subsequently execute a call to Wayland Composer 316 to recognize the new buffer, allowing Wayland Composer 316 to access the new buffer and facilitate the rendering of the modified graphical output. Alternatively, the Wayland client can reuse a previously instantiated buffer. Reusing a previously instantiated buffer may result in a race condition based on the buffer's previous content and the new content of the buffer if Wayland Composer 316 is still processing a portion of the old content of the buffer.

[0090] To optimize resource utilization and latency during graphics rendering, an integrated composition block architecture can be implemented within the Wayland compositor 316. For example, the integrated composition block architecture can internalize certain graphics processes (e.g., system-level processes, first-party applications or processes, other processes native to the operating system, etc.). These graphics processes can execute as child processes of the Wayland compositor 316 instead of as Wayland clients (e.g., as described in conjunction with the GPU context 220 of the integrated composition block above), thereby eliminating the need to execute certain Wayland clients and eliminating storage and processing resources dedicated to these Wayland clients, as well as the interaction between these Wayland clients and the Wayland compositor 316. Furthermore, because the graphics processes are internalized within the Wayland compositor 316, processing of graphics data for the graphics processes may be faster, and in some cases, processing of graphics data for the graphics processes takes precedence over processing of graphics data for the Wayland clients.

[0091] Figure 4A and Figure 4B Provided Figure 4B The integrated composite block architecture shown is Figure 4A The example graphical representation shown is an example of the current architecture compared to the implemented optimizations. A time allocation trace can be generated when the display device renders SmartCast Home (e.g., an exemplary GPU-intensive user interface used for benchmarking an integrated compositing block architecture), and this time allocation trace is configured to run at 60 frames per second. Flush 404 represents the time spent issuing commands to the GPU, glFinish represents the time spent waiting for the GPU to render, and pan 412 is the time spent waiting for VSynch after the GPU has finished rendering. GuiApp::doUpdate and SceneStackManager are the processes currently being rendered by the compositing architecture; the example graphical representation depicts the time allocation for each process, including the GPU's process.

[0092] Figure 4A An example diagram of the time allocation trace generated using the current architecture according to aspects of this disclosure is shown. The current compositing architecture is configured to render directly to the GOP0 display controller without using a DirectFB compositor. As shown in the exemplary diagram, Flush 404A takes 50 milliseconds because the GPU is blocked, waiting for the compositor to complete compositing of the previous frame (e.g., processing the buffers instantiated each time GuiApp::doUpdate and SceneStackManager generate graphics data). The time interval of Flush 404 limits the frame rate of the display device to 20 frames per second, instead of the 60 frames per second that the display device is configured to render.

[0093] Figure 4B An example diagram is shown illustrating time allocation traces generated using an integrated synthesis block architecture according to aspects of this disclosure. This integrated synthesis block architecture eliminates bottlenecks in the synthesis process by providing a single integrated synthesis block. Figure 4B As shown, Flush 404B is reduced to less than 2 milliseconds. Therefore, the latency of updating the user interface (in this example) is significantly reduced, which can increase the frame rate to the configured frame rate (e.g., 60 frames per second) and improve the responsiveness of the user interface to user interactions.

[0094] Figure 5 A flowchart illustrating an example process for detecting the boundary relative to a linear programming problem according to aspects of this disclosure is shown. At block 504, the graphics subsystem of the display device (e.g., monitor, television, etc.) (e.g., combined with...) Figure 2 The integrated composite block architecture described above can intercept function calls from a set of framebuffers to the Hardware Abstraction Layer (HAL). The HAL is a software layer situated between the operating system and the computer hardware, providing an interface for the operating system to interact with various hardware components, including graphics hardware. Function calls from each framebuffer in this set can be associated with a graphics component to be displayed on the screen. For example, a graphics component can correspond to a window, a user interface element, a user interface or any part thereof, an image, etc. A graphics component can be a new graphics component (e.g., a graphics component not yet displayed) or a modified version of a currently displayed graphics component (e.g., a modified user interface element to be displayed after receiving user input).

[0095] A framebuffer can be a component or process of an application or process executing on the display device. In some examples, a framebuffer can be a native operating system buffer accessed via / dev / fb0, where fb0 represents the first or main framebuffer, and can be replaced with another identifier associated with another framebuffer in the dev directory. In these cases, function calls can be used to write to or read from a framebuffer located at / dev / fb0 to render its contents. In other examples, a framebuffer can be a Direct Framebuffer (DirectFB) process configured to pass graphics data to the HAL's graphics processing pipeline. In still other cases, a framebuffer can be a buffer of a Wayland client configured to invoke a Wayland compositor to render graphics components. In these cases, the graphics subsystem intercepts function calls to the Wayland compositor (instead of the HAL).

[0096] The graphics subsystem can intercept function calls by: monitoring the framebuffer; capturing the function call; and redirecting the function call to the graphics subsystem. Alternatively, the framebuffer can be modified to redirect the function call to the graphics subsystem. For example, the framebuffer could be a DirectFB process instantiated from the DirectFB library. The DirectFB library can be modified so that the DirectFB process calls the graphics subsystem instead of the HAL.

[0097] At block 508, the graphics subsystem may composite graphics components from one or more framebuffers in a set of framebuffers into a single graphics plane. The compositor may be configured to generate a single graphics plane (e.g., a scene to be displayed on a display device's monitor) based on graphics data received in one or more intercepted function calls. In some examples, the compositor may retain graphics data received from the intercepted function calls, which can be used to maintain the graphics displayed when the intercepted function calls contain graphics data affecting parts of the graphics, windows displayed in the graphics, etc. For example, when a window of the user interface is updated, the compositor may receive a new graphics component corresponding to the updated portion of the window (or a new version of the entire window) and composite a new graphics plane containing that new graphics component (or modify a previously rendered graphics plane to incorporate the new graphics component). In some cases, the compositor may be configured to generate a single graphics plane based on two or more intercepted function calls received at intervals between each other.

[0098] In some cases, a graphics subsystem may include one or more subprocesses configured to internalize the graphical output of system-level processes (e.g., user interfaces associated with the operating system, native processes of the display device, etc.) and / or first-party applications or processes (e.g., native applications and / or processes of the display device, or applications and / or processes specified as native to the display device via user input, etc.). In these cases, the processes and / or first-party applications or processes providing system-level graphical output may not execute as external processes. The graphics subsystem can perform compositing using the same context used for rendering system-level graphical output and / or first-party applications or processes, without providing additional context. By internalizing system-level processes and / or first-party applications or processes into the graphics subsystem, the graphical output of system-level processes or first-party applications or processes can be facilitated (and / or prioritized) even when other processes are requesting compositing. This can optimize resource utilization for rendering graphical output and reduce latency in updating system-level processes or first-party applications or processes (e.g., when a user interacts with a system-level user interface, etc.).

[0099] At block 512, the graphics subsystem can transfer a single graphics plane to the hardware abstraction layer for rendering through the display device's graphics pipeline. By intercepting function calls from the framebuffer, the graphics subsystem can prevent the instantiation of additional buffers (e.g., one or more buffers are instantiated with each framebuffer call), which reduces the consumption of processing resources on the display device, increases the speed at which the graphics pipeline renders graphical content, and reduces latency when interacting with displayed graphical elements (e.g., user interface elements).

[0100] Figure 6 An exemplary computing device according to aspects of this disclosure is illustrated. For example, computing device 600 may implement any of the systems or methods described herein. In some cases, computing device 600 may be a component of or contained within a media device. Components of computing device 600 are shown to be in electrical communication with each other using connection 606 (e.g., a bus). The exemplary computing device architecture 600 includes a processor (e.g., CPU, processor, etc.) 604 and connection 606 (e.g., a bus, etc.) configured to couple components of computing device 600 (e.g., but not limited to memory 620, read-only memory (ROM) 618, random access memory (RAM) 616, and / or storage device 608) to processing unit 610.

[0101] Computing device 600 may include a cache 602 of high-speed memory, which is directly connected to, closely adjacent to, or integrated within processor 604. Computing device 600 may copy data from memory 620 and / or storage device 608 to cache 602 for faster access by processor 604. In this way, cache 602 can provide performance improvements and avoid latency while processor 604 waits for data. Alternatively, processor 604 may access data directly from memory 620, ROM 617, RAM 616, and / or storage device 608. Memory 620 may include various types of homogeneous or heterogeneous memory (e.g., but not limited to magnetic memory, optical memory, solid-state memory, etc.).

[0102] Storage device 608 may include one or more non-transitory computer-readable media, such as volatile memory and / or non-volatile memory. The non-transitory computer-readable media may store instructions and / or data accessible to computing device 600. Non-transitory computer-readable media may include, but are not limited to, magnetic tape, hard disk drive (HDD), flash memory, solid-state storage devices, digital multifunction disks, magnetic tape cassettes, optical discs, random access memory (RAM) 625, read-only memory (ROM) 620, and combinations thereof.

[0103] Storage device 608 may store one or more services, such as service 1 610, service 2 612, and service 3 614, which may be executed by processor 604 and / or other electronic hardware. The one or more services include instructions executable by processor 604 for: performing operations such as any techniques, steps, processes, blocks, and / or operations described herein; controlling the operation of means communicating with computing device 600; controlling the operation of processing unit 610 and / or any dedicated processor; and combinations thereof; etc. Processor 604 may be a system-on-a-chip (SoC) including one or more cores or processors, buses, memory, clocks, memory controllers, caches, other processor components, etc. Multi-core processors may be symmetric or asymmetric.

[0104] The computing device 600 may include one or more input devices 622, which may represent any number of input mechanisms, such as microphones, touchscreens for graphical input, keyboards, mice, motion input, voice, media devices, sensors, and combinations thereof. The computing device 600 may include one or more output devices 624 for outputting data to a user. Such output devices 624 may include, but are not limited to, media devices, projectors, televisions, speakers, and combinations thereof. In some cases, a multimodal computing device may allow the user to provide multiple types of input to communicate with the computing device 600. A communication interface 626 may be configured to manage user input and computing device output. The communication interface 626 may also be configured to manage communication with remote devices (e.g., establishing connections, receiving / transmitting communications, etc.) via one or more communication protocols and / or one or more communication media (e.g., wired, wireless, etc.).

[0105] The computing device 600 is not limited to Figure 6 The components shown. The computing device 600 may include other components not shown in the figures and / or the components shown may be omitted.

[0106] The following examples illustrate various aspects of this disclosure. As described below, any reference to a series of examples should be understood as an intermittent reference to each of these examples (e.g., "Examples 1 through 4" should be understood as "Examples 1, 2, 3, or 4").

[0107] Example 1 is a method comprising: intercepting function calls from a set of framebuffers to a hardware abstraction layer, wherein the function calls from each of the set of framebuffers are associated with a graphics component to be displayed in a graphical user interface; compositing the graphics components of one or more of the set of framebuffers into a single graphics plane, wherein the function calls of the one or more framebuffers are intercepted within a time interval; and transmitting the single graphics plane to the hardware abstraction layer for rendering on a display of a display device.

[0108] Example 2 is the method according to any of Examples 1 and 3 to 7, wherein at least one of the set of frame buffers is a direct frame buffer.

[0109] Example 3 is a method according to any of Examples 1 to 2 and 4 to 7, wherein the single graphics plane is transferred to the graphics output protocol buffer of the hardware abstraction layer.

[0110] Example 4 is the method described according to any of Examples 1 to 3 and 5 to 7, wherein the frame buffer is configured to use Arm frame buffer compression.

[0111] Example 5 is a method according to any of Examples 1 to 4 and 6 to 7, wherein intercepting function calls from a set of frame buffers includes unregistering the function calls to the hardware abstraction layer and redirecting the function calls to an integrated composition block.

[0112] Example 6 is the method according to any of Examples 1 to 5 and 7, wherein each frame buffer is associated with a different process executing on the display device, and each frame buffer is configured to output a graphics component.

[0113] Example 7 is the method described according to any of Examples 1 through 6, wherein function calls from a set of framebuffers are intercepted after the windowing process and before graphics rendering.

[0114] Example 8 is a system comprising: one or more processors; and a non-transitory computer-readable medium for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of methods 1 to 7.

[0115] Example 9 is a non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform any of methods 1 to 7.

[0116] Client devices, user devices, computing devices, computer resource provider devices, network devices, and other devices can be computing systems including one or more integrated circuits, input devices, output devices, data storage devices, and / or network interfaces. Integrated circuits may include, for example, one or more processors, volatile memory and / or non-volatile memory, and the components described herein. Input devices may include, for example, keyboards, mice, keypads, touch interfaces, microphones, camera devices, and / or other types of input devices, including but not limited to those described herein. Output devices may include, for example, displays, speakers, haptic feedback systems, printers, and / or other types of output devices, including but not limited to those described herein. Data storage devices (e.g., hard disks or flash memory) allow computing devices to temporarily or permanently store data. Network interfaces (e.g., wireless or wired interfaces) allow computing devices to communicate with a network. Examples of computing devices (e.g., computing device 902) include, but are not limited to, desktop computers, laptop computers, server computers, handheld computers, tablet computers, smartphones, personal digital representatives, digital home representatives, wearable devices, smart devices, and combinations of these devices and / or other such computing devices, as well as machines and equipment incorporating computing devices and / or virtual implementations of computing devices.

[0117] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in a variety of devices, such as general-purpose computers, wireless communication handheld devices, or multi-purpose integrated circuit devices, including applications in wireless communication handheld devices and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, these techniques can be implemented at least in part through a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include a memory or data storage medium, such as the memory or data storage medium described herein. Furthermore or alternatively, these techniques can be implemented at least in part through a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures, and that can be accessed, read, and / or executed by a computer, for example, by propagating signals or waves.

[0118] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided in a dedicated software or hardware module configured to implement a suspended database update system.

[0119] As used herein, the terms "machine-readable medium" and its equivalents "machine-readable storage medium," "computer-readable medium," and "computer-readable storage medium" refer to media including, but not limited to, portable or non-portable storage devices, optical storage devices, removable or non-removable storage devices, and other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored, and which do not contain carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (e.g., optical discs (CDs) or digital versatile optical discs (DVDs)), solid-state drives (SSDs), flash memory, memory, or memory devices.

[0120] Machine-readable media or machine-readable storage media may store code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted in any suitable manner, including memory sharing, message passing, token passing, network transmission, etc. Other examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include, but are not limited to, recordable media (e.g., volatile and non-volatile memory devices, floppy disks and other removable disks, hard disk drives, optical disks (e.g., CDs, DVDs, etc.)) and transmission media (e.g., digital and analog communication links).

[0121] As can be envisioned, although the examples in this document may indicate or refer to a single medium as a machine-readable medium or machine-readable storage medium, the terms "machine-readable medium" and "machine-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more sets of instructions. The terms "machine-readable medium" and "machine-readable storage medium" should also be understood to include any medium capable of storing, encoding, or carrying a set of instructions for execution by a system and causing the system to perform one or more of the methods or modules disclosed herein.

[0122] Some parts of this paper can be presented in the form of algorithms and symbolic representations of operations on data bits in computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to others skilled in the art. Algorithms are understood here and generally as self-consistent sequences of operations that ultimately produce the desired result. These operations require physical manipulation of physical quantities. Usually, but not necessarily, these physical quantities are in the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been shown that it is sometimes convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc., mainly for common reasons.

[0123] However, it should be noted that all these and similar terms should be associated with appropriate physical quantities; they are merely convenient labels applied to these physical quantities. Unless otherwise explicitly stated in the following discussion, it should be understood that throughout the description, discussions using terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or “generating” refer to the operation and processing of a computer system or similar electronic computing device that manipulates and converts data represented in physical (electronic) quantities in the computer system's registers and memories into other data similarly represented in physical quantities in the computer system's memory, registers, or other such information storage, transmission, or display devices.

[0124] It is also important to note that the various implementations can be described as processes, which may be represented by flowcharts, schematic diagrams, data flow diagrams, structure diagrams, or block diagrams (e.g., example process 800 in Figure 8). While flowcharts, schematic diagrams, data flow diagrams, structure diagrams, or block diagrams can describe operations as sequential processes, many of these operations can be executed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. The process shown in the figure terminates upon completion of an operation, but there may be additional steps not included in the figure. A process can correspond to a method, function, program, subroutine, subroutines, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0125] In some examples, machine learning or artificial intelligence algorithms can be used to implement one or more of the algorithms described in this article. Such machine learning or artificial intelligence algorithms can be trained using supervised learning, unsupervised learning, reinforcement learning, or other similar training techniques. For example, one of various machine learning algorithms can be used to analyze a dataset to identify correlations between different elements in the dataset without supervision and feedback (e.g., unsupervised training techniques). Machine learning data analysis algorithms can also be trained using sample data or real-time data to identify potential correlations. Such algorithms can include k-means clustering, fuzzy c-means (FCM), expectation-maximization (EM), hierarchical clustering, density-based noise applied spatial clustering (DBSCAN), and more. Other examples of machine learning or artificial intelligence algorithms include, but are not limited to, genetic algorithms, backpropagation, reinforcement learning, decision trees, linear classification, artificial neural networks, anomaly detection, and more. More generally, machine learning or artificial intelligence methods can include regression analysis, dimensionality reduction, meta-learning, reinforcement learning, deep learning, and other such algorithms and / or methods. It is conceivable that, due to the degree of overlap between these fields, the terms "machine learning" and "artificial intelligence" are often used interchangeably, and many publicly available technologies and algorithms employ similar approaches.

[0126] As an example of supervised training techniques, a dataset can be selected to train a machine learning model to facilitate the identification of correlations among members of the dataset. The machine learning model can then be evaluated to determine, based on the sample inputs provided to it, whether it accurately establishes correlations among members of the dataset. Based on this evaluation, the machine learning model can be modified to improve its likelihood of identifying the desired correlations. The machine learning model can also be dynamically trained by soliciting feedback from system users regarding the effectiveness of the correlations provided by the machine learning or artificial intelligence algorithm (i.e., supervision). The machine learning or artificial intelligence algorithm can utilize this feedback to improve the algorithm used to generate correlations (e.g., the feedback can be used to further train the machine learning or artificial intelligence to provide more accurate correlations).

[0127] The various examples of flowcharts, flowcharts, data flow diagrams, structural diagrams, or block diagrams discussed herein can also be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., a computer program product) used to perform the necessary tasks can be stored in computer-readable or machine-readable storage media (e.g., media for storing program code or code segments), such as those described herein. A processor implemented in an integrated circuit can perform the necessary tasks.

[0128] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the implementations disclosed herein can be implemented in the form of electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been generally described above according to their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

[0129] However, it is important to note that the algorithms and display methods presented in this article are not necessarily related to any particular computer or other device. Various general-purpose systems can be used with the programs taught in this article, or it can be demonstrated that it is convenient to build more specialized devices to perform some of the examples. The necessary structures for various such systems can be seen from the descriptions below. Furthermore, these techniques are not described with reference to any particular programming language, so various examples can be implemented using various programming languages.

[0130] In various implementations, the system operates as a standalone device or can be connected (e.g., networked) to other systems. In a networked deployment, the system can operate as a server or client system in a client-server network environment, or as a peer-to-peer system in a peer-to-peer (or distributed) network environment.

[0131] The system can be a server computer, client computer, personal computer (PC), tablet computer (such as iPad®, Microsoft Surface®, Chromebook®, etc.), laptop computer, set-top box (STB), personal digital assistant (PDA), mobile device (such as mobile phone, iPhone® and Android® devices, Blackberry®, etc.), wearable device, embedded computer system, e-book reader, processor, telephone, network device, network router, switch or bridge, or any system capable of executing a set of instructions (sequentially or otherwise) that specify the operations to be performed by the system. The system can also be a virtual system, such as a virtual version of the aforementioned device that can be hosted on another computer device (e.g., computer device 902).

[0132] Typically, routines executed to implement the embodiments of this disclosure may be implemented as an operating system or as part of a particular application, component, program, object, module, or sequence of instructions (collectively, "computer programs"). A computer program typically includes one or more instructions set at different times in various memories and storage devices in a computer, which, when read and executed by one or more processing units or processors in the computer, cause the computer to perform operations to execute elements relating to various aspects of this disclosure.

[0133] Furthermore, although examples have been described in the context of full-featured computers and computer systems, those skilled in the art will understand that various examples can be distributed as program objects in various forms, and this disclosure applies equally to any particular type of machine or computer-readable medium in which the distribution is actually implemented.

[0134] In some cases, the operation of a memory device, such as a change in state from binary 1 to binary 0 or from binary 0 to binary 1, may involve transitions, such as physical transitions. For certain types of memory devices, such physical transitions may include physically transforming an article of matter into a different state or thing. For example, but not limited to, for some types of memory devices, state changes may involve the accumulation and storage of charge or the release of stored charge. Similarly, in other memory devices, state changes may include a physical change or transition in magnetic orientation, or a physical change or transformation in molecular structure, such as a transition from a crystalline to an amorphous state or from an amorphous to a crystalline state. The foregoing is not intended to be an exhaustive list of all examples of transitions (such as physical transitions) that may occur in a memory device from binary 1 to binary 0 or from binary 0 to binary 1. Rather, the foregoing is intended as illustrative examples.

[0135] Storage media can typically be non-transitory, or include non-transitory devices. In this context, a non-transitory storage medium can include tangible devices, meaning that the device has a concrete physical form, although the device may change its physical state. Thus, for example, non-transitory means that the device remains tangible despite changes in its state.

[0136] The above description and accompanying drawings are merely illustrative and should not be construed as limiting or restricting the subject matter to the specific forms disclosed. Those skilled in the art will understand that many modifications and variations are possible based on the above disclosure, and can be modified and varied therewith without departing from the broader scope of the embodiments described herein. Numerous specific details have been described to provide a thorough understanding of this disclosure. However, in some cases, well-known or conventional details have not been described to avoid obscuring the description.

[0137] As used herein, the terms “connection,” “coupling,” or any variation thereof, when applied to modules of a system, refer to any direct or indirect connection or coupling between two or more elements; the connection or coupling between elements can be physical, logical, or any combination thereof. Furthermore, the terms “this article,” “above,” “below,” and similar terms as used in this application should refer to the entire application, and not any specific part of it. Where the context permits, singular or plural terms used in the above detailed description may also include either the plural or the singular, respectively. The word “or” refers to a list of two or more items and covers all of the following interpretations of the word: any item in the list, all items in the list, or any combination of items in the list.

[0138] The terms “a,” “an,” “the,” and other such singular referents used herein should be interpreted as including both singular and plural, unless otherwise stated herein or in obvious contradiction.

[0139] As used herein, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended (e.g., “including” should be interpreted as “including but not limited to”) unless otherwise stated or the context clearly contradicts it.

[0140] As used herein, the enumeration of value ranges is intended as a shorthand method for individually referencing each individual value falling within that range, unless otherwise stated or the context clearly contradicts it. Therefore, each individual value of the range is incorporated into the specification as if it were enumerated separately herein.

[0141] As used herein, the terms “set” (e.g., “item set”) and “subset” (e.g., “subset of an item set”) should be understood as non-empty sets containing one or more members, unless otherwise stated or the context clearly contradicts. Furthermore, unless otherwise stated or the context clearly contradicts, the term “subset of a corresponding set” does not necessarily mean a proper subset of the set, but rather that the subset and the set may contain the same elements (i.e., the set and the subset may be the same).

[0142] As used herein, the use of conjunctions such as "at least one of A, B, and C" should be interpreted as referring to one or more of A, B, and C (e.g., any of the following non-empty subsets of the set {A, B, C}, namely: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, or {A, B, C}), unless otherwise stated or clearly contradicted by the context. Therefore, conjunctions such as "at least one of A, B, and C" do not imply the requirement of at least one A, at least one B, and at least one C.

[0143] As used herein, the use of examples or exemplary language (e.g., "for example" or "as an example") is intended to illustrate implementation more clearly and does not impose a limitation on the scope unless otherwise stated in the claims. Such language in the specification should not be construed as requiring any unclaimed element to implement the examples described and claimed in this disclosure.

[0144] As used herein, when describing a component as being "configured" to perform certain operations, such configuration can be achieved by, for example, designing electronic circuits or other hardware, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits), or by any combination thereof.

[0145] Those skilled in the art will understand that the disclosed subject matter may be embodied in other forms and manners not shown below. It should be understood that the use of relational terms such as first, second, above, below, etc. (if any) is only for distinguishing one entity or action from another entity or action, and does not necessarily require or imply any actual relationship or order between these entities or actions.

[0146] Although processes or blocks are presented in a specific order, alternatives can be implemented in different orders, such as executing routines containing steps or using systems containing blocks. Furthermore, some processes or blocks can be deleted, moved, added, subdivided, replaced, merged, and / or modified to provide alternatives or subcombinations. Each of these processes or blocks can be implemented in multiple different ways. Additionally, although processes or blocks sometimes appear to execute sequentially, they can also execute in parallel or at different times. Moreover, any specific numbers mentioned herein are merely illustrative: alternatives can take on different values ​​or ranges.

[0147] The teachings provided in this document can be applied to other systems, not just those described above. Elements and actions from the various examples above can be combined to provide additional examples.

[0148] Any of the foregoing patents, applications, and other references, including any patents and applications that may be listed in the accompanying filings, are incorporated herein by reference. If necessary, aspects of this disclosure may be modified to incorporate the systems, functions, and concepts of the various references cited above to provide further examples of this disclosure.

[0149] These and other modifications may be made to this disclosure based on the detailed description above. While the foregoing description describes certain examples and outlines the expected best practices, these teachings can be practiced in a variety of ways, however detailed they may be in the text. The details of a system may vary considerably in their implementation details, but remain included within the subject matter of this disclosure. As noted above, specific terms used in describing certain features or aspects of this disclosure should not be construed as implying that such terms are redefined herein as limited to any particular characteristic, feature, or aspect of the disclosure associated with that term. In general, unless these terms are expressly defined in the detailed description above, the terms used in the appended claims should not be construed as limiting this disclosure to the specific implementations disclosed in the specification. Therefore, the actual scope of this disclosure includes not only the disclosed implementations but also all equivalent ways of practicing or implementing this disclosure under the claims.

[0150] Although certain aspects of this disclosure are presented below in the form of certain claims, the inventors contemplate covering all aspects of this disclosure in any number of claims. Any claim treated under 45 USC § 112(f) begins with "means for...". Therefore, the applicant reserves the right to add additional claims after filing the application to adopt such additional claim forms for other aspects of this disclosure.

[0151] The terms used in this specification generally have their ordinary meaning in the art, in the context of this disclosure, and in the specific context in which each term is used. Certain terms used to describe this disclosure have been discussed above or elsewhere in this specification to provide additional guidance to practitioners regarding the description of this disclosure. For convenience, some terms may be highlighted, for example, by using capital letters, italics, and / or quotation marks. The use of highlighting does not affect the scope and meaning of the terms; whether highlighted or not, the scope and meaning of the terms are the same in the same context. It should be understood that the same elements can be described in multiple ways.

[0152] Therefore, any one or more terms discussed herein may be used in alternative languages ​​and synonyms, and have no particular meaning whether or not the terms are elaborated or discussed herein. Synonyms for certain terms are provided. Listing one or more synonyms does not preclude the use of other synonyms. Examples used anywhere in this specification, including examples of any terms discussed herein, are illustrative only and are not intended to further limit the scope and meaning of this disclosure or any exemplary terms. Similarly, this disclosure is not limited to the various examples given in this specification.

[0153] Without intending to further limit the scope of this disclosure, examples of instruments, apparatus, methods, and related results based on this disclosure are given below. Note that headings or subheadings may be used in the examples for the reader's convenience, but this should in no way limit the scope of this disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In case of any conflict, this document (including the definitions) shall prevail.

[0154] Several sections of this specification describe examples of algorithms and symbolic representations of information operations. These algorithmic descriptions and representations are commonly used by those skilled in the art of data processing to effectively communicate the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, they should be understood to be implemented by computer programs or equivalent circuits, microcode, etc. Furthermore, it is sometimes convenient to arrange these operations as modules without loss of generality. The described operations and their associated modules can be implemented in software, firmware, hardware, or any combination thereof.

[0155] Any step, operation, or process described herein may be performed or implemented, alone or in combination with other means, using one or more hardware or software modules. In some examples, the software module is implemented using a computer program object, which includes a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0156] Examples may also relate to devices for performing the operations described herein. Such devices may be specifically constructed for the desired purpose, and / or may include general-purpose computing devices that can be selectively activated or reconfigured by a computer program stored in a computer. Such computer programs may be stored in a non-transitory, tangible, computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system mentioned in this specification may include a single processor or an architecture employing a multiprocessor design to enhance computing power.

[0157] Examples may also involve objects generated by the computational processes described herein. Such objects may include information generated by the computational processes, wherein the information is stored on a non-transitory, tangible, computer-readable storage medium, and may include any implementation of the computer program objects or other combinations of data described herein.

[0158] The language used in this specification has been chosen primarily for readability and instruction purposes and may not have been chosen to define or limit the subject matter. Therefore, the scope of this disclosure is not limited by this detailed description, but rather by any claims of the applications based thereon. Thus, the exemplary disclosure is intended to illustrate, but not limit, the scope of the subject matter set forth in the appended claims.

[0159] Specific details have been provided in the foregoing description to offer a thorough understanding of the various implementations of the system and components of the context-connected system. However, those skilled in the art will understand that the above implementations can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as block diagrams to avoid obscuring the implementation with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the implementation.

[0160] For illustrative and descriptive purposes, the above detailed description of the technology has been given. It is not intended to be exhaustive or to limit the technology to the precise forms disclosed. Many modifications and variations can be made based on the above teachings. The described embodiments were chosen to best explain the principles of the technology, its practical application, and to enable others skilled in the art to utilize the technology in various embodiments and make various modifications to suit a particular intended use. The scope of this technology is defined by the claims.

Claims

1. A method, the method comprising: Intercept function calls from a set of framebuffers to the hardware abstraction layer, wherein the function call from each of the set of framebuffers is associated with a graphical component to be displayed in the graphical user interface; Combining the graphics components of one or more framebuffers in the set of framebuffers into a single graphics plane, wherein the function calls of the one or more framebuffers are intercepted within a time interval; and The single graphics plane is transmitted to the hardware abstraction layer for rendering on the display of the display device.

2. The method according to claim 1, wherein, At least one of the frame buffers in the set of frame buffers is a direct frame buffer.

3. The method according to claim 1, wherein, The single graphics plane is transmitted to the graphics output protocol buffer of the hardware abstraction layer.

4. The method according to claim 1, wherein, The one or more frame buffers are configured to use Arm frame buffer compression.

5. The method according to claim 1, wherein, Intercepting function calls from the set of frame buffers includes unregistering function calls with the hardware abstraction layer and redirecting the function calls to an integrated composition block.

6. The method according to claim 1, wherein, Each frame buffer is associated with a different process executing on the display device, and each frame buffer is configured to output graphics components.

7. The method according to claim 1, wherein, Intercept function calls from a set of framebuffers after the windowing process and before graphics rendering.

8. A system comprising: One or more processors; A non-transitory computer-readable medium for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: Intercept function calls from a set of framebuffers to the hardware abstraction layer, wherein the function call from each of the set of framebuffers is associated with a graphical component to be displayed in the graphical user interface; Combining the graphics components of one or more framebuffers in the set of framebuffers into a single graphics plane, wherein the function calls of the one or more framebuffers are intercepted within a time interval; and The single graphics plane is transmitted to the hardware abstraction layer for rendering on the display of the display device.

9. The system according to claim 8, wherein, At least one of the frame buffers in the set of frame buffers is a direct frame buffer.

10. The system according to claim 8, wherein, The single graphics plane is transmitted to the graphics output protocol buffer of the hardware abstraction layer.

11. The system according to claim 8, wherein, The one or more frame buffers are configured to use Arm frame buffer compression.

12. The system according to claim 8, wherein, Intercepting function calls from a set of frame buffers includes unregistering function calls with the hardware abstraction layer and redirecting the function calls to an integrated composition block.

13. The system according to claim 8, wherein, Each frame buffer is associated with a different process executing on the display device, and each frame buffer is configured to output graphics components.

14. The system according to claim 8, wherein, Intercept function calls from a set of framebuffers after the windowing process and before graphics rendering.

15. A non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform an operation, the operation comprising: Intercept function calls from a set of framebuffers to the hardware abstraction layer, wherein the function call from each of the set of framebuffers is associated with a graphical component to be displayed in the graphical user interface; The graphics components of one or more framebuffers in the set of framebuffers are combined into a single graphics plane, wherein the function calls of the one or more framebuffers are intercepted within a time interval; and The single graphics plane is transmitted to the hardware abstraction layer for rendering on the display of the display device.

16. The non-transitory computer-readable medium according to claim 15, wherein, At least one of the frame buffers in the set of frame buffers is a direct frame buffer.

17. The non-transitory computer-readable medium according to claim 15, wherein, The single graphics plane is transmitted to the graphics output protocol buffer of the hardware abstraction layer.

18. The non-transitory computer-readable medium according to claim 15, wherein, The one or more frame buffers are configured to use Arm frame buffer compression.

19. The non-transitory computer-readable medium according to claim 15, wherein, Intercepting function calls from the set of frame buffers includes unregistering function calls with the hardware abstraction layer and redirecting the function calls to an integrated composition block.

20. The non-transitory computer-readable medium according to claim 15, wherein, Each frame buffer is associated with a different process executing on the display device, and each frame buffer is configured to output graphics components.

Images