Multi-screen unified frame buffer scene change area acquisition method and device, equipment and medium

Abstract

The embodiment of the application provides a multi-screen unified frame buffer scene change area acquisition method and device, electronic equipment and a storage medium, and relates to the technical field of computer graphics processing. By excluding the coverage window of a compositor and directly listening to change events of each application sub-window, the problem that the change area always returns the root window area in the multi-screen unified frame buffer scene is solved, and the acquisition of the accurate local change area is realized. Meanwhile, by using the cumulative merging mechanism based on the window identifier, the area fragmentation is reduced, the system overhead and bandwidth occupation are reduced, the best screen capture time is provided by the buffer exchange before and after listening, the best screen capture scheme is provided for the cloud desktop, and the scheme can be applied to most domestic operating system environments.

Description

Technical Field

[0001] This application relates to the field of computer graphics processing technology, specifically to a method and apparatus for obtaining scene change areas in a multi-screen unified frame buffer, an electronic device, and a storage medium. Background Technology

[0002] Currently, domestic operating systems such as Kylin V10 and UnionTech widely adopt multi-screen unified frame buffer scene window manager technology to improve the visual experience. In the traditional independent frame buffer window management mode, the window system server can accurately report the position and size of the screen's changing areas through compositor expansion, providing accurate incremental update basis for applications such as screen recording and remote desktop.

[0003] In related technologies, the main solutions for obtaining screen change areas in a multi-screen unified frame buffer scenario include: a full-screen refresh solution, which collects and encodes the entire screen each time; a solution that listens for changes in the root window content, which obtains the change area by listening to extended events of changes in the root window content; and a copy frame buffer difference comparison solution, which obtains the change area by comparing the differences between the data before and after the frame buffer.

[0004] In the process of implementing the embodiments of this application, it was found that at least the following problems exist in the related technology: in the multi-screen unified frame caching scenario, all window contents are rendered to off-screen textures and then uniformly synthesized into a full-screen covering window, which causes the compositor to always detect the size of the root window rather than the real local area, and it is impossible to distinguish which specific application window the change comes from, so it is impossible to obtain the accurate screen change area while retaining high-quality visual effects. Summary of the Invention

[0005] This application provides a method and apparatus for obtaining the scene change area of ​​a multi-screen unified frame buffer, an electronic device, and a storage medium.

[0006] The first aspect of this application provides a method for obtaining a change area in a multi-screen unified frame buffer scene, comprising: identifying the overlay window of the compositor and excluding listening to window content change notifications of the overlay window; maintaining a window listening set, the window listening set containing window identifiers of at least one application sub-window; in response to receiving a window content change notification event from an application sub-window, merging the current change area with the historical cumulative change area corresponding to the same window identifier according to the window identifier in the window content change notification event to obtain a merged change area; and outputting the merged change area.

[0007] A second aspect of this application provides a multi-screen unified frame buffer scene change area acquisition device, including a processor and a memory storing program instructions. The processor is configured to execute the multi-screen unified frame buffer scene change area acquisition method as described in the first aspect of this application when running the program instructions.

[0008] A third aspect of this application provides an electronic device, including: The electronic device body; and the multi-screen unified frame buffer scene change area acquisition device as described in the second aspect of the embodiments of this application, are installed on the electronic device body.

[0009] A fourth aspect of the embodiments of this application provides a computer-readable storage medium storing program instructions, which, when executed, cause a computer to perform the multi-screen unified frame buffer scene change area acquisition method as described in the first aspect of the embodiments of this application.

[0010] The method, apparatus, electronic device, and storage medium for obtaining the multi-screen unified frame buffer scene change area provided in this application embodiment have the following beneficial effects: This application's embodiments solve the problem that in multi-screen unified frame buffer scenarios, the compositor always returns to the root window's region when listening to the desktop, by excluding the compositor's overlay window and directly listening to change events in each application's sub-windows, thus achieving precise acquisition of locally changing regions. Simultaneously, through an accumulation and merging mechanism based on window identifiers, it reduces region fragmentation, lowering system overhead and bandwidth usage. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 (a) is a schematic diagram of an independent frame buffer scene rendering process provided in an embodiment of this application; Figure 1 (b) is a schematic diagram of a multi-screen unified frame caching scene rendering process provided in an embodiment of this application; Figure 2 (a) is a schematic diagram of a full-screen composition process of a window compositor in a multi-screen unified frame buffer scenario provided by an embodiment of this application; Figure 2 (b) is a schematic diagram illustrating the principle of message failure in a changing region under a multi-screen unified frame buffer scenario provided by an embodiment of this application; Figure 3 This is a flowchart illustrating a method for obtaining the scene change area of ​​a multi-screen unified frame buffer provided in an embodiment of this application; Figure 4This is a schematic diagram of a hash table and sub-window structure information provided in an embodiment of this application; Figure 5 This is a schematic diagram of a cloud desktop push service architecture provided in an embodiment of this application; Figure 6 This is a schematic diagram of the frame buffer structure in a cross-screen cloud desktop scenario provided by an embodiment of this application; Figure 7 This is a timing diagram of a vertical blanking signal synchronization mechanism provided in an embodiment of this application. Detailed Implementation

[0012] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0013] This application provides an electronic device including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it can implement the multi-screen unified frame buffer scene change area acquisition method provided in this application. The electronic device also includes communication interfaces, input / output interfaces, and other components. The processor is electrically connected to the above-mentioned components and is used to control the above-mentioned components to perform corresponding actions.

[0014] Based on the structure of the aforementioned electronic device, this application provides a method for obtaining a change area in a multi-screen unified frame buffer scene, comprising: identifying the overlay window of the compositor and excluding the listening to window content change notifications of the overlay window; maintaining a window listening set, the window listening set containing window identifiers of at least one application sub-window; in response to receiving a window content change notification event from an application sub-window, merging the current change area with the historical cumulative change area corresponding to the same window identifier according to the window identifier in the window content change notification event to obtain the merged change area; and outputting the merged change area.

[0015] In this embodiment, the core of the method lies in actively identifying and excluding compositor overlay windows in multi-screen unified frame buffer scenarios. This overlay window is the window output by the compositor after synthesizing the content of all application sub-windows. By detecting specific characteristics of this overlay window, it is excluded from the monitoring scope, thereby avoiding the problem that window content change notification events always return to the full-screen size area due to full-screen refresh of the overlay window. Figure 1 This is a diagram comparing the rendering process of an independent frame buffer window manager and a multi-screen unified frame buffer scene. Figure 1(a) uses XRender as its compositing backend and employs an independent frame buffer; Figure 1 (b) The 3D window manager uses OpenGL as the compositing backend and employs a unified frame buffer across multiple screens. Figure 1 (a) illustrates the normal flow in window manager mode with independent framebuffer mode. In window manager mode with independent framebuffer mode, multiple application child windows (window A, window B, and window C in the figure) are directly bound to the framebuffer (framebuffer in the figure). The X Window System Server (X Server in the figure) monitors screen changes through compositor extensions (X Damage extensions in the figure), which can accurately record the changed areas of each window. The final output window content change event (Damage event in the figure) contains precise position and size information (labeled as Damage:(x, y, width, height) in the figure). Figure 1 (b) illustrates the abnormal flow of the window manager in a multi-screen unified frame buffer scenario. In this scenario, each application's sub-windows are first rendered to off-screen textures (texture A, texture B, and texture C in the diagram), and then unified and composited by the graphics processor through a window compositor (compositer (ukui-kwin_x11) in the diagram). After compositing all window contents, the compositor outputs the full-screen content to an overlay window (overlay window in the diagram), whose window identifier is 0x66. Since the overlay window receives the full-screen composited image, the events of the changed area monitored by the compositor are always the size of the root window (Damage:(0,0,3840,1080) in the diagram), making it impossible to obtain precise local changed areas. This application solves this problem by identifying and excluding the overlay window and directly monitoring the changed area events of each application's sub-windows. Figure 1(b) illustrates the problem. For each application sub-window, maintain its independent window identifier and perform subsequent region accumulation and merging based on this identifier. When a window content change notification event is received from an application sub-window, extract the window identifier and the current change area from the event, and then search for the historical accumulated change area corresponding to the same window identifier in the window listener set. If the historical accumulated change area is empty, store the current change area as the historical accumulated change area; if the historical accumulated change area is not empty, geometrically merge the current change area with the historical accumulated change area to obtain the merged change area. Finally, output the merged change area to downstream modules, such as cloud desktop encoders or screen recording modules. The multi-screen unified frame buffer scene change area acquisition method provided in this application embodiment can accurately obtain the real change area of ​​each application sub-window while preserving high-quality visual effects, avoiding bandwidth waste caused by full-screen refresh, and reducing fragmentation processing overhead through region accumulation and merging.

[0016] Optionally, window content change notification is an extended mechanism provided by the window system server to monitor changes in the area, used to indicate the location and extent of changes within the window.

[0017] In this embodiment, when an application draws new content to a window, the window system server can record the rectangular area where the content has changed and generate a notification event for the changed area, which is then reported to the client that has registered to listen for the event. The event data structure of the changed area includes the identifier of the window that has changed, as well as the position and size of the changed area in the window's own coordinate system. Figure 2 (a) illustrates the complete process of a window compositor combining application sub-windows and outputting the full-screen output to the overlay window in a multi-screen unified frame buffer scenario. Figure 2 (b) shows a schematic diagram illustrating the principle of message failure in the changing region under this scenario. Figure 2 (a) The window layer includes three application sub-windows, each pointing to the window compositor (the 3D compositor (ukui-kwin_x11) in the diagram): a text editor (window identifier 0x45), a browser (window identifier 0x78), and a terminal (window identifier 0x23). The window compositor performs the following steps: Step 1, acquire the textures of each window; Step 2, perform off-screen rendering using the graphics processor; Step 3, composite the windows according to their stacking order; Step 4, add visual effects such as transparency, shadows, and animations; Step 5, output the composited full-screen image to... Figure 2(b) The overlay window (represented as Overlay Window in the diagram, window identifier 0x66, title 'Untitled', class name 'unknown'). Because the overlay window is fully refreshed every time, the notification events for changed areas detected by the window compositor's extended listener (Damage Notify event in the diagram) always return the full-screen size (Damage:(0,0,3840,1080) in the diagram), making it impossible to distinguish which specific window or which specific area within the window the change originates from. This application bypasses this overlay window by creating an object to listen for window events for each application child window, enabling the independent changed areas of each child window to be correctly captured. In this way, by utilizing the existing changed area listening extension mechanism of the window system server, accurate local window change information can be obtained without modifying the underlying window system, ensuring the compatibility and ease of deployment of the technical solution of this application.

[0018] Optionally, the window system server is the X Window System Server; the extended mechanism for listening to changed regions is provided by X services.

[0019] In this embodiment, the X Window System is a widely used window system on Unix-like operating systems, providing two programming interfaces: the XCB library and the Xlib library. The event structure of the X service extension contains a field representing the identifier of the window that has changed, and a field representing the changed area in the window coordinate system. In a window manager environment, due to the intervention of the compositor, directly using the X service's listening mechanism on the root window or overlay window will obtain change events for the entire full-screen area of ​​the root window. However, this application, by creating separate objects to listen for content changes for each application's child windows, can obtain the precise changed area for each child window. Thus, the technical solution of this application is highly compatible with the X Window System server and the X service extension, and can be directly deployed in the existing Xorg desktop environment without changing the window manager or upgrading the operating system, resulting in low modification costs.

[0020] Optionally, identifying the overlay window of the synthesizer includes: identifying the overlay window by at least one of the following features: the overlay window's window identifier is a fixed value, the overlay window's window class name is a predetermined string, or the overlay window's window title is a predetermined string.

[0021] In this embodiment, the overlay window in the window manager implementation of the UKUI desktop environment has stable and identifiable characteristics. A fixed window identifier means that the window number assigned to the overlay window is constant in each session. A predefined window class name means that the window class attribute of the overlay window is a specific string, which can be obtained by calling the window attribute retrieval function of the window system interface. A predefined window title means that the text displayed in the title bar of the overlay window is a specific string, which can also be obtained by the window attribute retrieval function. These three features can be used independently, meaning that satisfying any one of them is sufficient to identify an overlay window; they can also be used in combination to improve the accuracy of identification. Thus, through multi-feature combination identification, the compositor overlay window can be accurately distinguished from ordinary application child windows, avoiding the problem of mistakenly excluding ordinary windows from the listener or omitting overlay windows, ensuring the correctness of the listener list.

[0022] Optionally, the fixed value is 0x66; the predefined string is unknown or untitled.

[0023] In this embodiment, the aforementioned fixed value is an actual observed value in the window manager version of the UKUI desktop environment. It should be noted that different versions of the window manager or different operating system distributions may assign different window identifiers. Therefore, in practical applications, it is recommended to use a dynamic identification method, that is, to identify the window by reading its class name and title, rather than hardcoding the value. To ensure compatibility with different language environments, the identification logic should check multiple possible string values ​​simultaneously. Furthermore, the window manager may also identify overlay windows through other atomic attributes, such as checking whether the window's window type attribute is an overlay type. In this way, through configurable identification rules, this application can adapt to different versions and language environments of operating systems, exhibiting good portability and robustness.

[0024] Optionally, maintaining the window listener set includes: obtaining a list of child windows under the root window; and creating a window content change notification object for each application child window in the child window list after excluding the overlay window.

[0025] In this embodiment, obtaining the list of child windows under the root window can be achieved by calling the query tree interface of the window system server. This interface returns a list of direct child window identifiers of the root window and additional information for each child window. However, in a real system, the window hierarchy may have more than one level, and some application windows may be child windows of a container window under the root window rather than direct child windows of the root window. Therefore, when obtaining the list of child windows, the entire window tree should be recursively traversed to collect all window nodes, not just the direct child nodes of the root window. For each collected window, an overlay window identification is first performed; if it is an overlay window, it is skipped; if it is not an overlay window, it continues to determine whether the window is visible and whether it has actual content. For example, some windows only act as logical containers and do not participate in drawing; these windows do not need to create window content change notification objects. The process of creating a window content change notification object includes: calling the compositor's extended creation interface, passing in the window identifier and report level parameters. The report level can be set to a non-empty report level, that is, a notification event is generated immediately whenever a non-empty change area occurs within the window. In this way, through recursive enumeration and conditional filtering, the listening set can cover all application sub-windows that actually participate in the drawing, while excluding useless container windows and overlay windows, laying the data foundation for subsequent accurate region merging.

[0026] Optionally, the area of ​​change in this instance can be merged with the historical cumulative change area corresponding to the same window identifier, including: calculating the minimum bounding rectangle of the area of ​​change in this instance and the historical cumulative change area as the merged area of ​​change.

[0027] In this embodiment, let the coordinates of the top-left corner of the currently changed region be (x1, y1), the width be w1, and the height be h1. Let the coordinates of the top-left corner of the historical cumulative changed region be (x2, y2), the width be w2, and the height be h2. Then, the x-coordinate of the top-left corner of the merged changed region is min(x1, x2), and the y-coordinate of the top-left corner is min(y1, y2). The width of the merged changed region is max(x1+w1, x2+w2) minus its top-left corner x-coordinate, and the height is max(y1+h1, y2+h2) minus its top-left corner y-coordinate. This merged result is the smallest rectangle that can simultaneously contain two regions. When the number of regions to be merged is greater than 2, each new region can be sequentially processed with the current cumulative region using the above operations to finally obtain the smallest bounding rectangle of the union of all regions. Thus, the minimum bounding rectangle method requires only a few comparisons and addition / subtraction operations, with minimal computational overhead, making it suitable for use in high-frequency event processing paths. Although it introduces some region expansion, this expansion is usually small and its impact on coding efficiency is negligible.

[0028] Optionally, the method for obtaining the change area in a multi-screen unified frame cache scene also includes: when the display refresh synchronization signal is triggered, collecting the merged change areas corresponding to all window identifiers and clearing the historically accumulated change areas.

[0029] In this embodiment, at the end of each refresh cycle, the display controller emits a vertical synchronization signal. This signal can be captured by the application through window system extensions. When this signal is captured, a collection operation is performed: traversing all windows in the window listener set whose cumulative flag is true, for each such window, its historical cumulative change area is read, these areas are collected into a temporary list, then the window's cumulative flag is set to false, and the historical cumulative change area is cleared or reset to an empty rectangle. The collected list of areas represents all window areas that have changed during this cycle. After collection, these areas are passed to the subsequent global merging module for further processing. Figure 7 This is a timing diagram of the vertical blanking signal synchronization mechanism, showing the vertical blanking signal (VBLank signal in the diagram), with a period T of approximately 16.67 milliseconds, corresponding to a 60Hz monitor refresh rate. The diagram illustrates three vertical blanking cycles (cumulative cycle 1, cumulative cycle 2, and cumulative cycle 3). Within each cycle, multiple window content change events (Damage events in the diagram) arrive sequentially. Taking cumulative cycle 1 as an example: after the first window content change event occurs, the system records the changed area of ​​that event as R1; when the second window content change event arrives, the system merges this changed area with R1 to form R2; when the third window content change event arrives, the system merges this changed area with R2 to form R3. At the moment each vertical blanking signal arrives (marked "at each VBLank moment in the diagram"), the system performs the following operations: extracts the merged changed area (extracted and merged Damage area in the diagram), outputs a dirty area list (output dirty area list in the diagram), then clears the cumulative flag (cleared cumulative flag in the diagram), resets the historical cumulative areas of all windows, and begins the accumulation of the next vertical blanking cycle. This mechanism ensures that screen capture operations are synchronized with the display refresh, while reducing area fragmentation and encoder call counts through periodic accumulation and merging. Thus, the collection mechanism based on the display refresh synchronization signal ensures that screen capture operations remain synchronized with the display refresh, avoiding image tearing or content inconsistencies caused by improper capture timing; periodically clearing historical accumulated change areas ensures that each window restarts accumulation at the beginning of a new display cycle, guaranteeing data timeliness.

[0030] Optionally, the display refresh synchronization signal is a vertical blanking signal.

[0031] In this embodiment, the vertical blanking signal refers to the signal generated during the time period after each frame of image scanning ends, when the electron beam returns from the lower right corner of the screen to the upper left corner. During this signal period, the display does not update the image, making it the optimal time to capture and transmit the changed area of ​​the screen. By listening to this signal, it can be ensured that the merged changed area output by this application corresponds to the changed content in a complete frame. In practical applications, there are several ways to obtain this signal, including waiting for vertical synchronization through the frame buffer device interface provided by the operating system, or through the synchronization mechanism implicit in the swap buffer call of the graphics application interface. In the X Window system environment, the signal can be actively waited for through the vertical synchronization extension's wait vertical synchronization function. To improve efficiency, the listening thread of this application can set a timer with a period that is the same as or slightly less than the vertical blanking period to simulate the signal. In this way, using the vertical blanking signal as a synchronization anchor point can minimize screen tearing artifacts and improve the remote desktop experience; even if the hardware vertical synchronization signal is unavailable, the timer simulation method can provide a similar effect.

[0032] Optionally, the method for obtaining the scene change area of ​​the multi-screen unified frame buffer also includes: using a listening thread independent of the main screen capture thread, the listening thread establishing an independent connection with the window system server, and handling window events in an independent event loop.

[0033] In this embodiment, the number of window events in a multi-screen unified frame buffer scenario can be very large, especially when windows are frequently refreshed or users are interacting. If the listening task and the main screen capture task run in the same thread, event processing may block the main screen capture process, causing screen delay or frame drops. Therefore, this application uses an independent listening thread to specifically handle events from the window system server. When the listening thread starts, it calls the connection interface of the window system server to establish a new connection. This connection is independent of the connection used by the main screen capture thread, and each has its own event queue and file descriptor. The listening thread then enters an event loop, calling a function that blocks and waits for events within the loop. When no events arrive, the thread is suspended and does not consume CPU resources; when an event arrives, the thread is awakened and the corresponding processing function is called according to the event type. In this way, because the listening thread runs independently, even if a large number of window events flood in within a short period of time, it will not affect the normal operation of the main screen capture thread; the main screen capture thread can still periodically perform screen capture and encoding operations according to the rhythm of the vertical blanking signal, and the two do not interfere with each other.

[0034] Optionally, the window system server is an X Window System Server; the standalone connection is an XCB connection.

[0035] In this embodiment, XCB is a low-level library of the X Window System, providing an asynchronous, non-blocking interface. Using XCB connections can achieve better performance and lower memory overhead. An XCB connection is represented by a connection structure, which internally maintains the connection socket, event queue, and error queue. The listening thread calls the connection establishment function, passing in a display string parameter, which is usually obtained from the environment variable DISPLAY, with a default value of 0. After a successful connection, the listening thread calls the corresponding interface to select the event type for the window to be listened to. The event loop of the XCB connection can use a blocking wait event function, which returns an event object. The listening thread dispatches the event based on the response type field of the event object. In this way, using an independent XCB connection allows for the setting of an independent resource identifier allocator, avoiding resource contention with the main thread's connection; when the main thread's connection encounters an exception or needs to be reset, the listening thread's connection remains independent, improving the system's robustness.

[0036] Optionally, the listening thread and the main screen capture thread exchange information about changed regions through a lock-free circular buffer queue.

[0037] In this embodiment, the lock-free circular buffer is an efficient data structure for multi-threaded producer-consumer scenarios. The listening thread, acting as the producer, writes the resulting changed region information into the circular buffer after each received window content change event and region merging. The main screen-capturing thread, acting as the consumer, reads the changed region information from the circular buffer each time a vertical synchronization signal arrives. The core of the lock-free design lies in using atomic operations to update the read and write pointers, avoiding the overhead of mutex locks and potential lock contention. The size of the circular buffer needs to be pre-allocated, and each entry stores descriptive information about the changed region, including the window identifier, global coordinates, and size. When the buffer is full, the listening thread can choose to discard the latest changed region or overwrite the oldest entry. Since the changed region information is time-sensitive, an overwriting strategy is typically used to ensure that the main screen-capturing thread always obtains the latest changed region. Figure 5This is a schematic diagram of the cloud desktop push service architecture, dividing the entire system into two parts: a listening thread and a main screen capture thread. The listening thread includes: a sub-window listening module (shown in the diagram), responsible for establishing an independent connection with the X Window system server and receiving all window events; a window enumeration and state management module (shown in the diagram), maintaining window state information in a hash table and responding to events such as window creation, destruction, and configuration changes; and a window content change area merging module (shown in the diagram), which performs area accumulation and merging based on window identifiers according to received window content change notification events. The listening thread communicates with the main screen capture thread either through mutex lock shared memory (shown in the diagram) or through a lock-free circular buffer queue. The main screen capture thread includes: a changed region calculation module, responsible for handling special cases such as window position changes and transparency changes; a vertical blanking synchronization module (VBlank synchronization module in the diagram), which synchronizes with the display's vertical blanking signal and triggers the collection of merged changed regions; a region merging output module, which globally merges the changed regions of all windows, uses a scanline algorithm to merge overlapping regions into a minimum covering rectangle set, and degrades to full-screen refresh mode when the number of changed regions exceeds a preset threshold; finally, it outputs a list of dirty regions for the encoder to perform incremental encoding. The X service provides an event source for the listening thread, and the encoder receives the output of the main screen capture thread. This architecture achieves the separation of independent listening and main screen capture. In this way, compared with traditional queues protected by mutex locks, the lock-free circular buffer can significantly reduce latency and improve throughput in high-speed event flow scenarios, while avoiding memory allocation overhead.

[0038] Optionally, the window listener set is implemented using a hash table data structure, with the window identifier as the key, to store the state information of each application sub-window; the update of the state information adopts a copy-on-write strategy.

[0039] In this embodiment, the hash table provides insertion, deletion, and lookup operations with average constant time. The hash function can use the direct modulo method, where the remainder of the key value divided by the hash table size is used as the bucket index. Collision resolution uses a linked list method, maintaining a linked list in each bucket to store entries with the same hash value. To maintain the performance of the hash table, resizing and rehashing operations are required when the load factor exceeds a threshold. The copy-on-write strategy is a lock-free reading technique used in multi-threaded environments. The listening thread is responsible for updating window state information, while the main screen-capturing thread needs to read this information to calculate the changed area. If updates are performed directly on the original data structure, a mutex lock is required for protection, which may lead to thread blocking. With the copy-on-write strategy, when the listening thread needs to update the state information of a window, it does not directly modify the shared data structure, but instead copies the entry to be updated, modifies the copy, and then attempts to use an atomic operation to switch the shared pointer from pointing to the old entry to pointing to the new entry. Figure 4 This is a schematic diagram of the hash table and sub-window structure information. The hash table uses the window identifier as the key and is implemented using an array and linked lists. Each bucket in the array (indexed as 0, 1, 2, 3, and 4 in the diagram) stores a pointer to the head of the linked list, and window entries with the same hash value are connected through the linked list. Figure 4 The document also showcases the specific fields of a window entry (the `WindowDamageInfo` structure), including: window identifier (`window_id`, used as a hash key), window position (x, y, representing the x and y coordinates of the top-left corner of the window relative to the root window), window size (`width`, `height`, representing the width and height of the window), window opacity, a cumulative flag (`has_damage`, used to indicate whether a window content change event has been recorded in the current period), and a linked list pointer (`next`) to the next entry. This structure is identical to the content of each entry in the window listener set, supporting constant-time window lookup, facilitating rapid location of the historical cumulative change area of ​​the corresponding window upon receiving a window content change event. The state information is updated using a copy-on-write strategy to avoid lock contention between the listening thread and the main screen capture thread. This copy-on-write strategy eliminates the overhead of read-write locks, making it suitable for read-heavy, write-light scenarios. In the typical workload of this application, the frequency of window state changes is far lower than the processing frequency of window content change events, thus this strategy significantly improves concurrency performance.

[0040] Optionally, each entry in the window listener set includes: window identifier, window position, window size, window stacking order, window transparency, last accumulated change area, and accumulated flag.

[0041] In this embodiment, window position refers to the x and y coordinates of the top-left corner of the window in the root window's global coordinate system, in pixels. Window stacking order refers to the order of windows on the Z-axis; a larger value indicates the window is closer to the user, i.e., displayed above other windows. Window transparency refers to the window's opacity value, ranging from 0 to 1, where 0 represents complete transparency and 1 represents complete opacity. In the X window system, transparency is represented by a specific attribute whose value ranges from 0 to 1. The previous cumulative change area refers to the cumulative change area merged within the current vertical blanking cycle, represented by a rectangular structure containing the top-left corner coordinates and width and height. The cumulative flag is a Boolean value used to indicate whether at least one window content change event has been recorded within the current cycle. Thus, by maintaining complete state information for each window, the method can accurately perform region coordinate transformation, transparency cascading effect calculation, and old region marking during window movement, ensuring the comprehensiveness and accuracy of change area detection.

[0042] Optionally, the method for obtaining the change area in a multi-screen unified frame cache scene also includes: in response to receiving a window creation event, adding the new window to the window listener set and creating a window content change notification object for the new window.

[0043] In this embodiment, the data structure of the event includes the identifier of the new window and its parent window identifier. After the listening thread captures the creation event in the event loop, it first determines whether the new window is an overlay window. The determination method is the same as described above. If it is determined to be an overlay window, the window is ignored and no listening settings are applied to it. If it is determined to be a non-overlay application child window, an addition operation is performed: a new entry is created in the hash table, each field is initialized, the window identifier is set to the identifier of the new window, the window position and size can be obtained by querying the window attributes, the stacking order can be obtained by querying the window stacking order, the initial value of the transparency is set to completely opaque, the area of ​​the last accumulated change is set to empty, and the accumulated flag is set to false. Then, a window content change notification object is created for the new window, that is, the creation interface of the compositor extension is called, and the new window identifier is passed in. In this way, by dynamically responding to the window creation event, this application can capture newly launched application windows in real time and include them in the listening scope without restarting the service or re-enumerating windows, thus achieving good dynamic scalability.

[0044] Optionally, the method for obtaining the change area in a multi-screen unified frame cache scenario also includes: in response to receiving a window destruction event, removing the corresponding window identifier from the window listener set and destroying the corresponding window content change notification object.

[0045] In this embodiment, after the listening thread captures a destruction event, it extracts the window identifier from the event. It then searches for the corresponding entry in the hash table based on this window identifier. If the entry is not found, it indicates that the window may be an overlay window or another window not being listened to, and can be ignored. If the entry is found, it is deleted from the hash table. The deletion operation requires releasing the memory occupied by the entry and removing the entry from the linked list of the hash table. For thread safety, an appropriate synchronization mechanism should be used during deletion. For example, under a copy-on-write strategy, deletion can be achieved by replacing the entire hash table, or an atomic operation can be used to mark the entry as deleted. Simultaneously with deleting the entry, the destroy interface extended by the compositor needs to be called, passing in the object identifier of the previously created changed region, to release resources on the window system server side. This timely response to window destruction events and release of related resources avoids memory leaks and leaks of monitored window objects, ensuring the long-term stability of the system.

[0046] Optionally, the method for obtaining the change area in a multi-screen unified frame buffer scenario further includes: in response to receiving a window configuration change event, merging the old and new position areas of the corresponding window as the change area based on the window identifier in the window configuration change event.

[0047] In this embodiment, the event is triggered when a user drags the window title bar to move the window position or drags the window border to adjust the window size. When the window position changes, the window's coordinates in the root window's global coordinate system change, the window content at the original position disappears, and the window content at the new position appears; both areas should be considered as changed areas. Specifically, when the listening thread receives a window configuration change event, it extracts the window identifier, the new window position, and the new window size from the event. Then, it searches for the corresponding entry for the window in the hash table to obtain the old window position and the old window size. The old window area is calculated based on the old position and the old size, and the new window area is calculated based on the new position and the new size. The old window area and the new window area are used as the two current changed areas and merged with the window's historical accumulated changed areas. In this way, by including both the old and new positions of the window in the changed areas, it is ensured that after the window is moved or resized, the residual image at the original position and the image at the new position can be correctly updated, avoiding image residue or loss.

[0048] Optionally, the method for obtaining the change area of ​​a multi-screen unified frame cache scene also includes: in response to receiving a window configuration change event, updating the window position and window size of the corresponding window in the window listener set, and clearing the historical accumulated change area of ​​the corresponding window.

[0049] In this embodiment, after a window's position or size changes, the previously accumulated historical change areas are regions based on the old coordinate system, which are no longer meaningful in the new coordinate system. For example, if a window was originally located at a certain coordinate, its accumulated change area is recorded in the old coordinate system. When the window moves to a new position, the previously recorded accumulated change area actually points to a location that does not belong to the window in the root window's global coordinate system. If this area is not cleared and continues to be used, it will lead to incorrect area output. Therefore, when processing window configuration change events, in addition to using the old and new areas as change areas, it is also necessary to clear the window's historical accumulated change area, that is, reset the last accumulated change area to an empty rectangle and set the accumulated flag to false. In this way, clearing the historical accumulated change area avoids errors caused by coordinate system confusion, allowing window content change events at the new position to accumulate from a clean state, ensuring the correctness of area merging.

[0050] Optionally, the method for obtaining the change area in a multi-screen unified frame buffer scene further includes: in response to receiving a window attribute change event and the changed attribute being window transparency, according to the window identifier in the window attribute change event, taking the visible area of ​​the corresponding window and the overlapping area of ​​other windows with a stacking order lower than the corresponding window as the change area and performing a merging operation.

[0051] In this embodiment, in a multi-screen unified frame buffer scenario, the transparency of a window can be dynamically changed. When the transparency of a window changes, not only will the display effect of the window itself change, but the area of ​​the lower-level window visible through the transparent window will also be affected, because the compositor needs to blend the content of the transparent window with the content of the lower-level window when compositing the image. Specifically, after receiving an attribute change event, the listening thread checks whether the changed attribute atom is the atom corresponding to transparency. If so, it extracts the window identifier from the event. Based on the window identifier, it searches for the corresponding window entry in the hash table to obtain the visible area of ​​the window, i.e., the rectangular area of ​​the window in the root window's global coordinate system. Then, it obtains a list of the stacking order of all current windows. From the stacking order list, it finds all other windows with a stacking order lower than this window, i.e., windows drawn below this window. For each lower-level window, it calculates the overlap between its visible area and the visible area of ​​the transparent window, i.e., the intersection of the two rectangular areas. The visible area of ​​the transparent window itself and all calculated overlapping areas are sequentially used as the current change area and merged with the window's historical cumulative change area. In this way, by extending the influence of transparency changes to the overlapping area of ​​the lower-level windows, the semi-transparent effect is ensured to be correctly presented in the remote desktop, achieving complete adaptation to the compositing characteristics of multi-screen unified frame caching scenes.

[0052] Optionally, the method for obtaining the scene change area of ​​the multi-screen unified frame buffer also includes: maintaining the stacking order of each application sub-window in the window listening set, and determining other windows with a stacking order lower than the corresponding window based on the stacking order.

[0053] In this embodiment, the window manager maintains a window stack, with the top window displayed on top and the bottom window displayed on the bottom. The stacking order changes dynamically with user interaction. To accurately calculate the cascading effects of transparency changes, it is necessary to maintain the current stacking order of each window. This can be achieved by re-querying the window stacking order and updating the corresponding field in the hash table each time a window configuration change event or window focus event is received. A more efficient approach is to listen for changes in the window stack. When it is necessary to identify all windows with a stacking order lower than a given window, all window entries in the hash table can be traversed, the stacking order value of each window can be compared, and windows with a stacking order value lower than the given window can be selected. By maintaining the stacking order, the affected area during transparency changes can be accurately calculated, avoiding overlooking the refresh requirements of lower-level windows and ensuring the integrity of the composite image.

[0054] Optionally, the method for obtaining the change area in a multi-screen unified frame buffer scene also includes: in response to receiving a window mapping event or a window unmapping event, performing a merging operation on the entire visible area of ​​the corresponding window as the change area for this time.

[0055] In this embodiment, a window mapping event refers to an event that occurs when a window changes from a hidden state to a visible state, such as when a user launches a new application and its main window appears on the screen for the first time. A window unmapping event refers to an event that occurs when a window changes from a visible state to a hidden state, such as when a user minimizes or closes a window. When window mapping occurs, the entire area of ​​the window appears on the screen from nothing, so the entire window area should be considered a changed area. When window unmapping occurs, the area of ​​the window disappears from the screen, exposing the lower window area that was originally obscured by the window, so the entire area of ​​the window also needs to be marked as a changed area. The specific processing method is as follows: after the listening thread receives a mapping event or an unmapping event, it extracts the window identifier from the event, looks up the corresponding window entry in the hash table according to the window identifier, obtains the visible area of ​​the window, and uses this visible area as the changed area for this time, merging it with the window's historical accumulated changed areas. In this way, by processing window mapping and unmapping events, this application can correctly respond to the display and hiding of windows, ensuring the integrity of screen changed area detection throughout the window's lifecycle.

[0056] Optionally, the method for obtaining the change area in a multi-screen unified frame cache scene also includes: when the cumulative flag is false, directly recording the current change area in the window content change notification event as the historical cumulative change area and setting the cumulative flag to true; when the cumulative flag is true, performing a merging operation and updating the merged change area to the historical cumulative change area.

[0057] In this embodiment, the cumulative flag acts as a state machine, distinguishing different stages of the window within the current vertical blanking cycle. Initially, the cumulative flag is false, and the historical cumulative change area is empty or invalid. When the first window content change event arrives, since the flag is false, the branch performs a recording operation, directly assigning the current change area to the historical cumulative change area and flipping the flag to its value. Subsequently, all subsequent window content change events within this cycle will trigger a true branch, i.e., performing minimum bounding rectangle merging. This logic ensures that the first event does not require merging calculation, saving computational resources. When the vertical blanking signal is triggered, the system collects the historical cumulative change areas of all windows with true flags, then uniformly resets the flags of all windows to false and clears the historical cumulative change areas. In this way, the concise state machine design eliminates the need to introduce additional timestamps or counters to distinguish events in different cycles, reducing code complexity and error probability, while ensuring the correct timing of cumulative merging.

[0058] Optionally, after outputting the merged changed areas, the method further includes: performing a global merge on all the output changed areas, using a scanline algorithm to merge overlapping or adjacent areas into a minimum set of covering rectangles; and degrading to full-screen refresh mode when the number of changed areas exceeds a preset threshold.

[0059] In this embodiment, since the changing areas of multiple windows may overlap or be adjacent, directly providing these areas to the encoder one by one could lead to repeated encoding of overlapping areas or reduced encoding efficiency due to separate encoding of adjacent areas. The scanline algorithm uses the left and right boundaries of all rectangles as scan lines, sorts them by their horizontal coordinates, and then scans from left to right, maintaining the currently covered intervals. When the covered interval changes, a merged rectangle is output. This algorithm can merge any number of potentially overlapping rectangles into several non-overlapping rectangles, and the number of these rectangles is less than the original number of rectangles. When the number of changing areas exceeds a preset threshold, or the total area of ​​all changing areas exceeds a certain proportion of the screen area, it indicates that the changes on the screen are very drastic. In this case, processing the changing areas one by one is not very meaningful, and the algorithm directly degrades to a full-screen refresh mode, i.e., performing complete acquisition and encoding of the entire root window. In this way, global merging reduces the number of encoder calls and the encoding of duplicate areas, improving encoding efficiency; the degradation strategy ensures the stability and response speed of the system under extremely drastic change scenarios.

[0060] Optionally, the method for obtaining the scene change area of ​​the multi-screen unified frame buffer also includes: detecting the viewport position and size of all displays through the multi-display configuration interface, and establishing a mapping from global coordinates to display numbers; when the merged change area spans multiple displays, splitting the merged change area into multiple sub-regions, each sub-region corresponding to a display, and converting the coordinates of the sub-regions into the local coordinate system of the corresponding display.

[0061] In this embodiment, under the window manager, the content of all displays is composited into a unified root window buffer, whose total width is the sum of the widths of all displays, and whose height is the maximum height of each display. However, for cloud desktop transmission, image streams need to be transmitted separately for each display. The multi-display configuration interface provides a function to query the current display layout. Through this interface, information such as the name, viewport position (i.e., the coordinates of the top-left corner of the display in the global coordinate system of the root window), size (width and height), and refresh rate of each display can be obtained. Based on this information, a mapping function from the global coordinate point to the display number can be established. After calculating the merged change area, it is determined whether the area spans multiple displays. The determination method is: obtain the display number of the top-left corner of the area and the display number of the bottom-right corner of the area. If the two are different, it is a cross-display area. For cross-display areas, it is necessary to split them: find the intersection of the area with the viewport rectangle of each display, and each non-empty intersection is a sub-area. For each sub-region, its coordinates are originally in the global coordinate system of the root window, and need to be converted to the local coordinate system of the target display. The conversion method is: subtract the x-coordinate of the display viewport from the x-coordinate of the sub-region, and subtract the y-coordinate of the display viewport from the y-coordinate. Figure 6 This is a schematic diagram of the frame buffer structure in a cross-screen cloud desktop scenario (where the 3D window tube represents the compositor of the OpenGL backend). It shows the unified frame buffer after the composite of the multi-screen unified frame buffer scenario (the root window buffer in the figure), with a total resolution of 3840x1080, corresponding to a dual-screen scenario. Figure 6 The diagram also shows the viewports of two monitors: Monitor 0's viewport is located at (0,0) and has a size of 1920x1080; Monitor 1's viewport is located at (1920,0) and has a size of 1920x1080. Multiple windows are illustrated: Window A (located in the area of ​​Monitor 0), Window B (located in the area of ​​Monitor 0), Window C (located in the area of ​​Monitor 1), and Window D (spanning both monitors, spanning the area from x=0 to x=3840). When a window (such as Window D) spans multiple monitors, its changing area is split into multiple sub-regions: a portion located on Monitor 0 and a portion located on Monitor 1. The coordinates of each sub-region are converted to the local coordinate system of the corresponding monitor and then output separately. The viewport position and size of the monitors are obtained through the multi-monitor configuration interface (XRandR extension), establishing a mapping from global coordinates to monitor numbers and supporting dynamic mapping updates during monitor hot-swapping. Thus, through multi-screen adaptation and cross-screen area splitting, independent changing areas can be correctly output for each monitor, meeting the transmission requirements in cloud desktop multi-monitor scenarios.

[0062] Optionally, the multi-monitor configuration interface is an XRandR extension.

[0063] In this embodiment, XRandR is an extension of the X Window system, providing the ability to dynamically adjust screen resolution, rotation, reflection, and multi-monitor configuration. Through this extension, applications can obtain a list of all output devices in the system, with each output device corresponding to a physical monitor. For each output device, parameters such as its current connection status, resolution, refresh rate, viewport position, and rotation angle can be queried. Thus, using the XRandR extension provides the most accurate multi-monitor layout information and supports hot-swapping scenarios, improving the applicability of this application in complex hardware environments.

[0064] Optionally, the method for obtaining the scene change area of ​​the multi-screen unified frame buffer also includes: listening to the screen configuration change event of the multi-monitor configuration interface and dynamically updating the mapping from global coordinates to monitor number.

[0065] In this embodiment, the screen configuration change event is triggered when the display is connected, disconnected, or its resolution or position changes. During initialization, the listening thread should register the XRandR extended screen configuration change event in addition to registering window events. When this event is received, the listening thread re-detects the viewport position and size of all displays and reconstructs the global coordinate mapping table to the display number. Simultaneously, due to changes in the display layout, the entire screen's image may undergo a global change. In this case, the entire root window area can be marked as the changed area to ensure that the remote client's image is synchronized. This dynamic mapping update allows for adaptation to dynamic scenarios such as laptops connected to external monitors and projectors, seamlessly adapting to new display configurations without restarting the cloud desktop service.

[0066] To facilitate understanding of the technical solutions provided in the embodiments of this application, the following is combined with... Figure 3 The above plan will be explained in further detail. For example... Figure 3As shown, the process enumerates and establishes child window listening threads, obtains the root window identifier and child window identifiers, corresponding to the operations of establishing independent listening threads and obtaining all child windows under the root window; constructs a child window listening list with window identifiers as keys, which is implemented using a hash table data structure, and each entry contains state information such as window identifier, window position, window size, and transparency; enumerates windows and registers window content change notifications (i.e., window content change events) and window creation, destruction, configuration change, and attribute change events for each application child window excluding overlay windows; updates the child window listening list, maintains the state information of each window, including window position, size, stacking order, and transparency; and waits for the next XCB event, receiving and distributing the event. The event loop runs continuously, distributing events based on their type whenever a new event arrives. When a change region notification event is received, the change regions corresponding to the same window identifier are accumulated and merged, using the minimum bounding rectangle method to merge the current change region with the historical accumulated region. After merging, the accumulated region is updated, and the merged region is stored as the historical accumulated change region for that window. Other events, such as window position changes or window configuration change events, are handled, marking the old and new position regions as change regions. When the vertical blanking synchronization signal is triggered, the merged change regions of all windows are periodically collected, and the historical accumulated region is cleared to begin the next cycle of accumulation. The final dirty region list is output for the cloud desktop encoder to perform incremental encoding. In addition, all output change regions can be globally merged, and multi-screen adaptation and cross-screen region splitting can be performed. By identifying overlay windows, maintaining window listening sets, accumulating and merging regions based on window identifiers, collecting vertical blanking synchronization triggers, and performing multi-screen adaptation, screen change regions can be accurately obtained in a multi-screen unified frame buffer scenario.

[0067] This application provides a multi-screen unified frame buffer scene change area acquisition device, including a processor and a memory storing program instructions. The processor is configured to execute the multi-screen unified frame buffer scene change area acquisition method of this application when running the program instructions.

[0068] This application provides an electronic device, including: an electronic device body; and the aforementioned multi-screen unified frame buffer scene change area acquisition device, which is installed on the electronic device body.

[0069] This application provides a computer-readable storage medium storing program instructions, which, when executed, cause the computer to perform the above-described method for obtaining the scene change area of ​​a multi-screen unified frame buffer.

[0070] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for obtaining scene change areas in a multi-screen unified frame buffer, characterized in that, include: Identify the compositor's overlay window and exclude listening to notifications of changes to the overlay window's content; Maintain a set of window listeners, which contains the window identifiers of at least one application child window; In response to receiving a window content change notification event from an application child window, the changed area is merged with the historical cumulative changed area corresponding to the same window identifier according to the window identifier in the window content change notification event to obtain the merged changed area; Output the changed area after merging.

2. The method according to claim 1, characterized in that, The overlay window for identifying synthesizers includes: The overlay window is identified by at least one of the following features: the overlay window's window identifier is a fixed value, the overlay window's window class name is a predetermined string, or the overlay window's window title is a predetermined string.

3. The method according to claim 1, characterized in that, The area affected by this change will be merged with the historical cumulative change area corresponding to the same window identifier, including: The minimum bounding rectangle between the current change area and the historical cumulative change area is calculated as the merged change area.

4. The method according to claim 1, characterized in that, Also includes: When the display refresh synchronization signal is triggered, collect the merged changed areas corresponding to all window identifiers and clear the historical accumulated changed areas.

5. The method according to claim 1, characterized in that, Also includes: A listening thread independent of the main screen capture thread is used. The listening thread establishes an independent connection with the window system server and handles window events in an independent event loop.

6. The method according to claim 1, characterized in that, Also includes: The window listener set is implemented using a hash table data structure, with the window identifier as the key, to store the state information of each application sub-window; The status information is updated using a copy-on-write strategy.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: The viewport position and size of all monitors are detected through the multi-monitor configuration interface, and a mapping from global coordinates to monitor numbers is established. When the merged change area spans multiple displays, the merged change area is split into multiple sub-regions, each corresponding to one display, and the coordinates of the sub-regions are converted to the local coordinate system of the corresponding display.

8. A device for obtaining scene change areas in a multi-screen unified frame buffer, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the method for obtaining the scene change area of ​​the multi-screen unified frame buffer as described in any one of claims 1 to 7 when running the program instructions.

9. An electronic device, characterized in that, include: The electronic device itself; and, The multi-screen unified frame buffer scene change area acquisition device as described in claim 8 is installed on the electronic device body.

10. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed, they cause the computer to perform the method for obtaining the scene change area of ​​the multi-screen unified frame buffer as described in any one of claims 1 to 7.

Images