Remote display method, system and product based on linux kernel frame buffer

Abstract

The application discloses a Linux kernel frame buffer-based remote display method, system and product, comprising the following steps: obtaining original pixel data from a virtual frame buffer created by a kernel driver layer by using a back-end service layer, and generating target pixel data based on the original pixel data in response to a user access request; transmitting the target pixel data to a front-end rendering layer through a data channel established between the front-end rendering layer and the back-end service layer; mapping the target pixel data to a front-end display area by using the front-end rendering layer, and correcting the mapped data in the front-end display area according to a preset display format to obtain a target display picture. The embodiment is applied to a display system arranged in a Linux kernel, so that the display mechanism is independent of a user state graphic system or a virtualization environment, and a direct acquisition path of display data is established through creation and maintenance of the virtual frame buffer by the kernel driver layer, thereby remarkably expanding the application range of the remote display technology.

Description

Technical Field

[0001] This application relates to the field of display technology, specifically to a remote display method, system, and product based on the Linux kernel frame buffer. Background Technology

[0002] In the existing field of remote display on Linux systems, common technologies include Virtual Network Computing (VNC), open-source remote desktop protocol server (X Remote Desktop Protocol, XRDP), and Simple Protocol for Independent Computing Environments (SPICE), which is designed for virtualization scenarios.

[0003] However, the aforementioned remote display technologies are generally highly dependent on the graphics environment or virtualization environment and have high resource consumption, which leads to certain limitations of existing remote display technologies in embedded systems, system-level debugging, or resource-constrained environments. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the existing technology, it is desirable to provide a remote display method, system and product based on Linux kernel frame buffer. This method is applied to a display system set in the Linux kernel, so that the display mechanism does not depend on the user-space graphics system or virtualization environment. Furthermore, by creating and maintaining the virtual frame buffer through the kernel driver layer, a direct path for obtaining display data is established, so as to realize remote display capabilities in command-line environment, minimal system configuration and embedded bare metal environment, thereby significantly expanding the applicability of remote display technology.

[0005] In a first aspect, the present invention provides a remote display method based on a Linux kernel framebuffer. This method is applied to a remote display system based on a Linux kernel framebuffer. The system is set within a Linux kernel and includes a kernel driver layer, a backend service layer, and a frontend rendering layer. The method includes: The backend service layer obtains raw pixel data from the virtual frame buffer created by the kernel driver layer, and responds to user access requests to generate target pixel data based on the raw pixel data. The user access request is a request initiated by the user through the browser to access the target service address. The target pixel data is transmitted to the front-end rendering layer through the data channel established between the front-end rendering layer and the back-end service layer. The target pixel data is mapped to the front-end display area using the front-end rendering layer, and the mapped data in the front-end display area is corrected according to the preset display format to obtain the target display image.

[0006] In one possible implementation, the target pixel data is transmitted to the front-end rendering layer via a data channel established between the front-end rendering layer and the back-end service layer, including: When the target pixel data is uncompressed pixel data, the target pixel data is transmitted to the front-end rendering layer in binary frame form through the data channel established between the front-end rendering layer and the back-end service layer. When the target pixel data is compressed pixel data, the target pixel data is transmitted to the front-end rendering layer in the form of image frames through the data channel established between the front-end rendering layer and the back-end service layer.

[0007] In one possible implementation, the mapping data within the front-end display area is modified according to a preset display format, including: The mapping data is corrected based on the preset display resolution and / or the display characteristics of the front-end display area.

[0008] In one possible implementation, the raw pixel data is obtained in the following ways: The original pixel data is acquired periodically according to a preset frame rate; and / or Call the listener function to get update events. When the update event indicates that the state of the virtual frame buffer has changed, get the changed original pixel data.

[0009] In one possible implementation, the system also includes an application layer, and the method further includes: When the application layer performs graphics drawing operations or writes pixel data, it uses the kernel driver layer to mark the state of the virtual frame buffer as changing, and the kernel driver layer triggers an update event.

[0010] In one possible implementation, the method further includes: In response to a user's screen change request, the backend service layer transmits the screen configuration command corresponding to the screen change request to the kernel driver layer, which then updates the pixel data in the virtual frame buffer based on the screen configuration command.

[0011] Secondly, a remote display system based on the Linux kernel frame buffer is provided. This system is set in the Linux kernel and includes a kernel driver layer, a backend service layer, and a frontend rendering layer. The kernel driver layer is used to create virtual frame buffers and obtain raw pixel data from the virtual frame buffers; The backend service layer is used to respond to user access requests, generate target pixel data based on the original pixel data, and transmit the target pixel data to the frontend rendering layer through the data channel established between the frontend rendering layer and the backend service layer; where the user access request is a request initiated by the user through the browser to access the target service address; The front-end rendering layer is used to map target pixel data to the front-end display area and correct the mapped data in the front-end display area according to the preset display format to obtain the target display image.

[0012] In one possible implementation, the kernel driver layer is also used for, Create and manage standard frame buffer devices and dedicated control character devices, whereby the standard frame buffer devices are used to form pixel data channels and the dedicated control character devices are used to form control and event notification channels.

[0013] In one possible implementation, the kernel driver layer is also used for, Based on the dirty flag marking mechanism, when the state of the virtual frame buffer changes, an update event is triggered and the user space is notified.

[0014] Thirdly, a computer program product is provided, which contains instructions that, when the instructions are executed, are performed any of the methods described in the first aspect above.

[0015] The remote display method, system, and product based on Linux kernel frame buffer provided in this application construct a remote display mechanism that does not rely on user-space graphics systems and virtualization environments. By establishing a direct acquisition path for display data in the Linux kernel, remote display capabilities are realized in command-line environments, minimal system configurations, and embedded bare-metal environments, thereby significantly expanding the applicability of remote display technology. Secondly, by establishing a display data acquisition and output mechanism in the Linux kernel, the remote display function can be intervened during the system initialization phase, thereby improving system observability and debugging efficiency. Attached Figure Description

[0016] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the architecture of a remote display system 10 based on a Linux kernel frame buffer provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a remote display method based on the Linux kernel frame buffer provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the communication device provided in the embodiments of this application. Detailed Implementation

[0017] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments. Furthermore, the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The terms "first" and "second," etc., in the specification and claims of the embodiments of this application are used to distinguish different objects, not to describe a specific order of objects.

[0019] The following is an explanation of the terms used in this application: (1) Virtual network computing: Its architecture is a client-server architecture on the Linux platform. The core communication protocol is the Remote Framebuffer Protocol (RFB). This protocol uses pixels as the basic transmission unit and realizes cross-platform transmission and display of graphic data through capability negotiation between the client and the server.

[0020] Specifically, in terms of acquiring screen data, VNC servers typically rely on the underlying graphics system interface to obtain desktop content. For example, in the X Window system, the server obtains pixel data from the server's root window through a C language programming interface (Xlib interface) or a shared memory extension (MIT-SHM), and combines this with the XDamage extension to detect screen update areas, thereby achieving incremental updates based on the changed areas. In the Wayland environment, some remote display components integrated into the desktop environment obtain image output streams from the compositor through the PipeWire framework.

[0021] In terms of data transmission, Microsoft's Remote Desktop Protocol (RDP) supports various encoding methods to adapt to different bandwidth and performance requirements. Common encoding methods include Tight (combining zlib compression and JPEG encoding), Hextile (a block-based rectangular encoding), and TRLE (a run-length-based compression method). Correspondingly, the server selects the appropriate encoding format based on the client's declared capabilities and encodes the changing areas before transmission to reduce network bandwidth consumption.

[0022] On the client side, VNC clients (such as TigerVNC Viewer, Remmina, etc.) are used to implement the RFB protocol stack to decode the received data and reconstruct the remote desktop image locally, thereby enabling cross-platform remote graphical display.

[0023] (2) Open source remote desktop protocol server: It is an open source implementation of the Microsoft remote desktop protocol under the Linux platform. Its architecture is also a client-server architecture. Specifically, it can convert the output of the Linux graphics system into a data stream that conforms to the RDP protocol, thereby achieving interoperability with the standard RDP client.

[0024] Specifically, in terms of image data acquisition, XRDP typically integrates with the X Window system through a backend display module. Its mainstream implementation uses xorgxrdp as the driver module for the Xorg server, initiating an independent virtual Xorg session when establishing a remote connection. Correspondingly, this driver module acquires the image output through the Xorg driver interface and converts it into RDP graphics primitives. It should be noted that, compared to VNC, XRDP focuses on protocol-based semantic graphics representation, rather than simply pixel-level capture.

[0025] In terms of data encoding and transmission, XRDP utilizes various graphics encoding mechanisms provided by the RDP protocol, including a bitmap cache to reduce the transmission of duplicate data, a region-based update mechanism that transmits only changed graphics content, and various compression and encoding methods (e.g., RemoteFX or H.264-based encoding extensions). It should be noted that these mechanisms generally make it superior to pixel-based transmission schemes in terms of bandwidth utilization and interactive performance.

[0026] On the client side, the client is used to parse the RDP data stream and rebuild the graphical interface locally, thereby enabling access to and control of the remote Linux desktop; while XRDP is compatible with various standard RDP clients, including Windows Native Remote Desktop Connection, Remmina, FreeRDP, and Microsoft Remote Desktop.

[0027] (3) Remote display and interaction protocol designed for virtualization scenarios: It is widely used in virtualization platforms such as kernel-based virtual machine (KVM) / quick emulator (QEMU). Its architecture is also client-server architecture, but it focuses on high-performance graphics redirection and multimedia interaction capabilities.

[0028] Specifically, in terms of image data acquisition, a virtual graphics card device (e.g., a QXL graphics card or a Virtio Graphics Processing Unit, or Virtio-GPU for short) is integrated with the graphics stack inside the virtual machine. The graphics output within the virtual machine is captured by the SPICE Server, typically occurring at the virtual GPU driver layer or display protocol layer, to obtain higher semantic-level graphics operation information (e.g., drawing instructions, bitmap updates, etc.), and not just the final pixel data.

[0029] In terms of data encoding and transmission, the SPICE protocol employs a multi-channel architecture, transmitting display, input, audio, and USB redirection functions through different channels to improve overall throughput and response performance. Correspondingly, regarding graphics data processing, SPICE supports various encoding methods (including draw commands based on graphics instructions, image cache, and adaptive encoding of video streams (e.g., MJPEG or GStreamer-based video compression)). It also supports optimization for high-frequency changing areas (e.g., video playback windows) to reduce bandwidth consumption and improve display smoothness.

[0030] On the client side, SPICE provides dedicated clients (e.g., virt-viewer, remote-viewer, etc.) for connecting to the SPICE Server and parsing multi-channel data streams.

[0031] It should be noted that SPICE has a high performance advantage in virtualized desktop and cloud computing scenarios, but its technical approach relies on virtualization platforms and dedicated virtual graphics card drivers, resulting in a complex system architecture. It is also mainly designed for redirecting graphics output within virtual machines. Therefore, its applicability is limited in non-virtualized environments (such as bare-metal Linux systems or embedded devices).

[0032] Existing remote display technologies are relatively mature in general desktop or virtualization scenarios, but they still have certain limitations in embedded systems, system-level debugging, or resource-constrained environments. Specifically, they are highly dependent on the graphics or virtualization environment, have high system resource consumption, are unavailable in the early startup phase, lack platform adaptability and versatility, have dependence on specific clients leading to increased usage costs, and have difficulty obtaining the original output of the physical display path.

[0033] As a result, existing remote display technologies still suffer from problems such as dependence on the desktop environment, inability to be used in the early stages of startup, high resource consumption, and poor versatility.

[0034] In response, this application provides a remote display method based on the Linux kernel frame buffer. This method directly constructs the data acquisition path for display in the Linux kernel and uses the virtual frame buffer as a unified data source. It can achieve remote display without starting a graphical desktop, loading a display service process, or requiring window system support, thereby reducing dependence on the graphical environment and expanding the applicability of remote display technology.

[0035] In one possible implementation, Figure 1 This is a schematic diagram of the architecture of a remote display system 10 based on a Linux kernel framebuffer provided in this application embodiment, as shown below. Figure 1 As shown, the remote display system 10 adopts a layered architecture design, which can be divided into four layers: application layer 11, kernel driver layer 12, backend service layer 13, and frontend rendering layer 14. Among them, each layer interacts with the other through standardized interfaces.

[0036] For example, application layer 11 is deployed in the user space of the operating system and may include frame buffer configuration tools (e.g., the system tool fbset for viewing and modifying Linux kernel frame buffer device settings) and various graphical applications.

[0037] Specifically, the application layer 11 can interact with the kernel driver layer 12 through the standard frame buffer device interface provided by the operating system, using system calls such as memory mapping (mmap), writing (write), and control interface (ioctl), thereby enabling write, read, and parameter configuration operations on the virtual display device.

[0038] For example, kernel driver layer 12 runs in the kernel space of the operating system and can run as a kernel module, which is responsible for creating and managing virtual frame buffer devices.

[0039] Specifically, the kernel driver layer 12 can provide different device interfaces to both the application layer 11 and the backend service layer 13. The device interface provided to the application layer 11 is a standard frame buffer device interface (e.g., / dev / fbN) to ensure compatibility with the standard access methods of existing graphics applications; the device interface provided to the backend service layer 13 is a dedicated character device interface (e.g., / dev / webfb) to provide extended functions such as device information query, event notification subscription, and dynamic parameter configuration.

[0040] It should be noted that kernel driver layer 12 maintains a continuous virtual frame buffer as a shared pixel data storage area between layers.

[0041] For example, the backend service layer 13 runs in the user space of the operating system.

[0042] Specifically, the backend service layer 13 can obtain pixel data in the frame buffer by reading the device node created by the kernel driver layer 12, and perform color format conversion and image encoding processing on the obtained pixel data, so as to push the processed image frame to the frontend rendering layer 14; wherein, the backend service layer 13 pushes the image frame in two ways: full-duplex binary stream based on WebSocket and multipart image stream (MJPEG) based on HTTP.

[0043] Secondly, the backend service layer 13 can also provide an HTTP-based REST interface to receive configuration requests from the frontend rendering layer 14 and forward them to the kernel driver layer 12 for execution.

[0044] For example, the front-end rendering layer 14 runs in the client browser environment (i.e., the user space of the corresponding operating system).

[0045] Specifically, the front-end rendering layer 14 can receive frame data from the back-end service layer 13 via WebSocket or HTTP stream, and use the browser's built-in Canvas interface to draw pixel data onto the page canvas, or directly load the image stream into the page image elements to complete the display.

[0046] Based on this, the remote display system 10 provided in this embodiment is a remote display system based on a virtual framebuffer device and network streaming media transmission. In the remote display system 10, the kernel driver layer 12, as the core layer, can provide standard framebuffer access capabilities to the application layer 11 and frame data reading and event notification capabilities to the backend service layer 13. The backend service layer 13, as the bridge layer between the kernel driver layer 12 and the frontend rendering layer 14, is used to implement functions such as data acquisition, format conversion, encoding compression, and protocol adaptation. The frontend rendering layer 14, as the final human-computer interaction interface, is used to receive and display remote images.

[0047] In another embodiment of this application, a remote display method based on the Linux kernel frame buffer is also provided, which is applied to the remote display system 10 described above.

[0048] In one possible implementation, Figure 2 This is a flowchart illustrating a remote display method based on the Linux kernel framebuffer provided in an embodiment of this application, as shown below. Figure 2 As shown, the method specifically includes the following steps: Step S201: The backend service layer 13 obtains the original pixel data from the virtual frame buffer created by the kernel driver layer 12, and responds to the user access request to generate the target pixel data based on the original pixel data.

[0049] The user access request can be a request initiated by the user through a browser to access the target service address. Correspondingly, since the front-end rendering layer 14 runs in the client browser environment, the front-end rendering layer 14 can be loaded after the user accesses the back-end target service address through the browser and establish a connection with the back-end service layer 13.

[0050] Specifically, based on the WebSocket protocol stream service module in the backend service layer 13 and the WebSocket client in the frontend rendering layer 14, a persistent data channel can be established between the browser (i.e., the corresponding frontend rendering layer 14) and the backend service layer 13 to receive target pixel data through a long connection.

[0051] Optionally, based on the MJPEG protocol stream service module in the backend service layer 13 and the MJPEG receiver in the frontend rendering layer 14, a half-duplex data channel can be established between the browser (i.e., the corresponding frontend rendering layer 14) and the backend service layer 13 to receive target pixel data.

[0052] In one possible implementation, the frame capture module in the backend service layer 13 can be used to obtain the raw pixel data of the virtual frame buffer from the kernel driver layer 12.

[0053] For example, refer to Figure 1 The frame capture module can use the read system call to read the raw pixel data in the virtual frame buffer into the user space buffer, and reset the file offset before reading to ensure that complete frame data can be obtained every time.

[0054] In one example, raw pixel data can be acquired periodically according to a preset frame rate; that is, this acquisition mode is a timed trigger mode, which is more suitable for continuous refresh scenarios.

[0055] In another example, a listener function can be called to obtain update events. When an update event indicates a change in the state of the virtual frame buffer, the changed original pixel data can be obtained. This listener function is, for example, the poll function. That is, this acquisition mode is an event-driven mode, which specifically listens for update events through poll. It is more suitable for scenarios with high real-time requirements and irregular updates.

[0056] Correspondingly, when the application layer 11 performs graphics drawing operations or writes pixel data, the kernel driver layer 12 actively triggers an update event to mark the change in the state of the virtual frame buffer.

[0057] In one possible implementation, the encoding conversion module in the backend service layer 13 can be used to convert the original pixel data into target pixel data; wherein the image format of the target pixel data conforms to the network transmission format, and may include, for example, uncompressed pixel data and compressed pixel data.

[0058] For example, the core functions of the encoding conversion module include color format conversion and image encoding compression.

[0059] Specifically, color format conversion processes the pixel byte order of the virtual frame buffer (usually blue-green-red, BGR) to convert it into the standard red-green-blue-alpha (RGBA) format, thereby ensuring the consistency and accuracy of data transmission between different system levels.

[0060] In one example, for 32-bit per pixel BGRA format data, the encoding conversion module swaps the first byte (blue channel) and the third byte (red channel) of the original data, keeps the second byte (green channel) unchanged, and sets the alpha channel to be completely opaque.

[0061] In another example, for 24-bit per pixel BGR compact format data, the encoding conversion module swaps the first and third bytes of every three consecutive bytes and adds an alpha channel to the output to achieve a standard RGBA representation.

[0062] In another example, for RGB565 format data with 16 bits per pixel, the encoding conversion module expands the red, green, and blue components to eight bits of precision, completes the bit depth expansion by padding the low bits with the high bits, and supplements the alpha channel.

[0063] It should be noted that the conversion logic of the above three formats covers all pixel bit depth modes supported by the frame buffer device to ensure that color data remains consistent and accurate during transmission and display.

[0064] Specifically, image encoding compression encodes RGBA pixel data after color format conversion into compressed image data in JPEG format. During the encoding process, configurable image quality parameters are used to maintain a balance between compression ratio and image fidelity based on network bandwidth conditions and display requirements.

[0065] It should be noted that the above image encoding and compression is enabled after the current end service enables JPEG transmission, and the generated encoded image data can be directly used for HTTP real-time transmission to support remote display.

[0066] Step S202: The target pixel data is transmitted to the front-end rendering layer 14 through the data channel established between the front-end rendering layer 14 and the back-end service layer 13.

[0067] For example, when the target pixel data is uncompressed pixel data, the target pixel data is transmitted to the front-end rendering layer 14 in binary frame form through the data channel established between the front-end rendering layer 14 and the back-end service layer 13.

[0068] Specifically, the data channel between the front-end rendering layer 14 and the back-end service layer 13 is a WebSocket long-connection data channel, which transmits uncompressed target pixel data through the long connection based on the binary frame stream mode of WebSocket.

[0069] Correspondingly, the WebSocket client in the front-end rendering layer 14 continuously monitors the updates of pixel data, performs necessary format conversions on the received target pixel data, and transmits it to the rendering component for display.

[0070] The transmission method provided in this embodiment features low latency and lossless operation, making it suitable for local area networks or high-quality display scenarios.

[0071] Optionally, when the target pixel data is compressed pixel data, the target pixel data is transmitted to the front-end rendering layer 14 in the form of an image frame through the data channel established between the front-end rendering layer 14 and the back-end service layer 13.

[0072] Specifically, the data channel between the front-end rendering layer 14 and the back-end service layer 13 is an HTTP data transmission channel. JPEG image frames are continuously pushed through a streaming media transmission mode (e.g., multipart mode) based on the HTTP image frame stream mode (i.e., MJPEG stream mode).

[0073] Correspondingly, the MJPEG receiver in the front-end rendering layer 14 receives the encoded image stream through the standard Web protocol and uses the browser's native image decoding function to complete the display update for display on the browser side.

[0074] The transmission method provided in this embodiment significantly reduces bandwidth usage through compression, making it suitable for wide area networks or bandwidth-constrained scenarios.

[0075] Step S203: The target pixel data is mapped to the front-end display area using the front-end rendering layer 14, and the mapped data in the front-end display area is corrected according to the preset display format to obtain the target display screen.

[0076] In one possible implementation, the image rendering module in the front-end rendering layer 14 can be used to map the target pixel data to the front-end display area, and the mapping data in the front-end display area can be corrected according to the preset display format to obtain the target display image.

[0077] For example, the image rendering module can map target pixel data to the front-end display area through a browser display component, such as a Canvas.

[0078] For example, the image rendering module can modify the mapping data according to the preset display resolution and / or the display characteristics of the front-end display area.

[0079] For example, the image rendering module can adaptively scale the mapped data according to the preset display resolution and / or the display characteristics of the front-end display area to ensure the aspect ratio and clarity of the image.

[0080] In another embodiment of this application, other implementations of the remote display method based on the Linux kernel frame buffer are also provided.

[0081] In one possible implementation, in response to a user-submitted screen change request, the backend service layer 13 transmits the screen configuration instruction corresponding to the screen change request to the kernel driver layer 12, and the kernel driver layer 12 updates the pixel data in the virtual frame buffer based on the screen configuration instruction.

[0082] For example, a screen change request can be a resolution change request or a pixel bit depth change request, which can be submitted in the configuration interface of the front-end rendering layer 14.

[0083] Specifically, kernel driver layer 12 adopts a dual-device architecture, simultaneously creating and managing a standard frame buffer device (i.e., corresponding to / dev / fbX) and a dedicated control character device (i.e., corresponding to / dev / webfb). This allows the standard frame buffer device to form a pixel data channel, while the dedicated control character device forms a control and event notification channel, thereby decoupling the data path from the control path. The standard frame buffer device is provided for use by system applications, such as certain graphics applications.

[0084] Correspondingly, a dedicated control character device is used to adjust key display parameters such as resolution and pixel bit depth corresponding to the screen change request at runtime without uninstalling the kernel module, thereby realizing dynamic configuration function.

[0085] In another embodiment of this application, the module structure and specific implementation details of a remote display system 10 based on a Linux kernel frame buffer are also provided.

[0086] The following is an explanation of kernel driver layer 12: In one possible implementation, the kernel driver layer 12 completes device registration, memory allocation, and parameter initialization during the module initialization phase.

[0087] For example, the module supports multiple pixel formats, including 16-bit RGB565, 24-bit BGR, and 32-bit BGRA. In 24-bit and 32-bit modes, the pixel data adopts little-endian BGR / BGRA byte arrangement, which is beneficial for matching the data access characteristics of little-endian architecture processors and reducing format processing overhead.

[0088] In one example, when the application layer 11 performs a graphics drawing operation (e.g., rectangle filling, region copying, image bit block transfer, etc.) or writes pixel data, the kernel driver layer 12 marks the state of the virtual frame buffer as changed (i.e., marks the virtual frame buffer as modified) and triggers an update event.

[0089] That is, update events are actively triggered by the kernel driver during graphics operations or framebuffer data write paths, and are synchronized with the wait queue mechanism by marking with an internal dirty flag, so that memory-mapped (mmap) write operations can also generate perceptible update events.

[0090] In this embodiment, based on the above-mentioned kernel-triggered event-driven data synchronization mechanism, an event notification mechanism independent of the pixel data channel is constructed, enabling the backend service layer 13 to acquire data on demand when the virtual frame buffer content changes, thereby replacing the traditional fixed-frequency polling method, reducing invalid data acquisition and processing operations, reducing system resource consumption and improving real-time response.

[0091] In one possible implementation, the standard framebuffer device in the kernel driver layer 12 is registered in accordance with the operating system framebuffer subsystem specification, providing the application layer 11 with a standard access interface compatible with traditional framebuffers; wherein, the device implements a set of standard operations including opening, closing, reading, writing, memory mapping (mmap), rectangle filling, region copying, and image bit block transfer.

[0092] Specifically, the device's pixel data is stored in a page-aligned, virtual contiguous memory region (i.e., a virtual frame buffer), which is allocated through the kernel's virtual memory allocation mechanism and supports mapping to user space.

[0093] Correspondingly, the program in application layer 11 can map the buffer to the user space address through mmap to achieve direct access to pixel data, thereby constructing a zero-copy data writing path; it also supports data writing through the write system call.

[0094] Both access methods provided in this embodiment maintain the complete preservation of the original pixel format without introducing additional format conversion, thereby ensuring full compatibility with existing graphics applications.

[0095] In one possible implementation, a dedicated control character device in the kernel driver layer 12 is used to construct a control channel and a state management channel for the display system, thereby providing scalable control and event interaction capabilities in addition to the traditional frame buffer data access mechanism.

[0096] Specifically, the device provides unified control capabilities to user space through a custom ioctl interface, including device information query, update notification, dirty region marking, and dynamic configuration. Among these, device information query is used to obtain runtime parameters such as the width, height, pixel bit depth, line span, and buffer size of the virtual frame buffer; update notification is used to indicate changes in the virtual frame buffer content; dirty region marking is used to indicate the range of the changed rectangular region to support region-based local updates; and dynamic configuration is used to adjust key display parameters such as resolution and pixel bit depth at runtime without unloading kernel modules.

[0097] Secondly, regarding the event mechanism, the character device maintains frame buffer state flags on the kernel side and constructs an event notification path based on the wait queue. When the virtual frame buffer content changes or related state is updated, the kernel driver actively triggers the event and wakes up the wait queue, thereby publishing an update signal to user space.

[0098] Correspondingly, the user space can passively wait for frame buffer update events by performing a blocking poll operation on the device, thereby realizing an event-triggered data processing flow without periodically polling the frame buffer data.

[0099] The dedicated control character device provided in this embodiment forms a control and event channel independent of the pixel data read and write path in the system, realizing the decoupling of the frame buffer data channel and the control channel. While maintaining full compatibility with the standard frame buffer access interface, it introduces an event-driven mechanism oriented towards frame update semantics to the display system, which significantly improves data synchronization efficiency, reduces system resource consumption, and enhances the scalability and flexibility of the system in remote display and dynamic configuration scenarios.

[0100] The following is a description of backend service layer 13: In one possible implementation, the backend service layer 13 also includes a device management module.

[0101] For example, the device management module is used to initialize the backend service layer 13 and manage the device lifecycle; correspondingly, when the module starts, it automatically scans the frame buffer device nodes in the remote display system 10, verifies the validity of candidate devices (including resolution and pixel bit depth) through standard interfaces (e.g., ioctl to obtain parameters), and selects the first valid device as the data source.

[0102] At the same time, the device management module opens a dedicated control character device to obtain extended information, subscribe to update notifications, and issue dynamic configuration commands.

[0103] Secondly, the device management module maintains complete parameter information of the virtual frame buffer (including width, height, pixel bit depth, line span, and buffer size) and provides a thread-safe access interface to automatically update the local cache and notify relevant modules to adapt when parameters change.

[0104] In one possible implementation, the encoding conversion module of the backend service layer 13 can also dynamically adjust the compression parameters to optimize transmission efficiency according to real-time network conditions, while ensuring the visual integrity of key image information, providing a reliable data foundation for front-end display or subsequent image processing.

[0105] In one possible implementation, the WebSocket protocol stream service module of the backend service layer 13 pushes data at a preset frequency through a frame rate controller and adopts a multi-client broadcast mechanism.

[0106] For example, each connection is handled by an independent task, and data is distributed through a buffer queue to automatically discard intermediate frames when the client's processing power is insufficient, thereby avoiding blocking.

[0107] In one possible implementation, the MJPEG protocol stream service module of the backend service layer 13 also maintains an image cache to reuse the encoding results in multi-client scenarios and avoid the overhead of repeated encoding.

[0108] In one possible implementation, the backend service layer 13 also includes a configuration interface module.

[0109] For example, the configuration interface module is used to provide an HTTP-based REST interface to implement system configuration management.

[0110] Specifically, the query interface returns the current frame buffer parameters, frame rate, and client connection status; the configuration interface supports modifying the resolution, pixel bit depth, and frame rate.

[0111] In one example, when a request to change display parameters is received, the configuration is passed to the kernel driver layer 12 via the ioctl interface. The kernel driver layer 12 then reallocates the virtual frame buffer and updates the parameters accordingly, while notifying the relevant modules to complete the dynamic switch, thus achieving a no-downtime update. The frame rate adjustment is achieved by updating the timer parameters through internal control logic.

[0112] The following is an explanation of the front-end rendering layer 14: In one possible implementation, the WebSocket client of the front-end rendering layer 14 has automatic reconnection, heartbeat keep-alive, and state management mechanisms to maintain the continuity and reliability of remote display in the event of network anomalies or interruptions.

[0113] In this embodiment, the remote display system 10 achieves low-latency pixel-level remote display through the aforementioned WebSocket client, while ensuring stability under different network environments.

[0114] In one possible implementation, the MJPEG receiver of the front-end rendering layer 14 can achieve compatible rendering on different browsers and devices to ensure the continuity and consistency of remote images.

[0115] In one possible implementation, the front-end rendering layer 14 also includes a configuration interface.

[0116] For example, the configuration interface provides users with methods for interacting with the front-end rendering layer 14.

[0117] Specifically, the main functions of the configuration interface include viewing frame buffer parameters, switching screen transmission modes, adjusting display frame rate, resolution, and color depth, and supporting triggering full-screen display or scaling operations.

[0118] In this embodiment, the configuration interface interacts with the backend configuration interface to transform user operations into adjustments to the operating parameters of the remote display system, thereby enabling dynamic configuration and control of the remote display solution and meeting the visualization needs of different application scenarios.

[0119] In one possible implementation, the image rendering module in the front-end rendering layer 14 provides a unified pixel-level rendering interface through the Canvas display container to realize screen rendering under different data transmission modes, thereby achieving low latency and high consistency display.

[0120] For example, the image rendering module, in collaboration with a WebSocket client or an MJPEG receiver, ensures that the kernel frame buffer data can be completely and accurately restored on the browser side, thereby enabling closed-loop control and visual monitoring of the remote display system.

[0121] Based on the specific distribution of the remote display system 10 described above, another embodiment of this application also provides a specific operating mode of the remote display system 10.

[0122] In one possible implementation, the remote display system 10 includes a system startup phase, a steady-state operation phase, and a dynamic reconfiguration phase.

[0123] For example, during the system startup phase, the kernel driver layer 12 is first loaded into the operating system kernel. The kernel driver layer 12 completes device class creation, device number allocation, character device initialization, and framebuffer device registration, creating a standard framebuffer device and a dedicated control character device, and allocating initial framebuffer memory. Then, the backend service layer 13 process is started. The device management module of the backend service layer 13 automatically detects and opens the dedicated control character device, obtains initial device parameters, initializes the encoding conversion module and the dual-protocol stream service module, and begins listening for client connection requests. The frontend rendering layer 14 is loaded after the user accesses the backend service address through a browser, establishes a connection with the backend service layer 13, obtains current device parameters, and initializes the rendering engine.

[0124] For example, during steady-state operation, the application-layer graphics program (user program) continuously writes pixel data to the frame buffer, and the kernel driver layer 12 triggers an update notification after each write operation. Upon detecting the update notification, the frame capture module of the backend service layer 13 reads the frame buffer data, the encoding conversion module performs color format conversion and image encoding, and the WebSocket stream or MJPEG stream service module pushes the processed data to the frontend rendering layer 14. The image rendering module receives and renders the image frames, while simultaneously displaying runtime status information on the configuration interface.

[0125] For example, during the dynamic reconfiguration phase, the user submits a request to change the resolution or pixel bit depth through the configuration interface. The backend service layer 13 passes the configuration command to the kernel driver layer 12, which completes the reallocation of the frame buffer and parameter updates. After detecting the parameter changes, the backend service layer 13 and the frontend rendering layer 14 each complete their local state updates and reinitialize the rendering parameters. The system smoothly transitions without interrupting service and continues into the steady-state operation phase.

[0126] The following is for reference. Figure 3 , Figure 3 A schematic diagram of a communication device suitable for implementing embodiments of this application is shown, such as... Figure 3 As shown, the communication device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 302 or a program loaded from a storage section 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the system's operating instructions. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0127] The following components are connected to the input / output (I / O) interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the input / output (I / O) interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 310 as needed so that computer programs read from it can be installed into the storage section 308 as needed.

[0128] Specifically, according to embodiments of this application, the flowchart above refers to... Figure 2The described process can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program contains program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the system of this application.

[0129] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium compatible with computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operational instructions of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two connected blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operational instructions, or using a combination of dedicated hardware and computer instructions.

[0131] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor may be described as including a semantic extraction unit, a weight allocation unit, and a determination unit. The names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0132] On the other hand, this application also provides a computer-readable storage medium, which may be included in the communication device described in the above embodiments, or may exist independently and not assembled into the communication device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods described in this application. For example, it may execute... Figure 2 The steps of the method shown are as follows.

[0133] This application provides a computer program product including instructions that, when executed, cause the method described in this application to be performed. For example, it can execute... Figure 2 The steps of the method shown are as follows.

[0134] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A remote display method based on the Linux kernel framebuffer, characterized in that, The method is applied to a remote display system based on the Linux kernel framebuffer. The system is configured within the Linux kernel and includes a kernel driver layer, a backend service layer, a frontend rendering layer, and an application layer. The method includes: The kernel driver layer is used to create and manage standard frame buffer devices and dedicated control character devices, and the kernel driver layer maintains virtual frame buffers corresponding to the standard frame buffer devices; wherein, the standard frame buffer devices are used to form pixel data channels, and the dedicated control character devices are used to form control and event notification channels; The application layer writes pixel data to the virtual frame buffer through the standard frame buffer device, and when the write operation of the pixel data is detected, the kernel driver layer marks the change in the state of the virtual frame buffer and triggers the corresponding update event. The backend service layer obtains the update event by listening to the dedicated control character device, and after obtaining the update event, obtains the original pixel data from the virtual frame buffer; In response to a user access request, target pixel data is generated based on the original pixel data, wherein the user access request is a request initiated by the user through a browser to access the target service address; The target pixel data is transmitted to the front-end rendering layer through the data channel established between the front-end rendering layer and the back-end service layer. The target pixel data is mapped to the front-end display area using the front-end rendering layer, and the mapped data in the front-end display area is corrected according to a preset display format to obtain the target display image.

2. The remote display method based on the Linux kernel framebuffer according to claim 1, characterized in that, The step of transmitting the target pixel data to the front-end rendering layer through the data channel established between the front-end rendering layer and the back-end service layer includes: When the target pixel data is uncompressed pixel data, the target pixel data is transmitted to the front-end rendering layer in binary frame form through the data channel established between the front-end rendering layer and the back-end service layer. When the target pixel data is compressed pixel data, the target pixel data is transmitted to the front-end rendering layer in the form of image frames through the data channel established between the front-end rendering layer and the back-end service layer.

3. The remote display method based on the Linux kernel frame buffer according to claim 1, characterized in that, The step of correcting the mapping data in the front-end display area according to a preset display format includes: The mapping data is corrected based on the preset display resolution and / or the display characteristics of the front-end display area.

4. The remote display method based on the Linux kernel frame buffer according to claim 1, characterized in that, The methods for obtaining the raw pixel data include: The original pixel data is acquired periodically according to a preset frame rate; and / or Call the listener function to obtain update events. When the update event indicates that the state of the virtual frame buffer has changed, obtain the changed original pixel data.

5. The remote display method based on the Linux kernel framebuffer according to claim 4, characterized in that, The method further includes: When the application layer performs a graphics drawing operation, the kernel driver layer marks a change in the state of the virtual frame buffer, and the kernel driver layer triggers the update event.

6. The remote display method based on the Linux kernel framebuffer according to claim 1, characterized in that, The method further includes: In response to a user-submitted screen change request, the backend service layer transmits the screen configuration instruction corresponding to the screen change request to the kernel driver layer, and the kernel driver layer updates the pixel data in the virtual frame buffer based on the screen configuration instruction.

7. A remote display system based on the Linux kernel frame buffer, characterized in that, The system is set up in the Linux kernel and includes a kernel driver layer, a backend service layer, a frontend rendering layer, and an application layer. The kernel driver layer is used to create and manage standard frame buffer devices and dedicated control character devices, and maintains virtual frame buffers corresponding to the standard frame buffer devices; wherein, the standard frame buffer devices are used to form pixel data channels, and the dedicated control character devices are used to form control and event notification channels. The application layer is used to write pixel data to the virtual frame buffer through the standard frame buffer device, and when the write operation of the pixel data is detected, the kernel driver layer marks the change in the state of the virtual frame buffer and triggers the corresponding update event; The backend service layer is used to obtain the update event by listening to the dedicated control character device, and after obtaining the update event, obtain the original pixel data from the virtual frame buffer; The kernel driver layer is also used to respond to user access requests, generate target pixel data based on the original pixel data, and transmit the target pixel data to the front-end rendering layer through the data channel established between the front-end rendering layer and the back-end service layer; wherein, the user access request is a request initiated by the user through a browser to access the target service address; The front-end rendering layer is used to map the target pixel data to the front-end display area, and to correct the mapping data in the front-end display area according to a preset display format to obtain the target display image.

8. The remote display system based on the Linux kernel frame buffer according to claim 7, characterized in that, The kernel driver layer is also used for, Based on the dirty flag marking mechanism, when the state of the virtual frame buffer changes, an update event is triggered and the user space is notified.

9. A computer program product, characterized in that, The computer program product includes instructions that, when executed, cause the method as described in any one of claims 1-6 to be implemented.

Images