Video display and storage method and system based on wayland protocol and storage medium

Abstract

The invention belongs to the technical field of computers, and provides a video display and storage method and system based on a wayland protocol and a storage medium in order to solve the problems that an existing wayland video system is high in rendering delay, high in CPU load, low in storage efficiency and poor in cross-platform compatibility. Video frames are directly written into an Overlay layer through DRM-KMS and mixed by bypassing a synthesizer, meanwhile, CPU memory amplitude overhead is eliminated through zero copy transmission, an ISP-GPU-VPU direct transmission assembly line is constructed, full-link DMA-BUF direct transmission is achieved, and delay is greatly reduced; iSP preprocessing, GPU shader scaling and VPU coding are all executed by special hardware, so that the load is reduced, and the multi-path processing capability is improved; vPU dynamic code rate compression is utilized, segmented storage is carried out according to events / time, metadata is synchronized to a database, redundant storage is avoided, and space is saved.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and specifically relates to a video display and storage method, system and storage medium based on the Wayland protocol. Background Technology

[0002] In the field of video display and storage, systems based on the Wayland protocol need to simultaneously meet the requirements of high-resolution real-time display, efficient storage, and multi-scenario compatibility. However, existing technologies have significant shortcomings in terms of performance, reliability, and resource scheduling.

[0003] First, traditional application compatibility relies on the XWayland middleware layer, but this introduces significant performance losses and functional fragmentation when running X11 applications. For example, global shortcut keys become unusable, screen recording is limited, and the low-latency characteristics of the Wayland native protocol cannot be fully utilized. This compatibility compromise makes it difficult for the system to achieve efficient resource scheduling in mixed application environments (such as running traditional monitoring software and new AI analysis tools simultaneously), especially prone to rendering stutters or memory leaks in multi-tasking concurrent scenarios. Second, high-resolution video processing involves redundant operations. Existing technologies typically achieve resolution adaptation through brute-force cropping or pixel merging (e.g., dividing an 8K image into four 4K regions and then merging them). Such operations rely on repetitive computation by the CPU or GPU, increasing the load by 20%-30% and causing end-to-end latency to climb to over 50 milliseconds due to multiple memory copies. This makes it difficult to meet the demanding real-time requirements of fields such as industrial inspection and medical imaging. In addition, the video storage mechanism lacks native integration with the Wayland protocol and relies on external file systems or databases to achieve data persistence. This results in millisecond-level errors in the synchronization of metadata (such as timestamps and device status) with the video stream. When switching between master and slave nodes, it may cause loss of playback progress or screen tearing, which is particularly evident in disaster recovery and backup scenarios.

[0004] At the hardware collaboration level, existing solutions are inefficient at scheduling heterogeneous computing resources. For example, while the NPU unit of the Rockchip SoC has AI acceleration capabilities, there are bottlenecks in the data path between it and the VPU codec and GPU rendering pipeline. The MPP buffer needs to be converted multiple times (e.g., from YUV to RGB) when transferred across modules, resulting in wasted hardware computing power. At the same time, the V4L2 driver uses a static buffer allocation strategy when managing the high-bandwidth data stream of SDI cameras, which cannot dynamically adapt to changes in resolution or frame rate, easily causing buffer overflows or frame drops, especially under sudden high-load scenarios (e.g., rapid capture of moving targets) where stability is insufficient. The permission management mechanism also has serious flaws. Although the Wayland protocol improves security through client isolation, its overly strict resource access control prevents legitimate applications from directly operating the underlying hardware (e.g., bypassing the compositor to directly read and write video memory), limiting the functionality of professional video editing tools or low-latency monitoring systems. Existing permission frameworks (such as Flatpak Portal) are difficult to adapt uniformly due to differences in desktop environments, further increasing development complexity.

[0005] Regarding support for multiple monitors and dynamic layouts, existing technologies lack intelligent adaptation capabilities. Wayland's output protocol only provides basic multi-screen expansion functionality but cannot automatically optimize window layout and resolution matching based on content type (such as real-time video displayed on the main screen and analytical reports displayed on the secondary screen). Users need to manually adjust parameters, resulting in low operational efficiency. Furthermore, support for HDR (High Dynamic Range) and wide color gamut display is still in the experimental stage. The Wayland protocol lacks a standardized color space metadata transmission mechanism, leading to color information loss in the pipeline from acquisition to display, failing to meet the color accuracy requirements of professional scenarios such as film and television post-production. The storage system design also does not fully consider a distributed architecture; there is a lack of breakpoint resumption and consistency verification mechanisms between local NVMe storage and cloud synchronization, potentially causing data corruption due to network fluctuations during large-scale video stream archiving. Summary of the Invention

[0006] In order to solve at least one of the above-mentioned technical problems in the prior art, the present invention provides a video display and storage method, system and storage medium based on the Wayland protocol.

[0007] According to the first aspect, a video display and storage method based on the Wayland protocol includes the following steps: Step S1: Acquire raw RAW video streams from multiple SDI cameras via the V4L2 interface, and perform real-time preprocessing using the SOC's built-in ISP to generate standard video frames corresponding to each raw RAW video stream; the real-time preprocessing includes at least de-mosaicing, automatic exposure, and noise suppression. Step S2: Based on the GStreamer framework, an independent processing pipeline is established for each original RAW video stream, and the standard video frames are transferred to the GPU and VPU with zero copy through the DMA-BUF shared memory mechanism; wherein, the GPU performs format conversion and dynamic resolution scaling through shaders, and the VPU performs hardware encoding through the encoding engine; Step S3: Convert the standard video frame processed above into a wl_buffer supported by the Wayland protocol through waylandsink, and submit it to the Wayland compositor to define dynamic rendering parameters. Then, write the standard video frame directly into the Overlay layer of the display controller through the drmModeSetPlane function of the DRM-KMS interface. The dynamic rendering parameters are the rendering area coordinates and the layer level. Step S4: The processed standard video frames are dynamically bitrate compressed using the VPU and written directly to the NVMe storage device via the PCIe NVMe interface. At the same time, video metadata is stored in real time via SQLite. The dynamic bitrate compression adjusts the quantization parameters in real time according to the scene complexity and uses a segmented storage strategy to generate independent storage files.

[0008] Preferably, step S1 involves acquiring raw RAW video streams from multiple SDI cameras via the V4L2 interface, including: The SDI camera outputs a YUV422 10-bit uncompressed video stream through the 3G-SDI interface, which is then connected to the Rockchip SoC's sditodvp driver interface via the SDI-DVP converter and the sdi-dvp interface. The V4L2 interface driver calls ioctl to detect the device node, configures the SDI camera resolution and pixel format, requests a DMA-BUF buffer, and maps the acquired raw RAW video stream to user space.

[0009] Preferably, step S2 establishes an independent processing pipeline for each original RAW video stream based on the GStreamer framework, including: The ISP output data is written directly to the GPU memory via the DMA engine, and the GPU performs format conversion and dynamic resolution adaptation. The video frames processed by the GPU are hardware encoded by the VPU, and the hardware encoded data is directly written to the storage device via the PCIe NVMe interface.

[0010] Preferably, step S3 further includes: The Wayland compositor enables the zwp_linux_dmabuf_v1 extended protocol and extends the render-rectangle parameter in the Wayland protocol to dynamically specify the video display area. The z-axis layer level is set by the layer parameter, and the zpos property of DRM is used to control the stacking order of multiple layers.

[0011] Preferably, step S4 further includes: The system determines whether the storage link is congested. When the storage link is congested, non-reference frames are discarded first, key frames are retained, and the video metadata offset is recorded. At the same time, the display frame rate is dynamically adjusted through the QoS mechanism.

[0012] Preferably, it further includes: The timestamps of the SDI camera, encoder, and storage device are aligned using the PTP protocol.

[0013] According to a second aspect, a video display and storage system based on the Wayland protocol is capable of executing a video display and storage method based on the Wayland protocol as described in the first aspect and any preferred embodiment, including: The video acquisition and preprocessing module is used to connect multiple SDI cameras and use the SoC's built-in ISP to perform real-time preprocessing of the acquired raw RAW video streams, while calling the GPU for format conversion and dynamic resolution scaling; it also uses the DMA-BUF shared memory mechanism to achieve zero-copy direct data transmission across the entire link between the ISP, GPU, and VPU. The Wayland pass-through rendering engine is used to extend the render-rectangle and layer parameters in the Wayland compositor. It writes standard video frames directly to the overlay layer of the display controller via the DRM / KMS interface and uses the zpos property of DRM to control the stacking order of multiple layers. The intelligent dynamic storage module is used to dynamically adjust the bitrate using the VPU's encoding engine and dynamically adjust the quantization parameters according to the complexity of the scene; it adopts a segmented storage strategy to generate independent storage files based on event triggers or time thresholds; it directly writes to the storage device through the PCIe NVMe interface and synchronously stores video metadata through SQLite; The multi-channel parallel processing architecture is used to create independent processing pipelines for each raw RAW video stream based on the GStreamer framework, and manage resource allocation through a thread pool; it utilizes the DRM's Plane allocation strategy and the VPU's multi-instance encoding capabilities to achieve hardware-level isolation between display and encoding resources.

[0014] Preferably, it further includes: The real-time and reliability assurance module is used to align the timestamps of the SDI camera, encoding engine and storage device using the PTP protocol; and when the storage link is congested, non-reference frames are discarded first and key frames are retained, while the display frame rate is dynamically adjusted through the QoS mechanism.

[0015] Thirdly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute a video display and storage method based on the Wayland protocol as described in the first aspect or any corresponding embodiment.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention achieves pass-through rendering by extending the Wayland protocol, bypassing compositor mixing calculations and directly writing video frames to the display controller's Overlay layer via the drmModeSetPlane function. Simultaneously, zero-copy transmission eliminates CPU memory overhead, enabling end-to-end DMA-BUF direct transmission from ISP to GPU to VPU, significantly reducing latency. ISP preprocessing, GPU shader scaling, and VPU encoding are all performed by dedicated hardware, reducing workload and improving multi-processing capabilities. Furthermore, quantization parameters are adjusted in real-time based on scene complexity, and independent storage files are generated based on event triggers or time thresholds, avoiding redundant storage and saving space. This solution ensures video decodeability in extreme scenarios through keyframe retention and timestamp synchronization. Moreover, this invention's solution shields hardware differences through the HAL layer, improving cross-platform compatibility and reducing porting costs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a video display and storage method based on the Wayland protocol provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a video display and storage system based on the Wayland protocol provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that the structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should fall within the scope of the technical content disclosed in the present invention. It should be noted that in this specification, relational terms such as "first" and "second" are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0021] In this embodiment of the invention, the following is a list of definitions for abbreviations, English terms, and key terms: 1. CPU, Central Processing Unit; 2. GPU, Graphics Processing Unit; 3. SOC, System on a Chip; 4. VPU, Video Processing Unit; 5. ISP, Image Signal Processor; 6. DRM, Direct Rendering Manager; KMS, Kernel Mode-setting; 7. QoS, Quality of Service, a mechanism that dynamically adjusts the display frame rate.

[0022] like Figure 1 As shown in the figure, this embodiment of the invention provides a flowchart of a video display and storage method based on the Wayland protocol, including the following steps: Step S1: Acquire raw RAW video streams from multiple SDI cameras via the V4L2 interface, and perform real-time preprocessing using the SOC's built-in ISP to generate standard video frames corresponding to each raw RAW video stream; the real-time preprocessing includes at least de-mosaicing, automatic exposure, and noise suppression.

[0023] Optionally, in step S1, raw RAW video streams are acquired from multiple SDI cameras via the V4L2 interface, including: the SDI cameras output YUV422 10-bit uncompressed video streams via the 3G-SDI interface, which are then connected to the Rockchip SoC's sditodvp driver interface via the SDI-DVP converter and the sdi-dvp interface; the V4L2 interface driver calls ioctl to detect device nodes, configures the SDI camera resolution and pixel format, requests a DMA-BUF buffer, and maps the acquired raw RAW video streams to user space.

[0024] Step S2: Based on the GStreamer framework, an independent processing pipeline is established for each original RAW video stream, and the standard video frames are transferred to the GPU and VPU with zero copies through the DMA-BUF shared memory mechanism; wherein, the GPU performs format conversion and dynamic resolution scaling through shaders, and the VPU performs hardware encoding through the encoding engine.

[0025] Optionally, an independent processing pipeline is established for each raw RAW video stream based on the GStreamer framework, including: ISP output data is written directly to GPU memory via DMA engine, and the GPU performs format conversion and dynamic resolution adaptation; the video frames processed by the GPU are hardware encoded via VPU, and the hardware encoded data is written directly to the storage device via PCIe NVMe interface.

[0026] Step S3: Convert the standard video frame processed above into a wl_buffer supported by the Wayland protocol through waylandsink, and submit it to the Wayland compositor to define dynamic rendering parameters. Then, write the standard video frame directly into the Overlay layer of the display controller through the drmModeSetPlane function of the DRM-KMS interface. The dynamic rendering parameters are the rendering area coordinates and the layer level.

[0027] Optionally, it also includes: the Wayland compositor enables the zwp_linux_dmabuf_v1 extended protocol, and extends the render-rectangle parameter in the Wayland protocol to dynamically specify the video display area; the z-axis level is set through the layer parameter, and the zpos attribute of the DRM is used to control the stacking order of multiple layers.

[0028] Step S4: Use VPU to dynamically compress the processed standard video frames and write them to the NVMe storage device, while storing video metadata in real time via SQLite.

[0029] Optionally, it also includes: determining whether the storage link is congested; when the storage link is congested, prioritizing the discarding of non-reference frames, retaining key frames and recording video metadata offsets, and dynamically adjusting the display frame rate through a QoS mechanism.

[0030] Optionally, it also includes: aligning the timestamps of the SDI camera, encoding engine, and storage device using the PTP protocol.

[0031] like Figure 2 As shown, this embodiment of the invention also provides a video display and storage system based on the Wayland protocol, including: The video acquisition and preprocessing module is used to connect multiple SDI cameras and use the SoC's built-in ISP to perform real-time preprocessing of the acquired raw RAW video streams, while calling the GPU for format conversion and dynamic resolution scaling; it also uses the DMA-BUF shared memory mechanism to achieve zero-copy direct data transmission across the entire link between the ISP, GPU, and VPU. The Wayland pass-through rendering engine is used to extend the render-rectangle and layer parameters in the Wayland compositor. It writes standard video frames directly to the overlay layer of the display controller via the DRM-KMS interface and uses the zpos property of DRM to control the stacking order of multiple layers. The intelligent dynamic storage module is used to dynamically adjust the bitrate using the VPU's encoding engine and dynamically adjust the quantization parameters according to the complexity of the scene; it adopts a segmented storage strategy to generate independent storage files based on event triggers or time thresholds; it directly writes to the storage device through the PCIe NVMe interface and synchronously stores video metadata through SQLite; The multi-channel parallel processing architecture is used to create independent processing pipelines for each raw RAW video stream based on the GStreamer framework, and manage resource allocation through a thread pool; it utilizes the DRM's Plane allocation strategy and the VPU's multi-instance encoding capabilities to achieve hardware-level isolation between display and encoding resources.

[0032] Optionally, it also includes: a real-time and reliability assurance module, used to align the timestamps of the SDI camera, encoder and storage device using the PTP protocol; and when the storage link is congested, non-reference frames are discarded first and key frames are retained, while the display frame rate is dynamically adjusted through the QoS mechanism.

[0033] This invention aims to achieve low-latency video display and efficient storage using the Wayland protocol. The core hardware of the Wayland-based video display and storage system includes an SDI camera, a Rockchip SoC, a GPU, and NVMe storage devices. The software consists of a V4L2 driver layer, a Rockchip MPP, a GStreamer pipeline, a Wayland compositor, and a storage system. This invention uses the V4L2 driver layer to capture raw RAW video streams from an SDI camera, utilizes the Rockchip MPP for hardware-accelerated encoding and decoding, and then leverages the GStreamer framework to establish independent processing pipelines for each raw RAW video stream. Finally, the Wayland compositor enables display and storage on a local machine or in the cloud. This system supports real-time processing of high-resolution video streams (e.g., 4K@30fps) and reduces CPU load through hardware acceleration.

[0034] In this embodiment of the invention, the SDI camera of the hardware system outputs an uncompressed YUV422 10-bit raw RAW video stream through a 3G-SDI interface, which is then connected to the Rockchip SoC's sditodvp driver interface via an SDI-to-DVP converter. The Rockchip SoC is responsible for hardware encoding / decoding and AI analysis of the raw RAW video stream. Its VPU supports H.265 encoding, and its GPU (such as Mali-G52 or Mali-G610) accelerates graphics rendering to ensure smooth synthesis by the Wayland compositor. The NVMe storage device provides high-speed write capability, with sequential write speeds up to 1.5GB / s, and is used to store encoded video metadata. All hardware modules are interconnected through PCIe, DVP, and other interfaces, forming a complete link from acquisition to processing, display, and storage.

[0035] In this embodiment of the invention, the V4L2 (Video 4 Linux 2, the driver framework for video capture devices) driver in the software part is responsible for initializing the SDI camera, configuring the resolution and pixel format, and mapping the buffer of the original RAW video stream to user space through the DMA-BUF shared memory mechanism. The Rockchip MPP (Media Process Platform) uses a hardware codec to convert the original RAW video stream to H.265 format, while managing video memory through DRM-KMS to ensure efficient buffer sharing. The GStreamer framework, as an intermediate layer, constructs an independent processing pipeline of "capture-encoding-display / storage". It acquires the original RAW video stream through V4L2, performs hardware encoding using the hardware-accelerated H.264 encoding plugin MPP h264enc, and then distributes it to the display (waylandsink) and storage (filesink) paths through the tee element within the GStreamer framework. The Wayland compositor receives the buffer from the client, synthesizes it, and outputs it to the display through DRM / KMS, supporting zero-copy rendering to reduce latency. The storage system records video metadata (such as timestamps and resolution) using SQLite and encapsulates the encoded stream into MP4 or MKV format before writing it to the NVMe device.

[0036] During the video capture phase, after the SDI camera is powered on, it transmits raw data to the SoC via the DVP interface. The V4L2 driver uses ioctl to configure the device and start video stream capture, simultaneously requesting a DMA-BUF buffer and mapping it to user space. The Rockchip MPP obtains the buffer from the V4L2 and calls the VPU for H.265 encoding. The encoded standard video frame data is then distributed through the tee element of the GStreamer framework. In the display path, the video output plugin waylandsink in the GStreamer framework converts the encoded standard video frames into wl_buffer, a core graphics buffer mechanism supported by the Wayland protocol, and submits it to the wl_surface interface of the Wayland compositor. The GPU renders the composite image using OpenGLES or Vulkan and outputs it to the display. In the storage path, the encoded stream is parsed using the GStreamer framework's parsing and encapsulation tool h264parse, encapsulated into MP4 format using the GStreamer framework's QuickTime format multiplexer qtmux, and written to NVMe storage in non-blocking mode using the basic output plugin filesink in the GStreamer multimedia framework, avoiding pipeline blockage. Metadata is synchronized to the database in real time through a background service to ensure consistency between playback progress and file information.

[0037] In this embodiment of the invention, the video display and storage system based on the Wayland protocol achieves zero-copy data transmission through DMA-BUF. V4L2, RockchipMPP, and the Wayland compositor share physical memory, eliminating CPU copy overhead. The Wayland compositor enables the zwp_linux_dmabuf_v1 extended protocol to directly receive data transmitted via the client's DMA-BUF shared memory mechanism, reducing context switching. Simultaneously, to address hardware bottlenecks, RockchipMPP employs CBR bitrate control and dynamic GOP length adjustment to avoid overload; GPU rendering uses the Vulkan API to reduce driver overhead. The storage system improves throughput through RAID arrays or distributed storage and dynamically adjusts the bitrate based on AI analysis results, balancing image quality and storage costs.

[0038] In this embodiment of the invention, the hardware support in the video acquisition and preprocessing module is as follows: multiple SDI cameras are connected via the SDI-DVP interface, and the ISP (Image Signal Processor) built into the SoC is used to complete real-time processing of the raw RAW video stream for de-mosaicing, automatic exposure, and noise suppression; the software logic in the video acquisition and preprocessing module is as follows: the V4L2 plugin based on the GStreamer framework captures the raw RAW video stream, and calls the GPU shader to perform format conversion (NV12 to RGBA) and dynamic resolution adaptation (supporting real-time scaling from 4K to 1080P); the zero-copy transmission of the video acquisition and preprocessing module is achieved through the DMA-BUF shared memory mechanism, realizing direct data transmission of the entire link from ISP to GPU to VPU, avoiding CPU intervention and memory copying.

[0039] In this embodiment of the invention, the protocol extension in the Wayland pass-through rendering engine is as follows: the render-rectangle and layer parameters in the Wayland compositor (Weston) are extended to support dynamically defining the display area of ​​the video.<x,y,width,height> ) and layer hierarchy (Z-order); Hardware passthrough in Wayland passthrough rendering engine: Standard video frames are directly written to the overlay layer of the display controller (RockchipVOP's overlayPlane) via the DRM / KMS interface (drmModeSetPlane), bypassing unnecessary compositing calculations, with rendering latency ≤8ms; Multi-layer management in Wayland passthrough rendering engine: Supports the allocation of independent layers for each independent processing pipeline (e.g., 6 video streams can occupy 6 overlayPlanes), and controls the overlay order through the zpos attribute to ensure that critical images (alarm information in standard video frames) are always on the top layer.

[0040] In this embodiment of the invention, by deeply integrating the Wayland protocol with the display controller, the multi-layered compositing process in the traditional display architecture is completely eliminated, enabling direct rendering and precise control of video frames. Traditional solutions rely on the X11 protocol or general compositors, requiring multiple data copies and mixing calculations between the application layer buffer, the compositor buffer, and the display controller, resulting in high latency and resource waste. This invention extends the functionality of the Wayland protocol by introducing a dynamic rendering parameter configuration mechanism into the Waylandsink plugin. This includes: Region rendering control: defining the video display position (e.g., coordinates <800,0>) and size (e.g., 800x600) via the render-rectangle parameter, directly calling the drmModeSetPlane function of the DRM-KMS interface to write the framebuffer of key video frames into the overlay layer of the display controller (e.g., RockchipVOP's OverlayPlane), bypassing the compositor's pixel blending operation; Layer hierarchy management: specifying the Z-axis order of layers via the layer parameter (e.g., layer=0 indicates the bottom layer), and using the DRM's zpos attribute to control the overlay relationship of multiple video streams, ensuring that critical images (e.g., alarm information) are always on the top layer.

[0041] This invention significantly reduces CPU load and memory bandwidth usage through the collaborative work of end-to-end hardware modules (ISP, GPU, VPU) and zero-copy data transmission. Specific implementations include: ISP preprocessing and DMA direct transfer: The raw RAW video stream captured by the SDI camera undergoes de-mosaicing, automatic white balance, and noise suppression by the ISP hardware module, and is then directly written to the GPU memory area via the DMA engine, avoiding CPU intervention. In the experiment, the ISP2.0 module with the Rockchip RK3588 chip can process four 4K@30fps data streams in parallel, with each stream consuming less than 200mW.

[0042] GPU shader acceleration: In the video preprocessing stage, the GPU shader is used for format conversion (such as YUV to RGBA) and dynamic resolution scaling. Hardware acceleration units are called through OpenCL or Vulkan API, which improves processing efficiency by more than 5 times compared to pure CPU solutions.

[0043] VPU Hardware Encoding and Direct Storage Writing: Rendered standard video frames undergo dynamic bitrate compression via a VPU (such as an H.265 encoder), and the encoded data is directly written to an NVMe storage device through a PCIe NVMe interface, bypassing system memory. Experimental data shows that encoding and storing four 1080P video streams consumes only 3% of CPU resources, reducing CPU load by 85% compared to software encoding schemes (FFmpeg).

[0044] In this embodiment of the invention, the hardware encoding acceleration of the intelligent dynamic storage module is achieved by using the H.265 / H.264 encoding engine of the VPU (Video Processing Unit) for dynamic bitrate adjustment (VBR / CVBR), while adjusting the QP (Quantization Parameter) in real time according to the scene complexity (motion intensity, texture details, etc.), with a bitrate fluctuation range of up to 2Mbps~8Mbps; the software-level segmented storage strategy of the intelligent dynamic storage module is to generate independent storage files based on event triggers or time thresholds (such as generating an MP4 segment every 10 minutes or for each event), supporting fast retrieval and playback.

[0045] In this embodiment of the invention, the pipeline software processing instantiation of the multi-channel parallel processing architecture is as follows: an independent GStreamer pipeline is created for each video stream, and resource allocation is managed through a thread pool to support real-time processing of 8 channels of 4K@30fps video; the hardware resource isolation of the multi-channel parallel processing architecture is as follows: by utilizing the DRM's plane allocation strategy and the VPU's multi-instance encoding capability, it is ensured that each video has exclusive access to the display layer and encoding engine, avoiding resource contention.

[0046] In this embodiment of the invention, the specific functional execution steps of the software in the video display and storage system based on the Wayland protocol are as follows: V4L2 driver loading: Detect the SDI camera device node ( / dev / video0) via ioctl(VIDIOC_QUERYCAP), and configure the resolution (1920x1080), pixel format (YUV42210bit), and DMA-BUF buffer mode (V4L2_MEMORY_DMABUF).

[0047] Rockchip MPP codec initialization: Call mpp_create() to create the codec context, bind the VPU hardware unit, and set the H.265 encoding parameters (GOP=60, CBR bitrate 50Mbps).

[0048] Wayland synthesizer startup: Loads the zwp_linux_dmabuf_v1 extended protocol, initializes the DRM / KMS display backend, and creates the wl_display global object.

[0049] An independent processing pipeline based on the GStreamer framework is constructed: dynamically assembling the V4L2→mpp h264 enc→tee component pipeline, and selecting either waylandsink or xvimagesink as the display output according to hardware capabilities.

[0050] Hardware integration in a video display and storage system based on the Wayland protocol: SDI camera: After power-on, it outputs raw RAW video stream through the sdi-dvp interface, triggering the SoC's ISP (Image Signal Processor) to perform real-time preprocessing such as de-mosaicing, automatic exposure, and noise suppression.

[0051] Rockchip SoC: VPU loads H.265 firmware, NPU preloads AI models (such as super-resolution networks), GPU initializes Vulkan context, and memory manager pre-allocates DMA-BUF pool (4GB in size).

[0052] NVMe storage: Format the file system (EXT4 / ZFS), mount it to the / mnt / nvme path, and enable DirectI / O mode to bypass page caching.

[0053] To support parallel processing of multiple high-definition video streams, this invention designs a modular pipeline and a hardware resource isolation mechanism: Independent pipeline instantiation: An independent processing pipeline is created for each video stream, and resource allocation is managed through a thread pool. In an 8-channel video processing scenario, the system allocates independent v4l2src, waylandsink, and vaapiencode_h265 instances to each channel to avoid resource contention.

[0054] Display and encoding resource isolation: By using the DRM's plane allocation strategy, a dedicated overlay layer is bound to the video frames of each independent pipeline (such as Rockchip VOP supporting 6 independent layers) to ensure that the displays do not interfere with each other; at the same time, the multi-instance encoding capability of VPU is utilized (such as RK3588 supporting 4 channels of H.265 concurrent encoding) to achieve hardware-level isolation of encoding tasks.

[0055] Dynamic load balancing: When a video stream exceeds resource limits due to increased resolution or bitrate, the system automatically reduces the processing priority of non-critical streams (such as reducing the rendering area or lowering the frame rate) to ensure the real-time performance of core services.

[0056] In this embodiment of the invention, a Hardware Abstraction Layer (HAL) and protocol extension plugins are constructed to support deployment across chip platforms and operating systems, specifically including: Hardware Abstraction Interface: Defines a unified device control interface (such as set_render_rect(x,y,w,h) and set_layer(z)) to adapt to display controllers and encoding engines on different hardware platforms (such as Rockchip and NXP.MX8). For example, DCSS_OverlaySetPosition is called on the NXP platform, while DRM_PlaneSetRect is called on the Rockchip platform.

[0057] Wayland Protocol Extension: Develop a customized rkwaylandsink plugin to support Rockchip-specific layer attributes (such as rotation and transparency), and implement the DMA-BUF shared memory mechanism through the zwp_linux_dmabuf_v1 extension of the Wayland protocol to ensure compatibility with different synthesizers (such as Weston and KWin).

[0058] Operating system adaptation layer: Encapsulates the Linux DRM interface, Android SurfaceFlinger interface, and RTOS display driver, enabling the same application logic to run across platforms and reducing porting costs.

[0059] In this embodiment of the invention, for high real-time scenarios, a timestamp synchronization and fault-tolerance mechanism is proposed to ensure the stability of the display and storage links, specifically including: Global clock synchronization: The PTP (Precision Time Protocol) is used to align the timestamps of the SDI camera, encoder and NVMe storage device, keeping the error within 1ms and ensuring frame-level synchronization of multiple video streams.

[0060] Dynamic frame dropping strategy: When the storage link is congested (such as disk write latency exceeding the threshold), the system prioritizes dropping non-reference frames (B-frames / P-frames) and retains key frames (I-frames) to maintain video decodeability. At the same time, the display frame rate is dynamically adjusted through the QoS mechanism to avoid screen stuttering.

[0061] In another optional embodiment of the present invention, an optimized compositor based on the X11 protocol can also be selected. By improving the X11 compositor (such as using GPU-accelerated Mutter or Compton), the number of compositing layers can be reduced, or OpenGL direct rendering technology can be used to reduce display latency. However, in the experiment, the global locking mechanism and synchronization model of X11 make it impossible to completely eliminate intermediate layer data copying. For example, the client application needs to submit the rendering result to the XServer, and then the compositor mixes and outputs it. The compositing latency of multiple video streams is still higher than 30ms (tested in a 4K@60fps scene); the pixel blending operation of X11 requires a lot of GPU resources. The compositing calculation of 4 1080P video streams may cause the GPU utilization to exceed 70%, while the Wayland pass-through solution of the present invention only uses 20%-30%; although X11 is widely supported in the traditional Linux ecosystem, its protocol complexity is high (such as the XEXT extension) and its security is poor (such as the MIT-SHM shared memory vulnerability), making it difficult to adapt to the lightweight requirements of modern embedded devices.

[0062] In another optional embodiment of the present invention, the Android SurfaceFlinger compositor can be ported to a Linux system, utilizing its BufferQueue mechanism to directly manage layer overlays. However, SurfaceFlinger heavily relies on the AndroidHAL layer and Gralloc memory manager, and porting it to non-Android systems requires rewriting the hardware abstraction interface, resulting in extremely high development costs. For example, on the Rockchip platform, hwcomposer.rockchip.so needs to be adapted to support DRM / KMS, a workload far exceeding that of Wayland plugin development; SurfaceFlinger's display control interfaces (such as setLayer(), setPosition()) cannot flexibly support dynamic rendering area adjustments (such as dynamic resolution adaptation of render-rectangle), making it difficult to meet the multi-window layout requirements of industrial scenarios.

[0063] In another optional embodiment of the present invention, Wayland / X11 can be bypassed, and a private display protocol can be developed directly based on DRM / KMS to achieve zero-compositing layer rendering. However, private protocols require building their own application ecosystem, and mainstream GUI frameworks (such as Qt, GTK) and third-party applications (such as VLC, Chrome) are not directly compatible, making commercial promotion extremely difficult. For example, developers need to rewrite the window manager and input processing logic for private protocols, and the porting cost far exceeds the benefits of the protocol; the underlying interface of DRM / KMS changes frequently with kernel versions (such as the evolution of AtomicAPI), and private protocols need to continuously adapt to kernel and driver updates, making long-term maintenance costs unbearable; private protocols are usually customized for specific hardware (such as only supporting RockchipVOP), making cross-platform deployment difficult (such as NXPi.MX8 or Intel GPU), lacking the flexibility brought by the hardware abstraction layer design of the present invention.

[0064] The core of this invention lies in optimizing the entire video processing chain through deep collaboration between the Wayland protocol's intermediate-layer-free rendering architecture and the hardware codec engine. It offers the following advantages: Significantly reduced display latency: Wayland's pass-through rendering engine avoids the compositor overhead of X11, reducing real-world display latency from 50ms to less than 10ms compared to traditional solutions. For example, in 4K@60fps scenes, frame submission time is reduced by 80%.

[0065] Resource optimization: Hardware-accelerated preprocessing and encoding reduce CPU load from 80% to below 15%, and GPU utilization is stabilized at 30%-40%, supporting simultaneous processing of 8 1080P video streams.

[0066] Improved storage efficiency: Dynamic bitrate adjustment technology reduces storage space usage by 40%-60%, especially in monitoring scenarios, where the storage volume of static images can be compressed to 20% of the fixed bitrate.

[0067] Enhanced system compatibility: Supports various synthesizers (such as Weston and KWin) and hardware platforms (such as Rockchip and NXPi.MX8) that conform to the Wayland protocol, and is compatible with operating systems such as Linux and Android.

[0068] In this embodiment, a video display and storage system based on the Wayland protocol is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0069] This invention also provides a computer device having the above-described features. Figure 2 The video display and storage system shown is based on the Wayland protocol.

[0070] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 3 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0071] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0072] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0073] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0074] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0075] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0076] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0077] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0078] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A video display and storage method based on the Wayland protocol, characterized in that, Includes the following steps: Step S1: Acquire raw RAW video streams from multiple SDI cameras via the V4L2 interface, and perform real-time preprocessing using the SOC's built-in ISP to generate standard video frames corresponding to each raw RAW video stream; the real-time preprocessing includes at least de-mosaicing, automatic exposure, and noise suppression. Step S2: Based on the GStreamer framework, an independent processing pipeline is established for each original RAW video stream, and the standard video frames are transferred to the GPU and VPU with zero copy through the DMA-BUF shared memory mechanism; wherein, the GPU performs format conversion and dynamic resolution scaling through shaders, and the VPU performs hardware encoding through the encoding engine; Step S3: Convert the standard video frame processed above into a wl_buffer supported by the Wayland protocol through waylandsink, and submit it to the Wayland compositor to define dynamic rendering parameters. Then, write the standard video frame directly into the Overlay layer of the display controller through the drmModeSetPlane function of the DRM-KMS interface. The dynamic rendering parameters are the rendering area coordinates and the layer level. Step S4: The processed standard video frames are dynamically bitrate compressed using the VPU and written directly to the NVMe storage device via the PCIe NVMe interface. At the same time, video metadata is stored in real time via SQLite. The dynamic bitrate compression adjusts the quantization parameters in real time according to the scene complexity and uses a segmented storage strategy to generate independent storage files.

2. The video display and storage method based on the Wayland protocol according to claim 1, characterized in that, Step S1 involves acquiring raw RAW video streams from multiple SDI cameras via the V4L2 interface, including: The SDI camera outputs a YUV422 10-bit uncompressed video stream through the 3G-SDI interface, which is then connected to the Rockchip SoC's sditodvp driver interface via the SDI-DVP converter and the sdi-dvp interface. The V4L2 interface driver calls ioctl to detect the device node, configures the SDI camera resolution and pixel format, requests a DMA-BUF buffer, and maps the acquired raw RAW video stream to user space.

3. The video display and storage method based on the Wayland protocol according to claim 1, characterized in that, After establishing an independent processing pipeline for each raw RAW video stream based on the GStreamer framework in step S2, the following steps are also included: The ISP output data is written directly to the GPU memory via the DMA engine, and the GPU performs format conversion and dynamic resolution adaptation. The video frames processed by the GPU are hardware encoded by the VPU, and the hardware encoded data is directly written to the storage device via the PCIe NVMe interface.

4. The video display and storage method based on the Wayland protocol according to claim 1, characterized in that, Step S3 also includes: The Wayland compositor enables the zwp_linux_dmabuf_v1 extended protocol and extends the render-rectangle parameter in the Wayland protocol to dynamically specify the video display area. The z-axis layer level is set by the layer parameter, and the zpos property of DRM is used to control the stacking order of multiple layers.

5. A video display and storage method based on the Wayland protocol according to claim 1, characterized in that, Step S4 also includes: The system determines whether the storage link is congested. When the storage link is congested, non-reference frames are discarded first, key frames are retained, and the video metadata offset is recorded. At the same time, the display frame rate is dynamically adjusted through the QoS mechanism.

6. A video display and storage method based on the Wayland protocol according to claim 1, characterized in that, Also includes: The timestamps of the SDI camera, encoding engine, and storage device are aligned using the PTP protocol.

7. A video display and storage system based on the Wayland protocol, characterized in that, A video display and storage method based on the Wayland protocol, capable of executing any one of claims 1-6, includes: The video acquisition and preprocessing module is used to connect multiple SDI cameras and use the SoC's built-in ISP to perform real-time preprocessing of the acquired raw RAW video streams, while calling the GPU for format conversion and dynamic resolution scaling; it also uses the DMA-BUF shared memory mechanism to achieve zero-copy direct data transmission across the entire link between the ISP, GPU, and VPU. The Wayland pass-through rendering engine is used to extend the render-rectangle and layer parameters in the Wayland compositor. It writes standard video frames directly to the overlay layer of the display controller via the DRM-KMS interface and uses the zpos property of DRM to control the stacking order of multiple layers. The intelligent dynamic storage module is used to dynamically adjust the bitrate using the VPU's encoding engine and dynamically adjust the quantization parameters according to the complexity of the scene; it adopts a segmented storage strategy to generate independent storage files based on event triggers or time thresholds; it directly writes to the storage device through the PCIeNVMe interface and synchronously stores video metadata through SQLite; The multi-channel parallel processing architecture is used to create independent processing pipelines for each raw RAW video stream based on the GStreamer framework, and manage resource allocation through a thread pool; it utilizes the DRM's Plane allocation strategy and the VPU's multi-instance encoding capabilities to achieve hardware-level isolation between display and encoding resources.

8. A video display and storage system based on the Wayland protocol according to claim 7, characterized in that, Also includes: The real-time performance and reliability assurance module is used to align the timestamps of the SDI camera, encoder and storage device using the PTP protocol; and when the storage link is congested, non-reference frames are discarded first and key frames are retained, while the display frame rate is dynamically adjusted through the QoS mechanism.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images