Embeddable access 8k video frame pixel by pixel recombination system and method thereof

Abstract

This application discloses an embedded access 8K video frame pixel-by-pixel reassembly system and method, comprising: an embedded access and preprocessing module, which receives the raw 8K video signal and converts it into an IP video stream through a standard physical interface without changing the front-end video processing system; a network switching module, which encapsulates the pixel blocks and their source coordinates of the video frame into independent data packets and routes them to the target processing node according to a dynamically loadable pixel rearrangement mapping table; a rearrangement module, which reassembles the pixel data according to the mapping table to generate rearranged video frames; and a synchronization and control module, which provides a unified clock reference for each module and coordinates the processing timing. This invention serves as a transparent adaptation layer between the front-end standard video output and the back-end customized display input, solving the problem that existing technologies cannot perform real-time pixel-by-pixel rearrangement of 8K video frames without modifying the front-end video processing system.

Description

Technical Field

[0001] This invention relates to the field of video signal processing technology, and more specifically to an embedded 8K video frame pixel-by-pixel recombination system and method thereof. Background Technology

[0002] With the rapid development of ultra-high-definition video technology, 8K resolution (7680×4320 pixels, approximately 33 million pixels / frame) has become the core standard for next-generation display and content production. Its massive data volume poses unprecedented challenges to the entire video signal acquisition, processing, transmission, and display chain. To adapt to the input format requirements of different display devices (such as irregularly shaped LED splicing screens and multi-screen display walls) or to optimize transmission bandwidth, it is often necessary to rearrange and swap the pixels of video frames. However, existing technical solutions often suffer from high system coupling, high modification costs, or insufficient flexibility when achieving this goal.

[0003] 8K video processing involves multiple factors such as resolution, frame rate, color depth, and dynamic range (HDR). Any pixel-level operation must be performed while ensuring visual quality. Currently, technologies related to pixel rearrangement are mainly distributed in the fields of video compression, post-processing enhancement, and professional audio and video systems.

[0004] To reduce the transmission bandwidth of 8K video, pixel-domain processing based on compressed sensing and reconstruction algorithms is applied to the encoding end, merging sampling and compression and transferring complex computations to the decoding and reconstruction end. Related algorithms, such as block-based compressed sensing and reconstruction based on joint spatiotemporal features (e.g., adding spatiotemporal autoregression and multiple hypothesis regularization terms to the minimum total variation model), reconstruct high-resolution images at the decoding end using intra- and inter-frame correlations through prediction-residual methods. These methods essentially involve the reorganization and recovery of sampled data (which can be considered a subset of irregular pixels), but their processing flow is deeply embedded in the codec. The aim is to recover the image from compressed or undersampled data, rather than performing flexible, display-oriented pixel rearrangement on the decoded complete video frames.

[0005] In the field of video post-processing, artificial intelligence-based methods are used for image quality enhancement. For example, 3D convolutional neural networks (3D-CNNs) are used to enhance the quality of compressed videos, or sub-pixel convolutional layers are used in super-resolution networks (such as SD-VSRnet) to rearrange information in the channel dimension to a high-resolution spatial dimension. These techniques do achieve pixel data rearrangement and upsampling, but their goal is to improve the subjective or objective quality of the image (such as PSNR), and they usually exist as independent, computationally intensive post-processing modules. It is difficult to flexibly "embed" them to achieve only pixel position swapping without changing the existing processing pipeline.

[0006] Dedicated pixel switching systems. In the professional audio-visual field, there exist systems like PixelNet based on packet switching technology. These systems decompose video signals into pixel packets, which are then exchanged over a network, theoretically allowing any source pixel block to be displayed in any target display area. This approach offers high flexibility and lossless signal transmission capabilities. However, such systems are typically independent, closed, and dedicated, requiring dedicated input / output nodes and switching networks. Integrating them into existing video processing workflows often means replacing or significantly modifying the front-end signal acquisition, encoding, and transmission links, making it impossible to "embed functionality without changing the front-end video processing system."

[0007] To cope with the processing pressure of 8K video, existing system architecture solutions are also unable to meet the needs of lightweight, embedded pixel reassembly.

[0008] Heterogeneous and dedicated processing architectures include patented solutions that employ a heterogeneous architecture with a main chip and a dedicated 8K processing chip. When the main chip's computing power is insufficient, the dedicated chip takes over the display driving tasks. Another solution uses a master-slave multi-core architecture based on a high-performance SoC (such as the RK3588), pooling encoding and decoding resources through a master-slave approach. While these architectures improve processing power, their design goal is to handle core video encoding / decoding or rendering tasks. The systems are complex and highly integrated, not lightweight, pluggable modules solely responsible for pixel rearrangement.

[0009] For film and television production, distributed rendering using computer clusters or parallel acceleration using GPUs are common solutions. These solutions aim to process the generation or compositing of 8K content by distributing massive computing tasks across multiple nodes, rather than performing real-time, low-latency pixel rearrangement and format conversion on already generated video streams.

[0010] In summary, existing technologies offer a variety of approaches for processing 8K video: At the algorithm level, compressed sensing reconstruction and AI super-resolution technology implicitly involve pixel rearrangement operations, but their primary goal is compression or image quality enhancement, and they are deeply coupled with specific algorithm processes.

[0011] At the system level, while dedicated pixel switching systems can achieve pixel-level routing, they require the construction of an independent dedicated network; heterogeneous / distributed computing architectures, on the other hand, focus on improving overall processing power.

[0012] None of these solutions adequately address the core requirement: how to perform real-time, pixel-by-pixel arbitrary rearrangement and combination of output 8K video frames in an embeddable and configurable manner, without modifying existing and stable front-end video processing systems (such as cameras and encoders based on standard SDI or IP stream output), to directly meet the input format requirements of diverse back-end display units. Existing technologies either require modifications to the front end, result in excessively large systems, or address issues beyond simple pixel swapping. Therefore, a novel system architecture is urgently needed to achieve this goal. Summary of the Invention

[0013] To overcome the shortcomings of existing technologies, an embedded 8K video frame pixel-by-pixel rearrangement system and method are provided to solve the problem that existing technologies cannot perform real-time, pixel-by-pixel rearrangement and combination of 8K video frames without modifying the existing and stable front-end video processing system, so as to directly meet the input format requirements of diverse back-end display units.

[0014] To achieve the above objectives, an embedded access and preprocessing module is provided for a pixel-by-pixel reassembly system for 8K video frames that can be embeddedly accessed. This system receives the raw 8K video signal from the front-end video processing system through a standard physical interface and converts it into a video stream based on an IP network without changing the architecture and output of the front-end video processing system. This is used to divide each frame of pixel data in a video stream into multiple pixel blocks, encapsulate each pixel block and its associated source coordinate metadata into an independent pixel data packet, and route the pixel data packet to the network switching module of the target processing node according to a dynamically loadable pixel rearrangement mapping table, which is connected to the embedded access and preprocessing module; the pixel rearrangement mapping table records the mapping relationship between the coordinates of the source pixel or pixel block and the coordinates of the target pixel or pixel block and may contain processing parameters; The rearrangement module is used to reorganize, interpolate, or enhance pixel data to generate rearranged video frames based on the pixel rearrangement mapping table. It is connected to the network switching module, and multiple target processing nodes are each deployed with a rearrangement module. This synchronization and control module provides a unified clock reference for each module and coordinates the data processing timing between modules to ensure the synchronization of pixel data before and after reassembly. It is connected to the embedded access and preprocessing module, the network switching module, and the reassembly module.

[0015] The system serves as a transparent adaptation layer between the front-end standard video output and the back-end customized display input, and the front-end video processing system does not need to be aware of the existence of the system.

[0016] Furthermore, the embedded access and preprocessing module includes an FPGA processing unit and an ARM management unit. The FPGA processing unit is connected to the ARM management unit via a high-speed bus, and the ARM management unit is connected to the network switching module.

[0017] Furthermore, the embedded access and preprocessing module receives raw 8K video signals through physical interfaces such as HDMI, DVI, VGA, SDI, CVBS, and FIBER.

[0018] Furthermore, the network switching module is built on Ethernet and supports at least one of the following: MAC / RTP / UDP / IP / TCP protocol stack, multicast transmission, forward error correction (FEC) or negative acknowledgment (NACK) retransmission mechanism.

[0019] Furthermore, the rearrangement module includes an FPGA front-end processing unit and a GPU back-end computing unit. The FPGA front-end processing unit is connected to the GPU back-end computing unit, and the FPGA front-end processing unit is connected to a display module.

[0020] Furthermore, the synchronization and control module uses a network clock synchronization protocol to provide nanosecond-level clock synchronization for all network devices, and ensures the synchronization of video output frames from multiple target processing nodes through a threshold triggering mechanism of the frame buffer queue or a periodic timing packet calibration mechanism.

[0021] Furthermore, the pixel rearrangement mapping table is a dynamically loadable data structure that supports runtime updates to adapt to different backend display unit layout topologies. The pixel rearrangement mapping table may also include scaling algorithms, rotation angles, or transparency processing parameters.

[0022] Furthermore, the network switching module integrates an intelligent compression strategy, which includes at least one of inter-frame differential coding, pixel compression, and multi-format adaptive coding. Inter-frame differential coding transmits only the difference pixels and region coordinate information between adjacent frames, while multi-format adaptive coding generates both a downsampled preview stream and a full-resolution reconstructed information stream.

[0023] Furthermore, after receiving pixel data packets from the network, the FPGA front-end processing unit decapsulates them, extracts pixel data and source coordinate information, queries the pixel rearrangement mapping table stored locally or centrally to obtain the corresponding processing instructions, and distributes the pixel data that needs further processing to the GPU back-end computing unit through a high-speed interface. The GPU back-end computing unit performs sub-pixel level interpolation or AI image quality enhancement processing and then sends the data back to the FPGA front-end processing unit for reconstruction and output.

[0024] Furthermore, a FIFO buffer is set at the input front end of each rearrangement module. The system sets a uniform reading threshold. Subsequent reading and processing operations are triggered synchronously only when the amount of cached data in all relevant nodes reaches the reading threshold, so as to eliminate the delay caused by differences in network transmission paths.

[0025] This invention provides a method for pixel-by-pixel reassembly of 8K video frames in an embedded 8K video frame pixel-by-pixel reassembly system, comprising the following steps: S1. The embedded access and preprocessing module receives the raw 8K video signal from the front-end video processing system through a standard physical interface and converts it into a video stream based on the IP network without changing the architecture and output of the front-end video processing system. S2. The network switching module divides each frame of the video stream into multiple pixel data blocks and encapsulates each pixel block and its associated source coordinate metadata into an independent pixel data packet. S3. The network switching module routes pixel data packets to the target processing node based on a dynamically loadable pixel rearrangement mapping table. S4. At the target processing node, the rearrangement module reassembles, interpolates, or enhances pixel data packets according to the pixel rearrangement mapping table to generate rearranged video frames. The S5 synchronization and control module coordinates the entire process of signal reception and video frame reconstruction output of each module through a global synchronization mechanism, ensuring timing consistency.

[0026] The beneficial effects of this invention are that the embedded 8K video frame pixel-by-pixel reassembly system of this invention, through an independent embedded access and preprocessing module, can seamlessly receive standard 8K video output from existing front-end video processing systems (such as cameras and encoders based on SDI or ST2110 IP streams) without requiring any hardware or protocol modifications to the front-end equipment. This achieves "plug-and-play" pixel rearrangement capability, maximizing the protection of users' existing equipment assets and reducing system upgrade and modification costs.

[0027] The embedded 8K video frame pixel-by-pixel reassembly system of this invention utilizes a network switching module and a dynamically loadable pixel rearrangement mapping table. The system supports one-to-one and one-to-many arbitrary mappings from "source pixels / pixel blocks" to "target pixels / pixel blocks," and can be supplemented with processing parameters such as scaling, rotation, and transparency. This embedded 8K video frame pixel-by-pixel reassembly system allows for arbitrary pixel-level rearrangement, flexibly adapting to complex display needs. It can accurately meet the special requirements of diverse display units such as irregularly shaped LED splicing screens and multi-screen display walls for pixel spatial arrangement, breaking through the limitations of traditional fixed-format output or centralized processing. Attached Figure Description

[0028] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the structure of an 8K video frame pixel-by-pixel recombination system that can be embedded into an embodiment of the present invention; Figure 2 This is a schematic diagram of the embedded access and preprocessing module according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the embeddable 8K video frame pixel-by-pixel recombination method according to an embodiment of the present invention.

[0029] Figure label: Embedded access and preprocessing module 1, network switching module 2, rearrangement module 3, synchronization and control module 4. Detailed Implementation

[0030] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] Reference Figures 1 to 3 As shown, the present invention provides an 8K video frame pixel-by-pixel reassembly system that can be embedded, including: an embedded access and preprocessing module 1, a network switching module 2, a reassembly module 3, and a synchronization and control module 4.

[0033] Network switching module 2 is connected to embedded access and preprocessing module 1. Rearrangement module 3 is connected to network switching module 2. Rearrangement module 3 is deployed on multiple target processing nodes. Synchronization and control module 4 is connected to embedded access and preprocessing module 1, network switching module 2, and rearrangement module 3.

[0034] The embedded access and preprocessing module 1 is used to receive the raw 8K video signal from the front-end video processing system and convert it into a video stream based on the IP network.

[0035] The network switching module 2 is used to encapsulate the pixel data of the video frames of the video stream into pixel data packets with source coordinate information, and to route the pixel data packets to the target processing node according to a predefined pixel rearrangement mapping table.

[0036] Specifically, the pixel rearrangement mapping table is a dynamically loadable data structure that records the mapping relationship between the coordinates of the source pixel or pixel block and the coordinates of the target pixel or pixel block.

[0037] Ideally, the pixel rearrangement map also includes processing parameters such as scaling, rotation, or transparency.

[0038] The rearrangement module 3 is used to reorganize, interpolate, or enhance pixel data to generate rearranged video frames based on the pixel rearrangement mapping table.

[0039] Synchronization and control module 4 is used to provide a unified clock reference for each module and coordinate the data processing timing between modules to ensure the synchronization of pixel data before and after reconstruction.

[0040] The embedded 8K video frame pixel-by-pixel recombining system of the present invention aims to achieve real-time, pixel-by-pixel arbitrary rearrangement and combination of the output video stream without changing the architecture and output of the existing front-end video processing system (such as cameras and switchers based on SDI or standard IP stream output). This is achieved by introducing an independent intermediate layer based on networked switching and parallel processing, thereby directly adapting to the specific input format requirements of diverse back-end display units (such as irregularly shaped spliced ​​LED screens and multi-screen display walls).

[0041] The embedded 8K video frame pixel-by-pixel reassembly system of the present invention provides a non-intrusive pixel rearrangement solution.

[0042] The embedded 8K video frame pixel-by-pixel reassembly system of the present invention comprises four main parts: a network switching module, an embedded access and preprocessing module, a reassembly module, and a synchronization and control module. Its core idea is to decouple the pixel reassembly task from traditional centralized processing, transforming it into a collaborative process involving data distribution via the network, parallel computation on distributed nodes, and then network convergence.

[0043] The embedded 8K video frame pixel-by-pixel recombination system of the present invention serves as a "transparent" adaptation layer between the front-end standard video output and the back-end customized display input. The front-end system does not need to be aware of its existence, while the back-end display device can receive video data that meets its physical pixel arrangement requirements.

[0044] The system operates as a "transparent adaptation layer" in the following way: 1. On the front-end side, the embedded access and preprocessing module directly receives the existing 8K video output from front-end devices (such as cameras, switchers, and encoders) via a standard physical interface (such as a 75Ω BNC interface for 12G-SDI). The front-end devices do not require any hardware or protocol modifications, nor do they need to be aware of the system's existence—to the front-end device, its output signal is received by a module that "looks like a regular display device." For front-end devices that already support IP output (such as cameras compliant with ST2110), they can be directly connected to the network switching module via a 25G / 100G optical port.

[0045] 2. On the back-end side, each rearrangement module outputs rearranged video frames through standard high-speed video interfaces (such as HDMI 2.1 and DisplayPort 2.0) to directly drive the back-end display units (such as LED cabinets and displays). The back-end display units receive video signals that conform to their physical pixel arrangement requirements and do not need to perform any pixel rearrangement processing themselves.

[0046] 3. This "front-end standard input → intermediate pixel rearrangement → back-end standard output" architecture allows this system to be inserted as an independent module into existing video processing chains, achieving plug-and-play functionality. This is fundamentally different from closed, dedicated systems like PixelNet—the latter require dedicated input nodes, output nodes, and switching networks, and integrating them into existing workflows would mean replacing or significantly modifying the front-end signal acquisition, encoding, and transmission chains.

[0047] The embedded 8K video frame pixel-by-pixel recombining system of this invention enables real-time, pixel-by-pixel arbitrary swapping and recombination of 8K resolution video frames. By constructing a distributed processing architecture based on a high-speed switching network, it decomposes, routes, and recombines 8K video data with a single frame of up to 33 million pixels (7680×4320), supporting one-to-one or one-to-many mapping from "source pixel blocks" to "target pixel blocks," meeting the spatial rearrangement requirements with pixel-level precision.

[0048] The embedded access and preprocessing module 1 includes an FPGA processing unit and an ARM management unit. The FPGA processing unit is connected to the ARM management unit via a high-speed bus, and the ARM management unit is connected to the network switching module 2.

[0049] The embedded access and preprocessing module 1 receives raw 8K video signals via physical interfaces including HDMI, DVI, VGA, SDI, CVBS, and FIBER. The embedded access and preprocessing module 1 also includes IP video stream network input.

[0050] The video signal accessed by the embedded access and preprocessing module 1 includes a full frame of 8K signal, or multiple video frames after the 8K signal is divided.

[0051] See Figure 2 As shown, the FPGA side of the embedded access and preprocessing module is responsible for the underlying hardware interface and high-speed data processing: The SDI PHY receiver recovers data and clock from the 12G-SDI interface; Audio embedding / de-embedding units separate or embed audio data; The frame buffer and synchronization unit uses an external PTP or black field (BB) signal for phase locking to ensure input frame synchronization, with a phase locking error of less than 1 millisecond; The pixel format conversion unit converts the input YUV / RGB and other formats into a unified format required for internal processing or network transmission; The ST 2110 encapsulation engine then packages the processed video, audio, and auxiliary data into RTP / UDP / IP pixel packets conforming to the SMPTE ST2110-20, -30, and -40 standards, respectively.

[0052] The embedded access and preprocessing module runs a Linux operating system on its ARM side, integrating a complete TCP / IP protocol stack. It is responsible for network protocol processing, remote device management, LLDP (Link Layer Discovery Protocol) self-discovery, and communication with the control plane. The FPGA and ARM exchange control and data via a high-speed bus (such as PCIe or AXI). This module outputs standard 25G / 100G optical port IP streams, supporting seamless master / slave failover for ST 2022-7.

[0053] Network switching module 2 is built on Ethernet and supports at least one of the following: MAC / RTP / UDP / IP / TCP protocol stack, multicast transmission, forward error correction (FEC) or negative acknowledgment (NACK) retransmission mechanism.

[0054] The rearrangement module adopts a heterogeneous computing architecture, including FPGA, ASIC, SOC, etc., to decapsulate pixel data packets from the network switching module and query the pixel rearrangement mapping table to obtain pixel routing and processing instructions.

[0055] In this embodiment, the rearrangement module 3 includes an FPGA front-end processing unit and a GPU back-end computing unit. The FPGA front-end processing unit is connected to the GPU back-end computing unit, and the FPGA front-end processing unit is connected to a display module.

[0056] The reordering module of the embedded 8K video frame pixel-by-pixel recombining system of this invention adopts a heterogeneous collaborative architecture such as FPGA / ASIC / SOC, giving full play to its advantages in stream processing and low latency control. The reordering module, combined with hardware optimizations of sub-pixel-level processing algorithms (such as the parallel interpolation pipeline of FPGA), can process 7680×4320@high frame rate 8K video streams in real time, and control the end-to-end processing latency to the millisecond level, meeting the requirements of broadcast-level real-time applications.

[0057] The embedded 8K video frame-by-pixel reassembly system of this invention ensures high reliability and low latency throughout the reassembly process. By integrating precise clock synchronization, robust network transmission, and heterogeneous hardware acceleration, it meets the high bandwidth requirements of 8K video (such as tens of Gbps for uncompressed streams) while controlling the end-to-end latency of the system to the millisecond level. It is suitable for application scenarios with stringent timing requirements, such as on-site production and real-time monitoring.

[0058] The synchronization and control module 4 uses a network clock synchronization protocol to provide nanosecond-level clock synchronization for all network devices, and ensures the synchronization of video output frames of multiple target processing nodes through a threshold triggering mechanism of the frame buffer queue or a periodic timing packet calibration mechanism.

[0059] In this embodiment, the network switching module constitutes the data switching backbone of the system, responsible for the efficient and flexible routing of video pixel data packets.

[0060] The network architecture of the network switching module adopts an Ethernet-based switching network to achieve separate transmission of video, audio, and auxiliary data. The network supports multicast to meet one-to-many pixel distribution requirements. To ensure transmission reliability, forward error correction (FEC), negative acknowledgment (NACK), or multiple description coding (MDC) mechanisms can be introduced in weak network environments.

[0061] Data encapsulation and routing in the network switching module: The system logically or physically divides an input frame of 8K video data into multiple pixel blocks. Each pixel block and its associated metadata (such as source coordinates and target coordinate mapping) are encapsulated into an independent pixel data packet. Encapsulation follows protocol stacks such as MAC / RTP / UDP / IP / TCP, and a fragmented packet transmission strategy is used for large data units to adapt to the network MTU. The core switching device can route pixel data packets from any input port to any output port according to a pre-configured or dynamically issued mapping table, realizing arbitrary mapping of "source pixel block → target pixel block".

[0062] Each pixel data packet's payload contains one or more pixel blocks and their source coordinate information. The system maintains a global pixel rearrangement mapping table, where each record defines the correspondence between "source pixel (block) coordinates" and "target pixel (block) coordinates," and may include optional processing parameters such as scaling algorithms, rotation angles, and alpha blending.

[0063] The network switching module uses the pixel rearrangement mapping table to achieve precise routing of pixel data packets from the source address to the destination address, completing a one-to-one or one-to-many mapping of "source pixel block → destination pixel block", and the signal remains digitally attenuated throughout the process.

[0064] The coordination method between the independent pixel block packet and the mapping table routing is as follows: 1. During the data encapsulation phase, the network switching module logically or physically divides a frame of 8K video data into multiple pixel blocks. Unlike standard IP video transmission protocols such as ST 2110, which encapsulate complete video streams line by line or frame by frame, this system encapsulates each pixel block and its associated metadata (source coordinate information) into independent pixel data packets. This pixel-level decomposition allows each data packet to be independently routed to different target processing nodes, which is the basis for achieving arbitrary pixel-by-pixel rearrangement.

[0065] 2. The network switching module maintains a global pixel rearrangement mapping table. This mapping table is a dynamically loadable data structure that records the correspondence between source pixel (block) coordinates and target pixel (block) coordinates. Based on this mapping table, the network switching module routes pixel data packets carrying source coordinate information from any input port to the corresponding target processing node, completing a one-to-one or one-to-many mapping between "source pixel block → target pixel block".

[0066] 3. Since the mapping table can be dynamically loaded, when the backend display topology changes (such as adding display units or adjusting the splicing method), users can update the mapping table without restarting the system or redeploying the hardware, so that the system can adapt to different display layout requirements at runtime.

[0067] To reduce network bandwidth pressure, the network switching module integrates intelligent compression strategies including: inter-frame differential coding, which transmits only the difference pixels and region coordinate information between adjacent frames; pixel compression, which performs lossy or lossless scaling or representative value replacement on pixel data according to display requirements or region complexity; and multi-format adaptive coding, where the encoding end simultaneously generates a downsampled preview stream and a full-resolution reconstructed information stream, and the receiving end reconstructs the data as needed.

[0068] In this embodiment, the embedded access and preprocessing module is the key to achieving "embedding without changing the front-end system". It is responsible for lossless access to various video streams in various formats from the front end and converting them into standardized data streams that can be processed internally by the system.

[0069] The embedded access and preprocessing module provides multiple physical or network interfaces for directly receiving 8K video signals from traditional baseband devices or splitting them into multiple 8K signals. Through its built-in video-to-IP conversion gateway, the embedded access and preprocessing module converts baseband video signals into IP network streams in real time.

[0070] The embedded access and preprocessing module is an embedded processing board based on an FPGA+ARM architecture.

[0071] Signal inputs and interfaces: The module provides a standard 12G-SDI physical interface (75Ω BNC) to receive 8K video signals output from front-end devices. For 8K@60fps raw signals that a single 12G-SDI input cannot handle, it supports four-channel 12G-SDI bundled input.

[0072] For front-ends that already support IP output (such as cameras compliant with ST 2110), they can be directly connected to the pixel exchange network via 25G / 100G optical ports.

[0073] FPGA processing logic: The FPGA is responsible for the underlying hardware acceleration processing, including: physical layer reception, serial-to-parallel conversion, and clock recovery of SDI signals; de-embedding of embedded audio streams; and frame synchronization analysis of input video streams.

[0074] After completing the above processing, the FPGA converts the video pixel data into an internal processing format (such as YUV 4:2:2) and prepares it for IP packaging.

[0075] ARM System and IP Outputs: The ARM processor runs an embedded Linux system and integrates a complete network protocol stack. It receives video data processed by the FPGA and encapsulates the video and audio into independent IP multicast streams according to the SMPTE ST 2110-20 / -30 standard.

[0076] The module supports external synchronization phase-locking via PTP (IEEE 1588) precise time protocol or traditional black field (BB) signal to ensure that its output stream is strictly synchronized with other devices in the system. After phase-locking, the time difference between the IP output and the synchronization source is less than 1 millisecond.

[0077] Finally, the encapsulated IP stream is output to the pixel switching network via a 25G / 100G optical port. The module supports the LLDP protocol and can be automatically discovered and managed within the network.

[0078] In this embodiment, the rearrangement module is the core of the pixel rearrangement algorithm. It adopts a heterogeneous collaborative architecture of FPGA and GPU to balance low-latency processing and complex computing capabilities.

[0079] FPGA front-end processing unit: As a data pipeline and scheduling center, the FPGA first receives the IP pixel data packets required by the target pixel block from the network, performs network unpacking, and extracts the pixel data and source coordinate information.

[0080] Subsequently, the FPGA queries the pixel rearrangement mapping table stored locally or centrally based on the received target coordinates to obtain the corresponding source coordinates and processing parameters.

[0081] Next, the FPGA distributes the pixel data and parameters requiring further processing to the GPU via high-speed interfaces such as PCIe 5.0. The entire process employs a pipelined design, achieving microsecond-level processing latency.

[0082] GPU backend computing unit: The GPU receives data from the FPGA and uses its massively parallel computing capabilities to perform complex pixel-level operations. This includes, but is not limited to: Subpixel-level processing: such as bilinear or bicubic interpolation, used to achieve accurate resampling of non-integer pixel coordinates.

[0083] AI Enhancement Processing: Utilizing built-in AI acceleration units such as Tensor Cores, pre-trained neural network models (such as 3D-CNN) are run to enhance the image quality of the rearranged image through super-resolution, noise reduction, and other processes.

[0084] Encoding and Packaging: For outputting a compressed stream, efficient encoding can be achieved using the GPU's hardware encoder (such as NVENC). For chunked processing, **SFrame Coding (SFE)** can be used, which utilizes multiple encoding engines to process different parts of a frame in parallel.

[0085] Data feedback and output: Data processed by the GPU is sent back to the FPGA via the PCIe channel.

[0086] The FPGA is responsible for reassembling the final processed pixel data that conforms to the target display area format, and driving high-speed video interfaces such as HDMI 2.1 or DisplayPort 2.0 to output to the physical display unit.

[0087] The reordering module receives pixel data packets from the network, reassembles and buffers them, and performs basic, rule-based pixel operations. The advantage of the reordering module lies in its hardware programmability and spatially parallel architecture, enabling microsecond-level processing latency. It is ideally suited for pixel-level pipeline operations, such as initial coordinate mapping, data format conversion, and preparing data for subsequent complex processing.

[0088] The reordering module is responsible for computationally intensive global pixel reordering algorithms, performing large-scale parallel processing on entire frames or blocks of video data. Specific tasks of the reordering module include: performing spatial dimensional reordering based on subpixel convolution (such as reordering channel information to a high-resolution space), implementing subpixel interpolation calculations required for complex motion estimation and compensation (solving in the continuous domain using bilinear or bicubic interpolation algorithms), and running AI models for auxiliary processing such as image quality enhancement.

[0089] In this embodiment, the synchronization and control module ensures that all distributed processing nodes are strictly synchronized in time and data, which is the foundation for achieving seamless pixel recombination and display.

[0090] The synchronization and control module, based on the Network Time Protocol, establishes a unified hardware clock reference at the nanosecond level within the system to achieve global clock synchronization. The system deploys a PTP (IEEE 1588) Grandmaster clock, providing a unified clock reference with sub-microsecond accuracy to all access devices (embedded gateways, processing nodes, and switches) through network devices that support hardware timestamps.

[0091] Synchronization mechanism for data flow between synchronization and control modules: A FIFO (First-In, First-Out) buffer is set at the input front end of each processing node. The system sets a uniform read threshold, and subsequent read and processing operations are only triggered synchronously when the buffered data volume of all relevant nodes reaches this threshold. This "wait for the slowest" mechanism eliminates the initial latency caused by differences in network paths.

[0092] A master node is designated as the timing node, which periodically sends synchronization probe packets to other slave nodes to calculate and calibrate the clock offsets between them. Slave nodes use voltage-controlled adjustable crystal oscillators for frequency fine-tuning to ensure that the local clocks, timestamps, and frame rates of all nodes remain strictly consistent.

[0093] The synchronization and control module employs a multi-level synchronization strategy. First, a FIFO cache is set on each node, and a uniform read permission capacity threshold is set. Only when the cached data of all relevant nodes reaches the threshold is a synchronous read triggered, thereby eliminating the latency caused by differences in network transmission paths.

[0094] In the final video compositing and output stage, the synchronization and control module ensures that the frame rate and timestamp of all video streams to be stitched are completely consistent. The playback or display engine needs to trigger the drawing command simultaneously after all video frames have been decoded / processed, keeping the drawing time difference between multiple video streams within an extremely low range (such as within one frame, about 5-16.7ms), to achieve low-latency pixel rearrangement and compositing output.

[0095] See Figure 1 As shown, the system of this invention constructs a complete, non-intrusive processing link from the front-end video source to the back-end irregularly shaped display unit. Its core data flow is: The front-end 8K signal source outputs the raw video signal through a physical or network interface. This signal enters the embedded access and preprocessing module, which is usually implemented in the form of heterogeneous boards such as FPGA / ASIC / SOC. This module completes the reception of video signals, clock recovery, frame synchronization and format conversion, and encapsulates them into standardized IP unicast / multicast streams.

[0096] The encapsulated video stream enters the network switching module via Ethernet. This network module is based on a high-performance Ethernet switch and supports robust transmission mechanisms such as multicast, FEC / NACK / MDC. It is responsible for real-time routing and switching of video pixel data packets according to predefined mapping rules.

[0097] The swapped pixel data packets are distributed to one or more pixel rearrangement processing nodes. Each processing node adopts a heterogeneous architecture of end-to-end pipelined + parallel computing. After receiving the network pixel data packets via high-speed interconnects, it completes pixel-by-pixel remapping and interpolation (such as bilinear / bicubic) processing within millisecond latency, based on the mapping table, for 33 million pixels / frame. The processed video frames are then globally coordinated by the synchronization and control module (based on network clock and threshold synchronization mechanisms) and output through single or multiple physical video interfaces to directly drive the back-end irregularly shaped splicing LED screen or multi-screen display wall. The entire system is transparent to the front-end video processing system, realizing plug-and-play pixel-level arbitrary rearrangement functionality.

[0098] This invention provides a method for pixel-by-pixel reassembly of 8K video frames in an embedded 8K video frame pixel-by-pixel reassembly system, comprising the following steps: S1. Embedded access and preprocessing module 1 receives the raw 8K video signal from the front-end video processing system and converts it into a video stream based on the IP network.

[0099] S2, Network Switching Module 2 divides each frame of pixel data in the video stream into multiple pixel data packets and adds source coordinate information to each pixel data packet.

[0100] S3. According to the predefined pixel rearrangement mapping table, the network switching module 2 routes the pixel data packets to the target processing node via the network.

[0101] S4. At the target processing node, the rearrangement module 3 reassembles, interpolates, or enhances the pixel data packets according to the pixel rearrangement mapping table to generate rearranged video frames.

[0102] S5 and the synchronization and control module 4 coordinate the entire process of signal reception and video frame reconstruction output of each module through a global synchronization mechanism to ensure timing consistency.

[0103] The workflow of the embedded access-enabled 8K video frame pixel-by-pixel reassembly method of the present invention is as follows: after data access, preprocessing and distribution, the reassembly algorithm is executed, and the result is finally synchronized, spliced ​​and output driven (e.g., output through multiple HDMI 2.1 interfaces).

[0104] Combination Figure 3 As shown, the workflow of the 8K video frame pixel-by-pixel recombination method of the present invention is as follows: Signal input: The front-end 8K camera outputs the raw video stream through four bundled 12G-SDI cables.

[0105] IP-based conversion: The FPGA of the embedded access gateway receives the SDI signal and completes decoding, clock recovery, and audio de-embedding. The ARM system encapsulates the video data into an IP multicast stream conforming to the SMPTE ST 2110-20 standard and injects it into the pixel switching network through a 25G optical port.

[0106] Network switching and routing: The pixel switching network routes IP packets carrying pixel data from the source address to the network address of the target processing node based on the preloaded "pixel rearrangement mapping table".

[0107] Reordering Processing: The FPGA at the target processing node receives and unpacks network packets, and queries the mapping table to obtain processing instructions. Pixel data requiring complex calculations is sent to the GPU via PCIe 5.0 for sub-pixel interpolation or AI enhancement. The processed data is then transmitted back to the FPGA.

[0108] Synchronization and Output: Under the coordination of the synchronization and control module, all processing nodes complete processing synchronously. The FPGA of each node reassembles the pixel data of the display area under its responsibility into complete video frames, and outputs them synchronously to the corresponding display module (such as LED cabinet) through the HDMI 2.1 interface.

[0109] Final display: All display modules receive and display synchronized video signals, together presenting a complete 8K image that has been rearranged pixels and conforms to the physical layout of the irregularly shaped screen.

[0110] The modules of this invention work together in the following ways: The embedded access and preprocessing module converts heterogeneous front-end signals into standard IP streams, providing a unified data format foundation for subsequent pixel-coordinate-based routing. The network switching module decomposes video frames into independent data packets at the pixel level and uses a dynamically loadable mapping table as the routing basis, ensuring that pixel data packets are accurately distributed to the target processing nodes responsible for their respective display areas. The synchronization and control module ensures that all processing nodes synchronously trigger processing and output after data is complete through global clock synchronization and frame buffer threshold triggering mechanisms, eliminating display asynchrony caused by differences in network paths. The rearrangement module reassembles and enhances pixel data at each target processing node according to the mapping table, ultimately outputting it to the display unit through a standard video interface.

[0111] The aforementioned collaboration enables the system to achieve arbitrary pixel-by-pixel rearrangement with end-to-end millisecond latency at 8K resolution, while maintaining non-intrusive and transparent access to the front-end and back-end systems.

[0112] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. An 8K video frame-by-pixel reassembly system that can be embedded, characterized in that, Includes the following steps: An embedded access and preprocessing module for receiving raw 8K video signals from a front-end video processing system via a standard physical interface and converting them into IP network-based video streams without changing the architecture and output of the front-end video processing system. This is used to divide each frame of pixel data in the video stream into multiple pixel blocks, encapsulate each pixel block and its associated source coordinate metadata into an independent pixel data packet, and route the pixel data packet to the network switching module of the target processing node according to a dynamically loadable pixel rearrangement mapping table, which is connected to the embedded access and preprocessing module. The pixel rearrangement mapping table records the mapping relationship between the coordinates of the source pixel or pixel block and the coordinates of the target pixel or pixel block, and may include processing parameters; The rearrangement module is used to reorganize, interpolate, or enhance pixel data to generate rearranged video frames based on the pixel rearrangement mapping table. It is connected to the network switching module, and multiple target processing nodes are each deployed with a rearrangement module. This synchronization and control module provides a unified clock reference for each module and coordinates the data processing timing between modules to ensure the synchronization of pixel data before and after reassembly. It is connected to the embedded access and preprocessing module, the network switching module, and the reassembly module. The system serves as a transparent adaptation layer between the front-end standard video output and the back-end customized display input, and the front-end video processing system does not need to be aware of the existence of the system.

2. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 1, characterized in that, The embedded access and preprocessing module includes an FPGA processing unit and an ARM management unit. The FPGA processing unit is connected to the ARM management unit via a high-speed bus, and the ARM management unit is connected to the network switching module.

3. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 2, characterized in that, The embedded access and preprocessing module receives raw 8K video signals through physical interfaces such as HDMI, DVI, VGA, SDI, CVBS, and FIBER.

4. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 1, characterized in that, The network switching module is built on Ethernet and supports at least one of the following: MAC / RTP / UDP / IP / TCP protocol stack, multicast transmission, forward error correction (FEC) or negative acknowledgment (NACK) retransmission mechanism.

5. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 1, characterized in that, The rearrangement module includes an FPGA front-end processing unit and a GPU back-end computing unit. The FPGA front-end processing unit is connected to the GPU back-end computing unit, and the FPGA front-end processing unit is connected to a display module.

6. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 1, characterized in that, The synchronization and control module uses a network clock synchronization protocol to provide nanosecond-level clock synchronization for all network devices, and ensures the synchronization of video output frames of multiple target processing nodes through a threshold triggering mechanism of the frame buffer queue or a periodic timing packet calibration mechanism.

7. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 1, characterized in that, The pixel rearrangement mapping table is a dynamically loadable data structure that supports runtime updates to adapt to different backend display unit layout topologies. The pixel rearrangement mapping table may also include scaling algorithms, rotation angles, or transparency processing parameters.

8. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 1, characterized in that, The network switching module integrates an intelligent compression strategy, which includes at least one of inter-frame differential coding, pixel compression, and multi-format adaptive coding. Inter-frame differential coding transmits only the difference pixels and region coordinate information between adjacent frames, while multi-format adaptive coding generates both a downsampled preview stream and a full-resolution reconstructed information stream.

9. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 5, characterized in that, After receiving pixel data packets from the network, the FPGA front-end processing unit decapsulates them, extracts pixel data and source coordinate information, and queries the pixel rearrangement mapping table stored locally or centrally to obtain the corresponding processing instructions. The pixel data that needs further processing is distributed to the GPU back-end computing unit through a high-speed interface. The GPU back-end computing unit performs sub-pixel level interpolation or AI image quality enhancement processing and then sends the data back to the FPGA front-end processing unit for reconstruction and output.

10. The 8K video frame pixel-by-pixel reassembly system with embedded access according to claim 6, characterized in that, A FIFO buffer is set at the input front end of each rearrangement module. The system sets a uniform reading threshold. Subsequent reading and processing operations are triggered synchronously only when the amount of cached data in all relevant nodes reaches the reading threshold, so as to eliminate the delay caused by the difference in network transmission path.

11. A method for pixel-by-pixel reassembly of 8K video frames in an 8K video frame pixel-by-pixel reassembly system that can be embedded in any one of claims 1 to 10, characterized in that, Includes the following steps: S1. The embedded access and preprocessing module receives the raw 8K video signal from the front-end video processing system through a standard physical interface and converts it into a video stream based on the IP network without changing the architecture and output of the front-end video processing system. S2. The network switching module divides each frame of the video stream into multiple pixel data blocks and encapsulates each pixel block and its associated source coordinate metadata into an independent pixel data packet. S3. The network switching module routes pixel data packets to the target processing node based on a dynamically loadable pixel rearrangement mapping table. S4. At the target processing node, the rearrangement module reassembles, interpolates, or enhances pixel data packets according to the pixel rearrangement mapping table to generate rearranged video frames. The S5 synchronization and control module coordinates the entire process of signal reception and video frame reconstruction output of each module through a global synchronization mechanism, ensuring timing consistency.

Images