Multi-screen heterogeneous resolution adaptive video processing system and method

Abstract

The application discloses a multi-screen heterogeneous resolution adaptive video processing system and method; the method receives multiple heterogeneous signal sources, carries out timing synchronization alignment through unified global time stamp and a buffer queue based on delay gradient adaptive adjustment; a virtual canvas independent of the resolution of a physical display device is created, and the synchronized data is uniformly rendered to generate globally unique composite frame data; EDID information of multiple target display devices is acquired to calculate respective optimal output resolutions; in the case that the rendering action is not re-executed, an image resampling algorithm is used to independently scale and adapt the globally unique composite frame data to each display device. The application realizes 'one rendering, multiple heterogeneous resolution independent adaptation', effectively solves the problems of picture stretching, black edges and waste of computing power when multiple screens are mixed and accessed in different resolutions, and supports seamless topology reconstruction of display devices hot plug and frame-level synchronous superposition of real-time voice subtitles.

Description

Technical Field

[0001] This invention relates to the field of video signal processing and multimedia display control technology, and in particular to a multi-screen, multi-resolution adaptive video processing system and method. Background Technology

[0002] With the rapid development of display technology, the demand for multi-screen hybrid displays is increasing in scenarios such as conference systems, security monitoring, and command and dispatch. Existing multi-screen video processing technologies have the following shortcomings when facing complex application scenarios:

[0003] Synchronization difficulties of heterogeneous signal sources: When HDMI hardware capture stream and network Web rendering stream are mixed, due to network jitter and differences in encoding and decoding latency, conventional fixed delay or simple frame dropping methods are difficult to achieve frame-level alignment, which easily leads to audio and video desynchronization.

[0004] Rigid adaptation to different resolutions: Traditional multi-screen processors often use a fixed single output resolution. When outputting to screens with different resolutions such as 4K and 1080p, relying on the screen's own scaling can easily lead to image stretching or black borders. If each screen is rendered independently, the system's computing power will increase exponentially, which can easily cause overload.

[0005] Rigid topology and poor scalability: Traditional tightly coupled hardware architectures often require interrupting the current video stream for re-initialization when adding or removing display devices (hot-plugging), and system function upgrades are difficult. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a multi-screen heterogeneous resolution adaptive video processing system and method, aiming to achieve accurate synchronization of heterogeneous sources, lossless adaptation of multi-screen heterogeneous resolutions, and seamless dynamic topology reconstruction.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a multi-screen, heterogeneous resolution adaptive video processing method, comprising the following steps:

[0009] Step S1: Receive input data from multiple heterogeneous signal sources, including HDMI audio and video signals, web page content, and audio acquisition signals; add a unified global timestamp to each input data source, and establish a buffer queue based on the inherent delay characteristics of each signal source; perform timing synchronization and alignment processing on the input data through an adaptive buffering strategy.

[0010] Step S2: Create a virtual canvas independent of the physical resolution of the target display device, render the time-synchronized and aligned multi-channel input data to the virtual canvas, and generate globally unique composite frame data;

[0011] Step S3: Receive the screen layout configuration command submitted by the user through the Web terminal, perform position layout and screen mixing processing on the multi-channel input data in the virtual canvas according to the layout configuration command, and update the composite frame data;

[0012] Step S4: Obtain extended display identification data (EDID) information of multiple target display devices. Based on the resolution parameters in the EDID information of each target display device, calculate the optimal output resolution corresponding to each target display device. Without re-executing the rendering action of multiple input data, use an image resampling algorithm to independently scale and adapt the globally unique synthetic frame data to each optimal output resolution, and output it to the corresponding target display devices for display.

[0013] Preferably, in step S1, the timing synchronization and alignment processing of the input data using an adaptive buffering strategy specifically includes:

[0014] Analyze the acquisition delay, network transmission delay, and decoding delay of different signal sources, and establish a timing delay model;

[0015] Real-time monitoring of the delay gradient change between the arrival timestamp of each input data source and the global timestamp;

[0016] When the delay gradient change exceeds the preset jitter threshold, it is determined that the signal source has network jitter, the compensation time is calculated and the buffer queue depth corresponding to the signal source is dynamically expanded to absorb network jitter;

[0017] When the delay gradient recovers to a stable range, the buffer queue depth is reduced by a preset step size to reduce end-to-end latency and ensure that multiple data frames are strictly aligned at the output.

[0018] Preferably, in step S4, the process of independently scaling and adapting the synthesized frame data using an image resampling algorithm specifically includes:

[0019] The list of physical resolutions and color space capabilities supported by each target display device are extracted from the EDID information.

[0020] Look up the recommended output parameters that match each target display device in the preset resolution mapping table;

[0021] For each target display device, an independent output channel is established, and the bilinear interpolation algorithm or Lanczos resampling algorithm is called to resample the globally unique synthetic frame data into a target resolution format that matches each output channel.

[0022] Preferably, in step S3, performing screen blending processing within the virtual canvas according to the layout configuration instructions specifically includes:

[0023] A visual layout editor is provided on the web, which receives the position coordinates, size, layer order and transparency parameters of each signal source in the virtual canvas configured by the user through drag and drop.

[0024] Based on the hierarchical order and transparency parameters, alpha blending, chroma keying, or mask overlay processing are performed on video frames from multiple input data sources to generate a composite image with a multi-layered structure.

[0025] Preferably, it also includes a real-time voice-to-text overlay step, specifically including:

[0026] Extract the audio stream corresponding to the audio acquisition signal, use the integrated speech recognition engine for real-time recognition, and generate subtitle text with a time stamp that matches the global timestamp;

[0027] Convert the subtitle text rendering to a bitmap image;

[0028] Based on the user-configured subtitle style parameters, the bitmap image is frame-level aligned and superimposed with the video frame with the corresponding timestamp in the virtual canvas according to the time stamp, so that the subtitle screen in the synthesized video stream does not have a relative delay exceeding the human eye perception threshold when output to the display device, while the corresponding HDMI video screen and web screen are output.

[0029] Preferably, it also includes a multi-screen dynamic topology response step:

[0030] During continuous video stream output, if a new target display device is detected to be connected or an existing display device is disconnected, the EDID information of the currently online device is reread through a plug-and-play event.

[0031] Based on the updated EDID information, the corresponding output channels and scaling adaptation tasks are dynamically created or destroyed, while the rendering process of the virtual canvas and the generation of composite frame data are kept uninterrupted, thus achieving real-time seamless reconstruction of multi-screen heterogeneous topology.

[0032] Preferably, it also includes remote control steps based on a plug-in architecture:

[0033] The signal input, image processing, and display output modules are encapsulated as plug-ins with standardized interfaces;

[0034] The remote control layer uses a dual-protocol communication mechanism combining RESTful API and WebSocket to receive remote commands;

[0035] Based on remote instructions, the system dynamically executes plugin loading, unloading, hot updates, or version rollback operations without interrupting the main video processing stream and the generation of composite frame data.

[0036] Secondly, the present invention provides a multi-screen, heterogeneous resolution adaptive video processing system, comprising:

[0037] The signal access layer is used to receive HDMI audio and video signals, web page content, and audio acquisition signals. It adds a unified timestamp to each signal and performs timing synchronization alignment by dynamically adjusting the buffer queue depth based on the delay gradient.

[0038] The intelligent canvas engine layer is used to create a virtual canvas independent of the physical resolution of the display device, to uniformly render synchronized multi-channel data to generate globally unique composite frame data, and to perform blending processing within the virtual canvas according to the layout configuration instructions on the web side.

[0039] The display output layer is used to obtain EDID information of multiple target display devices, calculate the optimal output resolution for each device, and independently scale and adapt the synthesized frame data to each display device without re-executing the rendering process.

[0040] The remote control layer is used to manage the lifecycle of functional plugins with standardized interfaces through a dual-protocol communication mechanism.

[0041] Preferably, the system also includes:

[0042] The subtitle processing module, integrated into the intelligent canvas engine layer, is used to recognize the audio stream in real time, generate subtitle text with time stamps, convert the subtitle text into bitmap images, and then perform frame-level alignment and overlay with the corresponding video frames in the virtual canvas according to the time stamps.

[0043] The beneficial effects of this invention are: high computing power reuse: through the collaborative mechanism of "globally unique composite frame + no re-rendering + multi-path independent scaling", the system only needs to render once to adapt to multiple screens with different resolutions. Compared with the traditional multi-path independent rendering scheme, it greatly reduces the utilization rate of computing resources such as GPU.

[0044] Robust heterogeneous synchronization: Based on delay gradient and jitter threshold, the buffer queue depth is dynamically adjusted, which can effectively absorb short-term jitter of network Web signals, eliminate the relative delay between heterogeneous sources, and achieve strict frame-level alignment.

[0045] Seamless topology reconstruction: When display devices are hot-plugged, the output channels are dynamically created or destroyed through plug-and-play events, while the rendering process of the virtual canvas is not interrupted, thus realizing real-time seamless reconstruction of multi-screen heterogeneous topologies. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the architecture of the multi-screen, heterogeneous resolution adaptive video processing system of the present invention;

[0048] Figure 2 This is a flowchart illustrating the multi-screen, heterogeneous resolution adaptive video processing method of the present invention.

[0049] Figure 3 This is a schematic diagram of the intelligent canvas engine layer of the present invention;

[0050] Figure 4 This is a schematic diagram illustrating the adaptation of the multi-screen output with different resolutions according to the present invention;

[0051] Figure 5 This is a schematic diagram illustrating the multi-signal source mixing and subtitle overlay processing of the present invention;

[0052] Figure 6 This is a schematic diagram illustrating the communication between the remote control layer and the plug-in architecture of this invention. Detailed Implementation

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0055] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0056] Reference Figures 1-6 This is one embodiment of the present invention, which provides a multi-screen heterogeneous resolution adaptive video processing method, including the following steps:

[0057] Step S1: Receive input data from multiple heterogeneous signal sources, including HDMI audio and video signals, web page content, and audio acquisition signals; add a unified global timestamp to each input data source, and establish a buffer queue based on the inherent delay characteristics of each signal source; perform timing synchronization and alignment processing on the input data through an adaptive buffering strategy.

[0058] This step specifically aims to resolve the severe audio-visual asynchrony and stuttering issues caused by the significant difference in latency characteristics when traditional systems mix HDMI hardware streams (low latency, constant frame rate) and web streams (high jitter, variable frame rate). The implementation process includes the following sub-steps:

[0059] Sub-step S11: Heterogeneous source access and high-precision timestamp annotation based on a unified master clock

[0060] The system concurrently receives three types of heterogeneous signals: HDMI audio and video signals input through a hardware acquisition card, web page content captured based on a browser rendering engine, and audio acquisition signals input through a sound card.

[0061] The system maintains a high-precision unified master clock (such as a high-precision counter based on system time). When any signal enters the system's physical layer or network receiving layer, it is immediately intercepted and stamped with the current timestamp of this unified master clock as its "arrival timestamp". For web interfaces, the system also parses the RTP timestamp in its transmission protocol (such as WebRTC) and maps it to the global master clock domain to eliminate network reference time deviation.

[0062] Sub-step S12: Static timing modeling based on inherent delay characteristics

[0063] Different signal sources have inherent fundamental delays in the acquisition, transmission, and decoding stages. The system establishes a static timing delay model for each signal channel:

[0064] For HDMI signals, the inherent latency is usually the processing latency of the capture card (constant, such as 1-2 frame periods).

[0065] For web signals, the inherent delay is the sum of the network transmission delay and the browser decoding delay;

[0066] For audio signals, the inherent delay is the sound card buffer delay.

[0067] Based on the inherent delay characteristics mentioned above, the system calculates an "expected presentation time baseline" for each signal. The arrival timestamp minus this inherent delay is the theoretical acquisition timestamp of the data frame.

[0068] Sub-step S13: Adaptive buffer dynamic compensation based on delay gradient

[0069] Instead of setting fixed-length buffer queues for each signal, the system uses a dynamic state machine for real-time adjustment.

[0070] Delay gradient calculation: Real-time monitoring of the deviation between the actual arrival time of each input data frame and the "expected presentation time baseline" to calculate the delay gradient change rate.

[0071] Jitter Detection and Queue Expansion: When the latency gradient change rate of a certain network Web signal exceeds a preset jitter threshold (e.g., multiple consecutive frames with an increasing latency trend and an increment exceeding 20ms), the system determines that the signal is experiencing network congestion or sudden jitter. At this time, the system dynamically expands the depth of the video buffer queue corresponding to that Web signal (increasing the number of buffered frames) to absorb network jitter and avoid screen tearing caused by dropped frames. Simultaneously, to prevent audio from dropping due to excessive waiting time, the system sends a time-stretching command to the audio buffer queue, employing algorithms such as WSOLA (Waveform Similarity Overlap Addition) to perform microsecond-level time compression and expansion of the audio data without changing the audio pitch, thus actively extending the audio to wait for delayed video frames.

[0072] Queue reduction and delay recovery: When the rate of change of the delay gradient of the web stream recovers to the stable range (i.e., jitter is eliminated), in order to reduce the overall end-to-end latency, the system gradually reduces the buffer queue depth corresponding to the signal by a preset step size (such as 1ms per frame), discards redundant historical frames, and simultaneously releases the stretched state of the audio, so that the system can be restored to the steady-state alignment with the lowest latency.

[0073] Sub-step S14: Frame-level strict alignment and output release

[0074] After the aforementioned dynamic buffering and compensation, the system establishes a global synchronization threshold at the output. Data frames in each buffer queue must be released at the unified presentation time specified by the global synchronization threshold, based on their uniform global timestamps. Only when the timestamps of multiple video and audio frames all reach or exceed this presentation time will the system package this group of data frames and push it as "timing-aligned data at the same moment" to the subsequent virtual canvas rendering module.

[0075] Step S2: Create a virtual canvas independent of the physical resolution of the target display device, render the time-synchronized and aligned multi-channel input data to the virtual canvas, and generate globally unique composite frame data;

[0076] This step specifically aims to solve the technical challenges of "linear growth in rendering computing power with the number of screens" and "rigid adaptation caused by tight coupling of physical resolution" in traditional multi-screen processing systems. By constructing a logically independent virtual canvas at the GPU level, it achieves complete decoupling of content rendering and physical output. The specific implementation process includes the following sub-steps:

[0077] Sub-step S21: Decoupling the creation of the virtual canvas from the resolution

[0078] Unlike existing technologies that directly bind the rendering target to the physical display frame buffer, the system allocates an off-screen rendering surface, or "virtual canvas," in the graphics processing unit's (GPU) memory, independent of the physical resolution of any target display device.

[0079] The base resolution of this virtual canvas is dynamically calculated by the system based on the current hardware computing power limit and the highest resolution of the input signal (e.g., set to a uniform 7680×4320 or logical pixel coordinate space). This canvas does not contain any EDID information constraints specific to any particular display, fundamentally decoupling the strong relationship between the spatial layout of the screen content and the final physical output screen resolution.

[0080] Sub-step S22: Unified coordinate mapping of multi-path aligned data and single-pass composite rendering

[0081] The system receives the "time-synchronized and aligned multi-channel input data" output from step S1 (this data is usually decoded into a texture format that the GPU can recognize). The system maps each input texture to a pre-defined logical coordinate space of the virtual canvas.

[0082] In the GPU rendering pipeline, the system initiates a single draw call, rendering multiple heterogeneous textures (HDMI video frames, web frame frames, subtitle bitmaps, etc.) in parallel onto the off-screen surface of the aforementioned virtual canvas through rasterization operations. This process performs pixel-level operations such as alpha blending and color space conversion (e.g., from YUV to RGB), combining multiple independent input data into a complete image containing all image elements.

[0083] Sub-step S23: Generation and memory residency of globally unique composite frame data

[0084] After the aforementioned single rendering pipeline is completed, a "globally unique composite frame data" (represented as a global texture pointer or frame buffer object, FBO) is generated in the GPU memory. This composite frame data resides in the GPU memory and is not directly output to any physical display interface.

[0085] Regardless of whether the system needs to output to 2, 4, or more physical screens of different resolutions, this step will only perform one rendering calculation and generate only one texture data containing complete image information.

[0086] Step S3: Receive the screen layout configuration command submitted by the user through the Web terminal, perform position layout and screen mixing processing on the multi-channel input data in the virtual canvas according to the layout configuration command, and update the composite frame data;

[0087] This step specifically aims to address the system stuttering, frame drops, and wasted GPU computing power caused by triggering a full-screen re-rendering when users adjust the screen layout in real time (such as dragging windows, scaling, or switching layers). The specific implementation process includes the following sub-steps:

[0088] Sub-step S31: Web-based interaction parsing and incremental instruction generation

[0089] The system provides an HTML5-based visual layout editor on the web, responding in real-time to user interactions such as dragging and zooming. Unlike traditional solutions that submit the entire canvas state each time, the web client sends incremental layout configuration commands to the backend processing engine via a WebSocket persistent connection. These commands only contain the changed signal source ID, the new position coordinate bounding box, the size scaling ratio, the target hierarchy order, and the transparency parameter, significantly reducing communication latency.

[0090] Sub-step S32: Calculation and rendering range delineation of the dirty rectangular region

[0091] After receiving the incremental command, the backend engine compares the state of the signal source before and after, calculates the joint bounding box of the old and new positions, and thus defines the "dirty rectangular area" in the virtual canvas that needs to be recalculated.

[0092] The system strictly limits the GPU's rendering viewport to the dirty rectangular area, thereby avoiding a full cleanup and redraw of the entire 8K or 4K virtual canvas and minimizing the amount of rendering pixel operations caused by a single layout adjustment.

[0093] Sub-step S33: Local multi-layer blending and special effects processing

[0094] Within the defined dirty rectangular area, the system directly reuses the globally unique synthesized frame data residing in the video memory in step S2 as the underlying base.

[0095] Background layer restoration: If the signal source is displaced, the system first extracts the original texture of the occluded area from the underlying substrate to restore the background.

[0096] Layer overlay blending: Maps the altered signal source texture (which may include HDMI video streams, web images, caption bitmaps, etc.) to the dirty rectangular area according to the new coordinates and layer order.

[0097] Pixel-level blending: For this local area, the GPU shader is called to perform special effects processing such as alpha blending, chroma keying (such as removing green screen background) or mask overlay to generate a locally updated image.

[0098] Sub-step S34: In-situ update of synthesized frame data

[0099] After the local blending process is completed, the system uses a local write operation on the frame buffer object (FBO) to directly overwrite the corresponding position of the globally unique composite frame data in the video memory with the new pixel data within the aforementioned dirty rectangular area. This "in-situ update" process does not change the address of the global texture pointer, enabling the downstream multi-channel scaling output module (step S4) to directly obtain the complete composite image with the latest layout when the next vertical sync signal arrives.

[0100] Step S4: Obtain extended display identification data (EDID) information of multiple target display devices. Based on the resolution parameters in the EDID information of each target display device, calculate the optimal output resolution corresponding to each target display device. Without re-executing the rendering action of multiple input data, use an image resampling algorithm to independently scale and adapt the globally unique synthetic frame data to each optimal output resolution, and output it to the corresponding target display devices for display.

[0101] This step is specifically as follows:

[0102] This step aims to resolve the technical contradictions in traditional multi-screen systems when mixed access to screens of different resolutions (such as 4K and 1080p), where a single output resolution leads to image stretching and black borders, or where independent rendering for each screen results in computational exhaustion. As a closed loop of the "render once, adapt multiple times" architecture, this step's implementation includes the following sub-steps:

[0103] Sub-step S41: Dynamic resolution of multiple EDID channels and negotiation of optimal output resolution

[0104] The display output layer polls in real time or obtains Extended Display Identification Data (EDID) of multiple target display devices currently online through Plug and Play (HPD) interrupt events.

[0105] Unlike existing technologies that directly use the preferred timing from the EDID, this system establishes a dynamic negotiation mechanism: parsing the Detailed Timing Descriptor (DTD) and Established Time (EST) in the EDID to extract the maximum physical resolution and color space (such as RGB / YCbCr) supported by each screen. If a screen does not support the baseline resolution of the current virtual canvas, the system queries the built-in resolution mapping table to calculate an "optimal output resolution" for that screen that matches both the native pixels of its physical panel and maintains logical consistency with the aspect ratio of the global virtual canvas (for example, when the screen's native resolution is 1920×1080, its optimal output area is determined to be the corresponding segment in the composite frame).

[0106] Sub-step S42: Zero-copy reuse of global composite frame texture pointers

[0107] This is the core of computing power reuse. When performing scaling output, the system strictly cuts off the reconnection to the GPU rendering pipeline. The system no longer submits any vertex data or texture input requests to the GPU, but instead directly locks the memory pointer of the "globally unique composite frame data" generated in video memory in steps S2 and S3 and continuously updated in situ through the direct memory access (DMA) controller or the video post-processing unit (VPP).

[0108] This mechanism ensures that no matter how many external screens are connected, the system always reads only this one copy of the video memory texture, completely eliminating the huge overhead of the GPU repeatedly performing rasterization and hybrid operations in multi-screen scenarios.

[0109] Sub-step S43: Multi-channel parallel resampling and independent adaptation to different resolutions

[0110] The system establishes an independent hardware calibration channel for each target display device. Each channel reads the same global composite frame data in parallel, but executes different resampling instructions based on its negotiated optimal output resolution parameters.

[0111] Region cropping and coordinate mapping: Each channel extracts the corresponding source pixel region from the global composite frame according to the current multi-screen topology mode (such as stitching, cloning, and expansion).

[0112] Adaptive algorithm call: For high-resolution target screens (such as 4K), the Lanczos resampling algorithm is called for high-quality scaling to maintain image sharpness; for low-resolution screens (such as 1080p), the bilinear interpolation algorithm is called for fast and smooth scaling to reduce the computational load on the hardware scaler.

[0113] Pixel format conversion: The RGB format of the globally synthesized frame is converted into the corresponding HDMI output format (such as YCbCr 4:4:4 or 4:2:2) according to the color capabilities declared by each screen's EDID.

[0114] Sub-step S44: Multi-screen timing alignment and physical output

[0115] The resampling results for each channel are written to an independent output frame buffer. The system uses a unified vertical synchronization signal (V-Sync) as a reference to control each output interface (such as HDMI / DP transmitter) to synchronously push the adapted, different resolution images to the corresponding physical display devices within the same frame period.

[0116] Preferably, in step S1, the timing synchronization and alignment processing of the input data using an adaptive buffering strategy specifically includes:

[0117] Analyze the acquisition delay, network transmission delay, and decoding delay of different signal sources, and establish a timing delay model;

[0118] Real-time monitoring of the delay gradient change between the arrival timestamp of each input data source and the global timestamp;

[0119] When the delay gradient change exceeds the preset jitter threshold, it is determined that the signal source has network jitter, the compensation time is calculated and the buffer queue depth corresponding to the signal source is dynamically expanded to absorb network jitter;

[0120] When the delay gradient recovers to a stable range, the buffer queue depth is reduced by a preset step size to reduce end-to-end latency and ensure that multiple data frames are strictly aligned at the output.

[0121] Preferably, in step S4, the process of independently scaling and adapting the synthesized frame data using an image resampling algorithm specifically includes:

[0122] The list of physical resolutions and color space capabilities supported by each target display device are extracted from the EDID information.

[0123] Look up the recommended output parameters that match each target display device in the preset resolution mapping table;

[0124] For each target display device, an independent output channel is established, and the bilinear interpolation algorithm or Lanczos resampling algorithm is called to resample the globally unique synthetic frame data into a target resolution format that matches each output channel.

[0125] Preferably, in step S3, performing screen blending processing within the virtual canvas according to the layout configuration instructions specifically includes:

[0126] A visual layout editor is provided on the web, which receives the position coordinates, size, layer order and transparency parameters of each signal source in the virtual canvas configured by the user through drag and drop.

[0127] Based on the hierarchical order and transparency parameters, alpha blending, chroma keying, or mask overlay processing are performed on video frames from multiple input data sources to generate a composite image with a multi-layered structure.

[0128] Preferably, it also includes a real-time voice-to-text overlay step, specifically including:

[0129] Extract the audio stream corresponding to the audio acquisition signal, use the integrated speech recognition engine for real-time recognition, and generate subtitle text with a time stamp that matches the global timestamp;

[0130] Convert the subtitle text rendering to a bitmap image;

[0131] Based on the user-configured subtitle style parameters, the bitmap image is frame-level aligned and superimposed with the video frame with the corresponding timestamp in the virtual canvas according to the time stamp, so that the subtitle screen in the synthesized video stream does not have a relative delay exceeding the human eye perception threshold when output to the display device, while the corresponding HDMI video screen and web screen are output.

[0132] Preferably, it also includes a multi-screen dynamic topology response step:

[0133] During continuous video stream output, if a new target display device is detected to be connected or an existing display device is disconnected, the EDID information of the currently online device is reread through a plug-and-play event.

[0134] Based on the updated EDID information, the corresponding output channels and scaling adaptation tasks are dynamically created or destroyed, while the rendering process of the virtual canvas and the generation of composite frame data are kept uninterrupted, thus achieving real-time seamless reconstruction of multi-screen heterogeneous topology.

[0135] Preferably, it also includes remote control steps based on a plug-in architecture:

[0136] The signal input, image processing, and display output modules are encapsulated as plug-ins with standardized interfaces;

[0137] The remote control layer uses a dual-protocol communication mechanism combining RESTful API and WebSocket to receive remote commands;

[0138] Based on remote instructions, the system dynamically executes plugin loading, unloading, hot updates, or version rollback operations without interrupting the main video processing stream and the generation of composite frame data.

[0139] Secondly, the present invention provides a multi-screen, heterogeneous resolution adaptive video processing system, comprising:

[0140] The signal access layer is used to receive HDMI audio and video signals, web page content, and audio acquisition signals. It adds a unified timestamp to each signal and performs timing synchronization alignment by dynamically adjusting the buffer queue depth based on the delay gradient.

[0141] The intelligent canvas engine layer is used to create a virtual canvas independent of the physical resolution of the display device, to uniformly render synchronized multi-channel data to generate globally unique composite frame data, and to perform blending processing within the virtual canvas according to the layout configuration instructions on the web side.

[0142] The display output layer is used to obtain EDID information of multiple target display devices, calculate the optimal output resolution for each device, and independently scale and adapt the synthesized frame data to each display device without re-executing the rendering process.

[0143] The remote control layer is used to manage the lifecycle of functional plugins with standardized interfaces through a dual-protocol communication mechanism.

[0144] Preferably, the system also includes:

[0145] The subtitle processing module, integrated into the intelligent canvas engine layer, is used to recognize the audio stream in real time, generate subtitle text with time stamps, convert the subtitle text into bitmap images, and then perform frame-level alignment and overlay with the corresponding video frames in the virtual canvas according to the time stamps.

[0146] In summary, this invention overcomes the inherent shortcomings of traditional multi-screen video processing technologies in terms of computing power allocation, signal synchronization, and resolution adaptation through a collaborative architecture of "virtual canvas decoupled rendering" and "globally unique composite frame multi-path independent scaling," achieving the following significant beneficial effects:

[0147] 1. Heterogeneous multi-source precise synchronization, completely eliminating audio-visual tearing and stuttering.

[0148] Unlike existing synchronization methods that rely on fixed delays or simple frame drops, this invention introduces an adaptive buffering strategy based on delay gradients. For low-latency HDMI streams and high-jitter Web streams, it effectively absorbs network jitter through cross-media collaborative compensation of "dynamic video queue scaling + audio time compression," ensuring strict frame-level alignment of heterogeneous signals in complex network environments and greatly improving picture continuity and audio-visual synchronization quality when mixed signal inputs are used.

[0149] 2. High reuse of GPU computing power breaks through the performance bottleneck of multi-screen rendering.

[0150] This invention completely decouples "image content compositing" from "physical resolution output" at the GPU pipeline level. Regardless of the number of external display devices, the system performs rendering only once in the virtual canvas to generate globally unique composite frame data. The downstream scaling process is completed by reusing memory texture pointers and hardware scalers, achieving zero additional GPU rendering overhead. This prevents the system's computing power from increasing linearly or exponentially with the number of screens, fundamentally solving the problem of traditional multi-screen matrix systems easily crashing or dropping frames due to computing power exhaustion when fully loaded.

[0151] 3. Seamless and lossless adaptation to different resolutions, eliminating physical image distortion.

[0152] For scenarios involving multiple screens with varying resolutions such as 4K and 1080p, this invention dynamically analyzes the EDID information of each screen to calculate the optimal output resolution for each, and applies different resampling algorithms in parallel to scale the same composite frame data. This mechanism avoids the forced stretching, black borders, or cropping issues caused by traditional single-resolution output, achieving lossless point-to-point adaptation for physical displays with varying resolutions across multiple screens, resulting in a clearer and more natural image.

[0153] 4. A local incremental redraw mechanism ensures extremely low-latency interaction in layout configuration.

[0154] When responding to dragging, zooming, and other screen layout adjustments by web users, this invention employs a technical approach of "incremental instruction transmission + dirty rectangle region delimitation + in-situ update of video memory," compressing the global full redraw into updates of extremely small local pixel blocks. Even in scenarios with multiple 4K high-load signals, it maintains a high frame rate and real-time interactive experience, completely eliminating the operational latency and screen flickering inherent in traditional video processors when configuring the screen.

[0155] 5. The architecture is completely decoupled, supporting seamless dynamic topology reconfiguration.

[0156] Due to the decoupling of the rendering and output layers, when a display device is hot-plugged (connected or disconnected), the system only needs to reread the EDID and dynamically create or destroy the corresponding scaling output channel via a plug-and-play event. The rendering process of the virtual canvas and the generation of composite frame data remain completely unaffected and uninterrupted. This achieves zero-latency, seamless real-time reconstruction of multi-screen heterogeneous topologies, significantly improving the system's stability and flexibility in complex meeting or command and dispatch scenarios.

[0157] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-screen, heterogeneous resolution adaptive video processing method, characterized in that, Includes the following steps: Step S1: Receive input data from multiple heterogeneous signal sources, including HDMI audio and video signals, web page content, and audio acquisition signals; A unified global timestamp is added to each input data source, and a buffer queue is established based on the inherent delay characteristics of each signal source. The input data is then processed for timing synchronization and alignment using an adaptive buffering strategy. Step S2: Create a virtual canvas independent of the physical resolution of the target display device, and render the time-synchronized and aligned multi-channel input data onto the virtual canvas to generate globally unique composite frame data; Step S3: Receive the screen layout configuration instruction submitted by the user through the Web terminal, perform position layout and screen mixing processing on the multi-channel input data in the virtual canvas according to the layout configuration instruction, and update the composite frame data; Step S4: Obtain Extended Display Identification Data (EDID) information of multiple target display devices, and calculate the optimal output resolution corresponding to each target display device based on the resolution parameters in the EDID information of each target display device; Without re-executing the rendering of the multi-channel input data, an image resampling algorithm is used to independently scale and adapt the globally unique synthetic frame data to each optimal output resolution, and output it to the corresponding target display devices for display.

2. The multi-screen, heterogeneous resolution adaptive video processing method according to claim 1, characterized in that, In step S1, the timing synchronization and alignment processing of the input data using an adaptive buffering strategy specifically includes: Analyze the acquisition delay, network transmission delay, and decoding delay of different signal sources, and establish a timing delay model; Real-time monitoring of the delay gradient change between the arrival timestamp of each input data stream and the global timestamp; When the delay gradient change exceeds the preset jitter threshold, it is determined that the signal source has network jitter, the compensation time is calculated and the buffer queue depth corresponding to the signal source is dynamically expanded to absorb the network jitter; When the delay gradient recovers to a stable range, the depth of the buffer queue is reduced by a preset step size to reduce end-to-end latency and ensure that multiple data frames are strictly aligned at the output.

3. The multi-screen, heterogeneous resolution adaptive video processing method according to claim 1, characterized in that, In step S4, the image resampling algorithm is used to independently scale and adapt the synthesized frame data, specifically including: The list of physical resolutions and color space capabilities supported by each target display device are parsed from the EDID information. Look up the recommended output parameters that match each target display device in the preset resolution mapping table; For each target display device, an independent output channel is established, and the bilinear interpolation algorithm or Lanczos resampling algorithm is called to resample the globally unique synthetic frame data into a target resolution format that matches each output channel.

4. The multi-screen, heterogeneous resolution adaptive video processing method according to claim 1, characterized in that, In step S3, performing screen blending processing within the virtual canvas according to the layout configuration instructions specifically includes: A visual layout editor is provided on the web, which receives the position coordinates, size, layer order and transparency parameters of each signal source in the virtual canvas configured by the user through drag and drop. Based on the hierarchical order and transparency parameters, alpha blending, chroma keying, or mask overlay processing are performed on video frames of multiple input data to generate a composite image with a multi-layered structure.

5. The multi-screen, heterogeneous resolution adaptive video processing method according to claim 4, characterized in that, It also includes a real-time audio-visual caption overlay process, specifically including: The audio stream corresponding to the audio acquisition signal is extracted, and the integrated speech recognition engine is used for real-time recognition to generate subtitle text carrying a time stamp that matches the global timestamp; The subtitle text is rendered and converted into a bitmap image; Based on the user-configured subtitle style parameters, the bitmap image and the video frame with the corresponding timestamp in the virtual canvas are frame-level aligned and superimposed according to the timestamp, so that the subtitle screen in the synthesized video stream and the corresponding HDMI video screen and web screen do not have a relative delay exceeding the human eye perception threshold when output to the display device.

6. The multi-screen, heterogeneous resolution adaptive video processing method according to claim 1, characterized in that, It also includes multi-screen dynamic topology response steps: During continuous video stream output, if a new target display device is detected to be connected or an existing display device is disconnected, the EDID information of the currently online device is reread through a plug-and-play event. Based on the updated EDID information, the corresponding output channels and scaling adaptation tasks are dynamically created or destroyed, while the rendering process of the virtual canvas and the generation of composite frame data are kept uninterrupted, thus realizing real-time seamless reconstruction of multi-screen heterogeneous topology.

7. The multi-screen, heterogeneous resolution adaptive video processing method according to claim 1, characterized in that, It also includes remote control steps based on a plug-in architecture: The signal input, image processing, and display output modules are encapsulated as plug-ins with standardized interfaces; The remote control layer uses a dual-protocol communication mechanism combining RESTful API and WebSocket to receive remote commands; According to the remote command, the loading, unloading, hot update, or version rollback operations of the plugin are dynamically executed without interruption of the main video processing stream and the generation of composite frame data.

8. A multi-screen, heterogeneous resolution adaptive video processing system, characterized in that, include: The signal access layer is used to receive HDMI audio and video signals, web page content, and audio acquisition signals. It adds a unified timestamp to each signal and performs timing synchronization alignment by dynamically adjusting the buffer queue depth based on the delay gradient. The intelligent canvas engine layer is used to create a virtual canvas independent of the physical resolution of the display device, to uniformly render synchronized multi-channel data to generate globally unique composite frame data, and to perform blending processing within the virtual canvas according to the layout configuration instructions on the web side. The display output layer is used to obtain EDID information of multiple target display devices, calculate the optimal output resolution for each device, and independently scale and adapt the synthesized frame data to each display device without re-executing the rendering process. The remote control layer is used to manage the lifecycle of functional plugins with standardized interfaces through a dual-protocol communication mechanism.

9. The multi-screen, heterogeneous resolution adaptive video processing system according to claim 8, characterized in that, The system also includes: The subtitle processing module, integrated into the intelligent canvas engine layer, is used to recognize the audio stream in real time, generate subtitle text with time stamps, convert the subtitle text into bitmap images, and then perform frame-level alignment and overlay with the corresponding video frames in the virtual canvas according to the time stamps.

Images