A method for multi-screen fusion and adaptive window arrangement of TV wall for heterogeneous decoding terminals

Abstract

The application discloses a TV wall multi-screen fusion and adaptive window arrangement method for heterogeneous decoding terminals, and belongs to the technical field of video monitoring and display control. The application solves the problems of non-uniform screen identification system, poor cross-resolution layout precision, split business mode data link, insufficient fusion screen execution mapping and difficult window position recovery in the prior art by adopting the core technologies of global state management through a unified state management layer, screen unified semantic modeling, normalized coordinate conversion, window logical serial number mapping and multi-operation mode linkage scheduling. The application realizes unified scheduling of heterogeneous decoding terminals, semantic modeling of multi-screen objects, integrated processing of multi-business modes, accurate mapping of window coordinates and linkage synchronization of operation states, and significantly improves the control precision, operation efficiency and interactive stability of the TV wall system in a large-scale complex heterogeneous environment.

Description

Technical Field

[0001] This invention belongs to the field of video surveillance and display control technology, specifically relating to a method for multi-screen fusion and adaptive window arrangement of a video wall for heterogeneous decoding terminals. Background Technology

[0002] With the rapid development of video surveillance technology and the continuous deepening of security system construction, video walls, as the core display carrier of video surveillance systems, have been widely used in various scenarios such as command centers, urban smart systems, park security, and emergency dispatch. In these scenarios, video wall systems typically need to perform the following tasks: select video channels from the device tree or signal source, project the channels onto a designated screen or window, and support operations such as split-screen, windowing, rotation, playback, and temporary display.

[0003] Since video walls are often composed of multiple physical screens, merged screens, and heterogeneous decoding devices from different manufacturers, there are usually problems such as inconsistent representation, fragmented control links, and difficulty in synchronizing states between the front-end visual orchestration and the back-end device execution. Currently, the closest existing solution to this invention is usually a "fixed-rule video wall control platform," the typical implementation of which is as follows: 1. First, the user selects the video wall, and then selects the task or plan to be executed.

[0004] 2. The system generates several fixed split-screen windows (e.g., 1-split-screen, 4-split-screen, 9-split-screen) based on the static screen configuration saved in the background.

[0005] 3. Users can manually select camera channels from the device tree and assign them to a specific screen or sub-window.

[0006] 4. If a temporary display is required, the system will execute a separate "temporary display" process; if task-based management is required, it needs to be saved as a task separately.

[0007] 5. If geometric controls such as opening windows, dragging, scaling, and keeping windows on top are required, the physical coordinates of a window on a specific screen are usually controlled directly through the manufacturer's device SDK or backend interface.

[0008] 6. If polling is required, configure the polling order, dwell time, and stream type for each window separately.

[0009] 7. If there are merged screens, the existing system often handles merged screens and ordinary screens separately, and even requires maintenance personnel to manually maintain the correspondence between logical screen numbers and physical screen numbers.

[0010] 8. If the front-end page allows drag-and-drop interaction, it often only manifests as changes in browser pixel coordinates. However, back-end device execution requires parameters such as normalized coordinates, device number, and logical screen number, thus requiring complex secondary calculations.

[0011] The core problem with existing solutions of this kind is that the screen model, window model, channel model, and execution instruction model are usually not based on the same data structure. This means that the "visible layout" may not necessarily be "accurately executed," and the three modes of task display, temporary display, and real-time display are often disconnected from each other. Most similar traditional implementations in the industry focus on visual display, while the actual business execution relies heavily on decentralized backend interfaces and lacks a unified screen semantic mapping and write-back mechanism.

[0012] In summary, the existing technology has the following significant drawbacks: 1. Inconsistent identification system: Ordinary screens, merged screens, temporary screens, and open windows are usually handled separately, resulting in multiple numbering systems for the same video wall system, making it difficult to stably correspond between the front-end layout and the back-end execution objects.

[0013] 2. Poor layout accuracy: Window positions are often directly assigned using front-end pixel coordinates or simple ratios. Under different resolutions, container sizes, and blending relationships, windows are prone to offset, out-of-bounds, stretching distortion, and cannot guarantee that the layout result is consistent with what the user sees.

[0014] 3. Fragmented business models: Task display, temporary display and instant display often go through different data links, which means that the same channel needs to be modeled repeatedly in different modes, the operation process is fragmented and the status is difficult to synchronize.

[0015] 4. Insufficient depth of processing for merged screens: The processing of merged screens mostly stays at the display layer, without uniformly mapping child screens, parent screens, device codes, and logical screen numbers. When performing operations such as opening windows, moving, and rotating in merged screen scenarios, instructions are easily sent to the wrong objects.

[0016] 5. Lack of position restoration mechanism: Although split-screen is supported, there is no unified coordinate restoration mechanism between actions such as zooming in, restoring, dragging, and window overlaying in split-screen mode. This often results in the inability to accurately restore the original grid position after the user zooms in.

[0017] 6. Inefficient channel rotation management: Channel rotation typically only manages "playback content" without achieving a complete binding relationship between "window-channel-sequence-dwell time-bitrate". This results in an insufficiently refined rotation plan and difficulty in automatically synchronizing with window switching.

[0018] 7. Window opening control conflicts: Window opening control is mostly a single-point operation, lacking constraints on the number of windows, default placement strategy and top placement strategy, which can easily cause screen obstruction and control conflicts.

[0019] In summary, the inconsistent data structures of screen models, window models, channel models, and execution instruction models in video wall systems, coupled with the fragmentation of the three business modes—task-based display, temporary display, and real-time display—lead to a series of technical bottlenecks, including control link redundancy, difficulties in state synchronization, and display layout distortion. Therefore, achieving unified scheduling of heterogeneous decoding terminals, unified semantic modeling of multi-screen objects, precise control of window coordinates, and synchronized operation states has become a pressing technical challenge for those skilled in the art. Summary of the Invention

[0020] The purpose of this invention is to address the shortcomings of existing video wall control systems, such as inconsistent screen identification systems, poor accuracy and easy distortion of cross-resolution display layout, fragmented data links for various business modes (task / temporary / instant display), insufficient depth of execution mapping at the fusion screen's underlying layer, and difficulty in restoring window geometric positions. This invention provides a method for multi-screen fusion and adaptive window arrangement in video walls for heterogeneous decoding terminals. By constructing a unified state management model that includes screen modeling, window arrangement, geometric control, channel binding, and state synchronization, and utilizing a high-precision normalized coordinate conversion mechanism, window logical index number mapping logic, and heterogeneous screen semantic mapping strategy, this method solves the semantic inconsistency and control link fragmentation problems between the front-end visual arrangement and the back-end heterogeneous hardware execution layer. It achieves unified scheduling of heterogeneous decoding terminals, deep semantic modeling of multi-screen display objects, integrated closed-loop processing of various business modes, accurate mapping of window geometric coordinates, and linkage synchronization of multiple concurrent operation states. This significantly improves the control accuracy, operating efficiency, and interactive stability of video wall systems in large-scale, complex, heterogeneous environments, ensuring a high degree of consistency between the front-end visual arrangement layout results and the back-end hardware execution results.

[0021] To achieve the above objectives, this invention provides a method for multi-screen fusion and adaptive window arrangement of a video wall for heterogeneous decoding terminals. This method manages the global state of the video wall system through a unified state management layer, and includes the following steps: Step S1: State initialization and screen modeling; Read the TV wall screen configuration information and perform unified semantic modeling on ordinary physical screens and converged screens; calculate the coverage area of ​​the converged screen in the physical coordinate system and mark the ordinary physical screens within the coverage area as covered; sort all uncovered top-level screen objects according to their physical identifiers to establish a mapping relationship from screen service identifiers to temporary screen identifiers. Step S2: Assign logical numbers to the fused screen; For the fused screen, the temporary screen identifiers of all the sub-screens it covers are aggregated to establish a sub-screen mapping relationship set, and the extreme value of the sub-screen mapping relationship set is taken as the logical execution number of the fused screen. Step S3: Logical layout generation; In response to layout orchestration instructions, the logical grid size is determined based on the number of screens, multiple logical grid windows are generated within the target screen, and a globally unique window logical sequence number is assigned to each logical grid window; Step S4: Normalized coordinate transformation; Capture operation events of the front-end display viewport, project the absolute pixel coordinates generated by the operation events onto the discrete resolution reference space defined by a preset normalization upper limit constant, convert them into normalized execution coordinates decoupled from the display device resolution, and perform boundary clipping processing; Step S5: Data model construction and group scheduling; A unified window-channel data model is constructed, which includes the logical execution number, window logical sequence number, and video channel attributes. All logical grid window data under the same screen are flattened into a window dataset. The window dataset is grouped according to the physical device identifier of the heterogeneous decoding terminal, and the generated execution parameter sequence is sent to the corresponding heterogeneous decoding terminal. Step S6: Multi-mode linkage scheduling and state closure; The unified state pool maintained by the unified state management layer enables coordinated scheduling of three operating modes: task display, temporary display, and instant display. After a command is successfully issued, the window hardware identifier fed back by the heterogeneous decoding terminal is received and written back to the unified state pool, thereby achieving state synchronization between front-end business orchestration and back-end hardware execution.

[0022] Furthermore, it also includes the rendering pruning step: When rendering the TV wall layout at the front end, the ordinary physical screens marked as covered are skipped, and only the top-level screen object is rendered; when the heterogeneous decoding terminal executes control commands, the sub-screen mapping relationship set is used to penetrate and map to the specific sub-screen physical execution unit.

[0023] Furthermore, the calculation logic for converting to normalized execution coordinates in step S4 is as follows: Calculate the ratio between the absolute pixel coordinates and the current front-end display viewport size, and map the ratio to the quantization space defined by the preset normalization upper limit constant. Discretize the coordinates using a rounding function.

[0024] Furthermore, it also includes a step for accurate window geometry reconstruction based on the window logical sequence number: In response to the grid restoration command, the current screen's logical grid size is obtained; based on the target window's logical index and the logical grid size, mathematical modulo and rounding operations are performed to deduce the target window's row and column index positions in the original grid; the restored absolute pixel coordinates are calculated based on the row and column index positions, and then projected twice into normalized execution coordinates for distribution.

[0025] Furthermore, the coordinated scheduling of the three operating modes in step S6 specifically includes: When executing the task of displaying the video on the wall, the preset video wall number and task number are directly issued; When performing temporary wall display, the screen data is organized in real time and the logic execution number is calculated. The window channel data is flattened, and the sub-screen mapping relationship set is attached to the fused screen before the execution parameter sequence is issued. When performing instant video display, in response to the addition of a new video channel to the existing logical segment window, an incremental delivery command is directly sent to the corresponding heterogeneous decoding terminal without having to re-execute the complete video display process.

[0026] Furthermore, it also includes the procedure for preserving channels during layout changes: If the target screen has already bound video channels before the grid layout is changed, the window logical number of each window in the original layout is extracted, and the corresponding video channel list is migrated to the window object with the same window logical number in the new layout, so as to realize the logical binding between window position and channel attribute.

[0027] Furthermore, it also includes anti-collision control steps for overlapping window openings: Before performing the window overlay operation, count the number of existing overlay windows in the target screen. If the number reaches the preset limit, the window opening command is intercepted; otherwise, the initial normalized coordinates are assigned to the newly opened window according to the preset container width-to-height ratio and placement logic.

[0028] Furthermore, it also includes the steps of deduplication and automatic allocation of video channels: The video channels are checked for duplicates using the device code and channel number as the sole criteria for deduplication. Channels that pass the check are automatically assigned a round-robin sequence number, a default dwell time, and a bitstream attribute. These attributes are then encapsulated into the unified window-channel data model.

[0029] Furthermore, the data grouping based on the physical device identifier of the heterogeneous decoding terminal in step S5 specifically includes: Collect all pending window and channel data involved in the current display task; group the data according to the unique device identifier of the decoding terminal, generate an independent instruction payload for each device group, and distribute it to the corresponding heterogeneous decoding terminal through an asynchronous queue.

[0030] Furthermore, the global state management of the video wall system through a unified state management layer specifically includes: The unified state pool maintains the video wall identifier, task identifier, window position status, channel binding relationship, and layout modification handle in real time. When any state changes, the associated state is automatically updated synchronously to ensure the consistency of screen modeling results, layout data, geometric parameters, channel binding, and execution status in the global scope.

[0031] Compared with the prior art, the present invention has the following significant advantages: This invention establishes a mapping relationship from screen service identifiers to temporary screen identifiers by performing unified semantic modeling on ordinary physical screens and converged screens. This completely eliminates the problem of unstable correspondence between front-end layouts and back-end execution objects caused by the coexistence of multiple numbering systems in existing technologies. This unified identifier system enables standardized management of all display objects in the video wall system, significantly improving the maintainability and control accuracy of the system in large-scale and complex scenarios.

[0032] By employing a normalized coordinate conversion mechanism, the absolute pixel coordinates of the front end are projected onto a discrete quantization space, converting them into normalized execution coordinates decoupled from the display device resolution. This mechanism fundamentally solves technical bottlenecks such as window offset, out-of-bounds movement, and stretching distortion that easily occur under different resolutions and container sizes, ensuring a high degree of consistency between layout results and user operation intentions across devices and scenarios, and significantly improving the accuracy of visual presentation.

[0033] This invention uses a unified state pool to coordinate and schedule three operating modes: task display, temporary display, and instant display. This eliminates the shortcomings of existing technologies, such as fragmented data links between different modes, difficulty in synchronizing states, and the need for repetitive modeling. This integrated processing mode greatly simplifies the system architecture, reduces redundant data processing steps, and improves user operation response speed while ensuring a high degree of consistency in the overall business state of the system.

[0034] This invention achieves deep penetration mapping between the fused screen and the physical execution units of the sub-screens by establishing a sub-screen mapping relationship set and extracting the logic execution number. This solves the problem that existing processing often remains at the display layer, leading to instructions being easily sent to the wrong physical objects. It ensures that in complex fused screen merging and segmentation scenarios, various operations such as windowing, movement, and polling can be accurately sent to the underlying hardware.

[0035] By assigning globally unique serial numbers to logically segmented windows and combining mathematical algorithms to reverse-derive the row and column positions of windows, this invention solves the pain point in the industry of difficulty in accurately restoring the original grid position after split-screen zooming or dragging, achieving accurate restoration of the window's geometric position and ensuring the logical rigor and ease of operation of the TV wall layout structure in complex interactive processes.

[0036] By constructing a unified window and channel data model, this invention achieves complete binding relationship management of windows, channels, polling order, dwell time, and stream attributes. It solves the problems of coarse polling management and the inability to automatically synchronize with window layout changes in traditional solutions, supporting fine-grained configuration and automated execution of polling plans, and greatly improving the business reliability of video surveillance scheduling.

[0037] This invention achieves integrated control logic, including window quantity constraints, default placement strategies, and state closed-loop write-back, through a unified state management layer. This effectively solves the control conflicts and screen obstruction problems that may arise during concurrent operations. While ensuring stable system operation under multi-user, multi-task concurrent operations, it significantly enhances the system's throughput and user interaction experience under high-pressure environments.

[0038] This invention systematically solves all the technical defects existing in the existing video wall control system by constructing a unified state management architecture, and comprehensively improves the control accuracy, operating efficiency and interactive stability of the video wall system in a large-scale, complex and heterogeneous environment. It provides an industrial-grade video scheduling solution for various application scenarios such as command centers, urban brains, park security, and emergency dispatch. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural diagram of the system of the present invention; Figure 3 This is a flowchart of the unified mapping process between the fused screen and the ordinary screen in this invention; Figure 4 This is a flowchart illustrating the specific implementation process of the present invention. Detailed Implementation

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment provides a method for multi-screen fusion and adaptive window arrangement of a video wall for heterogeneous decoding terminals. It uses a unified state management layer to manage the global state of the video wall system. This unified state management layer is implemented using a memory-level state pool, supporting high-speed state read / write and synchronization to ensure real-time consistency between front-end interaction and back-end execution. Specifically, it includes the following steps: Step S1: State initialization and screen modeling; The system reads the configuration information of the video wall screens and performs unified semantic modeling on ordinary physical screens and converged screens. This unified semantic modeling adopts an object-oriented abstraction approach, separating the common and differential attributes of different screen types to achieve a unified data structure representation for multiple screen types. The system calculates the coverage area of ​​the converged screens in the physical coordinate system and marks ordinary physical screens within this coverage area as covered. All uncovered top-level screen objects are sorted according to their physical identifiers to establish a mapping relationship from screen service identifiers to temporary screen identifiers. This mapping relationship employs an efficient storage structure, supporting fast lookup and conversion. Through the marking of "covered state" and a unified object-oriented representation, semantic alignment of heterogeneous screens is achieved at the underlying logic level. This effectively avoids display conflicts between the merged screen and the ordinary physical screen it covers, ensuring that the same physical area is controlled by only one logical display object. This pre-eliminates redundant physical screen objects and improves the efficiency of screen initialization in heterogeneous environments.

[0042] Step S2: Assign logical numbers to the fused screen; For the fused screen, the temporary screen identifiers of all the sub-screens it covers are aggregated to establish a sub-screen mapping relationship set. The sub-screen mapping relationship set supports both sequential traversal and fast search by sub-screen identifier. The extreme value in the sub-screen mapping relationship set is taken as the logical execution number of the fused screen. This step establishes a unique logical handle for the complex fusion screen using the extreme value method, ensuring the uniqueness and stability of the number, avoiding conflicts with other ordinary screens, and facilitating the backend decoding device to quickly identify the fusion screen type. This also solves the command issuance conflicts caused by inconsistent identification when decoding chips from different manufacturers process fusion screens.

[0043] Step S3: Logical layout generation; In response to layout and arrangement instructions, the logical grid size is determined based on the number of screens. The logical grid size is automatically calculated using a corresponding algorithm to ensure that the aspect ratio of the grid layout matches the physical screen size. Multiple logical grid windows are generated within the target screen, and each logical grid window is assigned a globally unique window logical sequence number. The window logical sequence number is assigned using consecutive integers, serving as both a unique identifier and implicitly containing the window's position information within the grid. This transforms the physical screen space into a business-controllable digital matrix through the logical grid coordinate system, providing a standardized data foundation for subsequent rapid window reconstruction based on pure mathematical derivation.

[0044] Step S4: Normalized coordinate transformation; Capture operation events of the front-end display viewport, project the absolute pixel coordinates generated by the operation events onto the discrete resolution reference space defined by a preset normalization upper limit constant, convert them into normalized execution coordinates decoupled from the display device resolution, and perform boundary clipping processing; The discrete resolution reference space uses an integer coordinate system to avoid precision errors caused by floating-point arithmetic, and the boundary clipping process ensures that all coordinate values ​​fall within the legal range. This step decouples the front-end UI interaction from the back-end hardware parameters through a normalization algorithm, fundamentally eliminating the problem of windows displaying out of bounds and eliminating window offset and stretching distortion caused by changes in front-end container size or inconsistent terminal resolution.

[0045] Step S5: Data model construction and group scheduling; A unified window-channel data model is constructed, which includes the logical execution number, window logical sequence number, and video channel attributes. The model adopts a flat design, supports serialization and deserialization, and facilitates network transmission and persistent storage. All logical grid window data under the same screen are flattened into a window dataset. The window dataset is grouped according to the physical device identifier of the heterogeneous decoding terminal, and the generated execution parameter sequence is sent to the corresponding heterogeneous decoding terminal. The data grouping employs a corresponding grouping algorithm to achieve uniform distribution of instruction load. This flattened design transforms the complex tree-like UI structure into a linear instruction queue recognizable by heterogeneous decoders, greatly simplifying the complexity of distribution logic in heterogeneous environments, enabling parallel execution across multiple devices, and improving the concurrent scheduling capabilities of large-scale video wall systems.

[0046] Step S6: Multi-mode linkage scheduling and state closure; The unified state pool maintained by the unified state management layer enables coordinated scheduling of three operating modes: task display, temporary display, and instant display. The three operating modes reuse the same data structure and execution link to reduce code redundancy and maintenance costs. After the instruction is successfully issued, the window hardware identifier fed back by the heterogeneous decoding terminal is received and written back to the unified state pool. The state write-back is implemented using an asynchronous callback mechanism, which does not block front-end user interaction and ensures smooth system response. This establishes a complete closed-loop state feedback mechanism, enabling real-time synchronization between front-end business orchestration and back-end hardware execution, and preventing state disconnection.

[0047] As one implementation method, this embodiment also includes a rendering pruning step: When rendering the TV wall layout on the front end, the ordinary physical screens marked as covered are skipped, and only the top-level screen object is rendered; when the heterogeneous decoding terminal executes control commands, the sub-screen mapping relationship set is used to penetrate and map to the specific sub-screen physical execution unit. The pass-through mapping is implemented using a lookup table. Through rendering pruning and pass-through mapping, transparent control of "one logical layer and multiple physical layers" is achieved, significantly reducing the front-end rendering computation, improving page loading speed, and ensuring that operations on the merged screen can be accurately converted into physical operations on the sub-screen.

[0048] As one implementation method, the calculation logic for converting to normalized execution coordinates in step S4 of this embodiment is as follows: Calculate the ratio between the absolute pixel coordinates and the current front-end display viewport size, and map the ratio to the quantization space defined by the preset normalization upper limit constant. Discretize the coordinates using a rounding function.

[0049] The continuous pixel space is mapped to a discrete quantization space by using a rounding function. In this embodiment, the normalization upper limit constant is preferably 10000, so as to ensure the consistency of coordinate representation under different resolutions, and make the window edge positioning still maintain extremely high accuracy in a high-definition display environment.

[0050] As one implementation method, this embodiment also includes a step of accurately restoring the window geometry based on the window logical sequence number: In response to the grid restoration command, the current screen's logical grid size is obtained; based on the target window's logical index and the logical grid size, mathematical modulo and integer operations are performed to deduce the row and column index positions of the target window in the original grid. This step uses pure mathematical derivation to restore the position, without the need to store the window's historical position information. This saves memory storage space and completely solves the pain point that users cannot accurately restore the original grid position after zooming or dragging.

[0051] As one implementation method, the coordinated scheduling of the three operating modes in step S6 of this embodiment specifically includes: When executing the task of displaying the TV wall, the preset TV wall number and task number are directly issued. The task display mode adopts a full configuration pre-loading mechanism to support one-click restoration of the TV wall layout state. When performing temporary wall display, the screen data is organized in real time and the logic execution number is calculated. The window channel data is flattened, and the sub-screen mapping relationship set is attached to the fused screen before the execution parameter sequence is issued. The temporary wall display mode adopts a real-time calculation mechanism to support flexible layout adjustment. When executing instant video display, in response to the addition of a new video channel to the existing logical segment window, an incremental delivery command is directly sent to the corresponding heterogeneous decoding terminal. The instant video display mode adopts an incremental update mechanism, which improves the user experience through differentiated processing and meets the timeliness requirements of different scenarios.

[0052] As one implementation method, this embodiment also includes a channel retention step during layout changes: If the target screen has already bound video channels before the grid layout is changed, extract the window logical number of each window in the original layout and migrate the corresponding video channel list to the window object with the same window logical number in the new layout.

[0053] This automatic channel migration mechanism enables logical binding between window position and channel attributes, eliminating the need for users to rebind video sources when adjusting the layout and improving operational efficiency.

[0054] As one implementation method, this embodiment also includes a collision prevention control step for window overlay: Before performing the window overlay operation, the number of existing overlay windows on the target screen is counted. If the preset limit is reached, the window opening command is intercepted to avoid system performance degradation through a quantity constraint mechanism; otherwise, initial normalized coordinates are assigned to the newly opened window according to the preset container aspect ratio and placement logic. It also employs an intelligent placement algorithm to automatically avoid existing windows, thereby reducing the cost of manual adjustments for users and preventing display clutter.

[0055] As one implementation method, this embodiment also includes a video channel deduplication and automatic allocation step: The video channels are checked for duplicates using the device code and channel number as the sole criteria to prevent the same channel from being added repeatedly in the same window; and the channels that pass the check are automatically assigned round-robin parameters and encapsulated into a unified data model.

[0056] This automatic allocation mechanism simplifies the polling configuration process, allowing users to achieve fine-grained polling without manually configuring parameters, thus reducing configuration complexity.

[0057] As one implementation method, the data grouping according to the physical device identifier of the heterogeneous decoding terminal in step S5 of this embodiment specifically includes: Data is grouped according to the unique device identifier of the decoding terminal, an independent instruction payload is generated for each device group, and distributed to the corresponding heterogeneous decoding terminal through an asynchronous queue.

[0058] An asynchronous queue distribution mechanism is adopted to smooth out peaks and valleys in instruction processing. This improves the efficiency of parallel scheduling of heterogeneous terminals from multiple manufacturers while avoiding device overload caused by excessive instantaneous concurrency.

[0059] As one implementation method, the global state management of the video wall system through a unified state management layer in this embodiment specifically includes: The unified state pool maintains the state of the video wall, tasks, windows, and layout in real time. When any state changes, the associated state is automatically updated synchronously. The publish-subscribe model is used to achieve state linkage, ensuring atomic updates of the state, completely eliminating the problem of state inconsistency, and ensuring high synchronization of screen modeling, geometric parameters, and channel binding in a large-scale environment.

[0060] Example 2

[0061] This embodiment is a supplementary description to the foregoing embodiments. For example... Figure 2 As shown ( Figure 2 (See system architecture diagram). This embodiment of the video wall multi-screen fusion and adaptive window arrangement method for heterogeneous decoding terminals adopts a three-layer architecture design, which is jointly implemented by the front-end visualization arrangement layer, the unified state management layer, and the device execution interface layer. The system performs operations to maintain the global state through the state synchronization module, ensuring strong consistency of the state between each layer and each module.

[0062] I. System Overall Structure and Module Function Implementation; This system includes at least the following core functional modules, which support the entire business chain through logical collaboration: Device Source Access Module: This module enables unified access management of multi-source video signals. By loading the organizational hierarchy, device resource tree, and local signal source set, a complete video channel resource pool is constructed. Technically, this module adopts a hierarchical resource organization approach, standardizing and encapsulating video signals from different sources and of different types. It shields the differences between underlying devices and provides an abstract interface for projectable video channels to the upper-level video wall target window, thereby achieving unified access and management of cross-vendor and cross-protocol video sources and avoiding access complexity caused by device heterogeneity.

[0063] The video wall screen modeling module is used to achieve unified abstraction and modeling of video wall display units, corresponding to step S1 mentioned above. Technically, this module establishes a unified screen description model, uniformly modeling both conventional and integrated display screens. Through spatial positional relationship calculation and coverage detection algorithms, it automatically identifies and marks ordinary physical screen areas covered by integrated screens as "covered," performing intelligent filtering in the rendering and control process, thereby effectively solving the problems of repetitive rendering and control during hybrid deployments.

[0064] A window layout generation module is used to automatically generate an adapted window layout based on the characteristics of the target display area, corresponding to step S3 mentioned above. Technically, this module combines the physical characteristics of the display area with the user's arrangement requirements to automatically generate a segmented window layout, and assigns a globally unique identifier, screen association information, and window logical index to each window. This module supports flexible and diverse window division methods, significantly reducing the complexity of manual arrangement.

[0065] The window geometry control module is used to control the geometric shape and coordinate transformation of the window, corresponding to step S4 mentioned above. Technically, this module provides various geometric operation capabilities such as window creation, position dragging, size scaling, layer adjustment, grid zooming, and restoration. By establishing a mapping and transformation mechanism between the front-end browser pixel coordinates and the back-end device's recognizable coordinates, and utilizing normalization processing to achieve a unified coordinate system, it ensures accurate execution of orchestration operations on heterogeneous decoding terminals.

[0066] The channel binding and polling module, integrated into the unified state management layer, is used to associate and bind video channels with display windows. Technically, this module establishes the binding relationship between video channels and display windows, maintains configuration information such as playback order, switching interval, and bitrate type, and provides a start / stop control mechanism for polling playback. This module supports unified driving of both segmented windows and independent open windows, meeting the needs of multi-video source loop display in monitoring scenarios.

[0067] The video wall display execution module is deployed at the device execution interface layer and is used to execute and distribute video wall display tasks. Technically, this module supports multiple modes, including task-based display, temporary display, and instant display, and reuses the same set of window and channel data models across different modes. The module receives normalized coordinates and task parameters from the upper layer, generates an execution parameter sequence according to the device grouping strategy, and sends execution instructions to the underlying hardware through a standardized interface.

[0068] The State Synchronization Module is used to achieve unified management and synchronization of the system's global state. Technically, this module establishes a global state management mechanism to uniformly maintain the current video wall configuration, executed tasks, selected windows, open window lists, and various modified states. By employing a publish-subscribe mechanism, it achieves state notifications and data synchronization between modules, effectively solving the state consistency problem when multiple modules collaborate.

[0069] This invention constructs a unified data model around three core objects to achieve a standardized mapping between business logic and hardware instructions: ScreenObject: Used to abstractly describe the display screens of a video wall. It includes a type identifier to distinguish screen types, a parentId to describe hierarchical relationships, and a set of screen physical attributes. The type identifier distinguishes between regular screens and merged screens, while the parentId establishes a hierarchical structure, supporting the standardized expression of complex video wall topologies.

[0070] A WindowObject is used to abstractly describe the display windows of a video wall. It contains a unique window identifier, its associated screen identifier, and its logical index in the grid. Specifically, the logical index serves as the core index and is crucial for operations such as window zoom-in and restoration, polling and sorting, and real-time placement on the wall, supporting precise window positioning and state management.

[0071] ChannelObject: Used to abstractly describe video channels. It includes device association identifiers, unique channel identifiers, and polling configuration attributes. This model design is versatile, supporting both persistent storage for task-based video wall displays and incremental data transfer for temporary and real-time displays. Data model reuse enhances system reliability and maintainability.

[0072] like Figure 2 As shown, during system operation, the modules of the front-end visual orchestration layer generate interactive actions, the state synchronization module captures changes in real time and performs operations to maintain the global state, and updates the memory state pool in the unified state management layer.

[0073] The unified state management layer encapsulates business requirements into standardized, normalized coordinates and task parameters through channel binding and polling modules, and then transmits them to the device execution interface layer. The on-screen execution module groups data based on the physical device identifiers of heterogeneous decoding terminal A and heterogeneous decoding terminal B, and finally issues execution commands. This "data model-driven, layered architecture" design ensures a high degree of fidelity to the front-end orchestration intent in complex heterogeneous environments.

[0074] like Figure 3 As shown, this embodiment is a supplementary explanation to the foregoing embodiments, specifically illustrating the unified mapping process between the fused screen and the ordinary screen. This process is executed according to the following steps: After the screen preprocessing begins, the system first performs the operation of identifying all merged screens, and determines the existence of merged screens in the system based on the video wall configuration information.

[0075] If the video wall contains only regular screens, the system proceeds directly to the subsequent sorting and mapping steps; if it contains merged screens, the following processes are executed sequentially: First, the coverage area of ​​the merged screen is calculated to determine the area occupied by the merged screen in the physical coordinate system; then, the ordinary screens within the coverage area are marked as "covered screens"; in the subsequent front-end rendering and device control process, the system only retains the top-level screen object and automatically skips the processing of the covered screens.

[0076] After completing the converged screen coverage process, the system sorts all non-converged screens in ascending order of their identifiers and constructs a mapping table from screen service identifiers to temporary screen identifiers.

[0077] Next, the screen type determination operation is performed to distinguish between two processing branches: normal screen and fused screen. For ordinary screen branches, the temporary screen identifier is used directly as its logical execution number; For the merged screen branch, firstly, the temporary screen identifiers of all its sub-screens are aggregated to establish a sub-screen relationship data structure containing sub-screen mapping relationships. Then, the minimum value among the temporary screen identifiers of the sub-screens is taken as the logical execution number of the merged screen.

[0078] After the processing of the normal screen branch and the merged screen branch is completed, they are combined into the final step to generate unified backend control instructions.

[0079] This embodiment constructs a unified model around three types of objects: Screen object: Contains fields such as screenId, screenType, parentId, screenList, deviceCode, screenWidth, and screenHeight. The screenType distinguishes between regular screens and merged screens; the parentId describes the relationship between merged screens and child screens.

[0080] Window object: Contains fields such as window name (windowName), screen identifier (screenId), window logical index (sindex), split-screen mode (spiltMode), and window polling channel list (videoWindowPollingVos). Among them, the window logical index (sindex) represents the logical index of the window in the current screen grid, which is a key index for subsequent zoom-in restoration, polling sorting, and real-time wall positioning.

[0081] Channel object: Contains fields such as device code (deviceCode), channel sequence number (channelSeq), channel name (channelName), polling order (pollingSeq), polling dwell time (pollingTime), stream type (streamType), window identifier (windowId), temporary screen identifier (screenId_temp), and device unique identifier (devPuid). This model can be used for task saving, as well as for both immediate and temporary display.

[0082] The unified mapping method between the merged screen and the ordinary screen in this embodiment is as follows: like Figure 3 As shown, this embodiment first preprocesses the screens in the video wall: Identify all merged screens.

[0083] Traverse the physical coordinate range covered by the merged screen.

[0084] Mark the regular screens within this range as "covered screens".

[0085] During front-end rendering and subsequent control, skip the covered normal screen and only retain the top-level screen object.

[0086] Furthermore, before temporary wall display or device deployment, the system sorts all non-converged screens in ascending order of their identifiers and constructs a mapping table from screen service identifiers (screenId) to temporary screen identifiers (screenId_temp). For converged screens, it establishes child screen relationship data (childWallData) using the temporary screen identifiers (screenId_temp) of all its child screens, and takes the minimum value among the temporary screen identifiers (screenId_temp) as the logical execution number of the converged screen. In this way, "a converged screen in the front-end visual perception" is transformed into "a set of child screen relationships that can be recognized by the back-end device".

[0087] The method for generating the grid layout in this embodiment is as follows: When a user selects a screen and chooses the number of layouts, the system executes the following process: Obtain the target screen identifier (screenId).

[0088] The grid size is determined by the number of elements in the layout. For example, 1 corresponds to 1×1, 4 corresponds to 2×2, 9 corresponds to 3×3, and 16 corresponds to 4×4.

[0089] Generate multiple window objects in row and column order.

[0090] Write a unique window logical index (sindex) for each window.

[0091] If the original layout already has channels, the channel list will be migrated to the corresponding window in the new layout according to the original window logical index (sindex), thus preserving the channels when the grid is changed.

[0092] The advantage of this approach is that the window is not just a "display block," but a "controllable execution unit" with a stable logical number.

[0093] The window opening control method in this embodiment is as follows: In addition to basic segmented windows, this invention also supports overlaying windows on a single original screen. The specific process is as follows: Get the size of the parent container corresponding to the selected screen.

[0094] Count the number of windows that currently exist on this screen.

[0095] If the preset limit has been reached, then opening more windows will be refused.

[0096] When a new window opens, it defaults to a predetermined ratio of the width and height of the parent container.

[0097] Assign an initial position to the newly opened window according to the preset placement rules.

[0098] After converting the absolute pixel position and size into normalized coordinates, the data is sent to the backend windowing interface.

[0099] After the backend returns the window identifier (windowId), the opened window is written into the global state and used as the control object for subsequent moving, pinning, and instant display.

[0100] This makes the window opening behavior no longer just a temporary front-end element, but a complete device-executable window.

[0101] The method for converting pixel coordinates to normalized coordinates in this embodiment is as follows: This is one of the core aspects of this invention. Front-end dragging, scaling, zooming, and restoring yield absolute coordinates within the browser container, while back-end device interfaces require relative coordinates in a uniform proportional space. Therefore, this invention employs the following conversion: Let the width and height of the parent container be W and H, respectively, and the absolute coordinates and dimensions of the child window be x, y, w, and h, respectively. Let the upper limit of the normalized resolution be R = 10000, then we have: x'=round(R*x / W) y'=round(R*y / H) w'=round(R*w / W) h' = round(R*h / H) Simultaneously perform boundary clipping to ensure that x'+w'<=R and y'+h'<=R.

[0102] In this way, no matter how the front-end container is scaled, as long as the relative positional relationship remains the same, the coordinates received by the back-end device will always be consistent, thus ensuring the stability of layout control across resolutions.

[0103] The method for zooming in and restoring the segmented window in this embodiment: This embodiment supports double-clicking to zoom in on a specific grid window and then restoring it to its original grid position. The implementation steps are as follows: Select the target cell window.

[0104] When zoomed in, the target window position is set to full screen from the parent container.

[0105] Convert the absolute position after going full screen into normalized coordinates and issue movement commands.

[0106] During restoration, the grid size is deduced based on the current number of screen layouts.

[0107] Calculate the row and column of the window using its logical index sindex: ·rowIndex=floor(sindex / gridSize) ·colIndex=sindex%gridSize Calculate the absolute coordinates that the window should recover: x = colIndex * cellWidth ·y=rowIndex*cellHeight ·width=cellWidth ·height=cellHeight The coordinates are then converted back to normalized coordinates, and a movement command is issued.

[0108] This solution avoids the problem of "only knowing how to magnify, but not how to accurately restore".

[0109] The channel binding, real-time display, and round-robin scheduling methods in this embodiment When a user selects a channel from the device tree or signal source list on the left and adds it to the window, the system executes: Determine if the target window is currently selected.

[0110] The device code (deviceCode) and channel number (channelSeq) are used as the basis for deduplication to determine whether the same channel already exists in the window.

[0111] If it does not exist, a new channel object is created and appended to the current window's polling channel list videoWindowPollingVos.

[0112] Automatically assign pollingSeq to new channels, with default dwell time and default bitstream type.

[0113] If the system is currently in real-time display mode, it will not wait for the task to be saved, and will directly send the channel to the backend in real time according to the current window information.

[0114] If not in real-time display mode, only the local status will be updated, and the status will be distributed uniformly when the task is displayed or temporarily displayed.

[0115] Furthermore, at the start of the polling, the system reads the channel list of each window, generates execution parameters for each channel including window identifier (windowId), sequence identifier, device unique identifier (devPuid), stream type, dwell interval, and level index, then clears the window cache, adds polling items, and finally starts the polling.

[0116] This embodiment integrates temporary wall display with task-based wall display: In this embodiment, the temporary wall display is not regarded as an independent system, but as another execution mode of the same data model.

[0117] When a user already has a task, the system will execute the task and display it on the wall, only needing to issue the wall number and the task number.

[0118] When a user has not selected a task but a layout and channel relationship has already been formed on the page, the system will perform a temporary wall display.

[0119] Before temporarily displaying the screen, the system calls the screen data processing method: Unify and organize standard screens and converged screens; Calculate the temporary screen identifier (screenId_temp); Flatten all window channels on the same screen into window data (windowData); Add child screen relationship data (childWallData) to the merged screen; Finally, the execution parameters (wallData) array is formed by grouping the devices by their unique identifier (devPuid).

[0120] After successful distribution, the window identifier (windowId), logical screen number, and other information returned by the backend are written back to the current task object, so that the page state after temporary display can continue to be used for subsequent control such as immediate display, closing the window, moving, and rotating.

[0121] like Figure 4 As shown, this embodiment provides a specific implementation method. Taking a multi-screen physical video wall as an example, some physical screens are configured as a fused screen, while the rest are independent ordinary screens.

[0122] Implementation process After implementation, the system will proceed according to... Figure 4 The process shown is executed as follows: Screen recognition and preprocessing: Read the TV wall screen configuration and identify the merged screen (F1) and ordinary screens (S3, S4).

[0123] Logical number mapping: Generate logical numbers (P1-P4) by sorting by identifier, and establish a mapping of sub-screen relationship data (childWallData) for the merged screen.

[0124] Window layout arrangement: Select multi-grid layout on the merge screen (F1) to generate multiple sub-windows from window 0 (W0) to window 3 (W3), and assign a window logical number (sindex) to each window.

[0125] Channel binding and deduplication: Add camera channel 1 (C1) and channel 2 (C2) to the target window (W2), automatically sort them, and set the default dwell time and bitrate attributes.

[0126] Triggering temporary display: After the user clicks "Temporary Display," the system performs the following actions: Flatten the window channel data into window data (windowData); Execution parameters are generated in groups based on the device's unique identifier (devPuid); Send it to the backend and write back the window identifier.

[0127] User interaction: After the command is issued, the system supports three interaction branches: Double-click to zoom in: Zooming in controls the calculation of full-screen normalized coordinates and the issuance of movement commands; Double-click to restore: Precisely restores the column coordinates by reversing the order based on the window logical index (sindex) and grid size (gridSize); Instant Display: Real-time delivery adds channel 3 (C3) to the target window (W2), directly launching the delivery to the current window.

[0128] Polling scheduling: If the polling function is enabled, channels will be automatically switched within the window according to the polling sequence (pollingSeq).

[0129] The specific implementation process is as follows: 1. The system reads the video wall screen configuration and identifies one merged screen (F1) and two ordinary screens (S3, S4).

[0130] 2. The system sorts all non-converged physical screens by identifier and generates logical numbers: physical screen 1 (P1) corresponds to 0, physical screen 2 (P2) corresponds to 1, physical screen 3 (P3) corresponds to 2, and physical screen 4 (P4) corresponds to 3.

[0131] 3. For the merged screen (F1), establish child screen relationship data (childWallData) = [0,1], and set its logical execution number to 0.

[0132] 4. When the user selects a multi-pane layout on the merged screen (F1), the system generates four sub-windows: window 0 (W0), window 1 (W1), window 2 (W2), and window 3 (W3), and assigns the window logical index (sindex) to 0 to 3.

[0133] 5. The user adds camera channel 1 (C1) and channel 2 (C2) from the device tree to window 2 (W2). The system automatically deduplicates and sorts the data, and sets the default dwell time and default bitrate.

[0134] 6. When the user clicks "Temporary Display", the system flattens the segmented window channels of the converged screen (F1) into window data (windowData), then groups them according to the device unique identifier (devPuid) to form backend execution parameters and sends them down.

[0135] 7. When the user double-clicks window 2 (W2), the system enlarges it to full screen in the merged screen; the coordinate values ​​in the move command are not browser pixels, but normalized relative coordinates.

[0136] 8. When the user double-clicks to restore again, the system calculates the target cell position based on the window logical index (sindex) = 2 and the grid size (gridSize) = 2, and restores it accurately.

[0137] 9. After a user enables instant projection, if a new channel 3 (C3) is added from the device tree to window 2 (W2), the system does not need to re-execute the complete projection, but instead directly initiates instant projection to the current window.

[0138] 10. If the user enables polling, the system will display channel 1 (C1), channel 2 (C2), and channel 3 (C3) in window 2 (W2) in polling order (pollingSeq), with each channel playing according to a preset dwell time.

[0139] As can be seen from this implementation method, the present invention achieves unified coordination among fusion screen, segmented window, temporary wall display, instant wall display, and rotation control.

[0140] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for multi-screen fusion and adaptive window arrangement of a video wall for heterogeneous decoding terminals, characterized in that, Global state management of the video wall system is achieved through a unified state management layer, including the following steps: S1: Read the TV wall screen configuration information and perform unified semantic modeling on ordinary physical screens and converged screens; calculate the coverage area of ​​the converged screen in the physical coordinate system and mark the ordinary physical screens within the coverage area as covered; sort all uncovered top-level screen objects according to their physical identifiers to establish a mapping relationship from screen service identifiers to temporary screen identifiers. S2: For the fused screen, aggregate the temporary screen identifiers of all the sub-screens it covers to establish a sub-screen mapping relationship set, and take the extreme value of the sub-screen mapping relationship set as the logical execution number of the fused screen. S3: In response to the layout orchestration command, determine the logical grid size according to the number of screens, generate multiple logical grid windows within the target screen, and assign a globally unique window logical sequence number to each logical grid window; S4: Capture the operation events of the front-end display viewport, project the absolute pixel coordinates generated by the operation events onto the discrete resolution reference space defined by a preset normalization upper limit constant, convert them into normalized execution coordinates decoupled from the display device resolution, and perform boundary clipping processing; S5: Construct a unified window-channel data model that includes the logic execution number, window logic sequence number and video channel attributes, and flatten all logical grid window data under the same screen into a window dataset; group the window dataset according to the physical device identifier of the heterogeneous decoding terminal, and send the generated execution parameter sequence to the corresponding heterogeneous decoding terminal; S6: The unified state pool maintained by the unified state management layer coordinates the three operating modes of task display, temporary display and instant display. After the instruction is successfully issued, the window hardware identifier fed back by the heterogeneous decoding terminal is received and written back to the unified state pool to realize the state synchronization between the front-end business orchestration and the back-end hardware execution.

2. The method according to claim 1, characterized in that, It also includes the rendering pruning step: When rendering the TV wall layout at the front end, the ordinary physical screens marked as covered are skipped, and only the top-level screen object is rendered; when the heterogeneous decoding terminal executes control commands, the sub-screen mapping relationship set is used to penetrate and map to the specific sub-screen physical execution unit.

3. The method according to claim 1, characterized in that, The calculation logic for converting to normalized execution coordinates in step S4 is as follows: Calculate the ratio between the absolute pixel coordinates and the current front-end display viewport size, and map the ratio to the quantization space defined by the preset normalization upper limit constant. Discretize the coordinates using a rounding function.

4. The method according to claim 1, characterized in that, It also includes a step for accurate window geometry reconstruction based on window logical sequence number: In response to the grid restoration command, the current screen's logical grid size is obtained; based on the target window's logical index and the logical grid size, mathematical modulo and rounding operations are performed to deduce the target window's row and column index positions in the original grid; the restored absolute pixel coordinates are calculated based on the row and column index positions, and then projected twice into normalized execution coordinates for distribution.

5. The method according to claim 1, characterized in that, The coordinated scheduling of the three operating modes in step S6 specifically includes: When executing the task of displaying the video on the wall, the preset video wall number and task number are directly issued; When performing temporary wall display, the screen data is organized in real time and the logic execution number is calculated. The window channel data is flattened, and the sub-screen mapping relationship set is attached to the fused screen before the execution parameter sequence is issued. When performing instant video display, in response to the addition of a new video channel to the existing logical segment window, an incremental delivery command is directly sent to the corresponding heterogeneous decoding terminal without having to re-execute the complete video display process.

6. The method according to claim 1, characterized in that, It also includes the channel retention steps during layout changes: If the target screen has already bound video channels before the grid layout is changed, the window logical number of each window in the original layout is extracted, and the corresponding video channel list is migrated to the window object with the same window logical number in the new layout, so as to realize the logical binding between window position and channel attribute.

7. The method according to claim 1, characterized in that, It also includes anti-collision control steps for overlapping windows: Before performing the window overlay operation, count the number of existing overlay windows in the target screen. If the number reaches the preset limit, the window opening command is intercepted; otherwise, the initial normalized coordinates are assigned to the newly opened window according to the preset container width-to-height ratio and placement logic.

8. The method according to claim 1, characterized in that, It also includes the steps for deduplication and automatic allocation of video channels: The video channels are checked for duplicates using the device code and channel number as the sole criteria for deduplication. Channels that pass the check are automatically assigned a round-robin sequence number, a default dwell time, and a bitstream attribute. These attributes are then encapsulated into the unified window-channel data model.

9. The method according to claim 1, characterized in that, The data grouping based on the physical device identifier of the heterogeneous decoding terminal in step S5 specifically includes: Collect all pending window and channel data involved in the current display task; group the data according to the unique device identifier of the decoding terminal, generate an independent instruction payload for each device group, and distribute it to the corresponding heterogeneous decoding terminal through an asynchronous queue.

10. The method according to any one of claims 1 to 9, characterized in that, The aforementioned global state management of the video wall system through a unified state management layer specifically includes: The unified state pool maintains the video wall identifier, task identifier, window position status, channel binding relationship, and layout modification handle in real time. When any state changes, the associated state is automatically updated synchronously to ensure the consistency of screen modeling results, layout data, geometric parameters, channel binding, and execution status in the global scope.

Images