User terminal display signal IP coding and decoding and remote virtual multi-screen display system

Abstract

The application belongs to the field of IP transmission and remote display of user terminal display signals, and particularly relates to a user terminal display signal IP encoding and decoding and remote virtual multi-screen display system; the system comprises the following steps: an IP encoder receives a video signal sent by a signal source and performs video encoding processing; an IP decoder receives IP video packet data and decodes; a virtualized multi-screen controller receives an operation instruction and processes the same; an IP data service storage stores IP video packet data, and sends storage state information to an IP encoding and decoding controller; the IP encoding and decoding controller determines an IP encoding and decoding strategy; a multi-screen display video signal generator receives a video signal and generates a display video signal; a display terminal displays a video picture; the application can remotely transmit, store, call and display video signals output by different terminals distributed in different places through the Internet without consuming host resources and under the requirement of ensuring a certain video signal definition.

Description

Technical Field

[0001] This invention belongs to the field of IP-based transmission and remote display of user terminal display signals, specifically relating to an IP-based encoding and decoding system for user terminal display signals and a remote virtual multi-screen display system. Background Technology

[0002] With the increasing prevalence of video signals today, it is often necessary to remotely transmit multiple video signals simultaneously for monitoring and management of multiple computer screens. For example, when leaders are working remotely, they can remotely monitor the use of employees' computers; during the use of e-sports hotels, they can monitor for abnormal behavior of the computers in the e-sports hotels; supervisors can remotely view the content of various multi-channel monitoring systems, etc. It is necessary to realize the centralized display of multiple signals on a large screen on a remote terminal, and to be able to arbitrarily retrieve or review individual branch signals.

[0003] There are two main approaches to existing video signal transmission methods. The first is software-based simultaneous acquisition, encoding, and transmission at the video signal source. This involves installing application software on the host computer to acquire and transmit the video signal. The advantage of this method is that it can achieve long-distance signal transmission over a network, but the video signal transmission quality is low, and it impacts the host computer's performance. The second method is transmission via digital video interface cables. The advantage of this method is that it can transmit video signals with high quality, but it cannot achieve long-distance transmission.

[0004] For software-based simultaneous acquisition and encoding transmission, such as remote conferencing live streaming software, the advantages are as follows: 1. Acquiring and sending video signals through an application is simple to operate and easy for users to install and run directly on the host machine. 2. Post-operation and maintenance are convenient, as they can be completed through software updates and upgrades. 3. Hardware costs are relatively low. The disadvantages are as follows: 1. During transmission, issues such as blurry video, dropped frames, stuttering, and delays can easily occur, resulting in poor video signal transmission quality. 2. This operation consumes host machine resources; if the host machine is simultaneously performing other resource-intensive tasks, it can easily cause the host machine to lag.

[0005] For digital video interface cable transmission methods, such as HDMI, HDbaseT standard, SDI, etc., the advantages are as follows: 1. High video transmission quality, enabling high-definition video transmission. 2. High transmission rate, with no stuttering or latency. 3. Good device interoperability; any two devices supporting the relevant interfaces, regardless of manufacturer, can work under the highest version of their shared feature set. The disadvantages are as follows: 1. Multi-channel transmission results in hundreds or thousands of cables, making later management difficult. 2. Long-distance transmission cables are too expensive, limiting ultra-long-distance transmission; the maximum transmission distance of the HDbaseT standard is only 100 meters. 3. Difficulty in integrating and processing signals from different interfaces, and inconvenient for storing multiple signals.

[0006] Analysis revealed certain shortcomings in existing technologies: 1. While ensuring video signal transmission quality, ultra-long-distance transmission cannot be achieved. 2. Even when ultra-long-distance transmission is achieved, video smoothness cannot be guaranteed, and it consumes significant host resources.

[0007] In recent years, network technology and video encoding technology have developed rapidly and have broad application prospects. After encoding, high-definition video signals eliminate temporal and spatial redundancy in the encoded frames, compressing the data volume of the video information to a very low level without significantly affecting the visual experience. The encoded signal is then processed by a protocol encoding transmission module according to the network transmission protocol, becoming a specific IP signal protocol stream. This stream is then transmitted and encoded over the communication network before being transmitted. Therefore, smooth and clear playback of remote video signals becomes possible.

[0008] In summary, a video signal IP-based encoding and decoding and virtual multi-screen display solution needs to be designed to facilitate the long-distance transmission, storage, retrieval, and display of video signals output from different terminals located in various places via the Internet, without consuming host resources and while ensuring a certain level of video signal clarity. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes a user terminal display signal IP-based encoding and decoding and remote virtual multi-screen display system, comprising: an IP encoder, an IP decoder, an IP-based encoding and decoding controller, an IP-based data service storage, a virtual multi-screen controller, a multi-screen display video signal generator, and a display terminal;

[0010] The IP encoder operates at the transmitting end, receiving video signals from the signal source and performing video encoding processing to obtain IP-encoded video packet data; the IP-encoded video packet data is then transmitted over the network via a network protocol.

[0011] The IP decoder operates at the receiving end, and is used to receive IP video packet data sent by the network and decode it, and send the decoded video signal to the multi-screen display video signal generator.

[0012] The virtualized multi-screen controller is used to receive operation instructions issued by the operator and, after processing the operation instructions, send them to the multi-screen display video signal generator and the IP-based codec controller.

[0013] The IP-based data service memory is used to store and send IP-based video packet data to the IP decoder, sense the memory status and send the memory status information to the IP codec controller; receive terminal playback signal commands and send them to the IP decoder;

[0014] The IP-based codec controller is used to sense the IP network transmission environment status and determine the IP-based codec strategy based on the received instructions from the virtualized multi-screen controller and memory status information, and sends the determined codec instructions to the IP encoder and IP decoder.

[0015] The multi-screen display video signal generator is used to receive video signals sent by the IP decoder and combine them with the instructions received from the virtualized multi-screen controller to generate display video signals, and then send the display video signals to the display terminal.

[0016] The display terminal is used to display video images based on the display video signal.

[0017] Preferably, the IP encoder includes a control module, a video receiving module, a first buffer module, a video encoding module, a second buffer module, and an IP video data packet sending module;

[0018] The control module receives instructions transmitted by the IP-based codec controller and sets the encoding parameters;

[0019] The video receiving module receives the video signal transmitted from the video interface, determines the video signal type, and performs format conversion processing.

[0020] The first cache module caches the video signal after format processing.

[0021] The video encoding module encodes the video signal processed by the video receiving module into H.266 encoding.

[0022] The second caching module caches the encoded signal and waits for IP packets to be sent.

[0023] The IP video data packet sending module sends data waiting to be sent in the buffer via the TCP protocol.

[0024] Preferably, the process of the IP encoder performing video encoding processing on the video signal includes: setting IP encoder parameters according to the received IP encoding and decoding strategy; receiving the video signal according to the IP encoder parameters; converting the format of the video signal; and performing compression encoding processing on the converted video signal.

[0025] Preferably, the virtualized multi-screen controller includes a human-computer interaction module, an instruction processing module, and a network transmission module;

[0026] The human-computer interaction module receives instructions from the operator through the operation panel;

[0027] The instruction processing module processes the instructions received from the operation panel and generates signals that can be sent over the network.

[0028] The network sending module transmits instructions received from the operation panel via the network.

[0029] Preferably, the IP-based codec controller includes an instruction receiving module, a network sensing module, a memory sensing module, an instruction decision module, and an instruction sending module;

[0030] The instruction receiving module receives instructions from the virtualized multi-screen controller;

[0031] The network sensing module senses the status of the IP network transmission environment.

[0032] The memory sensing module receives information sent by the IP-based data service memory.

[0033] The instruction decision module determines the IP encoding and decoding strategy based on the instructions from the virtualized multi-screen controller, the IP network transmission environment status, and the information sent by the IP-based data service storage.

[0034] The instruction sending module sends the encoding and decoding instructions that implement the IP-based encoding and decoding strategy to the IP encoder and IP decoder.

[0035] Preferably, the process by which the IP-based codec controller determines the IP-based codec strategy includes:

[0036] Receive instructions from the virtualization multi-screen controller, including display resolution, whether the transmitted tributary signal needs to be stored, whether the display terminal needs to review the signal, encoding compression rate, and encoder signal acquisition rate;

[0037] The IP-based codec controller senses the status of the IP network transmission environment, including bandwidth, network latency, network congestion, and packet loss rate.

[0038] Receive storage status information sent by the IP-based data service storage, including storage duration, remaining storage capacity, and whether the stored data needs to be cleared;

[0039] The IP encoding / decoding strategy is determined based on the instructions of the virtualization multi-screen controller, the IP network transmission environment status, and the memory status information.

[0040] Preferably, the multi-screen display video signal generator includes a control module, an HDMI receiving module, an NDI conversion module, an NDI scheduling module, and a video generation module;

[0041] The control module receives operation instructions from the virtualized multi-screen controller, sets video display parameters, and determines whether the received video signal is a single-channel signal.

[0042] The HDMI receiving module receives video signals transmitted by multiple decoders.

[0043] The NDI conversion module converts the received video signal into an NDI protocol stream.

[0044] The NDI scheduling module schedules and processes multiple NDI signals to facilitate subsequent video signal generation;

[0045] The video generation module decodes the NDI signal, generates an HDMI multi-screen display video signal from the NDI signal selected by the scheduling module according to the video signal playback mode, and sends it to the display terminal through the HDMI interface.

[0046] Preferably, the process by which the multi-screen display video signal generator generates the display video signal includes:

[0047] Receive instructions from the virtualized multi-screen controller and set the video signal playback mode according to the instructions, including the playback signal branch range, display image format and display resolution;

[0048] The system receives video signals sent by the IP decoder and determines whether the video information is a single-channel signal. If it is a single-channel signal, the system directly feeds the HDMI signal recovered by the IP decoder to the display terminal. Otherwise, the HDMI interface receives HDMI signals sent by the IP decoder in parallel, performs NDI conversion on the HDMI signals to obtain the NDI protocol stream, and uses the NDI switching scheduling matrix to schedule and process the NDI protocol stream. Based on the selected NDI signal and the video signal playback mode, the system generates an HDMI multi-screen display video signal.

[0049] Preferably, the IP-based data service storage device includes a data storage module, an instruction feedback module, and a network transmission module;

[0050] The data storage module is used to store IP-encoded video packet data sent by the IP encoder;

[0051] The instruction feedback module has two functions: first, to send the memory status information to the IP-based codec controller; second, to receive the terminal playback signal instruction from the IP-based codec controller and send the terminal playback signal instruction stored in the memory to the IP-based decoder.

[0052] The network transmission module is used to send the stored IP-based video packet data to the IP decoder via the network.

[0053] The beneficial effects of this invention are as follows: This invention, without consuming host resources and while ensuring a certain level of video signal clarity, utilizes the internet for long-distance transmission, storage, retrieval, and display of video signals output from different terminals located in various locations. This invention solves the problem of existing software applications consuming host resources when generating video signals, as peripheral hardware does not consume host resources, thus improving host resource utilization. It solves the problem of long-distance HDMI signal transmission by using H.266 compression encoding and network transmission technology. It also solves the problem of digital multi-channel image processing; traditional methods using large-matrix parallel computation in digital image processing involve excessive computation, while the NDI protocol encoding and scheduling method significantly reduces the computational load. Attached Figure Description

[0054] Figure 1 This is a general block diagram of the user terminal display signal IP encoding and decoding and remote virtual multi-screen display system in this invention;

[0055] Figure 2 This is a diagram illustrating the structure of the IP encoder in this invention;

[0056] Figure 3 This is a flowchart of the IP encoder process in this invention;

[0057] Figure 4 This is a diagram illustrating the IP decoder configuration in this invention.

[0058] Figure 5 This is a flowchart of the IP decoder workflow in this invention;

[0059] Figure 6 This is a diagram illustrating the virtualized multi-screen controller configuration in this invention.

[0060] Figure 7 This is a diagram illustrating the configuration of the IP-based codec controller in this invention.

[0061] Figure 8 This is a diagram illustrating the structure of the multi-screen display video signal generator in this invention.

[0062] Figure 9 This is a flowchart illustrating the workflow of the multi-screen display video signal generator in this invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] This invention proposes an IP-based encoding and decoding system for user terminal display signals and a remote virtual multi-screen display system. Without consuming host resources and while ensuring a certain level of video signal clarity, it utilizes the internet to transmit, store, retrieve, and display video signals output from different terminals located in various locations over long distances. This overcomes the problems of existing systems where screen content signal acquisition and transmission consume host resources, high-definition video signals cannot be transmitted over long distances, transmission costs are high, interfaces are incompatible, and caching is difficult. Figure 1 As shown, the system specifically includes the following: a network video signal IP codec (hereinafter referred to as IP encoder / IP decoder), a video signal IP codec strategy controller (hereinafter referred to as IP codec controller), an IP data service storage, a virtualized multi-screen controller, a multi-screen display video signal generator, and a display terminal;

[0065] IP encoder:

[0066] The IP encoder operates at the transmitting end, receiving video signals from the signal source and performing video encoding processing to obtain IP-encoded video packet data; it then transmits the IP-encoded video packet data over the network via network protocols.

[0067] like Figure 2 As shown, the IP encoder mainly consists of the following six modules:

[0068] The first module is the control module, which receives instructions from the IP-based codec controller and sets the encoding parameters. Specifically, the control module receives instructions from the IP-based codec controller via the network and sets encoder parameters, including video reception and acquisition rate, H.266 encoding parameters, compression ratio, network transmission destination, and whether to send the signal to the IP data service storage device for storage.

[0069] The second module is the video receiving module, which receives high-definition video signals transmitted from the video interface, determines the video signal type, and performs format conversion processing. Specifically, the receiving module receives high-definition video signals sent from common interfaces such as HDMI, DP, DVI, and VGA, and transcodes the received video data into YUV420 format for subsequent encoding processing.

[0070] The third module is the first buffer module, which is responsible for buffering the video signal after format processing; specifically, the first buffer module uses RAM storage to buffer the format-processed video signal.

[0071] The fourth module is the video encoding module, which encodes the video signal processed by the video receiving module using H.266. Specifically, the video encoding module reads data from the first buffer module for encoding, and immediately clears the buffer after encoding is complete. The video encoding module adopts the H.266 / VVC encoding framework, which uses a hybrid encoding framework based on block prediction and transform. The framework includes modules such as intra-frame prediction, inter-frame prediction, transform, quantization, entropy coding, and loop filtering. Due to the introduction of the latest encoding and compression technologies at each stage, H.266 / VVC can compress the video signal to the greatest extent possible. The video encoding module uses an X266 encoder, and its online real-time video encoding workflow is as follows:

[0072] (1) Initialize and configure encoding parameters, set different encoding parameters, initialize the video parameter set and allocate memory space; set the data in the image structure and allocate memory space;

[0073] (2) Initialize the x266 encoder by passing the data from the encoding parameters and video parameters into the encoder;

[0074] (3) Determine whether the buffer contains a video stream. During the encoding process, the encoder reads the bit stream from the buffer. When the buffer is empty, the encoder needs to wait for the acquired video stream to be stored in the buffer first.

[0075] (4) Read YUV420 data frames from the buffer, encode each frame, and package the encoded video data.

[0076] (5) When no data frames are read in, check whether the encoder contains remaining bitstream data.

[0077] (6) Wait for the network transmission parameters to be set and the data to be encapsulated and transmitted to the receiving end through the network transmission protocol. When video data acquisition stops, release resources and shut down the x266 encoder.

[0078] The fifth module is the second buffer module, which buffers the encoded signal and waits for IP packets to be sent. Specifically, the second buffer module uses RAM storage to buffer the encoded video signal waiting to be sent by the network; the buffer is cleared immediately for data that has already been sent.

[0079] The sixth module is the IP video data packet sending module, which sends data waiting to be sent in the buffer via the TCP protocol. Specifically, the IP video data packet sending module uses the TCP protocol and a server / client network communication method. The encoder acts as a Socket communication server, and the software needs to provide the port number required for the TCP connection and bind it to the same IP address. Listening is started via command, waiting for client access. The encoder Socket communication process is as follows:

[0080] (1) Use the socket() function to create a Socket connection;

[0081] (2) Use the setsockopt() function to set Socket properties;

[0082] (3) Use the bind() function to bind information such as IP address and port number;

[0083] (4) Use the listen() function to start listening and wait for the client to connect;

[0084] (5) Use the accept() function to receive client connections;

[0085] (6) Use send() / recv() or read() / write() to send and receive data;

[0086] (7) Data transmission complete, close network connection;

[0087] (8) Turn off listening.

[0088] To further reduce latency, congestion control algorithms can be used for optimization.

[0089] The system uses TCP as the network transmission protocol. In video transmission systems, the choice of protocol has a significant impact on transmission latency. It is understood that congestion control algorithms adjust the slow start threshold (ssthresh) and congestion window (cwnd) when packet loss and retransmission occur, and these two parameters determine the amount of data transmitted. Congestion control algorithms can be categorized based on packet loss and latency. Packet loss-based congestion control algorithms include TCPTahoe, TCPReno, TCPNewReno, HSTCP, STCP, and HTCP, while latency-based congestion control algorithms primarily use TCPVegas.

[0090] The IP video packet data transmission module outputs through a high-speed network interface. The high-speed network interface is a high-speed data transmission device that uses technologies such as fiber optics or 10 Gigabit Ethernet to transmit data in the form of network signals. Among them, fiber optic networks have high transmission speed, large bandwidth and anti-interference capabilities, making them more suitable for long-distance network transmission of high-definition video streams.

[0091] like Figure 3 As shown, in some preferred embodiments, the operation steps of the IP encoder are as follows:

[0092] Step 1: Encoder parameter setting: The IP encoder parameters are set by receiving the IP encoding / decoding strategy from the IP encoding / decoding controller. The strategy is obtained by the IP controller through comprehensive decision-making after sensing the IP network environment (network bandwidth, latency), data storage server status, and user display requirements from the multi-screen controller.

[0093] The decision-making process (IP encoding / decoding strategy) enables the setting of video reception parameters and encoder encoding parameters for the IP encoder, such as parameters controlling video signal reception (resolution, acquisition rate, etc.), H.266 encoding parameters (compression rate, etc.), and IP signal transmission (priority, whether storage is required, etc.).

[0094] Step 2: Receive video signal: Video input interfaces include common interfaces such as HDMI, DP, DVI, and VGA to obtain the video signal sent by the video signal source.

[0095] Step 3: Video Signal Format Conversion: Since the received signals may be digital video signals from different digital interfaces, each with its own transmission format, to facilitate subsequent unified encoding processing, the video signal type is determined upon receipt, and then a transcoding operation is performed. The received video data is transcoded into YUV420 format. YUV420 is a data storage format with the advantage of low bandwidth usage, making it very suitable for storage in a buffer and facilitating subsequent encoding processing. After the format conversion is complete, the converted data stream is stored in a buffer.

[0096] Step 4: Video Signal Compression and Encoding Processing: The converted signal undergoes H.266 video compression and encoding processing, using the latest generation international video coding standard (H.266 / VVC) developed by JCT-VC. An H.266 encoder is used to perform video compression and encoding on the converted data.

[0097] Step 5: Network Transmission: The encoded video signal is transmitted in IP video data packets according to the network protocol. In this system, the encoder is located at the transmitting end, and the decoder is located at the receiving end. TCP, as a connection-oriented transport protocol, provides reliable end-to-end byte stream information transmission services, offering flow control and congestion control; therefore, TCP is chosen as the transmission protocol for video images.

[0098] The two ports use a classic client / server architecture to transmit data streams via a transport protocol. The encoder acts as the server in the client / server architecture, and the decoder acts as the client; both establish a TCP connection to transmit data.

[0099] Image data compressed using the H.266 compression protocol experiences minimal latency during transmission, resulting in smooth, real-time transmission without noticeable lag. To avoid network congestion, the TCP protocol can be optimized to further reduce latency.

[0100] IP decoder:

[0101] The IP decoder operates at the receiving end, receiving and decoding IP video packet data sent from the network, and sending the decoded video signal to the multi-screen display video signal generator.

[0102] like Figure 4 As shown, the IP decoder mainly consists of the following six modules:

[0103] The first module is the control module, which receives instructions from the IP-based codec controller over the network to set encoding parameters. Specifically, the control module receives instructions from the IP-based codec controller over the network to set decoder parameters, including H.266 decoding parameters, network receiving address, and whether to connect to the IP data service storage for signal playback.

[0104] The second module is the IP video packet data receiving module, which receives network signals from the network. Specifically, the IP video packet data receiving module uses the TCP protocol. The IP decoder acts as a client in the overall system architecture, and the IP video packet data receiving module serves as the functional entry point for the decoding terminal, requiring the implementation of network connection and network data acquisition functions. The network connection is designed and implemented based on the TCP protocol, using the same IP address and port number as the encoding terminal for data interaction. The video data receiving process is as follows:

[0105] (1) Use the socket() function to connect to the server using the port number and IP address;

[0106] (2) Use setReceiveBufferSize() to allocate buffer space to store data;

[0107] (3) Use getInputStream() to retrieve the video data from the cache;

[0108] (4) Set the buffer length for this data reading based on the video frame length marker;

[0109] (5) Use the write() function to write data into the buffer;

[0110] (6) Release the memory containing the data that has been read;

[0111] (7) Disconnect.

[0112] The third module is the first buffer module, which buffers signals forwarded from the network. Specifically, the first buffer module uses RAM to store the signals forwarded from the network for the decoder to read. For real-time video streams transmitted via TCP, the transmitted data is placed in the buffer, and the decoding module reads the bitstream from the buffer for decoding. When the buffer is empty, it needs to wait for the bitstream transmitted via TCP to be placed into the buffer.

[0113] The fourth module is the decoding module, which performs H.266 decoding on the signals sent from the network to obtain YUV420 video data. Specifically, the decoding module uses an H.266 video decoder based on FFmpeg. According to the decoding parameters initialized in the main function, it calls FFmpeg library functions to decode the data, converting the read-in H.266 data into YUV420 format video data. The online video real-time decoding workflow is as follows:

[0114] (1) When the terminal starts to receive the video stream transmitted by the TCP protocol, it counts the frame information in real time and allocates memory space, and allocates the stream buffer area according to the stream information.

[0115] (2) Initialize the FFmpeg decoder and pass the bitstream information in the buffer to the decoder;

[0116] (3) Determine whether the buffer contains transmitted bitstream data. If the buffer is empty, wait for the bitstream transmitted via TCP protocol to be placed into the buffer.

[0117] (4) Extract a frame from the bitstream, determine the type of the image frame, and decode it into YUV420 format video data;

[0118] (5) When no image frames are extracted from the bitstream, count the undecoded bitstream data in the decoder, determine the type of the remaining bitstream, and decode the remaining bitstream.

[0119] (6) Obtain YUV420 data through data format conversion.

[0120] (7) When the server stops sending the bitstream or the receiver stops receiving the bitstream, release the resources and close the decoder.

[0121] The fifth module is the second buffer module, which buffers the decoded signal and waits for format conversion. Specifically, the second buffer module uses RAM to buffer the video signal waiting for format conversion after decoding; the buffer is immediately cleared for data that has already been sent.

[0122] The sixth module is the format conversion module, which converts YUV420 data into HDMI signals and sends them to the multi-screen display video signal generator. Specifically, the HDMI format conversion module converts the YUV420 data stored in the buffer into a widely used HDMI video signal and sends it to the multi-screen display video signal generator through the HDMI interface.

[0123] like Figure 5 As shown, in some preferred embodiments, the IP decoder operates as follows:

[0124] Step 1: Receive instruction signals from the IP codec controller and set the decoder's decoding parameters; specifically: after receiving the IP control signal sent by the IP codec controller, set the basic parameters of the IP decoder, including parameters for controlling video signal reception (resolution, etc.), H.266 decoding parameters (compression rate, etc.), whether to connect to the IP video storage device, and whether to receive playback signals, etc.

[0125] Step 2: Receive IP video packet data: Receive the video encoded signal sent by the sender according to the TCP protocol. The decoder is the client in a C / S architecture.

[0126] Step 3: H.266 Decoding: Decode the video signal according to the H.266 / VVC standard to recover the YUV420 format data, and cache it in the buffer.

[0127] Step 4: Format Conversion: Since the HDMI interface is the typical mainstream display interface standard for most user terminals, the decoded and recovered output signal is mainly in HDMI format. Convert the decoded YUV420 format data to HDMI format.

[0128] Step 5: Send the HDMI signal to the multi-screen display video signal generator.

[0129] Virtualized multi-screen controller:

[0130] The virtualized multi-screen controller is used to receive operation instructions from operators, process the instructions, and then send them to the multi-screen display video signal generator or IP-based codec controller.

[0131] The virtualized multi-screen controller is essentially an operation panel, operating at the control terminal and directly facing the operator. Operators can use this device to control other devices in the system, set encoding / decoding parameters and operating modes. Specifically, the operator issues commands to the multi-screen display video signal generator and IP codec, and the virtualized multi-screen controller transmits these commands to the multi-screen display video signal generator and IP codec controller (which then makes a decision and issues it to the IP codec). This achieves system control. The main operation commands are as follows:

[0132] 1. Display screen format (multi-window, picture-in-picture, etc.), display resolution, display branch signal range (whether it is necessary to review the signals stored in the memory); after determining the display mode, the virtualization multi-screen controller will send instructions to the multi-screen display video signal generator.

[0133] 2. Parameter format settings for IP-based codecs, such as compression standard, signal acquisition rate, whether storage is required, storage duration, and whether to connect to an IP data storage device for signal playback. The virtualization multi-screen controller sends operation commands to the IP-based codec controller. After sensing the network environment and storage status, the IP-based codec controller makes a final decision and sends the decision to the IP-based codec.

[0134] like Figure 6 As shown, the virtualized multi-screen controller consists of three modules:

[0135] The first module is the human-computer interaction module, which receives instructions from the operator via a control panel. Specifically, the operator uses the control panel to set the operating modes of the multi-screen display video signal generator and the IP-based codec controller. Setting the operating mode of the multi-screen display video signal generator includes selecting the playback signal, whether to review the signal, and the display format. Setting the IP-based codec encoding parameters includes whether to store the data.

[0136] The second module is the instruction processing module, which processes the instructions to form instructions that can be sent.

[0137] The third module is the network transmission module, which transmits the generated command signals over the network. It directly sends commands to the multi-screen display video signal generator via the network. It also sends commands to the IP-based codec controller.

[0138] The virtualized multi-screen controller operates as follows:

[0139] Step 1, Receiving Instructions: The system uses a control panel to enable human-computer interaction and receive instructions from the operator.

[0140] Step 2, instruction processing: The received instructions are processed into transmissible IP instruction signals so that they can be sent to the multi-screen display video signal generator and the IP codec controller respectively.

[0141] Step 3, Command transmission: Send the command to the target device via the network.

[0142] IP-based data service storage:

[0143] The IP-based data service storage is responsible for storing the IP-based video packet data sent by the IP encoder and sending the IP-based video packet data to the IP decoder. At the same time, it feeds back information such as the storage status and its own capacity usage to the IP-based codec controller.

[0144] When signal playback is required, the IP-based data service storage connects to the IP decoder and sends the terminal's signal playback command to the IP decoder. Here, the IP-based data service storage acts as the server in a client / server architecture.

[0145] To facilitate subsequent playback of video signals on the terminal, the IP commands of the virtualized multi-screen controller need to be routed and forwarded to the IP-based data service storage device for retrieving stored signals.

[0146] The memory consists of three modules:

[0147] The first module is the data storage module, which stores the IP-based video packet data sent by the IP encoder. Since the Internet uses fiber optics for IP signal transmission, this module adopts a SAN storage method, which can well meet the functional requirements of the storage device.

[0148] The second module, the instruction feedback module, performs two functions: First, it transmits the memory status information of the IP-based data service memory to the IP codec controller, facilitating the controller's final IP codec strategy decision. Second, it receives terminal playback signal commands from the IP codec controller and sends them to the IP decoder. The memory status information includes storage duration, remaining storage capacity, and whether stored data needs to be cleared.

[0149] The third module is the network transmission module, which sends the stored IP-based video packet data to the IP decoder via the network.

[0150] IP-based codec controller:

[0151] The IP-based codec controller is used to sense the IP network transmission environment and determine the IP-based codec strategy based on the instructions and memory status information received from the virtualized multi-screen controller. The determined codec instructions are then sent to the IP encoder and IP decoder.

[0152] like Figure 7 As shown, the IP-based codec controller consists of 5 modules:

[0153] The first module is the instruction receiving module, which receives instructions from the virtualized multi-screen controller, including display resolution, whether the transmitted branch signal needs to be stored, whether the display terminal needs to review the signal, encoding compression rate, encoder signal acquisition rate, etc.

[0154] The second module is the network awareness module, which senses the status of the IP network transmission environment, including bandwidth, network latency, network congestion, packet loss rate, etc.

[0155] The third module is the memory awareness module, which receives information sent by the IP-based data service memory, and senses the memory status, storage duration, remaining storage capacity, and whether the stored data needs to be cleared.

[0156] The fourth module is the instruction decision module, which determines the final implementable encoding and decoding strategy based on the information received from the first three modules.

[0157] The fifth module is the instruction sending module, which sends the encoding and decoding instructions that ultimately implement the IP-based encoding and decoding strategy to the IP-based encoder / decoder.

[0158] In some preferred embodiments, the operation steps of the IP-based codec controller are as follows:

[0159] Step 1: Receive instructions and perceive the network environment. The main aspects are as follows:

[0160] 1. Receive instructions from the virtualization multi-screen controller, including display resolution, whether the transmitted branch signal needs to be stored, whether the display terminal needs to review the signal, encoding compression rate, and encoder signal acquisition rate.

[0161] 2. Sensing the status of the IP network transmission environment, including bandwidth, network latency, network congestion, and packet loss rate.

[0162] 3. Sensing the status of the storage device, including storage duration, remaining storage capacity, and whether stored data needs to be cleared.

[0163] Step Two: Command Interaction: The data obtained from the above sensing is summarized, and a final decision is made to determine the optimal encoding / decoding scheme, resulting in encoding / decoding instructions for implementing the IP-based encoding / decoding strategy. Specifically, based on the instructions from the virtualized multi-screen controller, the usage of the IP-based data service storage, and the current network conditions, the most suitable encoding parameters are selected. For example, if the control terminal requires high-definition signal and needs to store the signal on the server, but the server's storage space is insufficient, or the current network conditions are poor, transmitting high-quality signals may cause delays or stuttering, the encoding requirements should be appropriately reduced, and the optimal encoding parameter instructions should be transmitted to the IP encoder / decoder.

[0164] If signal playback is required, the IP-based codec controller needs to send the terminal's signal playback command to the IP decoder, which then controls the decoder to connect to the IP-based data service storage for signal playback.

[0165] Step 3: Sending instructions. Send the final instructions to the IP codec.

[0166] Multi-screen display video signal generator:

[0167] The multi-screen display video signal generator is used to receive video signals sent by the IP decoder and combine them with the instructions received from the virtualized multi-screen controller to generate display video signals, and then send the display video signals to the display terminal.

[0168] The multi-screen display video signal generator is a component designed to enable on-demand display on terminals. It generates display video signals to meet users' remote multi-screen display needs. (Note: The multi-screen display video signal generator can also operate at the source end, meeting users' local monitoring and multi-screen display needs for source-end video signals.)

[0169] like Figure 8 As shown, the multi-screen display video signal generator consists of 5 modules:

[0170] The first module is the control module. It receives operation commands from the virtualized multi-screen controller and sets the video signal playback mode, including the range of playback signal branches, display format (multi-window, picture-in-picture), resolution, etc. Simultaneously, the control module determines whether the video signal is a single-channel signal. If it is a single-channel signal for playback or reception, it sends the signal directly. Specifically, the control module determines the source and quantity of playback signals by receiving information from the HDMI receiving module (number of signal branches) and the playback signal branch range command from the virtualized multi-screen controller (playing the signal of the selected branch). If only a single-channel signal is received, or the playback command is to play a specific signal from multiple channels, the NDI conversion and NDI processing modules do not operate. After generating the display image, it is directly sent to the terminal display device via the HDMI interface.

[0171] The second module is the HDMI receiving module, which receives HDMI signals transmitted by multiple decoders in parallel. Specifically, the HDMI receiving module receives HDMI video signals transmitted by multiple decoders in parallel through multiple HDMI interfaces.

[0172] The third module is the NDI conversion module, which converts the received HDMI signal into an NDI protocol stream; specifically, the NDI conversion module converts the HDMI signal into an NDI protocol data stream that can be arbitrarily combined and processed.

[0173] The fourth module is the NDI scheduling module, which performs matrix processing on multiple NDI signals to facilitate subsequent video signal generation. Specifically, it adopts the NDI switching scheduling matrix algorithm, which directly schedules signals by exchanging NDI video signal metadata in the control network and through address redirection, but does not perform any physical centralization of the signals.

[0174] The fifth module is the video generation module, which decodes the NDI signal to generate an HDMI multi-screen display video signal and sends it to the terminal display device via the HDMI interface. Specifically, the video generation module uses real-world multi-screen system decoding software, which can theoretically create NDI windows without restriction according to business application needs. Each window can add multiple NDI video streams of various specifications, and the screen style is fully customizable. Multiple windows can be played, and switching between windows can be done with a single click. The video can also be sent to the terminal display device via the HDMI interface. As auxiliary functions, multi-screen displays typically also include speaker displays, safety frames, center lines, and borders. The video generation module generates an HDMI multi-screen display video signal from the NDI signal selected by the scheduling module according to the video signal playback mode and sends it to the display terminal via the HDMI interface.

[0175] like Figure 9 As shown, in some preferred embodiments, the operation steps of the multi-screen display video signal generator are as follows:

[0176] Step 1: Parameter Settings: Receive instructions from the virtualization multi-screen controller to set the video signal playback mode generated by the signal generator, including the range of playback signal branches, display format (multi-window, picture-in-picture), display resolution, etc.; Note: If only a single video signal display is required, the NDI module can bypass the signal and feed the HDMI signal recovered by the decoder to the display terminal, that is, send the signal received by the HDMI interface directly to the terminal display device through the HDMI interface, skipping steps 3 and 4.

[0177] Step 2: Signal Reception: Receive HDMI signals sent by the decoder in parallel via the HDMI interface.

[0178] Step 3: NDI Conversion: Due to the excessive computational load and heavy consumption of hardware and software resources in parallel processing of multiple video signals, this system adopts the NDI protocol to process multiple video signals instead of the multi-channel matrix operation method in traditional digital image processing, converting multiple HDMI signals into NDI protocol data streams that can be arbitrarily combined and processed.

[0179] Step 4: NDI Processing: The NDI switching scheduling matrix is ​​used to schedule the selected tributary signals based on the received IP control signals from the virtualized multi-screen controller. Specifically, the NDI scheduling module directly schedules signals by exchanging NDI video signal metadata located in the control network and using address redirection.

[0180] Step 5: Signal Transmission: Since most mainstream display devices use HDMI interfaces, the signal after scheduling is decoded to generate an HDMI signal. Specifically: Based on the video signal playback mode settings of the received virtualized multi-screen controller, a visual graphical interface is constructed. Based on the visual graphical interface, the NDI signal is converted into an HDMI multi-screen display video signal, and the display video signal is sent to the terminal display device through the HDMI interface.

[0181] Display terminal:

[0182] The display terminal is used to display video images based on the display video signal.

[0183] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A user terminal display signal IP-based encoding and decoding and remote virtual multi-screen display system, characterized in that, include: IP encoders, IP decoders, IP-based codec controllers, IP-based data service storage devices, virtualized multi-screen controllers, multi-screen display video signal generators, and display terminals; The IP encoder operates at the transmitting end, receiving video signals from a signal source and performing video encoding processing to obtain IP-encoded video packet data; it then transmits the IP-encoded video packet data over a network via a network protocol. The IP encoder includes a control module, a video receiving module, a first buffer module, a video encoding module, a second buffer module, and an IP video data packet transmission module. The control module receives instructions transmitted by the IP-based codec controller and sets the encoding parameters; The video receiving module receives the video signal transmitted from the video interface, determines the video signal type, and performs format conversion processing. The first cache module caches the video signal after format processing. The video encoding module encodes the video signal processed by the video receiving module into H.266 encoding. The second caching module caches the encoded signal and waits for IP packets to be sent. The IP video data packet sending module sends data waiting to be sent in the buffer via the TCP protocol; The IP decoder operates at the receiving end, and is used to receive IP video packet data sent by the network and decode it, and send the decoded video signal to the multi-screen display video signal generator. The virtualized multi-screen controller is used to receive operation instructions issued by the operator and, after processing the operation instructions, send them to the multi-screen display video signal generator and the IP-based codec controller. The IP-based data service storage is used to store and send IP-based video packet data to the IP decoder, sense the storage status, and send the storage status information to the IP codec controller; Receive the terminal's review signal command and send it to the IP decoder; The IP-based data service storage device includes a data storage module, an instruction feedback module, and a network transmission module; The data storage module is used to store IP-encoded video packet data sent by the IP encoder; The instruction feedback module has two functions: first, to send the memory status information to the IP-based codec controller; second, to receive the terminal playback signal instruction from the IP-based codec controller and send the terminal playback signal instruction stored in the memory to the IP-based decoder. The network transmission module is used to send the stored IP-based video packet data to the IP decoder via the network; The IP-based codec controller is used to sense the IP network transmission environment status and determine the IP-based codec strategy based on the received instructions from the virtualized multi-screen controller and memory status information, and sends the determined codec instructions to the IP encoder and IP decoder; the IP-based codec controller includes an instruction receiving module, a network sensing module, a memory sensing module, an instruction decision module, and an instruction sending module; The instruction receiving module receives instructions from the virtualized multi-screen controller; The network sensing module senses the status of the IP network transmission environment. The memory sensing module receives information sent by the IP-based data service memory. The instruction decision module determines the IP encoding and decoding strategy based on the instructions from the virtualized multi-screen controller, the IP network transmission environment status, and the information sent by the IP-based data service storage. The instruction sending module sends the encoding and decoding instructions that implement the IP-based encoding and decoding strategy to the IP encoder and IP decoder; The process by which the IP-based codec controller determines the IP-based codec strategy includes: Receive instructions from the virtualization multi-screen controller, including display resolution, whether the transmitted branch signal needs to be stored, whether the display terminal needs to review the signal, encoding compression rate, and encoder signal acquisition rate; The IP-based codec controller senses the status of the IP network transmission environment, including bandwidth, network latency, network congestion, and packet loss rate. Receive storage status information sent by the IP-based data service storage, including storage duration, remaining storage capacity, and whether the stored data needs to be cleared; The IP encoding and decoding strategy is determined based on the instructions of the virtualization multi-screen controller, the IP network transmission environment status, and the memory status information. The multi-screen display video signal generator is used to receive video signals sent by the IP decoder and combine them with the received instructions from the virtualized multi-screen controller to generate display video signals, which are then sent to the display terminal. The multi-screen display video signal generator includes a control module, an HDMI receiving module, an NDI conversion module, an NDI scheduling module, and a video generation module. The process of generating display video signals by the multi-screen display video signal generator includes: Receive instructions from the virtualized multi-screen controller and set the video signal playback mode according to the instructions, including the playback signal branch range, display image format and display resolution; The system receives video signals sent by the IP decoder and determines whether the video information is a single-channel signal. If it is a single-channel signal, the system directly feeds the HDMI signal recovered by the IP decoder to the display terminal. Otherwise, the HDMI interface receives HDMI signals sent by the IP decoder in parallel from multiple channels, performs NDI conversion on the HDMI signals to obtain the NDI protocol stream, and uses the NDI switching scheduling matrix to schedule and process the NDI protocol stream. Based on the selected NDI signal and the video signal playback mode, the system generates an HDMI multi-screen display video signal. The control module receives operation instructions from the virtualized multi-screen controller, sets video display parameters, and determines whether the received video signal is a single-channel signal. The HDMI receiving module receives video signals transmitted by multiple decoders. The NDI conversion module converts the received video signal into an NDI protocol stream. The NDI scheduling module schedules and processes multiple NDI signals to facilitate subsequent video signal generation; The video generation module decodes the NDI signal, generates an HDMI multi-screen display video signal from the NDI signal selected by the scheduling module according to the video signal playback mode, and sends it to the display terminal through the HDMI interface. The video generation module uses multi-screen system decoding software that is actually used in applications. It can create NDI windows according to business application requirements. Each window can add multiple NDI video streams of various specifications, and the screen style can be fully customized. It can play multiple windows, switch between windows with one click, and send the video to the terminal display device through the HDMI interface. The display terminal is used to display video images based on the display video signal.

2. The user terminal display signal IP encoding / decoding and remote virtual multi-screen display system according to claim 1, characterized in that, The process of video encoding processing of video signals by the IP encoder includes: setting IP encoder parameters according to the received IP encoding and decoding strategy; receiving video signals according to the IP encoder parameters; converting the format of the video signals; and compressing and encoding the converted video signals.

3. The user terminal display signal IP encoding / decoding and remote virtual multi-screen display system according to claim 1, characterized in that, The virtualized multi-screen controller includes a human-computer interaction module, an instruction processing module, and a network transmission module; The human-computer interaction module receives instructions from the operator through the operation panel; The instruction processing module processes the instructions received from the operation panel and generates signals that can be sent over the network. The network sending module transmits instructions received from the operation panel via the network.

Images