A Synchronous Control Method and System for Distributed Video Image Mosaic

Abstract

The present invention relates to the field of multimedia technology, and discloses a synchronous control method and system for a distributed video image splicing large screen. The method includes the following steps: S1: Construct a distributed video image splicing synchronous control system; S2: Multiple video input nodes respectively send the collected video image signals to the local main control computer, so that the video stream frame rate and time stamp of the video signals displayed on each group of unit display screens of the array display device are kept synchronous; S3: Synchronously control multiple remotely and distributedly arranged main control computers, so that the video stream frame rate and time stamp sent to the video input nodes of each local cloud splicing front-end processor and the array display device are kept consistent. The present invention adopts a combination of software and hardware, and a combination of local and remote control to ensure that the time stamp of each frame of video is consistent, and solves the problems of dislocation and tearing caused by asynchronous display of the distributed video image splicing large screen.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technologies, and particularly to a synchronous control method and system for distributed video image splicing. Background Art

[0002] In applications such as big data analysis, display platforms, intelligent monitoring platforms, and multimedia performance settings, it is often necessary to use an array of large screens (large video splicing walls) to display the programs running on the target terminal devices to be monitored and perform intelligent analysis and processing. With the development of display processing technologies from traditional single-machine splicing operations, these applications of large video splicing walls can no longer meet the requirements of multi-point and multi-distribution audio-visual display processing for interconnection and interoperability. For the extended network-distributed video processing products developed using existing technologies, due to limitations in network transmission bandwidth (channel congestion) and signal fluctuations, video frame delays or frame losses often occur. In practical applications, due to reasons such as different frame times of each decoding terminal, as the number of individual display screen units that make up the array of large video splicing walls increases, during the process of transmitting the spliced video source to each display screen unit for synchronous display, the situations of image asynchronization, image misalignment, and screen delay become more serious. To ensure the quality of the displayed image, the number of individual display screen units participating in the splicing of the large video splicing wall must be restricted, thereby greatly limiting the overall area and functionality of the large video splicing wall. To solve these problems, distributed synchronization technologies need to be adopted.

[0003] Chinese Invention Application No. CN201210029784.1 discloses a multi-screen display system picture synchronization technology based on network transmission. The multi-screen display system is composed of multiple display units spliced together, and the picture synchronization process is completed in the signal processing system. The signal processing system consists of distributed input nodes and output nodes. Each display unit is connected to a unique corresponding output node. The input nodes are responsible for collecting picture information and preprocessing the picture information, and then transmitting it to the corresponding output nodes. Under the control of a synchronization signal, all output nodes send the corresponding picture information to their respective display units each time to achieve the purpose of picture synchronization. However, this technology uses local area network cables to connect between each input node and output node, with poor scalability, and cannot be directly applied to wireless network connections. If wireless network signals are used to connect between its input nodes and output nodes, it still cannot solve the problem of signal asynchronization between each input node and output node caused by reasons such as external network transmission and synchronization signal reception delay after the signal processing system sends out the output signals of each node.

[0004] Chinese Invention Application No. CN201910847059.7 discloses a high-precision distributed display control frame synchronization method and system, including: performing network time synchronization operations on multiple distributed output nodes in a distributed splicing display system; obtaining a preset frame rate of the distributed splicing display system, and determining a phase to be synchronized according to the preset frame rate; adjusting the phase of the synchronization signal between the multiple distributed output nodes to be consistent with the phase to be synchronized; respectively obtaining the first timestamp information of the current display frame of the master node, the first time information of the first synchronization signal, and the second timestamp information and the second time information of the second synchronization signal of each slave node; determining the time offset information of the same frame between each slave node and the master node; and performing frame synchronization operations on each slave node relative to the master node based on the relationship between the time offset information and the preset frame rate. The core of this invention is to use the 1588 network time synchronization method to perform network time synchronization operations on multiple distributed output nodes in the distributed splicing display system. The 1588v2 time synchronization principle is: The PTP protocol - IEEE 1588V2 adopts a master-slave clock scheme, periodically publishes clocks, and the receiving party uses the symmetry of the network link to measure clock offset and delay measurement to achieve the synchronization of the frequency, phase, and absolute time of the master-slave clocks. However, the synchronization of this technical solution still requires obtaining the time offset information of the same frame between each slave node and the master node by synchronizing the time signal through an external network (such as 1588 network time). If the distributed splicing display system cannot receive this external network synchronization time signal, it cannot complete the time synchronization adjustment of each node, and cannot fundamentally solve the problem of the system's dependence on external network signals.

[0005] In addition, with the development of technology, existing distributed video processing products are limited by network transmission bandwidth limitations and fluctuations, which will cause delays or frame losses of video frames. In practical applications, it is difficult to simultaneously control the overlay, roaming, synchronization, etc. of local high-definition video signals and remote high-definition video signals; and due to reasons such as network latency and frame time asynchronization of each decoding terminal, image asynchronization, image misalignment, picture delay, jitter, frame loss, etc. will occur in the technology of video source splicing display, resulting in a poor image splicing display effect; especially when applied to small-pitch LEDs, due to the absence of physical seams and worse fault tolerance, the requirements for the picture display synchronization effect are more stringent. When playing videos with intense movement and overlay roaming applications, obvious tearing usually occurs because at this time, due to the problem of limited physical space in the data link, the conventional distributed splicing display system and control method can no longer meet its requirements, and can only be solved through new technical ideas. Summary of the Invention

[0006] (1) Technical problems to be solved

[0007] In view of the above deficiencies of the prior art, the present invention provides a synchronous control method and system for distributed video image splicing. By providing a set of methods that do not rely on external synchronous network signals, the AI server controls each distributed master computer, and the time reference signals generated by each local node and the timestamps of each node are compared. Through an intelligent time algorithm, the defects existing in the network transmission process are compensated. By calculating the time differences of each node and comparing them, the correct timestamp data is synchronized to each decoding terminal, so as to achieve the purpose of accurate synchronization of the video splicing signals transmitted to each single-group display screen unit both in the online and offline states. It can completely solve the problems of asynchronous, misaligned and torn images during the video stream decoding and splicing display of network distributed processors due to factors such as network transmission delay and bit error rate. Thus, the local high-definition video signal and the remote high-definition video signal are synchronously processed, and the superimposition, cutting, roaming and synchronous control of the video signal are realized.

[0008] (2) Technical solutions

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A synchronous control method for a distributed video image splicing large screen, characterized in that it includes the following steps:

[0011] S1: Construct a distributed video image splicing synchronous control system, which includes the following multiple distributed components interconnected through a network: multiple array display devices (distributed cloud splicing large screens) composed of M×N multiple groups of unit display screens, multiple cloud splicing front-end processors with built-in decoding and splicing matrix processing programs, multiple master computers with built-in distributed splicing control programs, multiple network switching devices, and at least one AI server with a built-in deep learning program; each local master computer is connected to one or more cloud splicing front-end processors, and each cloud splicing front-end processor is connected to one or more local array display devices; each cloud splicing front-end processor is provided with multiple video input nodes and multiple output nodes, with one of the video input nodes as the master input node and the others as ordinary input nodes, and one of the video output nodes as the master output node and the others as ordinary output nodes; high-precision crystal oscillators are provided in each of the video input nodes and output nodes.

[0012] S2: Multiple video input nodes respectively send the collected video image signals to the local master computer. The distributed splicing control program built in the master computer monitors, compares, and calibrates the timestamps of the video streams sent by each video input node. The calibration method is as follows: The high-precision crystal oscillators inside each video input node generate timestamps, which are sent to the AI server together with the processed collected video streams; The distributed splicing control program continuously and real-time monitors the video stream frame rates and timestamps of each output node playing the video streams of the input nodes, and makes comparisons. When a deviation in the timestamp of a certain node is detected, the correct timestamp is immediately sent to all video input nodes in the network for calibration, so that the video stream frame rates and timestamps of each video input node are kept consistent, and then processed and output to each video output node, so that the video stream frame rates and timestamps of each video output node are kept synchronized, thereby making the video stream frame rates and timestamps of the video signals displayed on each group of unit display screens of the array display device (distributed cloud splicing large screen) kept synchronized;

[0013] S3: The deep learning program built in the AI server synchronously controls multiple master computers set up remotely and distributively through deep learning algorithms and network buffering, so that the video stream frame rates and timestamps sent to the video input nodes of each local cloud splicing front-end processor and the array display device are kept consistent.

[0014] A synchronous control system for a distributed video image splicing large screen implementing the foregoing method, characterized in that it includes the following multiple distributed components interconnected through a network: multiple array display devices (distributed cloud splicing large screens) composed of M×N groups of unit display screens, multiple cloud splicing front-end processors with built-in decoding splicing matrix processing programs, multiple master computers with built-in distributed splicing control programs, multiple network switching devices, and at least one AI server with a built-in deep learning program; Each local master computer is connected to one or more cloud splicing front-end processors, and each cloud splicing front-end processor is connected to one or more local array display devices; Each cloud splicing front-end processor is provided with multiple video input nodes and multiple output nodes, with one of the video input nodes as the master input node and the others as ordinary input nodes, and one of the video output nodes as the master output node and the others as ordinary output nodes; High-precision crystal oscillators are provided inside each video input node and output node.

[0015] (III) Beneficial effects

[0016] Compared with the prior art, the synchronous control method and system for a distributed video image splicing large screen provided by the present invention have the following beneficial effects:

[0017] (1) The synchronous control method and system for distributed video image splicing provided by the present invention provide a set of methods that do not rely on external synchronous network signals. The AI server controls each distributed main control computer, and the time reference signals generated by each local node and the timestamps of each node are compared. Through an intelligent time algorithm, the defects existing in the network transmission process are compensated. By calculating the time differences of each node and comparing them, the correct timestamp data is synchronized to each decoding terminal, so as to achieve the purpose of accurate synchronization of the video splicing signals transmitted to each single-group display screen unit in both online and offline states. It can completely solve the problems of asynchronous, misaligned, and torn images caused by factors such as network transmission delay and bit error rate during the video stream decoding and splicing display of network distributed processors. Thus, the local high-definition video signal and the remote high-definition video signal are synchronously processed, and the superposition, cutting, roaming, and synchronous control of video signals are realized.

[0018] (2) The synchronous control method and system for distributed video image splicing provided by the present invention specifically solve the problem that with the development of display processing technology, traditional single-machine splicing operations can no longer meet the application requirements of multi-point and multi-distribution audio and video display processing for interconnection and interoperability in large video splicing walls. According to this problem and the development of technology, a variety of network distributed video processing products have emerged. However, due to the limitations of network transmission fluctuations, video frame delays or frame losses will occur. In actual applications, due to the asynchronous frame times of each decoding terminal, image asynchrony, image misalignment, picture delay, and frame loss phenomena will occur in the video source splicing display technology, greatly reducing the display effect. Especially in the application of small-pitch LEDs, due to the lack of physical splicing seams, the display synchronization effect is more stringent, and tearing is particularly obvious when playing videos with intense motion. Due to the problem of physical limitations in this link, conventional network splicing can no longer meet this kind of application. The present invention needs to adopt a method combining intelligent AI algorithms and hardware to make up for the defects existing in the prior art. By combining remote control and local control respectively, the video stream frame rates and timestamps of the video input nodes of each local cloud splicing front-end processor and array display device are kept consistent.

[0019] (3) The synchronous control method and system for distributed video image splicing provided by the present invention, through the combination of software and hardware, and the combination of remote control and local control, solve the problems of misalignment and tearing phenomena caused by asynchronous splicing due to bit error rate caused by network delay during video stream decoding and splicing display of network distributed processors. It can support the input signals of multiple levels and multiple locations of remote and local to achieve arbitrary position display, overlay roaming application, so that the splicing of each output card and display screen in the entire network uses network switching splicing, solve the problem of inconsistent timing of each output caused by packet loss in the network due to the disadvantages of network switching, and can ensure that there is no misalignment and tearing phenomenon in each frame of the picture image splicing when the screen and the screen are played, so that each group of unit display screens in the array display device in the network can be accurately synchronized with each other in the spliced picture of each frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the network composition structure of the synchronous control system for distributed video image splicing according to an embodiment of the present invention.

[0021] Figure 2 It is a schematic diagram of the composition structure of the synchronous control method and system module according to an embodiment of the present invention;

[0022] Figure 3 It is a schematic flow chart of the synchronous control method of the local cloud splicing front-end processor according to an embodiment of the present invention;

[0023] Figure 4 It is a schematic flow chart of the synchronous control method of the local array display device according to an embodiment of the present invention;

[0024] Figure 5 It is a schematic flow chart of the timestamp synchronization correction control method of the local video input node according to an embodiment of the present invention;

[0025] Figure 6 It is a schematic flow chart of the synchronous control method for the AI server to remotely control the local main control computer according to an embodiment of the invention;

[0026] Figure 7 It is a schematic flow chart of the synchronous control method for the remote local main control computer (slave) and the local main control computer (master) according to an embodiment of the invention;

[0027] Figure 8 It is a schematic diagram of the network topology structure of the synchronous control system according to an embodiment of the invention.

[0028] In the figure: 100, main control computer; 101, target application window information capturing program S end; 102, encoding module; 200, cloud splicing front-end processor; 203, FPGA module; 204, decoding module; 205, array display output module; 300, terminal for running the target application; 301, target application window information capturing program B end; 400, array display device. Detailed implementation manners

[0029] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] Embodiment

[0031] Embodiment 1:

[0032] Please refer to the attached Figures 1 - 8 , this embodiment is a specific application case of a synchronous control method and system for a distributed video image splicing large screen, which includes multiple remote video image splicing large screens (array display devices) and at least one local video image splicing large screen (array display device). Among them, the local array display device is composed of 36 splicing screens and is spliced in a 6×6 manner. The project requires that the remotely input signals (including real-time capture of remote and local) can be displayed at any position, with overlay roaming applications. The splicing of each output card and the output screen unit uses network switching splicing. In order to solve the problems of inconsistent video stream frame rate and timestamp of each output picture caused by network packet loss, latency, etc. in the network exchange between remote and local, resulting in misalignment, tearing and other phenomena, and to ensure that when each screen of each distributed local array display device in the entire network plays the screen image, the splicing of each frame of the picture image can be accurately ensured to be consistent, a combination of software and hardware and a combination of remote and local control are adopted to achieve frame synchronization of each frame of the picture spliced by any local array display device in the whole network.

[0033] The synchronous control system for a distributed video image splicing large screen provided in this embodiment can adopt a B / S network architecture or a C / S network architecture, or other combined heterogeneous network architectures. It includes the following multiple distributed components connected to each other through the network: multiple local array display devices (i.e., distributed cloud splicing large screens) composed of 6×6 total of 36 multi-unit display screens, and multiple distributed remote array display devices (attached Figure 8not shown); multiple cloud splicing front - end processors with built - in decoding splicing matrix handlers, multiple master computers with built - in distributed splicing control programs, multiple network switching devices, at least one AI server with built - in deep - learning program (not shown in the figure and actually connected to the master computer), multiple video signal acquisition terminal devices; wherein, each local master computer is connected to one or more cloud splicing front - end processors, and each cloud splicing front - end processor is connected to one or more local array display devices; each cloud splicing front - end processor is provided with multiple video input nodes and multiple output nodes, with one of the video input nodes as the master input node and the others as ordinary input nodes, and one of the video output nodes as the master output node and the others as ordinary output nodes; high - precision crystal oscillators are provided in each of the video input nodes and output nodes. Figure 8 not shown in the figure and actually connected to the master computer), multiple video signal acquisition terminal devices; wherein, each local master computer is connected to one or more cloud splicing front - end processors, and each cloud splicing front - end processor is connected to one or more local array display devices; each cloud splicing front - end processor is provided with multiple video input nodes and multiple output nodes, with one of the video input nodes as the master input node and the others as ordinary input nodes, and one of the video output nodes as the master output node and the others as ordinary output nodes; high - precision crystal oscillators are provided in each of the video input nodes and output nodes.

[0034] The synchronous control method for the distributed video image splicing large screen provided in this embodiment includes the following steps:

[0035] S1: First, construct a Figure 1 shown distributed video image splicing synchronous control system. This system includes the following multiple distributed components interconnected through a network: multiple array display devices (distributed cloud splicing large screens) composed of M×N groups of unit display screens, multiple cloud splicing front - end processors with built - in decoding splicing matrix handlers, multiple master computers with built - in distributed splicing control programs, multiple network switching devices, at least one AI server with built - in deep - learning program; each local master computer is connected to one or more cloud splicing front - end processors, and each cloud splicing front - end processor is connected to one or more local array display devices; each cloud splicing front - end processor is provided with multiple video input nodes and multiple output nodes, with one of the video input nodes as the master input node and the others as ordinary input nodes, and one of the video output nodes as the master output node and the others as ordinary output nodes; high - precision crystal oscillators are provided in each of the video input nodes and output nodes.

[0036] S2: Multiple video input nodes respectively send the collected video image signals to the local master computer. The distributed splicing control program built in the master computer monitors, compares, and calibrates the timestamps of the video streams sent by each video input node. The calibration method is as follows: The high-precision crystal oscillators inside each video input node generate timestamps, which are sent to the AI server together with the processed collected video streams. The distributed splicing control program continuously monitors the video stream frame rates and timestamps of each output node playing the video streams of the input nodes in real time, and makes comparisons. When a deviation in the timestamp of a certain node is detected, the correct timestamp is immediately sent to all video input nodes in the network for calibration, so that the video stream frame rates and timestamps of each video input node are kept consistent, and then processed and output to each video output node, so that the video stream frame rates and timestamps of each video output node are kept synchronized, thereby ensuring that the video stream frame rates and timestamps of the video signals displayed on each group of unit display screens of the array display device (distributed cloud splicing large screen) are kept synchronized.

[0037] S3: The deep learning program built in the AI server synchronously controls multiple master computers that are remotely and distributedly set through deep learning algorithms and network buffering, so that the video stream frame rates and timestamps sent to the video input nodes of each local cloud splicing front-end processor and array display device are kept consistent.

[0038] More specifically, in step S1, the signal sources of the multiple video input nodes include: remote video input signal sources and local video input signal sources; the network includes a wired network and a wireless network; each output node is respectively connected to a corresponding group of unit display screens.

[0039] In step S1, the synchronous control system for the distributed video image splicing large screen further includes multiple remote mobile control terminals connected to the master computer through the network; each remote mobile control terminal sends control signals to the master computer through the wireless network, and then controls other devices in the network to achieve synchronous playback of the distributed video image splicing large screen.

[0040] The AI deep learning program built in the AI server in step S1, according to the network fluctuation frequencies of each input node and output node connected to each distributed master computer in the network, especially the fluctuating network fluctuation frequencies of the master input node and master output node, obtains, through the main AI deep learning algorithm, an algorithm for the local master input node and master output node controlled by each master computer to correct the time of other nodes and reduce the frequency difference synchronization to within a set range, ensuring that the time difference and frequency difference synchronization of each node are adjusted to be consistent.

[0041] For the synchronous control method of the distributed video image splicing large screen, in step S2, the steps for the master computer to synchronize the video frames sent by each local video input node within the network are as follows:

[0042] S21: Start each input node and output node within the network, connect and calibrate the interface clocks of each input node after startup to ensure consistent time;

[0043] S22: Configure the clock parameters of each output node before system initialization. When initially starting the output video job, first call the clock interface of the initial output node according to the output process;

[0044] S23: Each input node starts the video acquisition job. The AI server calls the clock interfaces of each input node, turns on or off the clock interfaces, and restarts the clock interfaces and obtains correct clock data when bit errors occur;

[0045] S24: Under the control of the AI server, when the decoding and splicing matrix processing program of the cloud splicing front-end processor decodes and sends the video signals of the input nodes to the output nodes, it performs the actions of turning off and then turning on the clocks of each output node to ensure consistent timestamps for each output node; Turning off and turning on the clock is an operation on the corresponding bit of the clock. The time required to turn off and start the clock is 1 millisecond, which does not affect the displayed image effect;

[0046] S25: Under the control of the AI server, start each remote distributed master computer, cloud splicing front-end processor, and array display device in sequence. Each distributed cloud splicing front-end processor starts local decoding and sends the action of the video frames of the input nodes to the output nodes for decoding, ensuring that the frame rates and times are consistent when the video frames of each input node reach each output node. Each output node outputs to the corresponding unit display screen for presentation, achieving complete synchronous splicing of the video images of each group of unit display screens of the array display device;

[0047] S26: Under the control of the AI server, after the input nodes and output nodes of each array display device start, if a time difference is found in any input node, the AI server performs the action of comparing the timestamps of all input nodes within the network. If the AI server detects bit errors, it repeats steps S23, S24, and S25 above for recalibration, so that the frame rates and times of the input nodes and output nodes of each array display device are kept consistent;

[0048] S27: Each master output node of the cloud splicing front-end processor is built-in with a programmable differential high-precision crystal oscillator. This crystal oscillator is based on the basic circuit for clock production, and obtains the signal error code fluctuation range by calibrating the bit error rate of network transmission for each output node; edits the oscillation frequency offset data of the programmable crystal oscillator through the master output node, calls the preset comparative deep learning calculation method in the AI server, calculates the correction frequency data of the master control, and then uses this correction frequency data to correct the frequency data of all other input nodes within the same network, so that the frequency differences of all input nodes are reduced to the set range.

[0049] For the synchronous control method of the distributed video image splicing large screen, in step S3 thereof, the steps of synchronously controlling the videos of each output node and calibrating the timestamps are as follows:

[0050] S31: The AI server corrects the timestamps of multiple output nodes respectively by controlling the decoding and splicing matrix processor: when the video streams of each input node are sent to each output node for playing, calculates the frame rate and timestamp of the video streams of each output node; the master output node monitors the timestamp comparison of the nodes playing the video stream in real time. If it is found that there is an error code in node A among them, this master output node immediately sends the correct timestamp to all output nodes such as node B and node C within the same network for calibration, ensuring that the timestamps of each frame of video are all the same, and realizing the synchronous output of the videos of each unit display screen of each array display device.

[0051] S32: The steps of the AI server for synchronously controlling the videos of each output node and calibrating the timestamps are as follows: the AI server controls each output node to first perform video frame buffer processing and then perform frame synchronization call, specifically: first cache a video frame to be output in the renderer of each output node, and synchronize the timestamp of the cached frame to the AI server in real time after caching; when the AI server decides to render a certain frame, simultaneously sends a command to render a certain cached frame to multiple distributed master computers; if the video to be played is 30 frames per second, the AI server will send 30 commands to render the cached frame to the master computers of all output nodes per second; after the video frame cached in the memory is rendered and before it is presented on the screen, first copy the rendered data to the video memory of each output node with output. After each output node receives the rendering command, it can output without rendering delay.

[0052] S33: After each output node of the master computer receives the rendering command, before rendering, the output node corresponding to the main screen first checks the caches of all output nodes corresponding to the secondary screens. If the current frame to be rendered exists in all the caches of the secondary screens, immediately synchronously render, so as to make the timestamps of each output node the same and the frame rates the same when playing the video, and make the video data playback of each group of unit display screens synchronous.

[0053] See the appendix Figures 3 - 4 In this embodiment, for the distributed tiled display of the display screens of each unit in the local network, it consists of a master control output node, slave output nodes, signal access input nodes, and network switching devices. Between each node, the master node monitors and compares the timestamps of the video streams through the network. The time calibration is processed based on the internal high-precision crystal oscillator. It continuously monitors the video stream frame rate and timestamp of each output node playing the input node, and when a time deviation is detected at a certain node, it sends the correct timestamp for calibration processing. Each output node maintains the same time to achieve synchronization. For example, the master control output node 1 starts to send timestamps to the slave nodes 2, 3, and 4. The input nodes and slave nodes calibrate the time. After each node receives the master node, it continuously sends timestamp information to the master node. The timestamps output by each node are kept consistent. If the timestamp sent by one or more nodes deviates, or the master monitors incorrect timestamp information, it will resend the normal time information for real-time calibration. The processing time is at the millisecond level and does not affect the effect of the screen tiled display.

[0054] Among them, the synchronous service workflow steps for sending video frames composed of multiple video input nodes and output nodes within a local network are as follows:

[0055] Step 1: Enable the interface clocks of each input and output node to start and perform connection calibration to ensure the same time;

[0056] Step 2: Start the service. Before the system initialization, configure the clock parameters of the module. When initially starting the output service, still call the node clock interface according to the output process

[0057] Step 3: Call the clock interfaces of each node to call the open / close clock interface to restart and obtain the normal clock when an error code appears;

[0058] Step 4: When the input node sends a signal to the output node for decoding, re-close and then open the clocks of each output node to ensure that the timestamps of each node are the same. Closing and starting the clock only operates on the corresponding bits of the clock. The closing and starting time is at the millisecond level and does not affect the image effect being displayed;

[0059] Step 5: Start each device to start the action of decoding and sending the video frames of the input node to the output node for decoding to ensure that the frame rate and time of the video frames reaching the output node are the same, achieving complete synchronization of the tiling;

[0060] Step 6: After each service function starts, when a time difference appears, the master node performs a timestamp comparison action. After the master node detects an error code, it repeats the actions in steps 3, 4, and 5 above to re-calibrate the slave nodes to maintain the same tiling.

[0061] The synchronous control method and system for the distributed video image splicing large screen provided by this embodiment support the processing and distribution of real-time monitoring and real-time capture signals, can achieve real-time synchronization, cover real-time video distribution in multiple cities and multiple scenarios, support multi-site synchronous deployment, support flexible switching of the screen without roaming, and can be accessed at any time by monitoring centers, data centers, exhibition centers, etc., completely breaking through the limitations of different geographical distributions.

[0062] The cloud splicing front-end processor (i.e., the decoding splicing matrix processor) adopted in this embodiment supports multi-screen splicing and segmentation, with a maximum support for 36-screen segmentation and arbitrary segmentation within 1 - 36 screens; it supports multiple control methods (keyboard, mouse, tablet, mobile phone, etc.).

[0063] The array display device adopted in this embodiment, which consists of multiple M×N groups of unit display screens, is an overall seamless splicing curtain wall large screen composed of 6*6 LED panels spliced together; the resolution of each LED panel is 4K, and the resolution of the played video source is also 4K. Each LED is connected to a local main control computer (computer host), and local video synchronous playback is achieved through local area network communication. Since the entire large screen space displays a complete picture, it is required that the video frames of each LED can be precisely synchronized, otherwise the picture will be split. The traditional synchronization method is to send playback instructions to these several hosts simultaneously by the central control end after each host is ready, so that the videos play together. However, this often leads to deviations after playing for a period of time. Therefore, the control method provided by the present invention must be adopted to achieve local and remote distributed synchronous control.

[0064] The video frame buffer processing process adopted in this embodiment is as follows: The playback of a video usually goes through a separator -> decoder -> renderer. We cache the video frames in the renderer and synchronize the timestamps of the cached frames to the server in real time. When the server decides to render a certain frame, it sends instructions to render a certain cached frame to multiple hosts simultaneously. If the video to be played is 30 frames per second, then the server will send instructions to render the cached frames 30 times per second to all hosts. The video frames cached in the memory need to be copied to the video memory first to be presented on the screen. Since the resolution is usually very high, this copying also takes time. Fortunately, the video memories of current graphics cards are very large. For greater precision, the present invention directly caches the video frames in the video memory. This ensures that there is no delay in rendering after receiving the rendering instruction.

[0065] See Appendix Figure 5 , the high-precision crystal oscillator calibration judgment basis method for distributed video synchronization adopted by the present invention includes the following steps:

[0066] Each output node master controls the basic circuit produced based on the clock using a programmable differential high-precision crystal oscillator. The signal error fluctuation range is obtained by calibrating the bit error rate of the network transmission of each node. A set of comparison calculation methods is formulated by the master editing the frequency offset data of the programmable crystal oscillator. For example, when node A shows 0 and node B shows 1, the master uses the above-mentioned structure to obtain the calculation formula to judge and calibrate the basic bit error rate of each node to achieve the method of time consistency. By combining the network fluctuation frequencies of each node input node, output node, master node, etc. of the network, correction is performed by the master to reduce the frequency difference, ensuring that the frequency of the time difference of each node is reduced to achieve the method of accurate time.

[0067] See the appendix Figures 6 - 7 , the video synchronous playback communication protocol adopted in this embodiment is based on the UTP protocol. Among them, the main screen is the server side, and the secondary screen is the client side. The IP address and port of the main screen are registered on the secondary screen. Among them, the secondary screen actively connects to the main screen. If the connection is disconnected, it will reconnect infinitely; a status array is maintained on the main screen to record the status of all secondary screens in real time; each connection is maintained by a thread.

[0068] 1. Data packet format

[0069] Data packet start identifier: starts with 0x02 and ends with 0x03. Data packet content format: The content is 14 ascii characters. The first 4 characters represent the command, and the following 10 characters represent the parameters. Each data packet is 16 bytes.

[0070] 2. Command definition example

[0071] Send the HELO character; after the secondary screen connects to the main screen, it actively sends HELO. After the main screen receives HELO, it needs to immediately reply with a HELO; if the secondary screen does not receive the reply from the main screen within two seconds after sending HELO, it can be considered that the connection is not successful; the parameter format is 10 zeros; send PLAY, the main screen sends it to the secondary screen to play the video from the beginning. The parameter format is AA0000 + 4-bit video number. AA is a two-digit number, and the maximum amount of cached video frames; if the video number is "OPEN", or the secondary screen does not have this video number, it means that a file open dialog box needs to be displayed; send REND, the main screen sends it to the secondary screen to render the video frame, and the parameter format is a 10-bit timestamp. Send JUMP, the main screen sends it to the secondary screen to jump, and the parameter format is a 10-bit timestamp; after jumping, the secondary screen is automatically cleared. Send CACH, the secondary screen sends it to the main screen for the cached video frame, and the parameter format is a 10-bit timestamp. Send CACR; the secondary screen sends it to the main screen for the deleted cached video frame, and the parameter format is a 10-bit timestamp; CLCA, the secondary screen sends it to the main screen for deleting all cached video frames.

[0072] Before the main screen is rendered, check the caches of all secondary screens first. If the current frame to be rendered exists in the caches of all secondary screens, render it immediately in synchronization to ensure consistent timestamps and frame rates during video playback, thus achieving synchronized video playback.

[0073] The synchronous control method and system for the distributed video image splicing large screen provided in this embodiment support large screens with 6*6 splicing at a total of 4 locations, including local and remote locations. After actual testing, the present invention can achieve 4K ultra-high definition decoding with a resolution of 4KX2K (4096*2160), support image roaming and seamless switching, and can be widely applied to construction projects such as control centers in multiple fields including public security, fire protection, military, meteorology, railway, and aviation.

[0074] See the attached Figure 2 , the real-time video signal acquisition and network push method adopted in the embodiment of the present invention includes the following steps:

[0075] (1) Set a main control computer 100, a cloud splicing front-end processor 200 network-connected to the main control computer 100, and multiple terminal machines 300 running target application programs; the control programs built in the main control computer 100 and each terminal machine 300 are respectively installed with S-side and B-side target application program window information capture programs in the B / S architecture; the main control computer 100 is also built-in with an encoding module 102; the built-in B / S architecture program can work independently, for example, the FastStone Capture program can be used for screen capture, screen recording; or multiple programs can work in parallel: for example, the snipaste software can be used for screenshot and pasting; the source code for capturing the screen or the current active window can be obtained by calling the Win32 API (such as specifically calling the API function "BitBlt", or using GDI32.dll to achieve screen capture or capture of the current active window); software such as a window information acquirer and a window information extractor developed in c# can be used; audio data can be extracted using Pazera FreeAudio Extractor, Abelssoft MusicExtractor, etc.; each program runs in parallel, captures the corresponding data respectively, and sends it to the main control computer 100.

[0076] The cloud splicing front-end processor 200 includes: a target application program window information receiving module 201, an FPGA module 203, a decoding module 204, and an array display output module 205 that are interconnected;

[0077] (2) Power on and run. The target application window information capture program at the B side in each terminal captures the window information (including but not limited to the UI interface) data of the applications running on each terminal, and sends it to the main control computer 100 at the S side through the network. The target application window information capture program at the S side performs data processing such as sorting, grouping, and packing on it, and then the encoding module 102 performs audio and video encoding according to the set encoding standard;

[0078] (3) The main control computer 100 sends the processed window information (audio and video, images) encoded data to the target application window information receiving module 201 of the cloud splicing front-end processor 200 through the streaming media transmission network. After receiving it, the receiving module transfers it to the FPGA module 203;

[0079] (4) The FPGA module 203 processes the window information encoded data, including: switching and splitting one encoded window image into multiple window images, or splicing multiple encoded window images into one window image, and then performing grouping, packing, and sorting to generate new window information encoded data;

[0080] Among them, the FPGA module 203 of the cloud splicing front-end processor 200 performs arbitrary window arrangement display, layer overlay, window roaming, and picture segmentation operation processing on the received window information, and then sends the obtained streaming media (image video stream) information to the array display output module 205 for final integrated array window information display;

[0081] (5) The decoding module 204 decodes the new window information encoded data, restores the decoded data to window information according to grouping, packing, and sorting, and then generates a streaming media signal according to the rules set by the user and sends it to the array display output module 205 specified by the user;

[0082] (6) The array display output module 205 sends the streaming media signal to an external array display device (400) for integrated array window information display to complete information push.

[0083] The target application window information capture program at the S side or B side captures the window information of the target applications running on each terminal through window monitoring and remote control of the streaming media network; the target application window information includes text, symbols, video, audio, or image information that appears in the active window, window / object, rectangular area, hand-drawn area, entire screen, scrolling window, or fixed area of the target application.

[0084] The target application window information capture program S end 101 or B end 301 dynamically captures the multi-layer application window information of a single terminal (computer) through dynamic control of the streaming media transmission protocol.

[0085] In the embodiment of the present invention, the streaming media may adopt one of the following network protocols, including: real-time transport protocol RTP (Real-time Transport protocol); real-time transport control protocol RTCP (Real-time Transport Control protocol), real-time streaming protocol RTSP (Real Time Streaming protocol).

[0086] The encoding module (102) used in the present invention is a HiSilicon HI3521A encoding chip; the decoding module

[0087] (204) is a HiSilicon HI3536 decoding chip, which has a built-in SOC application processing program for receiving, grouping, packaging and sorting window information.

[0088] The cloud splicing front-end processor (200) is a GS6000 cloud splicing image processor; the multiple terminals running the target application programs are one of a computer terminal, an intelligent device terminal or a PLC terminal;

[0089] The embodiment of the present invention can realize the multi-target application window information capture and screenshot functions, and can capture: active window, window / object, rectangular area, hand-drawn area, entire screen, scrolling window, fixed area, screen recording (output format is WMV), and after processing, network push system according to customer needs. The multi-target video information capture of the present invention includes full-screen capture, current active window capture, capture of selected area, polygon capture and capture of scrolling page, etc.

[0090] The present invention adopts a combination of software and hardware, and local and remote control to correct the timestamps of multiple output nodes of the distributed splicing processor. When the video stream of the input node is given to each output node for playback, the output node performs comparative calculations based on the frame rate and timestamp of the video stream. The main control monitors the timestamp comparison of the node that is playing the video stream in real time. For example, when a bit error occurs at node A, the main control immediately sends the correct timestamp to node B or node C in the network for calibration, ensuring that the timestamp of each frame of video is consistent to achieve video synchronization effect. This solves the problem of the bit error rate caused by network delay when decoding, splicing and displaying the video stream of the network distributed processor, resulting in misalignment, tearing and other undesirable phenomena of splicing asynchrony, so that the display screens of each splicing unit of the large screen can display the video pictures smoothly and smoothly with no phenomena of sometimes fast and sometimes slow, jitter, error, tearing and the like.

[0091] The present invention performs distributed capture, editing, and integrated display of local and remote signal sources through network devices. All use network connections, with fewer devices used and high operating efficiency, and can be widely applied to technical fields such as big data platforms and integrated intelligent monitoring.

[0092] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A synchronous control method for a distributed video image stitching large screen, characterized in that, It includes the following steps: S1: Construct a distributed video image splicing and synchronization control system, which includes the following multiple distributed components interconnected through a network: multiple array display devices composed of multiple groups of unit display screens, multiple cloud splicing front-end processors with built-in decoding splicing matrix processing programs, multiple master computers with built-in distributed splicing control programs, multiple network switching devices, and at least one AI server with a built-in deep learning program; each local master computer is connected to one or more cloud splicing front-end processors, and each cloud splicing front-end processor is connected to one or more local array display devices; each cloud splicing front-end processor is provided with multiple video input nodes and multiple output nodes, with one of the video input nodes as the master input node and the others as ordinary input nodes, and one of the video output nodes as the master output node and the others as ordinary output nodes; high-precision crystal oscillators are provided in each of the video input nodes and output nodes; S2: Multiple video input nodes respectively send the collected video image signals to the local master computer. The distributed splicing control program built in the master computer monitors, compares, and calibrates the timestamps of the video streams sent by each video input node. The calibration method is as follows: Timestamps are generated by the high-precision crystal oscillators inside each video input node and sent to the AI server together with the processed collected video streams; the distributed splicing control program continuously and real-time monitors the video stream frame rate and timestamp of each output node playing the video stream of the input node, and makes a comparison. When a deviation in the timestamp of a certain node is detected, the correct timestamp is immediately sent to all the video input nodes in the network for calibration, so that the video stream frame rate and timestamp of each video input node are kept consistent, and then processed and output to each video output node, so that the video stream frame rate and timestamp of each video output node are kept synchronized, thereby keeping the video stream frame rate and timestamp of the video signals displayed on each group of unit display screens of the array display device synchronized; S3: The deep learning program built in the AI server synchronously controls multiple master computers set up remotely and distributedly through deep learning algorithms and network buffering, so that the video stream frame rate and timestamp sent to the video input nodes of each local cloud splicing front-end processor and array display device are also kept consistent.

2. The synchronous control method of the distributed video image stitching large screen according to claim 1, characterized in that, In the step S1 described above, the signal sources of the multiple video input nodes include: remote video input signal sources and local video input signal sources; the network includes a wired network and a wireless network; each of the output nodes is respectively connected to a corresponding group of unit display screens.

3. The synchronous control method of the distributed video image stitching large screen according to claim 2, wherein, In the step S1 described above, it also includes multiple remote mobile control terminals connected to the master computer through the network; each remote mobile control terminal sends control signals to the master computer through the wireless network, and then controls other devices in the network to achieve synchronous playback of the distributed video image splicing large screen.

4. The synchronous control method for the distributed video image stitching large screen according to claim 1, characterized in that, The AI deep learning program built into the AI server in step S1 obtains, through the main AI deep learning algorithm, the algorithm for time correction of other nodes by the local master input node and master output node controlled by each master computer based on the network fluctuation frequencies of each input node and output node connected to each distributed master computer in the network, including the master input node and master output node, so as to ensure that the time difference and frequency difference synchronization of each node are adjusted to be consistent.

5. The synchronous control method of the distributed video image stitching large screen according to claim 1, characterized in that In step S2 described above, the synchronization steps of the master computer for the video frames sent by each video input node in the network are as follows: S21: Start each input node and output node in the network, connect and calibrate the interface clocks of each input node after startup to ensure consistent time; S22: Configure the clock parameters of each output node before system initialization. When initially starting the output video job, first call the clock interface of the initial output node according to the output process; S23: Each input node starts the video acquisition job. The AI server calls the clock interfaces of each input node, turns on or off the clock interfaces, and restarts the clock interfaces and obtains correct clock data when errors occur; S24: Under the control of the AI server, when the decoding splicing matrix processing program of the cloud splicing front-end processor decodes and sends the video signal of the input node to the output node, it performs the actions of restarting and turning on the clocks of each output node to ensure that the timestamps of each output node are consistent; turning off and turning on the clocks is to operate on the corresponding bit of the clock. The time required to turn off and start the clock is 1 millisecond, which does not affect the displayed image effect; S25: Under the control of the AI server, start each distributed master computer, cloud splicing front-end processor, and array display device remotely in sequence. Each distributed cloud splicing front-end processor starts local decoding and sends the video frame action of the input node to the output node for decoding, ensuring that the frame rate and time are consistent when the video frames of each input node reach each output node. Each output node outputs to the corresponding unit display screen for presentation, achieving complete synchronous splicing of the video images of each group of unit display screens of the array display device; S26: Under the control of the AI server, after the input nodes and output nodes of each array display device are started, if a time difference is found in any input node, the AI server performs the action of comparing the timestamps of all input nodes in the network. If the AI server detects an error code, it repeats the above steps S23, S24, and S25 to recalibrate, so as to keep the frame rate and time of the input nodes and output nodes of each array display device consistent.

6. The synchronous control method for the distributed video image stitching large screen according to claim 5, characterized in that, In step S2 described above, the steps for synchronously controlling each input node video and calibrating the timestamp also include: S27: Each master output node of the cloud splicing front-end processor is built-in with a programmable differential high-precision crystal oscillator. This crystal oscillator is based on the basic circuit for clock production, and obtains the signal error code fluctuation range by calibrating the bit error rate of network transmission for each output node. By editing the frequency offset data of the programmable crystal oscillator at the master output node, calling the preset comparison deep learning calculation method in the AI server, calculating the correction frequency data of the master control, and then using this correction frequency data to correct the frequency data of all other input nodes in the same network, the frequency difference of all input nodes is reduced to the set range.

7. The synchronous control method for the distributed video image stitching large screen according to claim 1, characterized in that, In step S3 described above, the steps for synchronously controlling the videos of each output node and calibrating the timestamps are as follows: S31: The AI server controls the decoding and splicing matrix processor to correct the timestamps of multiple output nodes respectively: when the video streams of each input node are sent to each output node for playback, the frame rates and timestamps of the video streams of each output node are compared and calculated; the master output node monitors the timestamp comparison of the nodes playing the video stream in real time. If it is found that node A has a bit error, this master output node immediately sends the correct timestamp to all output nodes such as node B and node C in the same network for calibration, ensuring that the timestamps of each frame of video are the same, and realizing the synchronous output of the videos of each unit display screen of each array display device.

8. The synchronous control method of the distributed video image stitching large screen according to claim 7, characterized in that, Step S3 described above also includes: S32: The steps for the AI server to synchronously control the videos of each output node and calibrate the timestamps are as follows: the AI server controls each output node to first perform video frame buffer processing and then perform frame synchronization calls. Specifically: first, a video frame to be output is buffered in the renderer of each output node, and after the buffering is completed, the timestamp of the buffered frame is synchronized to the AI server in real time; when the AI server decides to render a certain frame, it simultaneously sends a rendering instruction for a certain buffered frame to multiple distributed master computers; if the video to be played is 30 frames per second, the AI server will send 30 rendering buffered frame instructions to the master computers of all output nodes per second; after the video frame buffered in the memory is rendered and before it is presented on the screen, the rendered data is first copied to the video memory of each output node with output. After each output node receives the rendering instruction, it can output without rendering delay.

9. The synchronous control method of the distributed video image stitching large screen according to claim 8, characterized in that, Step S3 described above also includes the following steps: S33: After each output node of the master computer receives the rendering instruction, before rendering, the output node corresponding to the main screen first checks the caches of all output nodes corresponding to the secondary screens. If the current frame to be rendered exists in all the caches of the secondary screens, it immediately synchronizes the rendering to achieve consistent timestamps and frame rates of each output node when playing the video, and make the video data playback of each group of unit display screens synchronous.

10. A synchronous control system for a large-screen distributed video image stitching screen implementing the method according to any one of claims 1-9, characterized in that It includes the following multiple distributed components interconnected through a network: multiple array display devices composed of multiple groups of unit display screens, multiple cloud splicing front-end processors with built-in decoding and splicing matrix processing programs, multiple master computers with built-in distributed splicing control programs, multiple network switching devices, and at least one AI server with a built-in deep learning program; each local master computer is connected to one or more cloud splicing front-end processors, and each cloud splicing front-end processor is connected to one or more local array display devices; each cloud splicing front-end processor is provided with multiple video input nodes and multiple output nodes, with one of the video input nodes as the master control input node and the others as ordinary input nodes, and one of the video output nodes as the master control output node and the others as ordinary output nodes; high-precision crystal oscillators are provided in each of the video input nodes and output nodes.

Images