Equipment and method for multi-element video fusion processing

Abstract

The invention discloses a method for multi-element video fusion processing, and relates to the field of video fusion, and the method comprises the steps: collecting multi-element video data; taking the preprocessed multi-element video data as input, carrying out cascade connection on a plurality of IPG gateways to form a signal scheduling matrix, and carrying out first scheduling and fusion processing on the multi-element video data; a 256 * 256 general control intersection point matrix is set; taking the signal processed by the signal scheduling matrix as an input signal, and performing secondary scheduling and processing on the input signal by controlling the switching and scheduling of the virtual cross point; creating different cross point applications for each output channel according to different application scenes in the general control cross point matrix; output channels of the general control cross point matrix are classified and grouped in a user-defined mode through a switching control module, one output channel is arranged in multiple groups according to different application scenes, and new groups are created according to preset groups; aiming at the poor expandability of video fusion processing equipment in the prior art, the expandability is improved.

Description

Technical Field

[0001] The present application relates to the field of video fusion, and in particular to a device and method for multi-video fusion processing. Background Art

[0002] As the application of video continues to expand, video surveillance, command and dispatch, live broadcast, remote conferencing and other scenarios have higher and higher requirements for video processing equipment. Most traditional video processing equipment uses a single input and output interface, such as SDI or HDMI, which has poor scalability and is difficult to meet the needs of large-scale and diversified video fusion processing.

[0003] Specifically, existing video fusion processing equipment has the following shortcomings: First, the video input and output interface types are single, usually only supporting a single type such as SDI or HDMI, and are not compatible with other heterogeneous video sources, resulting in limited application scope of the equipment. Second, the system scalability is poor. When the scale of video input and output needs to be expanded, it is limited by the hardware architecture and requires major changes to the equipment, resulting in high expansion costs and low flexibility.

[0004] Therefore, there is an urgent need for a new type of video fusion processing equipment that can support the access and fusion of multiple heterogeneous videos and has flexible expansion capabilities and dynamic scheduling mechanisms to adapt to the ever-evolving needs of diversified video applications. Summary of the invention

[0005] In response to the problem of poor scalability of video fusion processing equipment in the prior art, the present application provides a device and method for multi-video fusion processing, which uses cascaded IPG gateways to build a signal scheduling matrix for the first scheduling fusion, and then performs secondary scheduling processing through the master control crosspoint matrix, and performs group switching control on the output channels according to different application scenarios, thereby improving the scalability of the device.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] One aspect of the present application provides a device for multi-video fusion processing, including: an acquisition module, which acquires multi-video data, including an SDI or CVBS acquisition unit, a network communication unit and an interface processing unit, which are respectively used for SDI camera resources, external IP network video media resources and device desktop media resources; a preprocessing module, which preprocesses the acquired multi-video data; a control module, which performs fusion processing on the acquired multi-video data, including: a signal scheduling matrix, which is composed of multiple cascaded IPG gateways, and is used for the first scheduling and fusion processing of the multi-video data; a master control crosspoint matrix, which sets virtual crosspoints according to the output of the signal scheduling matrix, and is used for secondary scheduling and processing of signals processed by the signal scheduling matrix; a switching control module, which performs custom hierarchical grouping and switching control on the output channels of the master control crosspoint matrix; a transmission module, which transmits the data after fusion processing by the control module to a display device through a wireless communication link, a satellite communication link or a wired IP communication link.

[0008] Another aspect of the present application also provides a method for multi-video fusion processing, including: obtaining SDI camera resources through an SDI or CVBS acquisition unit, obtaining external IP network video media resources through a network communication unit, and obtaining device desktop media resources through an interface processing unit to obtain multi-video data; pre-processing the multi-video data; using the pre-processed multi-video data as input, cascading multiple IPG gateways to form a signal scheduling matrix, and performing a first scheduling and fusion processing on the multi-video data; according to the output of the signal scheduling matrix, setting a 256x256 master control cross point matrix, the master control cross point matrix including virtual cross points; The signal processed by the scheduling matrix is ​​taken as the input signal and input into the master control crosspoint matrix. By controlling the switching and scheduling of the virtual crosspoints, the input signal is secondary scheduled and processed. In the master control crosspoint matrix, different crosspoint applications are created for each output channel according to different application scenarios, and the applications include testing, live broadcast and standby broadcast. The output channels of the master control crosspoint matrix are customized and hierarchically grouped through the switching control module. According to different application scenarios, an output channel is set in multiple groups, and new groups are created according to preset groups. The video data after fusion and processing by the control module is transmitted to the display device for display through a wireless communication link, a satellite communication link or a wired IP communication link.

[0009] Furthermore, multiple video data are obtained, including: acquiring SDI signals or CVBS signals output by the SDI camera through an SDI or CVBS acquisition unit to obtain SDI video data or CVBS video data; establishing a communication connection with an external IP network through a network communication unit to receive network video media data transmitted by the external IP network to obtain network video data; establishing a connection with the video acquisition interface inside the device through an interface processing unit to acquire device desktop media data to obtain desktop video data.

[0010] Among them, SDI (Serial Digital Interface), SDI is a widely used digital video interface standard for transmitting uncompressed digital video signals. CVBS (Composite Video Baseband Signal), CVBS is an analog video signal, also known as a composite video signal. IP network (Internet Protocol Network), IP network is a data communication network based on Internet Protocol (IP), which realizes data transmission and exchange through interconnected network devices and communication protocols. Desktop video data refers to the video content on the display screen of a computer or other device. Desktop video data includes displayed graphics, text, images, and videos, etc., reflecting the current display status of the device. Desktop video data of the device can be collected through video capture interfaces such as VGA, DVI, HDMI or DisplayPort. Network video media data refers to video content data transmitted over an IP network. Network video media data usually adopts a specific video encoding format, such as H.264, H.265 or VP9, ​​to achieve efficient data compression and transmission. Network video media data can be transmitted through various network protocols, such as RTSP (Real-time Streaming Protocol), RTMP (Real-time Messaging Protocol), HLS (HTTP Live), etc. Common network video media data sources include network cameras, video servers, streaming media platforms, etc., which can be received and decoded by network communication units.

[0011] Furthermore, the multi-dimensional video data is preprocessed, including encoding the SDI video data, CVBS video data, network video data and desktop video data to obtain encoded multi-dimensional video data, wherein the audio data is encoded using an AAC encoding algorithm and the video data is encoded using an H.264 encoding algorithm; the encoded multi-dimensional video data is encapsulated using a streaming media transmission protocol; the streaming media protocol includes at least one of the RTMP protocol, the HLS protocol and the HTTP-FLV protocol.

[0012] Among them, AAC encoding algorithm (Advanced Audio Coding), AAC is an efficient audio compression encoding algorithm used to compress audio data into a smaller amount of data while maintaining high-quality audio restoration. H.264 encoding algorithm (H.264 / MPEG-4 Advanced Video Coding), is an efficient video compression encoding algorithm, also known as MPEG-4 AVC (Advanced Video Coding). RTMP protocol (Real-Time Messaging Protocol), RTMP is an application layer protocol based on TCP for real-time transmission of audio and video data. HLS protocol (HTTP Live Streaming), HLS is an HTTP-based streaming media transmission protocol launched by Apple. HLS divides video content into a series of short-duration video clips (usually 5-10 seconds) and generates a playlist file (M3U8) to manage these clips. HTTP-FLV protocol (HTTP-Flash Video), HTTP-FLV is an HTTP-based streaming media transmission protocol used to transmit audio and video data in FLV format in real time.

[0013] Preferably, the SDI video data, CVBS video data, network video data and desktop video data are encoded to obtain encoded multi-element video data, wherein the audio data is encoded using an AAC encoding algorithm, and the video data is encoded using an H.264 encoding algorithm; the encoded multi-element video data is encapsulated using a streaming media transmission protocol, and the streaming media transmission protocol is at least one of the RTMP protocol, the HLS protocol and the HTTP-FLV protocol; wherein the encoding process includes: inputting the SDI video data, the CVBS video data, the network video data and the desktop video data into corresponding encoders respectively; encoding the video data using an H.264 encoder to obtain H.264-encoded video data; and encoding the video data using an AAC encoding algorithm. The C encoder encodes the audio data to obtain AAC-encoded audio data; the H.264-encoded video data and the AAC-encoded audio data are input into the multiplexer, the encoded audio and video data are synthesized by the multiplexer, and an encapsulation format that complies with the streaming media transmission protocol is generated; wherein, the encapsulation process of the streaming media transmission protocol includes: when the RTMP protocol is adopted, a data packet in the RTMP encapsulation format is generated, specifically including: splitting the H.264-encoded video data into multiple NALU units, and adding NALU header information to construct an RTMP video data packet; adding ADTS header information to the AAC-encoded audio data to construct an RTMP audio data packet; multiplexing the constructed audio and video data packets according to timestamps to generate RTMP stream data.

[0014] When the HLS protocol is used, a TS encapsulation format data packet and an M3U8 index file are generated, specifically including: splitting the H.264-encoded video data into multiple NALU units, adding NALU header information, and constructing a TS video data packet; constructing a PES audio data packet from the AAC-encoded audio data; multiplexing the constructed audio and video PES packets according to timestamps and dividing them into multiple TS fragments, and generating corresponding M3U8 index files, and establishing a mapping relationship between TS fragments and indexes; when the HTTP-FLV protocol is used, a FLV encapsulation format data packet is generated, specifically including: splitting the H.264-encoded video data into multiple NALU units and constructing an FLV video TAG; constructing an FLV audio TAG from the AAC-encoded audio data; alternately multiplexing the constructed audio and video TAGs according to timestamps to generate FLV stream data. H.264 and AAC are used to efficiently encode video and audio respectively, and combined with mainstream streaming media encapsulation protocols such as RTMP, HLS, and HTTP-FLV, unified encapsulation of multiple heterogeneous video data is achieved. The use of mature and universal coding and encapsulation standards can significantly improve system compatibility and data transmission adaptability. At the same time, differentiated encapsulation processing flows are designed based on the characteristics of different protocols, which improves the flexibility of data encapsulation and transmission while ensuring video quality.

[0015] Further, the pre-processed multi-element video data is used as input, and multiple IPG gateways are cascaded to form a signal scheduling matrix, and the multi-element video data is first scheduled and fused, including: each IPG module has an internal switching matrix, set as A(i), where i represents the i-th IPG module, 1≤i≤N. The dimension of the switching matrix A(i) is X×Y, that is, the number of input ports is X, and the number of output ports is Y. The multi-element video data obtained by the SDI acquisition unit, the network communication unit and the interface processing unit are respectively input into the input interface of the corresponding IPG module. Let the input video data of the i-th IPG module be V(i, j), where 1≤j≤X, which represents the video data input to the j-th input port of the i-th IPG module. For the i-th IPG module, according to the preset scheduling strategy, by configuring the switching matrix A(i), the input X-channel video data V(i, j) is scheduled and fused. The scheduling strategy can be to exchange, synthesize or split the video data from different sources according to a predetermined rule. The output result of the switching matrix A(i) is Y-channel fused video data, denoted as U(i, k), where 1≤k≤Y, representing the video data of the kth output port of the i-th IPG module. The output data U(i, k) of N IPG modules are cascaded to form the output signal of the signal scheduling matrix. Let the cascaded output signal be W(l), where 1≤l≤N×Y, representing the video data of the lth output port of the signal scheduling matrix. The output signal W(l) of the signal scheduling matrix can be expressed as: W(l)=U(i, k), where k=(l-1)modY+1;In the above formula, represents rounding up operation, and mod represents modulo operation.

[0016] Among them, the IPG module (Integrated Processing Gateway) is an integrated video processing device for realizing the input, processing and output of multiple video signals. The IPG module usually has multiple input interfaces, such as SDI or HDMI interfaces, for receiving video signals from different video sources. The IPG module also has multiple output interfaces for outputting processed video signals, which can be SDI, HDMI or network interfaces. The IPG module integrates a video processing unit inside, which can perform various processing on the input video signal, such as encoding, decoding, scaling, mixing, superposition, etc. The IPG module supports flexible configuration and control, and its functions and parameters can be set and adjusted through software or control interface. The IPG module can be used alone or in cascade connection with other IPG modules to build a larger-scale video processing system. The signal scheduling matrix is ​​a video signal processing and distribution system composed of multiple IPG modules in cascade connection. The signal scheduling matrix interconnects multiple IPG modules to form a matrix structure with multiple input interfaces and output interfaces. The input interface of the signal scheduling matrix is ​​used to receive multi-dimensional video data from different video sources, such as video data obtained by the SDI acquisition unit, network communication unit and interface processing unit. By controlling the video processing unit inside the IPG module, the signal scheduling matrix can schedule and fuse the input multi-dimensional video data. Scheduling processing includes routing the input video data to different IPG modules to achieve the distribution and switching of video signals. Fusion processing includes mixing, superimposing, and screen segmentation of multiple video signals to generate fused video signals. The output interface of the signal scheduling matrix is ​​used to output the video signal after scheduling and fusion processing, and can be connected to a display device, a recording device, or other video processing equipment.

[0017] Furthermore, the preprocessed multivariate video data is used as input, multiple IPG gateways are cascaded to form a signal scheduling matrix, and the multivariate video data is first scheduled and fused, which also includes: setting 2 (N + 1) IPG modules, of which 2N IPG modules are used as main channels, and 2 IPG modules are used for backup channels. The IPG module of the main channel is numbered M (i), and the IPG module of the backup channel is numbered B (j), where 1 ≤ i ≤ 2N, 1 ≤ j ≤ 2. Each main channel IPG module M (i) has X input interfaces and Y output interfaces, and the backup channel IPG module B (j) has X input interfaces and Y output interfaces. The preprocessed multivariate video data V (i, k) is input to the input interface of the main channel IPG module M (i), where 1 ≤ k ≤ X, indicating the video data input to the kth input port of the i-th main channel IPG module. The main channel IPG module M(i) performs the first scheduling and fusion processing on the input video data to obtain the Y-channel output signal U(i, m), where 1≤m≤Y, which represents the video data of the m-th output port of the i-th main channel IPG module. In order to realize the main-backup switching, the Y-channel output signal U(i, m) of each main channel IPG module M(i) is simultaneously input to the input interface of the corresponding backup channel IPG module B(j), where That is, the output signals of the 1st to Nth main channel IPG modules are input to the backup channel IPG module B(1), and the output signals of the N+1th to 2Nth main channel IPG modules are input to the backup channel IPG module B(2). The backup channel IPG module B(j) performs backup processing on the input main channel output signals to obtain Y-channel backup output signals R(j, n), where 1≤n≤Y, representing the video data of the nth output port of the jth backup channel IPG module. By controlling the main channel IPG module M(i) and the backup channel IPG module B(j), the output signal U(i, m) of the main channel can be switched to the input interface of the corresponding backup channel IPG module B(j), thereby realizing the switching and backup of the main and backup channels. The output signals of the signal scheduling matrix include the NxY output signals of the main channel IPG module and the 2xY output signals of the backup channel IPG module, totaling NxY+2Y output signals, denoted as W(l), where 1≤l≤NxY+2Y. The corresponding relationship between the output signal U(i, m) of the main channel IPG module and the output signal W(l) of the signal scheduling matrix is: W(l)=U(i, m), where m=(l-1)modY+1, 1≤l≤NxY; the corresponding relationship between the output signal R(j,n) of the backup channel IPG module and the output signal W(l) of the signal scheduling matrix is: W(l)=R(j,n), where n=(l-NxY-1)modY+1, NxY+1≤l≤NxY+2Y.

[0018] Further, according to the output of the signal scheduling matrix, a 256x256 master control crosspoint matrix is ​​set, and the master control crosspoint matrix includes virtual crosspoints; the signal processed by the signal scheduling matrix is ​​used as an input signal and input into the master control crosspoint matrix, and the input signal is secondary scheduled and processed by controlling the switching and scheduling of the virtual crosspoints, including: setting a 256x256 master control crosspoint matrix, denoted as C, and the master control crosspoint matrix includes virtual crosspoints. Virtual crosspoints are used to implement signal scheduling and switching without involving actual physical connections. The NxY+2Y output signals W(l) of the signal scheduling matrix are used as input signals of the master control crosspoint matrix C, denoted as I(l), where 1≤l≤NxY+2Y. S channels are set, and each channel is set with a primary and backup channel. Therefore, 2(N+S) IPG modules need to be set, of which 2N IPG modules are used for the primary channel and 2S IPG modules are used for the backup channel. By controlling the virtual crosspoints in the master control crosspoint matrix C, the input signal I(l) is rescheduled and processed to obtain 2SxY output signals O(k, m) as the main and backup channel signals of S channels, where 1≤k≤2S, 1≤m≤Y. The main and backup channels of each channel have Y signal outputs. The relationship between the input signal I(l) and the output signal O(k, m) of the master control crosspoint matrix C can be expressed as: O(k, m)=C[I(l)], where C[·] represents the mapping relationship of the master control crosspoint matrix, which is determined by the connection status of the virtual crosspoints. The 2SxY output signals O(k, m) are output to the corresponding IPG modules respectively, where: the SxY signals of the main channel are output to 2N main channel IPG modules, denoted as M(i), 1≤i≤2N. The SxY signals of the backup channel are output to 2S backup channel IPG modules, denoted as B(j), 1≤j≤2S. The corresponding relationship between the input signal of the main channel IPG module M(i) and the output signal O(k, m) of the master control crosspoint matrix C is: M(i) = O(k, m), where m=(i-1)modY+1, 1≤i≤2N; the corresponding relationship between the input signal of the backup channel IPG module B(j) and the output signal O(k, m) of the master control crosspoint matrix C is: B(j)=O(k, m), where m=(j-1)modY+1, 1≤j≤2S.

[0019] The output signal of the signal scheduling matrix is ​​re-scheduled and processed using the master control crosspoint matrix to obtain the main and backup channel signals of S channels. Each channel has independent main and backup channels, which can be flexibly switched and scheduled as needed. At the same time, through the setting of virtual crosspoints, any combination and distribution of signals can be achieved, which improves the flexibility and reconfigurability of the system. In addition, the main and backup channel signals are output to the corresponding main channel and backup channel IPG modules respectively, which can achieve channel-level backup and redundancy, further improving the reliability and fault tolerance of the system. When the main channel of a channel fails, it can quickly switch to the corresponding backup channel to ensure the continuity and stability of the video transmission of the channel.

[0020] Among them, the channel in this solution corresponds to an independent video signal output channel, and each channel can be set to one or more output signals according to application requirements. For example, in the broadcasting system of a TV station, each channel can correspond to a TV program, such as a news channel, a sports channel, a film channel, etc.; in the video transmission system of a large event or conference, each channel can correspond to a different venue or a different perspective, such as the main venue channel, the branch venue channel, the panoramic channel, the close-up channel, etc. The setting of the channel is closely related to the application requirements. In different application scenarios, the number and content of the channels may be different. For example, in the broadcasting system of a TV station, the number of channels may be large, and the content of each channel is relatively fixed; while in the video transmission system of a large event or conference, the number of channels may be small, but the content of each channel may change at any time, requiring real-time scheduling and switching. In this solution, in order to improve the reliability and availability of the system, each channel is set up with a main and backup channel in the master control and broadcast. The main channel is used for normal signal transmission and broadcasting, and the backup channel is used to take over the work of the main channel when the main channel fails or needs maintenance, ensuring the continuity and stability of the signal.

[0021] Among them, the master control crosspoint matrix is ​​a large-scale video signal scheduling and processing system, which is used to realize flexible scheduling, switching and backup of multi-channel video signals. The master control crosspoint matrix usually adopts the concept of virtual crosspoints, that is, the video signals are logically connected and switched through software or control systems instead of physical hardware connections. The scale of the master control crosspoint matrix can be set according to demand, such as 256x256 means that the matrix has 256 input ports and 256 output ports. In the master control crosspoint matrix, the video signal can be routed from any input port to any output port to realize flexible signal scheduling and distribution. The master control crosspoint matrix can be connected to the IPG module, the output signal of the IPG module is used as the input of the matrix, and the output signal of the matrix is ​​transmitted to the IPG module for further processing. By controlling the virtual crosspoints in the master control crosspoint matrix, the input video signal can be secondary scheduled and processed to generate multiple output signals. The master control crosspoint matrix supports the setting of the main and backup channels, that is, the signal of the same channel is output to the main channel and the backup channel at the same time to realize the redundant backup of the signal. The main channel and backup channel can be connected to different IPG modules to process and transmit signals respectively, thus improving the reliability and fault tolerance of the system.

[0022] Furthermore, in the master control crosspoint matrix, different crosspoint applications are created for each output channel according to different application scenarios, and the applications include testing, live broadcasting and standby broadcasting. In the master control crosspoint matrix C, different application scenarios are created for each output channel, including testing, live broadcasting and standby broadcasting. The SDI interface, HDMI interface and IP interface of the IPG module are used to switch and schedule SDI signals, HDMI signals and IP signals. According to different application scenarios, virtual crosspoint matrices of different sizes are created by combining the crosspoints in the master control crosspoint matrix C, which are recorded as V_test, V_live and V_backup, corresponding to the test, live broadcasting and standby broadcasting application scenarios. The output signal O(k, m) of the master control crosspoint matrix C is used as the input signal and input into the corresponding virtual crosspoint matrix. According to different application scenarios, the virtual crosspoint matrix is ​​used for secondary processing and control of the signal: Test application scenario: Test and monitor the signal through V_test, record the signal quality parameters such as bit error rate, signal-to-noise ratio, etc., and realize fault detection and alarm. Live broadcast application scenario: Switch and schedule the main and standby signals through V_live, dynamically select the optimal signal source according to the preset strategy or manual intervention, and ensure the smoothness and stability of the live broadcast. Backup broadcast application scenario: Back up SDI, HDMI and IP signals through V_backup, and store the key video sources redundantly so that they can be quickly restored when a failure occurs. Through the control of the virtual crosspoint matrix, the output signal O(k, m) of the master control crosspoint matrix C is processed twice to obtain the output signal corresponding to the application scenario, which is recorded as follows: Output signal of the test application scenario: T(p, q), 1≤p≤P, 1≤q≤Q; Output signal of the live broadcast application scenario: L(r, s), 1≤r≤R, 1≤s≤S; Output signal of the backup broadcast application scenario: B(u, v), 1≤u≤U, 1≤v≤V; Among them, P, Q, R, S, U, V represent the number of output channels and the number of output paths of each channel in different application scenarios respectively. The processed output signal is output through the SDI interface, HDMI interface and IP interface of the IPG module, which are respectively recorded as: Test application scenario: T(p, q) is output to the test equipment and monitoring equipment. Live application scenario: L(r, s) is output to the encoder, transmission equipment and broadcasting equipment. Standby application scenario: B(u, v) is output to the storage device and the standby broadcasting equipment. The relationship between the virtual cross-point matrix and the input and output signals can be expressed as: Test application scenario: T(p, q) = V_test[O(k, m)], where V_test[·] represents the virtual cross-point matrix mapping relationship in the test application scenario. Live application scenario: L(r, s) = V_live[O(k, m)], where V_live[·] represents the virtual cross-point matrix mapping relationship in the live application scenario.Backup broadcast application scenario: B(u, v) = V_backup[O(k, m)], where V_backup[·] represents the virtual crosspoint matrix mapping relationship in the backup broadcast application scenario. The virtual crosspoint matrix is ​​used to realize secondary processing and control of the output signal of the master control crosspoint matrix. According to different application scenarios, virtual matrices for testing, live broadcast and backup broadcast are dynamically created, and signal resources are flexibly scheduled to meet various application requirements.

[0023] Among them, the virtual crosspoint matrix is ​​a logical matrix created by combining the crosspoints in the master control crosspoint matrix according to different application scenarios on the basis of the master control crosspoint matrix. The virtual crosspoint matrix is ​​a flexible video signal scheduling and processing mechanism that can be dynamically created and configured according to actual needs. Each virtual crosspoint matrix can have different sizes and functions to achieve video signal processing and control in specific application scenarios. The virtual crosspoint matrix receives the output signal of the master control crosspoint matrix as input, performs secondary processing and scheduling on the input signal, and generates an output signal corresponding to the application scenario. The virtual crosspoint matrix can be configured and controlled by software or a control system to achieve flexible signal switching, scheduling and processing. The virtual crosspoint matrix can be connected to different interfaces of the IPG module (such as SDI interface, HDMI interface, and IP interface) to achieve switching and scheduling of different types of signals.

[0024] Furthermore, the output channels of the master control crosspoint matrix are customized and hierarchically grouped by switching the control module, an output channel is set in multiple groups according to different application scenarios, and a new group is created according to the preset group, including: grouping and managing the output channels of the master control crosspoint matrix C, and setting the output channels according to different application scenarios. Preset multiple groups, each corresponding to a unique group number. Suppose there are G groups, the group number is G(i), 1≤i≤G. Preset multiple application scenarios, each corresponding to a unique scenario number. Suppose there are S application scenarios, the scenario number is S(j), 1≤j≤S. Establish a mapping table M between application scenarios and groups, and set the mapping relationship between output channels and groups under different application scenarios in the mapping table M. The rows of the mapping table M represent application scenarios, the columns represent groups, and the element M(j, i) represents the mapping relationship of the output channel belonging to the group G(i) under the application scenario S(j), and the value is 0 or 1, where: M(j, i) = 1 means that under the application scenario S(j), the output channel belongs to the group G(i); M(j, i) = 0 means that under the application scenario S(j), the output channel does not belong to the group G(i). Set the output channel configuration table T, set the corresponding relationship between the output channel entry and the group number in the output channel configuration table T, and associate one output channel entry with multiple group numbers. The rows of the output channel configuration table T represent the output channel entries, the columns represent the group numbers, and the element T(k, i) represents the corresponding relationship between the output channel entry k and the group number G(i), and the value is 0 or 1, where: T(k, i) = 1 means that the output channel entry k is associated with the group number G(i); T(k, i) = 0 means that the output channel entry k is not associated with the group number G(i). According to the current application scenario S(j), the switching control module queries the mapping table M of the application scenario and the group, obtains the mapping relationship between the output channel and the group in the scenario, and obtains a group mapping vector M_j, where M_j(i)=M(j, i). The switching control module dynamically generates new groups according to the group mapping vector M_j and the output channel configuration table T, and assigns the output channel entries to the corresponding groups. For each output channel entry k, grouping is performed according to the following rules: if there is a group number G(i) such that M_j(i)=1 and T(k, i)=1, the output channel entry k is assigned to the group G(i); if for all group numbers G(i), M_j(i)=0 or T(k, i)=0, the output channel entry k is assigned to a new group, and a new group number is assigned to the group. According to the generated new group, the switching control module updates the output channel configuration table T and the mapping table M of the application scenario and the group to reflect the latest corresponding relationship between the output channel entries and the groups. The switching control module sends the grouping information to the master control crosspoint matrix C to perform grouping control and switching on the output channels.The master control crosspoint matrix C dynamically adjusts the distribution and connection relationship of the output channels according to the received grouping information to realize signal scheduling and distribution in different application scenarios. Through the above scheme, the custom hierarchical grouping of the output channels of the master control crosspoint matrix is ​​realized, and the mapping relationship between the output channels and the groups is flexibly configured according to different application scenarios. By introducing the mapping table between application scenarios and groups and the output channel configuration table, new groups can be dynamically generated, and an output channel entry can be associated with multiple groups to meet diverse application needs.

[0025] Furthermore, a new group is created according to a preset group, including: setting a group configuration table G for storing group configuration information, the configuration information including a group number and a group name. Each row of the group configuration table G represents a group, and includes the following fields: G(i).id: group number, uniquely identifies a group, with a value range of 1 to G_max, where G_max is the maximum number of groups; G(i).name: group name, used to describe the attributes or purpose of the group, which is customized by the user. According to the user's instruction to create a new group, a new group is created in the group configuration table G: the user issues an instruction to create a new group, and the instruction includes the name of the new group new_name; the switching control module queries the group configuration table G and finds an unused group number new_id, with a value range of 1 to G_max; a new row is added to the group configuration table G, and G(new_id).id=new_id, G(new_id).name=new_name is set, indicating that a new group is created. In the output channel configuration table T, the new group number is associated with the output channel entry, and the output channel is associated with the new group: the switching control module selects the output channel entry that needs to be associated with the new group according to the user instruction or preset rule, which is recorded as T(k); in the output channel configuration table T, the value of T(k, new_id) is set to 1, indicating that the output channel entry k is associated with the new group new_id. According to the group configuration table G and the output channel configuration table T, a group control instruction reflecting the output channel group relationship is generated: the switching control module traverses each row of the group configuration table G, and generates a group control instruction C(i) for each group G(i); the group control instruction C(i) contains the following information: C(i).group_id: group number, that is, G(i).id; C(i).channel_list: a list of output channel entries belonging to the group G(i), which is generated according to the output channel entry k with T(k, i)=1 in the output channel configuration table T. The switching control module sends the generated group control instruction C(i) to the master control crosspoint matrix C. The master control crosspoint matrix C synchronously switches and controls the output channels belonging to the same group according to the received group control instructions C(i): for each group control instruction C(i), the master control crosspoint matrix C extracts the output channel entry list C(i).channel_list belonging to the group; by controlling the crosspoint switch, the audio and video data of the output channels in C(i).channel_list are synchronously switched and controlled to achieve independent control of the group G(i); the output channels of different groups can be switched and controlled independently without affecting each other. Through the above scheme, it is possible to create a new group according to the preset group and flexibly associate the output channel to the new group. Through the coordination of the group configuration table and the output channel configuration table, the switching control module can dynamically generate group control instructions to reflect the group relationship of the output channels.The master control crosspoint matrix performs independent synchronous switching and control of the output channels of different groups according to the group control instructions to meet the diverse group control requirements.

[0026] This group creation and control mechanism is highly flexible and scalable. Users can dynamically create and adjust groups according to actual application scenarios, and flexibly allocate output channels to different groups. Through group-level control instructions, independent control of output channels of different groups can be achieved, improving the availability and maintainability of the system. At the same time, this mechanism also facilitates resource sharing and collaboration between groups, improving the comprehensive utilization efficiency of the system.

[0027] Compared with the prior art, the advantages of this application are:

[0028] By cascading multiple IPG gateways with various interface types, a flexible and expandable signal scheduling matrix is ​​constructed. When the input and output scale needs to be expanded, only the number of IPG gateways needs to be increased without changing the original system architecture, which greatly improves the scalability and flexibility of the system.

[0029] The acquisition module supports the acquisition of video data from multiple sources and formats, such as SDI cameras, IP network videos, and device desktops. The heterogeneous data is converted into a unified format through encoding and packaging in the preprocessing module. The two-level scheduling processing of the signal scheduling matrix and the master control crosspoint matrix is ​​then used to achieve efficient fusion of multiple heterogeneous video data.

[0030] By introducing the master control crosspoint matrix, secondary scheduling of the fused video is realized on the basis of the signal scheduling matrix. Virtual crosspoints can be used to dynamically create a variety of matrix applications for different application scenarios, such as testing, live broadcast, standby broadcast, etc. The switching control module can group the output channels according to application requirements to achieve flexible adjustment of the output strategy.

[0031] The signal scheduling matrix adopts a distributed architecture with multiple IPG gateways cascaded, which can process multiple channels of video in parallel, breaking through the processing capacity limitations of a single device. The master control crosspoint matrix provides a flexible scheduling strategy to give full play to the utilization efficiency of the video link. The switching control module realizes the custom grouping of output channels, which can significantly reduce the number of switching operations compared to traditional single-channel switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0033] Figure 1is an exemplary flow chart of a method for multi-element video fusion processing according to some embodiments of the present application;

[0034] Figure 2 is an exemplary flow chart of multi-video fusion processing according to some embodiments of the present application;

[0035] Figure 3 is an exemplary flow chart of control module data processing according to some embodiments of the present application;

[0036] Figure 4 is a schematic diagram of an IPG module according to some embodiments of the present application;

[0037] Figure 5 It is another schematic diagram of an IPG module according to some embodiments of the present application. DETAILED DESCRIPTION

[0038] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0039] Figure 1 The present invention is an exemplary flow chart of a method for multi-element video fusion processing according to some embodiments of the present application, wherein SDI camera resources are obtained through an SDI or CVBS acquisition unit, external IP network video media resources are obtained through a network communication unit, and device desktop media resources are obtained through an interface processing unit to obtain multi-element video data; the multi-element video data is pre-processed; the pre-processed multi-element video data is used as input, multiple IPG gateways are cascaded to form a signal scheduling matrix, and the multi-element video data is firstly scheduled and fused; according to the output of the signal scheduling matrix, a 256x256 master control cross point matrix is ​​set, and the master control cross point matrix includes virtual cross points; The signal processed by the signal scheduling matrix is ​​used as the input signal and input into the master control crosspoint matrix. By controlling the switching and scheduling of the virtual crosspoints, the input signal is secondary scheduled and processed. In the master control crosspoint matrix, different crosspoint applications are created for each output channel according to different application scenarios, and the applications include testing, live broadcast and standby broadcast. The output channels of the master control crosspoint matrix are customized and hierarchically grouped through the switching control module. According to different application scenarios, an output channel is set in multiple groups, and new groups are created according to preset groups. The video data after fusion and processing by the control module is transmitted to the display device for display through a wireless communication link, a satellite communication link or a wired IP communication link.

[0040] Figure 2This is an exemplary flow chart of multi-video fusion processing shown in some embodiments of the present application. This solution is based on the hybrid master control conversion and scheduling technology of SDI+HDMI+IP. It first connects the SDI signal and HDMI signal through the IP network. While ensuring low latency in signal transmission, it can ensure that each system can obtain high-quality signals, and can support all applications of full-link acquisition, editing and broadcasting. Secondly, the object of the application determines what signal each system uses, and the signal format can be selected according to different application requirements. Finally, the advantage of SDI-HDMI-IP for connecting various systems is also reflected in the fact that the signals as resources can be flexibly scheduled and managed, and the audio and video processing capabilities can be distributed throughout the entire network, improving production capacity and management efficiency.

[0041] Figure 3 This is an exemplary flow chart of control module data processing according to some embodiments of the present application. In this solution, an IPG module with 16 SDI / HDMI inputs and 16 SDI / HDMI outputs is constructed; each IPG module integrates a 16×16 crosspoint matrix, which can independently complete the switching and scheduling of 16 signals; the IPG module supports conversion of SDI / HDMI signals into IP signals and transmission over an IP network.

[0042] Cascade and stack multiple IPG modules according to the upstream and downstream relationship; the output of the upstream IPG module is connected to the input of the downstream IPG module; each upstream IPG module only outputs one signal to the downstream IPG module to ensure the number of input signals of the downstream IPG module. By cascading multiple IPG modules, a large-scale SDI / HDMI matrix is ​​constructed; assuming that each IPG module has X SDI / HDMI input interfaces and Y SDI / HDMI output interfaces; use N upstream IPG modules (N≤X), each module outputs one signal to the downstream IPG module; the number of inputs of the downstream IPG module is N, and the number of outputs is Y; the input scale of the entire matrix is ​​N×X, and the output scale is Y, such as Figure 4 , Figure 5 shown.

[0043] The constructed large-scale matrix is ​​applied to the broadcast channel; each upstream IPG module corresponds to a broadcast channel and provides the SDI / HDMI signal required by the channel; the Y outputs of the downstream IPG module correspond to the input of the broadcast channel to complete the signal distribution and scheduling. If you need to set the main and standby SDI / HDMI channels, you can increase the number of IPG modules; set the number of IPG modules to 2(N+1), where N modules are used for the main channel and 1 module is used for the backup channel; the switching of the main and standby channels can be achieved by controlling the output of the IPG module. The routing control of the large-scale matrix is ​​completed in multiple IPG modules; each IPG module is responsible for the switching and scheduling of its own 16×16 crosspoint matrix; through a unified control system, the work of all IPG modules is coordinated and managed to achieve the routing control of the entire matrix. This solution uses the stacking combination of multiple IPG modules to build a large-scale SDI / HDMI matrix. The matrix can provide N×X SDI / HDMI inputs and Y SDI / HDMI outputs to meet the signal scheduling requirements of the broadcast channel. At the same time, the reliability and fault tolerance of the system are improved by setting the main and standby channels.

[0044] If there are multiple channels (for example, S channels), and there are primary and backup channels in both the master control and broadcast, the number of IPG modules to be configured shall be determined according to the number of channels and the requirements of the primary and backup channels; for S channels and there are primary and backup channels in both the master control and broadcast, the number of IPG modules shall be set to 2 (N+S); where N represents the number of IPG modules on the master control side, and S represents the number of broadcast channels. On the master control side, N IPG modules with X-channel SDI inputs are configured; the X-channel SDI inputs of each IPG module correspond to different signal sources, providing the signals required for master control scheduling; considering the primary and backup channels, the number of IPG modules actually configured on the master control side is 2N. On the broadcast side, an IPG module with Y-channel SDI outputs is configured for each channel; the IPG module of each channel provides the Y-channel signal output required by the channel; considering the primary and backup channels, the number of IPG modules actually configured for each channel is 2, and a total of 2S broadcast side IPG modules need to be configured. The 2N IPG modules on the master control side provide 2 (N×X) SDI input signals for master control scheduling; the SDI input signals on the master control side are routed and switched through the physical crosspoint matrix inside the IPG module; the master and standby IPG modules of each broadcast channel obtain the required Y-channel signals from the IPG module on the master control side to complete the scheduling and transmission of the signals. On the master control and broadcast sides, each channel has an independent master and standby IPG module; by controlling the output of the IPG module, the switching between the master and standby channels can be achieved; when the main channel fails, it can quickly switch to the backup channel to ensure the continuity and reliability of the signal. By increasing the number of IPG modules, the number of channels and signal routing capabilities of the system can be easily expanded; if you need to increase the broadcast channels, you only need to add the corresponding number of IPG modules on the broadcast side; if you need to expand the number of signal paths for master control scheduling, you can increase the number of IPG modules on the master control side. This solution uses IPG modules to build an SDI signal scheduling system that supports multiple channels and has master and standby channels. The IPG module on the master control side provides N×X SDI input signals, and the IPG module on the broadcast side provides Y-channel signal output for each channel. The system's reliability and fault tolerance are guaranteed by setting up the primary and backup channels.

[0045] By utilizing the IPG's multiple interface types and the flexibility of virtual crosspoints, virtual crosspoints of different scales and functions can be created according to application requirements to achieve mixed scheduling and broadcasting of SDI, HDMI and IP signals. IPG interface configuration: IPG has SDI interface, HDMI interface and IP interface, which can access and output different types of signals; by configuring the IPG interface, the input and output of SDI, HDMI and IP signals can be achieved; the IPG IP interface can convert SDI and HDMI signals into IP signals to achieve the fusion of different signal types. Creation of virtual crosspoints: By utilizing the IPG's virtual crosspoint feature, virtual crosspoints of different scales and functions can be created according to application requirements; virtual crosspoints can be consistent with physical crosspoints, or they can be the superposition and fusion of multiple physical crosspoints; through the control software, virtual crosspoints can be flexibly created and managed to achieve signal scheduling and switching. Creation of master control crosspoints: According to the needs of master control, a large-scale virtual crosspoint can be created, such as a 256×256 master control crosspoint; the master control crosspoint can be connected to various signal sources, including SDI, HDMI and IP signals; through the master control crosspoint, signal scheduling and switching can be achieved globally to meet the needs of master control.

[0046] Cascading of signal processing and virtual crosspoints: The processed signal can be used as a new signal source, which can be used as the input signal source of the virtual crosspoint; by cascading multiple virtual crosspoints, complex signal processing and scheduling processes can be realized; after each signal processing, the processed signal can be used as a new signal source and input into the next level virtual crosspoint. Virtual crosspoints of broadcast channels: Create independent virtual crosspoints for each broadcast channel, and set different scales and functions according to the needs of the channel; the virtual crosspoints of the broadcast channel can access the signal sources required by the channel, including SDI, HDMI and IP signals; through the virtual crosspoints of the channel, the signal scheduling and switching within the channel can be realized to meet the needs of broadcasting. Scenario-based virtual crosspoint applications: According to actual usage needs, virtual crosspoint applications in different scenarios can be created; when testing signals, virtual crosspoints of test applications can be created for signal testing and verification; when live broadcasting signals, virtual crosspoints of live broadcast applications can be created for primary and backup signals to realize the switching and scheduling of primary and backup signals; in the SDI, HDMI and IP mixed broadcasting domain, different backup broadcasting scenarios can be created for important signal broadcasting to ensure the reliability of broadcasting.

[0047] Combination of physical crosspoints and virtual crosspoints: Combining physical crosspoints and virtual crosspoints can give full play to the advantages of SDI and IP signals in their respective fields; physical crosspoints are suitable for stable and reliable SDI signal scheduling, and virtual crosspoints are suitable for flexible and changeable IP signal scheduling; through the integration of physical crosspoints and virtual crosspoints, the mixed use of SDI and IP can be realized, providing a more flexible and customizable signal flow. By utilizing the IP input and output and virtual crosspoint features of IPG, combined with control software, the switching and broadcasting of IP domains are realized. By creating virtual crosspoints of different scales and functions, the needs of master control, broadcast channels and different application scenarios can be met. At the same time, through the combination of physical crosspoints and virtual crosspoints, mixed scheduling of SDI and IP signals can be realized, providing a more flexible and reliable signal flow, and better serving safe broadcasting and definable signal flow.

[0048] This solution aims to achieve unified management of SDI, HDMI and IP devices and signals, covering major functions such as device management, switching control, multi-screen management, system monitoring and integrated control, and provides a converged master control management platform. Unified management of all devices in the system, including IP devices, IPG devices, HDMI devices and SDI devices; define and configure the properties and parameters of various types of devices, such as device type, interface type, IP address, etc.; group and classify devices for subsequent control and maintenance; define signal sources, map the input and output ports of the device to logical signal sources, and facilitate signal scheduling and switching.

[0049] Fully customized hierarchical grouping of destination channels. In the management platform, the function of destination channel grouping is provided, allowing users to create and manage different groups; each group can contain multiple destination channels, and a destination channel can also belong to multiple groups; groups can be managed hierarchically, supporting the creation of multi-level grouping structures, such as grading by region, function, etc.; users can customize the name, level and destination channels of the group according to actual needs. Copy group from default group: Provide a default group that contains all destination channels; users can select destination channels from the default group and create new groups; support one-click copying of default groups, quickly create new groups, and reduce repeated operations; after copying the group, users can edit and customize the new group, and add or delete destination channels.

[0050] Flexible switching control: In the management platform, an independent switching control interface is provided for each group; users can switch between different groups and switch and manage the destination channels in the current group; support setting different switching strategies and priorities for each group, and configure according to application requirements; switching control can be achieved through manual operation or automated rules, providing flexible control methods.

[0051] Drag-and-drop switching of signal sources and destination channels: In the graphical interface of the management platform, a visual representation of signal sources and destination channels is provided; signal sources can be displayed in the form of lists or icons, and destination channels can be displayed in the form of groups or grids; users can drag the signal source to the destination channel by dragging the mouse to switch the signal source to the destination channel; one-to-one switching is supported, that is, switching one signal source to one destination channel; one-to-many switching is supported, that is, switching one signal source to multiple destination channels at the same time; through intuitive drag-and-drop operations, the switching process between signal sources and destination channels is simplified, and operational efficiency is improved.

[0052] Multi-screen management: Provides intuitive multi-screen monitoring function to achieve real-time monitoring of all screen signal sources in the system; supports "what you see is what you get" operation mode, and operators can switch and manage screens through a visual interface; multi-screen monitoring can display all screen signal sources in the system and support real-time switching; through multi-screen management, improves the visibility and safety of operations and simplifies the operation process.

[0053] System monitoring: real-time monitoring of the system's operating status, especially the usage status of signals and links; specify the signal source in the signal source list, and analyze the usage of the signal source in the system in real time; list the signal source usage link diagram, showing which channels the signal source is used by, and update it in real time; establish a general protocol monitoring platform that supports SNMP, SSH, WebSocket and other protocols; through the general protocol monitoring platform, implement customized data-level monitoring and obtain device and signal status information.

[0054] Integrated control: Combine device management and channel definition to centralize protocol-related device management on a unified control platform; virtualize signals and requirements into operation objects, so operators only need to focus on how to perform operations on the objects; provide custom mode setting function to set functions such as device parameter modification, signal scheduling, screen layout switching and backup data in the system into the mode; through the one-key execution mode, fast configuration and operation of all functions in the system can be achieved; integrated control simplifies the operation process, improves operation efficiency and reduces the risk of operation errors.

[0055] The SDI+HDMI+IP hybrid broadcast control system described in this solution not only realizes the traditional SDI signal link scheduling function, but also uses the basic functions of the IPG device to complete the IPization of all SDI and HDMI signals in the system. Combined with the main and standby core switches, a completely independent IP architecture application link can be realized. This hybrid architecture broadcast control system can replace the traditional SDI matrix and still retain the SDI signal link function.

[0056] For the purpose of smooth transition, the traditional SDI broadcast control channel can be used under this architecture to keep the SDI and HDMI signal broadcast control links clear and intuitive. In the main and standby link mode, the single device failure point has little impact on the system; the IPG device's own frame synchronization, up and down conversion, multi-screen and embedding (ST2110 audio and video are transmitted separately) and other functions can also be used to save a lot of complicated peripheral equipment. At the same time, since the signal in the system has been IP-based, the IP-based production and broadcasting process can be realized at a relatively low cost with the support of this system.

[0057] IPG has SDI interface, HDMI interface, and IP interface. It can use the IP input and output of IPG and the IPG virtual crosspoint features, combined with control software, to more easily complete the switching and broadcasting of IP domains. Virtual crosspoints have more flexibility. They can be consistent with physical crosspoints, or they can be superimposed and integrated with multiple physical crosspoints. It is also possible to create different virtual crosspoint scales according to application needs. For example, you can create a 256×256 master control crosspoint, or you can use each processed signal as a new signal source as the input signal source of the virtual crosspoint, so as to complete more signal scheduling applications; you can also create crosspoint applications of different scales for each broadcast channel; it may be based on actual use needs, such as creating a crosspoint for test applications during signal testing, creating a crosspoint for live broadcast applications for the main and standby signals during signal live broadcast, and creating different standby broadcast scenarios for important signal broadcasts in the SDI, HDMI and IP mixed broadcast domains. The combination of physical crosspoints and virtual crosspoints can give full play to the advantages of SDI and IP signals in their respective fields, and can also integrate the mixed use of SDI and IP to better serve safe broadcasting and definable signal processes.

Claims

1. A device for multi-video fusion processing, characterized in that: include: The acquisition module acquires multiple video data, including an SDI or CVBS acquisition unit, a network communication unit, and an interface processing unit, which are used for SDI camera resources, external IP network video media resources, and device desktop media resources respectively; A preprocessing module preprocesses the collected multivariate video data; The control module performs fusion processing on the acquired multi-dimensional video data, including: The signal scheduling matrix, which consists of multiple cascade-connected IPG gateways, is used to perform the first scheduling and fusion processing of multi-dimensional video data; The master control crosspoint matrix sets virtual crosspoints according to the output of the signal scheduling matrix, and is used to perform secondary scheduling and processing on the signals processed by the signal scheduling matrix; The switching control module performs customized hierarchical grouping and switching control on the output channels of the master control crosspoint matrix; The transmission module transmits the data merged and processed by the control module to the display device through a wireless communication link, a satellite communication link or a wired IP communication link.

2. A method for multi-video fusion processing, characterized in that: include: Acquire SDI camera resources through SDI or CVBS acquisition unit, acquire external IP network video media resources through network communication unit, acquire device desktop media resources through interface processing unit, and obtain multi-dimensional video data; Preprocessing of multivariate video data; The pre-processed multi-dimensional video data is used as input, and multiple IPG gateways are cascaded to form a signal scheduling matrix to perform the first scheduling and fusion processing on the multi-dimensional video data; According to the output of the signal scheduling matrix, a 256x256 master control crosspoint matrix is ​​set, and the master control crosspoint matrix includes virtual crosspoints; the signal processed by the signal scheduling matrix is ​​used as an input signal and input into the master control crosspoint matrix, and the input signal is secondary scheduled and processed by controlling the switching and scheduling of the virtual crosspoints; In the master control crosspoint matrix, different crosspoint applications are created for each output channel according to different application scenarios, including testing, live broadcast and standby broadcast; The output channels of the master control crosspoint matrix are customized and grouped by switching control modules. According to different application scenarios, one output channel can be set in multiple groups, and new groups can be created based on preset groups. The video data after fusion and processing by the control module is transmitted to the display device for display via a wireless communication link, a satellite communication link or a wired IP communication link.

3. The method for multi-video fusion processing according to claim 2, characterized in that: Get multivariate video data, including: The SDI signal or CVBS signal output by the SDI camera is collected by the SDI or CVBS collection unit to obtain SDI video data or CVBS video data; Establishing a communication connection with an external IP network through a network communication unit, receiving network video media data transmitted by the external IP network, and obtaining network video data; A connection is established between the interface processing unit and the video acquisition interface inside the device to acquire the device desktop media data and obtain desktop video data.

4. The method for multi-video fusion processing according to claim 3, characterized in that: Preprocess the multivariate video data, including: SDI video data, CVBS video data, network video data and desktop video data are encoded to obtain encoded multi-dimensional video data, wherein the audio data is encoded using the AAC encoding algorithm and the video data is encoded using the H.264 encoding algorithm; The encoded multi-video data is encapsulated by using a streaming media transmission protocol; the streaming media protocol includes at least one of the RTMP protocol, the HLS protocol and the HTTP-FLV protocol.

5. The method for multi-video fusion processing according to claim 4, characterized in that: Multiple IPG gateways are cascaded to form a signal scheduling matrix, including: Set N IPG modules with X SDI or HDMI input interfaces and Y output interfaces, where N is less than or equal to X, and Y is less than or equal to X; N IPG modules are cascaded to form a signal scheduling matrix with NxX input interfaces and NxY output interfaces; Input the multi-dimensional video data acquired by the SDI acquisition unit, the network communication unit and the interface processing unit into the input interface of the corresponding IPG module respectively; By controlling the IPG module, the input multi-dimensional video data is first scheduled and fused to obtain NxY fused video signals; The NxY fused video signals are output through the output interface of the cascaded IPG modules as the output signals of the signal scheduling matrix.

6. The method for multi-element video fusion processing according to claim 5, characterized in that: Multiple IPG gateways are cascaded to form a signal scheduling matrix, which also includes: Set the main and backup output channels of SDI or HDMI, and set the number of IPG modules to 2(N+1), where 2N IPG modules are used as the main channels and 2 IPG modules are used as the backup channels; By controlling the IPG module of the main channel and the IPG module of the backup channel, the output signal of the main channel is switched to the input interface of the IPG module of the backup channel to switch and back up the main and backup channels; The NxY output signals of the main channel IPG module and the Y output signals of the backup channel IPG module, a total of NxY+Y signals, are used as output signals of the signal scheduling matrix.

7. The method for multi-video fusion processing according to claim 6, characterized in that: Set up a 256x256 master control crosspoint matrix, including: Set up a 256x256 master control crosspoint matrix, which includes virtual crosspoints; Set S channels, and each channel is set to a primary and backup channel, and set the number of IPG modules to 2 (N+S); among them, 2N IPG modules are used for the primary channel, and 2S IPG modules are used for the backup channel; By controlling the virtual crosspoints in the master control crosspoint matrix, the input NxY+Y signals are secondary scheduled and processed to obtain 2SxY output signals as the main and standby channel signals of S channels; wherein the main and standby channels of each channel have Y signal outputs; The 2SxY output signals are output to the corresponding IPG modules respectively, wherein the SxY signals of the main channel are output to 2N main channel IPG modules, and the SxY signals of the backup channel are output to 2S backup channel IPG modules.

8. The method for multi-video fusion processing according to claim 7, characterized in that: Create different crosspoint applications for each output channel, including: According to different application scenarios, create different application scenarios for each output channel in the master control crosspoint matrix; Use the SDI interface, HDMI interface and IP interface of the IPG module to switch and schedule SDI signals, HDMI signals and IP signals; According to different application scenarios, virtual crosspoint matrices of different sizes are created by combining the crosspoints in the master control crosspoint matrix; The output signal of the master control cross-point matrix is ​​used as an input signal and input into the virtual cross-point matrix; According to different application scenarios, control is performed through a virtual crosspoint matrix; Through the control of the virtual cross-point matrix, the output signal of the master control cross-point matrix is ​​processed twice to obtain the output signal corresponding to the application scenario, and the output signal is output through the SDI interface, HDMI interface and IP interface of the IPG module; Among them: the virtual cross-point matrix of the test application scenario is used for signal testing and monitoring; the virtual cross-point matrix of the live broadcast application scenario is used for switching and scheduling of main and standby signals; the virtual cross-point matrix of the standby broadcast application scenario is used for backup of SDI, HDMI and IP signals.

9. The method for multi-video fusion processing according to claim 8, characterized in that: The output channels of the master control crosspoint matrix can be customized and grouped by switching control modules, including: The output channels of the master control crosspoint matrix are managed in groups, and the output channels are set according to different application scenarios; Multiple groups are preset, each group corresponds to a unique group number; multiple application scenarios are preset, each application scenario corresponds to a unique scenario number; Establish a mapping table between application scenarios and groups, and set the mapping relationship between output channels and groups under different application scenarios in the mapping table; Set the output channel configuration table, the mapping table of application scenarios and groups, set the corresponding relationship between output channel entries and group numbers in the output channel configuration table, and associate one output channel entry with multiple group numbers.

10. The method for multi-video fusion processing according to claim 9, characterized in that: Create new groups based on preset groups, including: Set up a group configuration table to store group configuration information, the configuration information includes group number and group name; According to the user's instruction to create a new group, according to the configuration information of the preset group in the group configuration table, a new group number and group name are created in the group configuration table to create a new group; In the output channel configuration table, associate the new group number with the output channel entry, and associate the output channel with the new group; Generate a group control instruction reflecting the group relationship of the output channels according to the group configuration table and the output channel configuration table; The master control crosspoint matrix controls the crosspoint switches according to the group control instructions to synchronously switch and control the audio and video data of the output channels belonging to the same group in the group control instructions, thereby realizing independent control of the output channels of different groups.

Images