Audio and video signal exchange processing method, interface board and exchange device

Abstract

The application provides an audio and video signal exchange processing method, an interface board and an exchange device, and belongs to the technical field of audio and video exchange. The method comprises the following steps: a plurality of interface boards, each interface board comprising a plurality of input and output channels, and each input and output channel being used for connecting a remote device; a first interface board receives an input data stream from a first remote device through a first input and output channel connected with the first remote device; the first interface board packages the data stream according to the type of the data, obtains a multicast stream, and sends the multicast stream to a second interface board; and the second interface board unpackages the multicast stream according to the type of the multicast stream, obtains an output data stream, and sends the output data stream to a second remote device through a second input and output channel connected with the second remote device. The application can achieve the effects of small bandwidth, support of large-scale input and output of audio signals and video signals, low cost, and low requirements for CPU and GPU.

Description

Technical Field

[0001] This application relates to the field of audio and video switching technology, and more specifically, to an audio and video signal switching processing method, an interface board, and a switching device. Background Technology

[0002] In recent years, with the widespread application of multimedia devices, people have increasingly higher requirements for audio and video switching technology. How to achieve high-resolution switching of audio and video signals has become a major challenge limiting the development of audio and video technology.

[0003] In related technologies, audio and video signal switching is mainly achieved in two ways. The first is to receive video signals from various interfaces through SDI (Serial Digital Interface) or HDMI (High Definition Multimedia Interface), convert the received video signals into a unified format SDI or HDMI signal for transmission, and switch any input differential signal to any output differential signal to achieve audio and video signal switching. The second is to acquire multiple audio and video stream signals through a traditional video conferencing MCU (Multi-Control Unit). The audio and video stream signals are obtained by the terminal device connected to the MCU at the remote end through video encoding of the acquired audio and video signals. The MCU performs video encoding and decoding, audio encoding and decoding, and other processing, and sends the processed audio and video streams to the remote terminal to achieve audio and video signal switching and multi-screen mixing.

[0004] However, when switching audio and video signals based on related technologies, there are problems such as large bandwidth, large input and output scale, high cost, and high requirements for the CPU and GPU of the equipment. Summary of the Invention

[0005] The purpose of this application is to provide an audio and video signal switching and processing method, interface board and switching device, which can achieve the effects of small bandwidth, support for large-scale audio and video signal input and output, low cost and low requirements for CPU and GPU.

[0006] The embodiments of this application are implemented as follows:

[0007] A first aspect of this application provides an audio and video signal switching and processing method, comprising: multiple interface boards, each interface board including multiple input and output channels, each input and output channel being used to connect to a remote device;

[0008] The first interface board receives input data streams from the first remote device through a first input / output channel connected to the first remote device. The input data streams include audio streams or video streams.

[0009] The first interface board encapsulates the data stream according to the data type to obtain a multicast stream, and then sends the multicast stream to the second interface board.

[0010] The second interface board decapsulates the multicast stream according to its type to obtain the output data stream, and sends the output data stream to the second remote device through the second input / output channel connected to the second remote device.

[0011] As one possible implementation, sending the output data stream to the second remote device via a second input / output channel connected to the second remote device includes:

[0012] If the second interface board receives a data stream to be synchronized that is the same as the data source of the output data stream but of a different type, it performs timestamp synchronization processing on the output data stream and the data stream to be synchronized, and sends the processed output data stream and the data stream to be synchronized to the second remote device through the second input / output channel.

[0013] As one possible implementation, timestamp synchronization is performed on the output data stream and the data stream to be synchronized, including:

[0014] The timestamps corresponding to the initial audio stream or video stream are respectively recorded as the system audio timestamp and the system video timestamp;

[0015] Receive audio multicast streams or video multicast streams, and record the timestamp corresponding to the first frame of the audio multicast stream as the original audio timestamp, and the timestamp corresponding to the first frame of the video multicast stream as the original video timestamp.

[0016] For each received audio multicast stream or video multicast stream, a new audio timestamp and video timestamp are determined, wherein the new audio timestamp or new video timestamp of the previous frame is used as the original audio timestamp or original video timestamp of the next frame.

[0017] The timestamp corresponding to the output audio stream is determined based on the system audio timestamp, the original audio timestamp, and the new audio timestamp. The timestamp corresponding to the output video stream is determined based on the system video timestamp, the original video timestamp, and the new video timestamp.

[0018] As one possible implementation, it also includes:

[0019] If the second interface board receives a switching instruction from the server while outputting an input data stream from the first remote device through the second input / output channel, and the switching instruction includes the identifier of the third remote device to which the second input / output channel is to be switched, then the second interface board receives the input data stream from the third remote device through the second input / output channel after receiving the keyframe sent by the third remote device.

[0020] As one possible implementation, after receiving a keyframe from a third remote device, the second interface board outputs an input data stream from the third remote device through a second input / output channel, including:

[0021] The second interface board sends a request message to the third remote device through the second input / output channel. The request message is used to request the third remote device to send a key frame.

[0022] The second interface board receives key frames sent by the third remote device through the second input / output channel, and outputs the input data stream from the third remote device through the second input / output channel.

[0023] As one possible implementation, the switching equipment also includes: a mixing board;

[0024] Sending the multicast stream to the second interface board includes:

[0025] If the type of the multicast stream is audio, the first interface board sends the multicast stream to the mixing board;

[0026] The mixing board mixes the multicast stream to obtain a raw mixed audio stream, and then encapsulates the raw mixed audio stream to obtain a mixed multicast stream.

[0027] The mixing board sends the mixed multicast stream to the second interface board;

[0028] The second interface board decapsulates the multicast stream according to its type to obtain the output data stream, including:

[0029] The second interface board decapsulates the mixed multicast stream according to its type to obtain the output data stream.

[0030] As one possible implementation, a multicast stream includes: channel identifier, data type, whether it is a keyframe, frame start flag, frame end flag, frame sequence number, packet sequence number, timestamp, data length, and media data.

[0031] A second aspect of this application provides an interface board, which includes a memory, a processor, and multiple input / output channels. Each input / output channel is used to connect to a remote device. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the audio and video signal exchange processing method described in the first aspect.

[0032] A third aspect of the embodiments of this application provides a switching device, including: a plurality of interface boards, a switching board, a backplane, and a mixing board.

[0033] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the audio / video signal exchange processing method described in the first aspect.

[0034] The beneficial effects of the embodiments of this application include:

[0035] This application provides an audio / video signal switching and processing method applied to a switching device. The switching device includes multiple interface boards, each interface board including multiple input / output channels. Each input / output channel is connected to a remote device. The first interface board receives an input data stream from the first remote device through a first input / output channel connected to the first remote device. The first interface board encapsulates the data stream according to its data type to obtain a corresponding multicast stream and sends the multicast stream to a second interface board. The second interface board decapsulates the multicast stream according to its type to obtain an output data stream and sends the output data stream to the second remote device through a second input / output channel connected to the second remote device on the second interface board. In this system, the first remote device acts as the input data stream receiver, and the second remote device acts as the output data stream receiver. Multicast stream decapsulation and data stream packetization are performed on the interface board. Data exchange between the remote devices is also achieved through the various interface boards in the switching device. Multiple interface boards in the switching device enable large-scale data exchange between remote devices, and each interface board, through multiple input / output channels, can support data exchange between remote devices with a number matching the number of input / output channels on the interface board. This supports large-scale data stream and multicast stream exchange, reduces production costs, and lowers CPU requirements. Thus, it achieves the advantages of low bandwidth, support for large-scale audio and video signal input / output, low cost, and low CPU and GPU requirements. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0038] Figure 2 This is a schematic diagram of the system architecture of the switching device provided in the embodiments of this application;

[0039] Figure 3A flowchart illustrating an audio / video signal exchange processing method provided in this application embodiment;

[0040] Figure 4 This application provides a schematic diagram of input data stream classification as an embodiment.

[0041] Figure 5 A flowchart illustrating the interface board receiving direction provided in this application embodiment;

[0042] Figure 6 A flowchart illustrating the interface board transmission direction provided in this application embodiment;

[0043] Figure 7 A flowchart illustrating a timestamp synchronization processing method provided in this application embodiment;

[0044] Figure 8 A flowchart illustrating a timestamp synchronization processing system provided in this application embodiment;

[0045] Figure 9 A flowchart of a timestamp synchronization processing logic provided in an embodiment of this application;

[0046] Figure 10 A flowchart illustrating a data stream switching method provided in an embodiment of this application;

[0047] Figure 11 This is a flowchart of a mixing processing method provided in an embodiment of this application.

[0048] Attached image labels:

[0049] 100: Remote device; 200: System core terminal; 201: Switching device; 2011: Interface board; 2012: Mixing board; 2013: Backplane; 2014: Switching board. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0053] In related technologies, audio and video signal exchange is currently often achieved through the following two methods: First, by receiving video signals from various interfaces via SDI (Serial Digital Interface) or HDMI (High Definition Multimedia Interface), converting the received video signals into a unified SDI or HDMI signal format, and transmitting the signal on a PCB (Printed Circuit Board). This allows switching any input differential signal to any output differential signal, achieving a "matrix" effect and enabling audio and video signal exchange. Second, through a traditional video conferencing MCU (Multi-Control Unit), multiple audio and video streams are acquired from the local device. The remote terminal device connected to the MCU encodes the acquired audio and video streams into a network video stream, establishes a connection with the MCU, and sends it to the MCU. The MCU processes the received network video stream and sends the processed video stream to the remote device, achieving audio and video signal switching and multi-view mixing. However, the first method of audio and video signal exchange, implemented through a matrix, results in high video bandwidth, complex PCB layout, massive input / output scale, and high cost. The second method, using an MCU device for large-scale video and audio encoding / decoding, places high demands on the device's GPU and CPU, leading to high device cost and an inability to support large-scale audio and video signal exchange. Furthermore, the high bandwidth makes PCB layout difficult, and the fixed performance of the GPU and CPU in the device prevents support for large-scale audio and video exchange.

[0054] Therefore, this application provides an audio and video signal switching and processing method applied to a switching device. The method involves receiving an input data stream from a remote device via an interface board connected to its input / output channel. The interface board encapsulates the data stream according to its data type to obtain a corresponding multicast stream, and sends the multicast stream to the interface board that needs the input data. The interface board then decapsulates the multicast stream according to its type to obtain an output data stream, and sends the output data stream to the remote device that needs the multicast stream via the input / output channel connected to the remote device on the interface board. In this process, the decapsulation of multicast streams and the encapsulation of data streams are both performed on the interface board. Data exchange between various remote devices is also achieved through the various interface boards in the switching device. Multiple interface boards in the switching device can realize large-scale data exchange between remote devices, and each interface board can support data exchange between remote devices with the same number of input and output channels as the interface board. This can support the exchange of large-scale data streams and multicast streams, reduce production costs, and lower CPU requirements. It can achieve the effects of low bandwidth, support for large-scale audio and video signal input and output, low cost, and low requirements for CPU and GPU.

[0055] Figure 1 A schematic diagram of an application scenario provided in this application is shown below. Figure 1 It can be seen that this application scenario involves remote device 100 and system core terminal 200. There is a communication connection between remote device 100 and system core terminal 200. Remote device 100 and system core terminal 200 can exchange information to complete the exchange of audio and video signals.

[0056] Optionally, audio and video signal exchange often occurs in one-to-one, one-to-many, and many-to-many scenarios. The system core terminal 200 receives audio and video signals from multiple remote devices 100 via the Internet. The system core terminal 200 processes the received audio and video signals and sends the processed audio and video signals to the remote devices 100 that need the audio and video signals.

[0057] Optionally, the remote device 100 supports a maximum input resolution of 8K 30Hz, with an encoded stream bandwidth of 10-20Mbps. The encoded stream signal is then transmitted to the system core 200 via various streaming media transmission protocols. These protocols include UDP (User Datagram Protocol), SIP (Session Initialization Protocol), RTSP (Real Time Streaming Protocol), and SRT (Secure Reliable Transport), among others. This application does not specify any particular protocol.

[0058] Optionally, the system core terminal 200 processes the signals required by each remote device 100 and sends them to the remote devices 100 that need them, based on the application requirements of each remote device 100. Each remote device 100 receives the required signals and plays them.

[0059] Optionally, the system core terminal 200 can simultaneously connect to multiple remote devices 100, and can also simultaneously output multiple audio and video signals according to the needs of each remote device 100, so as to realize the exchange of audio and video signals between multiple remote devices 100.

[0060] In this embodiment, the audio and video signals from remote devices are processed by the system core and exchanged with the remote devices that require the signals. The remote devices decode the received audio and video signals to obtain the final displayed signals. This reduces the demands on the CPU and GPU of the devices and supports large-scale audio and video signal exchange. Thus, it achieves the effect of supporting large-scale input and output of audio and video signals while minimizing CPU and GPU requirements.

[0061] Figure 2 A system architecture diagram of a switching device provided in this application is shown in the attached diagram. Figure 2 It can be seen that the switching device 201 is composed of multiple interface boards 2011, mixing boards 2012, backplanes 2013, and switching boards 2014. Among them, the multiple interface boards 2011, mixing boards 2012, and switching boards 2014 are directly connected to each other via gigabit Ethernet.

[0062] Optionally, the system can adjust the number of interface boards 2011 according to the scale of input and output. Each interface board 2011 has multiple input and output interfaces, each of which can be individually connected to a remote device 100. Each input and output interface can serve as a receiving channel for receiving input data streams from the remote device 100, and also as an output channel for receiving data streams from the remote device. Each interface board 2011 supports the access and processing of up to 8 audio and video signals, and a single switching device 201 can support the access of up to 320 audio and video signals. This application does not impose a specific limit on the number of interface boards.

[0063] Optionally, the remote device 100 can be various video terminals, multimedia terminals, surveillance cameras, and other video devices. Each interface board 2011 connects to each remote device 100 via multiple streaming media transmission protocols. Data stream transmission between the interface board 2011 and the remote device 100 is completed through streaming media transmission protocols, realizing the exchange of audio and video signals between multiple remote devices. It is worth noting that each interface board 2011 supports the access and processing of a maximum of 8 audio and video signals, that is, each interface board 2011 supports the access of a maximum of 8 remote devices 100.

[0064] Optionally, after each interface board 2011 successfully connects its input / output channel to the remote device 100, it begins to receive audio and video signals sent by the remote device 100, processes the received audio and video signals, and sends the processed audio and video signals to the corresponding remote device that needs the signal.

[0065] Optionally, when the remote device 100 has a mixing requirement, the audio signal to be mixed is added to the mixing board 2012, and the added audio signal is mixed according to the volume detection algorithm and audio mixing logic. The processed mixed audio signal is then sent to the remote device 100 that needs it.

[0066] Optionally, the Switch 2014 can switch various IP signals and also supports IGMP (Internet Group Management Protocol) Snoopy function. It can control whether the IGMP data packets input and output on each interface board are forwarded to the various network channels on the Switch 2014.

[0067] In this embodiment, audio and video signals are exchanged with remote devices via input / output channels on the interface board. The number of interface boards is determined based on the input / output scale of the remote device. Increasing the number of interface boards enables large-scale audio and video signal exchange. Audio and video signal processing is performed on the corresponding interface boards, which reduces the demands on the device's CPU and GPU while supporting large-scale audio and video signal exchange. Thus, it achieves the advantages of supporting large-scale audio and video signal input / output, low cost, and low CPU and GPU requirements.

[0068] The audio and video signal exchange processing method provided in the embodiments of this application will be explained in detail below.

[0069] Figure 3 The flowchart of an audio and video signal switching processing method provided in this application is shown. This method can be applied to a switching device 201, which can be the aforementioned MCU device.

[0070] The switching equipment includes multiple interface boards, each of which includes multiple input / output channels, and each input / output channel is used to connect to a remote device.

[0071] Optionally, the number of interface boards in the switching equipment is set according to the audio and video switching requirements of the application scenario. Each interface board contains multiple input and output channels, and each interface board supports a maximum of 8 audio and video signal inputs. That is, each interface board has only 8 input and output channels, and each interface board can only connect to a maximum of 8 remote devices. Each input and output channel can only correspond to one remote device, which can ensure the independence of input and output.

[0072] It is worth noting that not all eight input / output channels of each interface board may be used; seven or six may be used, depending on the number of remote devices connected to the interface board. This application does not make any specific restrictions on this.

[0073] For example, if switching device A needs to connect to 35 remote devices, then switching device A shall be equipped with at least 5 interface boards for connecting these remote devices. It is possible that each input / output channel of four interface boards is occupied, with the last one occupying three input / output channels, or that each of the five interface boards occupies an average of seven input / output channels. This application does not make any specific limitation in this regard.

[0074] See Figure 3 This application provides an audio / video signal switching processing method, including:

[0075] S301, the first interface board receives input data streams from the first remote device through the first input / output channel connected to the first remote device. The input data streams include audio streams or video streams.

[0076] Optionally, the first interface board can be any switching board in the switching equipment. The first switching board can be connected to the first remote device through the input / output channel based on the streaming media transmission protocol. The first remote device refers to any remote device connected to the input / output channel on the first interface board. The input / output channel connecting the first interface board and the first remote device is called the first input / output channel. The first input / output channel can be any input / output channel on the first interface board, but the first input / output channel must be the channel connecting the first remote device and the first interface board.

[0077] Optionally, after the first interface board successfully connects to the first remote device, the first interface board begins receiving the input data stream from the first remote device based on the streaming media transmission protocol. The input data stream may include an audio stream or a data stream. It is worth noting that the input data stream transmitted by each remote device may include both audio and video streams simultaneously, or it may only contain audio or only video streams.

[0078] It is worth noting that each interface board initially only receives the input data stream transmitted by the remote device connected to that interface board, but the interface board can obtain the input data stream of other remote devices on other interface boards through the switch board and backplane.

[0079] Optionally, after receiving the input data stream from the first remote device, the first interface board divides the input data stream into audio stream and video stream according to the type of the received input data stream. The input data stream also contains the identifiers of the remote devices connected to each input / output channel, such as remote device 1, remote device 2, remote device 3, remote device 4, remote device 5, remote device 6, remote device 7, and remote device 8.

[0080] It is worth noting that the input audio stream obtained by the interface board from the input / output channels based on the streaming media transmission protocol is a raw PCM (Pulse Code Modulation) signal. Therefore, the audio stream needs to be decoded first when processing it. Furthermore, the video stream does not require decoding. Since a video stream consists of consecutive frames, decoding would be too burdensome, and the video stream can be recognized even without decoding; therefore, decoding the video stream is unnecessary.

[0081] Figure 4 This is a schematic diagram of input data stream classification provided in an embodiment of this application. See also... Figure 4The processor on the interface board can classify the input data stream into the following categories based on the information and data type contained in the input data stream: 1 video stream or 1 audio stream, 2 video streams or 2 audio streams, 3 video streams or 3 audio streams, 4 video streams or 4 audio streams, 5 video streams or 5 audio streams, 6 video streams or 6 audio streams, 7 video streams or 7 audio streams, and 8 video streams or 8 audio streams.

[0082] For example, switching device A is equipped with 5 interface boards a, b, c, d, and e for connecting 33 remote devices. Each interface board occupies seven input / output channels. Interface board a is connected to seven remote devices, namely a1, a2, a3, a4, a5, a6, and a7. The interface board receives one video stream and one audio stream from a1 as input data streams. The input / output channel between interface board a and remote device a1 is channel number 1.

[0083] Figure 5 For a flowchart illustrating the interface board receiving direction provided in this application embodiment, please refer to... Figure 5 The interface board communicates with the remote device via the input / output channels on the interface board based on the streaming media transmission protocol. It obtains the data stream from the remote device and determines whether the data stream is audio or video based on its data type. The processor on the interface board uses the IGMP signals emitted by each remote device to determine which data streams are needed by the remote device and which are not. Audio streams require priority decoding to determine which devices need them, while video streams can be identified without decoding. The necessary data streams are then packetized to obtain the corresponding multicast streams, which are then sent to the interface board where the remote device resides.

[0084] S302. The first interface board encapsulates the data stream according to the data type to obtain a multicast stream, and then sends the multicast stream to the second interface board.

[0085] Optionally, the first interface board encapsulates the input data stream according to the type of data in the input data stream. Audio streams are encapsulated to obtain audio multicast streams, and video streams are encapsulated to obtain video multicast streams. The encapsulation method for audio streams differs from that for video streams; audio and video streams need to be encapsulated separately. The encapsulation method for audio streams can be ACC (Advanced Audio Coding), MP3 (Moving Picture Experts Group Audio Layer 3), etc.; the encapsulation method for video streams can be H264 (Advanced Video Coding, AVC), Xvid (Advanced Video Interchange, AVI), etc. This application does not specifically limit the methods used.

[0086] Optionally, the processor on the first interface board encapsulates the data stream according to the type of data in the received input data stream, obtaining video multicast streams and audio multicast streams respectively, and then sends the multicast streams to the second interface board. The second interface board can be any interface board in the switching equipment, and it must have a remote device that needs the data contained in the multicast stream. The second interface board can be the same as the first interface board, or it can be a different interface board.

[0087] It is worth noting that when the first interface board and the second interface board are not the same interface board, the multicast stream cannot be directly transmitted from the first interface board to the second interface board. The multicast stream needs to be sent to the backplane via the first interface board, and the backplane sends the multicast stream to the switching board. The switching board determines the multicast stream to be sent to the second interface board based on the IGMP signal, thus realizing the transmission of the multicast stream from the first interface board to the second interface board.

[0088] For example, if interface board a in switching device A receives one video stream and one audio stream from remote device a1, and remote device c3 connected to input / output channel 3 of interface board c requires one video stream and one audio stream, interface board a performs packet processing on the one video stream and one audio stream to obtain one video multicast stream and one audio multicast stream. Interface board a sends the one video multicast stream and one audio multicast stream to the backplane, the backplane sends the multicast stream to the switching board, the switching board receives the IGMP signal sent by interface board c, determines that the multicast stream should be sent to interface board c, and the switching board sends the multicast stream to the network interface corresponding to interface board c. Interface board c receives the one video multicast stream and one audio multicast stream.

[0089] It is worth noting that packetizing the input data stream into a multicast stream for transmission between the interface board, background board, and switching board can effectively save space and prevent data loss. Data transmission between the interface board, background board, and switching board is carried out through packetized data.

[0090] S303 The second interface board decapsulates the multicast stream according to its type to obtain the output data stream, and sends the output data stream to the second remote device through the second input / output channel connected to the second remote device.

[0091] Optionally, the second interface board receives multicast streams and decapsulates them according to their data type to obtain the output data stream. Multicast streams are divided into video multicast streams and audio multicast streams. The different packetization methods of video and audio multicast streams lead to different decapsulation methods. Decapsulation of multicast streams involves first decapsulating the transmission protocol, and then decapsulating the multicast stream's encapsulation format to obtain the corresponding output data stream. It is worth noting that the decapsulated output audio stream is also a raw PCM signal and needs to be encoded before it can be output to the remote device.

[0092] Optionally, the second interface board will send the desealed output data stream to the second remote device through the second input / output channel. The second remote device is the remote device that needs the input data stream. The second input / output channel is the channel connecting the second interface board and the second remote device. The second remote device can be located on any input / output channel of the second interface board, but the second remote device must be connected to the second interface board.

[0093] For example, if interface board a in switching device A receives one video stream and one audio stream from remote device a1, and remote device c3 connected to input / output channel 3 of interface board c requires one video stream and one audio stream, interface board c receives one video multicast stream and one audio multicast stream, interface board c decapsulates the one video multicast stream and one audio multicast stream to obtain one video stream and one audio stream, and interface board c sends the one video stream and the encoded one audio stream to remote device c3 through input / output channel 3.

[0094] It is worth noting that there can be one or more remote devices that require a certain input data stream. These remote devices can be distributed on the same interface board or on different interface boards, but the principle of audio and video signal exchange is the same. This application takes the example of a remote device that requires a certain input data stream but is not on the same interface board, but it does not mean that this application only supports this one scenario.

[0095] Figure 6 For a flowchart illustrating the interface board transmission direction provided in this application embodiment, please refer to... Figure 6The interface board of the remote device that needs the input data stream obtains the multicast stream corresponding to the required data stream through the switching board and backplane. According to the data type of the multicast stream, the multicast stream is decapsulated to obtain the corresponding output data stream. The audio data stream in the output data stream needs to be encoded before it can be sent to the remote device. The interface board sends the output data stream to the remote device that needs the data stream through the input / output channel connected to the remote device. The remote device receives the required output data stream.

[0096] In this embodiment, input data streams from multiple remote devices are acquired through the input / output channels of multiple switching boards in the switching device. The interface board encapsulates the input data streams according to their data types to obtain different types of multicast streams. The remote device that needs the input data stream sends an IGMP signal. Based on this signal, the switching board realizes multicast stream exchange between interface boards and sends the multicast stream to the interface board where the remote device that needs the input data stream is located. The interface board decapsulates the multicast stream and sends the decapsulated output data stream to the remote device that needs it. By increasing the number of interface boards, the scale of input and output can be increased. In this way, the effects of small bandwidth, support for large-scale audio and video signal input and output, low cost, and low requirements for CPU and GPU can be achieved.

[0097] As an optional implementation, step S303 can be performed as follows:

[0098] If the second interface board receives a data stream to be synchronized that is the same as the data source of the output data stream but of a different type, it performs timestamp synchronization processing on the output data stream and the data stream to be synchronized, and sends the processed output data stream and the data stream to be synchronized to the second remote device through the second input / output channel.

[0099] Optionally, the second interface board decapsulates the received multicast stream to obtain the output data stream. When the output data stream contains two different types of data—that is, audio and video streams—and both types of data streams originate from the same remote device, synchronization processing is required for the audio and video streams. It's worth noting that when the output data stream contains only audio or only data streams, synchronization processing is not required, and the decapsulated audio stream needs to be encoded before synchronization processing.

[0100] Optionally, the synchronization processing of the output data stream involves synchronizing the timestamps of the audio stream and video stream contained in the output data stream. This synchronizes the timestamps of the audio stream and the video stream, reducing the time difference between them, improving video playback quality, achieving audio-visual synchronization, and enhancing the user experience.

[0101] Optionally, the second interface board sends the output data stream after timestamp synchronization processing to the remote device that needs the data stream through the input / output channel connected to the required remote device. The remote device receives the output data stream and displays the data stream on the remote device.

[0102] For example, if interface board c in switching device A receives one video multicast stream and one audio multicast stream, interface board c decapsulates the one video multicast stream and one audio multicast stream to obtain one video stream and one audio stream. Both the one video stream and one audio stream come from the input data stream sent by the remote device a1 connected to input / output channel 1 on interface board a. If there is a certain time interval between the one video stream and one audio stream, the processor of interface board c performs timestamp synchronization processing on the one video stream and one audio stream to synchronize the one video stream and one audio stream. Interface board c then sends the synchronized one video stream and the encoded one audio stream to the remote device c3 through input / output channel 3.

[0103] In this embodiment, the interface board of the remote device that needs to input the data stream in the switching device receives the multicast stream corresponding to the input data stream. The interface board of the remote device that needs to input the data stream decapsulates the multicast stream to obtain the corresponding output data stream. The interface board performs timestamp synchronization processing on the output data stream. The interface board sends the synchronized output data stream to the remote device that needs the data stream through the input / output channel. The remote device receives the required data stream and displays its data information, which can achieve the effect of small bandwidth and support for large-scale audio and video signal input and output.

[0104] Figure 7 For a flowchart of the timestamp synchronization processing method provided in this application, please refer to [link / reference]. Figure 7 The audio / video exchange processing method of this application performs timestamp synchronization processing on the output data stream and the data stream to be synchronized, including:

[0105] S701. Record the timestamps corresponding to the initial audio stream or video stream as the system audio timestamp and the system video timestamp, respectively.

[0106] S702: Receive audio multicast stream or video multicast stream, and record the timestamp corresponding to the first frame of the audio multicast stream as the original audio timestamp, and the timestamp corresponding to the first frame of the video multicast stream as the original video timestamp.

[0107] S703. For each received audio multicast stream or video multicast stream, a new audio timestamp and video timestamp are determined, wherein the new audio timestamp or new video timestamp of the previous frame is used as the original audio timestamp or original video timestamp of the next frame.

[0108] S704. Determine the timestamp corresponding to the output audio stream based on the system audio timestamp, the original audio timestamp, and the new audio timestamp; determine the timestamp corresponding to the output video stream based on the system video timestamp, the original video timestamp, and the new video timestamp.

[0109] Optionally, the timestamp corresponding to the video stream in the output data stream can be obtained using the following formula (1). Specifically, for each frame of the video stream, the output timestamp corresponding to that type of video multicast stream can be obtained by calling the following formula (1).

[0110] Tbv=tbv+(new_pts-last_pts)(1)

[0111] Where Tbv represents the output video timestamp, tbv represents the system video timestamp, new_pts represents the new video timestamp corresponding to each frame of the video multicast stream, and last_pts represents the original video timestamp and also the timestamp corresponding to the previous frame of the video multicast stream.

[0112] Optionally, tbv represents the initial timestamp of the start of the video multicast stream transmission. It is denoted as 0 and is also the system video timestamp. When receiving the first frame of the video multicast stream, tbv is directly assigned to Tbv for output, and the timestamp corresponding to the first frame of the video multicast stream is recorded as the original video timestamp. The original video timestamp is dynamic and will be replaced by the new timestamp of the next frame, becoming the original video timestamp of the frame after the next. That is, the original video timestamp corresponding to each frame of the video multicast stream is the new video timestamp corresponding to the previous frame of the video multicast stream.

[0113] Optionally, the timestamp is already present in the multicast stream and can be directly obtained from the multicast stream. The calculated timestamp of the output multicast stream can be delayed or fast-forwarded to synchronize the video with the audio.

[0114] For example, it is known that the initial time of the 1 video multicast stream on interface board a in switching device A corresponds to the system video timestamp tbv = 0, the original timestamp corresponding to the first frame of the video multicast stream is 1, and the timestamp corresponding to the output of the first frame of the video multicast stream is Tbv = 0. When the second frame of the video multicast stream is received, the timestamp corresponding to the first frame of the video multicast stream is recorded as the original video timestamp last_pts = 1, and the timestamp corresponding to the second frame of the video multicast stream is 3, then new_pts = 3. Therefore, the timestamp corresponding to the output of the second frame of the video multicast stream is Tbv = tbv + (new_pts - last_pts) = 0 + (3 - 1) = 2.

[0115] Optionally, the timestamp corresponding to the audio stream in the output data stream can be obtained using the following formula (2). Specifically, for each frame of the audio stream, the output timestamp corresponding to that type of audio multicast stream can be obtained by calling the following formula (2).

[0116] Tba=tba+(new_pts-last_pts)(2)

[0117] Where Tba represents the output audio timestamp, tba represents the system audio timestamp, new_pts represents the new audio timestamp corresponding to each frame of the audio multicast stream, and last_pts represents the original audio timestamp and also the timestamp corresponding to the previous frame of the audio multicast stream.

[0118] Optionally, tba represents the timestamp of the initial moment when the audio multicast stream begins transmission. It is recorded as 0 and is also the system video timestamp. When receiving the first frame of the audio multicast stream, tba is directly assigned to Tba for output, and the timestamp corresponding to the first frame of the audio multicast stream is recorded as the original audio timestamp. The original audio timestamp is dynamic. The original audio timestamp will be replaced by the new audio timestamp of the next frame and become the original audio timestamp of the frame after the next. That is, the original audio timestamp corresponding to each frame of the audio multicast stream is the new audio timestamp corresponding to the previous frame of the audio multicast stream.

[0119] For example, it is known that the initial time of the 1 audio multicast stream on interface board a in switching device A corresponds to the system video timestamp tba = 0, the original timestamp corresponding to the first frame of the audio multicast stream is 2, and the timestamp corresponding to the output of the first frame of the audio multicast stream is Tba = 0. When the second frame of the audio multicast stream is received, the timestamp corresponding to the first frame of the audio multicast stream is recorded as the original video timestamp last_pts = 2, and the timestamp corresponding to the second frame of the audio multicast stream is 3, then new_pts = 3. Therefore, the timestamp corresponding to the output of the second frame of the audio multicast stream is Tba = tba + (new_pts - last_pts) = 0 + (3 - 2) = 1.

[0120] Optionally, the output timestamps of the audio multicast stream and the video multicast stream can be compared. If the output timestamp of the video multicast stream is greater than that of the audio multicast stream (meaning the video multicast stream is faster than the audio multicast stream), the video should be read at a slower speed while the audio multicast stream is read normally. If the output timestamp of the video multicast stream is less than that of the audio multicast stream (meaning the audio multicast stream is faster than the video multicast stream), the video should be read faster while the audio is played normally.

[0121] Figure 8 A flowchart of a timestamp synchronization processing system provided in this application embodiment is shown below. Figure 8Initially, both audio and video timestamps are defined as 0 and recorded as system timestamps. `tbv` represents the video system timestamp, and `tba` represents the audio system timestamp. When the interface board starts receiving multicast streams, upon receiving the first frame of the audio or video multicast stream, `tbv` and `tba` are not modified. `tbv` is directly recorded as `Tbv` and assigned to the output module for output, and `tba` is recorded as `Tba` and assigned to the output module for output. The original timestamp in the first frame of the multicast stream is recorded as `last_pts`, where the original timestamp in the first frame of the audio multicast stream is recorded as `last_apts`, and the original timestamp in the first frame of the video multicast stream is recorded as `last_vpts`. Subsequent received frames of audio or video multicast streams... For each video multicast stream, the original timestamp new_pts of the audio and video streams in that frame is obtained. The original timestamp of each frame of the audio multicast stream is denoted as new_apts, and the original timestamp of each frame of the video multicast stream is denoted as new_vpts. The video timestamp Tbv or audio timestamp Tba of the output module is calculated, where Tbv = tbv + (new_pts - last_pts) and Tba = tba + (new_pts - last_pts). The calculated new Tbv or Tba is assigned to the output module, and new_pts is assigned to last_pts. The original timestamp of the previous frame is saved until the last frame of the multicast stream.

[0122] Optionally, the original remote device corresponding to the audio and video streams can be obtained through the packet structure of the multicast stream. When the audio and video streams are not sent from the same remote device, the audio and video streams are not strongly correlated and can tolerate the asynchrony between the audio and video streams. The audio and video only need to be read alternately according to their own system timestamps.

[0123] Optionally, when the audio stream and video stream are sent from the same remote device, the audio stream and video stream are highly correlated. It is necessary to require the input and output channels to perform timestamp synchronization processing on the audio stream and video stream, otherwise it will lead to poor audio-visual effects.

[0124] Figure 9 A flowchart of a timestamp synchronization processing logic provided in an embodiment of this application is shown below. Figure 9 It can be seen that the switching equipment prioritizes audio processing over video processing. It does not make many adjustments to the audio, but simply reads the original timestamp of the audio. The video data is adjusted to ensure that the audio is always continuous when the remote device outputs audio and video, and the slight delay or jitter caused by fast forward of the video data is ignored.

[0125] As an optional implementation, the above-described audio and video signal exchange processing method further includes:

[0126] If the second interface board receives a switching instruction from the server while outputting the output data stream from the first remote device through the second input / output channel, and the switching instruction includes the identifier of the third remote device to which the second input / output channel is to be switched, then after receiving the keyframe sent by the third remote device, the second interface board outputs the output data stream from the third remote device through the second input / output channel.

[0127] Optionally, in addition to receiving the data stream from the same remote device, the audio and video signal exchange may also switch to the data stream from other remote devices, which may be done once, twice, or even several times.

[0128] Optionally, when the second interface board outputs the input data stream of the first remote device through the second input / output channel, if the data stream output by the second input / output channel is to be switched to the input data stream of the third remote device, a switching instruction sent by the server is required to instruct the second input / output channel of the second interface board to perform the switching of the output data stream information. The switching instruction includes: the identifier of the third remote device to which the second input / output channel is to be switched. Here, the second interface board can be any interface board in the switching equipment, the second input / output channel is a channel on the second interface board that connects to any remote device, and the third remote device can be a remote device connected to other channels on the second interface board or a device connected to channels on other interface boards.

[0129] It is worth noting that the switching command originates from the output data switching requirements of the second input / output channel. Based on the output data switching requirements of the second input / output channel, the server issues a switching command to control the interface board to perform the switching operation.

[0130] Optionally, after receiving a switching command, the second input / output channel of the second interface board can identify the remote device corresponding to the output data stream to be switched from the switching command. The second interface board will only start receiving data stream information from the third remote device after receiving a key frame sent by the remote device. The third remote device can be any remote device on the second interface board other than the remote device corresponding to the second input / output interface. The third remote device can also be a remote device connected to another interface board.

[0131] It is worth noting that keyframes are used to indicate the start of the switch. Only after a keyframe in the switch data stream is received can the data stream information after the switch be sent. A keyframe is equivalent to a switch start button.

[0132] For example, if the remote device c2 connected to the second input / output channel of interface board c in switching device A needs to switch data streams, the remote device c2 first receives one audio multicast stream for 20 seconds and then receives three audio multicast streams until completion. Assume that the one audio multicast stream comes from the remote device a1 corresponding to the first input / output channel of interface board a, and the three audio multicast streams come from the remote device b3 corresponding to the third input / output channel of interface board b. According to the needs of remote device c2, server M sends a switching instruction m to interface board c, wherein the switching instruction includes the identifier of remote device b3. Interface board c first sends the one audio multicast stream from remote device a1 of interface board a to remote device c2. When interface board c receives the key frame Q sent by remote device b3, interface board c starts sending the three multicast streams from remote device b3 on interface board b to remote device c2.

[0133] In this embodiment, the interface board switches the output data stream by sending a switching command from the server, thereby meeting the data stream switching requirements of the remote device and realizing diversified audio and video data exchange processing. The audio and video exchange is mainly carried out on the interface board, and the server only gives simple command instructions, which can achieve the effect of not having high requirements for CPU and GPU.

[0134] Figure 10 A flowchart of a data stream switching method provided in an embodiment of this application is shown below. Figure 10 After receiving a keyframe from the third remote device, the second interface board outputs the input data stream from the third remote device through the second input / output channel, including:

[0135] S1001, the second interface board sends a request message to the third remote device through the second input / output channel. The request message is used to request the third remote device to send a key frame.

[0136] Optionally, when a remote device connected to the interface board needs to switch data streams, it sends a switching request through the input / output channel connected to the interface board. The interface board receives this switching request and identifies the remote device corresponding to the data stream being switched based on it. The switching request includes the remote device identifier before the switch, the target remote device identifier, and the node to be switched. The switching request indicates which data stream to switch to for output, such as switching from the audio stream of remote device L to the audio stream of remote device H. It is worth noting that the second interface board refers to the interface board containing the remote device with the switching request, and the second input / output channel is the channel that interfaces with the remote device with the switching request.

[0137] Optionally, the processor on the interface board identifies the target remote device to be switched based on the switching requirements of the remote device. The interface board sends a request message to the server through the channel where the remote device with the switching requirement is located. The request message includes the identifier and data type of the target remote device to be switched. The server identifies the switching target and the target interface board where the target remote device to be switched is located based on the request message from the interface board. The server sends a request switching command to the target interface board. Based on the request switching command, the target interface board instructs the target remote device to start sending a data stream containing keyframes. The target interface board encapsulates the data stream and sends it to the backplane. The backplane sends the multicast stream to the switching board, and the switching board sends the multicast stream to the backplane. The backplane then sends the multicast stream to the interface board where the remote device with the switching requirement is located.

[0138] It is worth noting that the interface board only begins receiving the data stream from the target remote device after recognizing the keyframe.

[0139] For example, if the remote device c2 connected to the second input / output channel of the interface board c in the switching device A has a need to switch data streams, the remote device c2 first receives one audio multicast stream for 20 seconds and then receives three audio multicast streams until the end. Based on the need to switch data streams, the remote device c2 sends a switching request to the interface board c. The interface board c2 sends the switching request to the server. Based on the switching request, the server sends a switching request to the source interface board b of the three audio multicast streams. Based on the switching request, the interface board b instructs the remote device b3 to input a data stream with keyframes.

[0140] S1002, the second interface board receives key frames sent by the third remote device through the second input / output channel, and outputs the input data stream from the third remote device through the second input / output channel.

[0141] Optionally, the second interface board receives key frames sent by the third remote device through the second input / output port, identifies the input data stream of the third remote device starting from the key frame, and sends the input data stream of the third remote device to the remote device with switching requirements through the second input / output port.

[0142] For example, if the remote device c2 connected to the second input / output channel of the interface board c in the switching device A has a need to switch data streams, the interface board c receives the data stream with key frames from the remote device b3. When the interface board c recognizes the key frames in the data stream, the interface board c sends the data stream from the second input / output port to the remote device c2.

[0143] Optionally, when the input / output port of the interface board receives the key frame of the remote device to be switched, the input / output channel stops receiving the data stream from the remote device before the switch and starts receiving the data stream from the remote device after the switch and sending it from the key frame. At this time, the interface board sends a command to the remote device before the switch to stop sending the data stream. The initial remote device stops sending the data stream and enters standby mode.

[0144] In this embodiment, according to the switching requirements of the remote device, the interface board sends a request message to the target remote device to be switched. The target remote device sends a data stream with key frames to the interface board with the switching requirement based on the request message. The interface board identifies the key frames in the data stream and transmits the data stream of the target remote device from the key frames to complete the data stream switching. This can realize diversified switching processing of audio and video exchange without relying on high-requirement CPUs and GPUs, achieving the effect of low CPU and GPU requirements.

[0145] As an optional implementation, audio and video signal exchange includes not only audio exchange, video exchange, and synchronized audio and video exchange, but also audio mixing exchange. For example, it may be necessary to mix the accompaniment of music with the singer's a cappella audio and send it to a remote device to achieve a mixing effect.

[0146] Figure 11 A flowchart of a mixing processing method provided in an embodiment of this application is shown below. Figure 11 The mixing function in the audio and video switching process relies on the mixing board in the switching equipment. The mixing method is as follows:

[0147] Sending the multicast stream to the second interface board includes:

[0148] Optionally, the remote device requiring mixing can interface with the second interface board to send the mixed multicast stream to the second interface board. The second interface board can be any interface board in the switching equipment, but the remote device requiring mixing must be connected to the second interface board.

[0149] S1101. If the type of the multicast stream is audio, the first interface board will send the multicast stream to the mixing board.

[0150] Optionally, if all the multicast streams that need to be mixed are audio, the interface board will send all the multicast streams that need to be mixed to the mixing board. Here, the first interface board refers to the set of interface boards in the switching device that contain the remote devices corresponding to the multicast streams that need to be mixed. The first interface board can be a single interface board in the switching device or a set of multiple interface boards.

[0151] It is worth noting that the data stream used for mixing is generally an audio stream, while video streams can only be switched or subjected to other operations.

[0152] For example, if the remote device c5 connected to the fifth input / output channel of interface board c in switching device A has a mixing requirement, and the mixing requirement of remote device c5 is to mix 1 audio multicast stream with 3 audio multicast streams, assuming that the 1 audio multicast stream comes from the remote device a1 corresponding to the first input / output channel of interface board a, and the 3 audio multicast streams come from the remote device b3 corresponding to the third input / output channel of interface board b, according to the mixing requirement of remote device c5, interface board a sends the 1 audio multicast stream to mixing board F, and interface board b sends the 3 audio multicast streams to mixing board F.

[0153] S1102. The mixing board performs mixing processing on the multicast stream to obtain the mixed audio raw stream, and then encapsulates the mixed audio raw stream to obtain the mixed multicast stream.

[0154] Optionally, the mixing board performs mixing processing on the received multicast stream. Specifically, the mixing board first decapsulates the received audio multicast stream to obtain the corresponding raw audio stream. Then, based on a volume detection algorithm and audio mixing logic, these raw audio streams are mixed to obtain the mixed audio raw stream. The processor on the mixing board then encapsulates the mixed audio raw stream based on the data type of the audio stream to obtain the mixed multicast stream. It is worth noting that the raw audio stream refers to the decoded pulse-mode audio signal.

[0155] For example, the mixing board F receives a 1-channel audio multicast stream from interface board a and a 3-channel audio multicast stream from interface board b. The processor on the mixing board F decodes the 1-channel and 3-channel audio multicast streams based on the data type of the audio streams to obtain a 1-channel raw audio stream and a 3-channel raw audio stream. According to the volume detection algorithm and audio mixing logic, the 1-channel and 3-channel raw audio streams are mixed to obtain a mixed audio raw stream n. The processor of the mixing board encapsulates the mixed audio raw stream n based on the data type of the mixed audio stream to obtain a mixed multicast stream N.

[0156] S1103, the mixing board sends the mixed multicast stream to the second interface board.

[0157] Optionally, the mixing board sends the mixed multicast stream to the interface board of the remote device that needs to be mixed.

[0158] S1104. The second interface board decapsulates the multicast stream according to its type to obtain the output data stream, including:

[0159] The second interface board decapsulates the mixed multicast stream according to its type to obtain the output data stream.

[0160] Optionally, the interface board receives the mixed multicast stream and decapsulates the multicast stream according to its data type to obtain the output data stream.

[0161] Optionally, the interface board's processor identifies the remote device that needs mixing based on the mixing request sent by the remote device corresponding to the input / output channel. The interface board then sends the decrypted mixed multicast stream through the input / output channel to the remote device that needs mixing, and the remote device receives and displays the output data stream.

[0162] For example, it is known that the remote device c5 corresponding to the fifth input / output channel of interface board c has a mixing requirement. Its mixing requirement is to mix one audio multicast stream from interface board a and three audio multicast streams from interface board b. Mixing board F mixes the one audio multicast stream and the three audio multicast streams according to the mixing requirement of c5 and obtains a mixed multicast stream N. Mixing board F sends the mixed multicast stream N to the backplane. The backplane sends the mixed multicast stream N to the switching board. The switching board exchanges IP addresses and then sends the mixed multicast stream N to the backplane. The backplane sends the mixed multicast stream N to interface board c. The processor of interface board c decapsulates the mixed multicast stream to obtain the mixed audio raw stream n. According to the mixing requirement of remote device c5, the processor encodes the mixed audio raw stream n and sends it to remote device c5 through the fifth input / output channel.

[0163] In this embodiment, based on the mixing requirements of the remote device, the audio multicast stream that meets the mixing requirements is mixed by the mixing board to obtain a mixed multicast stream. The mixed multicast stream is sent to the interface board where the remote device with mixing requirements is located. The interface board decodes the mixed multicast stream and sends the decoded mixed multicast stream to the remote device with mixing requirements through a specific input / output channel based on the mixing requirements of the remote device. The remote device receives the mixed data stream and displays the mixing effect, which can achieve the effect of low bandwidth and low requirements on CPU and GPU.

[0164] As an optional implementation, the multicast stream in step S302 above can specifically be:

[0165] Multicast streams include: channel identifier, data type, whether it is a keyframe, frame start flag, frame end flag, frame sequence number, packet sequence number, timestamp, data length, and media data.

[0166] As shown in Table 1, the multicast stream obtained after data stream encapsulation contains the following information:

[0167]

[0168] Optionally, the channel identifier includes the identifier of the interface board and the identifier of the channel on the interface board. The channel identifier can indicate which remote device on which interface board the input data stream comes from.

[0169] Optionally, the data type can indicate whether the input data stream corresponding to the multicast stream is a separate audio stream, a separate video stream, or a combination of both. Depending on the data type, it can indicate whether timestamp synchronization is required, and it can also indicate the method for unpacking the multicast stream.

[0170] Optionally, whether a frame is a keyframe can be determined based on the field name, data type, and value range of the data stream. Keyframes can be used to indicate the switching of multicast streams. Multicast stream switching can only be performed when a keyframe is identified, thus achieving seamless switching between video and audio streams.

[0171] Optionally, the start-of-frame flag is used to indicate the first frame of the multicast stream to be received, and the end-of-frame flag is used to indicate the last frame of the multicast stream to be received.

[0172] Optionally, the input data stream is encapsulated to obtain a multicast stream. The header of the multicast stream contains a frame sequence number and a packet sequence number. The frame sequence number can indicate whether frame loss has occurred during the transmission of the multicast stream, and the packet sequence number can indicate whether packet loss has occurred during the transmission of the multicast stream. The combination of the frame sequence number and the packet sequence number can indicate the interface board's processing method for the multicast stream.

[0173] Optionally, the timestamp is the original timestamp inherent in the audio and video streams of the input data stream. The timestamp can be used to instruct the interface board to perform timestamp synchronization processing on data streams from the same remote device, thereby improving the user's audiovisual experience.

[0174] Optionally, the input and output channels of the interface board can determine whether the audio stream and video stream in the input data stream come from the same remote device based on the channel identifiers corresponding to the acquired audio stream and video stream. When it is determined that the audio stream and video stream come from the same remote device, timestamp synchronization processing needs to be performed on the current audio stream and video stream.

[0175] Optionally, the data length is used to indicate the length of data contained in the input data stream, and can indicate the length of the input data stream data stored in the memory on the interface board.

[0176] Optionally, media data can be used to indicate the streaming media transmission protocol corresponding to the data stream transmission, indicating which streaming media transmission protocol the interface board's channel uses to interface with the remote device corresponding to the input data stream.

[0177] In the embodiments of this application, by using the channel identifier, data type, whether it is a keyframe, frame start flag, frame end flag, frame sequence number, packet sequence number, timestamp, data length and media data contained in the multicast stream, the interface board can easily realize various audio and video exchange functions, and can achieve the effect of not having high requirements for CPU and GPU.

[0178] The following describes the interface board, switching device, and computer-readable storage medium used to perform the audio and video signal switching processing provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0179] This application provides an interface board, which includes a memory, a processor, and multiple input / output channels. Each input / output channel is used to connect to a remote device. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements any of the above-described audio / video signal exchange processing method embodiments.

[0180] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0181] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs any of the above-described audio / video signal exchange processing method embodiments.

[0182] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0184] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for audio and video signal switching and processing, characterized in that, Applied to switching equipment, including: multiple interface boards, each interface board including multiple input / output channels, each input / output channel being used to connect to a remote device; The first interface board receives an input data stream from the first remote device through a first input / output channel connected to the first remote device. The input data stream includes an audio stream or a video stream. The first interface board encapsulates the data stream according to the data type to obtain a multicast stream, and sends the multicast stream to the second interface board. The multicast stream includes: channel identifier, data type, whether it is a keyframe, frame start flag, frame end flag, frame sequence number, packet sequence number, timestamp, data length, and media data. The multicast stream is divided into video multicast stream and audio multicast stream. The second interface board decapsulates the multicast stream according to its type to obtain an output data stream, and sends the output data stream to the second remote device through the second input / output channel connected to the second remote device. When the first interface board and the second interface board are not the same interface board, the multicast stream is sent to the backplane via the first interface board, and the backplane forwards the multicast stream to the second interface board based on the IGMP protocol; If the input data stream received by the first interface board is an audio stream, the first interface board first decodes the audio stream and then encapsulates the decoded audio stream. If the output data stream is an output audio stream, the second interface board encodes the decrypted output audio stream and sends it to the second remote device.

2. The audio and video signal switching and processing method according to claim 1, characterized in that, Sending the output data stream to the second remote device via the second input / output channel connected to the second remote device includes: If the second interface board receives a data stream to be synchronized that is the same as but different in type from the data source of the output data stream, it performs timestamp synchronization processing on the output data stream and the data stream to be synchronized, and sends the processed output data stream and the data stream to be synchronized to the second remote device through the second input / output channel.

3. The audio and video signal switching and processing method according to claim 2, characterized in that, The timestamp synchronization process for the output data stream and the data stream to be synchronized includes: The timestamps corresponding to the initial audio stream or video stream are respectively recorded as the system audio timestamp and the system video timestamp; Receive audio multicast streams or video multicast streams, and record the timestamp corresponding to the first frame of the audio multicast stream as the original audio timestamp, and the timestamp corresponding to the first frame of the video multicast stream as the original video timestamp. For each received audio multicast stream or video multicast stream, a new audio timestamp and video timestamp are determined, wherein the new audio timestamp or new video timestamp of the previous frame is used as the original audio timestamp or original video timestamp of the next frame. The timestamp corresponding to the output audio stream is determined based on the system audio timestamp, the original audio timestamp, and the new audio timestamp; the timestamp corresponding to the output video stream is determined based on the system video timestamp, the original video timestamp, and the new video timestamp.

4. The audio and video signal switching and processing method according to claim 1, characterized in that, Also includes: If the second interface board receives a switching instruction from the server while outputting an input data stream from the first remote device through the second input / output channel, and the switching instruction includes the identifier of the third remote device to which the second input / output channel is to be switched, then after receiving the keyframe sent by the third remote device, the second interface board outputs an input data stream from the third remote device through the second input / output channel.

5. The audio and video signal switching and processing method according to claim 4, characterized in that, After receiving the keyframe sent by the third remote device, the second interface board outputs the input data stream from the third remote device through the second input / output channel, including: The second interface board sends a request message to the third remote device through the second input / output channel. The request message is used to request the third remote device to send a keyframe. The second interface board receives the key frame sent by the third remote device through the second input / output channel, and outputs the input data stream from the third remote device through the second input / output channel.

6. The audio and video signal switching processing method according to any one of claims 1-5, characterized in that, The switching equipment also includes: a mixing board; Sending the multicast stream to the second interface board includes: If the type of the multicast stream is audio, then the first interface board sends the multicast stream to the mixing board; The mixing board performs mixing processing on the multicast stream to obtain a mixed audio raw stream, and encapsulates the mixed audio raw stream to obtain a mixed multicast stream; The mixing board sends the mixed multicast stream to the second interface board; The second interface board decapsulates the multicast stream according to its type to obtain an output data stream, including: The second interface board decapsulates the mixed multicast stream according to the mixed multicast stream type to obtain the output data stream.

7. An interface board, characterized in that, include: The system includes a memory, a processor, and multiple input / output channels, each used to connect to a remote device. The memory stores a computer program that can run on the processor, and when the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

8. A switching device, characterized in that, include: The interface board, switching board, backplane, and mixing board as described in multiple claims 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images