An apparatus and device based on dual system supporting audio and video decoding and SDI output

Abstract

The application relates to the technical field of audio and video processing, and particularly discloses a device and equipment supporting audio and video decoding and SDI output based on a double system, which comprises an ARM+FPGA system and a decoding SOC system, the decoding SOC system comprises an audio and video hardware module, an audio soft decoding module and an audio and video synchronization module, the audio and video hardware module is connected with an HDMI module, the audio and video hardware module and the audio soft decoding module are connected with the audio and video synchronization module; the audio and video synchronization module acquires decoded video data, synchronizes second audio data received with the video data, and then transmits the synchronized second audio data to the ARM+FPGA system; wherein the ARM+FPGA system processes the received video data, first audio data and second audio data, and then converts the data into an SDI standard format for output; the application adopts the ARM+FPGA system to increase the function of SDI output, supports video and double audio decoding, and supports a professional output interface SDI, can meet the decoding of numerous mainstream audio and video formats, and has the advantages of professionalism and cost performance.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, and in particular to a device and equipment based on a dual system that supports audio and video decoding and SDI output. Background Technology

[0002] Decoding is the reverse process of encoding. It requires decrypting the data obtained from the network to obtain the video's encapsulation packet, then unpacking it to obtain the encoded audio and video data. Finally, these audio and video data are decoded according to their corresponding encoding formats. After synchronizing the decoded audio and video, they can be played on the player.

[0003] In recent years, the price of professional decoding chip solutions from abroad has been rising, making decoders using these solutions less competitive in the market. However, existing consumer-grade decoding SOC chips cannot meet the decoding needs of many mainstream audio and video formats, cannot decode multiple audio streams, and lack a professional SDI output interface. Summary of the Invention

[0004] To address the issues of increasingly high prices for foreign professional decoding chip solutions and the inability of existing consumer-grade decoding SOC chips to meet the decoding requirements of many mainstream audio and video formats, to decode multiple audio streams, and to lack a professional SDI output interface, this invention provides a device and equipment based on a dual-system architecture that supports audio and video decoding and SDI output. It employs an ARM+FPGA system to add SDI output functionality, supports video and dual audio decoding, and supports the professional SDI output interface, satisfying the decoding needs of many mainstream audio and video formats. It is both professional and highly cost-effective.

[0005] To solve the above-mentioned technical problems, the specific solution provided by the present invention is as follows:

[0006] A device based on a dual system supporting audio and video decoding and SDI output includes: an ARM+FPGA system and a decoding SOC system. The decoding SOC system includes an audio and video hardware module, an audio software decoding module, and an audio and video synchronization module. The audio and video hardware module is connected to an HDMI module, and the audio and video hardware module and the audio software decoding module are connected to the audio and video synchronization module.

[0007] The ARM+FPGA system converts the received TS stream into a single-program TS stream and sends the single-program TS stream to the decoding SOC system.

[0008] The audio and video hardware module decodes the video and the first audio stream in a single program's TS stream, and transmits the decoded video data and the first audio data to the HDMI module.

[0009] The HDMI module converts the received video data and the first audio data into different formats and then transmits them to the ARM+FPGA system.

[0010] The audio software decoding module decodes the second audio stream in the TS stream of a single program and transmits the decoded second audio data to the audio-video synchronization module.

[0011] The audio-video synchronization module acquires the decoded video data, synchronizes the received second-channel audio data with the video data, and then transmits the synchronized second-channel audio data to the ARM+FPGA system; wherein,

[0012] The ARM+FPGA system processes the received video data, first audio data, and second audio data, and then converts them into the SDI standard format for output.

[0013] In some implementations, the decoded video data and the first audio data are synchronized using PCR synchronization to facilitate the synchronization of the second audio data with the video data.

[0014] In some implementations, the decoding SOC system further includes a multiplexing module, which connects the ARM+FPGA system, the audio / video hardware module, and the audio software decoding module;

[0015] The multiplexing module receives the single-program TS stream transmitted from the ARM+FPGA system, separates the data in the single-program TS stream, and transmits the part containing video, the first audio channel, and PCR information to the audio and video hardware module, and transmits the part containing the second audio channel to the audio software decoding module, thereby achieving the effect of decoding multiple audio channels.

[0016] In some implementations, the audio software decoding module decodes the second audio stream in the TS stream of a single program and transmits the decoded second audio data to the audio-video synchronization module. The second audio data carries the PTS and is transmitted to the audio-video synchronization module so that the audio-video synchronization module can obtain the PTS of the decoded video data and compare it with the PTS of the second audio data.

[0017] In some implementations, the audio-video synchronization module acquires the decoded video data, synchronizes the received second-channel audio data with the video data, and then transmits the synchronized second-channel audio data to the ARM+FPGA system. The synchronization process between the second-channel audio data and video data includes:

[0018] The audio-video synchronization module obtains the PTS of the decoded video data, compares the PTS of the video data with the PTS of the second audio data, and ensures that the difference between the PTS of the video data and the PTS of the second audio data is within a set threshold, thereby achieving synchronization between the second audio data and the video data.

[0019] In some implementations, comparing the PTS of the video data with the PTS of the second audio data, and ensuring that the difference between the PTS of the video data and the PTS of the second audio data is within a set threshold, includes:

[0020] If the difference between the PTS of the second audio data and the PTS of the video data exceeds a set threshold, some audio frames are discarded to keep the difference between the PTS of the second audio data and the PTS of the video data within the set threshold.

[0021] If the difference between the PTS of the second audio data and the PTS of the video data exceeds a set threshold, some audio frames are repeated to bring the difference between the PTS of the second audio data and the PTS of the video data within the set threshold, thus achieving synchronization between the second audio data and the video data.

[0022] In some implementations, transmitting the synchronized second audio data to the ARM+FPGA system includes:

[0023] The synchronized second audio data is transmitted to the ARM+FPGA system in SPDIF format.

[0024] In some implementations, the HDMI module includes an HDMI output and an HDMI receiver;

[0025] The decoded video data and the first audio data are transmitted to the HDMI receiver via the HDMI output;

[0026] The HDMI receiver transmits video data to the ARM+FPGA system in DVO format, and transmits the first audio data to the ARM+FPGA system in SPDIF format.

[0027] In some implementations, the ARM+FPGA system includes an FPGA SDI output module;

[0028] The ARM+FPGA system processes the received video data, first-channel audio data, and second-channel audio data, and then converts them into SDI standard format for output, including:

[0029] The FPGA SDI output module converts the first and second audio data in SPDIF format into the first and second audio data in AES3 format, and after delay control via DDR, embeds them into the cancellation area of ​​the SDI output frame according to the standard protocol.

[0030] Convert DVO format video data into frame format video data and process the data;

[0031] After embedding, the first audio data, the second audio data, and the video data are converted into the SDI standard format and output through the IP core.

[0032] This application also provides an apparatus comprising the dual-system support for audio / video decoding and SDI output described in any of the preceding claims.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a device based on a dual system that supports audio and video decoding and SDI output. It adopts an ARM+FPGA system to add SDI output function, supports professional output interface SDI, uses an audio hardware module to decode the video and the first audio in the TS stream, and uses an audio software decoding module and an audio-video synchronization module to decode the second audio and synchronize the decoded audio. It supports video and dual audio decoding, meets the decoding of many mainstream audio and video formats, and is professional while being highly cost-effective. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of a device based on a dual system that supports audio and video decoding and SDI output, provided in an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of a decoding SOC system provided in another embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the device provided in an embodiment of the present invention;

[0037] Wherein, 1-equipment; 2-device. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are some, but not all, of the embodiments of the present invention.

[0039] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0040] For example, a dual-system device supporting audio / video decoding and SDI output includes: an ARM+FPGA system and a decoding SOC system. The decoding SOC system includes an audio / video hardware module, an audio software decoding module, and an audio / video synchronization module. The audio / video hardware module is connected to an HDMI module. The audio / video hardware module and the audio software decoding module are connected to the audio / video synchronization module. The ARM+FPGA system converts the received TS stream into a single-program TS stream and sends the single-program TS stream to the decoding SOC system. The audio / video hardware module decodes the video and a first audio stream in the single-program TS stream and outputs the decoded video data and the first audio stream. Data is transmitted to the HDMI module; the HDMI module converts the received video data and the first audio data into a different format and then transmits them to the ARM+FPGA system; the audio software decoding module decodes the second audio stream in the single program's TS stream and transmits the decoded second audio data to the audio-video synchronization module; the audio-video synchronization module acquires the decoded video data, synchronizes the received second audio data with the video data, and then transmits the synchronized second audio data to the ARM+FPGA system; wherein, the ARM+FPGA system processes the received video data, the first audio data, and the second audio data, and then converts them into the SDI standard format for output.

[0041] This embodiment provides a dual-system audio and video decoding support method. It adopts an ARM+FPGA system to add SDI output functionality, supports the professional output interface SDI, uses an audio hardware module to decode the video and the first audio channel in the TS stream, and uses an audio software decoding module and an audio-video synchronization module to decode the second audio channel and synchronize the decoded audio. It supports video and dual audio decoding, meets the decoding requirements of many mainstream audio and video formats, and is both professional and cost-effective.

[0042] Example 1:

[0043] like Figure 1As shown, a dual-system device supporting audio and video decoding and SDI output adopts a decoding scheme combining a domestic consumer-grade decoding SOC system with an ARM+FPGA system. The device includes an ARM+FPGA system and a decoding SOC system. The decoding SOC system includes an audio and video hardware module, an audio software decoding module, and an audio and video synchronization module. The audio and video hardware module is connected to an HDMI module, and the audio and video hardware module and the audio software decoding module are connected to the audio and video synchronization module.

[0044] The ARM+FPGA system converts the received TS stream into a single-program TS stream and sends the single-program TS stream to the decoding SOC system.

[0045] There are two types of TS streams: one is a TS stream containing only the audio and video of a single program, called a single-program TS stream; the other is a TS stream multiplexed from multiple programs, called a multi-program multiplexed stream. In this example, we mainly focus on decoding and outputting the single-program TS stream. After being processed by the ARM+FPGA system, the TS stream becomes a single-program TS stream, which is then sent to the decoding SOC system.

[0046] The audio and video hardware module decodes the video and the first audio stream in a single program's TS stream, and transmits the decoded video data and the first audio data to the HDMI module.

[0047] The audio and video hardware module only supports decoding one video and one audio stream. Therefore, here, the audio and video hardware module is used to decode the video and the first audio stream in a single program's TS stream, and transmit the decoded video data and the first audio data to the HDMI module.

[0048] The HDMI module converts the received video data and the first audio data into different formats before transmitting them to the ARM+FPGA system.

[0049] The video data and the first audio data decoded by the audio / video hardware module do not meet the output requirements of the later stages. Therefore, an HDMI module is needed to convert the received video data and the first audio data before transmitting them to the ARM+FPGA system. In this example, the HDMI module converts the received video data to DVO format and transmits the first audio data to the ARM+FPGA system in SPDIF format to meet the processing requirements of the ARM+FPGA system for audio and video data. It should be noted that this example does not impose specific restrictions on the conversion formats of the video and audio data; the example is provided merely to facilitate understanding of the implementation scheme.

[0050] The audio software decoding module decodes the second audio stream in a single program's TS stream and transmits the decoded second audio data to the audio-video synchronization module.

[0051] The audio software decoding module is based on a software decoding framework and decoding library. It decodes the second audio channel in the TS stream of a single program, that is, it uses the CPU for decoding. The CPU in the decoding SOC system has strong performance and processing power, good compatibility with streaming media formats, and the decoding library is self-developed, which can better control the quality.

[0052] The audio-video synchronization module acquires the decoded video data, synchronizes the received second-channel audio data with the video data, and then transmits the synchronized second-channel audio data to the ARM+FPGA system.

[0053] The ARM+FPGA system processes the received video data, first audio data, and second audio data, and then converts them into the SDI standard format for output.

[0054] This example provides a dual-system approach to audio and video decoding. It utilizes an ARM+FPGA system to add SDI output functionality, supporting the professional SDI output interface. The audio hardware module decodes the video and the first audio channel in the TS stream, while the audio software decoding module and audio-video synchronization module decode the second audio channel and synchronize the decoded audio. It supports video and dual audio decoding, meets the decoding requirements of many mainstream audio and video formats, and is both professional and cost-effective.

[0055] Example 2:

[0056] The audio and video hardware module is used to decode the video and the first audio stream in a single program's TS stream, and transmits the decoded video data and the first audio data to the HDMI module. Here, the decoded video data and the first audio data are synchronized using PCR synchronization to facilitate the synchronization of the second audio data with the video data.

[0057] PCR is the system reference clock. When the encoder is encoding, it samples its own system clock and puts it into the TS stream in the form of PCR. When the decoder is decoding, it uses PCR to synchronize its own system clock to ensure that the encoder and decoder clocks are synchronized. Here, PCR synchronization is used to ensure the synchronization of the decoded video data and the first audio data.

[0058] In some implementation schemes, such as Figure 2 As shown, the decoding SOC system also includes a multiplexing module, which connects to the ARM+FPGA system, audio and video hardware module, and audio software decoding module.

[0059] The multiplexing module receives the single-program TS stream from the ARM+FPGA system, separates the data in the single-program TS stream, and sends the part containing video, the first audio channel, and PCR information to the audio and video hardware module, and sends the part containing the second audio channel to the audio software decoding module, thereby achieving the effect of decoding multiple audio channels.

[0060] Since the audio / video hardware module only supports decoding one video and one audio stream, the data in the TS stream of a single program is first separated by a multiplexing module. The portion containing video, the first audio stream, and PCR information is then sent to the audio / video hardware module, which decodes the video and the first audio stream of the single program's TS stream and sends the decoded video and first audio data to the HDMI module. The portion containing the second audio stream is sent to the audio software decoding module, which decodes the second audio stream of the single program's TS stream based on the software decoding framework and decoding library.

[0061] Example 3:

[0062] The audio software decoding module decodes the second audio stream in a single program's TS stream and transmits the decoded second audio data to the audio-video synchronization module. The second audio data carries the PTS (Pulse Test Status) and is transmitted to the audio-video synchronization module so that the audio-video synchronization module can obtain the PTS of the decoded video data and compare it with the PTS of the second audio data.

[0063] During the decoding process, the system clock is first reconstructed using PCR and synchronized with the encoder. After the system clock is restored, the synchronization between the video data and the second audio data is performed using PTS.

[0064] After the audio-video synchronization module acquires the decoded video data, it synchronizes the received second-channel audio data with the video data, and then transmits the synchronized second-channel audio data to the ARM+FPGA system. The synchronization process between the second-channel audio data and video data includes:

[0065] The audio-video synchronization module obtains the PTS of the decoded video data, compares the PTS of the video data with the PTS of the second audio data, and ensures that the difference between the PTS of the video data and the PTS of the second audio data is within a set threshold, thereby achieving synchronization between the second audio data and the video data.

[0066] Specifically, the PTS of the video data is compared with the PTS of the second audio data, and the difference between the PTS of the video data and the PTS of the second audio data is kept within a set threshold, including:

[0067] If the difference between the PTS of the second audio data and the PTS of the video data exceeds a set threshold, some audio frames are discarded to keep the difference between the PTS of the second audio data and the PTS of the video data within the set threshold.

[0068] If the difference between the PTS of the second audio data and the PTS of the video data exceeds a set threshold, some audio frames are repeated to bring the difference between the PTS of the second audio data and the PTS of the video data within the set threshold, thus achieving synchronization between the second audio data and the video data.

[0069] For example, if the threshold for the difference between the PTS of the second audio data and the PTS of the video data is set to 100ms, and one audio frame is approximately 20ms, and the current PTS of the second audio data is slower than the PTS of the video data by 110ms, then only one audio frame needs to be discarded. Similarly, if the current PTS of the second audio data is faster than the PTS of the video data by 110ms, then only one audio frame needs to be repeated.

[0070] Example 4:

[0071] In this example, transmitting the synchronized second audio data to the ARM+FPGA system includes transmitting the synchronized second audio data to the ARM+FPGA system in SPDIF format.

[0072] The HDMI module includes an HDMI output and an HDMI receiver. The decoded video data and the first audio data are transmitted to the HDMI receiver via the HDMI output. The HDMI receiver transmits the video data to the ARM+FPGA system in DVO format, and the first audio data is transmitted to the ARM+FPGA system in SPDIF format.

[0073] The ARM+FPGA system includes an FPGA SDI output module. Specifically, the ARM+FPGA system processes the received video data, first-channel audio data, and second-channel audio data, and converts them into SDI standard format for output, including:

[0074] The FPGA SDI output module converts the first and second audio data in SPDIF format into the first and second audio data in AES3 format, and after delay control via DDR, embeds them into the cancellation area of ​​the SDI output frame according to the standard protocol.

[0075] Video frames have blanking regions and valid frame regions. Video frames are stored in the valid frame regions, while audio data embedding refers to embedding audio data into the blanking region according to standard protocols.

[0076] Convert DVO format video data into frame format video data and process the data.

[0077] The aforementioned data processing process specifically includes parsing DVO format video data, converting it into frame format video data with line and field information as well as frame start and frame end signals, and then embedding auxiliary data according to relevant protocols, such as adding subtitles and audio data.

[0078] After embedding, the first audio data, the second audio data, and the video data are converted into the SDI standard format and output through the IP core.

[0079] Example 5:

[0080] refer to Figures 1-3 This application also provides an apparatus comprising the dual-system-based device for supporting audio and video decoding and SDI output as described in any of the preceding claims.

[0081] This invention provides a dual-system audio and video decoding support system. It uses an ARM+FPGA system to add SDI output functionality, supports the professional output interface SDI, uses an audio hardware module to decode the video and the first audio channel in the TS stream, and uses an audio software decoding module and an audio-video synchronization module to decode the second audio channel and synchronize the decoded audio. It supports video and dual audio decoding, meets the decoding requirements of many mainstream audio and video formats, and is both professional and cost-effective.

[0082] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0083] It should be understood that the present invention is not limited to the content already described above, and various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A device based on a dual-system architecture supporting audio / video decoding and SDI output, characterized in that, include: The system comprises an ARM+FPGA system and a decoding SOC system. The decoding SOC system includes an audio / video hardware module, an audio software decoding module, and an audio / video synchronization module. The audio / video hardware module is connected to an HDMI module, and the audio / video hardware module and the audio software decoding module are connected to the audio / video synchronization module. The ARM+FPGA system converts the received TS stream into a single-program TS stream and sends the single-program TS stream to the decoding SOC system. The audio and video hardware module decodes the video and the first audio stream in a single program's TS stream, and transmits the decoded video data and the first audio data to the HDMI module. The HDMI module converts the received video data and the first audio data into different formats and then transmits them to the ARM+FPGA system. The audio software decoding module decodes the second audio stream in the TS stream of a single program and transmits the decoded second audio data to the audio-video synchronization module. The audio-video synchronization module acquires the decoded video data, synchronizes the received second-channel audio data with the video data, and then transmits the synchronized second-channel audio data to the ARM+FPGA system; wherein, The ARM+FPGA system processes the received video data, first audio data, and second audio data, and then converts them into the SDI standard format for output. The decoded video data and the first audio data are synchronized using PCR synchronization. The decoding SOC system also includes a multiplexing module, which connects to the ARM+FPGA system, the audio and video hardware module, and the audio software decoding module; The multiplexing module receives the single-program TS stream transmitted from the ARM+FPGA system, separates the data in the single-program TS stream, and transmits the part containing video, the first audio channel, and PCR information to the audio and video hardware module, and transmits the part containing the second audio channel to the audio software decoding module.

2. The device based on a dual system supporting audio / video decoding and SDI output according to claim 1, characterized in that, The audio software decoding module decodes the second audio stream in the TS stream of a single program and transmits the decoded second audio data to the audio-video synchronization module. The second audio data carries the PTS and is transmitted to the audio-video synchronization module.

3. The device based on a dual system supporting audio / video decoding and SDI output according to claim 2, characterized in that, The audio-video synchronization module acquires the decoded video data, synchronizes the received second-channel audio data with the video data, and then transmits the synchronized second-channel audio data to the ARM+FPGA system. The synchronization process between the second-channel audio data and video data includes: The audio-video synchronization module obtains the PTS of the decoded video data, compares the PTS of the video data with the PTS of the second audio data, and ensures that the difference between the PTS of the video data and the PTS of the second audio data is within a set threshold.

4. The device based on a dual system supporting audio / video decoding and SDI output according to claim 3, characterized in that, The step of comparing the PTS of the video data with the PTS of the second audio data, and ensuring that the difference between the PTS of the video data and the PTS of the second audio data is within a set threshold, includes: If the difference between the PTS of the second audio data and the PTS of the video data exceeds a set threshold, some audio frames are discarded to keep the difference between the PTS of the second audio data and the PTS of the video data within the set threshold. If the difference between the PTS of the second audio data and the PTS of the video data exceeds a set threshold, some audio frames are repeated to bring the difference between the PTS of the second audio data and the PTS of the video data within the set threshold.

5. The device based on a dual system supporting audio / video decoding and SDI output according to claim 4, characterized in that, The process of transmitting the synchronized second audio data to the ARM+FPGA system includes: The synchronized second audio data is transmitted to the ARM+FPGA system in SPDIF format.

6. The apparatus for supporting audio and video decoding and SDI output based on a dual system according to claim 3, characterized in that, The HDMI module includes an HDMI output and an HDMI receiver; The decoded video data and the first audio data are transmitted to the HDMI receiver via the HDMI output; The HDMI receiver transmits video data to the ARM+FPGA system in DVO format, and transmits the first audio data to the ARM+FPGA system in SPDIF format.

7. The apparatus for supporting audio / video decoding and SDI output based on a dual system according to claim 6, characterized in that, The ARM+FPGA system includes an FPGA SDI output module; The ARM+FPGA system processes the received video data, first-channel audio data, and second-channel audio data, and then converts them into SDI standard format for output, including: The FPGA SDI output module converts the first and second audio data in SPDIF format into the first and second audio data in AES3 format, and after delay control via DDR, embeds them into the cancellation area of ​​the SDI output frame according to the standard protocol. Convert DVO format video data into frame format video data and process the data; After embedding, the first audio data, the second audio data, and the video data are converted into the SDI standard format and output through the IP core.

8. A device based on a dual-system architecture supporting audio / video decoding and SDI output, characterized in that, The device includes any one of claims 1-7, a dual-system apparatus supporting audio / video decoding and SDI output.

Images