Plug-in audio processing device and method

Abstract

The invention discloses a plug-in audio processing method and device, and the method comprises the following steps: directly uploading a plug-in audio to an audio / video server or uploading the plug-in audio to an audio / video server after packaging the plug-in audio according to the format of the plug-in audio; setting different URLs for the plug-in audio source and the audio and video source respectively; and when playing begins, judging whether the source needs the plug-in audio or not, and performing corresponding processing according to whether the source needs the plug-in audio or not. By using the method provided by the invention, when a new audio demand exists, only the plug-in audio is needed, the original video URL and the plug-in audio URL are subjected to state synchronization during playing, and the playing effect is consistent with the effect of a single audio video source.

Description

Technical Field

[0001] The invention belongs to the technical field of audio and video processing, and in particular relates to an external audio processing method and device. Background Art

[0002] With the development of audio and video technology and audio and video terminals, more and more scenes require better audio and video experience, better sound quality, better picture quality, better audio and video atmosphere and feeling. For early audio and video files whose sound quality does not meet the current requirements, they generally need to be re-transcoded and re-produced to convert them into high-quality audio and video.

[0003] However, the audio and video of current audio and video platforms are produced and stored on the server when the video content is uploaded to the video platform. For existing videos, there are still the following problems: one problem is that in general, audio and video have the same URL address in the cloud. If you want to support more audio formats, you need to transcode and produce both audio and video, which is undoubtedly a huge workload; second, for existing audio and video, when new and old videos are compatible, the same resource will form multiple copies of the video simply because of the difference in audio, resulting in redundant occupation of resources on the server. Summary of the invention

[0004] In view of the above problems, the present invention provides an external audio processing method and device. When there is a new audio demand, only the external audio is needed. During playback, the original video URL plus the external audio URL will be synchronized, and the playback effect is consistent with the single audio and video source effect.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The present invention provides a method for processing an external audio, comprising the following steps:

[0007] According to the format of the external audio, upload the external audio directly or after packaging to the audio and video server; set different URLs for the external audio source and the audio and video source respectively;

[0008] When playback starts, determine whether it is a source that requires external audio, and make corresponding processing based on whether it is a source that requires external audio.

[0009] In one possible implementation, corresponding processing is performed according to whether an external audio source is needed, including: if an external audio source is not needed, directly requesting the audio and video source URL from the audio and video server, decapsulating, separating the audio stream and the video stream, and then sending the audio stream and the video stream to the corresponding audio and video decoders for decoding, and after decoding, synchronizing the timestamps of the audio data and the video data, and after synchronization, rendering and playing through the rendering and playback module.

[0010] In one possible implementation, corresponding processing is performed according to whether the source of the external audio is needed, including: if the source of the external audio is needed, requesting the URL of the audio and video source and the URL of the external audio from the audio and video server respectively, and then parsing them; after obtaining the URLs of the audio and video source and the external audio, first decapsulating the audio and video source and the external audio respectively; for the audio and video source, first decapsulate and separate the audio and video in the corresponding slices, and take the video data therein for parsing; for the external audio, parse and process it according to the encapsulation format of the external audio to separate the audio; the parsed audio and video are decoded respectively, and then synchronized before rendering and playing.

[0011] In a possible implementation, parsing and processing according to the encapsulation format of the plug-in audio includes: if the mp4 format is used for encapsulation, first parsing the mp4 file header to obtain the audio information, and then separating the audio frames.

[0012] In a possible implementation, parsing and processing according to the encapsulation format of the plug-in audio includes: adopting a slicing mode, parsing out different addresses and information of the slices, and separating the audio.

[0013] Another aspect of the present invention provides an external audio processing device, including a playback control module, a transpackaging module, an audio and video server, a parsing module, an audio and video synchronization module and a rendering and playback module;

[0014] According to the format of the external audio, the external audio is directly or input into the transpackaging module for packaging and then uploaded to the audio and video server; different URLs are set for the external audio source and the audio and video source respectively;

[0015] When playback starts, the playback control module determines whether it is a source that requires external audio. Depending on whether the source requires external audio, the parsing module, the audio and video synchronization module, and the rendering and playback module make corresponding processing.

[0016] In one possible implementation, depending on whether an external audio source is needed, the parsing module, the audio and video synchronization module, and the rendering and playback module make corresponding processing, including: if the external audio source is not needed, directly request the audio and video source URL from the audio and video server, and after decapsulation, separate the audio stream and the video stream, and then send the audio stream and the video stream to the corresponding audio and video decoders for decoding respectively. After decoding is completed, the audio data and video data are synchronized with the timestamps through the audio and video synchronization module. After synchronization, they are rendered and played through the rendering and playback module.

[0017] In one possible implementation, depending on whether the source of the external audio is needed, the parsing module, the audio and video synchronization module and the rendering and playback module make corresponding processing, including: if the source of the external audio is needed, the URL of the audio and video source and the URL of the external audio are requested from the audio and video server respectively, and then parsed by the parsing module; after obtaining the URLs of the audio and video source and the external audio, the parsing module first decapsulates the audio and video source and the external audio respectively; for the audio and video source, first decapsulate and separate the audio and video in the corresponding slices, take the video data therein for parsing, and send it to the audio and video synchronization module; for the external audio, parse and process it according to the encapsulation format of the external audio, separate the audio, and send it to the audio and video synchronization module; take the video data therein for parsing and then decode them respectively, send them to the audio and video synchronization module for synchronization after decoding, and then render and play them through the rendering and playback module.

[0018] In a possible implementation, parsing and processing according to the encapsulation format of the plug-in audio includes: if the mp4 format is used for encapsulation, first parsing the mp4 file header to obtain the audio information, and then separating the audio frames.

[0019] In a possible implementation, parsing and processing according to the encapsulation format of the plug-in audio includes: adopting a slicing mode, parsing out different addresses and information of the slices, and separating the audio.

[0020] The use of the present invention has the following beneficial effects: when there is a new audio demand, only external audio is needed, and the original video URL plus the external audio URL will be synchronized during playback, and the playback effect is consistent with the single-audio video source effect, which saves the huge workload of producing and transcoding the existing videos, and also saves the huge redundant bandwidth and storage resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of the steps of the external audio processing method according to an embodiment of the present invention;

[0022] Figure 2 The figure is a principle block diagram of an external audio processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 , which is a flowchart of a method for processing external audio according to an embodiment of the present invention, comprising the following steps:

[0025] S10, uploading the external audio to the audio and video server directly or after packaging according to the format of the external audio; setting different URLs for the external audio source and the audio and video source;

[0026] S20, when the playback starts, determine whether it is a source that requires external audio, and make corresponding processing according to whether it is a source that requires external audio.

[0027] In one embodiment of the present invention, in S10, if the external audio is a raw audio stream, and no file layer encapsulation is performed, the main information of the entire external audio, such as the total length of the external audio, etc., cannot be accurately obtained when the audio and video start playing, and problems will arise when synchronizing with the video. If the external audio itself has been encapsulated, it can be repackaged into a mode supported by itself or directly uploaded to the audio and video server. For example, the external audio of Dolby Atmos can be encapsulated in mp4 format and uploaded to the audio and video server after encapsulation.

[0028] In one embodiment of the present invention, S20 makes corresponding processing according to whether the source of the external audio is needed, including: if the source of the external audio is not needed, directly request the audio and video source URL from the audio and video server, separate the audio stream and the video stream after decapsulation, and then send the audio stream and the video stream to the corresponding audio and video decoder for decoding, and after decoding, synchronize the timestamps of the audio data and the video data, and then render and play through the rendering and playback module. If this is the audio and video source of HLS, the URL is an m3u8 description file, and the m3u8 file records the information such as the slice address of the audio and video. Then, through the address URL of the slice, the audio and video data in the ts format of the corresponding slice is obtained, and the ts file is first decapsulated.

[0029] If the source of the plug-in audio is needed, the URL of the audio and video source and the URL of the plug-in audio are requested from the audio and video server respectively, and then parsed; after obtaining the URLs of the audio and video source and the plug-in audio, the audio and video source and the plug-in audio are decapsulated respectively; for the audio and video source, first decapsulate and separate the audio and video in the corresponding slice, and take the video data for parsing; for the plug-in audio, parse and process according to the encapsulation format of the plug-in audio, and separate the audio; the parsed audio and video are decoded respectively, and then synchronized, and then rendered and played. For the plug-in audio, it is parsed and processed according to the encapsulation format of the plug-in audio. If it is mp4 encapsulated, the mp4 file header is parsed first to obtain important audio information, and then the audio frame is separated; if it is a slice mode, it is similar to the m3u8 description file of HLS, parsing the different addresses and information of the slice, separating the audio, and sending it to the audio and video synchronization module. It is equivalent to audio and video coming from two different streams, obtaining video data from the audio and video source, and decapsulating and taking audio data from the plug-in audio. In a specific application example, the audio and video source is an m3u8 file, which uses ts slicing and contains its own audio and video. The external audio is Dolby Atmos, which is encapsulated using mp4. The audio and video streams in the m3u8 are first parsed to separate the video and audio, and the video is taken out for audio and video synchronization. The external audio is first decapsulated to obtain the mp4 file header, and important information such as the total audio length and sampling rate are obtained from the file header information for the next step of synchronization.

[0030] Through the external audio processing method set above, the synchronization of two audio and video sources with different URLs and external audio is realized. The audio and video sources are upgraded by means of external audio. When there is a new audio requirement, only the external audio is needed. When playing, the original video URL plus the external audio URL are parsed and the audio and video are played synchronously, which greatly saves the production and transcoding of a large amount of existing audio and video, as well as redundant bandwidth and storage expenses.

[0031] Corresponding to the method embodiment, participate Figure 2 The present invention also provides an external audio processing device, including a playback control module, a transpackaging module, an audio and video server, a parsing module, an audio and video synchronization module and a rendering and playback module; according to the format of the external audio, the external audio is directly or input into the transpackaging module for packaging and then uploaded to the audio and video server; different URLs are set for the external audio source and the audio and video source respectively; when playback starts, the playback control module determines whether it is a source that requires an external audio, and according to whether the external audio source is required, the parsing module, the audio and video synchronization module and the rendering and playback module make corresponding processing.

[0032] In one embodiment of the present invention, if the external audio is a raw audio stream, without file layer encapsulation, the main information of the entire external audio, such as the total length of the external audio, etc., cannot be accurately obtained when the audio and video start playing, and problems will arise when synchronizing with the video. If the external audio itself has been encapsulated, it can be repackaged into a mode supported by itself or directly uploaded to the audio and video server. For example, the external audio of Dolby Atmos can be encapsulated in mp4 format and uploaded to the audio and video server after encapsulation.

[0033] In one embodiment of the present invention, according to whether an external audio source is required, the parsing module, the audio and video synchronization module, the decoding module and the rendering and playback module make corresponding processing, including: if the external audio source is not required, directly request the audio and video source URL from the audio and video server, separate the audio stream and the video stream after decapsulation, and then send the audio stream and the video stream to the corresponding audio and video decoders for decoding respectively. After decoding, the audio data and the video data are synchronized with the timestamps through the audio and video synchronization module, and then rendered and played through the rendering and playback module after synchronization; if this is an HLS audio and video source, the URL is an m3u8 description file, and the m3u8 file records the audio and video slice address and other information, then the ts format audio and video data of the corresponding slice is obtained through the slice address URL, and the ts file is first decapsulated.

[0034] If the source of the plug-in audio is needed, the URL of the audio and video source and the URL of the plug-in audio are requested from the audio and video server respectively, and then parsed by the parsing module; after obtaining the URL of the audio and video source and the plug-in audio, the parsing module first decapsulates the audio and video source and the plug-in audio respectively; for the audio and video source, first decapsulate and separate the audio and video in the corresponding slice, take the video data therein for parsing, and send it to the audio and video synchronization module; for the plug-in audio, it is parsed and processed according to the encapsulation format of the plug-in audio, separate the audio, and send it to the audio and video synchronization module; take the video data therein for parsing and then decode it separately, send it to the audio and video synchronization module for synchronization after decoding, and then render and play it through the rendering and playback module. For the plug-in audio, it is parsed and processed according to the encapsulation format of the plug-in audio. If it is mp4 encapsulated, first parse the mp4 file header to obtain the important information of the audio, and then separate the audio frame; if it is a slice mode, it is similar to the m3u8 description file of HLS, parse the different addresses and information of the slice, separate the audio, and send it to the audio and video synchronization module. It is equivalent to the audio and video coming from two different streams, obtaining video data from the audio and video source, and decapsulating and obtaining audio data from the external audio. In a specific application example, the audio and video source is an m3u8 file, which uses ts slicing and contains its own audio and video; the external audio is Dolby Atmos, which is encapsulated using mp4. First, the audio and video streams in m3u8 are parsed, and the video and audio are separated. The video is taken out for audio and video synchronization; the external audio is first decapsulated to obtain the mp4 file header, and the total audio length, sampling rate and other important information are obtained from the file header information for the next step of synchronization.

[0035] Through the external audio processing device set up as above, the synchronization of two audio and video sources with different URLs and the external audio is realized. The audio and video sources are upgraded by means of the external audio. When there is a new audio demand, only the external audio is needed. When playing, the original video URL plus the external audio URL are parsed and the audio and video are played synchronously, which greatly saves the production and transcoding of a huge amount of existing audio and video, as well as redundant bandwidth and storage expenses.

[0036] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A plug-in audio processing method, characterized in that: The following steps are involved: According to the format of the external audio, upload the external audio directly or after packaging to the audio and video server; set different URLs for the external audio source and the audio and video source respectively; When playback starts, determine whether it is a source that requires external audio, and make corresponding processing based on whether it is a source that requires external audio.

2. The external audio processing method according to claim 1, characterized in that: Corresponding processing is performed according to whether an external audio source is needed, including: if an external audio source is not needed, the audio and video source URL is directly requested from the audio and video server, and after decapsulation, the audio stream and video stream are separated, and then the audio stream and video stream are sent to the corresponding audio and video decoders for decoding respectively. After decoding is completed, the audio data and video data are synchronized with the timestamps. After synchronization, they are rendered and played through the rendering and playback module.

3. The external audio processing method according to claim 1, characterized in that: Corresponding processing is performed according to whether the source of the external audio is needed, including: if the source of the external audio is needed, requesting the URL of the audio and video source and the URL of the external audio from the audio and video server respectively, and then parsing them; after obtaining the URLs of the audio and video source and the external audio, first decapsulating the audio and video source and the external audio respectively; for the audio and video source, first decapsulate and separate the audio and video in the corresponding slices, and take the video data therein for parsing; for the external audio, parse and process it according to the encapsulation format of the external audio to separate the audio; the parsed audio and video are decoded respectively, and then synchronized before rendering and playing.

4. The external audio processing method according to claim 3, characterized in that: The parsing process is performed according to the packaging format of the plug-in audio. For example, if the mp4 format is used for packaging, the mp4 file header is first parsed to obtain the audio information, and then the audio frames are separated.

5. The external audio processing method according to claim 3, characterized in that: The analysis and processing according to the encapsulation format of the plug-in audio includes: if a slice mode is adopted, different addresses and information of the slices are parsed to separate the audio.

6. An external audio processing device, characterized in that: It includes a playback control module, a transpackaging module, an audio and video server, a parsing module, an audio and video synchronization module, and a rendering and playback module; According to the format of the external audio, the external audio is directly or input into the transpackaging module for packaging and then uploaded to the audio and video server; different URLs are set for the external audio source and the audio and video source respectively; When playback starts, the playback control module determines whether it is a source that requires external audio. Depending on whether the source requires external audio, the parsing module, the audio and video synchronization module, and the rendering and playback module make corresponding processing.

7. The external audio processing device according to claim 6, characterized in that: Depending on whether an external audio source is needed, the parsing module, the audio and video synchronization module and the rendering and playback module make corresponding processing, including: if the external audio source is not needed, directly request the audio and video source URL from the audio and video server, decapsulate, separate the audio stream and video stream, and then send the audio stream and video stream to the corresponding audio and video decoders for decoding. After decoding, the audio data and video data are synchronized with the timestamps through the audio and video synchronization module. After synchronization, they are rendered and played through the rendering and playback module.

8. The external audio processing device according to claim 6, characterized in that: Depending on whether the source of the external audio is needed, the parsing module, the audio and video synchronization module and the rendering and playback module make corresponding processing, including: if the source of the external audio is needed, the URL of the audio and video source and the URL of the external audio are requested from the audio and video server respectively, and then parsed by the parsing module; after obtaining the URLs of the audio and video source and the external audio, the parsing module first decapsulates the audio and video source and the external audio respectively; for the audio and video source, first decapsulate and separate the audio and video in the corresponding slices, take the video data therein for parsing and then decode them respectively, and send them to the audio and video synchronization module for synchronization after decoding; for the external audio, it is parsed according to the encapsulation format of the external audio, the audio is separated and sent to the audio and video synchronization module; after the parsed audio and video are given to the audio and video synchronization module, after synchronization, they are rendered and played through the rendering and playback module.

9. The external audio processing device according to claim 8, characterized in that: The parsing process is performed according to the packaging format of the plug-in audio. For example, if the mp4 format is used for packaging, the mp4 file header is first parsed to obtain the audio information, and then the audio frames are separated.

10. The external audio processing device according to claim 8, characterized in that: The analysis and processing according to the encapsulation format of the plug-in audio includes: if a slice mode is adopted, different addresses and information of the slices are parsed to separate the audio.

Images