System for synchronously playing HDR Vivid video and Audio Vivid (AV3A) audio

Abstract

The invention relates to a system for synchronously playing an HDR Vivid video and an Audio Vivid (AV3A) audio, and belongs to the technical field of playing systems. The system architecture sequentially comprises an application layer and a core layer according to levels, and is divided into two paths of a video renderer and an audio renderer after passing through the core layer; the video renderer further comprises hardware decoding and displaying in sequence; the audio renderer further comprises software decoding and outputting in sequence; the working stage of the system comprises the following steps: 1) pre-checking a file; (2), an ExoPlayer is initialized; 3) analyzing the media source; 4) selecting and initializing a renderer; according to the technical scheme, the audio and video synchronous playing is achieved, the audio and video synchronous playing precision is greatly improved, the video decoding CPU occupation, the audio decoding CPU occupation and the total CPU occupation are reduced, the playing fluency can achieve the effect of no frame loss, the audio output delay is within 50-100 ms, and the starting time can reach 1-2 seconds.

Description

Technical Field

[0001] This invention relates to a system for synchronous playback of HDR Vivid video and Audio Vivid (AV3A) audio, belonging to the field of playback system technology. Background Technology

[0002] With the rapid development of ultra-high-definition audio-visual technology, users' demand for immersive audio-visual experiences is increasing. The combined application of the HDR Vivid video standard and the Audio Vivid (AV3A) audio standard has become a core requirement for scenarios such as home theaters, VR / AR devices, and professional audio-visual playback terminals. HDR Vivid video, with its advantages of wide dynamic range and high color fidelity, can present more realistic visual effects; Audio Vivid (AV3A) audio, through three-dimensional spatial audio technology, achieves immersive sound field envelopment. Synchronized playback of both is a key prerequisite for ensuring a good audio-visual experience.

[0003] While various audio-visual synchronization playback systems exist on the market, they still suffer from numerous technical shortcomings when adapting to the high-specification requirements of HDR Vivid video and Audio Vivid (AV3A) audio. First, existing systems rely heavily on complex timestamp comparison and dynamic calibration algorithms for synchronization, simultaneously handling audio-visual decoding, data transmission, and synchronization calibration. This results in high CPU usage, leading to overheating, shortened battery life, and impacting the multitasking capabilities of terminal devices, particularly in resource-constrained scenarios like mobile devices and embedded systems. Second, synchronization accuracy is insufficient. Traditional synchronization methods are susceptible to hardware performance fluctuations, decoding latency, and network jitter, making it difficult to achieve microsecond-level synchronization control. Millisecond-level deviations frequently occur between audio and video, compromising the integrity of the immersive experience. Furthermore, the massive data volume of high-definition audio and video, coupled with the lack of optimization in resource scheduling for decoding and synchronization processing in existing systems, easily leads to data transmission congestion and frame loss, resulting in playback stuttering and reduced smoothness, failing to fully leverage the technological advantages of HDR Vivid and Audio Vivid (AV3A).

[0004] The shortcomings of the aforementioned existing technologies severely limit the widespread application of HDR Vivid video and Audio Vivid (AV3A) audio combination solutions, failing to meet users' demands for high-quality, low-resource-consumption, and high-precision synchronized audio-visual experiences. Therefore, developing a synchronized playback system for HDR Vivid video and Audio Vivid (AV3A) audio with low CPU usage, high synchronization accuracy, and excellent playback smoothness has become an urgent technical problem to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a system for synchronous playback of HDR Vivid video and Audio Vivid (AV3A) audio to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A system for synchronized playback of HDR Vivid video and Audio Vivid (AV3A) audio, with the following components selected: Select ExoPlayer 2.19 as the playback engine; Choose MediaCodec for video decoding; Select FFmpeg for audio decoding; Select ALSA for audio output; Demultiplexing ExoPlayer Mp4Extractor; The system architecture consists of an application layer and a core layer, followed by two paths: a video renderer and an audio renderer. The video renderer further includes hardware decoding and display, while the audio renderer further includes software decoding and output. The system operates in the following stages: 1) Document pre-check Check if the file exists, use FFmpeg to detect the codec, and verify HDR Vivid and AV3A formats; 2) ExoPlayer initialization Create a CustomRenderersFactory; Register Simple12chAudioRenderer; Register the default video renderer MediaCodec; Configure Mp4Extractor to recognize av3a atom; 3) Media Source Analysis Mp4Extractor reads MP4 containers; Parse the atoms moov / trak / mdia / minf; Identify video track: video / hevc (HDR Vivid); Identify audio track: audio / av3a (Audio Vivid); Extract initialization data (extradata); 4) Renderer selection and initialization ExoPlayer selects the renderer based on the MIME Type; Video MediaCodecVideoRenderer; Audio Simple12chAudioRenderer; Initialize the decoder and output device; 5) Synchronized playback Simple12chAudioRenderer provides MediaClock (master clock); The video renderer adjusts the timing of frame display based on the audio clock; ExoPlayer coordinates the two renderers to keep them synchronized; 6) Data processing loop Video, Mp4Extractor→VideoRenderer→MediaCodec→Surface Display Audio output via Mp4Extractor→AudioRenderer→FFmpeg→ALSA; 7) Playback ends EOS (End of Stream) detected; The renderer flag isEnded() = true; Stop ExoPlayer from looping; Clean up resources.

[0007] A further improvement to the technical solution of this invention is as follows: The HDR Vivid video processing flow is as follows: 1) Mp4Extractor parses video tracks; 2) Constructing the Format object; 3) Renderer selection; 4) MediaCodec initialization; 5) Decoding loop; 6) Synchronization control.

[0008] A further improvement to the technical solution of this invention is as follows: The Audio Vivid (AV3A) audio processing flow is as follows: 1) Mp4Extractor parses audio tracks; 2) Constructing the Format object; 3) Renderer selection; 4) FFmpeg decoder initialization; 5) ALSA output initialization; 6) Decoding loop; 7) MediaClock provides a report of the current playback position to ExoPlayer for synchronization.

[0009] A further improvement to the technical solution of this invention is that the system operation process uses the ExoPlayer demultiplexing mechanism, as follows: 1) Launch ExoPlayer; 2) Create a MediaSource; 3) Mp4Extractor initialization; 4) Traverse trak and extract track information; 5) Parse the STSD (Sample Description), including video and audio; 6) Construct a Format object; 7) Notify ExoPlayer; 8) Read the mdat file and extract the sample; 9) The renderer receives data.

[0010] A further improvement to the technical solution of this invention is as follows: This system uses the ExoPlayer synchronization architecture, and the core of the ExoPlayer synchronization mechanism is as follows: 1) The master clock, MediaClock, is provided by Simple12chAudioRenderer; 2) ExoPlayerImplInternal playback loop thread; 3) Video rendering: Simultaneously check video frames and then render and output; Audio rendering involves synchronously checking audio packets before decoding and outputting them.

[0011] A further improvement to the technical solution of the present invention is that the system also includes performance optimization measures, as detailed below. 1) Video Decoding Use MediaCodec for hardware decoding; Configure the Surface appropriately for hardware rendering; Enable asynchronous mode; 2) Video Decoding FFmpeg compilation optimization, NEON instruction set; Use float32 to int16 for efficient conversion; Avoid unnecessary memory copies; 3) ALSA Output Configure an appropriate buffer Use the RW_INTERLEAVED access mode; Handling underrun automatic recovery; 4) Synchronization Mechanism MediaClock is based on System.nanoTime() for high precision. Simplify implementation and avoid complex calculations; Keep the original PTS value and do not make any corrections.

[0012] Due to the adoption of the above technical solution, the technical effects achieved by this invention are as follows: This technical solution achieves synchronized audio and video playback, significantly improving audio-visual synchronization accuracy; it also reduces CPU usage for video decoding, audio decoding, and overall CPU usage. In use, playback is smooth with no dropped frames. Audio output latency is within 50-100ms. Startup time is 1-2 seconds. Attached Figure Description

[0013] Figure 1 This is a hierarchical diagram of the system architecture of this invention; Figure 2 This is a diagram of the video processing architecture of the present invention; Figure 3 This is a diagram of the audio processing architecture of the present invention; Figure 4 This is a flowchart of the demultiplexing process of the present invention; Figure 5 This is a diagram of the ExoPlayer synchronization architecture of the present invention. Detailed Implementation

[0014] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0015] This technical solution is a system for synchronous playback of HDR Vivid video and Audio Vivid (AV3A) audio. By designing a completely new system architecture, it achieves synchronous playback of video and audio, enabling high-precision and high-smoothness playback.

[0016] Referring to the attached diagrams in the instruction manual Figure 1-5 This section provides a detailed explanation of the technical solution.

[0017] The system architecture of this technical solution includes, in order, the application layer and the core layer. After the core layer, it is divided into two paths: video renderer and audio renderer.

[0018] Furthermore, the video renderer further includes hardware decoding and display. The audio renderer further includes software decoding and output.

[0019] The system operates in the following stages: 1) Document pre-check Check if the file exists, use FFmpeg to detect the codec, and verify HDR Vivid and AV3A formats; 2) ExoPlayer initialization Create a CustomRenderersFactory; Register Simple12chAudioRenderer; Register the default video renderer MediaCodec; Configure Mp4Extractor to recognize av3a atom.

[0020] 3) Media Source Analysis Mp4Extractor reads MP4 containers; Parse the atoms moov / trak / mdia / minf; Identify video track: video / hevc (HDR Vivid); Identify audio track: audio / av3a (Audio Vivid); Extract initialization data (extradata).

[0021] 4) Renderer selection and initialization ExoPlayer selects the renderer based on the MIME Type; Video MediaCodecVideoRenderer; Audio Simple12chAudioRenderer; Initialize the decoder and output device.

[0022] 5) Synchronized playback Simple12chAudioRenderer provides MediaClock (master clock); The video renderer adjusts the timing of frame display based on the audio clock; ExoPlayer coordinates the two renderers to keep them synchronized.

[0023] 6) Data processing loop Video, Mp4Extractor→VideoRenderer→MediaCodec→Surface Display Audio output via Mp4Extractor→AudioRenderer→FFmpeg→ALSA.

[0024] 7) Playback ends EOS (End of Stream) detected; The renderer flag isEnded() = true; Stop ExoPlayer from looping; Clean up resources.

[0025] A further improvement to the technical solution of this invention is as follows: The HDR Vivid video processing flow is as follows: 1) Mp4Extractor parses video tracks; 2) Constructing the Format object; 3) Renderer selection; 4) MediaCodec initialization; 5) Decoding loop; 6) Synchronization control.

[0026] A further improvement to the technical solution of this invention is as follows: The Audio Vivid (AV3A) audio processing flow is as follows: 1) Mp4Extractor parses audio tracks; 2) Constructing the Format object; 3) Renderer selection; 4) FFmpeg decoder initialization; 5) ALSA output initialization; 6) Decoding loop; 7) MediaClock provides a report of the current playback position to ExoPlayer for synchronization.

[0027] A further improvement to the technical solution of this invention is that the system operation process uses the ExoPlayer demultiplexing mechanism, as follows: 1) Launch ExoPlayer; 2) Create a MediaSource; 3) Mp4Extractor initialization; 4) Traverse trak and extract track information; 5) Parse the STSD (Sample Description), including video and audio; 6) Construct a Format object; 7) Notify ExoPlayer; 8) Read the mdat file and extract the sample; 9) The renderer receives data.

[0028] A further improvement to the technical solution of this invention is as follows: This system uses the ExoPlayer synchronization architecture, and the core of the ExoPlayer synchronization mechanism is as follows: 1) The master clock, MediaClock, is provided by Simple12chAudioRenderer; 2) ExoPlayerImplInternal playback loop thread; 3) Video rendering: Simultaneously check video frames and then render and output; Audio rendering involves synchronously checking audio packets before decoding and outputting them.

[0029] In a specific embodiment, the components are selected as follows: The playback engine selected is ExoPlayer 2.19; it has a flexible architecture and supports custom renderers.

[0030] Video decoding is performed using MediaCodec (hardware); it supports HEVC, HDR, and Vivid, offering high performance. High efficiency and low power consumption.

[0031] For audio decoding, choose FFmpeg (software); Android MediaCodec does not support AV3A, while FFmpeg has full support.

[0032] Select ALSA for audio output; directly access Card 2 (12ch XLR) to bypass Android AudioTrack restrictions.

[0033] Demultiplexing ExoPlayer Mp4Extractor; supports recognizing 'av3a' atom and extracting HDR Vivid metadata.

[0034] The system mainly includes the following modules: Activity's functions include UI interaction and player lifecycle management.

[0035] The renderer factory is used to create custom renderers.

[0036] The audio renderer's functions include AV3A decoding, ALSA output, and clock provision.

[0037] Codec Assist, its function is to detect FFmpeg codecs.

[0038] The AV3A decoder is a software decoder for FFmpeg.

[0039] The ALSA output is a 12-channel PCM output.

[0040] Mp4Extractor is used to identify av3a atom.

[0041] The innovative features of this technical solution are as follows: 1. ExoPlayer + FFmpeg hybrid architecture Innovation points: Video: MediaCodec hardware decoding (standard path).

[0042] Audio: FFmpeg software decoding (custom path).

[0043] Synchronization: Unified coordination by ExoPlayer.

[0044] Advantages: Make full use of hardware acceleration (video).

[0045] Supports non-standard formats (AV3A).

[0046] A unified playback control and synchronization mechanism.

[0047] 2. Modification of Mp4Extractor source code Innovation points: Add av3a Atom recognition.

[0048] Custom MIME type audio / av3a.

[0049] Extract AudioSpecificConfig.

[0050] Advantages: The format support for ExoPlayer has been expanded.

[0051] Maintain compatibility with the ExoPlayer architecture.

[0052] It can be used as a template for other format extensions.

[0053] 3. Simple12chAudioRenderer Design Innovation points: A concise yet complete implementation of BaseRenderer.

[0054] The correct MediaClock is provided.

[0055] Critical fix for ByteBuffer.flip().

[0056] Complete stream lifecycle management.

[0057] Advantages: Single responsibility: Focus on AV3A + 12ch.

[0058] The code is clear, making it easy to maintain and extend.

[0059] Improved logs: Facilitate problem diagnosis.

[0060] 4. ALSA direct output Innovation points: Bypass Android AudioTrack restrictions.

[0061] Direct access to the hardware sound card (Card 2).

[0062] Supports 12-channel XLR output.

[0063] Advantages: Low latency (50-100ms vs AudioTrack 200-300ms).

[0064] Full multi-channel support.

[0065] Precise hardware control.

[0066] 5. Timestamp consistency handling Innovation points: Maintain the MP4 timecode offset.

[0067] MediaClock is based on the first frame PTS.

[0068] Do not modify the original timestamp.

[0069] The key performance indicators achieved by the playback system in this technical solution are as follows: The measured audio-visual synchronization accuracy is <100ms.

[0070] Video decoding CPU usage is 5-15%.

[0071] The CPU usage for audio decoding is 20-25%.

[0072] Total CPU usage is 25-40%.

[0073] Memory usage is 300-400MB.

[0074] The actual playback smoothness was measured at a stable 25fps.

[0075] The measured audio output delay is 50-100ms.

[0076] Startup time is 1-2 seconds.

[0077] The system also includes performance optimization measures, as detailed below. 1) Video Decoding Use MediaCodec for hardware decoding; Configure the Surface appropriately for hardware rendering; Enable asynchronous mode; 2) Video Decoding FFmpeg compilation optimization, NEON instruction set; Use float32 to int16 for efficient conversion; Avoid unnecessary memory copies; 3) ALSA Output Configure an appropriate buffer Use the RW_INTERLEAVED access mode; Handling underrun automatic recovery; 4) Synchronization Mechanism MediaClock is based on System.nanoTime() for high precision. Simplify implementation and avoid complex calculations; Keep the original PTS value and do not make any corrections.

[0078] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A system for synchronized playback of HDR Vivid video and Audio Vivid (AV3A) audio, characterized in that: The components are selected as follows: Select ExoPlayer 2.19 as the playback engine; Choose MediaCodec for video decoding; Select FFmpeg for audio decoding; Select ALSA for audio output; Demultiplexing ExoPlayer Mp4Extractor; The system architecture consists of an application layer and a core layer, followed by two paths: a video renderer and an audio renderer. The video renderer further includes hardware decoding and display, while the audio renderer further includes software decoding and output. The system operates in the following stages: 1) Document pre-check Check if the file exists, use FFmpeg to detect the codec, and verify HDR Vivid and AV3A formats; 2) ExoPlayer initialization Create a CustomRenderersFactory; Register Simple12chAudioRenderer; Register the default video renderer MediaCodec; Configure Mp4Extractor to recognize av3a atom; 3) Media Source Analysis Mp4Extractor reads MP4 containers; Parse the atoms moov / trak / mdia / minf; Identify video track: video / hevc (HDR Vivid); Identify audio track: audio / av3a (Audio Vivid); Extract initialization data (extradata); 4) Renderer selection and initialization ExoPlayer selects the renderer based on the MIME Type; Video MediaCodecVideoRenderer; Simple12chAudioRenderer; Initialize the decoder and output device; 5) Synchronized playback Simple12chAudioRenderer provides MediaClock (master clock); The video renderer adjusts the timing of frame display based on the audio clock; ExoPlayer coordinates the two renderers to keep them synchronized; 6) Data processing loop Video, Mp4Extractor→VideoRenderer→MediaCodec→Surface Display Audio output via Mp4Extractor→AudioRenderer→FFmpeg→ALSA; 7) Playback ends EOS (End of Stream) detected; The renderer flag isEnded() = true; Stop ExoPlayer from looping; Clean up resources.

2. The system for synchronized playback of HDR Vivid video and Audio Vivid (AV3A) audio according to claim 1, characterized in that: The HDR Vivid video processing workflow is as follows: 1) Mp4Extractor parses video tracks; 2) Constructing the Format object; 3) Renderer selection; 4) MediaCodec initialization; 5) Decoding loop; 6) Synchronization control.

3. The system for synchronized playback of HDR Vivid video and Audio Vivid (AV3A) audio according to claim 1, characterized in that: The Audio Vivid (AV3A) audio processing workflow is as follows: 1) Mp4Extractor parses audio tracks; 2) Constructing the Format object; 3) Renderer selection; 4) FFmpeg decoder initialization; 5) ALSA output initialization; 6) Decoding loop; 7) MediaClock provides a report of the current playback position to ExoPlayer for synchronization.

4. The system for synchronized playback of HDR Vivid video and Audio Vivid (AV3A) audio according to claim 1, characterized in that: The system uses the ExoPlayer demultiplexing mechanism in its operation, as follows: 1) Launch ExoPlayer; 2) Create a MediaSource; 3) Initialize Mp4Extractor; 4) Traverse trak and extract track information; 5) Parse the STSD (Sample Description), including video and audio; 6) Construct a Format object; 7) Notify ExoPlayer; 8) Read the mdat file and extract the sample; 9) The renderer receives data.

5. A system for synchronized playback of HDR Vivid video and Audio Vivid (AV3A) audio according to claim 1, characterized in that: This system uses the ExoPlayer synchronization architecture. The core of the ExoPlayer synchronization mechanism is as follows: 1) The master clock, MediaClock, is provided by Simple12chAudioRenderer; 2) ExoPlayerImplInternal playback loop thread; 3) Video rendering: Simultaneously check video frames and then render and output; Audio rendering involves synchronously checking audio packets before decoding and outputting them.

6. A system for synchronized playback of HDR Vivid video and Audio Vivid (AV3A) audio according to claim 1, characterized in that: The system also includes performance optimization measures, as detailed below. 1) Video Decoding Use MediaCodec for hardware decoding; Configure the Surface appropriately for hardware rendering; Enable asynchronous mode; 2) Video Decoding FFmpeg compilation optimization, NEON instruction set; Use float32 to int16 for efficient conversion; Avoid unnecessary memory copies; 3) ALSA Output Configure an appropriate buffer Use the RW_INTERLEAVED access mode; Handling underrun automatic recovery; 4) Synchronization Mechanism MediaClock is based on System.nanoTime() for high precision. Simplify implementation and avoid complex calculations; Keep the original PTS value and do not make any corrections.

Images