Low-latency decoding optimization method and system for android real-time video streaming

Abstract

The application provides a low-delay decoding optimization method and system for Android real-time video streaming. The method includes: receiving video encoding data from the cloud; identifying the chip platform type of the current Android terminal; selecting the corresponding low-delay decoding parameter combination from the preset configuration strategy set according to the chip platform type, and initializing and configuring the MediaCodec decoder of the Android system based on the parameter combination; inputting the video encoding data into the configured MediaCodec decoder for hardware decoding, and directly outputting the decoded video frame to the Surface; and controlling the display timing of the video frame output to the Surface to reduce the waiting time of the video frame in the display queue. The application reduces the end-to-end delay from decoding to display of video data through differentiated optimization, shortens the processing path and active timing control, and improves the real-time interaction experience.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a low-latency decoding optimization method and system for Android real-time video streams. Background Technology

[0002] Cloud gaming technology places game rendering and computation on cloud servers, transmitting real-time video streams to terminal devices via the network. As a common platform for cloud gaming, the latency of video decoding and rendering on Android devices directly impacts the user experience.

[0003] In existing technologies, cloud gaming video decoding solutions on the Android platform often employ general video decoding frameworks, such as those based on the WebRTC framework combined with the Android system's MediaCodec hardware decoder. This solution typically outputs the decoded video frames to a Surface for display. However, this general approach is primarily geared towards local video playback or traditional streaming media scenarios, and its default configuration prioritizes playback stability and image quality over the extremely low latency required for scenarios like cloud gaming.

[0004] Specifically, existing technologies suffer from the following drawbacks: First, the default parameter configuration of the MediaCodec decoder is not optimized for low-latency scenarios. For example, its internal input and output buffers are numerous, resulting in excessively long queuing times for video frames during decoding. Second, different chip manufacturers (such as HiSilicon and Goke Microelectronics) have varying levels of support for the underlying implementation and proprietary parameters of MediaCodec. Existing technologies often employ a unified decoding configuration, failing to optimize for the hardware characteristics of specific chip platforms. This leads to the hardware decoder's low-latency capabilities not being effectively activated on some terminals. Finally, after the decoded video frames are sent to the Surface display queue, their display timing is typically determined by the system's default strategy. The application layer lacks effective control mechanisms, causing video frames to potentially experience unnecessary waiting before display, further increasing end-to-end display latency. Therefore, reducing video decoding and display latency on the Android platform in low-latency scenarios has become a pressing technical problem to be solved in this field. Summary of the Invention

[0005] The purpose of this application is to provide a low-latency decoding optimization method and system for Android real-time video streams, in order to solve the problems of high overall display latency caused by unoptimized decoder configuration, lack of differentiated adaptation to different chip platforms, and insufficient control over display timing in existing video decoding solutions.

[0006] To achieve the above objectives, this application provides a low-latency decoding optimization method for Android real-time video streams, comprising the following steps:

[0007] It receives and parses video encoded data from the cloud to generate decodeable video frame data; Identify the chip platform type of the current Android terminal; Based on the chip platform type, select the corresponding low-latency decoding parameter combination from the preset configuration strategy set, and initialize the MediaCodec decoder of the Android system based on the parameter combination; The video frame data is input to the configured MediaCodec decoder for hardware decoding, and the decoded video frames are directly output to the Surface; The display timing of video frames output to the Surface is controlled to reduce the waiting time of video frames in the display queue.

[0008] Optionally, identifying the chip platform type of the current Android terminal includes: Read system properties or decoder information, and determine the chip platform type based on the decoder name or the manufacturer identifier contained in the system properties.

[0009] Optionally, the low-latency decoding parameter combination includes at least one of the following: Low-latency decoding mode enable parameters; The parameter limiting the number of decoder input and output buffers; Parameters used to disable B-frame decoding, post-processing, or frame rate conversion. Parameters used to enable automatic frame dropping by hardware; This is for the proprietary low-latency parameters defined by the pre-defined chip manufacturer.

[0010] Optionally, controlling the display timing of video frames output to the Surface includes: During the decoding output stage, the expected display timestamp of the video frame is dynamically calculated based on the system clock; Release the MediaCodec output buffer using the method described above, in order to control when video frames enter the Surface display queue.

[0011] Optionally, it also includes: When a backlog of video frames or a display delay exceeding a threshold is detected, the control logic prioritizes displaying the latest decoded video frame and actively discards expired video frames.

[0012] Optionally, before inputting the video frame data to the configured MediaCodec decoder for hardware decoding, the method further includes: Dynamically enumerate the available hardware decoders in the Android system and the color formats they support; When initializing the MediaCodec decoder, the color format is matched and set according to the dynamically enumerated color format results to ensure that the hardware decoder is successfully enabled.

[0013] To achieve the above objectives, this application also provides a low-latency decoding optimization system for Android real-time video streams, used to implement the above method. The system includes: The data receiving module is used to receive video encoded data from the cloud; The chip identification module is used to identify the chip platform type of the current Android terminal; The decoding control module is used to select the corresponding low-latency decoding parameter combination according to the chip platform type, and to perform initialization, data input and decoding operations on the MediaCodec decoder based on the parameter combination; The Surface display control module is used to directly bind video frames output by the MediaCodec decoder to the Surface display and control the display timing of the video frames.

[0014] Optionally, the decoding control module includes a custom video decoder factory, which is configured as follows: When creating a decoder instance, actively enumerate and match the color formats supported by the hardware decoder in the system, and force the return of the MediaCodec-based hardware decoder implementation to avoid the decoding process falling back to software decoding.

[0015] Optionally, the system further includes a decoding management and reconstruction module, which is used for: Separate the control logic of the MediaCodec decoder from the output thread; In the event of decoding stoppage or an exception, the MediaCodec decoder resources are safely released via the output thread. It also supports the reconstruction and rebinding of decoders when the Surface changes.

[0016] Compared with the prior art, this application has the following beneficial effects: By identifying the Android terminal's chip platform type and selecting the corresponding low-latency decoding parameter combination to configure the MediaCodec decoder, differentiated optimizations can be performed for the characteristics of different hardware platforms, fully activating and utilizing the hardware's low-latency decoding capabilities. By directly outputting the decoded video frames to the Surface, data copying and intermediate rendering steps are reduced, shortening the processing path. Furthermore, by actively controlling the display timing of video frames, invalid waiting times in the display queue are reduced. These methods work synergistically to significantly reduce end-to-end latency from video data decoding to display, improving the user experience. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Furthermore, these drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments.

[0018] Figure 1 This is a flowchart illustrating a low-latency decoding optimization method for Android real-time video streams provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the structure of a low-latency decoding optimization system for Android real-time video streams provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0021] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0022] (1) Cloud gaming: refers to a technology that centralizes the graphics rendering and logic operations of a game on a cloud server and transmits the real-time generated video stream to a terminal device for display via a network. User commands on the terminal device are sent back to the cloud, which updates the game screen according to the commands and pushes new video frames to the terminal device in real time. This scenario has extremely high requirements for end-to-end latency in video transmission, decoding, and display. If the latency is too high, the user experience will be very poor.

[0023] (2) MediaCodec: refers to a set of low-level application programming interfaces (APIs) provided by the Android operating system, which allows applications to access the platform's underlying multimedia hardware codecs. Through MediaCodec, applications can implement hardware-accelerated decoding or encoding of video and audio data, and it is the core component for implementing low-latency video decoding in this embodiment.

[0024] (3) Surface: In the Android graphics system, Surface is an abstract object used to represent image data producers. The decoder can directly output the decoded video frames to its associated Surface, and then the system's graphics compositor (such as SurfaceFlinger) consumes these image frames and renders them on the screen. By directly outputting to Surface, data copying and intermediate processing steps can be reduced.

[0025] (4) Chip platform type: refers to the manufacturer or series of the core processor (SoC, System on a Chip) carried by the Android terminal device. Since different manufacturers (such as HiSilicon, Goke Microelectronics, Qualcomm, MediaTek, etc.) have different implementations of the hardware decoder driver and MediaCodec interface of the Android system, the chip platform type is a key factor in determining the decoding strategy.

[0026] (5) Low-latency decoding parameter combination: refers to a set of key-value pairs used to configure the MediaCodec decoder. These parameters are specifically designed to instruct the decoder to prioritize decoding speed over image quality or stability during operation, such as enabling low-latency mode, limiting the internal buffer size, and disabling post-processing, thereby minimizing the latency introduced internally by the decoder.

[0027] (6) Display timing: This refers to the process of controlling the timing and rhythm of a decoded video frame from its output by the decoder to its final presentation on the screen. Precise display timing control aims to reduce the waiting time of video frames in the system display queue (such as the buffer queue of a Surface) and ensure that the image can be displayed as quickly as possible.

[0028] Reference Figure 1This invention provides a low-latency decoding optimization method for Android real-time video streams, aiming to solve the problem of high latency in cloud video display caused by long decoding and rendering links, unoptimized decoder configuration, and poor chip platform compatibility in existing technologies. This method significantly reduces the end-to-end latency of video frames from cloud reception to terminal display by collaboratively optimizing multiple key stages of the decoding process, thereby improving the user's real-time interactive experience.

[0029] like Figure 1 As shown, this method includes several core steps: Step S1: Receive and parse the video encoding data from the cloud to generate decodable video frame data.

[0030] Taking cloud gaming as an example, the cloud server continuously runs the game program and encodes the game screen into a video stream in real time, such as an H.264 or H.265 format. The Android terminal device, acting as a client, receives these continuous video encoded data packets from the cloud server via a network connection (such as a real-time transmission protocol based on WebRTC). After acquiring the video encoded data, format parsing and frame boundary identification are performed, such as parsing the H.264 / H.265 encoding format; extracting keyframe information; identifying frame types and related parameters, etc. This step is the data input source for the entire decoding process, and its real-time and complete reception is the foundation for subsequent low-latency processing.

[0031] Next, the method executes step S2: identify the chip platform type of the current Android terminal.

[0032] Due to the fragmented hardware nature of the Android ecosystem, chips from different manufacturers exhibit significant differences in hardware decoder implementation, supported features, and responses to MediaCodec parameters. Therefore, before configuring the decoder, it is essential to determine the hardware platform used by the current device. This step, by analyzing internal system information, provides a basis for subsequent decision-making regarding the selection of an appropriate decoding strategy and is a prerequisite for achieving differentiated and refined optimization.

[0033] After identifying the chip platform type, the method proceeds to step S3: Based on the chip platform type, a corresponding low-latency decoding parameter combination is selected from a preset configuration strategy set, and the MediaCodec decoder of the Android system is initialized and configured based on this parameter combination. This embodiment pre-establishes a series of corresponding optimized parameter sets for various mainstream or specific chip platforms (such as HiSilicon chip platforms, Guoke Microelectronics chip platforms, etc.). Once step S2 determines the current platform type, the system can accurately match and load the low-latency configuration suitable for that hardware for subsequent MediaCodec instance creation and configuration processes.

[0034] Subsequently, the method executes step S4: inputting the video encoded data into the configured MediaCodec decoder for hardware decoding, and directly outputting the decoded video frames to the Surface.

[0035] In this step, the video encoded data units (NALUs) received and potentially parsed in step S1 are fed into the input buffer of the MediaCodec decoder, which has been optimized for the specific chip platform. The decoder utilizes hardware acceleration to efficiently decode the compressed data into raw video image frames. A key optimization is that the decoder's output is directly configured onto a Surface object. This means that the decoded frames do not need to be returned to the application's memory space; instead, they are directly managed by the system's underlying layer and sent to the display pipeline, significantly shortening the processing path.

[0036] Finally, the method executes step S5: controlling the display timing of video frames output to the Surface to reduce the waiting time of video frames in the display queue.

[0037] Although the decoded frames are directly output to the Surface, they still queue in the Surface's internal buffer queue, waiting to be displayed. To minimize this latency, this step introduces a set of proactive control logic. This logic does not passively allow the system to schedule the display itself, but rather actively influences the timing of when video frames enter the display queue and the final display time through specific API calls and timing calculations, ensuring that decoded frames can be displayed as quickly as possible and avoiding unnecessary latency introduced by queuing.

[0038] In summary, the method proposed in this embodiment forms a complete closed-loop optimization process. It starts by identifying hardware differences (S2), performs targeted decoder configuration (S3), adopts the most efficient decoding output path (S4), and performs fine-grained timing control on the final display stage (S5). These steps are interconnected and work synergistically to shorten the processing time of the video stream at each stage within the terminal device, thereby achieving the ultimate goal of reducing overall latency.

[0039] The principle of this invention lies in the fact that video decoding latency on the Android platform originates from multiple sources, including decoder internal caching, data copying overhead, and display scheduling latency. Therefore, instead of employing a single optimization method, it proposes a combined strategy. By identifying the chip platform and applying customized low-latency parameters, it directly addresses the issues of suboptimal decoder internal caching and processing strategies. By directly connecting the decoded output to the Surface, it eliminates unnecessary data copying and thread switching overhead in the traditional "decoding to memory to GPU rendering" path. Finally, by actively controlling the display timing, it further compresses the waiting time of video frames in the display pipeline. This end-to-end, end-to-end optimization approach enables this method to achieve a more significant latency reduction compared to existing technologies.

[0040] In a preferred embodiment, the specific implementation of identifying the chip platform type of the current Android terminal in step S2 is refined. Specifically, identifying the chip platform type of the current Android terminal includes: reading system attributes or decoder information, and determining the chip platform type based on the decoder name or the manufacturer identifier contained in the system attributes.

[0041] For example, the program can call Android system APIs to query specific system attribute strings, such as ro.hardware and ro.board.platform. These attribute values ​​typically contain identifiers of the chip manufacturer or model. Another, more reliable approach is to enumerate all available decoders in the system using the MediaCodecList class and examine the name of each MediaCodecInfo object. Hardware decoder names usually contain vendor identifiers; for example, HiSilicon platform decoder names might begin with "OMX.hisi," while Google software decoders might begin with "OMX.google." By matching these specific vendor identifiers in the names, the chip platform type can be accurately determined. This specific implementation provides reliable input for subsequent strategy selection, ensuring accurate adaptation.

[0042] The technical advantage of the aforementioned identification method lies in providing a concrete, feasible, and highly compatible means to dynamically perceive the device's hardware environment. Compared to hard-coding or relying on unstable features, by reading system attributes and decoder names provided by the official API, it can run stably on most Android devices and accurately distinguish target platforms requiring special optimization (such as HiSilicon and Guoke Micro). This avoids compatibility issues or performance degradation that may result from applying general optimization strategies to all devices, making it a key step in achieving adaptive optimization.

[0043] Furthermore, in another preferred embodiment, the specific details of the low-latency decoding parameter combination mentioned in step S3 are described. This parameter combination may include a set of at least one or more of the following parameters: First, there's the low-latency decoding mode enabling parameter, such as MediaFormat.KEY_LOW_LATENCY defined in the Android standard API. Setting it to 1 tells the decoder to enter low-latency mode.

[0044] Secondly, there are parameters limiting the number of decoder input and output buffers. Some chip manufacturers (such as HisiSilicon) provide proprietary parameter keys (such as hisi-dec-input-buffer-num) to allow the application layer to reduce the length of the internal buffer queue of the decoder, thereby reducing frame queuing latency.

[0045] The next parameter is used to disable B-frame decoding, post-processing, or frame rate conversion. For example, by setting private keys such as disable-b-frame or no-post-process, you can turn off these functions that can improve image quality but increase processing latency.

[0046] In addition, parameters such as drop-frame, which allow the hardware to automatically drop frames, can be included to allow the decoder to actively drop some frames to catch up with the real-time progress when processing power is insufficient.

[0047] Finally, it can also include a large number of proprietary low-latency parameters defined for the preset chip manufacturers, such as vendor.goke.low-latency and vendor.amlogic.decoder.low-latency. These parameters can activate special low-latency processing logic at the chip's underlying layer.

[0048] By combining these parameters and selectively applying them based on the chip platform, the technical effect is to achieve deep customization and optimization of decoder behavior. Compared to simply enabling the standard KEY_LOW_LATENCY, this embodiment, by utilizing the vendor's proprietary API, can more thoroughly reduce the decoder's internal buffering, disable unnecessary processing steps, and establish a more aggressive frame processing strategy. This multi-dimensional, fine-grained parameter configuration is the core of fully utilizing the low-latency potential of the hardware decoder and achieving significant low-latency effects. For example, on the HiSilicon platform, simultaneously setting a low-latency mode, reducing the number of buffers, and disabling B-frames results in a latency reduction effect far exceeding that of optimizing a single parameter.

[0049] In a preferred embodiment, the process preceding step S4 is supplemented. Before inputting the video encoded data to the configured MediaCodec decoder for hardware decoding, the process may further include: dynamically enumerating the available hardware decoders in the Android system and their supported color formats; and when initializing the MediaCodec decoder, matching and setting the color format based on the dynamically enumerated color format results to ensure that the hardware decoder is successfully enabled.

[0050] Specifically, the program iterates through all decoders using MediaCodecList, filters out the hardware decoders, and then, for each hardware decoder, calls the getCapabilitiesForType() method to obtain its CodecCapabilities object, reading the colorFormats array from it. This array contains integer values ​​for all color formats (ColorFormat) actually supported by that decoder. During subsequent MediaCodec initialization, the system matches and sets the color formats based on this dynamically obtained, valid list. For example, it might select a color format supported by both the decoder and the rendering pipeline to configure MediaFormat.

[0051] The technical advantage of this step is that it significantly increases the probability of the hardware decoder being successfully enabled, thereby ensuring the effective execution of the entire low-latency solution. In Android practice, a common failure reason is that the color format requested by the application does not match the format actually supported by the hardware decoder, causing the system to silently fall back to the high-latency software decoding path. By proactively enumerating and using the color formats actually supported by the hardware in advance, this embodiment ensures that the MediaCodec.configure() call succeeds and truly hands over the decoding task to the hardware, avoiding performance bottlenecks caused by color format issues and enhancing the stability and universality of the solution.

[0052] Furthermore, the specific method for controlling the display timing of video frames output to the Surface in step S5 is described in detail. This step includes: during the decoding output stage, dynamically calculating the expected display timestamp of the video frame based on the system clock; and releasing the MediaCodec output buffer using the expected display timestamp to control the timing of video frames entering the Surface display queue.

[0053] Specifically, the control process mainly consists of two steps: First, during the decoding output stage, when the program obtains the index of a decoded output buffer from MediaCodec, it dynamically calculates the expected display timestamp of the video frame based on the system clock. For example, the current time can be obtained using SystemClock.elapsedRealtime(), and a reasonable future display time can be calculated based on a certain strategy (such as considering frame rate and network jitter). Second, when releasing the output buffer to submit it to the Surface for display, the MediaCodec.releaseOutputBuffer(int index, long renderTimestampNanos) method with a timestamp parameter is called, instead of the parameterless version. By passing the expected display timestamp (converted to nanoseconds) calculated in the previous step as the renderTimestampNanos parameter, the application can explicitly indicate the target display time of the frame to the system graphics stack (SurfaceFlinger).

[0054] The advantage of this proactive display timing control technology lies in shifting the scheduling of frame display from the system's default, potentially buffered mechanism to the application. By providing a nanosecond-precise "deadline" for each frame, it effectively prevents video frames from waiting haphazardly in the Surface's buffer queue, reducing unnecessary waiting. This allows video frames to be displayed more closely in sync with the real-time clock, significantly reducing the "last mile" latency between decoding completion and screen activation, which is crucial for scenarios requiring instant feedback, such as cloud gaming.

[0055] Based on the above embodiments for controlling display timing, a more preferred implementation is to introduce a dynamic frame dropping strategy. The method may further include: when video frame backlog or display delay exceeds a threshold, the control logic prioritizes displaying the latest decoded video frame and actively discards expired video frames.

[0056] Specifically, the control logic continuously monitors the backlog of video frames or display latency. For example, it can record the decoding completion timestamp of each frame and compare it with the current system time. When this time difference, i.e., the display latency, is detected to exceed a preset threshold (e.g., 30 milliseconds), or when too many frames are waiting to be displayed in the Surface queue, the system triggers frame dropping logic. This logic prioritizes displaying the latest decoded video frame and actively discards those "expired" old video frames that are backlogged in the decoder output buffer. In implementation, this can be achieved by repeatedly calling `dequeueOutputBuffer` and immediately releasing the old buffer until the latest frame is obtained before rendering with a timestamp.

[0057] The effectiveness of this technology lies in providing the system with a "self-healing" capability to cope with network fluctuations or performance jitter. In real-time scenarios such as cloud gaming, maintaining synchronization with the latest server feed is more important than displaying every frame completely. When latency accumulates, it's better to skip an outdated frame and directly display the latest one than to show an outdated one. This strategy effectively breaks the vicious cycle of latency accumulation, quickly restoring latency to a lower level even under poor network conditions or high terminal load, ensuring real-time feedback for user operations and greatly improving the user experience in weak network or complex environments.

[0058] This application also provides a low-latency decoding optimization system for Android real-time video streams, which is used to implement the methods described in any of the foregoing embodiments. (Refer to...) Figure 2 In a specific structure, the system may include: a data receiving module 10, a video data parsing module 20, a chip identification module 30, a decoding control module 40, and a Surface display control module 50.

[0059] Specifically, the data receiving module 10 has a function corresponding to step S1 in the aforementioned method, and is responsible for receiving video encoded data streams from the cloud server via the network.

[0060] The video data parsing module 20 is responsible for parsing the received raw data (video encoded data) and extracting video frame data that can be processed by the decoder, such as the NALU unit of H.264, in preparation for subsequent decoding.

[0061] The chip identification module 30, whose function corresponds to step S2, is responsible for identifying the hardware chip platform type of the current terminal.

[0062] The decoding control module 40, whose functions correspond to S3 and S4, matches an optimal set of low-latency decoding parameters for the current device from a preset strategy library based on the identification results of the chip identification module 30, and uses this configuration to initialize and control a MediaCodec instance. Simultaneously, it is also responsible for sending video frame data from the video data parsing module 20 into the MediaCodec for hardware decoding.

[0063] The Surface display control module 50, whose function corresponds to the output part of step S4 and step S5, directly outputs the video frames decoded by the decoding control module 40 to a pre-bound Surface and implements a set of fine-grained timing control logic to manage the display rhythm of the decoded frames in order to minimize the display waiting time.

[0064] During system operation, these modules work collaboratively. The data receiving module 10 continuously receives data, which is then processed by the video data parsing module 20 before flowing to the decoding control module 40. Furthermore, the decoding control module 40, based on the recognition results from the chip recognition module 30, configures the decoder accordingly and uses the configured decoder to decode and output the video frame data from the video data parsing module 20. After decoding, the Surface display control module 50 takes over, ensuring that these frames are presented to the user with minimal latency. The entire system design embodies the end-to-end, collaborative optimization philosophy consistent with the aforementioned methods.

[0065] In a preferred system implementation, the decoding control module 40 internally includes a custom video decoder factory. This factory is specifically configured to execute the color format matching logic described in the foregoing method embodiments when creating a decoder instance. That is, it actively enumerates all hardware decoders in the system and obtains a list of color formats they each support. When WebRTC or other frameworks request the creation of a decoder, the factory uses this information to select a guaranteed compatible color format and forces the return of a MediaCodec-based hardware decoder implementation. This design ensures that the decoding process will not unexpectedly fall back to the software decoding path due to color format mismatch or other issues.

[0066] The technical advantage of this custom video decoder factory lies in its ability to block a major entry point for hardware decoding failure at the architectural level. By firmly controlling the hardware capability detection and decoder instantiation process, the system avoids dependence on the default behavior of upper-layer frameworks (such as WebRTC), thereby significantly improving the success rate of stably enabling hardware decoding on various complex Android devices. This embodies the institutionalization and modularization of optimization strategies, enhancing the system's robustness.

[0067] In another preferred system implementation, the system further includes a decoding management and reconstruction module. This module is designed to improve the stability and robustness of the decoding process. Its specific functions include: First, separating the control logic of the MediaCodec decoder (such as calling configure, start, etc.) from the output thread of the decoded data (i.e., the thread polling dequeueOutputBuffer). This separation effectively avoids thread deadlocks in complex asynchronous operations. Second, when decoding stops or an unrecoverable exception occurs, the independent output thread safely performs resource release operations such as stop and release, ensuring that MediaCodec resources are cleanly reclaimed. Finally, this module also supports rebuilding and rebinding the decoder when the Surface changes (e.g., when the application switches from the background to the foreground and the Surface is rebuilt), enabling seamless recovery of video playback.

[0068] The technical advantage of this decoding management and reconstruction module lies in its ability to solve several technical challenges encountered in managing the MediaCodec lifecycle in practical engineering projects. The MediaCodec API has strict thread requirements and complex exception handling. By introducing such a dedicated management module, which encapsulates complex logic such as resource control, exception handling, and lifecycle management, the entire decoding system not only exhibits low latency but also demonstrates greater stability and reliability in the face of various runtime changes and exceptional situations, thereby improving the system's stability and reliability in real-world operating environments.

[0069] The following describes an embodiment that integrates the aforementioned preferred technical features to demonstrate the technical effects achieved when these features work together. This embodiment also embodies the core ideas of the aforementioned methods and systems.

[0070] In this embodiment, an Android application designed for cloud gaming is launched. First, the application creates a custom TYVideoDecoderFactory instance, a class that implements the aforementioned video decoder factory functionality. During its instantiation, it immediately calls MediaCodecList to scan all codecs on the device, filtering out all hardware decoders provided by third-party vendors (i.e., whose names do not begin with official identifiers such as 'OMX.google') that support H.264 / H.265 decoding, and records a list of color formats supported by each hardware decoder, forming a global view of hardware capabilities.

[0071] When a cloud gaming connection is established and a video decoder needs to be created, the application layer requests a decoder from the TYVideoDecoderFactory. The factory first identifies the current device's chip platform type based on the decoder name (e.g., containing "hisi" or "goke"). Next, it retrieves a specific set of low-latency decoding parameters from a pre-defined Map structure, tailored to the chip platform type. This set of parameters is quite comprehensive; for example, for the HiSilicon platform, it might include: {KEY_LOW_LATENCY:1,hisi-dec-input-buffer-num:3,hisi-dec-output-buffer-num:2,drop-frame:1} etc.

[0072] Next, the factory creates a custom TYAndroidVideoDecoder2 decoder instance. During the initialization process of this instance, it creates a MediaFormat object and sets all the low-latency parameters obtained in the previous step. Simultaneously, it uses the color format information collected during factory initialization to select the optimal color format for configuration, ensuring the hardware decoder can be successfully configured. Finally, it calls the MediaCodec.configure() method and directly binds the decoder output to a Surface object passed in from the application layer.

[0073] After the decoder starts, a separate decoding output thread begins working. Video data received by the network module is sent to the decoder's input buffer. The output thread continuously calls `MediaCodec.dequeueOutputBuffer()` in a loop. When a decoded frame is acquired, it doesn't release it immediately, but first checks the difference between the current system time and the frame's timestamp. If it finds that the delay has accumulated and exceeded a set threshold (e.g., 40ms), it enters "catch-up mode": quickly releasing the current and several subsequent decoded older frames until the latest frame is acquired. For the frame ultimately decided to be displayed, the thread calls `SystemClock.elapsedRealtime()` to calculate a compact future display time and passes it as a parameter to `MediaCodec.releaseOutputBuffer(index, renderTimeNanos)`, precisely instructing the system when to display this frame.

[0074] Furthermore, the entire lifecycle of the decoder, including its creation, stopping, release, and recovery logic during Surface destruction and reconstruction, is managed uniformly by a dedicated decoding and reconstruction module, ensuring stable operation under various application state transitions and abnormal conditions.

[0075] The complete workflow of this embodiment can be summarized as follows: Through a highly customized factory mode, hardware capability detection, chip platform identification, color format optimization, and precise matching of low-latency parameters are completed at the beginning of decoder creation; during the decoding operation phase, significant compression of the entire decoding link latency is achieved through the direct connection path between decoding output and display, as well as dynamic frame dropping and precise timestamp delivery strategies based on real-time latency monitoring; at the same time, the stability and robustness of the entire complex system are ensured through an independent management module.

[0076] By combining all the aforementioned technical features, this embodiment achieves significant overall technical benefits. Firstly, regarding latency, compared to solutions using only Android's default decoding mechanism, end-to-end video display latency can be reduced from hundreds of milliseconds to tens of milliseconds, greatly improving the playability of cloud gaming. Secondly, in terms of compatibility and stability, through dynamic hardware capability detection and fine-grained resource management, this solution can stably enable hardware decoding on Android devices with various chip platforms, including HiSilicon and Goke Microelectronics, avoiding common crashes and rollback issues. Finally, regarding responsiveness, the dynamic frame dropping strategy ensures that even under network jitter, the user's view is always as up-to-date as possible, guaranteeing real-time feedback. The combined effect of these benefits ultimately provides cloud gaming users on the Android platform with a smooth interactive experience approaching that of local games.

[0077] It is understood that the technical solutions provided in the embodiments of this invention are not limited to cloud gaming. Any real-time video application on the Android platform with stringent requirements for video latency can benefit from this invention. For example, in remote desktop or virtual desktop infrastructure (VDI) applications, low-latency screen streaming is directly related to the efficiency and user experience of remote work, and the solution of this invention can ensure smooth remote desktop operation.

[0078] Another typical application scenario is first-person view (FPV) video transmission systems for drones or robots. In these applications, operators need to control the device based on real-time video feeds, and any significant delay can lead to operational errors or even accidents. This invention provides operators with a stronger sense of presence and more precise control capabilities by implementing low-latency video decoding on the Android receiver.

[0079] Furthermore, in telemedicine, particularly in remote surgical guidance, doctors need to observe real-time high-definition video of the surgical area. Reduced video latency means that guidance can be delivered more promptly, which is crucial for the accuracy and safety of the surgery. The technical solution of this invention can provide reliable low-latency video technology support for such critical applications with high real-time requirements.

[0080] In interactive live streaming, such as online auctions and interactive Q&A games, the interaction delay between the streamer and the audience directly affects the sense of participation. By applying the decoding method of this invention, the time it takes for viewers to see the streamer's screen can be shortened, making interactive feedback such as bullet comments and gifts more immediate and improving the immediacy of the interactive experience.

[0081] In summary, this invention provides a real-time video decoding solution optimized for the Android platform. By identifying the hardware platform, customizing application parameters, optimizing data paths, and controlling display timing, this solution addresses the issues of inconsistent decoding performance and difficulties in latency optimization caused by hardware ecosystem fragmentation. Therefore, it provides an effective technical solution for applications requiring low-latency real-time video streaming on Android terminals.

[0082] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0083] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.

[0084] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0085] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0086] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0089] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A low-latency decoding optimization method for Android real-time video streams, characterized in that, Includes the following steps: It receives and parses video encoded data from the cloud to generate decodeable video frame data; Identify the chip platform type of the current Android terminal; Based on the chip platform type, a corresponding low-latency decoding parameter combination is selected from a preset configuration strategy set, and the MediaCodec decoder of the Android system is initialized and configured based on the parameter combination. The video frame data is input to the configured MediaCodec decoder for hardware decoding, and the decoded video frames are directly output to the Surface; The display timing of video frames output to the Surface is controlled to reduce the waiting time of video frames in the display queue.

2. The method according to claim 1, characterized in that, The identification of the chip platform type of the current Android terminal includes: Read system properties or decoder information, and determine the chip platform type based on the decoder name or the manufacturer identifier contained in the system properties.

3. The method according to claim 1 or 2, characterized in that, The low-latency decoding parameter combination includes at least one of the following: Low-latency decoding mode enable parameters; The parameter limiting the number of decoder input and output buffers; Parameters used to disable B-frame decoding, post-processing, or frame rate conversion. Parameters used to enable automatic frame dropping by hardware; This is for the proprietary low-latency parameters defined by the pre-defined chip manufacturer.

4. The method according to claim 1, characterized in that, Controlling the display timing of video frames output to the Surface includes: During the decoding output stage, the expected display timestamp of the video frame is dynamically calculated based on the system clock; Release the MediaCodec output buffer using the method described above, in order to control when video frames enter the Surface display queue.

5. The method according to claim 4, characterized in that, Also includes: When a backlog of video frames or a display delay exceeding a threshold is detected, the control logic prioritizes displaying the latest decoded video frame and actively discards expired video frames.

6. The method according to claim 1, characterized in that, Before inputting the video frame data into the configured MediaCodec decoder for hardware decoding, the method further includes: Dynamically enumerate the available hardware decoders in the Android system and the color formats they support; When initializing the MediaCodec decoder, the color format is matched and set according to the dynamically enumerated color format results to ensure that the hardware decoder is successfully enabled.

7. A low-latency decoding optimization system for Android real-time video streams, characterized in that, The system for implementing the method as described in any one of claims 1 to 6 comprises: The data receiving module is used to receive and parse video encoded data from the cloud to generate decodable video frame data; The chip identification module is used to identify the chip platform type of the current Android terminal; The decoding control module is used to select the corresponding low-latency decoding parameter combination according to the chip platform type, and to perform initialization, data input and decoding operations on the MediaCodec decoder based on the parameter combination; The Surface display control module is used to directly bind video frames output by the MediaCodec decoder to the Surface display and control the display timing of the video frames.

8. The system according to claim 7, characterized in that, The decoding control module includes a custom video decoder factory, which is configured as follows: When creating a decoder instance, actively enumerate and match the color formats supported by the hardware decoder in the system, and force the return of the MediaCodec-based hardware decoder implementation to avoid the decoding process falling back to software decoding.

9. The system according to claim 7, characterized in that, The system also includes a decoding management and reconstruction module, which is used for: Separate the control logic of the MediaCodec decoder from the output thread; In the event of decoding stoppage or an exception, the MediaCodec decoder resources are safely released via the output thread. It also supports the reconstruction and rebinding of decoders when the Surface changes.

Images