Multimedia system for mobile client platforms

Abstract

A method for multimedia playback and transmission to wireless clients is described. A host webserver transcodes a live digital or analog audio-visual or audio broadcast signal and splits the input stream into small multimedia objects of an efficient compression such as MPEG4 / AAC, and then immediately deploys the objects to distributed content servers for a geographically dispersed population of wireless clients. A java applet object player, downloaded to wireless clients at the beginning of the multimedia on-demand session, interprets and decodes the multimedia objects as they are received, using multiple levels of optimization. The applet uses novel video and audio decoding optimizations which can be generically applied to many digital video and audio codecs, and specifically decodes Simple Profile MPEG4 video and Low Complexity AAC audio.

Description

FIELD OF INVENTION [0001] The invention relates to issues of the wireless Internet, specifically to methods of multimedia transmission and playback for mobile clients. BACKGROUND OF THE INVENTION [0002] The current methods of distributing multimedia data over the wireless Internet to mobile clients are constrained by existing wireless bandwidth, and the real-time decoding, processing and displaying of multimedia content with limited hardware capabilities. These hardware limitations include slow CPUs, high memory latencies, slow drawing capabilities and the absence of YUV to RGB conversion in the hardware. [0003] Video and audio playback exist on certain cell phone handsets, but this technology is embedded and takes advantage of low-level hardware processing to enable the performance required for media playback. Through embedded media players, several cell phone handsets and handheld computers can play either streamed video or audio. Popular digital video encoding standards for some ...

AI Technical Summary

Benefits of technology

[0014] The multimedia object creator produces discrete multimedia objects from video and audio segments of a continuous stream. If the stream is MPEG4, multimedia objects can also be segments of multiple component video and audio streams. In the case of multiple MPEG4 component streams, per object segmentation and decoding can enable the composition of a single scene from several temporally-independent multimedia objects. This provides the possibility of decoding only a limited number of multimedia objects, and not all objects, to provide an object-based scalability.

[0018] The wireless networks over which the limited devices operate often have a very high latency. The HTTP 1.1 protocol and some implementations of HTTP 1.0 allow the use of a persistent connection over TCP to perform multiple requests. Furthermore, some HTTP 1.1 implementations allow the use of pipelined connections allowing the HTTP client to perform many requests in rapid succession decreasing the latency between the request and reception of each multimedia object. When possible, the invention can take advantage of this capability.

[0021] In a third aspect, the invention provides novel optimizations for digital video decoding. Some of these optimizations can then be used by an expert assessment process, whereby, the decoder maintains a state information list of short-cuts, related to perceived frame rate in a sorted list starting with those that will decrease output quality the least, to those that will decrease output quality the most but have the most impact on decoding speed. The client player-decoder dynamically adjusts how many shortcuts must be taken. These short-cuts are specifically designed to drastically reduce the number of computations necessary at certain critical steps in the video decoding process at the cost video output quality. This allows the video decoder to scale in complexity based on the processing power of the device being used. It also allows users to experience multimedia playback despite the limitations of the device they may be using.

Problems solved by technology

These hardware limitations include slow CPUs, high memory latencies, slow drawing capabilities and the absence of YUV to RGB conversion in the hardware.

At the time of this writing, most cell phones cannot support multimedia playback because they are only capable of supporting one or a few concurrent processing threads.

On handsets that have embedded media players, video Is also limited to very low frame rates and the bandwidth available for streaming in North America is low, varying from 2-3 kybtes / second to ISDN speeds of 64 Kbits.

But again, this approach does not use MPEG4 video encoding.

Similarly, although Java decoders exist that play MP3 ringtones on cell phones, no Java players are known that will play AAC.

In the case of wireless networks, bit streaming in terms of audio / video content is usually is done over a non-reliable transport like UDP and requires a lot of error correction and duplication of content (extra stream correction data).

However, these do not address specific continuity and deployment issues for wireless multimedia transmission and concurrent playback on a limited-tasking cell phone handsets.

Images