Establishing video conference during phone call
A dual-camera mobile device with real-time adjustment capabilities addresses the challenge of transmitting multiple video feeds, enhancing video conferencing by allowing simultaneous and flexible video stream management.
Patent Information
- Application Number
- JP2024106235
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2010-06-06
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-13
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing mobile devices with a single camera struggle to transmit live video streams efficiently, as they lack the capability to process and transmit multiple video feeds simultaneously, and existing methods do not allow for real-time viewing of captured video.
A mobile device equipped with two cameras, capable of capturing and transmitting video from both cameras simultaneously, with features for real-time adjustment and composite display of multiple video feeds, including a picture-in-picture mode and dynamic switching between cameras.
Enables seamless transmission and display of multiple video streams during video conferences, allowing for enhanced interaction and flexibility in video conferencing scenarios.
Smart Images

Figure 2024160220000001_ABST
Abstract
Description
[Technical field]
[0001] [Background technology]
[0002] Many of today's mobile devices, such as smartphones, offer video capture capabilities. A mobile device user can capture both still images and video through the phone's camera. However, to transmit the captured video to another party, the user must typically either send the video directly to the other party after the video capture is complete, or upload the video to another location (e.g., an Internet video hosting site). Unfortunately, this does not allow the other party to watch a live video stream while the video is being captured by the mobile device.
[0003] Furthermore, a typical mobile device is equipped with only one camera, and processing the information from this single camera is quite difficult. An ideal device would have multiple cameras and be able to transmit live video that is a composite of the videos from at least two cameras. This is a particularly challenging problem, given the limited resources available to mobile devices, both in terms of the device processing the multiple captured video streams and in terms of the network to which the device is connected handling the transmission of the live video stream. Summary of the Invention
[0004] Some embodiments of the present invention provide a portable device having two cameras capable of taking photos and videos. The portable device of some embodiments has a display screen for displaying captured photo and video images. The portable device also includes a storage device for storing captured images for later transmission to another device. The device further includes a network interface capable of transmitting the captured images to one or more devices during a real-time communication session between users of the devices. The device also includes an encoder that can be used to encode the captured images for local storage or transmission to another device. The portable device further includes a decoder that enables the device to decode images captured by another device during a real-time communication session or to decode locally stored images.
[0005] One example of a real-time communication session that includes the transmission of captured video images is a video conference. In some embodiments, the mobile device can transmit only the captured video images of one camera at any given time during a video conference. However, in other embodiments, the mobile device can transmit captured video images from both of its cameras simultaneously during a video conference or other real-time communication session.
[0006] During a videoconference with another device, the mobile device of some embodiments can transmit other types of content along with the video captured by one or both of its cameras. One example of such other content includes low- or high-resolution photographic images captured by one of the device's cameras while the other camera of the device captures the video used in the videoconference. Other examples of such other content include (1) files and other content stored on the device, (2) the device's screen display (i.e., content displayed on the device's screen), and (3) content received from another device during the videoconference or other real-time communication session.
[0007] The mobile device of some embodiments employs a novel in-conference adjustment technique for making adjustments during a video conference. For example, the mobile device of some embodiments can dynamically switch to transmit video captured by its other camera while transmitting only the video captured by one camera during a video conference. In such a situation, the mobile device of some embodiments notifies any other devices participating in the video conference of the switch so that the other devices can realize a smooth transition between the videos captured by the two cameras on their side.
[0008] In some embodiments, camera switching requests can come not only from a "local" device switching its camera during a video conference, but also from other "remote" devices receiving the video captured by the local device. Furthermore, one device being able to instruct another device to switch cameras is only one example of the remote control capabilities of the devices in some embodiments. Examples of other operations that can be remotely instructed on devices in some embodiments include exposure adjustment operations (e.g., auto exposure), focus adjustment operations (e.g., auto focus), and the like. Another example of a novel in-conference adjustment that can be specified locally or remotely is the specification of a region of interest (ROI) in the captured video, where the ROI identification is used to change the behavior of the capturing camera, to change the image processing operations of the device that has the capturing camera, or to change the encoding operations of the device that has the capturing camera.
[0009] Yet another example of novel in-meeting adjustments of some embodiments includes real-time modification of a composite video display generated by the device. Specifically, in some embodiments, the mobile device generates a composite display that simultaneously displays multiple videos captured by multiple cameras of one or more devices. In some cases, the composite display places the videos in adjacent viewports (e.g., adjacent windows). In other cases, the composite display is a picture-in-picture (PIP) display that includes at least two viewports showing two different videos, one of which is a background main viewport and the other is a foreground inset viewport that overlaps the background main viewport.
[0010] In some embodiments, real-time modification of the composite video display includes moving one or more of the viewing areas within the composite display in response to user selections and movement of the viewing areas. Some embodiments also rotate the composite display during a video conference if the screen of the device that provided the composite display is rotated. Some embodiments of the mobile device also allow the device user to swap video within a PIP display (i.e., displaying video from a background main display in a foreground inset display and displaying video from a foreground inset display in a background main display).
[0011] The above summary of the invention is intended to serve as a brief description of some embodiments of the present invention. It is not intended to be a description or overview of all inventive subject matter disclosed herein. In the following detailed description of the invention and the drawings referenced in the detailed description of the invention, the embodiments described in the summary of the invention as well as other embodiments will be further described. Therefore, a thorough review of the summary of the invention, the detailed description of the invention, and the drawings is required to understand all embodiments described herein.
[0012] The novel features of the invention are set forth in the appended claims, but for purposes of illustration, certain embodiments of the invention are shown in the following figures. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 illustrates a composite display of some embodiments. [Diagram 2] FIG. 13 illustrates another composite display of some embodiments. [Diagram 3] FIG. 2 conceptually illustrates the software architecture for the video processing and encoding module of the dual camera mobile device of some embodiments. [Figure 4] FIG. 2 conceptually illustrates a captured image processing unit of some embodiments. [Diagram 5] FIG. 2 conceptually illustrates an example of different frame rates based on different vertical blanking intervals (VBIs). [Figure 6] FIG. 2 conceptually illustrates an example of different interleaved frame rates based on different VBIs. [Figure 7] FIG. 2 conceptually illustrates another captured image processing unit of some embodiments. [Figure 8] FIG. 2 conceptually illustrates another captured image processing unit of some embodiments. [Figure 9] FIG. 2 conceptually illustrates the software architecture for the videoconferencing and processing module of the dual camera mobile device of some embodiments. [Figure 10] FIG. 2 conceptually illustrates an exemplary videoconference request message sequence of some embodiments. [Figure 11] FIG. 1 illustrates a user interface of some embodiments for a video conference setup operation. [Figure 12] 1 illustrates a user interface of some embodiments for accepting a videoconference invitation. [Figure 13] FIG. 2 illustrates another user interface of some embodiments for accepting a videoconference invitation. [Figure 14] FIG. 13 illustrates another user interface of some embodiments for a video conference setup operation. [Figure 15]FIG. 2 conceptually illustrates a process in some embodiments for setting the bit rate of a video conference. [Figure 16] FIG. 2 conceptually illustrates another software architecture for the videoconferencing and processing module of the dual camera mobile device of some embodiments. [Figure 17] FIG. 2 conceptually illustrates another software architecture for a dual camera mobile device of some embodiments. [Figure 18] FIG. 17 conceptually illustrates a process performed by a videoconference manager of some embodiments, such as the process illustrated in FIG. 16. [Figure 19] FIG. 2 conceptually illustrates a software architecture for a temporal noise reduction module of some embodiments. [Figure 20] FIG. 2 conceptually illustrates a process of some embodiments for reducing noise in an image of a video. [Figure 21] FIG. 10 conceptually illustrates a process performed by an image processing manager of some embodiments, such as the process illustrated in FIG. 9 . [Figure 22] 13 illustrates a user interface of some embodiments for exposure adjustment operations. [Figure 23] FIG. 13 illustrates a user interface of some embodiments for focus adjustment operations. [Figure 24] FIG. 17 conceptually illustrates a perspective correction process performed by an image processing manager of some embodiments, such as the process shown in FIG. 16. [Diagram 25] FIG. 1 conceptually illustrates an exemplary perspective correction operation of some embodiments. [Figure 26] FIG. 17 conceptually illustrates a software architecture for an encoder driver of some embodiments, such as the software architecture illustrated in FIG. [Figure 27] FIG. 27 conceptually illustrates an image resizing process performed by an encoder driver of some embodiments, such as the encoder shown in FIG. [Figure 28] FIG. 17 conceptually illustrates a software architecture for a decoder driver of some embodiments, such as the software architecture shown in FIG. [Figure 29] FIG. 29 conceptually illustrates an image extraction process performed by a decoder driver of some embodiments, such as the decoder driver shown in FIG. 28. [Diagram 30] FIG. 2 illustrates an encoder driver of some embodiments that includes two rate controllers. [Diagram 31] FIG. 17 conceptually illustrates a software architecture for a networking manager of some embodiments, such as the networking manager shown in FIG. [Diagram 32] FIG. 1 illustrates a user interface of some embodiments for a snap-to-corner operation. [Diagram 33] FIG. 13 illustrates another user interface of some embodiments for snapping to a corner. [Diagram 34] 13 illustrates a user interface of some embodiments for a PIP display rotation operation. [Diagram 35] 13 illustrates another user interface of some embodiments for a PIP display rotation operation. [Diagram 36] 13 illustrates another user interface of some embodiments for a PIP display rotation operation. [Figure 37] 13 illustrates another user interface of some embodiments for a PIP display rotation operation. [Figure 38] 1 illustrates a user interface of some embodiments for resizing the foreground inset area of a PIP display. [Figure 39] 13 illustrates another user interface of some embodiments for resizing an insert viewport of a PIP display. [Diagram 40] 13 illustrates another user interface of some embodiments for resizing an insert viewport of a PIP display. [Diagram 41] 13 illustrates another user interface of some embodiments for resizing an insert viewport of a PIP display. [Diagram 42] 1 illustrates a user interface of some embodiments for identifying an area of interest in a display. [Diagram 43] FIG. 13 illustrates another user interface of some embodiments for identifying an area of interest in a display. [Diagram 44] FIG. 13 illustrates another user interface of some embodiments for identifying an area of interest in a display. [Diagram 45] FIG. 2 illustrates a process of some embodiments for performing local camera switching operations on a dual camera mobile device. [Figure 46] FIG. 1 illustrates a user interface of some embodiments for camera switching operations. [Figure 47] FIG. 13 illustrates another user interface of some embodiments for camera switching operations. [Figure 48] FIG. 13 illustrates another user interface of some embodiments for camera switching operations. [Figure 49] FIG. 13 illustrates another user interface of some embodiments for camera switching operations. [Figure 50] FIG. 2 illustrates a process of some embodiments for performing remote camera switching operations on a dual camera mobile device. [Figure 51] 1 illustrates a user interface of some embodiments for remote control camera switching operations. [Figure 52] FIG. 13 illustrates another user interface of some embodiments for remote control camera switching operations. [Figure 53] FIG. 13 illustrates another user interface of some embodiments for remote control camera switching operations. [Figure 54] FIG. 13 illustrates another user interface of some embodiments for remote control camera switching operations. [Figure 55] FIG. 2 conceptually illustrates a process of some embodiments for performing an exposure adjustment operation. [Figure 56] 13 illustrates a user interface of some embodiments for exposure adjustment operations. [Figure 57] FIG. 13 illustrates another user interface of some embodiments for exposure adjustment operations. [Figure 58] FIG. 13 illustrates another user interface of some embodiments for exposure adjustment operations. [Figure 59] FIG. 17 conceptually illustrates an exposure adjustment process performed by an image processing manager of some embodiments, such as the process illustrated in FIG. 16. [Figure 60] 13A-13C conceptually illustrate an example exposure adjustment operation of some embodiments. [Figure 61] FIG. 2 conceptually illustrates a process of some embodiments for performing a focus adjustment operation. [Figure 62] FIG. 13 illustrates a user interface of some embodiments for focus adjustment operations. [Figure 63] FIG. 13 illustrates another user interface of some embodiments for focus adjustment operations. [Figure 64] FIG. 13 illustrates another user interface of some embodiments for focus adjustment operations. [Figure 65] 1A-1C illustrate different display configurations of some embodiments for videos captured from one or more dual camera mobile devices. [Figure 66] 1 illustrates a user interface of some embodiments for overlaying an insert video foreground onto a background video in a PIP display. [Figure 67] 1 illustrates a technique according to some embodiments for determining the foreground of a video image. [Figure 68] 1 illustrates a user interface of some embodiments for replacing an inset view of a PIP view with a background view during a video conference. [Figure 69]1 illustrates a user interface of some embodiments for snapping to a corner. [Figure 70] 1 illustrates a user interface of some embodiments for snap and push actions towards corners. [Figure 71] 13 illustrates a user interface of some embodiments for a PIP display rotation operation. [Figure 72] 13 illustrates another user interface of some embodiments for a PIP display rotation operation. [Figure 73] 1 illustrates a user interface of some embodiments for selecting one video from two remote videos during a video conference. [Figure 74] 1 illustrates a user interface of some embodiments for selecting one video from two local videos during a video conference. [Figure 75] 1 illustrates a user interface of some embodiments for pre-meeting selection of video to be used for a video conference. [Figure 76] FIG. 1 illustrates an example of bandwidth allocation between two videos captured by a dual camera mobile device of some embodiments. [Figure 77] FIG. 2 conceptually illustrates an arbitrator module of some embodiments for managing a rate controller in a dual camera mobile device. [Figure 78] FIG. 2 conceptually illustrates a method of some embodiments for encoding images captured by the cameras of a dual camera mobile device. [Figure 79] FIG. 2 conceptually illustrates another method of some embodiments for encoding images captured by the cameras of a dual camera mobile device. [Figure 80] FIG. 80 illustrates an exemplary image synthesis for the method shown in FIG. 79. [Figure 81] FIG. 2 conceptually illustrates another method of some embodiments for encoding images captured by the cameras of a dual camera mobile device. [Figure 82] FIG. 2 conceptually illustrates a method of some embodiments for decoding images captured by the cameras of a dual camera mobile device. [Figure 83] FIG. 2 conceptually illustrates another method of some embodiments for decoding images captured by the cameras of a dual camera mobile device. [Figure 84] FIG. 2 conceptually illustrates another software architecture for the videoconferencing and processing module of the dual camera mobile device of some embodiments. [Figure 85] FIG. 2 illustrates a user interface of some embodiments for multi-participant video conferencing. [Figure 86] FIG. 2 illustrates another user interface of some embodiments for multi-participant video conferencing. [Figure 87] FIG. 2 illustrates another user interface of some embodiments for multi-participant video conferencing. [Figure 88] FIG. 1 conceptually illustrates an application programming interface (API) architecture of some embodiments. [Figure 89] FIG. 2 illustrates an architecture for a dual camera mobile computing device of some embodiments. [Figure 90] FIG. 1 conceptually illustrates a touch input / output (I / O) device of some embodiments. [Figure 91] FIG. 1 conceptually illustrates an exemplary communication system of some embodiments. [Figure 92] FIG. 2 conceptually illustrates another exemplary communication system of some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] In the following description, numerous details are set forth for purposes of explanation. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the description of the present invention with unnecessary detail.
[0015] Some embodiments of the present invention provide a mobile device with two cameras capable of taking photos and videos. Examples of mobile devices include mobile phones, smartphones, personal digital assistants (PDAs), laptops, tablet personal computers, or any other type of mobile computing device. As used herein, photos refer to still photographic images taken by the camera, one at a time in single-photo mode, or multiple at a time in fast-action mode. Video, on the other hand, refers to a sequence of video images captured by the camera at a particular speed, which is often referred to as the frame rate. Typical frame rates for capturing video are 25 frames per second (fps), 30 fps, and 60 fps. The camera of the mobile device of some embodiments can capture video images (i.e., video frames) at these and other frame rates.
[0016] In some embodiments, the portable device can (1) display captured photographic and video images, (2) store captured images for later transmission to another device, (3) transmit captured images to one or more devices during a real-time communication session between users of the devices, and (4) encode captured images for local storage or transmission to another device.
[0017] One example of a real-time communication session involving the transmission of captured video images is a video conference. In some embodiments, the mobile device can only transmit the captured video images of one camera at any given time during the video conference. However, in other embodiments, the mobile device can transmit captured video images from both of its cameras simultaneously during the video conference or other real-time communication session.
[0018] Some embodiments of mobile devices generate a composite display that includes a simultaneous display of multiple videos captured by multiple cameras of one or more devices. In some cases, the composite display places the videos in adjacent display areas (e.g., adjacent windows). Figure 1 shows one such example of a composite display 100 that includes two adjacent display areas 105 and 110 that simultaneously display two videos captured by two cameras of one device or by two cameras of two different devices during a video conference.
[0019] In other cases, the composite display is a PIP display that includes at least two viewports showing two different videos, one of which is a background main viewport and the other of which is a foreground inset viewport that overlaps the background main viewport. FIG. 2 shows one such example of a composite PIP display 200. The composite PIP display 200 includes a background main viewport 205 and a foreground inset viewport 210 that overlaps the background main viewport. The two viewports 205 and 210 simultaneously display two videos captured by two cameras of one device or captured by two cameras of two different devices during a video conference. The exemplary composite PIP display shown and described herein is similar to the composite PIP display 200 in which the foreground inset viewport 210 is displayed entirely within the background main viewport 205, although other composite PIP displays having a foreground inset viewport 210 that overlaps but is not entirely within the background main viewport 205 are possible.
[0020] In addition to transmitting video content during a videoconference with another device, the mobile device of some embodiments can transmit other types of content along with the video content of the conference. One example of such other content is a low- or high-resolution photographic image captured by one of the device's cameras while another camera of the device captures the video used in the videoconference. Other examples of such other content include (1) files and other content stored on the device, (2) the device's screen display (i.e., the content displayed on the device's screen), and (3) content received from another device during a videoconference or other real-time communication session.
[0021] The mobile device of some embodiments employs a novel in-conference adjustment technique for making adjustments during a video conference. For example, the mobile device of some embodiments can dynamically switch to transmit video captured by the other camera while transmitting only the video captured by one camera during a video conference. In such a situation, the mobile device of some embodiments notifies any other devices participating in the video conference of the switch, so that the other devices can realize a smooth transition between the videos captured by the two cameras on their side.
[0022] In some embodiments, camera switching requests can originate not only from a "local" device that switches its camera during a video conference, but also from other "remote" devices that are receiving video captured by the local device. Furthermore, one device being able to instruct another device to switch cameras is only one example of the remote control capabilities of the devices in some embodiments. Other examples of operations that can be remotely instructed on devices in some embodiments include exposure adjustment operations (e.g., auto exposure), focus adjustment operations (e.g., auto focus), etc. Another example of a novel in-conference adjustment that can be specified locally or remotely is the identification of a region of interest (ROI) in the captured video, which is used to change the behavior of the capturing camera, to change the image processing operations of the device that has the capturing camera, or to change the encoding operations of the device that has the capturing camera.
[0023] Yet another example of novel in-conference adjustments of some embodiments involves real-time modification of a composite video display generated by the device. Specifically, in some embodiments, real-time modification of the composite video display involves moving one or more of the viewing areas within the composite display in response to a user's selection and movement of the viewing areas. Also, in some embodiments, the composite display is rotated during a video conference if the screen of the device that provided the composite display is rotated. Also, the mobile device of some embodiments allows the device user to reverse the order of the videos in a PIP display (i.e., to have video from the background main display displayed in the foreground inset display while video from the foreground inset display is displayed in the background main display).
[0024] Some more detailed embodiments are described below: Section I provides a description of the video processing architecture of some embodiments. Then, Section II describes the captured image processing unit of some embodiments. In some embodiments, this unit is a component of the device responsible for processing RAW images captured by the device's camera.
[0025] Section III then describes the videoconferencing architecture of some embodiments. This section describes some schemes for setting up a videoconference with a single camera as well as the videoconferencing module of some embodiments. Section IV then describes the in-conference coordination and control operations of some embodiments. Section V then describes the videoconferencing functionality of embodiments that transmit and display multiple videos from individual devices during a videoconference. Section VI then describes transmitting real-time video along with non-real-time content during a videoconference. Finally, Section VII describes the hardware architecture of the dual camera device of some embodiments.
[0026] I. Video Capture and Processing 3 conceptually illustrates a video processing and encoding module 300 of a dual camera mobile device of some embodiments. In some embodiments, the module 300 processes images captured by the cameras of the dual camera mobile device and encodes video also captured by the cameras of the dual camera mobile device. As shown in FIG. 3, the module 300 includes a captured image processing unit (CIPU) driver 305, a media exchange module 310, an encoder driver 320, and a video processing module 325.
[0027] In some embodiments, the media exchange module 310 allows programs on the device that are consumers and producers of media content to exchange media content and instructions for processing that media content. In the video processing and encoding module 300, the media exchange module 310 of some embodiments routes instructions and media content between the video processing module 325 and the CIPU driver 305, and between the video processing module 325 and the encoder driver 320. To facilitate such routing of instructions and media content, the media exchange module 310 of some embodiments provides a collection of application programming interfaces (APIs) for use by consumers and producers of media content. In some of such embodiments, the media exchange module 310 is a collection of one or more frameworks that are part of the operating system running on the dual camera mobile device. One example of such a media exchange module 310 is the Core Media framework provided by Apple Inc.
[0028] The video processing module 325 performs image processing on images and / or video captured by the device's camera. Examples of such operations include exposure adjustment operations, focus adjustment operations, perspective correction, dynamic range adjustment, image resizing, image compositing, etc. In some embodiments, some image processing operations may also be performed by the media replacement module 310. For example, as shown in FIG. 3, the media replacement module 310 of some embodiments performs a temporal noise reduction (TNR) operation (e.g., by TNR 315) that reduces noise in video images captured by the device's camera. Further examples of such image processing operations of the video processing module 325 and the media replacement module 310 are provided below.
[0029] As mentioned above, the video processing module 325 communicates with the CIPU driver 305 and the encoder driver 320 through the media exchange module 310. The CIPU driver 305 serves as a communications interface between a captured image processing unit (CIPU) 330 and the media exchange module 310. As described further below, the CIPU 330 is the component of the dual camera device responsible for processing images captured during image or video capture operations of the device's cameras. The CIPU driver 305 receives requests for images and / or video from one or both of the device's cameras from the video processing module 325 through the media exchange module 310. The CIPU driver 305 relays such requests to the CIPU 330 and, in response, receives the requested images and / or video from the CIPU 330, which the CIPU driver 305 then sends to the video processing module 325 through the media exchange module 310. In some embodiments, the video processing module 325 also sends instructions to the CIPU 330 through the CIPU driver 305 and the media exchange module 310 to change some of its own operations (e.g., changing the camera frame rate, exposure adjustment operations, focus adjustment operations, etc.).
[0030] The encoder driver 320 serves as a communication interface (e.g., an encoder chip, an encoding component on a system-on-chip, etc.) between the media exchange module 310 and the encoder hardware 335. In some embodiments, the encoder driver 320 receives an image and a request to encode the image from the video processing module 325 through the media exchange module 310. The encoder driver 320 sends the image to be encoded to the encoder 335, which then performs picture encoding or video encoding on the image. When the encoder driver 320 receives an encoded image from the encoder 335, the encoder driver 320 sends the encoded image back to the video processing module 325 through the media exchange module 310.
[0031] In some embodiments, the video processing module 325 can perform various operations on the encoded images received from the encoder, such as storing the encoded images in a storage device of the device, transmitting the encoded images during a video conference call over a network interface of the device, etc.
[0032] In some embodiments, some or all of the modules of the video processing and encoding module 300 are implemented as part of an operating system. For example, in some embodiments, all four components 305, 310, 320, and 325 of this module 300 are implemented as part of the device's operating system. In other embodiments, the media exchange module 310, the CIPU driver 305, and the encoder driver 320 are implemented as part of the device's operating system, and the video processing module 325 is included as an application running on the operating system. Additionally, other implementations of the module 300 are possible.
[0033] The operation of the video processing and encoding module 300 during a video capture session is described below. To start a video capture session, the video processing module 325 initializes several components required for a video capture session. In some embodiments, these components include: (1) the CIPU 330, (2) a scaling and compositing module (not shown) of the video processing module 325, (3) an image processing module (not shown) of the video processing module 325, and (4) an encoder 335. In some embodiments, the video processing module 325 also initializes a network manager (not shown) when joining a video conference.
[0034] The video processing module sends its initialization request, through the media exchange module 310 and the CIPU driver 305, to the CIPU 330 to cause one or both of the device's cameras to begin video capture. In some embodiments, the request specifies a particular frame rate, exposure level, and scale size for each camera that needs to capture video. In response to the request, the CIPU 330 begins returning video images from the requested cameras at the specified rate(s), exposure level(s), and scale size(s). These video images are returned through the CIPU driver 305 and the media exchange module 310 to the video processing module 325, which performs TNR operations on the video images as described above before providing them to the video processing module 325. At the video processing module 325, the video images are stored in a buffer (not shown) for further image processing.
[0035] An image processing module of the video processing module 325 retrieves the buffered video images for further video processing. A scaling and compositing module then retrieves the processed video images for scaling, if necessary, for real-time display on the display screen of the device. In some embodiments, this module creates a composite image from images captured by two cameras of a device together with a camera(s) of another device during a video conference, or from images captured by a camera(s) of a device, to provide real-time display of the captured video images on the device, or to create a composite video image for encoding.
[0036] The processed and / or composited video images are provided to encoder 335 via encoder driver 320 and media exchange module 310. Encoder 335 then encodes the video images. The encoded images are then returned to video processing module 325 (also via encoder driver 320 and media exchange module 310) for storage on the device or transmission during the videoconference. When the device is participating in a videoconference, a network manager (initialized by video processing module 325) then reads these encoded images and packetizes and transmits them to one or more other devices through the device's network interface (not shown).
[0037] II. Processing the captured images A. Single Pipeline The images captured by the cameras of the dual camera mobile device of some embodiments are unprocessed raw images. These images require conversion to a particular color space before they can be used for other operations, such as transmitting the images to another device (e.g., during a video conference), storing the images, or displaying the images. Furthermore, the images captured by the cameras may need to be processed to correct errors and / or distortions, adjust the image color, size, etc. Therefore, in some embodiments, some processing operations are performed on the images before storing, transmitting, and displaying such images. Some of the processing of such images is performed by the CIPU 330.
[0038] An example of such a CIPU is shown in FIG. 4. Specifically, this figure conceptually illustrates a captured image processing unit (CIPU) 400 of some embodiments. This CIPU 400 includes a single processing pipeline 485 that processes images from only one of the device's cameras at a time, or processes images from both of the device's cameras simultaneously in a time-multiplexed (i.e., time-interleaved) manner. The processing pipeline 485 of the CIPU 400 may be configured differently to accommodate different characteristics and / or operational settings of different cameras. Examples of different camera characteristics in some embodiments include different resolutions, noise sensors, lens types (fixed lens or zoom lens), etc. Also, examples of different operational settings at which the device can operate the cameras in some embodiments include image resolution size, frame rate, zoom level, exposure level, etc.
[0039] 4, CIPU 400 includes a sensor module 415, a line / frame buffer 417, a bad pixel correction (BPC) module 420, a lens shading (LS) module 425, a demosaicing module 430, a white balance (WB) module 435, a gamma module 440, a color space conversion (CSC) module 445, a hue, saturation, and contrast (HSC) module 450, a scaler module 455, a filter module 460, a statistics engine 465, two sets of registers 470, and a controller module 475. In some embodiments, all of the modules of CIPU 400 are implemented as hardware (e.g., ASIC, FPGA, SOC with a microcontroller, etc.), while in other embodiments, some or all of the modules of CIPU 400 are implemented as software.
[0040] 4, the sensor module 415 is communicatively coupled to two pixel arrays 410a and 410b of the device's two cameras and two sets of sensors 405a and 405b. In some embodiments, this communicative coupling is facilitated by a Mobile Industry Processor Interface (MIPI) on each camera sensor.
[0041] This communicative coupling allows the sensor module 415 to transfer instructions to the cameras to control various aspects of each camera's operation, such as its power level, zoom level, focus, exposure level, etc. In some embodiments, each camera has four operating power modes. In a first operating power mode, the camera is powered off. In a second operating power mode, the camera is powered on but not yet configured. In a third operating power mode, the camera is powered on, the camera's sensor is configured, and the camera sensor's pixels are collecting photons and converting the collected photons into digital values; however, the camera sensor has not yet sent an image to the sensor module 415. Finally, in a fourth operating power mode, the camera is in the same operating power mode as the third power mode, except that the camera is currently sending images to the sensor module 415.
[0042] During operation of the device, the camera may switch from one operating power mode to another any number of times. When switching operating power modes, some embodiments require the camera to switch between operating power modes in the sequence described above. Thus, in these embodiments, the camera in the first operating power mode can only switch to the second operating power mode. When in the second operating power mode, the camera can switch to either the first or third operating power mode. Similarly, from the third operating power mode, the camera can switch to either the second or fourth operating power mode. When in the fourth operating power mode, the camera can only switch back to the third operating power mode.
[0043] Furthermore, it takes a certain amount of time to switch from one operating power mode to the next or previous operating power mode. Thus, switching between two or three operating power modes is slower than switching between one operating power mode. Also, different operating power modes consume different amounts of power. For example, the fourth operating power mode consumes the most amount of power, the third operating power mode consumes more power than the first and second operating power modes, and the second operating power mode consumes more power than the first operating power mode. In some embodiments, the first operating power mode does not consume any power.
[0044] The camera may remain in one of the other operating power modes when not in the fourth operating power mode to capture an image. The decision of which operating mode to leave the camera in when not in use depends on how much power the camera is permitted to consume and how quickly the camera may need to respond to a request to begin image capture. For example, a camera configured to operate in the third operating power mode (e.g., standby mode) consumes more power than a camera configured to be in the first operating power mode (i.e., powered off). However, when the camera is commanded to capture an image, a camera operating in the third operating power mode can switch to the fourth operating power mode more quickly than a camera operating in the first operating power mode. In this way, the camera may be configured to operate in different operating power modes based on various requirements (e.g., response time to a request to capture an image, power consumption) when not capturing an image.
[0045] As described further below, when the video processing module 325 requests one or both cameras to begin capturing images, and the sensor module 415 receives this request through the controller module 475, the sensor module 415 can instruct one or both of the camera sensor sets through a communicative coupling with each camera to begin capturing images. A Bayer filter is convolved with each of the camera sensors, such that each camera sensor outputs a Bayer pattern image that is stored in a pixel array associated with each camera sensor. A Bayer pattern image is an image in which each pixel stores only one of the colors red, blue, or green.
[0046] The sensor module 415 reads out the Bayer pattern raw images stored in the camera pixel arrays 410a and 410b through its coupling with the pixel arrays 410a and 410b. By controlling the rate at which the sensor module 415 reads out images from the camera's pixel arrays, the sensor module 415 can control the frame rate of the video images being captured by a particular camera. By controlling its own image readout rate, the sensor module 415 can also interleave the fetching of images captured by different cameras in order to interleave the image processing of images captured from different cameras by the CIPU processing pipeline 485. The control of the image readout of the sensor module 415 is further described below in subsections II.A.1 and II.A.2.
[0047] The sensor module 415 reads image lines (i.e., rows of pixels of an image) from the pixel arrays 410a and 410b and stores them in the line / frame buffer 417. Each image line in the line / frame buffer 417 is processed by the CIPU processing pipeline 485. As shown in FIG. 4, the CIPU processing pipeline 485 is formed by the BPC module 420, the LS module 425, the demosaicing module 430, the WB module 435, the gamma module 440, the CSC module 445, the HSC module 450, the scaler module 455, and the filter module 460. In some embodiments, the CIPU processing pipeline 485 processes the image from the line / frame buffer 417 line-by-line (i.e., row-by-row), while in other embodiments, the CIPU processing pipeline 485 processes the entire image from the line / frame buffer 417 frame-by-frame.
[0048] In the exemplary pipeline shown in FIG. 4, the BPC module 420 is a module that reads out images from the line / frame buffer 417. This module performs a bad pixel removal operation that attempts to correct bad pixels in the read out image, which may be caused by defects in one or more of the camera sensors (e.g., a defective photosensor not detecting any light or detecting light incorrectly). In some embodiments, the BPC module 420 detects bad pixels by comparing a particular pixel in the image to one or more neighboring pixels in the image. If the difference between the value of the particular pixel and the neighboring pixels is greater than a threshold amount, the value of the particular pixel is replaced by the average of the values of several neighboring pixels of the same color (i.e., red, green, and blue) as the particular pixel.
[0049] The operation of BPC module 420 is controlled in part by values stored in two sets of registers 470 of CIPU 400 for this module. Specifically, to process images captured by two different cameras of the device, in some embodiments, CIPU processing pipeline 485 is configured differently for each camera, as described above. CIPU processing pipeline 485 is configured for the two different cameras by storing two different sets of values in two different sets of registers 470a (Ra) and 470b (Rb) of CIPU 400. Each set of registers 470 includes one register (Ra or Rb) for each of modules 420-460 in CIPU processing pipeline 485. Each register in each register set stores a set of values that define the operation of one processing pipeline module. Thus, as shown in FIG. 4, register set 470a is for indicating the mode of operation of each processing pipeline module for one camera (camera A) of the dual camera mobile device, and register set 470b is for indicating the mode of operation of each module for the other camera (camera B) of the dual camera mobile device.
[0050] One example of configuring CIPU processing pipeline 485 differently for each camera is to configure the modules of CIPU processing pipeline 485 to process images of different sizes. For example, if camera sensor 405a is 640x480 pixels and camera sensor 405b is 2048x1536 pixels, then a set of registers 470a is configured to store values that instruct the modules of CIPU processing pipeline 485 to process a 640x480 pixel image, and a set of registers 470b is configured to store values that instruct the modules of CIPU processing pipeline 485 to process a 2048x1536 pixel image.
[0051] In some embodiments, different processing pipeline configurations (i.e., register values) are stored in different profile settings. In some of such embodiments, a user of the mobile device is permitted to select one of the profile settings (e.g., by a user interface displayed on the mobile device) to configure the operation of the camera(s). For example, the user can select a profile setting to configure the camera to capture high-resolution video, a profile setting to configure the same camera to capture low-resolution video, or a profile setting to configure both cameras to capture high-resolution still images. Various configurations are possible and may be stored in many different profile settings. In other such embodiments, instead of having the user select a profile setting, the profile setting is automatically selected based on the application or activity selected by the user. For example, if the user selects a videoconferencing application, a profile that configures both cameras to capture video is automatically selected, if the user selects a photo application, a profile that configures one of the cameras to capture still images is automatically selected, and so on.
[0052] After the BPC module 420, the LS module 425 receives the bad pixel corrected image. The LS module 425 performs a lens shading correction operation to correct image defects caused by the camera lens that produce a light falloff effect (i.e., light decreases towards the edge of the camera sensor). Such effects cause the image to be unevenly lit (e.g., darker in the corners and / or edges). To correct these image defects, the LS module 425 of some embodiments estimates a mathematical model of the lens' illuminance falloff. The estimated model is then used to compensate for the lens falloff of the image to uniformly illuminate the unevenly lit parts of the image. For example, if the brightness of the corners of the image is half that of the center of the image, the LS module 425 of some embodiments doubles the pixel values of the corners to generate a uniform image.
[0053] Demosaicing module 430 performs a demosaicing operation to generate a full color image from the sampled color image. As previously described, the camera sensor outputs a Bayer pattern image, but this Bayer pattern image is incomplete because each pixel in the Bayer pattern image stores only one color value. Demosaicing module 430 reconstructs a red, green, blue (RGB) image from the Bayer pattern image by interpolating color values for each set of colors in the Bayer pattern image.
[0054] The WB module 435 performs a white balance operation on the RGB image received from the demosaicing module 430 so that the color of the image content resembles the color of such content as perceived by the human eye in real life. The WB module 435 adjusts the white balance by adjusting the color of the image to properly render neutral colors (e.g., gray, white, etc.). For example, an image of a blank piece of paper under incandescent light may appear yellow, but the human eye perceives the paper as white. To account for differences between the color of the image captured by the sensor and the color perceived by the human eye, the WB module 435 adjusts the color values of the image so that the captured image properly reflects the color perceived by the human eye.
[0055] The statistics engine 465 collects image data at various stages of the CIPU processing pipeline 485. For example, FIG. 4 shows the statistics engine 465 collecting image data after the LS module 425, the demosaicing module 430, and the WB module 435. Various embodiments collect data from any number of various stages of the CIPU processing pipeline 485. The statistics engine 465 processes the collected data and adjusts the operation of the camera sensors 405a and 405b through the controller module 475 and the sensor module 415 based on the processed data. Examples of such operations include exposure and focus. Although FIG. 4 shows the statistics engine 465 controlling the camera sensors 405a and 405b through the controller module 475, other embodiments of the statistics engine 465 control the camera sensors just through the sensor module 415.
[0056] The processed data may also be used to adjust the operation of various modules of the CIPU 400. For example, the statistics engine 465 of some embodiments adjusts the operation of the WB module 435 based on data collected after the WB module 435. In some of such embodiments, the statistics engine 465 provides an automatic white balance (AWB) function by adjusting the white balance operation of the WB module 435 using the processed data. In other embodiments, the processed data collected from any number of stages of the CIPU processing pipeline 485 may be used to adjust the operation of any number of modules in the CIPU processing pipeline 485. Additionally, the statistics engine 465 may receive instructions from the controller module 475 to adjust the operation of one or more modules of the CIPU processing pipeline 485.
[0057] After receiving the image from the WB module 435, the gamma module 440 performs a gamma correction operation on the image to encode and decode the luminance or tristimulus values of the camera system. In some embodiments, the gamma module 440 corrects the gamma by converting a 10-12 bit linear signal into an 8 bit non-linear encoded signal to correct the gamma of the image. In some embodiments, the gamma is corrected by using a look-up table.
[0058] The CSC module 445 converts the image received from the gamma module 440 from one color space to another. Specifically, the CSC module 445 converts the image from the RGB color space to a luma and chroma (YUV) color space. However, other embodiments of the CSC module 445 can convert the image from any number of color spaces to any number of color spaces.
[0059] HSC module 450 may adjust the hue, saturation, contrast, or any combination thereof, of the images received from CSC module 445. HSC module 450 may adjust these properties, for example, to reduce noise or enhance the image. For example, the saturation of an image captured by a low noise camera sensor may be increased to make the image appear sharper. In contrast, the saturation of an image captured by a high noise camera sensor may be decreased to reduce color noise in such images.
[0060] After the HSC module 450, a scaler module 455 can resize the image to adjust the pixel resolution of the image or to adjust the data size of the image. The scaler module 455 may, for example, reduce the size of the image to fit a smaller display. The scaler module 455 can scale the image in a number of different ways. For example, the scaler module 455 can enlarge (i.e., stretch) or reduce (i.e., shrink) the image. The scaler module 455 can also scale the image proportionally or scale the image anamorphically.
[0061] The filter module 460 applies one or more filter operations to the image received from the scaler module 455 to modify one or more attributes of some or all of the pixels in the image. Example filters include low pass filters, high pass filters, band pass filters, bilateral filters, and Gaussian filters, among other examples. As such, the filter module 460 can apply any number of different filters to the image.
[0062] The controller module 475 of some embodiments is a microcontroller that controls the operation of the CIPU 400. In some embodiments, the controller module 475 controls (1) the operation of the camera sensor through the sensor module 415 (e.g., exposure levels), (2) the operation of the CIPU processing pipeline 485, (3) the timing of the CIPU processing pipeline 485 (e.g., when to switch camera sensors, when to switch registers, etc.), and (4) the flash / strobe (not shown) that is part of the dual camera handheld device of some embodiments.
[0063] Some embodiments of the controller module 475 process instructions received from the statistics engine 465 and the CIPU driver 480. In some embodiments, the instructions received from the CIPU driver 480 are instructions from the dual camera mobile device (i.e., received from a local device), while in other embodiments, the instructions received from the CIPU driver 480 are instructions from another device (e.g., a remote control during a video conference). The controller module 475 can adjust the operation of the CIPU 400 by programming the values of the registers 470 based on the processed instructions. Additionally, the controller module 475 can dynamically reprogram the values of the registers 470 during operation of the CIPU 400.
[0064] As shown in Figure 4, CIPU 400 includes multiple modules in CIPU processing pipeline 485. However, those skilled in the art will appreciate that CIPU 400 may be implemented with only a few of the described modules or with different additional modules. Furthermore, the processing performed by different modules may be applied to an image in an order different from that shown in Figure 4.
[0065] An exemplary operation of the CIPU 400 will now be described with reference to Figure 4. For purposes of illustration, a set of registers Ra is used to process images captured by the camera sensor 405a of the dual camera mobile device, and a set of registers Rb is used to process images captured by the camera sensor 405b of the dual camera mobile device. The controller module 475 receives instructions from the CIPU driver 480 to generate images captured by one of the cameras of the dual camera mobile device.
[0066] The controller module 475 then initializes the various modules of the CIPU processing pipeline 485 to process an image captured by one of the cameras of the dual camera mobile device. In some embodiments, this involves the controller module 475 making sure that the appropriate set of registers of the registers 470 is being used. For example, if the CIPU driver 480 instructs the controller module 475 to generate an image captured by the camera sensor 405a, the controller module 475 makes sure that the set of registers Ra is the set of registers that the modules of the CIPU 400 will read. If not, the controller module 475 switches the set of registers so that the set of registers Ra is the set that will be read by the modules of the CIPU 400.
[0067] For each module of CIPU processing pipeline 485, a mode of operation is indicated by a value stored in a set of registers Ra. As previously mentioned, the values in the set of registers 470 may be dynamically reprogrammed during operation of CIPU 400. Thus, the processing of one image may differ from the processing of the next image. Although this description of the exemplary operation of CIPU 400 describes each module of CIPU 400 reading a value stored in a register that indicates the module's mode of operation, in some software-implemented embodiments, parameters are instead passed to the various modules of CIPU 400.
[0068] In some embodiments, the controller module 475 initializes the sensor module 415 by instructing the sensor module 415 to delay a particular amount of time after reading out an image from the pixel array 410a. In other words, the controller module 475 instructs the sensor module 415 to read out images from the pixel array 410a at a particular rate.
[0069] The controller module 475 then commands the camera sensor 405a through the sensor module 415 to capture an image. In some embodiments, the controller module 475 also provides the camera sensor 405a with exposure and other camera operation parameters. In other embodiments, the camera sensor 405a uses default values for the camera sensor operation parameters. The camera sensor 405a captures a RAW image that is stored in the pixel array 410a based on the parameters. The sensor module 415 reads the RAW image from the pixel array 410a and sends the image to the line / frame buffer 417 for storage before the CIPU processing pipeline 485 processes the image.
[0070] In certain circumstances, images may be discarded by the line / frame buffer 417. If the camera sensors 405a and / or 405b are capturing images at a high rate, the sensor module 415 may receive and store images in the line / frame buffer 417 faster than the BPC module 420 can read images from the line / frame buffer 417 (e.g., capturing high frame rate video), and the line / frame buffer 417 may become full. When this occurs, the line / frame buffer 417 of some embodiments discards images (i.e., frames) on a first-in, first-out basis. That is, when the line / frame buffer 417 discards an image, it discards the image that it received before all other images in the line / frame buffer 417.
[0071] The processing of an image by the CIPU processing pipeline 485 begins with the BPC module 420 reading the image from the line / frame buffer 417 to correct the bad pixels in the image. The BPC module 420 then sends the image to the LS module 425 to correct the uneven illumination of the image. After the illumination of the image is corrected, the LS module 425 sends the image to the demosaicing module 430, which processes the RAW image to generate an RGB image from the RAW image. The WB module 435 then receives the RGB image from the demosaicing module 430 and adjusts the white balance of the RGB image.
[0072] As mentioned above, the statistics engine 465 may collect some data at various points in the CIPU processing pipeline 485. For example, the statistics engine 465 collects data after the LS module 425, the demosaicing module 430, and the WB module 435, as shown in FIG. 4. Based on this collected data, the statistics engine 465 can adjust the operation of the camera sensor 405a, the operation of one or more modules of the CIPU processing pipeline 485, or both, to adjust the capture of subsequent images from the camera sensor 405a. For example, based on the collected data, the statistics engine 465 may determine that the exposure level of the current image is too low, and therefore instruct the camera sensor 405a through the sensor module 415 to increase the exposure level for subsequently captured images. Thus, the statistics engine 465 of some embodiments operates as a feedback loop for some processing operations.
[0073] After adjusting the white balance of the image, the WB module 435 sends the image to the gamma module 440 for gamma correction (e.g., to adjust the gamma curve of the image). The CSC module 445 receives the gamma-corrected image from the gamma module 440 and performs color space conversion. In this example, the CSC module 445 converts the RGB image to a YUV image. In other words, the CSC module 445 converts the image represented in the RGB color space to an image represented in the YUV color space. The HSC module 450 receives the YUV image from the CSC module 445 and adjusts the hue, saturation, and contrast attributes of various pixels of the image. After the HSC module 450, the scaler module 455 resizes the image (e.g., stretches or shrinks the image). The filter module 460 applies one or more filters to the image after receiving the image from the scaler module 455. Finally, the filter module 460 sends the processed image to the CIPU driver 480.
[0074] In this example of the operation of CIPU 400 described above, each module of CIPU processing pipeline 485 processed an image in some manner. However, other images processed by CIPU 400 may not require processing by all modules of CIPU processing pipeline 485. For example, an image may not require white balancing, gamma correction, scaling, or filtering. Thus, CIPU 400 can process an image in any of a number of ways based on various received inputs, such as instructions from CIPU driver 480 or data collected by statistics engine 465.
[0075] Different embodiments control the speed at which images are processed (i.e., frame rate) differently. One way to control the frame rate is through manipulation of the vertical blanking interval (VBI). In some embodiments that read out image lines to process images line by line, the VBI is the time difference between reading out the last line of an image of a video captured by the camera of the dual camera mobile device from the pixel array and reading out the first line of the next image of the video from the pixel array. In other embodiments, the VBI is the time difference between reading out one image of a video captured by the camera of the dual camera mobile device from the pixel array and reading out the next image of the video from the pixel array.
[0076] One example where the VBI can be used is between the sensor module 415 and the pixel arrays 410a and 410b. For example, some embodiments of the sensor module 415 read images from the pixel arrays 410a and 410b line by line, while other embodiments of the sensor module 415 read images from the pixel arrays 410a and 410b image by image. Thus, the frame rate can be controlled by adjusting the VBI of the sensor module 415. Increasing the VBI decreases the frame rate, and decreasing the VBI increases the frame rate.
[0077] 1. Using VBI for Single Camera: Frame Rate Control FIG. 5 conceptually illustrates examples of different frame rates 505, 510, and 515 based on different VBIs. Each sequence shows an image of a person with a guitar at various times 525-555 along a timeline 520 captured by one of the cameras of a dual camera mobile device. Furthermore, the time between each time 525-555 is the same and is referred to as one unit of time. For purposes of explanation, FIG. 5 will now be described with reference to the sensor module 415 and pixel array 410a of FIG. 4. Thus, each image represents a time along the timeline 520 at which the sensor module 415 reads out an image from the pixel array 410a.
[0078] In the exemplary frame rate 505, the VBI of the sensor module 415 for the pixel array 410a is set (e.g., by the controller module 475) to three time units. That is, the sensor module 415 reads images from the pixel array 410a every three time instances along the timeline 520. As shown in the exemplary frame rate 505, the sensor module 415 reads images at times 525, 540, and 555. Thus, the exemplary frame rate 505 has a frame rate of one image every three time units.
[0079] The exemplary frame rate 510 is similar to the exemplary frame rate 505, except that the VBI is set to two units of time. Thus, the sensor module 415 reads images from the pixel array 410a every two time instants along the timeline 520. The exemplary frame rate 510 shows the sensor module 415 reading images from the pixel array 410a at times 525, 535, 545, and 555. Because the VBI of the exemplary frame rate 510 is smaller than the VBI of the exemplary frame rate 505, the frame rate of the exemplary frame rate 510 is higher than the frame rate of the exemplary frame rate 505.
[0080] The exemplary frame rate 515 is similar to the exemplary frame rate 505, except that the VBI of the sensor module 415 for the pixel array 410a is set to one unit time. Thus, the sensor module 415 is instructed to read out images from the pixel array 410a every time along the timeline 520. As shown, the sensor module 415 reads out images from the pixel array 410a at times 525-555. The VBI of the exemplary frame rate 515 is less than the VBI of the exemplary frame rates 505 and 510. Thus, the frame rate of the exemplary frame rate 515 is greater than the exemplary frame rates 505 and 510.
[0081] 2. Using VBI for two cameras In some embodiments, it may be desirable to operate both cameras of a dual camera mobile device simultaneously (e.g., to transmit video from both cameras during a video conference). Different embodiments of a dual camera mobile device that includes a single processing pipeline provide different mechanisms for operating both cameras of the dual camera mobile device simultaneously.
[0082] One such mechanism is to interleave the processing of images captured by both cameras by controlling the VBI of each camera, i.e., one or more images captured by one camera are captured and processed during the VBI of the other camera, and one or more images captured by the other camera are captured and processed during the VBI of one camera. This mechanism may be implemented in the CIPU 400 of some embodiments, since the CIPU 400 described above has a single processing pipeline 485. In such an embodiment, the sensor module 415 reads out an image from one of the pixel arrays 410a and 410b, and the read out image is processed by the CIPU 400 during the VBI for the other pixel array of the sensor module 415.
[0083] The VBI of the sensor module 415 for each pixel array may be set to a specific value. However, in some embodiments, the VBI is not set to a value smaller than the time it takes the CIPU 400 to read and process one image. In some embodiments, the VBI of the sensor module 415 for each pixel array is set to the same value. For example, the sensor module 415 reads images alternately from the pixel arrays 410a and 410b, where the VBI of the sensor module 415 for each pixel array is set to the same value. In other embodiments, the VBI of the sensor module 415 for each pixel array is set to a different value. In some of such embodiments, the VBI of the sensor module 415 for one pixel array is set to a multiple of the VBI of the sensor module 415 for the other pixel array. For example, the VBI of the sensor module 415 for one pixel array is set to 2 units of time, and the VBI of the sensor module 415 for the other pixel array is set to 4 units of time. In this example, the sensor module 415 reads out two images from one pixel array for every image the sensor module 415 reads out from the other pixel array.
[0084] Figure 6 conceptually illustrates examples of different interleaved frame rates 605, 610, and 615 based on different VBIs. Figure 6 is similar to Figure 5, except that Figure 6 includes thirteen times 625-685 along a timeline 620. Additionally, an image of a person holding a guitar represents times along the timeline 620 at which images are read out from one pixel array, while an image of a person wearing a college cap (i.e., mortarboard) represents times along the timeline 620 at which images are read out from the other pixel array.
[0085] For purposes of explanation, assume that the image of the person holding the guitar is captured by camera sensor 405a of the dual camera mobile device, and assume that the image of the person wearing the college cap is captured by camera sensor 405b of the dual camera mobile device. Additionally, Figure 6 will now be described with reference to sensor module 415 and pixel arrays 410a and 410b of Figure 4.
[0086] In the exemplary interleaved frame rate 605, the VBI of the sensor module 415 for both pixel array 410a and pixel array 410b is set to two units of time. As shown in the exemplary interleaved frame rate 605, the sensor module 415 reads images from pixel array 410a at times 625, 635, 645, 655, 665, 675, and 685 along the timeline 620, and the sensor module 415 reads images from pixel array 410b at times 630, 640, 650, 660, 670, and 680 along the timeline 620. That is, the sensor module 415 alternately reads images from the pixel arrays every unit of time.
[0087] The exemplary interleaved frame rate 610 is similar to the exemplary interleaved frame rate 605, except that the VBI of the sensor module 415 for both pixel array 410a and pixel array 410b is set to 4 time units. The exemplary interleaved frame rate 610 shows the sensor module 415 reading images from pixel array 410a at times 625, 645, 665, and 685 along the timeline 620, and the sensor module 415 reading images from pixel array 410b at times 635, 655, and 675 along the timeline 620. Because the VBI of the exemplary interleaved frame rate 610 is higher than the VBI of the exemplary interleaved frame rate 605, the frame rate of the exemplary interleaved frame rate 610 is less than the frame rate of the exemplary interleaved frame rate 605.
[0088] The exemplary interleaved frame rate 615 is similar to the exemplary interleaved frame rate 605, except that the VBI of the sensor module 415 for both pixel array 410a and pixel array 410b is set to 6 time units. As shown in FIG. 6, the sensor module 415 reads images from pixel array 410a at times 625, 655, and 685 along the timeline 620, and the sensor module 415 reads images from pixel array 410b at times 640 and 670 along the timeline 620. The VBI of the exemplary interleaved frame rate 615 is greater than the VBI of the exemplary interleaved frame rates 605 and 610. Thus, the frame rate of the exemplary interleaved frame rate 615 is less than the exemplary interleaved frame rates 605 and 610.
[0089] B. Multiple Pipelines 7 conceptually illustrates a captured image processing unit (CIPU) 700 of some embodiments. CIPU 700 performs the same functions as CIPU 400 described above, except that CIPU 700 is implemented with two front-end processing pipelines, a storage device, and a back-end processing pipeline instead of a single processing pipeline. Therefore, a description of the functionality of CIPU 700 will be described with reference to the modules of CIPU 400.
[0090] As shown, CIPU 700 includes a front-end processing pipeline 715 for camera sensor 405a and pixel array 410a, a front-end processing pipeline 720 for camera sensor 405b and pixel array 410b, a memory device 725, a controller module 730, and a back-end processing pipeline 735. In some embodiments, camera sensors 405a and 405b are the cameras of a dual camera mobile device.
[0091] The front-end processing pipelines 715 and 720 of some embodiments perform a portion of the image processing of the CIPU 400. Thus, different embodiments may include a different number of modules of the CIPU 400. For example, the front-end processing pipelines 715 and 720 of some embodiments each include the sensor module 415, the BPC module 420, the LS module 425, the demosaicing module 430, the WB module 435, and the statistics engine 465 of the CIPU 400.
[0092] Although front-end processing pipelines 715 and 720 perform the same type of image processing by having the same modules, each module of each of front-end processing pipelines 715 and 720 may be configured differently with different register values, as described above with respect to CIPU 400. Furthermore, because each of camera sensors 405a and 405b has its own front-end processing pipeline, front-end processing pipelines 715 and 720 can process images independently of each other. For example, front-end processing pipelines 715 and 720 can process images in parallel (i.e., simultaneously) at different times and at different speeds.
[0093] In some embodiments, each of the front-end processing pipelines 715 and 720 can read images from its corresponding camera sensor and pixel array. For example, front-end processing pipeline 715 reads images captured by camera sensor 405a from pixel array 410a, and front-end processing pipeline 720 receives images captured by camera sensor 405b from pixel array 410b. When one of the front-end processing pipelines 715 and 720 reads an image from its corresponding camera sensor and pixel array, that front-end processing pipeline processes the image and sends the processed image to storage device 725. Similarly, each of the front-end processing pipelines 715 and 720 communicates with the controller module 730 (e.g., through the statistics engine of each front-end processing pipeline) as described above.
[0094] The storage 725 of some embodiments stores partially processed images for the back-end processing pipeline 735 to complete processing. In these embodiments, the storage 725 receives partially processed images from the front-end processing pipelines 715 and 720 and sends the partially processed images to the back-end processing pipeline 735. In some embodiments, the storage 725 is implemented as a volatile storage device (e.g., random access memory (RAM)), while in other embodiments, the storage 725 is implemented as a non-volatile storage device (e.g., flash memory, hard disk, optical disk, etc.). Furthermore, the storage 725 of some embodiments is an internal storage device (e.g., RAM), while the storage 725 of other embodiments is an external storage device (e.g., Compact Flash (CF) card, Secure Digital (SD) card, etc.).
[0095] Some embodiments of the back-end processing pipeline 735 perform a portion of the image processing of the CIPU 700. In some embodiments, the back-end processing pipeline 735 includes modules of the CIPU 400 that the front-end processing pipelines 715 and 720 do not include. For example, referring to the above example, the back-end processing pipeline 735 would include the CSC module 445, the gamma module 440, the HSC module 450, the scaler module 455, and the filter module 460 of the CIPU 400. Thus, the back-end processing pipeline 735 of such an embodiment performs the remaining image processing of the CIPU 400 that the front-end processing pipelines 715 and 720 do not perform. Thus, the back-end processing pipeline 735 reads the partially processed image from the storage device 725 and performs the remaining image processing on the partially processed image. After processing the image, the back-end processing pipeline 735 sends the processed image to the CIPU driver 480.
[0096] The controller module 730 performs the same functions as described above with respect to Figure 4. As shown in Figure 7, the controller module 730 interacts with front-end processing pipelines 715 and 720 as well as a back-end processing pipeline 735. In some embodiments, the controller module 730 is included in the back-end processing pipeline 735, while in other embodiments, the controller module 730 is included in one of the front-end processing pipelines 715 and 720.
[0097] The operation of CIPU 700 will now be described with respect to camera sensors 405a and 405b, pixel arrays 410a and 410b, front-end processing pipelines 715 and 720, storage 725, and back-end processing pipeline 735 shown in Figure 7. When one of front-end processing pipelines 715 and 720 reads an image from its corresponding camera sensor and pixel array, that front-end processing pipeline processes the image and sends the partially processed image to storage 725. For example, front-end processing pipeline 715 may read an image captured by camera sensor 405a from pixel array 410a, or front-end processing pipeline 720 may read an image captured by camera sensor 405b from pixel array 410b. As previously mentioned, each front-end processing pipeline 715 and 720 can process images in parallel.
[0098] The back-end processing pipeline 735 reads the partially processed image from the storage device 725 and processes the partially processed image to complete image processing for the image. In some embodiments, the back-end processing pipeline 735 reads and processes the images stored in the storage device 725 on a first-in, first-out basis. In other words, a particular image in the storage device 725 is processed after all images received and stored in the storage device 725 before the particular image, but the particular image is processed before images received and stored in the storage device 725 after the particular image. After the back-end processing pipeline 735 processes an image, it sends the processed image to the CIPU driver 480.
[0099] 8 conceptually illustrates another captured image processing unit (CIPU) 800 of some embodiments. CIPU 800 performs the same functions as CIPU 400 described above, except that CIPU 800 is implemented with two separate processing pipelines, with each camera sensor having its own separate processing pipeline. Accordingly, a description of the functionality of CIPU 800 will be described with reference to the modules of CIPU 400.
[0100] As shown, CIPU 800 includes a processing pipeline 815 for camera sensor 405a and pixel array 410a, and a processing pipeline 820 for camera sensor 405b and pixel array 410b. Each of the processing pipelines 815 and 820 of some embodiments includes all of the modules included in CIPU 400. Thus, the operation of each of the processing pipelines 815 and 820 of these embodiments is the same as the operation of CIPU 400.
[0101] Because each of camera sensors 405a and 405b has its own processing pipeline, processing pipelines 815 and 820 can process images independently of one another. For example, processing pipelines 815 and 820 can process images in parallel (i.e., simultaneously) at different times and at different rates. Furthermore, each of the processing pipelines 815 and 820 of some embodiments can be configured differently through different register values, as described above with reference to CIPU 400.
[0102] In some embodiments, some modules of the CIPU 400 include one or more line / frame buffers to perform some or all of the module's operations. For example, the filter processing module 460 of some embodiments is implemented to perform a 3×3 low-pass filter. In such an embodiment, the 3×3 low-pass filter processes three consecutive lines in an image, such as applying a 3×3 low-pass filter to the middle line of the three consecutive lines. Thus, the filter processing module 460 of such an embodiment requires at least three line / frame buffers to perform a 3×3 low-pass filter. Other modules of the CIPU 400 also include one or more line / frame buffers, such as the BPC module 420 and the LS module 425.
[0103] Each of the processing pipelines of the CIPU 800 may have a different line / frame buffer size to customize image processing to the characteristics of its corresponding camera. For example, if one camera of a dual camera mobile device has a 2048×1500 pixel sensor, the processing pipeline for the 2048×1500 pixel sensor may include a line / frame buffer that is 2048 pixels wide. Similarly, if the other camera of the dual camera mobile device has a 640×480 pixel sensor, the processing pipeline for the 640×480 pixel sensor may include a line / frame buffer that is 640 pixels wide. That is, the size of the line / frame buffer included in the module of one processing pipeline may be different from the size of the line / frame buffer included in the module of another processing pipeline.
[0104] III. Videoconferencing A. Videoconferencing Architecture 9 conceptually illustrates a software architecture for a videoconferencing and processing module 900 of a dual camera mobile device of some embodiments. The videoconferencing and processing module 900 includes a CIPU driver 905, a media exchange module 910, and an encoder driver 920, which are similar to the corresponding modules and drivers 305, 310, and 320 described above with respect to FIG. 3. The videoconferencing and processing module 900 also includes a videoconferencing module 925, a videoconferencing client 945, and a network interface 950 for performing various videoconferencing functions. Like the video processing and encoding module 300, the videoconferencing and processing module 900 processes and encodes images captured from the cameras of the dual camera mobile device.
[0105] As described above with respect to FIG. 3, by using the media exchange module 910, consumers and producers of media content within the device exchange media content and instructions for processing the media content, the CIPU driver 905 serves as a communication interface with a captured image processing unit (CIPU) 955, and the encoder driver 920 serves as a communication interface with encoder hardware 960 (e.g., an encoder chip, an encoding component on a system-on-chip, etc.).
[0106] The videoconferencing module 925 of some embodiments handles various videoconferencing functions such as image processing, videoconference management, and networking. As shown, the videoconferencing module 925 interacts with the media exchange module 910, the videoconferencing client 945, and the network interface 950. In some embodiments, the videoconferencing module 925 receives instructions from and sends instructions to the videoconferencing client 945. The videoconferencing module 925 of some embodiments also transmits data to and receives data from a network (e.g., a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a network of networks, a code division multiple access (CDMA) network, a GSM network, etc.) through the network interface 950.
[0107] The videoconferencing module 925 includes an image processing layer 930, a management layer 935, and a network layer 940. In some embodiments, the image processing layer 930 performs image processing operations on the videoconference images. For example, the image processing layer 930 of some embodiments performs exposure adjustment, image resizing, perspective correction, and dynamic range adjustment, as described in more detail below. The image processing layer 930 of some embodiments transmits requests for images from the CIPU 955 through the media exchange module 910.
[0108] The management layer 935 of some embodiments controls the operation of the videoconferencing module 925. For example, in some embodiments, the management layer 935 initializes the camera / cameras of a dual camera mobile device, processes images and audio for transmission to a remote device, and processes images and audio received from a remote device. In some embodiments, the management layer 935 generates composite (e.g., PIP) displays for the devices. Additionally, the management layer 935 may modify the operation of the videoconferencing module 925 based on networking reports received from the network layer 940.
[0109] In some embodiments, the network layer 940 performs some or all of the networking functions for the video conference. For example, the network layer 940 of some embodiments establishes a network connection (not shown) between the video conference dual camera mobile device and a remote device, transmits images to the remote device, and receives images from the remote device, among other functions, as described below. In addition, the network layer 940 receives networking data such as packet loss, one-way latency, and round trip delay time, among other types of data, processes such data, and reports this data to the management layer 935.
[0110] In some embodiments, the videoconferencing client 945 is an application that may use the videoconferencing functionality of the videoconferencing module 925, such as a videoconferencing application, a Voice Over IP (VOIP) application (e.g., Skype), or an instant messaging application. In some embodiments, the videoconferencing client 945 is a standalone application, while in other embodiments, the videoconferencing client 945 is integrated into another application.
[0111] In some embodiments, network interface 950 is a communications interface that enables videoconferencing module 925 and videoconferencing client 945 to send data and receive data over a network (e.g., a cellular network, a local area network, a wireless network, a network of networks, the Internet, etc.) through network interface 950. For example, if videoconferencing module 925 wants to send data (e.g., an image captured by the camera of a dual camera mobile device) to another device over the Internet, videoconferencing module 925 transmits the image to the other device through network interface 950.
[0112] B. Videoconferencing Setup 10 conceptually illustrates an exemplary videoconference request message sequence 1000 of some embodiments. The diagram illustrates a videoconference request message sequence 1000 between a videoconference client 1010 running on a device 1005, a videoconference server 1015, and a videoconference client 1025 running on a device 1020. In some embodiments, the videoconference clients 1010 and 1025 are the same as the videoconference client 945 illustrated in FIG. 9. As illustrated in FIG. 10, one device (i.e., device 1005) requests a videoconference and another device (i.e., device 1020) responds to such a request. The dual camera mobile device described in this application can perform both operations (i.e., make a request and respond to a request).
[0113] The videoconference server 1015 of some embodiments routes messages between multiple videoconference clients. In some embodiments, the videoconference server 1015 is implemented on one computing device, while in other embodiments, the videoconference server 1015 is implemented on multiple computing devices. In some embodiments, the videoconference server is a publicly accessible server that can handle and route messages for multiple conferences at once. Each of the videoconference clients 1010 and 1025 of some embodiments communicates with the videoconference server 1015 over a network (e.g., a cellular network, a local area network, a wireless network, a network of networks, the Internet, etc.) through a network interface, such as the network interface 950 described above.
[0114] The videoconference request message sequence 1000 of some embodiments begins when the videoconference client 1010 receives (in action 1) a request to start a videoconference with the device 1020 from a user of the device 1005. The videoconference client 1010 of some embodiments receives the videoconference start request when a user of the device 1005 selects a user interface (UI) item in a user interface displayed on the device 1005. Examples of such user interfaces are shown in Figures 11 through 14, described below.
[0115] After the videoconference client 1010 receives the request, it sends a videoconference request to the videoconference server 1015 (in operation 2) that indicates the device 1020 as a recipient based on the input from the user. The videoconference server 1015 forwards the videoconference request to the videoconference client 1025 of the device 1020 (in operation 3). In some embodiments, the videoconference server 1015 forwards the videoconference request to the videoconference client 1025 using push technology. That is, the videoconference server 1015 begins transmitting the videoconference request to the videoconference client 1025 upon receiving it from the videoconference client 1010, rather than waiting for the client 1025 to send any message request.
[0116] When the videoconference client 1025 of some embodiments receives the videoconference request, a user interface is displayed on the device 1020 to indicate to the user of the device 1020 that the user of the device 1005 has sent a videoconference start request and to prompt the user of the device 1020 to accept or reject the videoconference request. An example of such a user interface is shown in FIG. 12, described below. In some embodiments, when the videoconference client 1025 receives a videoconference request accept request from the user of the device 1005 (in operation 4), the videoconference client 1025 sends a videoconference accept to the videoconference server 1015 (in operation 5). The videoconference client 1025 of some embodiments receives the videoconference request accept request, for example, when the user of the device 1020 selects a user interface item in the user interface shown in FIG.
[0117] After the videoconference server 1015 receives the videoconference acceptance from the videoconference client 1025, the videoconference server 1015 forwards the videoconference acceptance (at operation 6) to the videoconference client 1010. Some embodiments of the videoconference server 1015 forward the videoconference acceptance to the videoconference client 1010 using push technology, as described above.
[0118] Upon receiving the videoconference acceptance, in some embodiments, a videoconference is established (at act 7) between device 1005 and device 1020. Different embodiments establish the videoconference differently. For example, videoconference establishment in some embodiments includes negotiating a connection between device 1005 and device 1020, determining a bit rate at which to encode the video, and exchanging video between device 1005 and device 1020.
[0119] In the above example, the user of the device 1020 accepts the videoconference request. In some embodiments, the device 1020 may be configured (e.g., by a preference setting of the device) to automatically accept incoming videoconference requests without displaying a UI. Additionally, the user of the device 1020 may also reject the videoconference request (in act 4) (e.g., by selecting a user interface item in a user interface displayed on the device 1020). Instead of sending a videoconference accept, the videoconference client 1025 sends a videoconference rejection to the videoconference server 1015, which forwards the videoconference rejection to the videoconference client 1010. In that case, the videoconference is not established.
[0120] 1. Videoconferencing Setup User Interface In some embodiments, a videoconference is initiated based on an ongoing call. That is, while a user of a mobile device is engaged in a call with a second user, the user can turn the call into a videoconference, with the permission of the other party. In some embodiments of the present invention, FIG. 11 illustrates the initiation of such a videoconference with a dual camera handheld mobile device 1100. The figure illustrates the initiation of the videoconference as five operational stages 1110, 1115, 1120, 1125, and 1130 on a user interface ("UI") 1105 of the device 1100.
[0121] As shown in FIG. 11, UI 1105 includes a name field 1135, a selection menu 1140, and selectable UI items 1145. Name field 1135 displays the name of the person on the other end of the call to whom the user may wish to make a video conference request. In this example, selectable UI item 1145 (which may be implemented as a selectable button) provides a "end call" option that the user may select to end the call. Selection menu 1140 displays a menu of selectable UI items, such as speakerphone item 1142, mute item 1144, keypad item 1146, phone book item 1148, hold item 1152, and video conference item 1154. Different embodiments display the selection menu differently. In the embodiment illustrated by FIG. 11, selection menu 1140 includes several icons of equal size, each representing a different action. Other embodiments provide a scrollable menu or prioritize certain items (e.g., by making the items larger).
[0122] The operation of the UI 1105 will now be described with respect to its states in the five stages 1110, 1115, 1120, 1125, and 1130 shown in FIG. 11. In the first stage 1110, a call is established between the handheld portable device user and Nancy Jones. The second stage 1115 displays the UI 1105 after the user selects a selectable videoconferencing option 1154 (e.g., through a single finger tap with finger 1150) to activate a videoconferencing tool. In this example, the videoconferencing option 1154 (which may be implemented as a selectable icon) allows the user to initiate a videoconferencing session during a call. In the second stage, the videoconferencing option 1154 is highlighted to indicate that the videoconferencing tool has been activated. Different embodiments may indicate such a selection in different ways (e.g., by highlighting the item's border or text).
[0123] The third stage 1120 displays the UI 1105 after the device 1100 initiates the videoconference process by selecting the videoconference option 1154. The third stage is a transitional waiting stage while the device waits for the videoconference to be established (e.g., while the device waits for the other device to accept or decline the videoconference). In the third stage 1120, the user of the device 1100 can still talk to the user of the other device (i.e., Nancy Jones) while the videoconference connection is established. Additionally, according to some embodiments, the user of the device 1100 can cancel the videoconference request in the third stage 1120 by selecting a selectable UI item (not shown) displayed in the UI 1105 for canceling the videoconference request. During this waiting stage, different embodiments use different displays of the UI 1105 to indicate the waiting state.
[0124] As shown in FIG. 11, in some embodiments, the wait state of the third stage is shown as a full screen display of the video captured by the device 1100, with a "preview" notation at the bottom of this video. Specifically, in FIG. 11, the third stage 1120 indicates the start of the videoconferencing process by displaying in the display area 1160 of the UI 1105 a full screen presentation of the video being captured by the device's camera. In some embodiments, the front camera is the default camera selected by the device at the start of the videoconference. Often, this front camera points to the user of the device at the start of the videoconference. Thus, in the example shown in FIG. 11, the third stage 1120 shows the device 1100 as presenting a full screen video of the user of the device 1100. The wait state of the device is further highlighted by a "preview" indicator 1165 below the video that appears in the display area 1160 during the third stage 1120.
[0125] The transitional third wait stage 1120 may be represented differently in some embodiments. For example, some embodiments allow a user of the device 1100 to select the rear camera as the camera for initiating a video conference. To enable this selection, some embodiments allow the user to designate the rear camera as the default camera for initiating a video conference (e.g., via a menu preference) and / or the user can select the rear camera from a menu that displays the rear and front cameras after selecting the video conference option 1154. In either of these situations, the UI 1105 (e.g., display area 1160) displays the video captured by the rear camera during the third wait stage 1120.
[0126] Similarly, in other embodiments, activation of the videoconferencing tool may be indicated by displaying a miniature version of the video captured by device 1100, by displaying a still image stored on device 1100, by providing a message highlighting the waiting state of the device (e.g., by indicating "establishing conference"), by not displaying a "preview" indicator, etc. Similarly, in the third stage 1120, the UI 1105 of some embodiments provides an end button (not shown) to allow the user to cancel entering the videoconference and return to the call if the user decides not to enter the videoconference at this stage (e.g., while the user is waiting for the remote user to respond to the request).
[0127] A fourth stage 1125 illustrates the UI 1105 in a transitional state after the remote user accepts the videoconference request and the videoconference connection is established. In this transitional state, the size of the viewport 1160 displaying the local user's video (captured by the front camera in this example) is gradually decreased (i.e., gradually shrunk) as indicated by arrow 1175. The viewport 1160 (i.e., the local user's video) is shrunk so that the UI 1105 can display a viewport 1170 (e.g., a display window 1170) containing the video from the remote device's camera behind the viewport 1160. In other words, the local user's video 1160 is shrunk to generate a PIP view 1180 having a foreground inset view 1160 of the local user's video and a background main view 1170 of the remote user. In this example, background main display 1170 presents a video of a woman captured by a front camera of a remote device (e.g., Nancy Jones, a user of the remote device) or a woman whose video is captured by a rear camera of the remote device (e.g., a woman whose video is captured by Nancy Jones). Those skilled in the art will appreciate that the fourth stage of the transition shown in FIG. 11 is only one exemplary approach used by some embodiments, and that other embodiments may animate the fourth stage of the transition differently.
[0128] The fourth stage 1125 also shows selectable UI items 1132 in the bottom display area 1155. The selectable UI items 1132 (which may be implemented as selectable buttons) provide a selectable end meeting option 1132 below the PIP display 1180. The user may select the end meeting option 1132 (e.g., through a single finger tap) to end the videoconference. Different embodiments may cause the user to end the conference in different ways, such as by flipping a switch on a mobile device, by giving a voice command, etc. Additionally, different embodiments may cause the end meeting option 1132 to disappear during the videoconference, such that the PIP display 1180 occupies the entire display area 1185. A single finger tap on the bottom of the display area 1185 may then redisplay the end meeting option 1132 and provide the user with access to the end meeting option 1132. In some embodiments, the layout of the display area 1155 is the same as the display area 1155 described in more detail below.
[0129] The fifth stage 1130 shows the UI 1105 after the animation of the fourth transition state 1125 has finished. Specifically, the fifth stage 1130 shows a PIP display 1180 presented by the UI 1105 during a video conference. As mentioned above, this PIP display 1180 includes two video displays: a larger background display 1170 from a remote camera and a smaller foreground inset display 1160 from a local camera.
[0130] This PIP display 1180 is just one way of presenting a composite view of the video being captured by the remote and local devices. In addition to this composite view, some embodiment devices provide other composite views. For example, instead of having a larger background display 1170 relative to the remote user, the larger background display 1170 can be relative to the local user and a smaller foreground inset display 1160 can be relative to the remote user. As described further below, some embodiments allow users to switch between local and / or remote cameras during a video conference as the cameras for the inset and main views of the PIP display 1180.
[0131] Similarly, in some embodiments, the local and remote videos may be displayed in two side-by-side viewing areas (e.g., left-right or top-bottom viewing windows) or two diagonally aligned viewing areas in the UI 1105. As described further below, the manner in which the PIP is displayed or the default viewing mode may, in some embodiments, be specified by the user through a device preference setting or through a user-selectable control during the videoconference.
[0132] When the user of device 1100 of FIG. 11 invites a remote user to a videoconference, the remote user may accept or decline the invitation. FIG. 12 shows UI 1205 of remote user's device 1200 at six different stages 1210, 1215, 1220, 1225, 1230, and 1235 illustrating a sequence of operations for presenting and accepting a videoconference invitation at the remote user's device. The following description of UI 1205 refers to the user of device 1200 (i.e., the device that receives the videoconference request) as the invitation recipient and the user of device 1100 (i.e., the device that sends the videoconference request) as the invitation requester. Also, in this example, it is assumed that the invitation recipient's device 1200 is a dual camera device, like the invitation requester's device. However, in other examples, one or both of these devices are single camera devices.
[0133] A first stage 1210 shows the UI 1205 when the invitation recipient receives an invitation to a videoconference from an invitation requester, John Smith. As shown in FIG. 12, the UI 1205 at this stage includes a name field 1235, a message field 1240, and two selectable UI items 1245 and 1250. The name field 1235 displays the name of the person requesting the videoconference. In some embodiments, the name field 1235 displays the phone number of the person requesting the videoconference instead of the person's name. The message field 1240 displays an invitation from the invitation requester to the invitation recipient. In this example, the "Videoconference invitation" in the field 1240 indicates that the invitation requester is requesting a videoconference with the invitation recipient. The selectable UI items 1245 and 1250 (which may be implemented as selectable buttons) provide selectable request reject option 1245 and request accept option 1250 for the invitation recipient to use to reject or accept the invitation. Different embodiments may display these options differently and / or display other options.
[0134] When the invitation recipient sees the "Video Conference Invite" notation displayed in the message field 1240, they can reject or accept the request by selecting a reject request option 1245 or an accept request option 1250, respectively, in the UI. A second step 1215 shows the user selecting the accept request option 1250 in the example shown in FIG. 12. In this example, the selection is made by the user's finger tapping the accept request option 1250, which is indicated by a highlighting of the option 1250. In some embodiments, other techniques (e.g., double tapping, etc.) are provided for selecting the accept request option 1245 or the reject request option 1250 to indicate a selection (e.g., highlighting the border or text of the UI item).
[0135] A third stage 1220 shows the UI 1205 after the invitee has agreed to join the videoconference. At this stage, the UI 1205 enters a preview mode in which a full-screen presentation of video from the front camera of the remote device is shown in the viewport 1244. The front camera is, in this case, pointed at the user of the remote device (i.e., Nancy Jones in this example). Thus, in this preview mode, her image is shown. This preview mode allows the invitee to ensure that their video appears properly and that they are satisfied with their appearance before the videoconference begins (e.g., before the actual transmission of video begins). In some embodiments, a notation such as a "Preview" notation may be displayed below the viewport 1244 to indicate that the invitee is in preview mode.
[0136] As described further below, some embodiments allow the invitee to select the rear camera as the default camera when initiating a videoconference, or to select the front or rear camera at the beginning of the videoconference. In other embodiments, a preview view of the invitee is displayed differently (e.g., in a smaller image placed in the corner of the viewing area 1244). Still other embodiments do not include this preview mode, and the videoconference begins immediately after the invitee accepts the request.
[0137] In a third stage, the UI 1205 shows two selectable UI items 1275 and 1246, one overlapping the display area 1244 and the other below the display area 1244. Selectable UI item 1275 is an accept button 1275 that the user may select to start the videoconference. Selectable UI item 1246 is an end button 1246 that the invite recipient may select if they decide not to join the videoconference at this stage.
[0138] A fourth stage 1225 shows the UI 1205 after the invitation recipient has selected the accept button 1275. In this example, the accept button 1275 is highlighted to indicate that the invitation recipient is ready to begin the videoconference. Such a selection may be indicated differently in other embodiments.
[0139] A fifth stage 1230 shows the UI 1205 in a transitional state after the invitee accepts the video conference request. In this transitional stage, the viewport 1244 displaying the invitee's video (captured by the front camera in this example) is gradually reduced in size (i.e., gradually shrinks) as indicated by arrow 1260. The invitee's video shrinks so that the UI 1205 can display a viewport 1265 (e.g., a display window 1265) that includes video from the inviter's camera behind the viewport 1244. In other words, the invitee's video shrinks to generate a PIP display 1280 having a foreground inset viewport 1244 of the invitee's video and a background main display 1265 of the inviter.
[0140] In this example, the background main display 1265 presents a video of a man (i.e., John Smith, a user of the local device 1100) whose video is captured by a front camera of the local device. In another example, the video can be a video of a man whose video is captured by a rear camera of the local device (e.g., a video captured by John Smith). Different embodiments can animate this fifth stage of the transition differently.
[0141] The UI in the fifth stage 1230 also displays a display area 1155 (e.g., a toolbar or menu bar) that includes a selectable UI item 1285 (e.g., a mute button 1285) for muting other users' audio during the videoconference, a selectable UI item 1287 (e.g., an end conference button 1287) for ending the videoconference, and a selectable UI item 1289 (e.g., a switch camera button 1289) for switching cameras, which will be described in more detail below. Thus, the invitation recipient can select any of the selectable UI items 1285-1289 (e.g., by a single finger tap) to perform a desired action during the videoconference. Different embodiments allow the invitation recipient to perform any of the actions in different ways, such as, for example, by toggling a switch on a mobile device, by giving a voice command, etc.
[0142] 12 shows an example layout of the display area 1155, some embodiments provide different layouts of the display area 1155, such as the layout of the display area 1155 of FIG. 11 that includes only an end meeting selectable UI item 1132 for ending the videoconference. Other layouts of the display area 1155 can include any number of different selectable UI items for performing different functions. Additionally, the fifth stage 1230 shows the display area 1155 displayed at the bottom of the UI 1205. Different embodiments of the display area 1155 may be displayed in different locations within the UI 1205 and / or defined as different shapes.
[0143] FIG. 12 illustrates the viewport 1155 as a static viewport (i.e., the viewport 1155 is always displayed). However, in some embodiments, the viewport 1155 is a dynamic viewport. In some such embodiments, the viewport 1155 is not normally displayed. Rather, the viewport 1155 is displayed when a triggering event is received (e.g., a user selection to tap the viewport 1280 once in this manner, a voice command, etc.). The viewport 1155 is hidden when a user selection is received (e.g., selecting the selectable mute UI item 1285) or after a defined amount of time (e.g., 3 seconds), which may be specified by the user through a preference setting of the mobile device or videoconferencing application. In some such embodiments, the viewport 1155 is automatically displayed after the videoconference begins and is hidden in the same manner described above.
[0144] A sixth stage 1235 shows the UI 1205 after the animation of the fifth transition stage has ended. Specifically, the sixth stage shows a PIP display 1280 presented by the UI 1205 during a video conference. As described above, the PIP display 1280 includes two video displays: a larger background display 1265 from a local camera and a smaller foreground inset display 1244 from a remote camera. The PIP display 1280 is just one way of presenting a composite view of the videos being captured by the remote and local devices. In addition to this composite view, the devices of some embodiments provide other composite views. For example, instead of having a larger background display related to the invitation recipient, the larger background display can be related to the invitation requester and the smaller foreground inset display can be related to the invitation recipient. As described further below, some embodiments allow a user to control the inset view and main view of the PIP display to switchably display the local and remote cameras. Similarly, in some embodiments, the local and remote videos may be displayed in two side-by-side display areas (e.g., left and right display windows, or top and bottom display windows) or two diagonally aligned display areas within the UI 1205. As described further below, the manner or default display mode of the PIP display may, in some embodiments, be specified by the user through a device preference setting or through a user-selectable control during the videoconference.
[0145] 12 illustrates the sequence of operations for submitting and accepting a videoconference invitation as six distinct operational steps, in some embodiments the operations may be performed in fewer steps. For example, some such embodiments may omit presenting the third step 1220 and the fourth step 1225, and proceed from the second step 1215 to the fifth step 1230 after the user selects the request accept option 1250. In other embodiments that perform the operations (i.e., submitting and accepting a videoconference invitation) in fewer steps, the first step 1210 and the second step 1215 may be omitted, and the user may be presented with the third step 1220 when the invitation recipient receives the videoconference invitation from the invitation requester.
[0146] FIG. 13 illustrates an example of performing the operations shown in FIG. 12 in a fewer number of stages by combining stages 1 and 3 into one stage and stages 2 and 4 into one stage. Specifically, this figure illustrates the UI 1205 of the remote user's device 1200 in five different stages 1390, 1392, 1394, 1230, and 1235. The first stage 1390 is similar to stage 1110, except that the name field 1235 displays the name "John Smith" to indicate the name of the person on the other end of the call. That is, a call is established between the user of the remote mobile device and the user of the local device (i.e., John Smith in this example). The second stage 1392 and the third stage 1394 are similar to the first stage 1210 and the second stage 1215 of FIG. 12, except that the second stage 1392 and the third stage 1394 also show a preview of the user of the remote mobile device (i.e., Nancy Jones in this example). The fourth step 1230 and the fifth step 1235 are the same as the fifth step 1230 and the sixth step 1235 of FIG.
[0147] In addition to invoking the videoconferencing tool through a selectable option during a call, some embodiments allow a user of a dual camera device to directly initiate a videoconference without having to first make a call. Figure 14 illustrates another such alternative method of initiating a videoconference. The figure shows a UI 1405 in seven different stages 1410, 1415, 1420, 1425, 1430, 1435, and 1440 illustrating alternative sequences of operations for initiating a videoconference.
[0148] In a first step 1410, the user searches through the mobile device's contact list for people with whom the user wants to engage in a video conference, similar to how one might find a contact to call. In a second step 1415, the user selects a person 1455 with whom the user wants to videoconference (e.g., through a single finger tap 1460 on the person's name 1455). This selection triggers the UI 1405, which displays the contact's information and various user-selectable options. In this example, Johnson's name 1455 is highlighted to indicate that this is the person with whom the user wants to videoconference. Different embodiments may indicate such a selection in different ways. In the second step 1415, the user of the device 1400 can select a person with whom the user wants to videoconference through a contact list, although some embodiments allow the user to select a person through a "recent" call history, which lists specific numbers or names of people with whom the user of the device 1400 has recently videoconferenced or called.
[0149] In a third stage 1420, the UI 1405 displays the selected person's information 1462 and various selectable UI items 1468, 1472, and 1470 after the person's name 1455 is selected. In this example, one of the various selectable UI items 1472 (which may be implemented as selectable icons or buttons) provides a videoconferencing tool. Using the videoconferencing option 1472, the user can invite the person identified by the contacts 1466 to a videoconference. In different embodiments, the information 1462 and the selectable UI items 1468, 1472, and 1470 are displayed differently (e.g., in different configurations).
[0150] A fourth step 1425 shows the user selecting (e.g., via a single finger tap) the videoconferencing option 1472. In this example, the videoconferencing option 1472 is highlighted to indicate that the videoconferencing tool 1472 has been activated. Such a selection may be indicated differently in different embodiments (e.g., by highlighting the text or border of the selected icon).
[0151] The fifth stage 1430, the sixth stage 1435 and the seventh stage 1440 are similar to the third stage 1120, the fourth stage 1125 and the fifth stage 1130 shown in FIG. 11 and may be understood by referring to the description of these stages. In short, the fifth stage 1430 shows a transitional waiting stage for a remote user to respond to the videoconference invitation. The sixth stage 1435 shows that after the remote user accepts the videoconference request, the size of the viewport 1480 (displaying the local user's video) is gradually reduced so that the UI 1405 can show a viewport 1492 containing the video from the remote user's camera behind the viewport 1480. In the seventh stage 1440, a PIP display 1447 is presented by the UI 1405 during the videoconference. In some embodiments, the layout of the viewport 1155 in the sixth stage 1435 and the seventh stage 1440 resembles the layout of the viewport 1155 in FIG. 12 described above.
[0152] Figures 10, 11, 12, 13 and 14 show some ways of establishing a videoconference. In some embodiments, during a phone call, audio data (e.g., voice) is transmitted over one communication channel (over a communication network such as a circuit-switched or packet-switched communication network), and during a videoconference, the audio data is transmitted over another communication channel. Thus, in such embodiments, before the videoconference is established, the audio data (e.g., voice) is transmitted over a communication channel, and once the videoconference is established, the audio is transmitted over a different communication channel (instead of the communication channel used during the phone call).
[0153] To provide a seamless transition (e.g., handoff) of audio data from the call to the videoconference, in some embodiments, the call is not ended before the videoconference is established. For example, in some embodiments, the peer-to-peer videoconference connection is established (e.g., after completing the message sequence shown in FIG. 10) before the call is ended and the audio / video data begins to be transmitted over the peer-to-peer communication session. Alternatively, in other embodiments, the peer-to-peer videoconference connection is established (e.g., after completing the message sequence shown in FIG. 10) before the call is ended and the received audio / video data begins to be presented.
[0154] The peer-to-peer videoconference connection of some embodiments allows the mobile devices in the videoconference to communicate directly with each other (e.g., instead of communicating through a central server). Some embodiments of peer-to-peer videoconferences allow the mobile devices in the videoconference to share resources with each other. For example, as described in more detail below, through the control communication channel of the videoconference, one mobile device can remotely control the operation of another mobile device during the videoconference by sending commands from one mobile device to the other mobile device to instruct the other mobile device to process images differently (i.e., share its image processing resources), such as exposure adjustment operations, focus adjustment operations, and / or camera switching operations.
[0155] 2. Dynamic bitrate setup Typically, mobile devices in a video conference communicate data (e.g., audio and video images) with each other through communication channels of various types of communication networks, such as various private and public wireless communication networks (e.g., cellular networks such as GSM, UMTS, etc.) Examples of such wireless communication networks are described below with respect to Figures 91 and 92.
[0156] Due to the varying number of mobile devices accessing the communication network at a given time, the available bandwidth of the communication network for conducting a video conference varies from time to time. The available bandwidth may even change during the video conference. Furthermore, it is undesirable to flood the communication network with high bit rates or large amounts of signaling during the video conference in an attempt to track the optimal video conference bit rate.
[0157] For these reasons, some embodiments employ a novel method for identifying an optimal initial bit rate for a videoconference, which begins the videoconference at a particular bit rate and gradually increases the bit rate at specific intervals if these embodiments do not detect network conditions that degrade the videoconference quality.
[0158] An example of such an embodiment is shown in Figure 15, which conceptually illustrates a process 1500 of some embodiments of the present invention for setting bit rates for a videoconference. Process 1500 is performed as part of a videoconference setup (e.g., as part of the establishment of the videoconference shown in Figure 10) to dynamically determine the bit rates for transmitting data (e.g., audio and video images) based on various network conditions. In some embodiments, process 1500 is performed by management layer 935 of videoconferencing module 925 described above with respect to Figure 9. A more detailed version of this videoconferencing module is described below with respect to Figure 16.
[0159] 15, process 1500 begins (at 1505) by setting the bit rate to an initial bit rate. In some embodiments, the initial bit rate is a default baseline rate for the device. However, in some embodiments, the initial bit rate can be specified by a user. At 1505, process 1500 also initiates a videoconference that transmits data (e.g., audio and video images) to a remote device over one or more communication channels at the initial bit rate.
[0160] Next, process 1500 identifies (at 1510) a network condition parameter set received from a remote device in the videoconference. In some embodiments, the local device receives the network condition parameter set from the remote device through a Real-time Transport Protocol (RTP) communication session established at the start of the videoconference. For example, some embodiments provide the network condition parameters through an extension to RTP. Furthermore, the RTP extension of some embodiments indicates the presence of an extension header in the RTP packet header and can be used to transmit any type of information (such as a network condition parameter set) by defining an extension header for the additional information.
[0161] In different embodiments, the devices in the video conference relay different sets of network condition / congestion parameters. In the embodiments described below, the set of network condition parameters includes one-way latency and bandwidth estimated bit rate. In other embodiments, the set of network condition parameters includes packet loss data and round trip time (RTT) delay data. Thus, different embodiments may include any number of different network condition parameters in the set of network condition parameters.
[0162] In some embodiments, the set of network condition parameters received from the remote device of the videoconference is based on data (e.g., audio and video) transmitted from the local mobile device (i.e., the mobile device executing process 1500) to the remote device during the videoconference at the initial bit rate set in operation 1505. For example, in some embodiments, the remote device can determine the one-way latency by calculating the time it takes for an audio packet to propagate through a network connection from the local mobile device to the remote device using the timestamp of the audio packet. Specifically, in some embodiments, each audio packet is time-stamped. In the absence of packet delay, the remote device would receive the audio packet at a set interval equal to the difference in the timestamps. However, in the presence of one-way latency delay, the remote device receives the audio packet at an interval that is greater than the difference in the timestamps.
[0163] Also, in some embodiments, the remote device determines the bandwidth estimated bit rate by examining the time a video packet was received, the time a successive video packet was received, and the size of the successive video packets. That is, the difference in the reception times of two successive video packets and the size of the second video packet are used to estimate the available bandwidth of the network connection. In some embodiments, the bandwidth estimated bit rate is determined by examining multiple pairs of successive video packets. The above example uses a particular type of data (i.e., audio data to determine the one-way latency and video data to determine the bandwidth estimated bit rate). However, in some embodiments, other types of data communicated over the network connection between the local mobile device and the remote device may also be used.
[0164] After identifying (at 1510) the set of network conditions, process 1500 then determines (at 1515) whether the one-way latency has deteriorated beyond a defined threshold amount. In some embodiments, the threshold amount is defined as a particular latency amount, and the one-way latency is determined to have deteriorated beyond the threshold amount if the difference between the current one-way latency and the previous one-way latency exceeds the particular latency amount. In other embodiments, the threshold amount is defined as a particular rate of change of the one-way latency. Thus, the one-way latency is determined to have deteriorated beyond the threshold amount if the rate of change of the set of one-way latencies (e.g., the current one-way latency and the previous one-way latency) exceeds the particular rate of change.
[0165] If it is determined that the one-way latency degrades beyond a threshold amount, the process 1500 ends. Otherwise, the process 1500 determines (at 1520) whether the current bitrate has reached the bandwidth estimation bitrate. In some embodiments, the bandwidth estimation bitrate indicates the amount of bandwidth available for the network connection (e.g., 15 kilobits per second (kbps)). If the process 1500 determines that the current bitrate has exceeded the bandwidth estimation bitrate, the process 1500 ends. If the process 1500 determines that the current bitrate has not exceeded the bandwidth estimation bitrate, the process 1500 proceeds to operation 1525.
[0166] At 1525, process 1500 determines whether the current bit rate has reached a specified maximum bit rate. If process 1500 determines that the current bit rate has exceeded the specified maximum bit rate, process 1500 ends. If not, process 1500 proceeds to operation 1530, where the current bit rate is increased by a specified amount. Different embodiments define the amount by which the bit rate is increased differently. Examples of the specified amount by which the current bit rate is increased include 32 kbps, 64 kbps, among any number of other amounts by which the bit rate is increased.
[0167] The process then determines (at 1535) whether a prescribed amount of time has elapsed. This prescribed amount of time may be 1 second, 2 seconds, 5 seconds, or any other possible amount of time, as different embodiments define the amount of time differently. Process 1500 waits for the prescribed amount of time to elapse in order for the remote device to receive (at operation 1530) the data (e.g., audio and video images) transmitted from the local mobile device at the new increased bit rate and determine network condition parameters based on the new increased bit rate. If process 1500 determines that the prescribed amount of time has not elapsed, process 1500 returns to operation 1535 until the prescribed amount of time has elapsed. If process 1500 determines that the prescribed amount of time has elapsed, process 1500 returns to operation 1510. Operation of process 1500 proceeds from 1510 onward as described above until process 1500 ends.
[0168] When process 1500 ends (i.e., after operation 1515, 1520, or 1525), the bitrate setup for the videoconference is complete and the optimal bitrate has been determined. Since the bandwidth available for the videoconference may change during the videoconference, some embodiments continue to adjust the bitrate based on a set of network condition parameters received from the remote devices (i.e., one-way latency and bandwidth estimated bitrate). The bitrate may be adjusted during the videoconference by increasing the bitrate. For example, if process 1500 ends because the one-way latency has deteriorated beyond a specified threshold amount and the one-way latency has improved during the videoconference, in some embodiments the bitrate is increased. Similarly, if process 1500 ends because the bitrate has exceeded the bandwidth estimated bitrate and the bandwidth estimated bitrate has increased during the videoconference, in some embodiments the bitrate is increased.
[0169] In contrast, the bitrate may be adjusted during the videoconference by decreasing the bitrate. For example, if one-way latency continues to degrade during the videoconference beyond a specified threshold amount, in some embodiments the bitrate is decreased. Also, if the bitrate continues to exceed the bandwidth estimated bitrate during the videoconference (e.g., the bandwidth estimated bitrate continues to decrease), in some embodiments the bitrate is decreased.
[0170] Additionally, the description of process 1500 uses one-way latency and bandwidth estimated bitrate to determine whether to increase the bitrate. However, those skilled in the art will appreciate that in different embodiments, any number of network condition parameters can be used to determine whether to increase the bitrate. For example, the decision to increase the bitrate can be based solely on RTT delay data or packet loss data.
[0171] C. Videoconferencing Architecture As mentioned above, Figure 16 conceptually illustrates a software architecture for a videoconferencing and processing module 1600 of a dual camera mobile device of some embodiments. As shown, the videoconferencing and processing module 1600 includes a client application 1665, a videoconferencing module 1602, a media exchange module 1620, a buffer 1625, a captured image processing unit (CIPU) driver 1630, an encoder driver 1635, and a decoder driver 1640. In some embodiments, the buffer 1625 is a frame buffer that stores images of video for display on a display 1645 of the dual camera mobile device.
[0172] In some embodiments, the client application 1665 is the same as the videoconferencing client 945 of Figure 9. As mentioned above, the client application 1665 may be integrated into another application or implemented as a stand-alone application. The client application 1665 may be an application that uses the videoconferencing functionality of the videoconferencing module 1602, such as a videoconferencing application, a Voice over IP (VOIP) application (e.g., Skype), or an instant messaging application.
[0173] In some embodiments, the client application 1665 sends instructions to the videoconferencing module 1602, such as instructions to start a conference and instructions to end a conference, receives instructions from the videoconferencing module 1602, routes instructions from a user of the dual camera mobile device to the videoconferencing module 1602, and generates a user interface that is displayed on the dual camera mobile device to enable the user to interact with the application.
[0174] D. Videoconferencing Manager 16, videoconferencing module 1602 includes videoconferencing manager 1604, image processing manager 1608, networking manager 1614, and buffers 1606, 1610, 1612, 1616, and 1618. In some embodiments, videoconferencing module 1602 is the same as videoconferencing module 925 shown in FIG. 9 and thus performs some or all of the same functions described above with respect to videoconferencing module 925.
[0175] In some embodiments, the videoconference manager 1604 is responsible for initializing some or all of the other modules of the videoconference module 1602 (e.g., image processing manager 1608 and networking manager 1614) when a videoconference begins, controlling the operation of the videoconference module 1602 during the videoconference, and ceasing the operation of some or all of the other modules of the videoconference module 1602 when the videoconference ends.
[0176] The videoconference manager 1604 of some embodiments also processes images received from one or more devices in the videoconference and images captured by one of both cameras of the dual camera mobile device for display on the dual camera mobile device. For example, the videoconference manager 1604 of some embodiments reads decoded images received from another device participating in the videoconference from the buffer 1618 and reads images processed by the CIPU 1650 (i.e., images captured by the dual camera mobile device) from the buffer 1606. In some embodiments, the videoconference manager 1604 also scales and composites images before displaying them on the dual camera mobile device. That is, the videoconference manager 1604 generates a PIP view or other composite view for display on the mobile device in some embodiments. Some embodiments scale images read from the buffers 1606 and 1618, while other embodiments scale only images read from one of the buffers 1606 and 1618.
[0177] 16 illustrates the videoconference manager 1604 as part of the videoconference module 1602, some embodiments of the videoconference manager 1604 are implemented as a separate component from the videoconference module 1602. Thus, a single videoconference manager 1604 may be used to manage and control several videoconference modules 1602. For example, in some embodiments, a separate videoconference module runs on a local device to interact with each party during a multi-party conference, and each of these videoconference modules on the local device is managed and controlled by one videoconference manager.
[0178] An image processing manager 1608 of some embodiments processes images captured by the cameras of the dual camera mobile device before the images are encoded by the encoder 1655. For example, some embodiments of the image processing manager 1608 perform one or more of exposure adjustment, focus adjustment, perspective correction, dynamic range adjustment, and image resizing on images processed by the CIPU 1650. In some embodiments, the image processing manager 1608 controls the frame rate of encoded images transmitted to other devices in a video conference.
[0179] Some embodiments of the networking manager 1614 manage one or more connections between the dual camera mobile device and other devices participating in the videoconference. For example, the networking manager 1614 of some embodiments establishes connections between the dual camera mobile device and other devices participating in the videoconference at the start of the videoconference and disconnects these connections at the end of the videoconference.
[0180] During the videoconference, the networking manager 1614 transmits images encoded by the encoder 1655 to other devices in the videoconference and routes images received from other devices in the videoconference to the decoder 1660 for decoding. In some embodiments, the networking manager 1614, rather than the image processing manager 1608, controls the frame rate of images transmitted to other devices in the videoconference. For example, some such embodiments of the networking manager 1614 control the frame rate by discarding (i.e., not transmitting) some of the encoded frames that would have been transmitted to other devices in the videoconference.
[0181] As shown, the media exchange module 1620 of some embodiments includes a camera source module 1622, a video compressor module 1624, and a video decompressor module 1626. The media exchange module 1620 is the same as the media exchange module 310 shown in Figure 3, which will be described in more detail below. The camera source module 1622 routes messages and media content between the videoconferencing module 1602 and the CIPU 1650 through a CIPU driver 1630, the video compressor module 1624 routes messages and media content between the videoconferencing module 1602 and the encoder 1655 through an encoder driver 1635, and the video decompressor module 1626 routes messages and media content between the videoconferencing module 1602 and the decoder 1660 through a decoder driver 1640. In some embodiments, the TNR module 315 (not shown in FIG. 16) included in the media exchange module 310 is implemented as part of the camera source module 1622, while in other embodiments, the TNR module 315 is implemented as part of the video compressor module 1624.
[0182] In some embodiments, the CIPU driver 1630 and the encoder driver 1635 are the same as the CIPU driver 305 and the encoder driver 320 shown in Figure 3. The decoder driver 1640 of some embodiments serves as a communications interface between the video decompressor module 1626 and the decoder 1660. In such an embodiment, the decoder 1660 decodes images received from other devices in the video conference through the networking manager 1614 and routed through the video decompressor module 1626. After the images are decoded, they are sent back to the video conference module 1602 through the decoder driver 1640 and the video decompressor module 1626.
[0183] In addition to performing video processing during a video conference, the videoconferencing and processing module 1600 for a dual camera mobile device of some embodiments also performs audio processing operations during a video conference. Figure 17 illustrates such a software architecture. As shown, the videoconferencing and processing module 1600 includes a videoconferencing module 1602 (including a videoconferencing manager 1604, an image processing manager 1608, and a networking manager 1614), a media exchange module 1620, and a client application 1665. Other components and modules of the videoconferencing and processing module 1600 shown in Figure 16 are omitted in Figure 17 for ease of illustration. The videoconferencing and processing module 1600 also includes frame buffers 1705 and 1710, an audio processing manager 1715, and an audio driver 1720. In some embodiments, the audio processing manager 1715 is implemented as a separate software module, while in other embodiments, the audio processing manager 1715 is implemented as part of the media exchange module 1620.
[0184] The audio processing manager 1715 processes audio data captured by the dual camera mobile device for transmission to other devices in the video conference. For example, the audio processing manager 1715 receives audio data captured by the microphone 1725 through the audio driver 1720, encodes the audio data, and then stores the encoded audio data in the buffer 1705 for transmission to other devices. The audio processing manager 1715 also processes audio data captured by and received from other devices in the video conference. For example, the audio processing manager 1715 reads audio data from the buffer 1710, decodes the audio data, and then outputs it to the speaker 1730 through the audio driver 1720.
[0185] In some embodiments, the videoconferencing module 1602, along with the audio processing manager 1715 and its associated buffers, is part of a larger conferencing module. When a multi-participant audio conference takes place between several devices without exchanging video content, the videoconferencing and processing module 1600 uses only the networking manager 1614 and the audio processing manager 1715 to facilitate the exchange of audio over the Internet Protocol (IP) layer.
[0186] The operation of the videoconference manager 1604 of some embodiments will now be described with respect to Figure 18. Figure 18 conceptually illustrates a process 1800 performed by a videoconference manager of some embodiments, such as the videoconference manager 1604 shown in Figure 16, which may be equivalent to that performed by the management layer 935 of Figure 9. In some embodiments, the videoconference manager 1604 performs the process 1800 when a user of the dual camera mobile device accepts a videoconference request (e.g., through a user interface displayed on the dual camera mobile device) or when a user of another device accepts a request sent by a user of the dual camera mobile device.
[0187] Process 1800 begins by receiving (at 1805) an instruction to initiate a videoconference. In some embodiments, the instruction is received from a client application 1665 or from a user through a user interface displayed on the dual camera mobile device and forwarded by the client application 1665 to the videoconference manager 1604. For example, in some embodiments, if a user of the dual camera mobile device accepts a videoconference request, the instruction is received through the user interface and forwarded by the client application. On the other hand, if a user of another device accepts a request sent from the local device, in some embodiments, the instruction is received from the client application without any user interface interaction (although there may have been a previous user interface interaction to send the initial request).
[0188] Next, the process 1800 initializes (at 1810) a first module that interacts with the videoconference manager 1604. The modules in some embodiments that interact with the videoconference manager 1604 include the CIPU 1650, the image processing manager 1608, the audio processing manager 1715, and the networking manager 1614.
[0189] In some embodiments, initializing the CIPU 1650 includes instructing the CIPU 1650 to start processing images captured by one or both cameras of the dual camera mobile device. In some embodiments, the image processing manager 1608 is initialized by instructing the image processing manager 1608 to start reading images from the buffer 1610 and processing and encoding the read images. To initialize the audio processing manager 1715, in some embodiments, the audio processing manager 1715 is instructed to start encoding audio data captured by the microphone 1725 and decoding audio data (received from other devices) stored in the buffer 1710 for output to the speaker 1730. In some embodiments, initializing the networking manager 1614 includes instructing the networking manager 1614 to establish a network connection with other devices in the video conference.
[0190] The process 1800 then determines (at 1815) whether there are no modules left to initialize. If there are modules left to initialize, the process 1800 returns to operation 1810 to initialize another of the modules. If all the necessary modules have been initialized, the process 1800 generates (at 1820) an image composite for display on the dual camera mobile device (i.e., a local display). These composite images may include the images shown in FIG. 65 (i.e., a PIP or other composite display), described below, and may include various combinations of images from the cameras of the local dual camera mobile device and images from the cameras of other devices participating in the videoconference.
[0191] Next, the process 1800 determines (at 1825) whether changes have been made to the videoconference. In some embodiments, changes to the videoconference are received through user interaction with a user interface displayed on the dual camera mobile device, while in other embodiments, changes to the videoconference are received from other devices through the networking manager 1614 (i.e., a remote control). In some embodiments, changes to the videoconference settings may be received from the client application 1665 or other modules within the videoconferencing module 1602. The videoconference settings may change due to changes in network conditions.
[0192] If a change has been made, process 1800 determines (at 1830) whether the change to the videoconference is a change to a network setting. In some embodiments, the change is either a change to a network setting or a change to an image capture setting. If the change to the videoconference is a change to a network setting, the process changes (at 1840) the network setting and then proceeds to operation 1845. The change to the network setting in some embodiments includes a change to the bit rate at which images are encoded or the frame rate at which images are transmitted to other devices.
[0193] If the change to the video conference is not a change to the network settings, process 1800 determines that the change is a change to the image capture settings and then proceeds to operation 1835. Process 1800 then performs (at 1835) the change to the image capture settings. In some embodiments, the change to the image capture settings may include camera switching (i.e., switching which camera on the dual camera mobile device captures video), focus adjustment, exposure adjustment, displaying or not displaying images from one or both cameras of the dual camera mobile device, and zooming in or out of the image displayed on the dual camera mobile device, among other setting changes.
[0194] At operation 1845, the process 1800 determines whether to end the videoconference. If the process 1800 determines that the videoconference is not to end, the process 1800 returns to operation 1820. If the process 1800 determines that the videoconference is to end, the process 1800 ends. Some embodiments of the process 1800 determine that the videoconference is to end when the process 1800 receives an instruction from the client application 1665 to end the videoconference (i.e., due to an instruction received through a user interface of the local dual-camera mobile device or from another device participating in the videoconference).
[0195] In some embodiments, videoconference manager 1604 performs various operations not shown in process 1800 when the videoconference ends, such as instructing CIPU 1650 to stop generating images, instructing networking manager 1614 to disconnect network connections with other devices in the videoconference, and instructing image processing manager 1608 to stop generating and encoding images.
[0196] E. Temporal Noise Reduction Some embodiments include a separate temporal noise reduction module for processing the video images to reduce noise in the video. The temporal noise reduction module of some embodiments compares subsequent images of a video sequence to identify and eliminate undesirable noise from the video.
[0197] Figure 19 conceptually illustrates a software architecture for such a temporal noise reduction (TNR) module 1900 of some embodiments. In some embodiments, the TNR module 1900 is implemented as part of an application (e.g., as part of the media exchange module shown in Figure 3), while in other embodiments, the TNR module 1900 is implemented as a standalone application for use by other applications. In yet other embodiments, the TNR module 1900 is implemented as part of an operating system running on a dual camera mobile device. In some embodiments, the TNR module 1900 is implemented by a collection of APIs that provide some or all of the functionality of the TNR module 1900 to other applications.
[0198] As shown in Figure 19, the TNR module 1900 includes a TNR manager 1905, a difference module 1910, a pixel averaging module 1915, and a motion history module 1920. Although Figure 19 shows the three modules 1910, 1915, and 1920 as separate modules, in some embodiments, the functions of these modules are performed in a single module, as described below. The TNR module 1900 of some embodiments receives as input an input image, a reference image, and a motion history. In some embodiments, the input image is the image currently being processed, and the reference image is the previous image in the video sequence to which the input image is compared. The TNR module 1900 outputs an output image (a noise-reduced version of the input image) and an output motion history.
[0199] A TNR manager 1905 of some embodiments directs the flow of data through the TNR module 1900. As shown, the TNR manager 1905 receives an input image, a reference image, and a motion history. The TNR manager 1905 also outputs an output image and an output motion history. The TNR manager 1905 sends the input image and the reference image to a difference module 1910, and receives a difference image from the difference module 1910.
[0200] In some embodiments, the difference module 1910 processes data received from the TNR manager 1905 and sends the processed data to the TNR manager 1905. As shown, the difference module 1910 receives an input image and a reference image from the TNR manager 1905. The difference module 1910 of some embodiments generates a difference image by subtracting pixel values of one image from pixel values of the other image. The difference image is sent to the TNR manager 1905. The difference image of some embodiments shows the difference between the two images to identify areas of the input image that have changed and areas of the input image that have remained the same compared to the previous image.
[0201] The TNR manager 1905 also sends the input image and the reference image to the pixel averaging module 1915. As shown, in some embodiments, it also sends the motion history to the pixel averaging module 1915. However, in other embodiments, it could send only the input image and the reference image without sending the motion history. In either embodiment, the TNR manager 1905 receives the processed images from the pixel averaging module 1915.
[0202] The pixel averaging module 1915 of some embodiments uses the motion history to determine whether to average pixels from the input image and the reference image for a particular location of the image. In some embodiments, the motion history includes a probability value for each pixel of the input image. The particular probability value represents the probability that the corresponding pixel of the input image has changed relative to the corresponding pixel of the reference image (i.e., is a dynamic pixel). For example, if the probability value of a particular pixel of the input image is 20, it indicates that there is a 20% probability that the particular pixel of the input image has changed relative to the corresponding pixel of the reference image. As another example, if the probability value of a particular pixel of the input image is 0, it indicates that the particular pixel of the input image has not changed relative to the corresponding pixel of the reference image (i.e., is a static pixel).
[0203] Different embodiments store the probability values of the input image differently. In some embodiments, a single data array may store the probability values for each pixel of the input image. In other embodiments, a matrix (e.g., an array of arrays) may store the probability values having the same dimensions as the resolution of the images in the video. For example, if the resolution of the images in the video is 320x240, then the matrix is also 320x240.
[0204] When the pixel averaging module 1915 receives the input image and the reference image as well as the motion history from the TNR manager 1905, the pixel averaging module 1915 reads a probability value for each pixel of the input image. If the probability value for a particular pixel of the input image is below a specified threshold (e.g., 5%, 20%), the pixel averaging module 1915 averages the particular pixel value with the corresponding pixel value of the reference image based on the premise that there is a low probability of motion at the particular pixel, and therefore the difference between images at that pixel may be due to noise.
[0205] If the probability for a particular pixel of the input image is not below a specified threshold, the pixel averaging module 1915 does not change the particular pixel of the input image (i.e., the pixel value of that pixel remains the same as in the input image). This is because there is a high probability of motion at the particular pixel, and therefore the difference between the images is likely not the result of noise. In some embodiments, if no motion history is sent to the pixel averaging module 1915, the pixel averaging module 1915 averages each pixel of the input image with the corresponding pixel of the reference image. The processed image output by the pixel averaging module 1915 and sent to the TNR manager 1905 includes the input image pixel value for every pixel that was not averaged and the averaged pixel value for every pixel that was averaged by the pixel averaging module 1915.
[0206] In some embodiments, the motion history module 1920 processes data received from the TNR manager 1905 and sends result data back to the TNR manager 1905. The motion history module 1920 of some embodiments receives an input image and a motion history from the TNR manager 1905. In some embodiments, this data is input to a Bayesian estimator to generate a new motion history (i.e., a set of probability values) that can be used for pixel averaging for the next input image. In other embodiments, other estimators are used to generate the new motion history.
[0207] The operation of the TNR module 1900 will now be described with respect to Figure 20, which conceptually illustrates a process 2000 of some embodiments for reducing temporal noise in images of a video. The process 2000 begins with the TNR manager 1905 receiving (at 2005) an input image, a reference image, and a motion history. The input image is the image currently being processed for noise reduction. In some embodiments, the reference image is the immediately previous image in the sequence of images of the video received from the CIPU. However, in other embodiments, the reference image is an output image (i.e., the output of the TNR module 1900) generated from processing the immediately previous input image. The motion history is the output motion history generated from processing the immediately previous input image.
[0208] If the input image is the first image of a video, the TNR module 1900 of some embodiments does not process the first image (i.e., does not apply TNR to the first image). In other words, the TNR manager 1905 receives the first image and simply outputs the first image. In other embodiments, if the input image is the first image of a video, the first image is used as the input image and the reference image, and the TNR module 1900 processes the images as described below. Furthermore, if the input image is the first image of a video, the motion history is empty (e.g., null, all zeros, etc.), and the TNR manager 1905 simply outputs the empty motion history as the output motion history.
[0209] The TNR manager 1905 then determines (at 2010) whether the input image is static. To make this determination, some embodiments send the input image and a reference image to a difference module 1910 and receive a difference image from the difference module 1910. If the difference between the two images is below a specified threshold (e.g., a 5% difference, a 10% difference, etc.), then some embodiments classify the input image as static.
[0210] If the input image is a static image, the TNR manager 1905 sends the input image and the reference image to a pixel averaging module 1915 for averaging (at 2015) the pixels of the input image with the pixels of the reference image in order to reduce noise from the static image. The process then proceeds to 2040, which is described below.
[0211] If the input image is not a static image, the TNR manager sends the input image, the reference image, and the motion history to a pixel averaging module 1915 for processing. The pixel averaging module 1915 selects (at 2020) a pixel of the input image. As described above, the pixel averaging module 1915 uses the motion history to determine (at 2025) whether the probability of motion of the pixel is below a certain threshold.
[0212] If the probability of the selected pixel is below a particular threshold, the pixel averaging module 1915 averages (at 2030) the pixel of the input image with the corresponding pixel of the reference image. If not, the pixel is not averaged and the output image is the same as the input image at that particular pixel. The pixel averaging module 1915 then determines (at 2035) whether any unselected pixels remain in the input image. If any pixels have not been processed, the process returns to operation 2020 to select the next pixel. The pixel averaging module 1915 performs operations 2020-2030 until all pixels have been evaluated.
[0213] The process then updates (at 2040) the motion history. As shown in Figure 19 and described above, the motion history module 1920 updates the motion history based on the input image. The new motion history is output by the TNR manager along with the processed image from the pixel averaging module.
[0214] F. Image Processing Manager and Encoder In some embodiments, in addition to the temporal noise reduction and image processing operations performed by the CIPU and / or CIPU driver, various image processing operations are performed in image processing layer 930 of videoconferencing module 925. These image processing operations may include exposure adjustment, focus adjustment, perspective correction, dynamic range adjustment, and image resizing, among others.
[0215] Figure 21 conceptually illustrates a process 2100 for performing such image processing operations. In some embodiments, some or all of the operations of process 2100 are performed by a combination of image processing manager 1608 and encoder driver 1635 of Figure 16. In some such embodiments, image processing manager 1608 performs pixel-based processing (e.g., resizing, dynamic range adjustment, perspective correction, etc.). In some embodiments, process 2100 is performed during a videoconference on images that are to be transmitted to another device participating in the videoconference.
[0216] Process 2100 will now be described with respect to Figure 16. The process begins by reading (at 2105) an image from buffer 1606. In some embodiments, the image read is an image of a video (i.e., an image in an image sequence). The video may have been captured by a camera of the device on which process 2100 is performed.
[0217] Next, the process 2100 performs (at 2110) an exposure adjustment on the retrieved image. In some embodiments, the exposure adjustment is performed through a user interface displayed on the dual camera mobile device. Figure 22 illustrates an exemplary exposure adjustment operation of such an embodiment.
[0218] This figure illustrates the exposure adjustment operation with respect to three stages 2210, 2215, and 2220 of the UI 2205 of the device 2200. The first stage 2210 shows the UI 2205 including a viewport 2225 and a viewport 1155. As shown, the viewport 2225 displays an image 2230 of the sun and a man with a dark face and body color. The dark face and body color indicates that the man is not properly exposed. The image 2230 could be a video image captured by the camera of the device 2200. As shown, the viewport 1155 includes a selectable UI item 2250 for ending the video conference. In some embodiments, the layout of the viewport 1155 is the same as the layout of the viewport 1155 of FIG. 12 described above.
[0219] A second step 2215 shows a user of the device 2200 beginning an exposure adjustment operation by selecting an area of the viewport 2225. In this example, the selection is made by placing a finger 2235 anywhere within the viewport 2225. In some embodiments, the user selects the exposure adjustment from a menu of possible image setting adjustments.
[0220] A third stage 2220 shows an image 2240 of the man after the exposure adjustment operation has been completed. As shown, image 2240 is similar to image 2230, except that the man in image 2240 is properly exposed. In some embodiments, the properly exposed image is an image captured after the improperly exposed image. The exposure adjustment operation initiated in second stage 2215 adjusts the exposure of subsequent images captured by the camera of device 2200.
[0221] Returning to Figure 21, process 2100 then performs (at 2115) a focus adjustment on the image. In some embodiments, the focus adjustment is performed through a user interface displayed on the dual camera mobile device. Figure 23 conceptually illustrates one example of such a focus adjustment operation.
[0222] FIG. 23 illustrates the focus adjustment operation with respect to three different stages 2310, 2315, and 2320 of the UI 2305 of the device 2300. The first stage 2310 shows the UI 2305 including a viewport 2325 and a viewport 1155. The viewport 2325 presents a blurred image 2330 of a man captured by a camera of the device 2300. The blur indicates that the image 2330 of the man is out of focus. That is, the lens of the camera was not focused on the man when the image 2330 of the man was captured by the camera. The image 2330 may also be a video image captured by the camera of the device 2300. As shown, the viewport 1155 includes a selectable UI item 2350 for ending the video conference. In some embodiments, the layout of the viewport 1155 is the same as the layout of the viewport 1155 of FIG. 12 described above.
[0223] A second step 2315 shows a user of the device 2300 beginning the focus adjustment operation by selecting an area of the viewport 2325. In this example, the selection is made by placing a finger 2335 anywhere within the viewport 2325. In some embodiments, the user selects the focus adjustment from a menu of possible image setting adjustments.
[0224] A third stage 2320 shows an image 2340 of the man after the focus adjustment operation has been completed. As shown, image 2340 is the same as image 2330, but the man in image 2340 appears clearer. This indicates that the camera lens has properly focused on the man. In some embodiments, the properly focused image is an image captured after the improperly focused image. The focus adjustment operation initiated in second stage 2315 adjusts the focus of subsequent images captured by the camera of device 2300.
[0225] Returning to Figure 21, process 2100 performs (at 2120) image resizing on the image. In some embodiments, image resizing is performed on the image in order to reduce the number of bits used to encode the image (i.e., to lower the bit rate). In some embodiments, process 2100 performs image resizing as described below with respect to Figure 26.
[0226] The process 2100 then performs (at 2125) perspective correction on the image. In some embodiments, the process 2100 performs perspective correction as described below in FIG. 24. Such perspective correction involves using data obtained by one or more accelerometer and / or gyroscope sensors to identify the orientation and movement of the dual camera mobile device. This data is then used to modify the image to correct for the misaligned perspective.
[0227] After perspective correction has been performed on the image, process 2100 adjusts (at 2130) the dynamic range of the image. In some embodiments, the dynamic range of an image is the range of possible values that each pixel in the image can have. For example, an image with a dynamic range of 0-255 may be adjusted to a range of values of 0-128 or any other range. Adjusting the dynamic range of an image can reduce the amount of bits used to encode the image (i.e., lower the bit rate), thereby smoothing the image.
[0228] Adjusting the dynamic range of an image can also be used for various other purposes. One purpose is to reduce image noise (e.g., the image was captured by a noisy camera sensor). To reduce noise, the dynamic range of the image is adjusted so that the black level is redefined to include lighter blacks (i.e., crush blacks). In this way, image noise is reduced. Another purpose of dynamic range adjustment is to adjust one or more colors or color ranges to improve image quality. For example, in some embodiments, it may be assumed that an image captured by a front camera is an image of a person's face. Thus, the dynamic range of the image may be adjusted to increase red and pink colors to make the person's cheeks appear rosy / rosier. The dynamic range adjustment operation may also be used for other purposes.
[0229] Finally, the process 2100 determines (at 2135) one or more rate controller parameters to be used to encode the image. Such rate controller parameters, in some embodiments, may include a quantization parameter and a frame type (e.g., P (predictive), B (bi-directional), I (intra-coded)). The process then terminates.
[0230] Although various operations of process 2100 are shown to be performed in a particular order, one skilled in the art will appreciate that many of these operations (exposure adjustment, focus adjustment, perspective correction, etc.) can be performed in any order and are not dependent on one another. That is, the process of some embodiments may perform focus adjustment before exposure adjustment, or may perform modifications similar to the process shown in FIG.
[0231] 1. Perspective correction As mentioned above, in some embodiments, perspective correction is performed on an image before the image is displayed or transmitted. In some cases, one or more of the cameras on a dual camera mobile device is not properly oriented to the subject, which appears distorted in the uncorrected image. Perspective correction can be used to process the image so that it more closely reflects how the objects in the image appear to a person.
[0232] Figure 24 conceptually illustrates a perspective correction process 2400 performed by an image processing manager of some embodiments, such as the process shown in Figure 16. Process 2400 of some embodiments is performed by image processing layer 930 (which may include image processing manager 1608) shown in Figure 9. In some embodiments, process 2400 is performed at operation 2125 of process 2100 to correct the perspective of a recently captured video image before displaying or transmitting the image.
[0233] Process 2400 begins by receiving (at 2405) data from an accelerometer sensor, which in some embodiments is part of a dual camera mobile device. The accelerometer sensor of some embodiments measures the rate of change of the device's velocity (i.e., the acceleration of the device) along one or more axes. The process also receives (at 2410) data from a gyroscope sensor, which in some embodiments may also be part of the dual camera mobile device. The gyroscope sensor and accelerometer sensor of some embodiments may be used individually or in combination to identify the orientation of the dual camera mobile device.
[0234] Next, the process 2400 determines (at 2415) the amount of perspective correction to perform based on data obtained from the accelerometer and gyroscope sensors. Generally, the further off-axis the orientation, the more perspective correction is needed to produce an optimal image. In some embodiments, a warp parameter is calculated that represents the amount of perspective correction based on the device orientation.
[0235] After determining the amount of perspective correction to perform, the process 2400 receives (at 2420) images captured by the cameras of the dual camera mobile device. This process may be performed for each image of a video sequence captured by the cameras. In some embodiments, separate calculations may be performed for images arriving from each of the two cameras on the dual camera mobile device.
[0236] The process then modifies (at 2425) the image based on the determined amount of perspective correction. In some embodiments, the process uses a baseline image or other information (e.g., user-inputted points regarding which correction should be performed) in addition to the distortion parameters or other representation of the amount of perspective correction. After modifying the image, the process 2400 ends.
[0237] 25 conceptually illustrates exemplary image processing operations of some embodiments. The diagram illustrates a first image processing operation 2505 performed by a first image processing module 2520 that does not use perspective correction, and a second image processing operation 2550 performed by a second image processing module 2565 that uses perspective correction.
[0238] As shown, the first image processing operation 2505 is performed on a first image 2510 of a block 2515 from an atmospheric perspective looking downward at an angle towards the block. From that perspective, the top of the block 2515 is closer than the bottom of the block. Thus, the block 2515 appears tilted towards the camera that captured the first image 2510. Figure 25 also shows a processed first image 2525 after processing by the first image processing module 2520. As shown, the block 2515 of the processed first image 2525 appears the same post-processing because the first image processing module 2520 did not perform perspective correction.
[0239] A second image processing operation 2550 is performed on the second image 2555 in block 2560. Block 2560 is the same as block 2515 of the first image 2510. FIG. 25 also shows a processed second image 2575 after processing of the second image 2555 by a perspective corrector 2570 of the second image processing module 2565. The perspective corrector 2570 may use the process 2400 to correct the perspective of the second image 2555. Based on data from the accelerometer and gyroscope indicating that the camera that captured the second image 2555 was tilted at a downward angle (and possibly based on other data), the perspective corrector 2570 can correct the second image so that the blocks appear to appear straight on the processed second image 2575.
[0240] 2. Resizing and Bitstream Manipulation Among the functions performed by image processing layer 930 of some embodiments, and discussed above with respect to Figure 21, are image resizing and bitstream manipulation. In some embodiments, image resizing (performed in operation 2130) includes scaling the image up or down (i.e., changing the number of pixels used to represent the image). In some embodiments, bitstream manipulation includes inserting data into the bitstream that indicates the size of the resized image. This resizing and bitstream manipulation is performed by an encoder driver (e.g., driver 1635) in some embodiments.
[0241] FIG. 26 conceptually illustrates a software architecture for such an encoder driver 2600 of some embodiments, showing exemplary resizing and bitstream manipulation operations performed by the encoder driver 2600 on an exemplary image 2605. In some embodiments, the image 2605 is an image of video captured by the camera of a dual camera mobile device for transmission to another device(s) in a video conference. Referring to FIG. 16, in some embodiments, the video image will go from the CIPU 1650 through the CIPU driver 1630 and the camera source module 1622 to the buffer 1606 from which it is read by the image processing manager 1608. After being subjected to image processing (e.g., focus adjustment, exposure adjustment, perspective correction) in the image processing manager 1608, the image is sent to the encoder driver 1635 through the buffer 1610 and the video compressor module 1624.
[0242] As shown, the encoder driver 2600 includes a processing layer 2610 and a rate controller 2645. An example of the rate controller of some embodiments is shown in FIG. 30 described below. The processing layer 2610 includes an image resizer 2615 and a bitstream manager 2625. In some embodiments, these modules perform various operations on images both before and after the images are encoded. In this example, the image resizer is shown as part of the processing layer 2610 of the encoder driver 2600, but in some embodiments, the image resizer is implemented as part of the image processing manager 1608 rather than the encoder driver 2600 (i.e., image resizing occurs before sending the image and size data to the encoder driver).
[0243] As shown, image resizer 2615 resizes the image before it is sent through rate controller 2645 to encoder 2650. Image 2605 is sent through resizer 2615 and reduced to image 2630. In some embodiments, in addition to reducing an image, the image may also be enlarged.
[0244] 26, in some embodiments, an incoming image (e.g., image 2605) is scaled down and then the scaled down image (e.g., image 2630) is superimposed on a spatially redundant image (e.g., image 2635) that is the same size (in pixels) as the incoming image (i.e., the number of rows and columns of pixels in image 2605 is the same as the number of rows and columns of pixels in spatially redundant image 2635). In some embodiments, the scaled down image 2630 is superimposed on the top left corner of the spatially redundant image (to generate composite image 2640 as shown), while in other embodiments, the scaled down image is superimposed on different sections of the spatially redundant image (e.g., center, top right, top center, bottom center, bottom right, etc.).
[0245] In some embodiments, spatially redundant images are images that are substantially all one color (e.g., black, blue, red, white, etc.) or have a repeating pattern (e.g., checks, stripes, etc.). For example, the spatially redundant image 2635 shown in FIG. 26 has a repeating pattern of crosses. Because of their repetitive nature, the spatially redundant portions of the composite image 2640 may be easily compressed by an encoder into a small amount of data. Moreover, if a sequence of images is all scaled down and the spatially redundant images used are the same for each image in the sequence, temporal compression may be used to further reduce the amount of data required to represent the encoded image.
[0246] Some embodiments of the image resizer 2615 also generate size data 2620 indicating the size of the resized image (e.g., the size of the reduced image 2630) and send this generated size data 2620 to the bitstream manager 2625. The size data 2620 of some embodiments indicates the size of the resized image 2630 in terms of the number of pixel rows and the number of pixel columns (i.e., height and width) of the resized image 2630. In some embodiments, the size data 2620 also indicates the position of the resized image 2630 within the composite image 2640.
[0247] After the image is resized, the composite image 2640 is sent to the encoder 2650 through a rate controller 2645, which in some embodiments controls the bit rate (i.e., data size) of the image output by the encoder 2650, as described in more detail below. The encoder 2650 in some embodiments compresses and encodes the image. The encoder 2650 may use H.264 encoding or another encoding method.
[0248] The bitstream manager 2625 of some embodiments receives a bitstream of one or more encoded images from the encoder 2650 and inserts size data into the bitstream. For example, in some embodiments, the bitstream manager 2625 receives size data 2620 from the image resizer 2615 and inserts the size data 2620 into the bitstream 2655 of the encoded composite image 2640 received from the encoder 2650. The output of the bitstream manager 2625 is then a modified bitstream 2660 that includes the size data 2620. In different embodiments, the size data 2620 is inserted into different positions in the bitstream 2655. For example, the bitstream 2660 shows the size data 2620 inserted at the beginning of the bitstream 2660. However, in other embodiments, the size data 2620 is inserted at the end of the bitstream 2655, in the middle of the bitstream 2655, or at any other position within the bitstream 2655.
[0249] In some embodiments, the bitstream 2655 is a bitstream of a sequence of one or more encoded images, including the composite image 2640. In some such embodiments, the images in the sequence are all resized to the same size, and the size data 2620 indicates the size of those resized images. After the images are transmitted to the other device in the video conference, the receiving device can extract the size information from the bitstream and use the size information to properly decode the received images.
[0250] 27 conceptually illustrates an image resizing process 2700 performed by an encoder driver of a dual camera mobile device, such as driver 2600. Process 2700 begins by receiving (at 2705) an image (e.g., image 2605) captured by a camera of the dual camera mobile device. If the dual camera device is capturing images with both cameras, then in some embodiments process 2700 is performed on images from both cameras.
[0251] Next, process 2700 resizes (at 2710) the received image. As previously mentioned, different embodiments resize image 2605 differently. For example, image 2605 of FIG. 26 is scaled down and superimposed onto spatially redundant image 2635 to generate composite image 2640.
[0252] The process 2700 then sends (at 2715) the resized image (e.g., a composite image 2640 including the resized image 2630) to the encoder 2650 for encoding. In some embodiments of the process 2700, the resized image 2630 (included in the composite image 2640) is sent to the encoder 2650 through a rate controller that determines the bit rate at which the encoder will encode the image. The encoder 2650 of some embodiments compresses and encodes the image (e.g., using a discrete cosine transform, quantization, entropy coding, etc.) and returns a bitstream having the encoded image to the encoder driver 2600.
[0253] The process 2700 then sends (at 2720) data indicating the size of the resized image (e.g., size data 2620) to the bitstream manager. As shown in Figure 26, this operation is performed within the encoder driver 2600 in some embodiments (i.e., one module of the encoder driver 2600 sends the size data to another module of the encoder driver 2600).
[0254] After the resized image is encoded by the encoder 2650, the process 2700 receives (at 2725) a bitstream from the encoder. As shown, in some embodiments, the bitstream is received at a bitstream manager, which also receives the size data. The received bitstream includes the encoded composite image and may also include one or more additional images in the video sequence.
[0255] Process 2700 then terminates by inserting (at 2730) data indicating the size of the resized image (e.g., size data 2620) into the bitstream. As shown in Figure 26, this operation may also be performed by a bitstream manager in some embodiments. As discussed above, different embodiments insert the size data into different parts of the bitstream. In the illustrated example, size data 2620 is inserted at the beginning of bitstream 2655, as shown in resulting bitstream 2660. This bitstream can now be transmitted to other devices participating in the videoconference and decoded and viewed on the other devices.
[0256] In some embodiments, a decoder driver (e.g., driver 1640) performs the opposite function of the encoder driver: it extracts size data from a received bitstream, passes this bitstream to a decoder, and resizes the decoded image using the size data. Figure 28 conceptually illustrates a software architecture for such a decoder driver 2800 of some embodiments, and shows example bitstream manipulation and resizing operations performed by the decoder driver 2800 on an example bitstream 2825.
[0257] In some embodiments, the bitstream 2825 is a bitstream containing encoded images of a video captured by a device's camera in a video conference (e.g., a bitstream from an encoder driver such as driver 2600) and transmitted to a device on which the decoder driver 2800 operates. With reference to Figure 16, in some embodiments, the bitstream is received by the networking manager 1614, sent to a buffer 1616, read from the buffer 1616 by the video decompressor module 1626, and sent to the decoder driver 1640.
[0258] As shown, the decoder driver 2800 includes a processing layer 2805. The processing layer 2805 includes an image resizer 2810 and a bitstream manager 2820. In some embodiments, these modules 2810 and 2820 perform various operations on received images both before and after the images are encoded. In this example, the image resizer 2810 is shown as part of the processing layer 2805 of the decoder driver 2800, although some embodiments implement the image resizer as part of the image processing manager 1608 rather than the decoder driver (i.e., image resizing occurs after sending the image from the decoder driver 2800).
[0259] As shown, the bitstream manager 2820 of some embodiments receives a bitstream of one or more encoded pictures (i.e., pictures in a video sequence) and extracts size data from the bitstream before sending the bitstream to a decoder 2835 for decoding. For example, as shown in Figure 28, the bitstream manager 2820 receives a bitstream 2825 of an encoded picture, extracts size data 2815 from the bitstream 2825, and sends the resulting bitstream 2830 (without size data 2815) to the decoder 2835 for decoding. As shown, the bitstream manager 2820, in some embodiments, sends the extracted size data 2815 to the image resizer 2810.
[0260] The size data 2815 of some embodiments is the same as the size data 2620 inserted into the bitstream by the encoder driver 2600. As discussed above in the description of FIG. 26, the size data 2815 of some embodiments indicates the size of the subimage 2845 as the number of rows of pixels and the number of columns of pixels of the subimage 2845. The size data 2815 may also indicate the location of the subimage 2845 within the larger spatially redundant image 2840. In this example, the bitstream 2825 indicates the size data 2815 inserted at the beginning of the bitstream 2825. However, as discussed above, different embodiments insert the size data 2815 at different locations in the bitstream 2825.
[0261] The image resizer 2810 of some embodiments extracts sub-images from an image using size data received from the bitstream manager 2820. For example, Figure 28 shows the image resizer 2810 receiving an image 2840, which includes a sub-image 2845, from the decoder 2835. As shown, the image resizer 2810 of some embodiments extracts the sub-image 2845 from the image 2840. This extracted image may then be displayed on the dual camera mobile device.
[0262] 29 conceptually illustrates an image extraction process 2900 of some embodiments performed by a decoder driver of a device participating in a videoconference, such as driver 2800. The process begins by receiving (at 2905) a bitstream of encoded images (e.g., bitstream 2825). The bitstream may be transmitted from a device participating in the videoconference with the device on which the decoder driver is running, or may be stored in the device's storage. If the device is receiving images from multiple sources, some embodiments perform process 2900 for images from each source.
[0263] Next, the process 2900 extracts (at 2910) size data from the bitstream. As previously mentioned, this size data may be found in various locations within the bitstream. Some embodiments know where to look for the size data, while other embodiments look for a specific signature that indicates where the size data is located within the received bitstream. In some embodiments, the size data indicates the size (e.g., the number of pixels in each row and the number of pixels in each column) and location of the sub-image within the encoded image.
[0264] The process 2900 then sends (at 2915) the extracted size data to the image resizer. As shown in Figure 28, in some embodiments, this operation is performed within the decoder driver (i.e., one module of the decoder driver sends the size data to another module of the decoder driver).
[0265] The process 2900 also sends (at 2920) the bitstream to a decoder for decoding, which in some embodiments decompresses and decodes the bitstream (e.g., using an inverse discrete cosine transform, inverse quantization, etc.) and returns a reconstructed image to a decoder driver.
[0266] After the bitstream is decoded by the decoder, the process 2900 receives (at 2925) the decoded image from the decoder. As shown, some embodiments receive the image at an image resizer, which also receives size data from the bitstream manager. The process then extracts (at 2930) a sub-image from the decoded image using the received size data. As shown, a sub-image 2845 is extracted from the top left of the decoded image 2840, as indicated by the size data 2815. This extracted sub-image may now be displayed on a display device (e.g., the screen of a dual camera mobile device).
[0267] 3. Rate Controller In some embodiments, the two cameras of the device have different sets of characteristics. For example, in some embodiments, the front camera is a lower resolution camera optimized for capturing motion video images and the rear camera is a higher resolution camera optimized for capturing still images. For reasons such as cost, functionality and / or shape of the device, other embodiments may use a different combination of cameras with different characteristics.
[0268] Cameras with different characteristics may result in different artifacts. For example, a higher resolution camera may reveal more noise than a lower resolution camera. Images captured by a higher resolution camera may exhibit a higher level of spatial or temporal complexity than images captured by a lower resolution camera. Similarly, different cameras with different optical properties may result in different gamma values in the captured images. Different light sensing mechanisms used by different cameras to capture images may also result in different artifacts.
[0269] Some of these camera-specific artifacts hide artifacts produced from other sources. For example, in an image captured by a high-resolution camera with a high level of noise, artifacts that are by-products of the video encoding process are less visible. When encoding noise (such as quantization distortion) in a way that hides the camera-specific artifacts, the video encoding process can use a larger quantization step size to achieve a lower bit rate. On the other hand, when the camera introduces fewer artifacts (as in the case of a lower resolution camera), the video encoding process can use a finer quantization step size to avoid unacceptable levels of visual distortion due to quantization. Thus, a video encoding process optimized to exploit or compensate for these camera-specific characteristics can achieve a better rate-distortion tradeoff than a video encoding process that does not consider these camera-specific characteristics.
[0270] To take advantage of these camera-specific characteristics for the purpose of performing a rate-distortion tradeoff, some embodiments implement two video encoding processes, each optimized for each of the two cameras. Figure 30 shows an example of a system having two video encoding processes for two cameras 3060 and 3070. As shown in Figure 30, the system 3000 includes an encoder driver 3010, rate controllers 3020 and 3040, and a video encoder 3030. The encoder 3030 encodes the video images captured from the video cameras 3060 and 3070 into bitstreams 3080 and 3090.
[0271] In some embodiments, the video encoder driver 3010 is a software module that executes on one or more processing units. The video encoder driver 3010 provides an interface between the video encoder 3030 and other components of the system, such as the video camera, the image processing module, the network management module, and the storage buffer. The encoder driver 3010 controls the flow of captured video images from the camera and image processing module to the video encoder 3030, and also provides a conduit (conduit 1) for the encoded bitstreams 3080 and 3090 to the storage buffer and the network management module.
[0272] As shown in Figure 30, the encoder driver 3010 includes two different instances of a rate controller, 3020 and 3040. These multiple instances may be two different rate controllers for two different cameras, or may be one rate controller configured in two different ways for two different cameras. Specifically, in some embodiments, the two rate controllers 3020 and 3040 represent two separate rate controllers. Alternatively, in other embodiments, the two rate controllers 3020 and 3040 are two different configurations of a single rate controller.
[0273] FIG. 30 also illustrates the encoder driver 3010 including a state buffer 3015 that stores encoded state information for rate control operations used during the videoconference. Specifically, in some embodiments, two different rate controllers or two different configurations of the same rate controller share the same encoding state information stored in the state buffer 3015 during the videoconference. Sharing state information in this manner allows for uniform rate controller operation in a dual video capture videoconference. This sharing also allows for optimal video encoding during camera switching operations during a single video capture videoconference (i.e., allowing the rate control operation for encoding the video captured by the current camera to use the encoding state information maintained by the rate control operation for encoding the video captured by the immediately preceding camera). Although FIG. 30 illustrates the state buffer 3015 as part of the encoder driver 3010, other embodiments implement the state buffer 3015 outside of the encoder driver 3010.
[0274] In the state buffer 3015, different embodiments store different types of data (e.g., different types of encoding parameters) that represent encoding state information. One example of such encoding state information is the current target bit rate for a video conference. One scheme for identifying the target bit rate is described above in Section III.B. Other examples of such encoding state information include one or more of buffer fullness, maximum buffer fullness, and bit rate of recently encoded frames, among other encoding state information.
[0275] The rate controller can then use the target bit rate (or another encoding state parameter stored in the state buffer) to calculate one or more parameters used in its rate control operation. For example, as described further below, the rate controller of some embodiments uses the current target bit rate to calculate a quantization parameter QP for a macroblock or frame. By way of example, in some embodiments, the current target bit rate is used to calculate a quantization adjustment parameter that derives the quantization parameter QP for the macroblock and / or frame. Thus, during a camera switching operation of a video conference, by sharing the target bit rate between two rate control operations (of two rate controllers, or of two different configurations of one rate controller), the rate control operation for encoding the video captured by the current camera benefits from the encoding state data from the previous rate control operation for encoding the video captured by the previous camera.
[0276] FIG. 30 illustrates the encoder driver 3010 as including two different rate-controller instances 3020 and 3040. However, in other embodiments, these rate controller instances 3020 and 3040 are incorporated into a video encoder 3030. The video encoder 3030 encodes video images captured from cameras 3060 and 3070 into digital bitstreams 3080 and 3090. In some embodiments, the video encoder generates bitstreams that conform to conventional video encoding standards (e.g., H.264 MPEG-4). In some of these embodiments, the video encoder performs encoding operations including motion estimation, discrete cosine transform ("DCT"), quantization, and entropy coding. The video encoder also performs a decoding operation, which is the inverse function of the encoding operations.
[0277] In some embodiments, the encoder 3030 includes a quantizer module 3032 for performing quantization. The quantizer module is controlled by a quantization parameter 3022 or 3042 from a rate controller 3020 or 3040. In some embodiments, each quantization parameter is set by a corresponding rate controller and is a function of one or more attributes of the camera associated with the rate controller, as described further below. The rate controller can decrease the number of bits used for encoding according to a coarser quantization step size setting or increase the number of bits used by a finer quantization step size setting. By controlling the quantization step size, the rate controller also determines how much distortion is being introduced into the encoded video image. Thus, the rate controller can perform a tradeoff between bit rate and image quality. In performing the rate-distortion tradeoff, the rate controller monitors the bit rate to avoid overflowing a memory buffer, underflowing a memory buffer, or exceeding the capacity of the transmission channel. The rate controller must also control the bit rate to provide the highest possible image quality and to avoid unacceptable distortion of image quality due to quantization. In some embodiments, each rate controller stores the monitored data as a collection of state data values in a state buffer 3015. In some embodiments, the rate controllers 3020 and 3040 use camera-specific attributes to optimize the rate-distortion tradeoff.
[0278] In some embodiments, each rate controller optimizes the rate-distortion tradeoff by directly applying modification factors to its quantization parameters. In some of these embodiments, the modification factors are predetermined and built into the device along with the camera. The device does not need to dynamically calculate these modification factors. In other embodiments, the system uses the incoming images captured by the camera to dynamically determine appropriate modification factors specific to the camera. In some of these embodiments, the system analyzes a sequence of incoming video images captured by the camera over multiple encoding passes to gather camera-specific statistics. The system then uses these statistics to derive modification factors for the camera-optimized quantization parameters.
[0279] In some embodiments, these camera-specific modification factors are applied to the quantization parameters via visual masking attributes of the video images. The visual masking attributes of an image or image portion are an indication of how many coding artifacts are tolerable for the image or image portion. Some embodiments calculate visual masking attributes that quantify the luminance energy of the image or image portion, while other embodiments calculate visual masking attributes that quantify the activity energy or complexity of the image or image portion. Regardless of how the visual masking attributes are calculated, some embodiments use the visual masking attributes to calculate modified or masked quantization parameters for the video frames. Some of these embodiments use the visual masking attributes to calculate modified or masked quantization parameters for the video frames. Some of these embodiments use the visual masking attributes to calculate the masked quantization parameters via a frame-level visual masking attribute φ frame and the reference visual masking attribute φ R In some embodiments, the visual masking attribute φ frame and φ R The quantization parameter changed by is expressed as follows:
[0280] MQP frame =QP nom +β frame*(φ frame -φ R ) / φ R (1) In the above formula, MQP frame is the masked or modified quantization parameter of the frame, QP nom is the initial or nominal quantization value, and β frame is a constant that fits the local statistics. In some embodiments, the reference visual masking attribute φ R and the nominal quantization parameter QP nom is predetermined from an initial or periodic assessment of network conditions.
[0281] In some embodiments, the visual masking attribute φ in Eq. frame is calculated as follows:
[0282] φ frame = C (E avgFrameLuma) β (D avgFrameSAD) α (2) where avgFrameLuma is the average luminance value of the frame and avgFrameSAD is the average sum of absolute differences of the frame. The constants α, β, C, D and E are adapted to local statistics. These constants are adapted to the intrinsic properties of the camera in some embodiments.
[0283] Some embodiments also calculate a masked quantization parameter for a portion of a video image, such as a macroblock. In these examples, the masked quantization parameter is calculated based on the macroblock visual masking attribute φ, as follows: MB It is calculated as a function of
[0284] MQP MB =MQP frame +β MB *(φ MB -φ frame ) / φ frame (3) In the above formula, β MB is a constant that fits the local statistics, and MQP frameIn some embodiments, the visual masking attribute φ in Equation (3) is calculated using Equation (1) and Equation (2). MB is calculated as follows:
[0285] φ MB = A (C avgMBLuma) β (B MBSAD) α (4) where avgMBLuma is the average luminance value of the macroblock and avgMBSAD is the average sum of absolute differences of the macroblock. The constants α, β, A, B and C are adapted to local statistics. These constants are adapted to the specific characteristics of the camera in some embodiments.
[0286] Some embodiments perform camera-specific rate control by calculating the quantization parameter using only a single camera-specific coefficient, rather than calculating a modified quantization parameter using multiple camera-specific constants as described above. For example, the visual masking attribute φ frame and φ MB and the quantization parameter QP frame Given this, some embodiments use a single camera-specific coefficient μ to calculate the quantization parameter for a macroblock as follows:
[0287] QP MB =μ*(φ frame -φ MB )+QP frame (5) To calculate equation (5), some embodiments may use the frame and macroblock complexity measures as visual masking attributes φ frame and φ MB Use as.
[0288] Some embodiments include QP MB For example, in some embodiments, we apply different camera-specific coefficients to the calculation of QP MB is calculated as follows:
[0289] QP MB =ρ·(1-φMB / φ frame )·QP frame +QP frame (6) In the above equation, ρ is a coefficient that is tailored to the inherent characteristics of the camera.
[0290] As mentioned above, state buffer 3015 stores encoding state information that two different rate controller instances 3020 and 3040 may share during a videoconference in order to obtain better encoding results from their rate control operations. T is an example of such shared state information in some embodiments. This rate is the desired bit rate for encoding the sequence of frames. Typically, this bit rate is expressed in bits per second and is determined based on a process such as the process described above in Section III.B.
[0291] As mentioned above, the rate controller of some embodiments uses the target bitrate to calculate the frame and / or macroblock quantization parameter(s) QP to output to the video encoder 3030. For example, some embodiments use the current target bitrate to calculate a quantization adjustment parameter from which the quantization parameter QP for the macroblock and / or frame is derived. In some embodiments, the quantization adjustment parameter is expressed as a ratio calculated by dividing either the bitrate of the previous frame or a running average of the bitrates of the previous frames by the current target bitrate. In other embodiments, the adjustment parameter is not calculated exactly in this way, but is more generally (1) proportional to the bitrate of the previous frame or a running average of the bitrates of the previous frames, and (2) inversely proportional to the current target bitrate.
[0292] After calculating such quantization adjustment parameters, the rate controller of some embodiments uses the parameters to calculate and adjust the quantization parameter(s) for the macroblock and / or frame. One way to make such an adjustment is to multiply the calculated quantization parameter(s) for the macroblock and / or frame by the quantization adjustment parameters. Another way to make this adjustment is to calculate a quantization parameter offset value from the quantization adjustment parameters and then apply (e.g., subtract) this parameter offset to the calculated quantization parameter(s) for the macroblock and / or frame. The rate controller of these embodiments then outputs the adjusted quantization parameter(s) for the macroblock and / or frame to the video encoder 3030.
[0293] In other embodiments, the rate controller uses the target bit rate to calculate other parameters used in its rate control operations, for example, in some embodiments the rate controller uses the bit rate to modify the visual masking strength of a macroblock or frame.
[0294] G. Networking Manager Figure 31 conceptually illustrates a software architecture for a networking manager 3100 of some embodiments, such as the networking manager 1614 shown in Figure 16. As described above, the networking manager 3100 manages the network connection (e.g., establishing the connection, monitoring the connection, adjusting the connection, disconnecting the connection, etc.) between the dual camera mobile device on which the networking manager 3100 operates and the remote devices in a videoconference. During the videoconference, the networking manager 3100 of some embodiments also processes data for transmission to the remote devices and processes data received from the remote devices.
[0295] As shown in FIG. 31, the networking manager 3100 includes a session negotiation manager 3105, a transmitter module 3115, a generic transmission buffer 3120, a generic transmission buffer manager 3122, a virtual transport protocol (VTP) manager 3125, a receiver module 3130, and a media transport manager 3135.
[0296] The session negotiation manager 3105 includes a protocol manager 3110. The protocol manager 3110 ensures that the sender module 3115 transmits data to remote devices during a videoconference using the appropriate communication protocol and enforces the rules of the communication protocol used. Some embodiments of the protocol manager 3110 support several communication protocols, such as Real-time Transport Protocol (RTP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Hypertext Transfer Protocol (HTTP), among others.
[0297] The session negotiation manager 3105 is responsible for establishing connections between the dual camera mobile device and one or more remote devices participating in a videoconference and for tearing down these connections after the conference. In some embodiments, the session negotiation manager 3105 is also responsible for establishing a multimedia communication session (e.g., for sending and receiving video and / or audio streams) between the dual camera mobile device and the remote devices in the videoconference (e.g., using Session Initiation Protocol (SIP)).
[0298] The session negotiation manager 3105 also receives feedback data from the media transport manager 3135 and determines the operation of the generic transmission buffer 3120 through the generic transmission buffer manager 3122 (e.g., whether to transmit or discard packets / frames) based on the feedback data. In some embodiments, the feedback may include one-way latency and bandwidth estimated bit rate. In other embodiments, the feedback includes packet loss information and round trip delay time (e.g., determined based on packets sent to a remote device in a video conference and receiving an acknowledgment from that device). Based on information from the media transport manager 3135, the session negotiation manager 3105 can determine whether too many packets are being sent and instruct the generic transmission buffer manager 3122 to send fewer packets to the generic transmission buffer 3120 (i.e., adjust the bit rate as described in FIG. 15).
[0299] The sender module 3115 reads the encoded images (e.g., as a bitstream) from a video buffer (e.g., buffer 1612 of FIG. 16) and packetizes the images for transmission to a remote device in a video conference through a generic transmission buffer 3120 and a virtual transport protocol manager 3125. The manner in which the encoded images are created and sent to the sender module 3115 may be based on instructions or data received from the media transport manager 3135 and / or the session negotiation manager 3105. In some embodiments, packetizing the images also includes breaking down the received bitstream into a group of packets, each having a particular size (i.e., a size specified by the session negotiation manager 3105 according to a particular protocol), and adding any necessary headers (e.g., address headers, protocol-specific headers, etc.).
[0300] The generic transmit buffer manager 3122 controls the operation of the generic transmit buffer 3120 based on data and / or instructions received from the session negotiation manager 3105. For example, the generic transmit buffer manager 3122 may be instructed to instruct the generic transmit buffer 3120 to send data, to stop sending data, to discard data, etc. As mentioned above, in some embodiments, a remote device participating in a conference may appear to be discarding packets, but this may be recognized based on an acknowledgement received from the remote device. To reduce packet discards, the generic transmit buffer manager 3122 may be instructed to transmit packets to the remote device at a slower rate.
[0301] The generic transmit buffer 3120 stores data received from the transmitter module 3115 and transmits the data to remote devices via the VTP manager 3125. As previously described, the generic transmit buffer 3120 may discard data (e.g., images of video) based on instructions received from the generic transmit buffer manager 3122.
[0302] In some embodiments, RTP is used to communicate data packets (e.g., audio and video packets) over UDP during a video conference. Other embodiments use RTP to communicate data packets over TCP during a video conference. In different embodiments, other transport layer protocols may be used.
[0303] Some embodiments define a particular communication channel between two mobile devices by a pair of port numbers (i.e., source port number and destination port number). For example, one communication channel between the mobile devices can be defined by one port number pair (e.g., source port 50 and destination port 100), and another different communication channel between the mobile devices can be defined by another different port number pair (e.g., source port 75 and destination port 150). Some embodiments also use a pair of Internet Protocol (IP) addresses to define the communication channels. In some embodiments, different communication channels are used to transport different types of data packets. For example, video data packets, audio data packets, and control signal data packets can be transported in separate communication channels. Thus, a video communication channel carries video data packets, and an audio communication channel carries audio data packets.
[0304] In some embodiments, the control communication channel is for transmitting messages between the local mobile device and the remote device during the videoconference. Examples of such message transmission include sending and receiving requests, notifications, and acknowledgements to such requests and notifications. Another example of message transmission includes sending remote control command messages from one device to another device. For example, remote control operations described below (e.g., commanding a device to only transmit images from a particular camera or to only capture images with a particular camera) may be performed by sending commands from the local device to the remote device through the control communication channel in order for the local device to remotely control the operation of the remote device. Different embodiments implement the control communication using various protocols such as Real-time Transport Control Protocol (RTCP), RTP extensions, SIP, etc. For example, some embodiments use RTP extensions to relay one set of control messages between two mobile devices in a videoconference and use SIP packets to relay another set of control messages between the mobile devices during the videoconference.
[0305] The VTP manager 3125 of some embodiments allows different types of data packets designated to be transmitted over different communication channels (e.g., using different port number pairs) to be transmitted over a single communication channel (e.g., using the same port number pair). One technique for doing this includes identifying the data packet type, identifying the communication channel designated to transmit the data packet over by extracting the data packet's designated port number pair, and designating the data packets to be transmitted over the single communication channel by changing the data packet's port number pair to be the port number pair of the single communication channel (i.e., all data packets are transmitted over the same port number pair).
[0306] To keep track of the original port number pairs for each type of data packet, some embodiments store a mapping of original port number pairs to data packet types. Some of these embodiments then use the packet type field of the protocol to distinguish different packets that are multiplexed into one communication channel. For example, some embodiments that have a VTP manager multiplex audio, video, and control packets into one RTP stream use the RTP packet type field to distinguish audio, video, and control packets that are transmitted on one RTP channel to other devices in a video conference. In some of these embodiments, the VTP manager also routes control message transmissions in SIP packets to other devices.
[0307] In some embodiments, data packet signatures (i.e., packet header formats) are identified and examined to distinguish between different packets communicated using different protocols (e.g., to distinguish between packets carried using RTP and packets carried using SIP). In such embodiments, after data packets of different protocols are determined, fields of data packets using the same protocol (e.g., audio and video data using RTP) are examined as described above to identify different data types. In this manner, VTP manager 3125 transmits different data packets intended to be transmitted over different communication channels over a single communication channel.
[0308] Although one means of combining different types of data over a single communication channel has been described above, other embodiments utilize other techniques to multiplex different packet types into one communication stream. For example, one technique in some embodiments involves tracking the original port number pairs of the data packets and storing the original port number pairs in the data packets themselves for later retrieval. Still other ways exist for combining different types of data between two video conference participants into one port pair channel.
[0309] When the VTP manager 3125 receives a data packet from a remote device through a virtual communication channel, the VTP manager 3125 examines the signature of the data packet to identify different packets transmitted using different protocols. Such signatures can be used to distinguish SIP packets from RTP packets. The VTP manager of some embodiments also uses some or all of the packet type fields of the packets to demultiplex various different types of packets (e.g., audio packets, video packets, and control packets) that are multiplexed onto a single virtual channel. After identifying these different types of packets, the VTP manager associates each different type of packet with its corresponding port pair number based on the port pair number to packet type mappings that it manages. The VTP manager 3125 then modifies the port number pair of the data packet with the identified port number pair and forwards the data packet to be demultiplexed. In other embodiments that use different techniques to multiplex different packet types onto a single channel, the VTP manager uses different techniques to parse the packets.
[0310] By using such techniques for multiplexing and demultiplexing different packets, the VTP manager 3125 creates a single virtual communication channel (e.g., a single port number pair), transmits video data, audio data, and control signal data through this single virtual communication channel, and receives audio packets, video packets, and control packets from a remote device through this single virtual communication channel. Thus, from the perspective of the network, data is transmitted through this single virtual communication channel, but from the perspective of the session negotiation manager 3105 and the protocol manager 3110, video data, audio data, and control signal data are transmitted through different communication channels.
[0311] Images transmitted from a remote device in a video conference, as well as images transmitted to a remote device in a video conference, are received in packet format. The receiver module 3130 receives the packets, depacketizes the packets to reconstruct the image, and then stores the image to be decoded in a video buffer (e.g., buffer 1616 of FIG. 16). In some embodiments, depacketizing the image includes removing any headers and reconstructing a bitstream having only image data (and possibly size data) from the packets.
[0312] The media transport manager 3135 processes feedback data received from the network (e.g., one-way latency, bandwidth estimated bit rate, packet loss data, round trip delay time data, etc.) to dynamically and adaptively adjust the speed (i.e., bit rate) of data transmission. The media transport manager 3135 may also send feedback data to the videoconferencing manager 1604 to control error resilience based on the processed feedback data in some other embodiments and to adjust other operations of the videoconferencing module 1602, such as scaling, resizing, and encoding. In addition to having the general transmission buffer discard packets when remote devices in a conference cannot process all of the packets, the videoconferencing module and encoder can use a lower bit rate to encode the images so that fewer packets are transmitted per image.
[0313] In some embodiments, the media transport manager 3135 may also monitor other variables of the device, such as power consumption and thermal levels, which may affect how the camera's operating power mode is configured, as described above. This data may be used as additional input to the feedback data (e.g., if the device is getting too hot, the media transport manager 3135 may try to slow it down).
[0314] Some example operations of the networking manager 3100 will now be described with respect to Fig. 16. The transmission of images captured by the cameras of the dual camera mobile device to a remote device in a video conference will be described first, followed by the reception of images from the remote device. The transmitter module 3115 reads the encoded image from the buffer 1612, which can be transmitted to the remote device in the video conference.
[0315] The protocol manager 3110 determines the appropriate protocol to use (e.g., RTP for transmitting audio and video), and the session negotiation manager 3105 informs the sender module 3115 of such protocol. The sender module 3115 then packetizes the images and sends the packetized images to the generic transmission buffer 3120. The generic transmission buffer manager 3122 receives instructions from the session negotiation manager 3105 and instructs the generic transmission buffer 3120 to transmit or discard the images. The VTP manager 3125 receives the packets from the generic transmission buffer 3120 and processes the packets for transmission to a remote device over a single communication channel.
[0316] When receiving an image from a remote device, the VTP manager 3125 receives the packetized image from the remote device through a virtualized single communication channel and processes the packets to direct the image to the receiver module 3130 through a communication channel (e.g., a video communication channel) assigned to receive the image.
[0317] The receiver module 3130 depacketizes the packets to reconstruct the image and sends the image to the buffer 1616 for decoding by the decoder 1660. The receiver module 3130 also forwards control signal messages (e.g., acknowledgments of packets received from remote devices in a video conference) to the media transport manager 3135.
[0318] Several exemplary operations of the networking manager 3100 have been described above. These are merely illustrative examples because various other embodiments perform these or different operations using different modules or with functionality spread differently among the modules. Moreover, additional operations such as dynamic bit rate adjustment may be performed by the modules of the networking manager 3100 or other modules.
[0319] IV. Coordination and control actions during meetings A. Picture in Picture Changes 1. Corner snap According to some embodiments of the present invention, a user of a dual camera mobile device modifies the composite display displayed on the device by moving one or more of the display areas that form the composite display. One such example is moving an inset display area of a PIP display. Figure 32 illustrates such an example performed during a video conference. In a video conference, a user may want to move a foreground inset display area for various reasons, such as if this area is occluding an area of interest in the background display area.
[0320] FIG. 32 illustrates the movement of the inset viewport 3240 in the UI 3205 of a device as five different stages 3210, 3215, 3220, 3225, and 3230 of the UI. The first stage 3210 illustrates the UI 3205 during a videoconference between a local user of the device and a remote user of a remote device. The UI 3205 of FIG. 32 illustrates a PIP display that is the same PIP display shown in the fifth stage of FIG. 11 after the videoconference has begun. In this example, a video captured by the local user's device is displayed in the inset viewport 3240, and a video captured by the remote user's device is displayed in the background viewport 3235. As shown, the viewport 1155 includes a selectable UI item 3245 for ending the videoconference. In some embodiments, the layout of the viewport 1155 is the same as the layout of the viewport 1155 of FIG. 12 described above.
[0321] A second phase 3215 indicates that the user initiates the snap-to-corner action by selecting the insertion viewport 3240. In this example, the selection is made by placing a finger 3255 anywhere within the insertion viewport 3240. As shown, the selection is indicated as a thick border 3260 of the insertion viewport 3240. Different embodiments may indicate such a selection in different ways, such as by highlighting the viewport 3240, by vibrating the viewport 3240, etc.
[0322] The third stage 3220 shows the UI 3205 after a user has begun to move the insert viewer 3240 of the PIP display 3250 from one area within the PIP display 3250 to another area within the display. In this example, the insert viewer 3240 begins to move from the lower left corner of the PIP display 3250 to the lower right corner of the display, as indicated by arrow 3265. In this example, the insert viewer 3240 is moved by the user dragging his or her finger 3255 toward the lower right corner of the PIP display 3250 after selecting the insert viewer in the second stage 3215. Some embodiments provide other techniques for moving the insert viewer 3240 in the PIP display 3250.
[0323] The fourth stage 3225 shows the UI 3205 in a state after the user has lifted their finger 3255 off the screen of the device 3200. In this state, the inset view 3240 is still moving towards the bottom right corner of the PIP display 3250, which was identified based on the movement of the user's finger in the third stage 3220. In other words, after the finger 3255 starts moving the inset view 3240 towards the bottom right corner of the PIP display 3250, the UI 3205 maintains this movement even after the finger 3255 is lifted. To maintain this movement, the UI 3205 of some embodiments requires that the user's dragging motion be greater than a certain threshold amount (e.g., greater than a certain distance or a certain length of time) before the user lifts their finger 3255. Otherwise, these embodiments keep the inset view 3240 in its original left corner position after moving it slightly or not at all.
[0324] However, some embodiments allow the inset viewport to move even after the user stops the dragging motion before the inset viewport reaches its new position, while other embodiments require the user to maintain their dragging motion until the inset viewport reaches its new position. Some embodiments provide still other techniques for moving the inset viewport. For example, some embodiments may require the user to specify where the inset viewport 3240 should be oriented before it actually begins to move. Some embodiments may allow the viewport to slide and corner-oriented snapping motions to occur simply by tilting the mobile device to different angles.
[0325] The fifth stage 3230 shows the UI 3205 after the inset viewport 3240 has reached its new position at the bottom right corner of the PIP view 3250. The removal of the thick border 3260 in the fifth stage 3230 indicates that the snap to corner operation is complete.
[0326] To facilitate the movements illustrated in the third stage 3220, the fourth stage 3225, and the fifth stage 3230 above, the UI 3205 of some embodiments employs snapping rules that allow the inset viewport 3240 to quickly snap to a corner of the PIP display 3250 as the user moves the inset viewport 3240 toward that corner. For example, if the user drags the inset viewport 3240 toward a particular corner more than a threshold amount, the UI 3205 of some embodiments identifies the direction of the movement of the inset viewport 3240, determines that the movement has exceeded the threshold amount, and then automatically moves the inset viewport 3240 without further user input of the next grid point in the UI 3205 to which the inset viewport 3240 can be snapped. In some embodiments, the only grid points provided to snap the inset viewport 3240 to are the grid points at the four corners of the PIP display 3250. Other embodiments provide other grid points within the UI 3205 (eg, within the PIP display 3250) to which the inset viewport 3240 can snap (i.e., be placed or aligned with an edge or vertex of the area 3240).
[0327] Still other embodiments may not employ grid points so that the inset viewport may be placed at any point on the PIP display 3250. Still other embodiments provide the ability for a user to turn on or off the snap to grid points feature of the UI. Additionally, different embodiments may allow a user to snap to corners using various items, such as icons, in addition to the video captured from the device.
[0328] 33 illustrates two other examples 3330 and 3335 of corner-directed snap actions in the UI 3205. These other corner-directed snap actions show that the inset viewport 3240 is moved vertically or diagonally within the PIP display 3250 based on a vertical or diagonal dragging action of the user.
[0329] 32 and 33 illustrate moving an inset view in a PIP display, one skilled in the art will appreciate that other embodiments utilize similar techniques to move view areas in other types of PIP displays or other types of composite displays. For example, as described further below, the PIP display of some embodiments has two or more foreground inset views, which may be moved in the PIP display using techniques similar to those described above with respect to FIG. 32 and FIG. 33. Some embodiments also use similar techniques to move view areas in a composite display (e.g., moving a view area from the left side of the screen to the right side of the screen through a user drag movement). Moreover, moving view area(s) in a composite display may cause changes in image processing operations of the dual camera mobile device, such as causing the videoconferencing manager 1604 to recombine the view area in the composite display in response to user input. As described further below, some embodiments employ a snap and push technique in which pushing a first view area from a first position moves a second view area from a third position to the first position.
[0330] 2. Rotation Some embodiments rotate the PIP display presented during a video conference when a user of a portable device used for video conferencing rotates the device during the conference. FIG. 34 illustrates the rotation of the UI 1105 of the device 3400 when the device is rotated from a vertical position to a horizontal position. The device 3400 is held vertically when the long side of the screen is vertical, and the device 3400 is held horizontally when the long side of the screen is horizontal. In the example shown in FIG. 34, the UI 1105 rotates from a portrait view optimized for holding the device vertically to a landscape view optimized for holding the device 3400 horizontally. This rotation feature allows the user to see the UI 1105 displayed in an upright position whether the portable device 3400 is held vertically or horizontally.
[0331] FIG. 34 illustrates the rotation of the UI 1105 as six different operational stages 3410, 3415, 3420, 3425, 3430, and 3435. The first stage 3410 illustrates the UI 1105 during a videoconference between a local user of the device and a remote user of a remote device. The UI 1105 of FIG. 34 illustrates a PIP display 1180, which is the same PIP display shown in the fifth stage of FIG. 11 after the videoconference has been established. In this example, a video captured by the local user's device is displayed in the inset display area 1160, and a video captured by the remote user's device is displayed in the background display area 1170. The display area 1155 below the PIP display 1180 includes a selectable UI item 3485 (e.g., an end conference button 3485) that a user may select (e.g., via a single finger tap) to end the videoconference.
[0332] A second stage 3415 shows the UI 1105 after the user begins to tilt the device 3400 sideways. In this example, the user begins to tilt the device 3400 from being held vertically to being held horizontally, as indicated by arrow 3460. The appearance of the UI 1105 does not change. In other situations, the user may want to tilt the device 3400 from being held horizontally to being held vertically, and in these situations, the UI 1105 switches from a horizontally optimized view to a vertically optimized view.
[0333] A third stage 3420 shows the UI 1105 in a state after the device 3400 has been tilted from being held vertically to being held horizontally. In this state, the appearance of the UI 1105 has not yet changed. In some embodiments, a rotation action is triggered after the device 3400 has been tilted past a threshold amount and held beyond this point for a period of time. In the example shown in FIG. 34, it is assumed that the threshold amount and speed of rotation will not rotate the UI 1105 until a short interval has elapsed after the device is placed in a horizontal position. Different embodiments have different threshold amounts and wait periods for triggering a rotation action. For example, some embodiments may have a lower threshold for triggering a rotation action such that the UI 1105 always appears to be displayed in an upright position regardless of the orientation of the device 3400. In other embodiments, the user of the device 3400 may specify (e.g., through a menu preference) when a rotation action can be triggered. Also, some embodiments may not delay rotation after the device has been tilted beyond a threshold amount. Additionally, different embodiments may allow the rotation action to be triggered in different ways, such as by flipping a switch on the mobile device, by giving a voice command, upon selection through a menu, etc.
[0334] A fourth stage 3425 shows the UI 1105 after a rotation action has begun. Some embodiments animate the rotating viewports to provide the user with feedback regarding the rotation action. FIG. 34 shows an example of such an animation. Specifically, FIG. 34 shows the viewports 1180 and 1155 beginning to rotate together in the fourth stage 3425. The viewports 1180 and 1155 rotate about an axis 3465 (i.e., the z-axis) that passes through the center of the UI 1105. The viewports 1180 and 1155 rotate by the same amount in the opposite direction of the rotation of the device 3400 (e.g., through tilting the device 3400). In this example, because the device 3400 has rotated 90 degrees in a clockwise direction (by going from being held vertically to being held horizontally), the rotation action rotates the viewports 1180 and 1155 by 90 degrees in a counterclockwise direction. As the viewports 1180 and 1155 are rotated, the viewports 1180 and 1155 shrink proportionately to fit the UI 1105 so that the viewports 1180 and 1155 are still entirely visible on the UI 1105. Some embodiments may provide a message indicating this state of the device 3400 (e.g., by displaying the words "Rotating").
[0335] The fifth stage 3430 shows the UI 1105 after the viewports 1180 and 1155 have been rotated 90 degrees counterclockwise from a portrait view to a landscape view. At this stage, the viewports 1180 and 1155 have been rotated but have not yet expanded to fill the entire width of the UI 1105. Arrows 3475 indicate that at the end of the fifth stage, the viewports 1180 and 1155 begin to expand laterally to fill the entire width of the UI 1105. Different embodiments may not include this stage, as the expansion can be performed simultaneously with the rotation in the fourth stage 3425.
[0336] A sixth stage 3435 shows the UI 1105 after the viewports 1180 and 1155 have been expanded to occupy the entire display of the UI 1105. As noted above, other embodiments may implement this rotation differently. In some embodiments, simply rotating the device's screen beyond a threshold amount may trigger a rotation action, regardless of the device's 3400 orientation.
[0337] Also, other embodiments may provide different animations to indicate the rotation action. The rotation action performed in FIG. 34 involves the viewports 1180 and 1155 rotating about the center of the UI 1105. Alternatively, the viewports may rotate individually about the central axis of their respective viewports. One such approach is shown in FIG. 35. FIG. 35 shows an alternative way of animating the rotation of the viewports 1170 and 1160 of the PIP display 1180 of the UI 1105. The PIP display 1180 shown in FIG. 35 is the same PIP display 1180 shown in FIG.
[0338] Figure 35 illustrates the rotation of the PIP display 1180 as six different stages of operation 3410, 3415, 3420, 3525, 3530, and 3535. The first three stages of operation of the UI 1105 are identical to the first three stages of operation described with respect to the UI 1105 of Figure 34. In the third stage for both Figures 34 and 35, the device 3500 has moved from being held vertically to being held horizontally and rotation of the UI 1105 has not yet begun.
[0339] A fourth stage 3525 illustrates an alternative method of animating a rotation. In this stage, the rotation operation begins. Specifically, in the fourth stage 3525, the viewports 1170 and 1160 are shown beginning to rotate. The viewports 1170 and 1160 rotate about axes 3567 and 3565 (i.e., the z-axis), respectively, which pass through the respective centers of the viewports. The viewports 1170 and 1160 rotate by the same amount in the opposite direction of the rotation of the device 3500 (e.g., due to tilting the device 3500). Similar to what is shown in the fourth stage 3425 of FIG. 34 above, the rotation operation rotates the viewports 1170 and 1160 by 90 degrees in the counterclockwise direction because the device 3500 has rotated 90 degrees in the clockwise direction (by going from being held vertically to being held horizontally). When the viewports 1170 and 1160 are rotated, the viewports 1170 and 1160 shrink proportionately to fit the UI 1105 so that the viewports 1170 and 1160 are still entirely visible on the UI 1105.
[0340] The fifth stage 3530 shows the UI 1105 after each of the viewports 1170 and 1160 has been rotated 90 degrees counterclockwise from a portrait view to a landscape view. At this stage, the viewports 1170 and 1160 have been rotated but have not yet expanded to the entire width of the UI 1105. Furthermore, the viewport 1160 has not been moved to its final position. The final position of the inset viewport 1160 within the PIP display 1180 is determined by the position of the inset viewport 1160 within the PIP display 1180 shown in the first stage 3410 (e.g., the inset viewport 1160 in the lower left corner of the PIP display 1180). At this stage, the inset viewport 1160 is still in the upper right corner of the UI 1105.
[0341] Arrow 3580 indicates that at the end of the fifth stage 3530, viewports 1170 and 1160 begin to expand laterally until main viewport 1170 fills the entire width of the device's UI 1105, which is held horizontally. Additionally, arrow 3575 indicates that inset viewport 1160 slides into the lower left corner of PIP display 1180.
[0342] Different embodiments may implement this differently. In some embodiments, the movement of the insert viewport 1160 may occur simultaneously or sequentially with the expansion of the main viewport 1170. Additionally, some embodiments may resize the insert viewport 1160 before, during, or after the expansion of the main viewport 1170 to generate the new PIP display 1180. In this example, the viewport 1155 is hidden while the viewports 1160 and 1170 are rotated. However, in some embodiments, the viewport 1155 may remain in the UI 1105 during the rotation and rotate along with the viewports 1160 and 1170.
[0343] The sixth stage 3535 shows the UI 1105 after the inset viewport 1160 has reached its new position and the viewports 1160 and 1170 have expanded appropriately to fill the entire width of the UI 1105. In this example, the inset viewport 1160 is now in the lower left corner of the PIP display 1180 and overlaps the main viewport 1170. The PIP display 1180 now has the same display configuration as the PIP display 1180 in the first stage 3410. The appearance of the viewport 1155 below the PIP display 1180 in the sixth stage indicates that the rotation operation is complete. As previously mentioned, simply rotating the device's screen beyond a threshold amount can trigger a rotation operation, regardless of the orientation of the device 3500.
[0344] In the examples discussed above with respect to Figures 34 and 35, the orientation of the viewport 1170 also changes (i.e., from portrait to landscape). That is, after the viewport 1170 is rotated in the third stage 3420, the orientation of the viewport 1170 changes from portrait to landscape by expanding the PIP display 1180 horizontally to fill the entire UI 1105. In some embodiments, when the device 3500 is rotated, the video captured by the remote device rotates, but the orientation of the viewport displaying the video captured by the remote device remains unchanged. One such example is shown in Figure 36, which is similar to Figure 35, except that the video displayed in the viewport 1170 rotates, but the viewport 1170 remains displayed in portrait orientation.
[0345] Figure 36 also illustrates an example of a rotation operation in which the viewport 1155 remains in the same position (rather than rotating and expanding horizontally to fill the PIP display 1180 as shown in Figure 35). Additionally, this figure includes a layout of the viewport 1155 that is the same as the layout of the viewport 1155 in Figure 12 described above. As shown, as the device 3500 rotates in steps 3640, 3645, 3650, 3655, 3685, and 3690, the viewport 1155 remains in the same position.
[0346] Some embodiments provide a rotation operation in which the orientation of the viewport displaying the video captured by the local device changes to reflect the orientation of the local device after the rotation operation is performed on the local device (rather than remaining in the same orientation as shown in FIG. 35). FIG. 36 illustrates an example of such a rotation operation of the UI 1105 as six different stages 3640, 3645, 3650, 3655, 3685, and 3690. In this figure, the first stage 3640 shows the inset viewport 1160 displaying the video captured by the camera of the device 3500 in a portrait orientation. The second stage 3645 and the third stage 3650 are similar to the second stage 3415 and the third stage 3420 of FIG. 35 as they show the tilt of the device 3500 at various stages of the rotation operation. At this point, the camera of the device 3500 is capturing an image in a landscape orientation. To illustrate this transition, some embodiments provide the animation shown in the fourth stage 3655 and the fifth stage 3685, while other embodiments do not provide any animation at all.
[0347] In a fourth stage 3655, the image displayed in the inset viewing area 1160 is rotated, but the inset viewing area 1160 itself is not rotated, because the tilt of the device 3500 in the second stage 3445 and the third stage 3650 rotated the inset viewing area 1160 sideways. In a fifth stage 3685, the rotated image in the inset viewing area 1160 expands horizontally to fill the inset viewing area 1160, and the inset viewing area 1160 begins to move toward the lower left area of the PIP display 1180 to place the inset viewing area 1160 in the same relative position as the inset viewing area 1160 in the PIP display of the first stage 3640.
[0348] In some embodiments, the orientation of the viewport displaying the video captured by the remote device also changes to reflect the orientation of the remote device after a rotate operation is performed on the remote device. Figure 37 shows four different stages of the UI 1105 of device 3500 in which (1) the orientation of the viewport displaying the video captured by the local device (viewport 1160 in this example) changes to reflect the orientation of the local device after a rotate operation is performed on the local device, and (2) the orientation of the viewport displaying the video captured by the remote device (viewport 1170 in this example) changes to reflect the orientation of the remote device after a rotate operation is performed on the remote device.
[0349] In the first stage 3705, the UI 1105 is the same as the UI 1105 of FIG. 36. Specifically, the first stage 3705 shows the viewports 1160 and 1170 in a portrait orientation because the device 3500 is shown in a portrait orientation and the remote device is in a portrait orientation (not shown). From the first stage 3705 to the second stage 3710, a rotation operation is performed on the local device by rotating the device 3500 90 degrees from an upright position to a landscape position. The second stage 3710 shows the UI 1105 after the rotation operation of the device 3500 is completed. At this stage, the video displayed in the viewports 1170 and 1160 has been rotated to an upright position. However, because the rotation operation was performed only on the local device (i.e., device 3500), only the viewport 1160 of the locally captured video has been rotated from portrait to landscape. The viewport 1170 remains in a portrait orientation.
[0350] From second stage 3710 to third stage 3715, a rotation operation is performed on the remote device (not shown) by rotating the remote device from an upright position to a landscape position. Third stage 3715 shows UI 1105 after the rotation operation on the remote device is completed. At this stage, the video displayed in viewport 1170 and remotely captured video viewport 1170 have been rotated from portrait to landscape orientation because the rotation operation is performed only on the remote device. Thus, this stage of UI 1105 displays both locally captured video viewport 1160 and remotely captured video viewport 1170 in landscape orientation.
[0351] From third stage 3715 to fourth stage 3720, a rotation operation is performed on the local device by rotating device 3500 90 degrees from a landscape position to an upright position. Fourth stage 3720 shows UI 1105 after this rotation operation is completed. In this fourth stage 3720, the video displayed in viewports 1160 and 1170 has been rotated to an upright position. However, because the rotation operation was performed only on the local device (i.e., device 3500), only viewport 1160 of the locally captured video has been rotated from landscape to portrait. Viewport 1170 remains in landscape orientation.
[0352] From fourth stage 3720 to first stage 3705, a rotation operation is performed on the remote device (not shown) by rotating the remote device 90 degrees from a landscape position to an upright position. In this case, first stage 3705 shows viewport 1170 after this rotation operation is completed. Thus, UI 1105 at this stage shows viewports 1160 and 1170 in a portrait orientation. Although Figure 37 shows a sequence of different rotation operations, other embodiments may perform any number of rotation operations in any number of different sequences.
[0353] 34, 35, 36 and 37 illustrate rotation operations performed at a local device and a remote device during a video conference. When a rotation operation is performed at a local mobile device, some embodiments notify the remote device of the rotation operation so that the remote device can also perform any changes to the local device's video (such as rotating a viewport displaying the local device's video). Similarly, when a rotation operation is performed at a remote device, the remote device notifies the local device of this operation so that the local device can also perform any changes to the remote device's video. Some embodiments provide a control communication channel for communicating notifications of rotation operations between the local device and the remote device during a video conference.
[0354] Although Figures 34, 35, 36, and 37 show different ways in which the rotation animation can be performed, one skilled in the art will appreciate that other embodiments may display the rotation animation in other different ways. Additionally, the animation of the rotational movement may cause the videoconferencing manager 1604 to recompose the viewport(s) at a different angle in the UI 1105, cause changes to the image processing operations of the local mobile device, such as scaling the images displayed in the viewport(s).
[0355] 3. Adjust the window size Some embodiments allow a user of a mobile device to adjust the size of the inset viewport of a PIP display presented during a video conference. Different embodiments provide different techniques for resizing the inset viewport. Figure 38 illustrates one approach for resizing the inset viewport. In this approach, a user of a mobile device adjusts the size of the inset viewport by selecting a corner of the inset viewport and then expanding or contracting the inset viewport.
[0356] In FIG. 38, the UI 3800 of a mobile device 3825 presents a PIP display 3865 during a video conference with a remote user of another mobile device. The PIP display 3865 includes two video displays: a background main display area 3830 and a foreground inset display area 3835. The background main display area 3830 occupies the majority of the PIP display 3865, while the foreground inset display area 3835 is smaller than and overlaps the background main display area 3830. In this example, the background main display area 3830 presents a video of a person holding a guitar, who is assumed to be a person whose video is captured by a front camera of a remote device or a person whose video is captured by a rear camera of a remote device. The foreground inset display area 3835 presents a video of a person wearing a hat, who is assumed to be a person whose video is captured by a front camera of a local device or a person whose video is captured by a rear camera of a local device in this example. Below the PIP display 3865 is the display area 1155 that includes a selectable UI item 3860 (eg, button 3860) labeled "End Conference" that the user can select to end the videoconference.
[0357] This PIP display 3865 is just one way of presenting a composite view of the videos being captured by the remote and local devices. Some embodiments may provide other composite views. For example, instead of having a larger background display for the video from the remote device, the larger background display 1170 may be of the video from the local device and a smaller foreground inset display may be of the video from the remote device. Similarly, according to some embodiments, the local and remote videos may be displayed in two display areas side-by-side (e.g., left and right display windows or top and bottom display windows) or two diagonally aligned display areas in the UI 3800. In some embodiments, the manner or default display mode of the PIP display may be specified by the user. In other embodiments, the PIP display may also include a larger background display and two smaller foreground inset displays.
[0358] 38 illustrates the resize operation as four stages of operation in the UI 3800. In the first stage 3805, the foreground inset display 3835 is much smaller than the background main display area 3830. Also, in this example, the foreground inset display area 3835 is in the lower right corner of the PIP display 3865. In other examples, the foreground inset display area 3835 may be a different size or may be in a different area within the PIP display 3865.
[0359] In the second stage 3810, the resize action is initiated. In this example, the action is initiated by the user selecting the corner of the insert viewer 3835 that they wish to resize (e.g., by pressing the top left corner of the insert viewer 3835 with finger 3840). The second stage 3810 of the UI 3800 indicates this selection as a thick border 3845 of the insert viewer 3835. In this stage, the user can make the insert viewer 3835 larger or smaller (e.g., by dragging their finger 3840 on the PIP display 3865 away from or towards the insert viewer 3835).
[0360] A third stage 3815 shows the UI 3800 after the user has begun to expand the insertion viewport 3835 by moving their finger 3840 away from the insertion viewport 3835 (i.e., in this example, by moving their finger diagonally toward the upper left corner of the UI 3800), as indicated by arrow 3850. The movement of finger 3840 also causes the insertion viewport 3835 to expand proportionally in both height and width, as indicated by arrow 3855. In other examples, the user can shrink the insertion viewport 3835 using the same technique (i.e., by dragging the finger toward the insertion viewport 3835).
[0361] The fourth stage 3820 displays the UI 3800 after resizing of the insert viewport 3835 is complete. In this example, the user completes the resizing of the insert viewport 3835 by stopping dragging their finger 3840 and removing their finger from the PIP display 3865 when the insert viewport 3835 reaches the desired size. As a result of this procedure, the resized insert viewport 3835 is larger than its original size in the first stage 3805. The removal of the thick border 3845 indicates that the insert viewport resize operation is now complete.
[0362] Some embodiments provide other techniques to allow a user to resize the inset viewer 3835 in a PIP view 3865 during a video conference. One such other technique is illustrated in Figure 39. This figure illustrates a technique for resizing the inset viewer 3835 by selecting an edge of the inset viewer 3835 (i.e., one of the sides of the viewer 3835) and then expanding or contracting the inset viewer 3835.
[0363] Figure 39 illustrates this resizing operation as four stages of operation in the UI 3800 of Figure 38. The first stage 3805 of Figure 39 is the same as the first stage 3805 of Figure 38. Specifically, at this stage, the UI 3800 of the device 3925 shows a PIP display 3865 with a larger background main viewport 3830 and a smaller foreground inset viewport 3835 in the lower right corner of the PIP display 3865. Although Figures 38 and 39 illustrate two different techniques for resizing the inset viewport 3835 in the same UI 3800, one skilled in the art will understand that some embodiments do not provide both of these techniques in the same UI.
[0364] The second stage 3910 indicates the start of the resize action. In this example, the user begins the action by selecting the edge of the insert viewport 3835 that the user wishes to resize (e.g., by placing finger 3840 on the top or side edge of the insert viewport 3835). In this example, the user places their finger 3840 on the top edge of the insert viewport 3835 to make this selection. The second stage 3910 indicates this selection as a thick border 3845 of the insert viewport 3835.
[0365] A third stage 3915 shows the UI 3800 after the user has begun to expand the insert viewport 3835 by moving their finger 3840 away from the insert viewport 3835 (i.e., vertically toward the top of the PIP display 3865), as indicated by arrow 3950. The movement of finger 3840 also causes the insert viewport 3835 to expand proportionally in both height and width, as indicated by arrow 3955. In other examples, the user can shrink the insert viewport 3835 using the same technique (e.g., by dragging finger 3840 toward the insert viewport 3835).
[0366] A fourth stage 3920 displays the UI 3800 after resizing of the insert viewport 3835 is complete. In this example, the user completes the resizing of the insert viewport 3835 by stopping dragging their finger 3840 and lifting their finger 3840 off the device's display screen when the insert viewport 3835 reaches the desired size. As a result of this procedure, the resized insert viewport 3835 is larger than its original size in the first stage 3805. Removal of the thick border 3845 indicates that the insert viewport resize operation is now complete.
[0367] As illustrated by Figures 38 and 39, some embodiments adjust the size of the inset viewport 3835 proportionally in height and width in response to a drag action. In other embodiments, the user may be able to adjust the height and / or width of the inset viewport 3835 without affecting other attributes. Figure 40 shows an example of such a resizing technique.
[0368] Specifically, FIG. 40 illustrates a UI 3800 for a mobile device 4025 similar to the UI 3800 of FIG. 38, except that the UI 3800 of FIG. 40 allows the inset viewport 3835 to expand horizontally and / or vertically if one of the edges of the inset viewport 3835 is selected and moved horizontally or vertically. To simplify the description of the UI 3800, FIG. 40 illustrates a PIP display 3865 in a UI 3800 similar to the PIP display 3865 of FIG. 38, except that the inset viewport 3835 is now in the upper right corner of the PIP display 3865. The PIP display 3865 includes two video displays: a background main viewport 3830 and a foreground inset viewport 3835. In this example, the background main viewport 3830 presents video being captured by a front or rear camera of the remote device. The foreground inset viewport 3835 presents video being captured by a front or rear camera of the local device.
[0369] Like Figure 38, Figure 40 illustrates the resize operation as four operation stages in the UI 3800. The first stage 4005 is similar to the first stage 3805 of Figure 38, except that the insertion viewport 3835 is now in the upper right corner. The other three stages 4010, 4015, and 4020 are similar to the other three stages 3910, 3915, and 3920, except that the selection and movement of the bottom edge of the insertion viewport 3835 has expanded the insertion viewport 3835 only vertically, without affecting the width of the insertion viewport 3835.
[0370] 38, 39, and 40 provide example embodiments in which a user can resize the insertion viewport 3835 of a PIP view 3865 by selecting a corner or edge of the insertion viewport 3835. Some embodiments provide other techniques for resizing the insertion window 3835. For example, FIG. 41 illustrates that some embodiments allow the insertion viewport 3835 to be resized by selecting inside the insertion viewport 3835. In this approach, a user adjusts the size of the insertion viewport 3835 by placing two fingers 4155 and 4156 on the screen and dragging the fingers either away from or towards each other.
[0371] In Figure 41, a UI 3800 of a mobile device 4140 provides a PIP display 3865 during a video conference with a remote user of another mobile device. To simplify the description of the UI 3800, Figure 41 shows a PIP display 3865 in this UI 3800 that is similar to the PIP display 3865 in Figure 38.
[0372] FIG. 41 illustrates the resize operation as seven operation stages of the UI 3800. The first four stages 3805, 4110, 4115, and 4120 illustrate the expansion of the inset viewport 3835, while the last three stages illustrate the contraction of the inset viewport 3835. The first stage 3805 of FIG. 41 is the same as the first stage 3805 of FIG. 38. Specifically, in this stage, the UI 3800 illustrates a PIP display 3865 having a larger background main viewport 3830 and a smaller foreground inset viewport 3835. In this example, the background main viewport 3830 presents video captured by a front or rear camera of the remote device. The foreground inset viewport 3835 presents video captured by a front or rear camera of the local device.
[0373] Second stage 4110 shows UI 3800 after a resize action has been initiated. In this example, the user initiates the action by selecting the insertion area 3835 that the user wants to resize (e.g., by placing two fingers 4155 and 4156 within the insertion area 3835). Second stage 4110 of UI 3800 indicates this selection as a thick border 4190 of the insertion area 3835.
[0374] A third stage 4115 shows the UI 3800 after the user has begun to expand the insertion area 3835 by moving fingers 4155 and 4156 away from each other (i.e., moving finger 4155 toward the upper left corner of PIP display 3865 and moving finger 4156 toward the lower right corner of PIP display 3865) as indicated by arrow 4160. The movement of fingers 4155 and 4156 causes the insertion area 3835 to expand proportionally in both height and width, as indicated by arrow 4165.
[0375] The fourth stage 4120 shows the UI 3800 after resizing of the insertion viewport 3835 is complete. In this example, the user completes the resizing of the insertion viewport 3835 by stopping dragging fingers 4155 and 4156 and removing their fingers 4155 and 4156 from the device's display screen. As a result of this procedure, the resized insertion viewport 3835 is larger than its original size in the first stage 3805. The removal of thick border 4190 indicates that the insertion viewport resize operation is complete at this stage.
[0376] In a fifth stage 4125, the user reselects the insertion viewport 3835 by placing two fingers 4155 and 4156 on the insertion viewport 3835. A sixth stage 4130 shows the UI 3800 after the user has begun to shrink the insertion viewport 3835 by moving fingers 4155 and 4156 toward each other, as indicated by arrow 4170. The movement of fingers 4155 and 4156 causes the insertion viewport 3835 to shrink proportionally in both height and width, as indicated by arrow 4175.
[0377] The seventh stage 4135 is similar to the fourth stage 4120 of Figure 41, except that the operation results in a reduction in size of the insert viewport 3835. The removal of the thick border 4190 indicates that the insert viewport resize operation is complete at this stage.
[0378] The above description of Figures 38-41 illustrates several exemplary user interfaces that allow a user to resize the inset area of the PIP display. In some embodiments, resizing the inset area results in a change to the image processing operations of the dual camera mobile device such that the videoconferencing manager 1604 changes the scaling and compositing of the inset area in the PIP display in response to user input. Furthermore, in some embodiments, the layout of the viewport 1155 of Figures 38-41 is the same as the layout of the viewport 1155 of Figure 12 described above.
[0379] 4. Identifying Regions of Interest Some embodiments allow a user to identify regions of interest (ROIs) in a video displayed during a video conference in order to modify the image processing (e.g., image processing manager 1608 of FIG. 16), encoding (e.g., encoder 1655 of FIG. 16), behavior of the mobile device and its camera during the video conference, or a combination thereof. Different embodiments provide different techniques for identifying such regions of interest in a video. FIG. 42 illustrates a user interface of some embodiments for identifying a region of interest in a video to improve the image quality of the video.
[0380] In FIG. 42, the UI 4200 of the mobile device 4225 presents a PIP display 4265 during a video conference with a remote user of another mobile device. The PIP display of FIG. 42 is substantially similar to that of FIG. 41. Specifically, the PIP display 42 of FIG. 42 includes two video displays: a background main display 4230 and a foreground inset display 4235. In this example, the background main display area 4230 presents a video of a tree and a person wearing a hat, which is assumed to be a tree and person whose video is captured by a front camera of a remote device or a tree and person whose video is captured by a rear camera of a remote device. The foreground inset display 4235 presents a video of a man, which is assumed to be a man whose video is captured by a front camera of a local device or a man whose video is captured by a rear camera of a local device in this example. Below the PIP display is a display area 1155 that includes a selectable UI item 4260 (eg, button 4260) labeled "End Conference" that the user can select to end the videoconference.
[0381] This PIP display is just one way of presenting a composite view of the videos being captured by the remote and local devices. Some embodiments may provide other composite views. For example, instead of having a larger background display for the video from the remote device, the larger background display may be of the video from the local device and the smaller foreground inset display may be of the video from the remote device. Similarly, according to some embodiments, the local and remote videos may be displayed in two display areas side-by-side (e.g., left and right display windows or top and bottom display windows) or two diagonally aligned display areas in the UI. In other embodiments, the PIP display may also include a larger background display and two smaller foreground inset displays. In some embodiments, the manner or default display mode of the PIP display may be specified by the user.
[0382] 42 illustrates the ROI identification operation as four operational stages in UI 4200. As shown in a first stage 4205, the video presented in background display 4230 has very low quality (i.e., the video image is blurred). In this example, a user of mobile device 4225 wishes to identify an area of background display 4230 in which a person's face 4270 is displayed as a region of interest.
[0383] In a second step 4210, an operation to identify a region of interest is initiated. In this example, the operation is initiated by the user selecting an area within the video presented in background display 4230 that the user wishes to identify as a region of interest (e.g., by tapping the device screen with finger 4250 in a location around a person's face 4270 displayed in background display 4230).
[0384] As shown in the third stage 4215, upon the user's selection of the area, the UI 4200 draws an enclosure 4275 (e.g., a dashed rectangle 4275) surrounding the area of the user's selection. A fourth stage 4220 displays the UI 4200 after the identification of the region of interest is complete. As a result of this process, the quality of the video within the region of interest is substantially improved from the quality in the first stage 4205. Removal of the enclosure 4275 indicates that the ROI selection operation is now complete. In some embodiments, the ROI identification process also causes the same changes of the same video displayed on the remote device as it does to the local device 4225. In this example, for example, the image quality within the region of interest of the same video displayed on the remote device is also substantially improved.
[0385] In some embodiments, the user may stretch or shrink the enclosure 4275 in the third stage 4215 (e.g., by pressing the display with finger 4250 and moving finger 4250 toward the top right corner of the screen to stretch the enclosure 4275 or moving finger 4250 toward the bottom left corner of the screen to shrink the enclosure 4275). Some embodiments also allow the user to reposition the enclosure 4275 in the third stage 4215 (e.g., by pressing the display with finger 4250 and moving finger 4250 horizontally or vertically across the display). In some other embodiments, the selection of an area may not require the UI 4200 to draw the enclosure 4275 in the third stage 4215 at all.
[0386] Other embodiments provide different techniques for allowing a user to identify a region of interest in a video. Figure 43 illustrates one such other technique. In Figure 43, a user identifies a region of interest by drawing a shape that bounds the region. In this example, the shape is a rectangle, but it can be other shapes (e.g., any other polygon, circle, ellipse, etc.). Some embodiments provide this alternative technique of Figure 43 in a device UI that also provides the technique shown in Figure 42. Other embodiments, however, do not provide both of these techniques in the same UI.
[0387] Figure 43 illustrates this ROI identification operation as five operational stages in a UI 4200. The first stage 4205 of Figure 43 is identical to the first stage 4205 of Figure 42. Specifically, in this first stage 4205, the UI 4200 shows a PIP display 4265 with a larger background main viewport 4230 and a smaller foreground inset viewport 4235 in the lower right corner of the PIP display 4265.
[0388] In a second step 4310, an operation to identify a region of interest is initiated. In this example, the operation is initiated by selecting for a period of time a first location in the video presented in the background viewport 4230 to define the region of interest (e.g., by pressing finger 4350 for a period of time on the device's screen at a location around a person's face 4270 displayed in the background viewport 4230). In a third step 4315, the UI 4200 indicates that the first location 4370 has been selected as a dot 4355 on the background viewport 4230 next to the selected first location.
[0389] A fourth stage 4320 shows the UI 4200 after the user has selected a second location 4375 for defining the region of interest. In this example, the user selects this second location 4375 by dragging finger 4350 across the device screen from the first location after point 4355 is displayed, as indicated by arrow 4360, and stopping at a location between the displayed hat and the displayed tree in the background display area 4230. As shown in the fourth stage, this drag causes the UI 4200 to draw a rectangular boundary 4365 for the region of interest area having the first location 4370 and the second location 4375 at its opposite vertices.
[0390] The fifth stage 4325 shows the UI 4200 after the identification of the region of interest is complete. In this example, the user completes the identification of the region of interest by stopping dragging the finger 4350 and lifting the finger 4350 off the display screen of the device once the desired region of interest area is identified. The fifth stage 4325 shows that as a result of the drawing process, the quality of the video within the region of interest has been substantially improved from the quality in the first stage 4205. In some embodiments, the drawing process also causes the same changes to the display on the remote device that were caused to the local device 4225. In this example, for example, the image quality within the region of interest of the same video displayed on the remote device is substantially improved.
[0391] The above description of Figures 42 and 43 illustrates different ways of identifying regions of interest in a video to enhance the image quality of the identified regions. In some embodiments, enhancing the image quality of the identified regions of interest results in changes to the encoding operations of the dual camera mobile device, such as allocating more bits to the identified regions when encoding the video.
[0392] According to some embodiments, a user can identify an area of interest in a video in order to make various changes to the mobile device or its camera. For example, Fig. 44 shows an example of identifying an area of interest in a video in order to expand or shrink the area of interest on the display. In this approach, a user identifies an area of interest in a video by selecting an area on the display as the center of the area of interest and then expanding or shrinking the area of interest.
[0393] In Figure 44, the UI 4400 of a mobile device 4425 presents a PIP display 4265 during a videoconference with a remote user of another mobile device. The PIP display 4265 of Figure 44 is substantially similar to the PIP display 4265 of Figure 42, except that the foreground inset display 4235 of Figure 44 is in the lower left corner of the PIP display 4265.
[0394] 44 illustrates an ROI selection operation as four operational stages in UI 4400. As shown in a first stage 4405, a background display 4430 presents a video having a man to the left of display 4430 and a tree 4440 to the right. Furthermore, tree 4440 is relatively small and occupies only the right side of background display area 4430. In this example, a user of mobile device 4425 wishes to identify the area in display 4430 in which tree 4440 appears as a region of interest.
[0395] In a second stage 4410, the operation of identifying a region of interest is initiated. In this example, the operation is initiated by selecting an area 4440 in the video presented in the background display 4430 that the user wishes to identify as a region of interest (e.g., by pressing with two fingers 4445 and 4446 on the background display area 4430 where the tree 4440 is displayed). In this stage 4410, the user can expand the region of interest area 4440 by dragging the fingers 4445 and 4446 further away from each other, thereby capturing a larger portion of the background display area 4430. The user can also shrink the region of interest 4440 by dragging the fingers 4445 and 4446 closer together, thereby capturing a smaller portion of the background display area 4430.
[0396] A third stage 4415 shows the UI 4400 after the user has begun to expand the region of interest 4440 to occupy a larger portion of the background view area 4430 by moving their fingers 4445 and 4446 away from each other (i.e., moving finger 4445 toward the top left corner of the background view area 4430 and finger 4446 toward the bottom right corner of the view 4430) as indicated by arrow 4450. In some embodiments, the finger movement also causes the same changes to the remote device's display as it did to the local device. In this example, for example, the region of interest of the same video expands to occupy a larger portion of the remote device's background view area 4430. In some embodiments, the expansion of the region of interest in the local and / or remote display causes one or both of the mobile device or its camera to change one or more of their other operations, as described further below.
[0397] A fourth step 4420 displays the UI 4400 after the identification of the region of interest is complete. In this example, the user completes the identification of the region of interest by stopping dragging their fingers 4445 and 4446 and lifting their fingers 4445 and 4446 off the device's display screen when the region of interest reaches a desired proportion in the background display area 4430. As a result of this process, the region of interest occupies the majority of the background display 4430. The identification of the region of interest action is complete at this point.
[0398] Some of the above examples illustrate how a user can identify a region of interest in a video to improve image quality within the selected region of interest of the video (e.g., by increasing the bit rate for encoding the region of interest portion of the video). In some embodiments, identifying the region of interest in the video results in changes to the image processing operations of the mobile device, such as exposure, scaling, focus, etc. For example, identifying the region of interest in the video (e.g., by identifying the region of interest to zoom on) may cause the videoconference manager 1604 to scale and composite the image of the video differently.
[0399] In other embodiments, identifying a region of interest in the video results in changes to the operation of the camera(s) of the portable device (e.g., frame rate, zoom, exposure, scaling, focus, etc.). In still other embodiments, identifying a region of interest in the video results in changes to the encoding operation of the portable device, such as allocating more bits to the identified region, scaling, etc. Additionally, while the exemplary ROI identification operation described above may result in only one of the above-described changes to the portable device or its camera, in some other embodiments, the ROI identification operation may result in more than one of the changes to the operation of the portable device or its camera. Additionally, in some embodiments, the layout of the display area 1155 of FIGS. 42-44 is the same as the layout of the display area 1155 of FIG. 12 described above.
[0400] B.Camera switching Some embodiments provide a procedure for switching cameras (i.e., changing the camera from which an image is captured) during a videoconference. Different embodiments provide different procedures for performing a camera switching operation. Some embodiments provide a procedure performed by a dual camera mobile device to switch the device's camera (i.e., local switching), while other embodiments provide a procedure for the dual camera mobile device to instruct another dual camera mobile device in a videoconference to switch the other device's camera (i.e., remote switching). Still other embodiments provide procedures for both. Section IV.B.1 describes a process for performing a local camera switching operation on a dual camera mobile device. Section IV.B.2 describes a process for performing a remote camera switching operation on a dual camera mobile device.
[0401] 1.Local camera switching Figure 45 illustrates a process 4500 that some embodiments perform on a local dual camera mobile device to switch between two cameras of the device during a videoconference with a remote mobile device that includes at least one camera. In some embodiments, the process 4500 is performed by the videoconference manager 1604 shown in Figure 16. For purposes of explanation, this description will refer to one camera of the local dual camera mobile device as camera 1 and the other camera of the local dual camera mobile device as camera 2.
[0402] The process 4500 begins by initiating (at 4505) a video conference between a local dual camera mobile device and a remote mobile device. The process 4500 then transmits (at 4510) a video image from a currently selected camera (e.g., camera 1) of the local dual camera mobile device to the remote mobile device for display on the remote mobile device. At 4510, the process also generates and displays a composite display based on this video image and the video image received from the remote mobile device.
[0403] The process 4500 then determines (at 4515) whether a videoconference end request has been received. As mentioned above, in some embodiments, the videoconference can be ended at the request of a user of the local dual camera mobile device (e.g., through a user interface of the local dual camera mobile device) or at the request of a user of a remote mobile device (e.g., through a user interface of the remote mobile device). If the process 4500 receives a videoconference end request, the process 4500 ends.
[0404] If the process 4500 does not receive a videoconference end request, the process 4500 then determines (at 4520) whether the user of the local dual camera mobile device has instructed the device to switch cameras for the videoconference. If the process 4500 determines (at 4520) that the device has not been instructed to switch cameras, the process 4500 returns to operation 4510. However, if the process 4500 determines (at 4520) that the device has been instructed to switch cameras, the process 4500 moves to 4525.
[0405] The process 4500 sends a notification to the remote mobile device indicating that the local dual camera mobile device is switching cameras at 4525. In some embodiments, the process 4500 sends the notification over a video conference control channel multiplexed with the audio and video channels by the VTP manager 3125 as described above.
[0406] After sending that notification, process 4500 performs (at 4530) a camera switching operation. In some embodiments, performing the camera switching operation (at 4530) includes instructing the CIPU to stop capturing video images with camera 1 and begin capturing video images with camera 2. These instructions may simply instruct the CIPU to switch to capturing images from a pixel array associated with camera 2 and begin processing those images. Alternatively, in some embodiments, the instructions to the CIPU are accompanied by a set of initialization parameters that instruct the CIPU to (1) operate camera 2 based on a particular set of settings, (2) capture video generated by camera 2 at a particular frame rate, and / or (3) process video images from camera 2 based on a particular set of settings (e.g., resolution, etc.).
[0407] In some embodiments, the camera switching instructions (at 4530) also include instructions for switching the unused camera to the fourth operating power mode described above. In this example, the camera switching instructions include instructions for camera 2 to switch to its fourth operating power mode. Additionally, the camera switching instructions also include instructions for camera 1 to switch from its fourth operating power mode to another operating power mode, such as the power-saving first operating power mode, or to the third operating power mode, so that if an image is requested to be captured, it can quickly switch to the fourth operating power mode and begin capturing. The camera switching operation 4530 also includes combining an image captured by camera 2 of the local dual camera mobile device (rather than an image captured by camera 1) with an image received from the remote mobile device for display on the local dual camera mobile device.
[0408] After indicating a camera switch at 4530, the process 4500 performs (at 4535) a camera switch animation on the local dual camera mobile device to display a transition between displaying an image from camera 1 and displaying an image from camera 2. Following the camera switch animation on the local dual camera mobile device, the process 4500 cycles back through operations 4510-4520 until a videoconference end request or a new camera switch request is received.
[0409] 46 illustrates an example of how a camera switching operation can be requested via the UI 1105 of a dual camera device according to some embodiments, and how these embodiments animate the camera switching operation. The figure illustrates the camera switching operation as eight different operation stages 4610, 4615, 4620, 4625, 4630, 4635, 4640, and 4645 in the UI 1105 of the device. The first four stages 4610, 4615, 4620, and 4625 of the UI 1105 illustrate an example of receiving a user's request to switch cameras. In some embodiments of the invention, the user of the device has other mechanisms for making such a request.
[0410] 11, showing the UI 1105 after the video conference has been set up. At this stage, the UI 1105 displays a PIP display that includes two video displays: a larger background display from a remote camera and a smaller foreground inset display from a local camera. In this example, the background main viewport 1170 presents a video of a woman, assumed to be a woman in this example, whose video is captured by a remote device, and the foreground inset viewport 1160 presents a video of a man, assumed to be a man in this example, whose video is captured by the front camera of the local device.
[0411] Thereafter, a second stage 4615 illustrates the initiation of a camera switching operation through a selection of the PIP display area 1180 of the UI 1105. As shown, the selection is made by placing a user's finger 4670 on the PIP display 1180. A third stage 4620 illustrates the UI 1105 including a selectable UI item 4675 (e.g., a camera switching button 4675) for requesting switching between cameras of the local device 4600 during the videoconference. A fourth stage 4625 illustrates the UI 1105 after a user of the local device 4600 selects the selectable UI item 4675 (e.g., by a single finger tap), with the selection indicated through a highlighting of the selectable UI item 4675. By selecting this selectable UI item 4675, the user is instructing the device 4600 to switch from the front camera of the device 4600 to the rear camera of the device 4600 during the videoconference. In another example, where the rear camera of device 4600 is capturing video, a user's selection of selectable UI item 4675 instructs device 4600 to switch from the rear camera of device 4600 to the front camera of device 4600. After the fourth stage, the videoconference manager sends instructions to the CIPU and the remote device to initiate the camera switching operation.
[0412] The last four stages 4630, 4635, 4640 and 4645 of UI 1105 show an example of a camera switching animation on the local device. The animation is intended to provide the impression that video captured from the front and back cameras of the local device are currently displayed on two opposing sides of a viewing pane, only one of which is visible by the user at any one time. When a camera switch is requested during the course of a video conference, the viewing pane is made to appear to rotate about a vertical axis, so that the presentation of one camera's video on one side of the viewing pane that previously showed the user that camera's video rotates away from the user until it is replaced by the other side of the viewing pane, which shows the other camera's video. This animation and appearance of perceived view pane rotation is achieved by (1) gradually shrinking the video image from one camera in the viewing area and applying a perspective correction operation to it, followed by (2) gradually expanding the video image from the other camera in the viewing area and decreasing the perspective correction operation to it.
[0413] Thus, the fifth step 4630 indicates the start of a "view pane rotation" about the vertical axis 4682. To give the appearance of a view pane rotation, the UI 1105 reduced the size of the front camera video image in the video viewport 1160 and applied a perspective operation so that the right side of the video image appears farther away from the user compared to the left side of the video image.
[0414] A sixth stage 4635 indicates that the view pane has rotated 90 degrees so that the user can only see the edge of this pane, as indicated by a thin line 4686 displayed in the center of the view area 1160. A seventh stage 4640 indicates that the view pane continues to rotate so that the back side of the view pane 4688 is now gradually revealed to the user to show video captured from the user's rear camera. Again, in some embodiments, this display of the rotation animation is achieved by reducing the size of the rear camera video image in the video view area 4688 and applying a perspective operation so that the left side of the video image appears further away from the user compared to the right side of the video image.
[0415] An eighth stage 4645 illustrates the completion of the animation illustrating the camera switching action. Specifically, this stage displays in the viewport 1160 a video image of the automobile being captured by the rear camera of the device 4600.
[0416] The example above with respect to FIG. 46 initiates the camera switching action through a camera switching user interface. Other embodiments initiate the camera switching action differently. For example, some embodiments initiate the camera switching action by having a camera switching selectable UI item permanently displayed in a UI during a video conference, such as UI 1105 of FIG. 47. In FIG. 47, a camera switching button 1289 is shown in the display area 1155 along with a mute button 1285 and an end conference button 1287. In some embodiments, the layout of the display area 1155 is the same as the layout of the display area 1155 described above with respect to FIG.
[0417] FIG. 47 illustrates the camera switching operation of the UI 1105 as six stages: 4610, 4790, 4630, 4635, 4640, and 4645. The first stage 4610 of FIG. 47 is similar to the first stage 4610 of FIG. 46, except that the layout of the viewport 1155 shows a mute button 1285, an end meeting button 1287, and a camera switching button 1289, rather than a single end meeting button. The second stage 4790 illustrates the UI 1105 after a user of the local device 4600 selects the camera switching selectable UI item 1289 (e.g., via a single finger tap using finger 4670). In this example, by selecting this selectable UI item 1289, the user instructs the device 4600 to switch from the front camera of the device 4600 to the rear camera of the device 4600 during the video conference. The last four stages of FIG. 47 are similar to the last four stages of FIG. 46 except that the layout of the display area 1155 is the same as the layout described above in the first stage 4610, and therefore will not be described further so as not to obscure the description of the invention with unnecessary detail.
[0418] In some embodiments, when the remote mobile device receives images from a different camera of the local dual camera mobile device (i.e., the local dual camera mobile device switches cameras), the remote mobile device also performs a camera switching animation to display a transition between displaying images from one camera of the local dual camera mobile device and displaying images from the other camera of the local dual camera mobile device. FIG. 48 shows an example of one such camera switching animation as five operation stages 4810, 4815, 4820, 4825, and 4830 of UI 4805. This figure shows an example camera switching animation on a remote mobile device 4800. The operation stages are the same as the example animation of FIG. 46, except that the animation is performed on an image displayed in a display area 4835, which is where the image from the local dual camera mobile device is displayed on the remote mobile device 4800. Thus, the image of the man displayed in viewport 4835 is animated to appear to rotate 180 degrees about a vertical axis 4855 in the center of viewport 4850 to indicate the transition between the display of the image of the man in viewport 4835 and the display of the image of the car 4870. The implementation of the camera switch animation in some embodiments is the same as the implementation of the animations described above.
[0419] The above example shows a camera switching animation on a remote device having a particular user interface layout. Other embodiments may perform this camera switching animation on a remote device having a different user interface layout. For example, FIG. 49 shows such an example of a remote device 4800 having a different user interface layout 4805. In particular, the UI 4805 of FIG. 49 includes a mute button 1285, an end conference button 1287, and a camera switching button 1289 included in a display area 1155, which is permanently displayed on one side of a composite display 4850 during a video conference. The layout of the three buttons was described above with respect to FIG. 48. Other than the different user interface layout, the five stages 4810, 4815, 4820, 4825, and 4830 of FIG. 49 are identical to the five stages 4810, 4815, 4820, 4825, and 4830 of FIG. 48.
[0420] 2. Remote camera switching 50 shows a process 5000 for switching between two cameras of a remote dual camera device during a videoconference. The process 5000 is performed by a videoconference manager of a device that includes at least one camera. In the following description, a device where a user directs a remote camera switch is referred to as a local device, and a device that switches its two cameras is referred to as a remote device. Similarly, in the following description, a remote device is said to switch between its front camera (i.e., camera 1) and its back camera (i.e., camera 2).
[0421] The process 5000 of Fig. 50 will be described with reference to Fig. 51, Fig. 52, Fig. 53 and Fig. 54. Fig. 51 shows a UI 5105 of a local device 5100 through which a user requests that a remote device switch its two cameras during a video conference. This figure shows eight different operation stages 5110, 5115, 5120, 5125, 5130, 5135, 5140 and 5145 of this UI 5105. Fig. 54 shows a UI 5405 of a remote device 5400 receiving a camera switch request from the local device 5100. Fig. 54 shows six different operation stages 5410, 5415, 5420, 5425, 5430 and 5435 of the UI 5405.
[0422] As shown in Figure 50, process 5000 begins by initiating (at 5005) a videoconference between a local device and a remote device. Process 5000 then receives (at 5010) images from one camera of each device (e.g., from a front camera of each device) and generates a composite view for the videoconference based on these images. At 5010, process 5000 also transmits video images from the local device to the remote device.
[0423] Thereafter, process 5000 determines (at 5015) whether a videoconference end request has been received. As mentioned above, a videoconference may be ended in some embodiments at the request of a user of a local device or a remote device. If process 5000 receives a videoconference end request, process 5000 ends.
[0424] If process 5000 does not receive a videoconference end request, then process 5000 determines (at 5020) whether a user of the device on which process 5000 is running (i.e., the user of the local device) has instructed the device to request that the remote device switch its camera for the videoconference. If process 5000 determines (at 5020) that a remote camera switch has not been instructed to initiate, then process 5000 returns to operation 5010. If process 5000 determines (at 5020) that the device has been instructed to initiate a remote camera switch, then process 5000 transitions to 5025, which is further described below.
[0425] The first four stages 5110, 5115, 5120, and 5125 of the UI 5105 of FIG. 51 show an example of receiving a user's request to switch cameras on a remote device. The first stage 5110 and the second stage 5115 are the same as the first stage 4610 and the second stage 4615 of FIG. 46. The third stage 5120 is the same as the third stage 4620, except that the third stage 5120 includes a selectable UI item 5180 for requesting the remote device 5100 to switch cameras in addition to a selectable UI item 5175 for requesting the local device 5100 to switch cameras. The fourth stage 5125 shows the user of the local device 5100 selecting the UI item 5180 for requesting the remote device to switch cameras (e.g., via a single finger tap 5170 on the selectable UI item 5180). Selection is indicated by highlighting of the selectable UI item 5180. FIG. 51 shows one example of performing this operation, although other embodiments may perform the operation to request the remote device to switch cameras differently.
[0426] The example above with respect to Figure 51 initiates the remote camera switching action through a remote camera switching user interface. Other embodiments initiate the remote camera switching action differently. For example, some embodiments initiate the camera switching action by having a UI permanently display a camera switching selectable UI item during a video conference, such as UI 5105 of Figure 52. In Figure 52, a remote camera switching button 5288 is shown in the display area 1155 along with a mute button 5282, an end conference button 5284, and a local camera switching button 5286.
[0427] Figure 52 illustrates the remote camera switching operation of the UI 5105 of the device 5100 as six different stages 5110, 5290, 5130, 5135, 5140, and 5145. The first stage 5110 of Figure 52 is similar to the first stage 5110 of Figure 51, except that the layout of the viewport 1155 shows a mute button 5282, a local camera switch button 5286, a remote camera switch button 5288, and an end conference button 5284. The second stage 5290 shows the UI 1105 after a user of the local device 5100 selects the selectable remote camera switch UI item 5288 (e.g., via a single finger tap 5170). The last four stages of FIG. 52 are similar to the last four stages of FIG. 51 except that the layout of the display area 1155 is the same as the layout described above in the first stage 5110, and therefore will not be described further so as not to obscure the description of the invention with unnecessary detail.
[0428] Some embodiments provide a layout similar to that shown in Fig. 52, except that the selectable remote camera switching UI items are displayed in the PIP display 5165 rather than in the display area 1155. Fig. 53 illustrates such a layout 5105. In particular, this figure shows a PIP display with a selectable remote camera switching UI item 5180 and a display area 1155 with a mute button 5282, a local camera switching button 5286, and an end conference button 5284.
[0429] As described above, if the user requests a remote camera switch, process 5000 moves to 5025. At 5025, process 5000 sends a camera switch request to the remote device. In some embodiments, this request is sent over a videoconference control channel multiplexed with the audio and video channels by VTP manager 3125 as described above.
[0430] After the camera switch request is received, the process 5000 determines (at 5030) whether the remote device has responded to the camera switch request. In some embodiments, the remote device automatically sends an acknowledgment response (i.e., sends an acknowledgement response) to the local device over the videoconference control channel. However, in other embodiments, the user of the remote device must acknowledge the request through a user interface of the remote device.
[0431] The first two stages 5410 and 5415 of the UI 5405 of Figure 54 show an example of a remote user accepting a request to switch cameras of the remote device 5400. The first stage 5410 shows (1) a display area 5440 for displaying text informing the remote user of the request, (2) a selectable UI item 5465 (e.g., an Allow button 5465) for accepting the remote device's request to switch cameras, and (3) a selectable UI item 5470 (e.g., a Reject button 5470) for rejecting the remote device's request to switch cameras. The second stage 5415 then shows the UI 5405 after the user of the remote device has selected (e.g., via a single finger tap 5480) the UI item 5465 for accepting the request to switch cameras, as indicated by a highlighting of the selectable UI item 5465.
[0432] If process 5000 determines (at 5030) that a response has not yet been received from the remote device, process 5000 determines (at 5035) whether a videoconference end request has been received. If a videoconference end request has been received, process 5000 ends. Otherwise, the process receives (at 5040) images from the currently used cameras of the remote device and the local device, generates a synthetic view for the videoconference based on these images, transmits the video image of the local device to the remote device, and then transitions to 5030.
[0433] If process 5000 determines (at 5030) that a response has yet been received from the remote device, process 5000 determines (at 5045) whether the remote device acknowledged the camera switch request. If not, process 5000 returns to operation 5010 and continues to receive images from the camera of the other device. If not, the process receives images (at 5050) from the other camera of the remote device and then performs a camera switch animation (at 5055) on the local device to display a transition between the video of the previously utilized remote camera and the video of the currently utilized remote camera (i.e., the images received in operation 5050). After 5055, the process transitions to 5010 as described above.
[0434] The last four operation stages 5130, 5135, 5140, and 5145 shown in UI 5105 of Figure 51 show an example of such a remote camera switching animation on local device 5100. The example animation is similar to the example animation shown in stages 4815, 4820, 4825, and 4830 of Figure 48, except that Figure 51 shows in viewport 5150 an animation replacing a video of a woman captured by the front camera of the remote device with a video of a tree captured by the rear camera of the remote device. The last four stages of Figures 52 and 53 show the same animation as that of Figure 51, except that viewport 1155 of Figures 52 and 53 includes different selectable UI items than viewport 1155 of Figure 51.
[0435] In some embodiments, when the remote device switches cameras, the UI of the remote device also performs a camera switching animation to show the transition between the two cameras. The last four operation stages 5420, 5425, 5430, and 5435 shown in UI 5405 of FIG. 54 show an example of a camera switching animation displayed on the remote device 5400 when the remote device 5400 switches cameras. This animation is similar to the animation shown in stages 4630, 4635, 4640, and 4645 of FIG. 46, except that the animation in viewport 5445 replaces the video of a woman captured by the front camera of the remote device 5400 with a video of a tree captured by the rear camera of the remote device 5400.
[0436] As mentioned above, Figures 46, 47, 48, 49, 51, 52, 53, and 54 show various examples of camera switching animations performed in the user interface. In some embodiments, the camera switching animations cause changes to the image processing operations of the respective dual camera mobile devices, such as scaling, compositing, and perspective distortion, which may be performed by the videoconferencing manager 1604 and the image processing manager 1608, for example.
[0437] C. Exposure adjustment During a video conference between a dual camera mobile device and another mobile device, different embodiments provide different techniques for adjusting the exposure of an image captured by the camera of either mobile device. Some embodiments provide techniques for a user of the dual camera mobile device to adjust the exposure of an image captured by the camera of the other device, while other embodiments provide techniques for a user to adjust the exposure of an image captured by the camera of the dual camera mobile device. Some example techniques are described in detail below.
[0438] Fig. 55 illustrates a process 5500 for performing a remote exposure adjustment operation on a dual camera mobile device of some embodiments during a video conference. In the following description, the device where a user instructs a remote device to adjust its exposure level is referred to as the local device. In some embodiments, the process 5500 is performed by a video conference manager of the local device. The process 5500 is further described with respect to Figs. 56, 57 and 58, which illustrate various ways in which a user of a local device can request a remote device to perform an exposure adjustment operation.
[0439] As shown in Figure 55, process 5500 begins by initiating (at 5505) a videoconference between a local device and a remote device. Process 5500 then receives (at 5510) video from the remote device for display on a display screen of the local device. Process 5500 then determines (at 5515) whether a videoconference end request has been received. As discussed above, some embodiments can receive a videoconference end request from a user of the local device or the remote device. If process 5500 receives a videoconference end request, then process 5500 ends.
[0440] However, if the process 5500 does not receive a videoconference end request, the process 5500 then determines (at 5520) whether a request to adjust the exposure of the remote device's camera has been received. If the process 5500 determines that a request to adjust the exposure of the remote device's camera has been received, the process 5500 returns to operation 5510 to receive additional video captured from the remote device. Figures 56, 57 and 58 show three different examples that provide a way for a user to make such a request. In Figures 56, 57 and 58, the first stages 5610, 5710 and 5810 all show PIP displays 5625, 5750 and 5835 of the local devices 5600, 5700 and 5800 that display two videos, a video captured by the camera of the local device and a video captured by the camera of the remote device. In the first stages 5610, 5710 and 5810, the man in the background representations 5635, 5760 and 5845 is dark, indicating that the man is not properly exposed.
[0441] 56 shows one manner in which a user of the local device 5600 may request the remote device to perform an exposure adjustment by selecting the remote device's video (e.g., through a single tap on background display 5635). In this manner, the UI 5605 automatically associates the user's selection of the region of interest defined by box 5645 with the user's desire to instruct the remote device to perform an exposure adjustment on the region of interest, and thus instruct the local device's videoconferencing manager to contact the remote device to perform the exposure adjustment operation. The defined region of interest is used by the remote device in calculating the exposure adjustment.
[0442] Like second stage 5615 of Fig. 56, second stage 5715 of Fig. 57...
Claims
1. A computer readable storage medium for a first portable device, comprising: The storage medium stores a computer program; The computer program comprises: a set of instructions for presenting, during an audio call over a wireless communication network with a second device, a selectable user interface (UI) item at the first mobile device for switching from the audio call to a video conference; a set of instructions for receiving a selection of the selectable UI item; a set of instructions for initiating the videoconference without terminating the audio call; and a set of instructions for terminating the audio call before enabling the first device and the second device to present audio and video data exchanged over the video conference.
2. 2. The storage medium of claim 1, wherein the set of instructions for presenting the selectable UI items includes a set of instructions for continuously presenting the selectable UI items within a display area viewable on a display screen of the first portable device during the audio call.
3. The computer program comprises: a set of instructions for displaying a first video captured at the first portable device in a first display area while waiting for a second video captured at the second device to arrive at the first portable device; 2. The storage medium of claim 1, further comprising a set of instructions for reducing a size of the first display area when the second video is received at the first portable device so as to present the second video in a second display area of the first portable device.
4. 4. The storage medium of claim 3, wherein the first display area overlaps the second display area when the first display area is reduced in size.
5. 4. The storage medium of claim 3, wherein the first display area and the second display area form a picture-in-picture display such that when the size of the first display area is reduced, the first display area functions as an inset photograph overlaid on the second display area.
6. 10. The storage medium of claim 1, wherein the audio call ends before enabling the first device and the second device to exchange audio and video data over the videoconference.
7. 2. The storage medium of claim 1, wherein the audio call ends after the first device and the second device begin exchanging audio and video data over the video conference but before the first device and the second device begin presenting the exchanged audio and video data.
8. 2. The storage medium of claim 1, wherein the computer program further comprises a set of instructions for establishing a communication channel from the first mobile device to the second device to establish the audio call between the first device and the second device.
9. 2. The storage medium of claim 1, wherein the first portable device is a portable communication device for communicating with other devices in the wireless communication network.
10. The storage medium of claim 1 , wherein the first mobile device is a smartphone.
11. The storage medium of claim 1 , wherein the first device and the second device are provisioned in the wireless communication network to conduct audio calls with other devices over the wireless communication network.
12. 2. The storage medium of claim 1, wherein the first portable device is provisioned on the wireless communication network to make audio calls with other devices over the wireless communication network, but the second device is not provisioned to make audio calls with other devices over the wireless communication network.
13. 13. The storage medium of claim 12, wherein the wireless communication network is a first wireless communication network, and the second device is provisioned to conduct audio calls with other devices over a second wireless communication network different from the first wireless communication network or over a different communication network.
14. The storage medium according to claim 1 , wherein the first device and the second device are smartphones.
15. 2. The storage medium of claim 1, wherein the communications network is a private wireless communications network.
16. 2. The storage medium of claim 1, wherein the communications network is a public wireless communications network.
17. 2. The storage medium of claim 1, wherein the wireless communication network includes a circuit switch for routing audio calls and a packet switch for routing data, the audio calls being routed through the circuit switch of the wireless communication network, the audio data and the video data exchanged during the video conference being exchanged as packets through the packet switch, and the packets being transported between the first device and the second device over an Internet Protocol (IP) communication link.
18. A computer readable storage medium for a first portable device, comprising: the storage medium storing a computer program for setting up a video conference during a call between the first mobile device and a second device; The computer program comprises: a set of instructions for presenting, during a call between the first device and the second device, a first user interface (UI) layout at the first mobile device having a selectable UI item for switching from the call to the videoconference; a set of instructions for presenting a second UI layout having a first video captured by the first device in response to receiving a selection of the selectable UI item; and a set of instructions for reducing a size of the first video while overlaying the first video over a second video captured by the second device to present a transition from the second UI layout to a third UI layout in response to receiving a grant of the video conference request from the second device; The storage medium, wherein the third UI layout indicates a picture-in-picture (PIP) layout in which the first video overlaps at least a portion of the second video.
19. 20. The storage medium of claim 18, wherein the first device and the second device are mobile devices provisioned to conduct calls over at least one public wireless communication network of a wireless service carrier.
20. 20. The storage medium of claim 18, wherein the PIP display has a main display area showing the second video and an inset display area showing the first video, the inset display area being smaller than the main display area.
21. 20. The storage medium of claim 18, wherein the computer program further comprises a set of instructions for providing an animated initial presentation of the second video in response to receiving a grant of the video conference request from the second device by reducing a size of a first display area displaying the first video to reveal a second display area displaying the second video behind the first display area.
22. 20. The storage medium of claim 18, wherein the third UI layout has a composite display having a first display area showing the first video and a second display area showing the second video, the first display and the second display being the same size and positioned side-by-side.
23. 20. The storage medium of claim 18, wherein the first UI layout, the second UI layout, and the third UI layout are within a same display area of a UI of the first portable device.
24. A computer-readable storage medium for a first portable device, the storage medium storing a computer program, the computer program comprising: a set of instructions for presenting at the first device a composite display having a first video captured by the first device and a second video captured by a second device participating in a videoconference with the first device; a set of instructions for receiving input at the first device for modifying the composite display during the videoconference; and a set of instructions for modifying the composite display based on the received input.
25. the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video, the inset display being smaller than the main display and overlapping at least a portion of the main display; the set of instructions for receiving input includes: (i) a set of instructions for receiving a selection of the inset representation; and (ii) a set of instructions for receiving an input indicating a direction of movement of the inset representation; 25. The storage medium of claim 24, wherein the set of instructions for modifying includes a set of instructions for moving the inset representation in the movement direction.
26. the first portable device has a touch sensitive display screen; the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video, the inset display being smaller than the main display and overlapping at least a portion of the main display; the set of instructions for receiving the input includes: (i) a set of instructions for associating a position of the inset display with a contacted position of the touch-sensitive display screen; (ii) a set of instructions for selecting the inset display in response to the association; and (iii) a set of instructions for determining that a position of contact with the touch-sensitive display screen has moved while the contact is maintained; 25. The storage medium of claim 24, wherein the set of instructions for modifying the composite representation includes a set of instructions for moving the inset representation in the direction of the movement of the contact.
27. 27. The storage medium of claim 26, wherein the set of instructions for moving the insertion display includes a set of instructions for moving the insertion display even after contact with the screen has ended while the insertion has not reached a snap position within the composite display specified as a location within the composite display for placing one end of the insertion display.
28. 28. The storage medium of claim 27, wherein the snap location is a corner of the PIP display.
29. The set of instructions for moving the inset representation includes: a set of instructions for moving the inset display even after contact with the screen has ended if movement of the contact location exceeds a specified amount; and a set of instructions for not moving the inset representation after the contact ends if the movement of the contact location has not exceeded the specified amount.
30. the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video, the inset display being smaller than the main display and overlapping at least a portion of the main display; the set of instructions for receiving an input includes: (i) a set of instructions for receiving a selection of the inset representation; and (ii) a set of instructions for receiving an input indicating a movement of the inset representation; 25. The storage medium of claim 24, wherein the set of instructions for modifying includes instructions for moving the insertion area.
31. 25. The storage medium of claim 24, wherein the inset display moves from a first corner of the PIP display to a second corner of the PIP display.
32. the input is generated when a display screen of the first device is rotated; 25. The storage medium of claim 24, wherein the set of instructions for modifying includes a set of instructions for rotating the composite representation.
33. 33. The storage medium of claim 32, wherein the screen of the first device is rotatable independently of the body of the first device.
34. 33. The storage medium of claim 32, wherein the screen of the first device rotates when the first device is rotated.
35. 33. The storage medium of claim 32, wherein the set of instructions for rotating the composite display includes a set of instructions for presenting an animation illustrating rotation of the composite display about an axis passing through a center of the composite display.
36. 36. The storage medium of claim 35, wherein the animation includes shrinking the composite display so that the entirety of the composite display is displayed while the composite display is being rotated.
37. 37. The storage medium of claim 36, wherein at the end of the animation the composite display is expanded to match the screen orientation.
38. 33. The storage medium of claim 32, wherein the screen of the first device is rotated in a first direction and the composite display is rotated in a second direction opposite the first direction.
39. the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video; the inset display is smaller than the main display and overlaps at least a portion of the main display; 33. The storage medium of claim 32, wherein if the main view and the inset view are in portrait orientation before the rotation, the set of instructions includes instructions for changing the orientation of the inset view to landscape.
40. the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video; the inset display is smaller than the main display and overlaps at least a portion of the main display; 33. The storage medium of claim 32, wherein the computer program further comprises a set of instructions for receiving notification of a rotation of the second device and a set of instructions for changing an orientation of the main display from a first orientation to a second orientation.
41. the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video; the inset display is smaller than the main display and overlaps at least a portion of the main display; The input is generated when the first device is rotated; 25. The storage medium of claim 24, wherein the set of instructions for modifying includes a set of instructions for rotating the PIP display.
42. the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video; the inset display is smaller than the main display and overlaps at least a portion of the main display; the set of instructions for receiving input includes a set of instructions for receiving a selection of a portion of the inset representation and a movement of the portion; 25. The storage medium of claim 24, wherein the set of instructions for modifying includes a set of instructions for modifying a size of the inset display.
43. 43. The storage medium of claim 42, wherein the moving of the portion comprises moving the portion to a center of the inset indication, and the modifying comprises reducing a size of the inset indication.
44. 43. The storage medium of claim 42, wherein the moving of the portion comprises moving the portion away from a center of the inset indication, and the modifying comprises increasing a size of the inset indication.
45. 43. The storage medium of claim 42, wherein the selected portion is an edge of the inset representation.
46. 43. The storage medium of claim 42, wherein the selected portion is a corner of the inset representation.
47. the composite display is a picture-in-picture (PIP) display having an inset display showing the first video and a main display showing the second video; the inset display is smaller than the main display and overlaps at least a portion of the main display; 25. The storage medium of claim 24, wherein the set of instructions for modifying includes a set of instructions for swapping the insert display and the main display such that the insert display shows the second video and the main display shows the first video.
48. 25. The storage medium of claim 24, wherein the set of instructions for modifying includes a set of instructions for swapping a position of the first video and a position of the second video in the composite display.
49. the set of instructions for receiving the input includes a set of instructions for selecting the first video; 25. The storage medium of claim 24, wherein the set of instructions for modifying includes a set of instructions for moving the first video with respect to the composite display.
50. 1. A computer-readable storage medium storing a computer program for execution by a processor, the computer program for coordinating operation of a first mobile device in a videoconference with a second mobile device, the computer program comprising: a set of instructions for receiving video captured by a camera of the second mobile device; a set of instructions for displaying, at the first device, a picture-in-picture representation of the video captured by the second device and a video captured by the first device; a set of instructions for detecting a change in screen orientation of the first portable device; and a set of instructions for rotating the PIP display on the screen of the first device based on the change in screen orientation of the first device.
51. 1. A computer-readable storage medium storing a computer program for execution by a processor, the computer program for coordinating operation of a first mobile device in a videoconference with a second mobile device, the computer program comprising: a set of instructions for receiving a video captured by the second portable device, the video being displayed on a screen of the first portable device; a set of instructions for generating a display of the video captured by the second portable device along with another display of the video captured by the first portable device; a set of instructions for receiving a user input directing a display of one of the videos to move toward a screen orientation of the first portable device; and a set of instructions for moving the representation of the one of the videos to a different location on a screen of the first portable device based on the user instruction.
52. 1. A method for processing images for a first camera and a second camera of a mobile device using a shared pipeline, comprising: receiving a first set of images captured by the first camera of the mobile device; processing the first set of images using a first configuration of the shared pipeline; receiving a second set of images captured by the second camera of the mobile device; and processing the second set of images using a second configuration of the shared pipeline that is different from the first configuration.
53. 53. The method of claim 52, wherein the first configuration of the shared pipeline is stored in a first set of registers included in the shared pipeline.
54. 54. The method of claim 53, wherein the second configuration of the shared pipeline is stored in a second set of registers included in the shared pipeline.
55. 55. The method of claim 54, wherein the first set of registers and the second set of registers are distinct.
56. 54. The method of claim 53, further comprising the step of reading values stored in the first set of registers prior to the step of processing the first image set.
57. 55. The method of claim 54, further comprising the step of reading the values stored in the second set of registers prior to the step of processing the second set of images.
58. 1. A method for processing images for a first camera and a second camera of a mobile device using a shared pipeline, comprising: processing a first image from the first camera using the shared pipeline during a vertical blanking interval of the second camera; and processing a second image from the second camera using the shared pipeline during a vertical blanking interval of the first camera.
59. 60. The method of claim 58, wherein the vertical blanking period of the first camera and the vertical blanking period of the second camera are equal.
60. 1. A method for controlling a frame rate of a camera of a mobile device, comprising: capturing a first set of images at a first frame rate using a first vertical blanking interval of the camera of the mobile device; adjusting the first vertical blanking interval of the camera to a second vertical blanking interval for capturing images at a second frame rate; and capturing a second set of images at the second frame rate using the second vertical blanking interval of the camera.
61. transmitting the first set of images at the first frame rate to a second device during the videoconference; 61. The method of claim 60, further comprising transmitting the second set of images to the second device at the second frame rate during the videoconference.
62. 1. A computer readable storage medium for a mobile device having at least a first camera and a second camera, the storage medium storing a computer program for encoding video pictures captured by the cameras, the computer program comprising: a set of instructions for identifying a first rate controller parameter based at least on a first value quantifying a characteristic of the first camera; a set of instructions for using the first rate controller parameters to configure a set of rate controllers to adjust a bit rate for encoding video pictures captured by the first camera; a set of instructions for identifying a second rate controller parameter based at least on a second value quantifying the same characteristic of the second camera, the first value and the second value being different such that the first rate controller parameter and the second rate controller parameter are different; and and a set of instructions for using the second rate controller parameters to configure the set of rate controllers to adjust a bit rate for encoding video pictures captured by the second camera.
63. 63. The storage medium of claim 62, wherein the collection of rate controllers includes one rate controller.
64. 63. The storage medium of claim 62, wherein the set of instructions for using the first rate controller parameters to configure the rate controller collection comprises a set of instructions for configuring a first rate controller, and the set of instructions for using the second rate controller parameters to configure the rate controller collection comprises a set of instructions for configuring a second rate controller.
65. 63. The storage medium of claim 62, wherein the set of instructions for identifying the first rate controller parameter comprises a set of instructions for calculating the first rate controller parameter.
66. 63. The storage medium of claim 62, wherein the characteristic of the first camera and the second camera is camera noise, which represents an amount of picture artifacts introduced by the camera.
67. 63. The storage medium of claim 62, wherein the characteristic of the first camera and the second camera is camera resolution.
68. the first value and the second value are visual masking strengths that quantify the visibility of coding artifacts in coding of video pictures captured by the camera; 63. The storage medium of claim 62, wherein the first rate controller parameter and the second rate controller parameter are a first quantization parameter and a second quantization parameter.
69. the first camera and the second camera are for capturing video while the mobile device is being used for video conferencing; 63. The storage medium of claim 62, wherein the computer program further comprises instructions for switching from the first camera to the second camera during the videoconference.
70. The computer program comprises: a set of instructions for storing a set of state information while encoding video pictures from the first camera; and a set of instructions for using the set of state information to configure the set of rate controllers to adjust a bit rate for encoding the video pictures captured by the second camera after switching from the first camera to the second camera.
71. 71. The storage medium of claim 70, wherein the set of state information includes a target bit rate for encoding captured video pictures.
72. 71. The storage medium of claim 70, wherein the set of state information includes at least one of a buffer fullness, a maximum buffer fullness, and data regarding a size of a recently encoded video picture.
73. 1. A mobile device comprising: a first camera and a second camera for capturing video pictures; a rate controller set having (i) a first configuration based on a first rate controller parameter identified based on a first value quantifying a characteristic of the first camera, and (ii) a second configuration based on a second rate controller parameter identified based on a second value quantifying the same characteristic of the second camera, the first value and the second value being different such that the first rate controller parameter and the second rate controller parameter are different, and the rate controller set generates a quantization parameter based on the rate controller parameters; a video encoder for encoding a video picture using the quantization parameters received from the set of rate controllers.
74. 74. The mobile device of claim 73, wherein the collection of rate controllers includes one rate controller, and the first configuration and the second configuration are different configurations of the rate controller.
75. 74. The mobile device of claim 73, wherein the rate controller set includes a first rate controller and a second rate controller, and (i) the first configuration is a configuration of the first rate controller, and (ii) the second configuration is a configuration of the second rate controller.
76. the first value and the second value are visual masking strengths that quantify the visibility of coding artifacts in coding of video pictures captured by the camera; 74. The mobile device of claim 73, wherein the first rate controller parameter and the second rate controller parameter are a first quantization parameter and a second quantization parameter.
77. the first camera and the second camera are for capturing video while the mobile device is being used for video conferencing; The mobile device comprises: a module for enabling switching from one camera to another during said videoconference; a storage device for storing the set of state information during encoding of the video pictures from any of the cameras; 74. A mobile device as recited in claim 73, wherein after a switch is made from one camera to another, the set of state information is used to configure the set of rate controllers to adjust a bit rate for encoding the video pictures captured by the other camera.
78. 74. The mobile device of claim 73, wherein the video encoder comprises a quantizer that quantizes the encoding of the video picture based on the quantization parameter.
79. 74. The mobile device of claim 73, wherein the characteristic of the first camera and the second camera is camera noise, which represents an amount of picture artifacts introduced into the camera.
80. 74. The mobile device of claim 73, wherein the characteristic of the first camera and the second camera is camera resolution.
81. 1. A method for encoding video captured by a first camera and a second camera of a mobile device, comprising: defining a first process for generating first rate controller parameters based at least on a first value quantifying a characteristic of the first camera; defining a second process for generating second rate controller parameters based at least on a second value quantifying the same characteristic of the second camera, the first value and the second value being different; and defining a set of rate controllers for adjusting a bit rate for encoding video pictures captured by either the first camera or the second camera based on the generated first rate controller parameters or the generated second rate controller parameters.
82. 82. The method of claim 81, wherein the collection of rate controllers includes one rate controller, and the first configuration and the second configuration are different configurations of the rate controller.
83. 82. The method of claim 81 , wherein the set of rate controllers includes a first rate controller and a second rate controller, and (i) the first configuration is a configuration of the first rate controller, and (ii) the second configuration is a configuration of the second rate controller.
84. 1. A computer-readable storage medium for a first portable device including a first camera and a second camera, the storage medium storing a computer program for video conferencing between the first portable device and the second device, the computer program comprising: a set of instructions for selecting the first camera for capturing an image; a set of instructions for transmitting an image captured by the first camera to the second device; a set of instructions for receiving a selection of the second camera for capturing an image during the videoconference; and a set of instructions for terminating transmission of images captured by the first camera and transmitting images captured by the second camera of the first portable device to the second device during the videoconference.
85. 85. The storage medium of claim 84, wherein the computer program further comprises a set of instructions for displaying an image captured by each camera of the first portable device on a display of the first portable device whenever the image captured by that camera is transmitted to the second device.
86. 86. The storage medium of claim 85, wherein the computer program further comprises a set of instructions for animating a transition from the video image captured by the first camera to the video image captured by the second camera.
87. The computer program comprises: a set of instructions for receiving an image captured by a camera of the second device during the videoconference; 85. The storage medium of claim 84, further comprising: a set of instructions for displaying on a display of the first mobile device the received image captured by the camera of the second device.
88. The computer program comprises: a set of instructions for receiving an image captured by a camera of the second device during the videoconference; and a set of instructions for displaying the received images captured by the camera of the second device on a display of the first portable device simultaneously with images captured by the camera of the first portable device that is capturing images being transmitted during the videoconference.
89. 90. The storage medium of claim 88, wherein the computer program further comprises a set of instructions for simultaneously displaying the image captured and transmitted by the first camera while simultaneously displaying a selectable user interface item for instructing the first mobile device to switch cameras during the videoconference.
90. 85. The storage medium of claim 84, wherein the set of instructions for selecting the second camera further comprises a set of instructions for receiving input through a user interface of the first portable device to instruct the first portable device to select the second camera.
91. 85. The storage medium of claim 84, wherein the set of instructions for selecting the second camera further comprises a set of instructions for receiving a request from the second device to switch from the first camera to the second camera.
92. 92. The storage medium of claim 91, wherein the first device and the second device exchange control messages during the videoconference, and the request from the second device is one of the control messages exchanged during the videoconference.
93. 1. A computer-readable storage medium for a first portable device including a first camera and a second camera, the storage medium storing a computer program for conducting a video conference between the first portable device and a second device, the computer program comprising: a display area for displaying images captured by a camera of the first mobile device and transmitted to the second mobile device during a real-time video conference session; a selectable user interface (UI) item for switching the cameras of the first portable device during the videoconference, where when the camera switching occurs, the first portable device terminates transmission of images captured by one of the first portable device's cameras and transmits images captured by the other of the device's cameras to the second device; A storage medium having a graphical user interface (GUI) having:
94. 94. The storage medium of claim 93, wherein the display area is also for displaying images transmitted by the second device during the videoconference.
95. 94. The storage medium of claim 93, wherein the display area and the selectable UI items are displayed simultaneously during the videoconference.
96. 94. The storage medium of claim 93, wherein the computer program comprises a network manager for receiving a request from the second device to switch cameras of the first mobile device during a videoconference.
97. 97. The storage medium of claim 96, wherein the first device and the second device exchange control messages during the videoconference, and the request from the second device is one of the control messages exchanged during the videoconference.
98. 13. A method for defining a computer program for a first portable device to videoconference with a second device, the first portable device comprising a first camera and a second camera, the method comprising: (1) defining a display area for displaying an image captured by a camera of the first mobile device and (2) transmitted to the second mobile device during a real-time video conference session; and defining a selectable user interface (UI) item for switching the cameras of the first portable device during the videoconference, wherein when the camera switching occurs, the first portable device terminates transmission of images captured by one of the cameras of the first portable device and transmits images captured by the other of the device's cameras to the second device.
99. 100. The method of claim 98, wherein the display area is also for displaying images transmitted by the second device during the videoconference.
100. 100. The method of claim 98, wherein the display area and the selectable UI items are displayed simultaneously during the videoconference.
101. 100. The method of claim 98, further comprising: providing a network manager for receiving a request from the second device to switch cameras of the first mobile device during a videoconference.
102. 102. The method of claim 101, wherein the first device and the second device exchange control messages during the videoconference, and the request from the second device is one of the control messages exchanged during the videoconference.
103. A computer-readable storage medium of a first portable device, the storage medium storing a computer program for enabling a first device in a videoconference with a second portable device to remotely control the second portable device, the computer program comprising: a set of instructions for transmitting an image captured by a camera of the first device to the second device; a set of instructions for receiving an image captured by a camera of the second device; and a set of instructions for transmitting a command to the second device over a communication channel of a real-time communication session to instruct the second device to perform an action that modifies the image captured by the camera of the second device.
104. 104. The storage medium of claim 103, wherein the communication channel is a control channel.
105. 104. The storage medium of claim 103, wherein the action is an exposure action.
106. 104. The storage medium of claim 103, wherein the operation is a focus operation.
107. 104. The storage medium of claim 103, wherein the operation is a zoom operation.
108. 104. The storage medium of claim 103, wherein the computer program further comprises a set of instructions for displaying the image captured by the camera of the second device.
109. 104. The storage medium of claim 103, wherein the command is received through a graphical user interface of the first device.
110. 110. The storage medium of claim 109, wherein the command is transmitted by touching a display screen of the first device at a location that displays the image captured by a camera of the second device.
111. 1. A computer-readable storage medium of a first portable device, the storage medium storing a computer program for enabling a first device in a videoconference with a second portable device to remotely control the second device, the computer program having a graphical user interface (GUI), the GUI comprising: a first display area for displaying an image captured by a camera of the first device during a real-time communication session; a second display area for displaying an image captured by a camera of the second device during the real-time communication session; and a user interface (UI) item for sending a command to the second device during the real-time communication session instructing the second device to perform an action that modifies the image captured by the camera of the second device.
112. 112. The storage medium of claim 111, wherein the action is an exposure action.
113. 112. The storage medium of claim 111, wherein the action is a focus action.
114. 112. The storage medium of claim 111, wherein the operation is a zoom operation.
115. 112. The storage medium of claim 111, wherein the command is transmitted over a communication channel of the real-time communication session.
116. 116. The storage medium of claim 115, wherein the communication channel is a control channel of the real-time communication session.
117. 112. The storage medium of claim 111, wherein the first display area is an inset within the second display area.
118. 1. A method for defining a computer program for enabling a first portable device to remotely control a second portable device during a videoconference, comprising: defining a display for displaying the video captured from the second portable device on the first portable device; defining a camera adjustment action for receiving input at the first portable device for instructing the second portable device to perform an action during the videoconference that modifies the image captured by the camera of the second device; The method according to claim 1, further comprising:
119. 119. The method of claim 118, further comprising: defining a first network manager for the first portable device that relays the received input to a second network manager of the second portable device during the videoconference.
120. 120. The method of claim 119, wherein the first network manager relays data to the second network manager during the videoconference via packets transmitted over Internet Protocol (IP), the packets including control packets for relaying input data received at the first portable device to control the operation of the second portable device.
121. 119. The method of claim 118, further comprising defining a user interface for receiving the input by detecting when a location on the display is touched during the videoconference.
122. 119. The method of claim 118, wherein the action is an exposure action.
123. A computer-readable storage medium of a first portable device, the storage medium storing a computer program for managing a videoconference between the first portable device and a second portable device, the computer program comprising: a set of instructions for identifying a first allowable bit rate for transmitting videoconference data over a communication channel to the second device; a set of instructions for identifying a current bitrate that is less than the first allowed bitrate; a set of instructions for receiving network data from the second device regarding the communication channel; a set of instructions for determining from the received network data that the communication channel will sustain an increase in the current bit rate; an instruction set for increasing the current bit rate; a set of instructions for repeatedly performing the receiving, determining, and increasing operations until a determination is made that the communication channel will not support an increase in the current bit rate; A storage medium comprising:
124. 124. The storage medium of claim 123, wherein the second device is a mobile device that communicates with the first mobile device over a wireless communication network having a limited bandwidth.
125. 124. The storage medium of claim 123, wherein the set of instructions for repeatedly performing the receiving, determining and increasing operations is repeatedly executed at a particular time rate that enables accurate network data to be relayed from the second device based on a previous increment of the current bit rate.
126. The computer program comprises: a set of instructions for receiving network data from the second device regarding the communication channel, even after the repeated execution of the receiving, determining, and increasing operations has been completed, and identifying from the received network data a change in available resources of the communication channel; a set of instructions for increasing or decreasing the current bit rate based on the identified change in the available resources of the communication channel; and 124. The storage medium of claim 123, further comprising:
127. 124. The storage medium of claim 123, wherein the set of instructions for determining includes a set of instructions for determining when the current bitrate has reached the first allowed bitrate.
128. the received data includes available bandwidth measurement data provided by the second device; 124. The storage medium of claim 123, wherein the set of instructions for determining includes a set of instructions for determining that the current bit rate has reached a second allowed bit rate that is lower than the first allowed bit rate, the second allowed bit rate being derived from the available bandwidth measurement data provided by the second device.
129. 124. The storage medium of claim 123, wherein the received data includes a one-way signal delay calculated by the second device.
130. 130. The storage medium of claim 129, wherein the one-way signal delay is calculated by the second device by examining the delay of audio packets transmitted from the first portable device to the second device.
131. 131. The storage medium of claim 130, wherein the audio packet delay is checked by checking differences in time codes within the audio packets against differences in receipt of the audio packets by the second device.
132. 124. The storage medium of claim 123, wherein the received data includes available bandwidth measurement data calculated by the second device by examining the size of video packets sent to the second device against the difference in time at which the second device received the packets.
133. 124. The storage medium of claim 123, wherein the received network data includes packet loss data.
134. 124. The storage medium of claim 123, wherein the received network data includes round trip delay data.
135. 124. The storage medium of claim 123, wherein the set of instructions for receiving network data includes a set of instructions for receiving network data transmitted in a Real-time Transport Protocol (RTP) stream and identified in the RTP stream by an RTP extension header.
136. 1. A method performed by a first portable device for managing a videoconference between a first portable device and a second device, comprising: identifying a first allowable bit rate for transmitting videoconference data over a communication channel to the second device; identifying a current bitrate that is less than the first allowed bitrate; receiving network data from the second device regarding the communication channel; determining from the received network data that the communication channel will support an increase in the current bit rate; increasing the current bit rate; and repeatedly performing said receiving, determining, and increasing operations until a determination is made that said communications channel will not support an increase in said current bit rate.
137. 137. The method of claim 136, wherein the second device is a mobile device that communicates with the first mobile device over a wireless communication network having a limited bandwidth.
138. 137. The method of claim 136, wherein the steps of repeatedly performing the receiving, determining, and increasing operations are performed at a particular time rate that enables accurate network data to be relayed from the second device based on a previous increment of the current bit rate.
139. receiving network data from the second device regarding the communication channel even after the repeated execution of the receiving, determining, and increasing operations has been completed, and identifying from the received network data a change in available resources of the communication channel; 137. The method of claim 136, further comprising: increasing or decreasing the current bit rate based on the identified change in the available resources of the communication channel.
140. 137. The method of claim 136, wherein said determining step comprises determining that the current bit rate has reached the first allowed bit rate.
141. the received data includes available bandwidth measurement data provided by the second device; 137. The method of claim 136, wherein the determining step includes determining that the current bit rate has reached a second allowed bit rate that is lower than the first allowed bit rate, the second allowed bit rate being derived from the available bandwidth measurement data provided by the second device.
142. 137. The method of claim 136, wherein the received data includes a one-way signal delay calculated by the second device.
143. 143. The method of claim 142, wherein the one-way signaling delay is calculated by the second device by examining the delay of audio packets transmitted from the first mobile device to the second device.
144. 137. The method of claim 136, wherein the received data includes available bandwidth measurement data calculated by the second device by examining the size of video packets sent to the second device against the difference in time at which the second device received the packets.
145. 137. The method of claim 136, wherein receiving the network data includes receiving network data transmitted in a Real-time Transport Protocol (RTP) stream and identified in the RTP stream by an RTP extension header.
Citation Information
Patent Citations
Information output unit and information outputting method
JP2003299051A
Video conference system, conference terminal, and video server
JP2006254166A
