Methods and systems for dynamic display of video communication data

Abstract

This disclosure relates to the dynamic display of video communication data. The invention discloses a computing device capable of dynamically displaying video communication data. The video communication data can be received at the computing device where another application is currently displaying image data on an electronic display. The display position for the video communication data can be determined based on display attributes configured by the other application at runtime. Once determined, the video communication data can be displayed at the determined position. In some embodiments, the video communication data can be integrated with other data displayed on the electronic display for the other application.

Description

[0001] This application is a divisional application of the invention patent application filed on May 28, 2015, with application number 201580026580.4 and invention title "Dynamic Display of Video Communication Data". Technical Field

[0002] This disclosure relates to the dynamic display of video communication data. Background Technology

[0003] The advent of mobile computing, combined with enhanced networking capabilities, has transformed the way individuals communicate. Previously, dedicated networks had to be configured with specific devices to provide different types of communication. For example, video conferencing relied on dedicated telephone networks, radio frequencies, or closed-circuit television systems to provide two-way or multi-way video communication between users in different (and often fixed) locations. Because broadband or other high-capacity networking technologies have allowed for greater and more flexible access to multiple different systems or devices, such as via the Internet, different types of communication have evolved to take advantage of these enhanced networking capabilities. For example, instant messaging technology provides real-time transmission of text messages over the Internet, allowing participants to chat between two Internet-enabled devices.

[0004] Video conferencing has evolved to facilitate video communication between network-connected devices. Video communication data can be transmitted to other participants via networks such as the Internet, providing a video conferencing-like user experience without the limitations of past and more static communication technologies. However, because many devices, such as laptops, mobile phones, tablets, or other mobile computing devices, can now easily run multiple programs or functions simultaneously, video communication may be interrupted, neglected, or contested by the computing resources of the device performing the video communication, thus diminishing the scope or effectiveness of new video communication technologies. Summary of the Invention

[0005] This document describes several implementation schemes for dynamically displayed video communication data. Video communication between two or more parties can be facilitated based on the transmission of video communication data between them. However, other applications may currently be running at one or more of these parties, which may be controlling or displaying image or video data on an electronic display. The video communication data received as part of the video communication can then be displayed within a portion of the electronic display that can be configured by the currently running application. The currently running application can configure one or more display attributes at runtime that indicate the display location of the video communication data. In some implementations, the video communication data may be mixed with other image or video data from the currently running application to generate a single video stream for display. Various inputs, such as pausing, resuming, or ending the video communication, can be received via touch input at the display location of the video communication data. Attached Figure Description

[0006] Figure 1A and Figure 1B Some implementation schemes demonstrate peer-to-peer video communication between different participants.

[0007] Figure 2A This is a block diagram illustrating the dynamic display of video communication data at the video communication participants, based on some implementation schemes.

[0008] Figure 2B According to some implementation schemes, a participant device for displaying video communication data in an integrated display mode is shown.

[0009] Figure 2C According to some implementation schemes, a participant device for displaying video communication data in a full-screen display mode is shown.

[0010] Figure 3 This is a block diagram of a video communication module based on some implementation schemes.

[0011] Figure 4 This is a high-level flowchart illustrating methods and techniques for dynamically displaying video communication data, based on some implementation schemes.

[0012] Figures 5 to 6C According to some implementation schemes, video communication data is dynamically displayed on the electronic displays of the participating devices.

[0013] Figure 7 This is a high-level flowchart illustrating methods and techniques for dynamically displaying video communication data for multiple video communication participants, based on some implementation schemes.

[0014] Figure 8According to some implementation schemes, video communication data is dynamically displayed on the electronic display of the participating device for multiple video communication participants.

[0015] Figure 9 This is a high-level flowchart illustrating methods and techniques for processing inputs used in dynamically displayed video communication data, based on some implementation schemes.

[0016] Figures 10A to 10C Examples of how to process inputs for dynamically displayed video communication data are shown according to some implementation schemes.

[0017] Figure 11 This is a block diagram illustrating an exemplary transmit / receive manager module for a video communication module that dynamically pauses and resumes the transmission of video communication data, according to some implementation schemes.

[0018] Figure 12 This is a high-level flowchart illustrating methods and techniques for dynamically detecting pause or resume events in video communication, based on some implementation schemes.

[0019] Figure 13 This is a high-level flowchart illustrating methods and techniques for determining active exchange indicators in video communications, based on some implementation schemes.

[0020] Figure 14 It is a high-level flowchart illustrating methods and techniques for the dynamic transition from video messaging to video communication, based on some implementation schemes.

[0021] Figure 15 This is a high-level flowchart illustrating methods and techniques for monitoring the quality of video communication connections during dynamic transitions between video communication and video messages, based on some implementation schemes.

[0022] Figure 16A and Figure 16B Some implementation schemes illustrate the dynamic transitions of video communication in multiple participating directions within a video message.

[0023] Figure 17 This is a block diagram illustrating the connection manager of a video communication module that implements dynamic transitions between video communication and video messages, based on some implementation schemes.

[0024] Figure 18 This is a high-level flowchart illustrating methods and technologies for real-time video communication based on some implementation schemes.

[0025] Figure 19A and Figure 19B This is a high-level flowchart illustrating methods and techniques for obtaining authorization to establish a real-time video communication connection, based on some implementation schemes.

[0026] Figure 20 This is a high-level flowchart illustrating methods and techniques for displaying video communication data via a real-time video communication connection, based on some implementation schemes.

[0027] Figure 21 This is a block diagram illustrating the connection manager of a video communication module that implements real-time video communication, based on some implementation schemes.

[0028] Figures 22 to 24 This is a block diagram illustrating an exemplary portable multi-functional computing device according to some implementation schemes.

[0029] Figure 25 An exemplary computing system according to some implementation schemes is shown.

[0030] While the invention has been described herein with the aid of examples of several embodiments and exemplary drawings, those skilled in the art will recognize that the invention is not limited to the described embodiments or drawings. It should be understood that the drawings and their detailed description are not intended to limit the invention to the specific forms disclosed, but rather, the invention is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. The headings used herein are for organizational purposes only and are not intended to limit the scope of the specification. As used throughout the patent application, the word “may” is used in a permissive sense (i.e., meaning possible) rather than a mandatory sense (i.e., meaning must). Similarly, the words “comprising,” “including,” and “having” mean including, but not limited to, these. Detailed Implementation

[0031] This document describes several implementation schemes for dynamic display of video communication. Video communication can occur between different participating devices. These participating devices can capture video and audio data (typically from a user) at their respective participating devices and transmit the captured video communication data to other participants in the video communication for display. While the video communication data can be displayed under the control of a video communication application, such as in full-screen mode, other applications may be running simultaneously on participating devices. These applications can control or use some or all of the display of their image data or other graphics. Dynamic display of video communication allows currently running applications to control, instruct, or otherwise configure where the video communication data received in the video communication is displayed on participating devices.

[0032] This article describes several implementation schemes for the dynamic suspension and resumption of video communication. As mentioned above, other applications or tasks may be running on the devices of the participants in a video communication. Therefore, the level of information participation or active exchange via video communication may sometimes fluctuate. Evaluation of the communication data captured for video communication can indicate whether some of the video communication data needs to be transmitted or whether there is any video communication data that needs to be transmitted. Dynamic suspension and resumption of video communication can provide automated or intelligent detection of suspension events, where active transmission of some or all video communication data can be suspended without prohibiting information exchange between active participants via video communication. Similarly, automated or intelligent detection of resumption events can allow the resumption of suspended video communication data transmission. This can thus dynamically save computing resources at the participating devices.

[0033] This article describes several implementation schemes for the dynamic transition from video messaging to video communication. Video messages are used to send video to a receiving device that can be played back at the recipient's convenience. For users, exchanging information in this way can be less efficient than multi-party communication, such as video communication. A dynamic transition from video messaging can be implemented so that if a user on the receiving device receives a video message and wishes to respond or engage in further communication, video communication can be established instead of responding with another video message. In this way, users have greater flexibility in responding to video messages: they can establish video communication or save the video message for later playback and / or response.

[0034] This article describes several implementation schemes for instant video communication connections. For communication between certain users, video communication may be advantageous compared to other communication technologies such as voice or text. Family members, friends, colleagues, or other communication relationships can utilize fast and efficient video communication. Instant video communication allows users to authorize pre-established video communication connections between the respective participating devices, making the video communication appear to begin instantly. For example, video communication can be displayed as soon as it is received, thus minimizing the time required before video communication can commence. Users can manage or authorize which other users (or their respective participating devices) can establish instant video communication with their own respective participating devices.

[0035] Video communication can occur between two or more participating devices over a wide area network (WAN). Participating devices may implement some type of application, module, or component that facilitates video communication (from establishing a connection to capturing, transmitting, and processing received video communication data). Different types of participating devices and different numbers of participants can establish or participate in video communication, such as two-way or multi-way video communication. In various implementations, video communication can be peer-coordinated or managed. For example, participating devices conducting video communication may utilize peer-to-peer communication policies or protocols to transmit and receive communication data. It should be noted that peer-to-peer managed video communication may not exclude the use of third-party systems or services to facilitate video communication, such as cellular or other types of wireless network services that provide participating devices with WAN access, authentication or identification services, or simply relaying or retransmitting messages or data to ensure video communication reaches all participants. In at least some implementations, peer-to-peer managed video communication may not include a central video communication coordinator, server, system, or service, but instead may allow participating devices to act as peers to manage or coordinate the forwarding of video communication data between participating devices.

[0036] Figure 1A and Figure 1B Video communication for peer-to-peer management between different participants is illustrated according to some implementation schemes. Participant devices (e.g., participant devices 102, 112, 122, 132, 142, 152, and 162) may be implemented on different devices or different device types. Examples of participant devices may include... Figures 22 to 24 Portable multifunction devices such as the Multifunction Device 3000, other types of mobile computing devices such as telephones, computers, laptops, personal digital assistants (PDAs), or any computing systems such as those described in the reference below. Figure 25 The computing system 4000 is described above. Corresponding video communication modules (e.g., video communication modules 104, 114, 124, 134, 144, 154, and 164) can be implemented at each participating device to facilitate video communication over the wide area network 100. The video communication modules can be implemented in a variety of different ways, as detailed below. Figure 3 An example of some implementation schemes for a video communication module is provided.

[0037] Different video communications can be established via video communication connections over a wide area network (WAN) 100 (e.g., the Internet). For example, in Figure 1A In this process, video communication between the two parties can be established between participant device 102 and participant device 104. And... Figure 1BIn this context, multi-party video communication can be established among participating devices 124, 134, 144, 154, and 164. Participating devices can be configured to access WAN 100 in different ways. For example, participating device 102 can be configured to access a specific wireless network, such as a cellular network, that can communicate with WAN 100 via a gateway. The gateway can provide packet-oriented mobile data services (such as General Packet Radio Service (GPRS)) or other mobile data services that allow the wireless network to transmit data to other networks (such as WAN 100). Similarly, different access devices or access points (such as IEEE 802.11g wireless access routers or devices) can provide communication access to WAN 100. Participating device 112 can, for example, be configured (or authorized) to use an access device to obtain access to WAN 100. Therefore, in various embodiments, participating devices can access WAN 100 in different ways and still establish video communication. Therefore, the foregoing examples are not intended to limit other methods that can provide participating devices with access to WAN 100.

[0038] Video communication data can be exchanged using various transmission formats or protocols suitable for or capable of being transmitted over a wide area network (WAN), including but not limited to SIP (Session Initiation Protocol), STUN (Session Traversal Utility for NAT), TURN (NAT Traversal via Relay), ICE (Interactive Connection Establishment), RTP (Real-Time Transport Protocol), and SRTP (Secure RTP). The video communication module can be configured to format, process, transmit, send, or receive video communication data (e.g., audio and / or video data) according to various transmission protocols, and to encrypt, compress, encode, decrypt, decompress, and decode video communication data.

[0039] Figure 2A This is a block diagram illustrating the dynamic display of video communication data at participating devices in a video communication process, based on some implementation schemes. Participating device 200 can be participating device 102, 112, 122, 132, 142, 152, or 162, as described below. Figures 22 to 24 The portable multi-functional device 3000, or as described below, Figure 25 Any one of the aforementioned computing systems 4000.

[0040] Participant device 200 may implement multi-layered functionality for performing various tasks or applications, such as video communication. Application layer 210 may allow the implementation of applications (such as video communication application 214 (similar to the video communication module described above with reference to Figure 1)) and several other applications 212. Applications currently running on participant device 200 and implemented in application layer 210 may access the corresponding data, features, services, controls, or other functions of participant device 200 via one or more different objects or programming interfaces (APIs) implemented in middleware / operating system layer 220 for the data, features, services, controls, or other functions of participant device 200. Participant device 200 may implement various middleware layers 220 and / or operating system layers 220 to provide services. The operating system, for example, may provide access to and / or management of hardware devices (such as display device 232 or other devices, components, or subsystems or circuits) implemented in hardware layer 230. Participant device 200 may include a hardware layer 230, which includes various hardware components such as an electronic display 232, other input / output devices (e.g., microphone, keyboard, touch-sensitive area (which may also be an electronic display 232), speaker) or other hardware components implementing participant device 200, processor, graphics accelerator, or other components (such as those described below with reference to portable multifunction device 3000 or computing system 4000).

[0041] like Figure 2A As shown, video communication application 214 can be implemented in application layer 210. Video communication application 214 (such as that described above with reference to Figure 1 or referred to below) Figure 3 The video communication modules described in more detail can perform video communication with other participants over a network, as referenced above. Figure 1A and Figure 1B The video communication data 202 may be received at the video communication application 214 as part of a current (or initiated) video communication with one or more other participating devices. The video communication application 214 may be configured to process the video communication data 202 (e.g., decrypt, decompress, decode) or otherwise prepare the video communication data for display. Then, as part of performing the current video communication, the video communication application 214 may make one or more calls or requests to the audio / video management module 222 to ultimately display the video communication data on the display 232. In various embodiments, the video communication data may include video communication data from multiple different participants who are involved as part of the same video communication (e.g., multi-party video communication).

[0042] In various implementations, video communication 214 can operate in different display modes. For example, a full-screen display mode allows video communication application 214 to control most or all of display 232 to implement video communication, including additional video / image data of user interface elements or graphics (e.g., windows, toolbars, or buttons) for formatting or frame processing of the displayed video communication data. In at least some implementations, an integrated display mode can be used while other applications 212 or components (e.g., operating system interfaces) are displaying image data, video data, or any other form of graphical display on display 232. Audio / video management component 222 can provide software hooks, display attributes, or other technologies to indicate the display of video communication data in full-screen mode or integrated display mode. For example, if video communication application 214 is operating in full-screen mode, the audio / video management component can determine the display position within display 232 for video communication data 202 specified or requested by video communication application 214, which may consume most of display 232. For example... Figure 2C According to some implementation schemes, a participant device is shown for displaying video communication data in full-screen mode for video communication.

[0043] In some implementations, the video application module 214 may operate in an integrated display mode. For example, another application 212 may be currently running. As part of the operation of the other application 212, application image data 204 (or video or other graphical data) may be acquired, received, or generated and displayed on the display 232, as indicated by the arrow from the other application 212 through the audio / video management module 222 to the display 232. Video communication data 202 initiated before or during the operation of the other application 212 may also still be received. The audio / video management module 222 may be configured to identify the display location within the display 232 for the video communication data 202. This identification may be dynamic, as the display attributes for the video communication data may be configured by the other application 212 at runtime. For example, the other application 212 may define an event detection mechanism or software hook to configure display attributes upon receiving video communication data for display. Thus, video communication data 202 can be displayed in a location optimized to allow application image data 204 to be displayed as desired, even if current video communication may be in progress, as indicated by the arrow within the audio / video management module 222. In at least some embodiments, video communication data can be directly mixed or integrated with application image data 204 or other video or graphics data encoded as a single data stream for display, as referenced below. Figure 5 The audio / video management module 222 can coordinate or manage software or hardware resources (such as video encoders) that can be used to apply image data 204 and video communication data 202.

[0044] Figure 2B According to some embodiments, a participant device is shown displaying video communication data in an integrated display mode for video communication. The video communication data 202 is integrated with application image data 204 according to display attributes configured by another application 212. The video communication data 202 is placed within the participant device's display 232 in a location that does not obstruct the display of the application image data 204 (e.g., a blank area or dead zone, or within image data that can be easily adjusted to work around the video communication data 202). It should be noted that while the above exemplary integrated display mode is given with reference to the application in application layer 210, it is equally possible that components of middleware / operating system layer 220 (e.g., operating system user interface) can also be configured to display the video communication data 202 in a manner suitable for displaying images or other graphical data for components of middleware / operating system layer 220.

[0045] In various implementations, the audio / video management module 222 may be configured to determine the display mode of the current video communication, whether it is full-screen mode or integrated display mode. However, in some implementations, the video module application 214 may determine or maintain the state of the display mode and provide instructions to the audio / video management component 222 or format requests to the audio / video management component 222, causing the audio / video management module 222 to process the video communication data 202 according to the current display mode. In some implementations, input may be received to switch between full-screen display mode and integrated display mode.

[0046] The video application 214 can also be configured to provide user interface elements for display to the audio / video management module 222. These user interface elements can also be displayed based on a determined location (e.g., within a determined location specified by the video application module 214) or various other locations on the display 232 (e.g., indicated by other display properties that may be configured by the other application 212 at runtime). The video communication application 214 can process input received via these user interface elements (e.g., pause, resume, end).

[0047] Video communication applications can be implemented in a variety of different ways to facilitate video communication over a network. Figure 3 This is a block diagram of a video communication module according to some implementation schemes, which can implement video communication. Note that... Figure 3 The various components shown can be implemented in hardware, software, or any combination thereof. Where a single component is shown, it is understood that multiple components may alternatively be implemented to provide the same functionality. Therefore, the following regarding... Figure 3The discussion is not intended to limit the specific implementation of the video communication module. The video communication module 300 can be implemented on a variety of different types of participant devices, such as mobile computing devices, mobile phones, laptops, or other portable multi-functional devices (such as...). Figure 22 A portion of the portable multifunction device 3000 implements a video communication module 3039 or a computing device (such as those referred to below). Figure 25 The aforementioned computing system 4000).

[0048] In various implementations, the video communication module 300 may implement a user interface 310 to receive, process, parse, or manipulate input from a user for the video communication module 300. For example, in some implementations, the user interface 310 may provide multiple user interface elements selectable via I / O devices. For example, if a participating device implements a touch-sensitive display, the user interface elements may be selected by touch, gesture, or otherwise conveying selection of a particular user interface element. In some implementations with a touch-sensitive display, the user interface elements do not need to be displayed; instead, touch gestures themselves may be interpreted or parsed as indicating or conveying selection of certain actions via the user interface. The user interface 310 may implement multiple different elements for the video communication module 300, including but not limited to initiating video communication (two-way or multi-way mode with other participating devices), ending video communication, pausing video communication, resuming video communication, switching between display modes, formatting the display of video communication data, selecting an input device (e.g., a camera), muting audio, recording audio / video / still images of video communication data, or any other component that provides input or instructions from the user to the video communication module 300. In some implementations, other input devices such as computer mice, keyboards, voice commands or recognition, and physical movement of participant devices (rotating, tilting, or flipping devices) can also be inputs, which are transformed or detected via user interface component 310.

[0049] In various implementations, the video communication module 300 may implement a connection manager 320. The connection manager 320 may be configured to establish a connection for video communication with the video communication participant 304. For example, the connection manager 320 may access receiver contact data 250 for identification information to establish a connection with one or more participant devices (as requested by a user). The connection manager 320 may implement various different protocols or formats to establish video communication connections, such as those referenced above. Figure 1A and Figure 1B Those described above. In some implementations, the connection manager 320 may be configured to maintain on-demand or pre-established connections, making some video communications appear to be initiated and / or displayed in real time (see reference below). Figures 18 to 21(as described above). In various embodiments, the connection manager 320 may also be configured to handle the dynamic conversion of video messages received from participating devices into real-time video communications, and the conversion back to offline video message formats (see reference below). Figures 14 to 17 The above).

[0050] In at least some embodiments, the video communication module 300 may have access to one or more persistent storage devices or data stores that hold receiver contact data 350 and stored video communication data 360. In some embodiments, receiver contact data 350 may be identity, address, or other network / transmission information used to establish a video communication connection with a specific video communication participant and to transmit / receive video communication data via that connection. In some embodiments, the stored video communication data 360 may be a stored version of a currently ongoing or previous video communication (and other associated data, such as still images or audio recordings). The stored video communication data 360 may be accessed by the video communication module 300 of a participant's device or by another application or component for playback or display. In some embodiments, the stored video communication data 360 may not be stored locally relative to the participant's device implementing the video communication module 300, but rather may be stored in a remote storage location (e.g., a media server) accessible to the video communication module 300.

[0051] like Figure 3 As shown, the video communication module 300 can receive video communication data (video and / or audio data) from the video communication capture device 302. For example, an on-board camera and / or an on-board microphone (e.g., Figure 22 The optical sensor 3064 and / or audio circuitry 3030, along with an externally coupled camera and / or microphone, can be used to capture video and / or audio data for transmission to the video communication participant 304. In some embodiments, the video communication module 300 may implement a transmit / receive manager 300 for processing outgoing and incoming video communication data for video communication. For example, the transmit / receive manager may encode, encrypt, compress, and / or otherwise format the captured video communication data for transmission to the video communication participant 304, and instruct the transmission of the video communication data to the video communication participant 304 according to various communication protocols, such as those referenced above. Figure 1A and Figure 1BAs described above, for incoming video communication data, the transmit / receive manager 330 can be configured to decode, decrypt, decompress, or otherwise reformat the data for display. In some embodiments, the display manager 340 can instruct the received video communication data to be displayed according to a full-screen display mode and / or an integrated display mode. The display manager 340 can also provide the video communication data to an audio / video management component (e.g., the audio / video management module 222 in FIG. 2) or another application for further processing prior to display.

[0052] In various implementation schemes, the communication module 300 may be implemented as described below. Figures 4 to 21 Each of the methods, techniques, and / or components described herein. However, some implementations may implement only certain specific ones, such as implementing only... Figure 4 , Figure 12 , Figure 14 or Figure 17 The technology. Therefore. Figure 3 It is not limited to any particular combination or arrangement of the components or specific implementation of the technology described below.

[0053] As previously mentioned, the dynamic display of video communication data can be implemented as part of various components or devices of the participating equipment. An application running on the participating equipment itself can perform techniques to incorporate video communication data into graphics displayed on an electronic screen for that application. Similarly, the video application module itself can be configured to implement these techniques, dynamically determining the display position of the video communication data even if another application is controlling the electronic display. Alternatively, as described above... Figure 2A As shown, the middleware / operating system component can implement the following technologies. Figure 4 Therefore, a high-level flowchart is shown, based on some implementation schemes, of methods and techniques for dynamically integrating video communication data, which can be implemented by various system components or devices.

[0054] As shown in 410, video communication data can be received for display as part of the current video communication. The video communication data may include video and / or audio data to be displayed or presented. In various embodiments, if the video communication application is operating in full-screen mode, the video communication data may be displayed on the electronic display according to the full-screen mode, as shown in 420. In some embodiments, the full-screen display mode may be controlled by the video communication application. However, in some embodiments, other applications may also trigger or initiate the full-screen display mode. In some embodiments, video communication may be displayed in full-screen mode by default, unless otherwise specified by another application, operating system / middleware component, or the video communication application. If no other application is controlling the electronic display, as shown in the negative branch of 430, the video communication data may continue to be displayed in full-screen mode (or as otherwise specified by the video communication application).

[0055] In various implementations, if another application is launched, initiated, resumed, woken up, or otherwise started such that the other application is controlling an electronic display to display image data, as shown in the affirmative branch of 430, the display location of the video communication data can be dynamically determined based on display attributes configured at runtime as part of the execution of the other application, as shown in 440. The launched other application can be any application that uses or controls the electronic display, from games and other media to other communication applications (including other video communication applications or screen sharing applications), and various tools, tasks, or other processes that the application can be configured to perform. Software hooks, event listeners, middleware, or API calls provided by the operating system, or other devices, can allow the other application to define where, how, and / or when the video communication data is displayed, thereby defining and redefining display attributes that the application deems appropriate. In several implementations, dynamically determining the display location can allow the other application to mix or integrate video communication data into data received, generated, or otherwise obtained via the current video application. For example… Figure 5 The participating device 200 is shown displaying integrated video communication data 500. Figure 5In this context, the game application can receive video communication data (e.g., an image of a user on the other participating device), which can then integrate the user's image data into a specific scene within the electronic display (e.g., as an opponent in a domino game). More generally, the other application can control, transform, modify, or otherwise adjust the video communication data to present it on the electronic display in a way that differs from how the video communication data would have been displayed upon initial reception. Therefore, the video communication data can rely on the other application to generate a single video stream for displaying the video communication data, rather than simply displaying separate video data streams.

[0056] As shown in 450, the video communication data can then be displayed on the electronic display at the determined display position. If no display mode change is made (e.g., switching to full-screen mode), as shown in the negative branch of 460, the determination of the display position of the video communication data and the display of the video communication data in 440 and 450 can be repeated. Thus, the other application can change the display position of the video communication data over time. Figures 6A to 6C This illustrates the different display positions of video communication data that can be determined over time. For example, it can be initially determined when the other application just starts running. Figure 6A The video communication data is displayed at position 610. However, if the other application (e.g., in response to user input) changes the display of data on the electronic display, the display position of the video communication data may also change. For example, consider the other application as an example of a shared book reading experience via an electronic display. Figure 6A and Figure 6B In between, the virtual pages in the shared book application were "flipped," causing the optimal display position for the video communication data to change to display position 620. Similarly, the display position can be... Figure 6B and Figure 6C The display position changes again, from 620 to 630.

[0057] As shown in the affirmative branch of 460, switching to full-screen mode allows control of the displayed video communication data to revert back to the video communication application. It should be noted that in some implementations, video communication can be initiated while the video communication application is considered to be in integrated display mode. For example, the other application may already be running and displaying graphics on an electronic display. Therefore, after receiving the video communication data, as shown in element 410, the display position for the received video communication data can then be dynamically determined, as shown in 440 (instead of initially displaying the video communication in full-screen mode). The following discussion... Figure 7A similar example of initiating video communication in an integrated display mode is provided. Although it describes multi-party video communication, the same technique can be applied to video communication between two parties in at least some implementations.

[0058] Figure 7 This is a high-level flowchart illustrating methods and techniques for dynamically displaying video communication data for multiple video communication participants, according to some embodiments. As shown in 710, image data (or other video or graphics) can be displayed within an electronic display for an application. While the application is running, video communication can be initiated with multiple users, as shown in 720. For example, interface elements or other components can be selected to initiate video communication. Alternatively, in some embodiments, an incoming connection for video communication with multiple participants can be received.

[0059] As shown in 730, in various embodiments, video communication data of the video communication can be received, including video communication data for each of the plurality of participants. For example, individual data streams transmitted from each of the participants can be received. One or more display positions for the video communication data can be dynamically determined within an electronic display based on display attributes configured at runtime as part of running the application, as shown in 740. Once display positions are determined for the plurality of participants, the corresponding video communication data for each participant can be displayed in the determined display position, as shown in 750. It should be noted that video communication data from different participants may arrive at different times, so the video communication data of each participant can be displayed independently of other participants. For example, the video communication data of participant C may be displayed before the video communication data of other participants. In some embodiments, when a participant joins or leaves the video communication, the display of the corresponding video communication data of this additional participant can be added or removed (based on the dynamically determined display positions).

[0060] For example Figure 8 According to some embodiments, video communication data dynamically displayed on the electronic displays of participating devices for multiple video communication participants is shown. Different display positions on the electronic displays of participating device 200 can be determined for each participant. For the video communication data of participant A, display position 810 is determined. For the video communication data of participant B, display position 820 is determined. For the video communication data of participant C, display position 830 is determined. In some embodiments, the other application may be another video communication or transmission application, such as a screen sharing application. For example, other video communication data 840 may be information shared between the respective electronic displays by the video communication participants via a screen sharing application.

[0061] Even if another application is running and controlling the electronic display, input from video communication in the integrated display mode can still be processed. In some implementations, the video communication application can process the received input and simulate an appropriate response. Figure 9 This is a high-level flowchart illustrating methods and techniques for processing input for dynamically displayed video communication data, according to some embodiments. As shown in 910, video communication data can be displayed within an electronic display according to a dynamically determined display position. As mentioned above, the determination of the display position can be performed with reference to display attributes configured as part of the operation of an application different from the video communication application. As shown in 920, input for video communication can be detected. For example, in some embodiments, the electronic display can be touch-sensitive, and one or more touch gestures or inputs can be detected via the electronic display. Other input devices such as a computer mouse, trackball, keyboard, or voice commands can also transmit input for video communication.

[0062] In various implementations, if input is detected, as shown in the affirmative branch of 920, the input can be analyzed or parsed to determine a response, as shown in 930. For example, one or more user interface elements can be displayed, and the selection of a specific user interface element corresponding to a specific response can be determined. In some implementations, these user interface elements can also be displayed dynamically under the control of the other application. As another example, a specific gesture for a touch-sensitive input device (such as a touch-sensitive electronic display) can be interpreted and correspond to a specific action.

[0063] One definite response is to terminate the video communication, as shown in 940. For example, a user interface element for terminating the video communication can be selected, and in response, the video communication connection can be terminated. Similarly, a specific gesture (such as a specific touch swipe in a specific direction) can also be interpreted as a request to terminate the video communication, such as... Figure 10A The touch input 1012 shown.

[0064] One definite response is to display user interface elements for video communication, as shown in 950. For example, Figure 10A The participating device 200 is shown displaying video communication data for the current video communication at display location 1010. Touch input 1012 can be detected within display location 1010. Figure 10B As shown, additional user interface elements 1020 may be displayed within display location 1020, which allows for additional control over video communication (e.g., pausing, resuming, ending, or switching to full-screen display mode). In some embodiments, additional touch input relative to these user interface elements 1020 may then be detected.

[0065] Another definite response is the restoration of video communication, as shown in 960. In some implementations, for example, Figure 10A Touch input can be a specific gesture (e.g., a click) used to resume video communication from a paused state. Similarly, another determinable response is to pause video communication, as shown in 970. For example, a pause user interface element can be selected, or a specific action or touch gesture can be performed on a participating device (e.g., rotating the participating device in another direction).

[0066] One response could be switching from integrated display mode to full-screen display mode, as shown in 980. In some implementations, for example, Figure 10A The touch input 1012 can be a specific gesture (such as a double-tap or directional swipe) used to switch to full-screen display mode. Figure 10C As shown, participant device 200 in Figure 10A The video communication is then displayed in full-screen mode 1030 after input 1020 is shown.

[0067] As described above, according to some implementation schemes, dynamic pausing and resuming of video communication can be implemented. Mobile computing devices such as mobile phones, laptops, personal digital assistants (PDAs), and portable multifunction computing devices (e.g.) Figures 22 to 24 The connectivity or access to a network on which video communication can be transmitted in a portable multi-functional computing device (3000) or computing system (e.g., computing system 4000) may be limited. Video communication, like a face-to-face meeting, may experience fluctuations in the amount of active information exchange. However, video communication may continue to transmit video communication data even when very little or no active information exchange is taking place. Dynamic pause and resume allow the transmission of video communication data to correspond to the information exchange between the video communication participants. Data transmission for video communication from the participant devices can be evaluated to dynamically determine whether the transmission of video communication data (or at least some of the video communication data) can be paused or resumed.

[0068] Figure 11 This is a block diagram illustrating an exemplary transmit / receive manager module for a video communication module that dynamically pauses and resumes the transmission of video communication data, according to some implementation schemes. The video communication module 300 may implement a transmit / receive manager 330, as referenced above. Figure 3The captured video communication data 1102 can be received at the transmit / receive manager 330 for transmission 1106 to the participating device as part of the video communication. The stream of the captured video communication data can be evaluated. For example, the transmit / receive manager 330 can implement an active exchange dynamic evaluator 1110 to dynamically evaluate the video communication data stream to detect pause or resume events. The active exchange dynamic evaluator 1110 can implement various video analytics techniques (e.g., facial recognition), audio analytics techniques, other sensor information, and user-provided information (e.g., interactions with applications at the participating device) to evaluate the video communication data stream. This is discussed below. Figure 13 Further examples of techniques that can be implemented to detect pause or resume events are provided. Once detected, the pause or resume event can be transmitted to the transport status manager 1140.

[0069] The transmit / receive manager 330 may implement the transmit status manager 1140 to maintain the current transmit status of the video communication. For example, the transmit status manager 1140 may indicate whether the video communication is paused, and what video communication data (if any) is allowed to be transmitted when the video communication is paused. For example, if video data is filtered out while audio data of the video communication is still transmitted, the transmit status manager 1140 may indicate that the video data is filtered out. In various embodiments, indications for manually pausing and resuming video communication 1104 (rather than pause and resume events detected by the active exchange dynamic evaluator) may also be received and the transmit status indicated at the transmit status manager 1140 may be updated.

[0070] In some implementations, the transmit / receive manager 330 may implement a video communication data stream filter 1120. In some implementations, the video communication data stream filter 1120 may pause or filter some transmitted data, such as video data, audio data, or both, depending on the current transmission state. Even when not paused (when some receiving devices may only receive a portion of the video communication data stream), the filter 1120 may filter or select different amounts of video communication data streams to transmit (e.g., only audio or a small amount of video data). The transmit / receive manager 330 may implement a video communication data stream transmission formatter 1130. The filtered data (or unfiltered data if full transmission resumes) may then be formatted at the video communication data stream transmission formatter 1130 for transmission according to the established connection for video communication. Various techniques may be implemented for encoding, encrypting, compressing, and / or formatting communication data, and for constructing messages, packets, or other transmission containers conforming to communication protocol formats (such as Real-Time Protocol (RTP) formats) to prepare the video communication data for transmission. The video communication data may then be transmitted.

[0071] Figure 12 This is a high-level flowchart illustrating methods and techniques for dynamically detecting pause or resumption events in video communication, based on some implementation schemes. In some implementations, such as 1210, the video communication data stream can be captured for transmission to the participants in the current video communication. (See above reference...) Figure 3 The capture of video communication data can be performed by various optical and / or auditory sensors (e.g., cameras and / or microphones). In various embodiments, video communication can be managed peer-to-peer, as described above with reference to Figure 1. As shown in Figure 1222, in some embodiments, the video communication data stream can be dynamically evaluated. The following discussion... Figure 13 A discussion of various evaluation techniques is provided. For example, video and / or audio data within video communication data, as well as various other inputs from other sensors or components, can be analyzed. Based on the evaluation, pause events can be detected, as shown in 1230.

[0072] If a pause event is detected, as shown in the affirmative branch of 1230, the video communication can be paused (for this purpose, at least a portion of the captured video communication data stream is then not transmitted to the participants in the video communication). It should be noted that in various implementations, not all receiving devices that are participants in the video communication can receive the same amount of data. For example, some participating devices may be more resource-constrained than others (e.g., preventing devices from effectively processing the entire video communication data stream). Also, different users may have different access rights or interests to the video communication data stream from the transmitting device, or a user of the transmitting device may wish to mask graphic effects or other information for a specific recipient in a multi-party video communication. Accordingly, pausing the transmission of video communication data can reduce or stop transmission based on the corresponding amount of the originally transmitted video communication data stream. If, for example, one recipient only receives audio data as part of the video communication, while another recipient receives both audio and video data, then in the case of a pause at the transmitting device, the first recipient may not receive any video communication data stream, while the second recipient may only receive audio data. Alternatively, in some implementations, a pause can reduce all transmissions of video communication data streams to a specific amount (or not at all).

[0073] Alternatively, if the current transmission state of the video communication is paused, a resumption event can be detected, as shown in the affirmative branch of 1250. The transmission of at least a portion of the video communication data stream can be resumed or retransmitted, as shown in 1260. For example, if no data (or only audio data) is transmitted during the pause, both audio and video data can be transmitted to the receiving participant upon detection of the resumption event. Similarly, as described above, different recipients in a multi-party video communication can receive different amounts of video communication data streams. When transmission of at least a portion of the video communication data stream resumes, the amount of video communication data originally sent to the participating recipients can be restored (e.g., if only audio was sent previously and no data was sent during the pause, only audio can be sent upon resumption). The situation of the participants in a multi-party video communication may change during the pause of the transmitting device. For example, a previously resource-constrained receiving device may no longer be resource-constrained (and alternatively, the receiving device may become resource-constrained). In some embodiments, regardless of the type of pause, upon resumption, an amount of video communication data stream different from the amount originally transmitted before the pause can be sent to the receiving device.

[0074] In some implementations, manual input for pausing and resuming events can be received. As shown in 1220, indications of user selections regarding the state of video communication transmission can be received. For example, user interface elements can be selected, and touch input, audio input, or some other form of input can be received and interpreted as corresponding to pausing or resuming video communication.

[0075] Dynamic detection of pause and resume events can be advantageous when operating multi-party video communications and / or displaying video communication data in integrated display modes. For example, when multiple different parties are transmitting and receiving video communication data, dynamic pausing of the video communication data can reduce the computational burden on the transmitting and receiving parties' devices. Similarly, automated or dynamic detection of pause and resume events can prevent users from manually pausing and resuming transmissions while operating other applications simultaneously with video communication (a situation that may occur in integrated display modes).

[0076] Figure 13This is a high-level flowchart illustrating methods and techniques for determining active exchange indicators in video communications, according to some embodiments. As described above, video communication data can be captured 1300 for transmission. In various embodiments, the video data can be analyzed 1310. For example, facial recognition technology can be performed to identify whether someone is in the line of sight of a participant's device. Other forms of video analysis can be used. For example, light levels can be determined, or the amount of motion can be detected. As shown 1312, in some embodiments, audio data can be analyzed. Speech analysis can be used to detect the amount of conversation or speech. Background noise or other audio signals can also be evaluated or identified (and ignored). For example, it can be determined that the speech being spoken is not directed at other participants in the video communication. Other sensor inputs 1314 can also be analyzed. For example, such as those included in the references below. Figure 22 The orientation sensor in the portable multi-functional device 3000 can be used to detect movement of participating devices. In some embodiments, other sensor inputs 1314 can indicate whether another participating device linked to the same user account as the device transmitting data is also transmitting video communication data and / or whether another participating device linked to the same user account is near the device. If it can be determined, based on analysis of audio or video data received from the other device, that a user is instead using the other device to participate in video communication, it can be indicated whether to pause video communication at that device. For example, if a mobile phone and a tablet are both linked to the same user account and both devices are participating in video communication, a pause event can be detected on one of these devices (e.g., the tablet) based on active information exchange determined at the mobile phone and / or based on the mobile phone being near the tablet.

[0077] As shown in 1320, in some embodiments, an active exchange indicator can be determined. For example, in some embodiments, various weighting schemes for different determinations at elements 1310, 1312, and 1314 can be implemented. A face identified within the line of sight of a participating device, combined with a specific amount of detected speech, can be weighted as more likely to indicate active information exchange. Information provided by other users can be considered when determining the active exchange indicator. For example, if no face is recognized, but it is known that an outward-facing camera has been selected (not pointing at the user, but at a camera used to display something that is not the user's receiver), the lack of face recognition can be ruled out. Therefore, in various embodiments, the source or specific device capturing the video communication data stream can be considered when evaluating the data stream to detect pause and resume events.

[0078] Once the active exchange indicator is determined, it can be compared to a specific exchange threshold. Whether this triggers a resume event or a pause event depends on the current transmission state of the video communication. As shown in the negative branch of 1330, if the current transmission state is not paused, the active exchange indicator can be compared to the exchange threshold. If the active exchange indicator is below the exchange threshold, as shown in the positive branch of 1340, a pause event can be triggered, as shown in 1350. Otherwise, as shown in the negative branch of 1340, more data can be captured, and the analysis can be performed again. As shown in the positive branch of 1330, a comparison can be made between the active exchange indicator and whether it is above the exchange threshold, as shown in 1360. If the active exchange indicator is above the exchange threshold, as shown in the positive branch of 1360, a resume event can be triggered. Otherwise, as shown in the negative branch of 1360, more data can be captured, and the analysis can be performed again.

[0079] Initiating video communication typically involves both an initiating device and a receiving device. If the receiving device accepts the video communication, a video connection can be established. As video communication becomes more prevalent, many entities unknown to the receiving device may request video communication, or even attempt to conduct more video communication between known parties. However, video communication may not always be as desired unless the subject matter of the communication can be determined. Instead of initiating video communication, a video message can be sent to the receiving device, and the video message can be replayed to determine the nature of the video communication. If desired, the receiving device can initiate a dynamic transformation of the video message into video communication. Figure 14 It is a high-level flowchart illustrating methods and techniques for the dynamic transition from video messaging to video communication, based on some implementation schemes.

[0080] In various implementations, as shown in 1410, video communication data can be received from a remote mobile computing device as part of a video message from the remote mobile computing device via a video messaging protocol. For example, the video communication data can be sent and / or received via a Multimedia Messaging (MMS) protocol (which may be performed partially on a network different from the wide area network over which the video communication is transmitted, such as a wireless telephone network). Alternatively, the video communication data of the video message can be sent via an instant messaging or chat protocol.

[0081] When video communication data of a video message is received, the video message can be displayed on an electronic display at the receiving device, as shown in 1420. For example, the video message can be displayed at the receiving device in near real-time, either by "pop-up" or automatically, even if another application is currently displaying or controlling the electronic display, or if the electronic display is in a "sleep" or inactive state, similar to the integrated display mode discussed above. The video message can also be recorded for subsequent display at the receiving device, as shown in 1430. For example, the video communication data of the video message can be stored locally on the receiving storage device. In various implementations, the video communication data can be recorded at a lower quality or a different version compared to the data quality or version at the time of reception (e.g., it can be implemented by another video codec such as OpenFormat encoding).

[0082] In at least some embodiments, an instruction to initiate video communication with a remote mobile computing device can be received, as shown in the affirmative branch of 1440. In various embodiments, user selection of user interface elements or some other form of input received at the receiving device can indicate the intention to initiate video communication. For example, a touch gesture such as tapping a video message displayed on a touch-sensitive electronic display can indicate the desire to initiate video communication.

[0083] In various implementations, in response to receiving an instruction to initiate video communication, a video communication connection can be established with the remote mobile computing device, as shown in 1450. For example, various handshake messages or other credential exchanges can be performed to establish the connection. Subsequently received video communication data can then be received via the established video communication connection, which may differ from a video messaging protocol. For example, video communication can be performed via a specific implementation of a video transport protocol such as a peer-to-peer and / or peer-to-peer management implementation of the Real-Time Transport Protocol (RTP). Furthermore, video communication data can be captured and transmitted from the receiving device to the remote mobile computing device via the established connection (e.g., as part of a two- or multi-party video communication). In at least some implementations, video communication data received on the established video communication connection can continue to be recorded and stored along with video communication data received via a video messaging protocol for subsequent playback.

[0084] In at least some implementations, although a video communication connection is established and real-time communication is permitted, the captured video communication data can still be exchanged as messages via the established video communication channel. For example, a user can indicate in the selection of establishing a video communication connection that the session can still proceed "offline" (i.e., by sending and receiving video messages). The exchanged video messages can be sent via the established connection (and according to various protocol requirements). Furthermore, instructions can be received to switch between an offline session and a real-time session via the established video communication connection. However, in some implementations, the method by which "offline" video messages can be transmitted can be determined based on cost or responsiveness determinations. For example, if the resources used to establish the video communication connection are available for other purposes, the video communication can be abandoned, and video messages can be exchanged via a video messaging protocol.

[0085] In at least some embodiments, no instruction to initiate video communication may be received, or a negative instruction not to initiate video communication may be received, as shown in the negative branch of 1440. In some embodiments, this instruction may indicate that the display of video communication data is expected to stop upon receipt, as shown in 1460. For example, a touch gesture such as swiping away from a video message displayed on a touch-sensitive electronic display may indicate that the display of video communication data is expected to stop (and video communication is not initiated). However, recording of the video message may continue. In some embodiments, the recorded and stored video communication data may subsequently be displayed in response to a playback request. Although not shown, in some embodiments, an instruction to establish video communication may subsequently be received (after the initial instruction not to display video communication). For example, a voice command may be detected to instruct the display and establishment of video communication with a transmitting device that has sent (or is still sending) a video message.

[0086] The transition from one-way or one-sided video messaging to two-way or multi-way video messaging can also be used to address network changes that may disrupt video communication (e.g., due to poor network connection quality). Figure 15 This is a high-level flowchart illustrating methods and techniques for monitoring the quality of a video communication connection during dynamic transitions between video communications and video messages, according to some implementation schemes. As shown in 1510, video communication data can be captured for transmission to a remote mobile computing device as part of a video communication. The captured video communication data can be sent to one or more remote mobile computing devices via an established connection. As shown in 1520, video communication can be monitored to determine connection quality indicators. For example, various signal strength metrics (for wireless connections) or network performance metrics (e.g., available bandwidth or connection speed) can be used to determine the connection quality indicators.

[0087] If the video communication maintains the connection quality indicator at or above a certain tolerance threshold, the video communication can continue, as shown in the negative branch of 1530. However, in various embodiments, if the connection quality indicator drops below the tolerance threshold, a switch to a video messaging protocol can be made. For example, as shown in the positive branch of 1530, an indication of a switch to a video messaging protocol can be provided, as shown in 1540. In various embodiments, this indication can be a prompt, notification, message, or information display at the transmitting participant's device. In some embodiments, notifications or other indications can also be provided to other participants in the video communication to indicate that a particular participant's device has switched to a video messaging protocol. As shown in 1550, the captured video communication data can be recorded for transmission as a video message via a video messaging protocol to a remote mobile computing device. Although not shown, in at least some embodiments, the reverse switch can be made, for example, if the connection quality indicator reaches or increases above the tolerance threshold, the captured video communication data can be transmitted again via the established video communication connection. For example, in some implementations, the connection quality indicator may be determined to be equal to or higher than the connection tolerance threshold, and in response, transmission of the captured video communication data may be resumed via the video communication connection.

[0088] While the preceding discussion focused on initiating two-way video communication by receiving video messages, the same techniques can be applied to larger groups of participating devices. For example, multiple participating devices communicating via group messaging threads or video chat threads can implement these techniques. Figure 16A and Figure 16B Some implementation schemes illustrate the dynamic transitions of video communication in multiple participating directions within a video message. Figure 16A Group message thread 1600 is shown. Different participants in group message thread 1600 can send / publish different video messages, such as participant A video message 1610, participant B video message 1620, and participant C video message 1630. Video messages can be displayed when received. Additionally, see the reference above. Figure 14 The device can receive instructions to initiate video communication, such as touch input on one or more of the video messages 1610, 1620, and / or 1630. Figure 16BThe diagram illustrates a dynamic transition to video communication with participants in group messaging thread 1600. Video communication data 1612 from participant A, 1622 from participant B, and 1632 from participant C can be displayed as part of a multi-party video communication. It should be noted that while all participants are illustrated as participating in multi-party communication, not all participants in the group messaging thread need to or accept participation in real-time video communication. In some implementations, for those participants who are not participating, video messages containing communication data can be transmitted or published to the group messaging thread. Thus, some participants in the group messaging thread can passively participate in group communication. In some implementations, participants who later join the group chat, messaging thread, or multi-party video communication may also receive (and may display) previous video communications or messages from the group communication (e.g., replaying session history).

[0089] As referenced above Figure 14 As described above, in various implementations, video communication data can be recorded, whether transmitted via a video messaging protocol or a video communication connection. In some implementations, the communication data can be stored locally at each participant's device. However, the large volume of video communication data may exceed local storage resources. Therefore, the video communication data can be stored in another format (e.g., a lower resolution). In some implementations, the video communication data can be stored on a remote storage device or data storage location, such as those referenced above. Figure 3 The video communication data stored in 360 is described.

[0090] However, various privacy protections and protocols can be implemented for any video communication data stored remotely. For example, explicit consent may be required for recording video communications remotely (or locally on another participant's device) (e.g., selecting or approving confirmation of dialog interface elements may be required). Video communication data may be encrypted, allowing only the participants in the video communication to access the stored video communication data. In some implementations, strict data retention policies may be implemented to delete the stored communication data after a short period of time (e.g., 24 hours or 3 days). For each access to the stored video communication data by another participant, current consent from those participants may be required. Alternatively, in some implementations, participant devices may be able to stream or send video communication data that may be stored only on that one participant device. For example, each participant device may only retain the video communication data captured on that participant device, bypassing any remote storage for the video communication.

[0091] Figure 17This is a block diagram illustrating a connection manager for a video communication module that implements dynamic transitions between video communication and video messaging, according to some implementation schemes. The connection manager 320 may implement a video message transition coordinator 1720, which can be configured to instruct various steps necessary for the transition between video messaging and video communication. For example, a start or stop command 1704 for video communication may be sent to the transmit / receive manager 330 to start or stop video communication. Similarly, the video message transition coordinator may request another application, such as an application, module, or component responsible for implementing video message transmission and reception (e.g., ...). Figure 22 The instant messaging module 3041 in the portable multifunction device 3000 processes video communication data as video messages (e.g., when the video connection quality drops below a certain threshold). A video message transition coordinator can determine when to transition between video message transmission and reception and video communication based on various inputs. For example, it can receive an instruction or user selection 1706 to initiate video communication for the currently displayed video message, which the video message transition coordinator can rely on to initiate the transition to video communication.

[0092] The connection manager 320 can also implement the connection quality monitor 1710, which can be configured to perform the above-mentioned reference. Figure 15 The various techniques described herein are used to evaluate connection quality data 1702 and determine when a transition to and from video message transmission can be performed. The connection quality monitor 1710 can also provide input to the video message transition coordinator to indicate when such a transition may need to be performed.

[0093] The connection manager 320 may also implement a video connection interface 1730. The video connection interface 1730 can establish a connection 1708 with a remote mobile computing device for conversion to video communication. The connection information 1708 may be provided to the transmit / receive manager 330, or other similar components responsible for transmitting captured video communication data via the established connection 1708.

[0094] The transmit / receive manager 330 can record / store the transmitted and / or received video communication data 1732 in the stored video data 360. The stored video data 360 can be a local persistent data store, or it can be a remote data store that stores the video communication data on a storage server or location or on multiple storage servers or locations (e.g., a single remote device or a distributed system).

[0095] Another technique to improve the ease and flexibility of performing video communications may involve reducing the amount of time required to establish video communications. Figure 18This is a high-level flowchart illustrating methods and techniques for real-time video communication according to some implementation schemes. Real-time video communication can be implemented based on pre-established video communication connections maintained for different participating devices. Each participating device can manage a list of authorized participating devices, and can receive or send real-time video communication to authorized participating devices. It should be noted that the term "real-time" can refer to the transmission time of video communication data between participating devices, without necessarily performing some connection establishment techniques before transmitting the video communication data. Therefore, in various implementation schemes, "real-time" video communication can be a video communication connection for which previously performed connection establishment steps have been performed, allowing the transmission of video communication data to begin "instantly" (from the perspective of the user of the participating device), without performing connection establishment steps. Therefore, the term "real-time" should not be understood as requiring absolutely instantaneous communication.

[0096] As shown in 1810, a request can be received to initiate real-time video communication with a remote mobile computing device. For example, some remote mobile computing devices that support / are authorized for real-time communication may, by default, initiate a request for real-time video communication when a request for video communication with one of those mobile computing devices is received. In response to receiving the request, it can be determined whether a real-time connection is authorized for a particular user, as shown in 1820. For example, a list of stored participant connection information, or other organizational schemes, can be applied, where certain participants are designated as authorized for real-time connections. If a participant device is on the authorized list, the real-time video connection is considered authorized.

[0097] If authorized, as shown in the affirmative branch of 1820, access can be made to the stored connection information for the remote mobile computing device, as shown in 1830. In various embodiments, this stored connection information can be maintained or updated to ensure successful maintenance of the connection with the remote mobile computing device. For example, heartbeats or other polling techniques can periodically or non-periodically send acknowledgment or reauthorization requests to the remote mobile computing device to obtain updated connection information or verify connection information. In some embodiments, the remote mobile computing device can proactively send updates to participating devices authorized to maintain access to real-time video communications to update or verify the connection information. The maintained connection information can be various identifiers, keys, credentials, tokens, addresses, or other information necessary for transmitting video communication data via a previously authorized and established video communication connection.

[0098] As shown in 1840, in various implementations, video communication data captured for real-time video communication can be transmitted based on a real-time connection stored for a remote mobile computing device. In this way, the communication data can be quickly received and displayed at the remote mobile computing device, thereby providing the user of the mobile computing device with a near real-time video communication experience. The following description... Figure 20 Further details are provided regarding the display of video communication data for real-time video communication. If the real-time connection is not authorized, as shown in the negative branch of 1820, a video communication connection may first be established with the remote computing device, as shown in 1850. For example, various identification and authentication exchanges may be performed with the remote mobile computing device and / or other third-party systems such as credential provisioning or identification systems to facilitate video communication with the remote mobile computing device. After the video communication connection is established, the video communication data captured for the video communication may be transmitted according to the established video communication connection, as shown in 1860.

[0099] Figure 19A and Figure 19B This is a high-level flowchart illustrating methods and techniques for obtaining authorization to establish a real-time video communication connection, based on some implementation schemes. Figure 19A This section illustrates the authorization to establish an instant video communication connection from the perspective of the requesting participating device. As shown in 1910, a request for authorization to establish an instant video communication connection can be sent to the remote mobile computing device. The request may provide various identifying credentials for the requesting device, such as a user identifier, contact identifier, or other information. As shown in 1920, in various embodiments, in response to receiving authorization, an instant video communication connection can be established with the remote mobile computing device. For example, various identification and authentication exchanges can be performed with the remote mobile computing device and / or other third-party systems, such as credential provisioning or identification systems, to facilitate video communication with the remote mobile computing device. Once established, the connection information for the instant video communication connection can be stored, as shown in 1930, and can subsequently be accessed when initiating instant video communication with the remote mobile computing device.

[0100] Figure 19B The techniques for establishing an instant video communication connection are illustrated from the perspective of a remote mobile computing device or a participating device that confirms or denies authorization for instant video communication. As shown in 1940, a request for authorization to establish an instant video communication connection can be received from the remote mobile computing device. As mentioned above, the request may include various identification information. Based on this identification information, it can be determined whether to accept authorization for the instant video communication connection, as shown in 1950. For example, the identification of the requesting participating device (or the user of that participating device) and an indication or notification of the authorization request can be directed to the user of the remote mobile computing device (e.g., displaying a notification or sending an email) to confirm or deny the instant connection authorization request. Confirmation input can be received, as shown in the affirmative branch of 1950, and a response with connection information for establishing the instant video communication connection can be sent to the remote mobile computing device, as shown in 1960. In some embodiments, the receiving participating device itself can initiate the process for establishing the instant video communication connection. If not authorized, the authorization request can be denied, as shown in 1970.

[0101] Figure 20 This is a high-level flowchart illustrating methods and techniques for displaying video communication data via a real-time video communication connection, according to some embodiments. As shown in 2010, video communication data for real-time video communication can be received from a remote mobile computing device. Before displaying the video communication data, it can be determined whether real-time video communication from this remote mobile computing device is authorized, as shown in 2020. For example, a list of authorized users / devices or contact information can be evaluated for a specific remote mobile computing device. If real-time video communication is authorized, the video communication data for real-time video communication can be displayed, as shown in 2030. In some embodiments, the video communication data can be displayed by default in an integrated display mode. In other embodiments, a full-screen display mode can be used to display the video communication data, thereby dynamically determining the display position. If real-time video communication is not authorized, real-time video communication can be rejected, as shown in 2040. In some embodiments, rejection can trigger a non-real-time video communication establishment process at the remote mobile computing device and / or the receiving device.

[0102] Figure 21 This is a block diagram illustrating a connection manager for a video communication module that implements real-time video communication, according to some implementation schemes. In various implementations, connection manager 320 may implement real-time connection manager 1740, which can be configured to coordinate authorization and perform real-time video communication. (See references above.) Figure 19A The instant connection authorization establishment request 1743 may be received at the instant connection manager 1740, which requests and / or receives user authorization 1748 for establishing instant video communication (or denies it). For authorized instant video communication, the instant connection manager 1740 may notify the video communication interface 1730 to establish an instant video communication connection with the requesting mobile computing device. The instant connection manager 1740 may maintain the connection information of the established instant video communication connection as part of the authorized instant connection participant information 1760.

[0103] Similarly, when the instant connection manager 1740 (for which it has not yet received a request from the specific user or participating device) receives a user authorization 1748 (or request) for establishing an instant connection with a specific user or mobile computing device, the instant connection manager 1740 may instruct the video communication connection interface 1730 to request authorization from the specific user or remote mobile computing device. Instant connection acceptance 1746 may be received for these requests, and subsequently acquired connection information may be retained as authorized instant connection participant information 1760 (which may be retained in the above reference). Figure 3 The aforementioned participant contact information is stored in 350.

[0104] In response to a request to initiate an instant connection 1744, the instant connection manager 1740 determines whether such an instant connection is authorized. If authorized, it instructs the video communication connection interface 1730 to provide instant connection information 1752 to the transmission / reception manager 330. Similarly, for incoming instant video communication, the instant connection manager 1740 determines whether the incoming instant video communication is authorized and instructs the display manager 340 to display instant video communication data 1754.

[0105] The methods described herein can be implemented in various embodiments using any combination of hardware and software. For example, in one embodiment, these methods can be implemented via a mobile computing device, a portable multifunction device (e.g., a processor that executes program instructions stored on a computer-readable storage medium coupled to a processor), or a portable multifunction device. Figures 22 to 24 Portable multi-functional devices 3000 or computer systems (e.g.) Figure 25 The computer system (4000) is used to implement this. These program instructions can be configured to implement the functions described herein (e.g., to implement the functions of various components of the video communication application described herein). The various methods shown in the accompanying drawings and described herein represent exemplary embodiments of the methods. The order of any methods can be changed, and various elements can be added, reordered, combined, ignored, and modified, etc.

[0106] This document describes implementations of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some implementations, the device is a portable communication device such as a mobile phone that also includes other functions such as PDA and / or music player functionality. Other portable electronic devices may also be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some implementations, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0107] The following discussion describes an electronic device that includes a display and a touch-sensitive surface. However, it should be understood that the electronic device may include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0108] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0109] Various applications that can run on a device can use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device can be adjusted and / or differ for each application, and / or can be adjusted and / or differ within the corresponding application. In this way, the device's common physical architecture (such as the touch-sensitive surface) can support various applications using a user interface that is intuitive and transparent to the user.

[0110] Attention is now directed to embodiments of portable devices having touch-sensitive displays. Figure 22 This is a block diagram illustrating a portable multi-functional device 3000 with a touch-sensitive display 3012 according to some embodiments. The touch-sensitive display 3012 is sometimes referred to as a "touchscreen" for convenience, and may also be referred to as a touch-sensitive display system. The device 3000 may include a memory 3002 (which may include one or more computer-readable storage media), a memory controller 3022, one or more processing units (CPUs) 3020, a peripheral interface 3018, RF circuitry 3008, audio circuitry 3010, a speaker 3011, a microphone 3013, an input / output (I / O) subsystem 3006, other input or control devices 3016, and an external port 3024. The device 3000 may include one or more optical sensors 3064. These components may communicate via one or more communication buses or signal lines 3003.

[0111] It should be understood that device 3000 is only one example of a portable multifunctional device, and device 3000 may have more or fewer parts than shown, may combine two or more parts, or may have different configurations or arrangements of these parts. Figure 22 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.

[0112] The memory 3002 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory 3002 by other components of the device 3000 (such as the CPU 3020 and the peripheral interface 3018) may be controlled by the memory controller 3022.

[0113] Peripheral interface 3018 can be used to couple the device's input and output peripherals to CPU 3020 and memory 3002. The one or more processors 3020 run or execute various software programs and / or instruction sets stored in memory 3002 to perform various functions of device 3000 and process data.

[0114] In some implementations, the peripheral interface 3018, CPU 3020, and memory controller 3022 may be implemented on a single chip, such as chip 3004. In other implementations, they may be implemented on separate chips.

[0115] RF (Radio Frequency) circuit 3008 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 3008 converts electrical signals into electromagnetic signals and vice versa, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 3008 may include well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 3008 can communicate wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). Wireless communication may use any of a number of communication standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Message Processing Field Protocol (XMPP), Session Initiation Protocol Extended for Instant Messaging Field Support (SIMPLE), Instant Messaging Field Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the date of this application.

[0116] Audio circuitry 3010, speaker 3011, and microphone 3013 provide an audio interface between the user and device 3000. Audio circuitry 3010 receives audio data from peripheral device interface 3018, converts the audio data into electrical signals, and transmits the electrical signals to speaker 3011. Speaker 3011 converts the electrical signals into sound waves audible to the human ear. Audio circuitry 3010 also receives electrical signals converted from sound waves by microphone 3013. Audio circuitry 3010 converts the electrical signals into audio data and transmits the audio data to peripheral device interface 3018 for processing. The audio data can be retrieved from memory 3002 and / or RF circuitry 3008 and / or transmitted to memory and / or RF circuitry by peripheral device interface 3018. In some embodiments, audio circuitry 3010 also includes a headset jack (e.g., ...). Figure 24(3212 in the text). The headset jack provides an interface between the audio circuitry 3010 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., mono or binaural headphones) and input (e.g., a microphone).

[0117] I / O subsystem 3006 couples input / output peripherals on device 3000, such as touchscreen 3012 and other input control devices 3016, to peripheral interface 3018. I / O subsystem 3006 may include display controller 3056 and one or more input controllers 3060 for use with other input or control devices. The one or more input controllers 3060 receive electrical signals from / send electrical signals to other input or control devices 3016. The other input control devices 3016 may include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click dials, etc. In some alternative embodiments, one or more input controllers 3060 may be coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing device such as mouse. The one or more buttons (e.g., ... Figure 24 3208) may include volume up / down buttons for volume control of speaker 3011 and / or microphone 3013. The one or more buttons may include a push-button (e.g., Figure 22 (3206 in the middle).

[0118] The touch-sensitive display 3012 provides input and output interfaces between the device and the user. The display controller 3056 receives electrical signals from the touchscreen 3012 and / or sends electrical signals to the touchscreen. The touchscreen 3012 displays visual output to the user. Visual output may include graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output may correspond to user interface objects.

[0119] Touchscreen 3012 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 3012 and display controller 3056 (together with any associated modules and / or instruction set in memory 3002) detect contact on touchscreen 3012 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 3012. In one exemplary embodiment, the contact point between touchscreen 3012 and the user corresponds to the user's finger.

[0120] The touchscreen 3012 may use LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies may be used in other embodiments. The touchscreen 3012 and display controller 3056 may utilize any of a number of known or future-developed touch sensing technologies to detect contact and any movement or interruption thereof, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touchscreen 3012.

[0121] The touchscreen 3012 may have a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 1060 dpi. Users can interact with the touchscreen 3012 using any suitable object or attachment such as a stylus, finger, etc. In some embodiments, the user interface is designed to operate primarily with finger-based contact and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0122] In some embodiments, in addition to the touchscreen, device 3000 may include a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that differs from the touchscreen and does not display visual output. The touchpad may be a touch-sensitive surface separate from the touchscreen 3012, or an extension of the touch-sensitive surface formed by the touchscreen.

[0123] The device 3000 also includes a power system 3062 for supplying power to various components. The power system 3062 may include a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0124] The device 3000 may also include one or more optical sensors 3064. Figure 22An optical sensor is shown coupled to an optical sensor controller 3058 in an I / O subsystem 3006. The optical sensor 3064 may include a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 3064 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 3043 (also referred to as a camera module), the optical sensor 3064 can capture still images or video. In some embodiments, the optical sensor is located at the rear of the device 3000, opposite to the touchscreen display 3012 at the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located at the front of the device, allowing the user to obtain an image of that user for use in the video conference while viewing other video conference participants on the touchscreen display.

[0125] The device 3000 may also include one or more proximity sensors 3066. Figure 7 A proximity sensor 3066 is shown coupled to a peripheral device interface 3018. Alternatively, the proximity sensor 3066 may be coupled to an input controller 3060 in an I / O subsystem 3006. In some embodiments, the proximity sensor turns off and disables the touchscreen 3012 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0126] Device 3000 includes one or more orientation sensors 3068. In some embodiments, these one or more orientation sensors include one or more accelerometers (e.g., one or more linear accelerometers and / or one or more rotational accelerometers). In some embodiments, these one or more orientation sensors include one or more gyroscopes. In some embodiments, these one or more orientation sensors include one or more magnetometers. In some embodiments, these one or more orientation sensors include one or more Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), and / or other global navigation system receivers. GPS, GLONASS, and / or other global navigation system receivers can be used to obtain information about the position and orientation (e.g., longitudinal or lateral) of device 3000. In some embodiments, these one or more orientation sensors include any combination of orientation / rotation sensors. Figure 22 One or more orientation sensors 3068 are shown coupled to the peripheral device interface 3018. Alternatively, the one or more orientation sensors 3068 may be coupled to the input controller 1060 in the I / O subsystem 3006. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from the one or more orientation sensors.

[0127] In some embodiments, the software components stored in memory 3002 include an operating system 3026, a communication module (or instruction set) 3028, a contact / motion module (or instruction set) 3030, a graphics module (or instruction set) 3032, a text input module (or instruction set) 3034, a Global Positioning System (GPS) module (or instruction set) 3035, and an application program (or instruction set) 3036. Furthermore, in some embodiments, memory 3002 stores device / global internal state 3057, such as... Figure 22 As shown in the figure. Device / global internal state 3057 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views or other information occupy various areas of the touch screen display 3012; sensor state, which includes information obtained from the device's various sensors and input control device 3016; and position information about the device's position and / or orientation.

[0128] Operating system 3026 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0129] The communication module 3028 facilitates communication with other devices via one or more external ports 3024 and includes various software components for processing data received by the RF circuitry 3008 and / or the external ports 3024. The external ports 3024 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, Wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used on iPod (Apple Inc. trademark) devices.

[0130] The contact / motion module 3030 can detect contact with the touchscreen 3012 (in conjunction with the display controller 3056) and other touch-sensitive devices (e.g., touchpads or physical click-based rotary dials). The contact / motion module 3030 includes multiple software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining whether there is movement of the contact and tracking that movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has terminated (e.g., detecting a finger lift event or contact interruption). The contact / motion module 3030 receives contact data from the touch-sensitive surface. Determining the movement of the contact point may include determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations may be applied to a single contact point (e.g., a fingertip) or multiple simultaneous contacts (e.g., "multi-touch" / multiple fingertip contacts). In some implementations, the contact / motion module 3030 and the display controller 3056 detect contact on the touchpad.

[0131] The touch / motion module 3030 can detect user gesture input. Different gestures on a touch-sensitive surface have different contact patterns. Therefore, gestures can be detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the icon location). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0132] The graphics module 3032 includes several known software components for rendering and displaying graphics on the touchscreen 3012 or other displays, including components for changing the intensity of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0133] In some implementations, the graphics module 3032 stores data to be used to represent graphics. Each graphic may be assigned a corresponding code. The graphics module 3032 receives one or more codes specifying the graphic to be displayed from applications, etc., and also receives coordinate data and other graphic attribute data if necessary, and then generates screen image data to output to the display controller 3056.

[0134] The text input module 3034, which can be a component of the graphics module 3032, provides a soft keyboard for entering text in various applications (e.g., contacts 3037, email 3040, instant messaging 3041, browser 3047, and any other application that requires text input).

[0135] GPS module 3035 determines the location of the device and provides that information for use in various applications (e.g., to phone 3038 for use in location-based dialing, to camera 3043 as image / video metadata, and to applications that provide location-based services such as weather desktop apps, local yellow pages desktop apps, and map / navigation desktop apps).

[0136] Application 3036 may include the following modules (or instruction sets) or subsets or supersets thereof:

[0137] • Contacts module 3037 (sometimes called address book or contact list);

[0138] • Telephone module 3038;

[0139] • Video communication module 3039;

[0140] • Email client module 3040;

[0141] • Instant Messaging (IM) module 3041;

[0142] Fitness support module 3042;

[0143] • Camera module 3043 for still images and / or video images;

[0144] • Image management module 3044;

[0145] • Browser module 3047;

[0146] • Calendar module 3048;

[0147] • Desktop mini-program module 3049 may include one or more of the following: Weather desktop mini-program 3049-1, Stock market desktop mini-program 3049-2, Calculator desktop mini-program 3049-3, Alarm clock desktop mini-program 3049-4, Dictionary desktop mini-program 3049-5, and other desktop mini-programs obtained by the user and desktop mini-programs created by the user 3049-6;

[0148] • Desktop Mini Program Creator Module 3050 used to create user-created desktop mini programs 3049-6;

[0149] • Search module 3051;

[0150] • Video and music player module 3052, which can be composed of a video player

[0151] It consists of a module and a music player module;

[0152] Notepad module 3053;

[0153] • Map module 3054; and / or

[0154] • Online video module 3055.

[0155] Examples of other applications 3036 that may be stored in memory 3002 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech copying.

[0156] In conjunction with the touchscreen 3012, display controller 3056, contact module 3030, graphics module 3032, and text input module 3034, the contact module 3037 can be used to manage an address book or contact list (e.g., stored in the application internal state 3092 of the contact module 3037 in memory 3002), including: adding one or more names to the address book; deleting one or more names from the address book; associating one or more phone numbers, one or more email addresses, one or more physical addresses, or other information with a name; associating an image with a name; categorizing and classifying names; providing a phone number or email address to initiate and / or facilitate communication via telephone 3038, video conferencing 3039, email 3040, or IM 3041; and so on.

[0157] Combining RF circuitry 3008, audio circuitry 3010, speaker 3011, microphone 3013, touchscreen 3012, display controller 3056, contact module 3030, graphics module 3032, and text input module 3034, telephone module 3038 can be used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in the address book 3037, modify entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication can use any of a variety of communication standards, protocols, and technologies.

[0158] Incorporating RF circuitry 3008, audio circuitry 3010, speaker 3011, microphone 3013, touchscreen 3012, display controller 3056, optical sensor 3064, optical sensor controller 3058, contact module 3030, graphics module 3032, text input module 3034, contact list 3037, and telephone module 3038, the video communication module 3039 includes executable instructions for initiating, conducting, and terminating video communication between the user and one or more other participants, as described above with reference to Figures 1 to 21.

[0159] Incorporating RF circuitry 3008, touchscreen 3012, display controller 3056, contact module 3030, graphics module 3032, and text input module 3034, email client module 3040 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 3044, email client module 3040 makes it very easy to create and send emails containing still images or video images captured by camera module 3043.

[0160] In conjunction with RF circuitry 3008, touchscreen 3012, display controller 3056, contact module 3030, graphics module 3032, and text input module 3034, instant messaging module 3041 includes executable instructions for inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS protocols for internet-based instant messaging), receiving an instant message, and viewing received instant messages. In some embodiments, the sent and / or received instant messages may include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0161] Incorporating RF circuitry 3008, touchscreen 3012, display controller 3056, contact module 3030, graphics module 3032, text input module 3034, GPS module 3035, map module 3054, and music player module 3046, fitness support module 3042 includes executable instructions for: creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (mobile devices); receiving fitness sensor data; calibrating sensors used for monitoring fitness; selecting and playing fitness music; and displaying, storing, and transmitting fitness data.

[0162] Incorporating a touchscreen 3012, a display controller 3056, an optical sensor 3064, an optical sensor controller 3058, a contact module 3030, a graphics module 3032, and an image management module 3044, the camera module 3043 includes executable instructions for capturing still images or videos (including video streams) and storing them in a memory 3002, modifying the characteristics of still images or videos, or deleting still images or videos from the memory 3002.

[0163] Incorporating a touchscreen 3012, a display controller 3056, a contact module 3030, a graphics module 3032, a text input module 3034, and a camera module 3043, the image management module 3044 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or photo album), and storing still images and / or video images.

[0164] Combining RF circuit 3008, touch screen 3012, display system controller 3056, contact module 3030, graphics module 3032, and text input module 3034, browser module 3047 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) according to user instructions.

[0165] Combining RF circuitry 3008, touchscreen 3012, display system controller 3056, contact module 3030, graphics module 3032, text input module 3034, email client module 3040, and browser module 3047, calendar module 3048 includes executable instructions for creating, displaying, modifying, and storing calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0166] In conjunction with RF circuitry 3008, touchscreen 3012, display system controller 3056, contact module 3030, graphics module 3032, text input module 3034, and browser module 3047, desktop applet module 3049 is a micro-application that can be downloaded and used by a user (e.g., weather desktop applet 3049-1, stock market desktop applet 3049-2, calculator desktop applet 3049-3, alarm clock desktop applet 3049-4, and dictionary desktop applet 3049-5) or a user-created micro-application (e.g., user-created desktop applet 3049-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop applet includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop applet).

[0167] Combining RF circuit 3008, touch screen 3012, display system controller 3056, contact module 3030, graphics module 3032, text input module 3034, and browser module 3047, the desktop applet creator module 3050 can be used by users to create desktop applets (e.g., transferring user-specified portions of web pages to desktop applets).

[0168] Combining the touchscreen 3012, display system controller 3056, contact module 3030, graphics module 3032, and text input module 3034, the search module 3051 includes executable instructions for searching the memory 3002 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0169] Incorporating a touchscreen 3012, a display system controller 3056, a contact module 3030, a graphics module 3032, an audio circuitry system 3010, a speaker 3011, a radio frequency circuitry system 3008, and a browser module 3047, the video and music player module 3052 includes executable instructions that allow a user to download and play recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and that the executable instructions display, present, or otherwise play video (e.g., on the touchscreen 3012 connected via an external port 3024 or on an external display). In some embodiments, the device 3000 may include the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0170] Combining the touchscreen 3012, display controller 3056, contact module 3030, graphics module 3032, and text input module 3034, the notepad module 3053 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0171] Combining RF circuitry 3008, touchscreen 3012, display system controller 3056, contact module 3030, graphics module 3032, text input module 3034, GPS module 3035, and browser module 3047, map module 3054 can be used to receive, display, modify, and store maps and map-related data (e.g., driving routes; data about businesses and other points of interest at or near a specific location; and other location-based data) according to user instructions.

[0172] In conjunction with touchscreen 3012, display system controller 3056, contact module 3030, graphics module 3032, audio circuitry 3010, speaker 3011, RF circuitry 3008, text input module 3034, email client module 3040, and browser module 3047, online video module 3055 includes instructions that allow a user to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 3024), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 3041, instead of email client module 3040, is used to send links to specific online videos.

[0173] Each of the modules and application programs described above corresponds to a set of executable instructions for performing one or more of the functions described above, as well as the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as standalone software programs, processes, or modules; therefore, various subsets of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, memory 3002 may store a subset of the modules and data structures described above. Furthermore, memory 3002 may store additional modules and data structures not described above.

[0174] In some implementations, device 3000 is the only device that performs operations of a predefined set of functions on the device via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 3000, the number of physical input control devices (such as push-buttons, dial pads, etc.) on device 3000 can be reduced.

[0175] The predefined set of functions, uniquely executable via a touchscreen and / or touchpad, includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates the device 3000 from any user interface that can be displayed on the device 3000 to the main interface menu, home menu, or root menu. In such embodiments, the touchpad may be referred to as a "menu button." In some other embodiments, the menu button may be a physical push-button or other physical input control device, rather than a touchpad.

[0176] Figure 23 This is a block diagram illustrating exemplary components for event handling according to some embodiments. In some embodiments, memory 3002 (in...) Figure 23 The (in the middle) includes an event classifier 3070 (e.g., in operating system 3026) and a corresponding application 3036-1 (e.g., any one of the aforementioned applications 3037-3051, 3055).

[0177] Event classifier 3070 receives event information and determines the application 3036-1 to which the event information should be delivered and the application view 3091 of application 3036-1. Event classifier 3070 includes event monitor 3071 and event dispatcher module 3074. In some embodiments, application 3036-1 includes application internal state 3092, which indicates one or more current application views displayed on touch-sensitive display 3012 when the application is active or executing. In some embodiments, device / global internal state 3057 is used by event classifier 3070 to determine which application(s) is currently active, and application internal state 3092 is used by event classifier 3070 to determine the application view 3091 to which the event information should be delivered, such as which applications can control the display of data on display 3012 and the display properties for configuring video communication, as described above with reference to Figures 1 through 10.

[0178] In some implementations, the application internal state 3092 includes additional information such as one or more of the following: recovery information to be used when the application 3036-1 resumes execution, user interface state information indicating information being displayed by the application 3036-1 or information ready to be displayed by the application 3036-1, a state queue for enabling the user to return to the previous state or view of the application 3036-1, and a repeat / undo queue for the user's previous actions.

[0179] Event monitor 3071 receives event information from peripheral device interface 3018. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 3012 as part of a multi-touch gesture). Peripheral device interface 3018 transmits information it receives from I / O subsystem 3006 or sensors (such as proximity sensor 3066, orientation sensor 3068, and / or microphone 3013 (via audio circuitry 3010)). The information received by peripheral device interface 3018 from I / O subsystem 3006 includes information from touch-sensitive display 3012 or touch-sensitive surfaces.

[0180] In some implementations, event monitor 3071 sends requests to peripheral device interface 3018 at predetermined intervals. In response, peripheral device interface 3018 transmits event information. In other implementations, peripheral device interface 3018 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0181] In some implementations, the event classifier 3070 also includes a hit view determination module 3072 and / or an activity event recognizer determination module 3073.

[0182] When the touch-sensitive display 3012 displays more than one view, the hit view determination module 3072 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0183] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected may correspond to a procedural level within the application's procedural hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected may be called the hit view, and the set of events recognized as correct input may be determined at least in part based on the hit view of the initial touch that initiated the touch-based gesture.

[0184] The hit view determination module 3072 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 3072 identifies the hit view as the lowest-level view in the hierarchical structure that handles the sub-events. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that forms an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source to which it is identified as the hit view.

[0185] The activity event recognizer determination module 3073 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 3073 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 3073 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher-level views in the hierarchy will still remain actively participating views.

[0186] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 3080). In embodiments that include active event identifier determination module 3073, event assigner module 3074 delivers event information to the event identifier determined by active event identifier determination module 3073. In some embodiments, event assigner module 3074 stores event information in an event queue, which is retrieved by the corresponding event receiver module 3082.

[0187] In some embodiments, the operating system 3026 includes an event classifier 3070. Alternatively, the application 3036-1 includes an event classifier 3070. In another embodiment, the event classifier 3070 is a separate module or part of another module (such as a contact / motion module 3030) stored in memory 3002.

[0188] In some embodiments, application 3036-1 includes a plurality of event handlers 3090 and one or more application views 3091, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 3091 of application 3036-1 includes one or more event recognizers 3080. Typically, each application view 3091 includes a plurality of event recognizers 3080. In other embodiments, one or more event recognizers among the event recognizers 3080 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 3036-1 inherits methods and other properties. In some embodiments, the corresponding event handlers 3090 include one or more of the following: a data updater 3076, an object updater 3077, a GUI updater 3078, and / or event data 3079 received from an event classifier 3070. Event handler 3090 may utilize or invoke data updater 3076, object updater 3077, or GUI updater 3078 to update the application's internal state 3092. Alternatively, one or more application views in application view 3091 include one or more corresponding event handlers 3090. Additionally, in some embodiments, one or more of data updater 3076, object updater 3077, and GUI updater 3078 are included in the corresponding application view 3091.

[0189] The corresponding event recognizer 3080 receives event information (e.g., event data 3079) from the event classifier 3070 and identifies events from the event information. The event recognizer 3080 includes an event receiver 3082 and an event comparator 3084. In some embodiments, the event recognizer 3080 further includes at least a subset of the following: metadata 3083 and event delivery instructions 3088 (which may include sub-event delivery instructions).

[0190] Event receiver 3082 receives event information from event classifier 3070. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information may also include the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device pose).

[0191] Event comparator 3084 compares event information with predefined event or sub-event definitions and determines, or determines or updates, the state of an event or sub-event based on the comparison. In some embodiments, event comparator 3084 includes event definition 3086. Event definition 3086 contains definitions of events (e.g., a predefined sequence of sub-events), such as event 1 (3087-1), event 2 (3087-2), and other events. In some embodiments, sub-events in event 3087 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (3087-1) is defined as a double-click on a displayed object. A double-click includes, for example, a first touch (touch start) on the displayed object for a predetermined stage, a first lift (touch end) for a predetermined stage, a second touch (touch start) on the displayed object for a predetermined stage, and a second lift (touch end) for a predetermined stage. In another example, event 2 (3087-2) is defined as dragging on a displayed object. Dragging includes, for example, a touch (or contact) on a displayed object for a predetermined stage, movement of the touch on the touch-sensitive display 3012, and lifting of the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 3090.

[0192] In some implementations, event definition 3087 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 3084 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view where three user interface objects are displayed on a touch-sensitive display 3012, when a touch is detected on the touch-sensitive display 3012, event comparator 3084 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 3090, the event comparator uses the result of the hit test to determine which event handler 3090 should be activated. For example, event comparator 3084 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0193] In some implementations, the definition of the corresponding event 3087 also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0194] When the corresponding event recognizer 3080 determines that the sub-event sequence does not match any event in the event definition 3086, the corresponding event recognizer 3080 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process ongoing sub-events based on touch gestures.

[0195] In some embodiments, the corresponding event recognizer 3080 includes metadata 3083 having configurable attributes, tags, and / or lists indicating how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 3083 includes configurable attributes, tags, and / or lists indicating how event recognizers can interact with each other. In some embodiments, the metadata 3083 includes configurable attributes, tags, and / or lists indicating whether a sub-event is delivered to a different level of change in a view or programmatic hierarchy.

[0196] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 3080 activates the event handler 3090 associated with the event. In some implementations, the corresponding event recognizer 3080 delivers event information associated with the event to the event handler 3090. Activating the event handler 3090 is different from sending (and delaying) the sub-event to the corresponding hit view. In some implementations, the event recognizer 3080 throws a flag associated with the identified event, and the event handler 3090 associated with the flag receives the flag and executes a predefined procedure.

[0197] In some implementations, event delivery instruction 3088 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event string or to an actively participating view. The event handler associated with the sub-event string or the actively participating view receives the event information and executes a predetermined procedure.

[0198] In some implementations, data updater 3076 creates and updates data used in application 3036-1. For example, data updater 3076 updates phone numbers used in contact module 3037 or stores video files used in video player module 3045. In some implementations, object updater 3077 creates and updates objects used in application 3036-1. For example, object updater 3076 creates new user interface objects or updates the location of user interface objects. GUI updater 3078 updates the GUI. For example, GUI updater 3078 prepares display information and sends it to graphics module 3032 for display on a touch-sensitive display.

[0199] In some embodiments, one or more event handlers 3090 include a data updater 3076, an object updater 3077, and a GUI updater 3078, or have access to the data updater 3076, object updater 3077, and GUI updater 3078. In some embodiments, the data updater 3076, object updater 3077, and GUI updater 3078 are included in a single module of the corresponding application 3036-1 or application view 3091. In other embodiments, they are included in two or more software modules.

[0200] It should be understood that the preceding discussion of event handling for user touch on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 3000 (not all of which are initiated on a touchscreen), such as coordinated mouse movement and mouse button presses (with or without single or multiple keyboard presses or holds), user movement taps, drags, scrolls, etc. on the touchpad, stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof, which can be used as input corresponding to sub-events of events defined to be recognized.

[0201] Figure 24A portable multifunction device 3000 with a touchscreen 3012 is shown according to some embodiments. The touchscreen can display one or more graphics within a user interface (UI) 3200. In this embodiment, and in other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, with one or more fingers 3202 (not drawn to scale in the figures) or with one or more styluses 3203 (not drawn to scale in the figures). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, gestures may include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 3000. In some embodiments, unintentional contact with a graphic will not select the graphic. For example, a swipe gesture over an application icon will not select the corresponding application when the gesture corresponding to selection is a tap.

[0202] Device 3000 may also include one or more physical buttons, such as a "home" button or a menu button 3204. As previously described, menu button 3204 can be used to navigate to any application 3036 in a set of applications that can be executed on device 3000. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 3012.

[0203] In one embodiment, device 3000 includes a touchscreen 3012, a menu button 3204, a push-button 3206 for powering on / off and locking the device, one or more volume control buttons 3208, a SIM card slot 3210, a headset jack 3212, and a docking / charging external port 3024. The push-button 3206 can be used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, device 3000 can also accept speech input via microphone 3013 for activating or deactivating certain functions.

[0204] It should be noted that although many of the examples below will be given with reference to input on touch screen 3012 (which is a combination of touch-sensitive surface and display), a touch-sensitive surface independent of the display can be used instead of touch screen 3012.

[0205] Implementations of the various methods and technologies for video communication described herein can run on one or more computer systems capable of interacting with various other participant devices, such as those referenced above. Figures 22 to 24 The portable multi-functional computing device 3000 is described above. Figure 25 The illustration depicts one such computer system. In various embodiments, the computer system 4000 can be any device of a variety of types, including but not limited to personal computer systems, desktop computers, laptops, notebooks, or netbooks, mainframe computers, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, consumer devices, video game controllers, handheld video game devices, application servers, storage devices, peripherals (such as switches, modems, routers), or any type of computing or electronic device in general.

[0206] In the illustrated implementation, computer system 4000 includes one or more processors 4030 coupled to system memory 4020 via input / output (I / O) interface 4030. Computer system 4000 also includes a network interface 4040 coupled to I / O interface 4030, and one or more input / output devices 4050, such as cursor control device 4060, keyboard 4070, display 4080, and various audio / optical sensors 4090 for capturing video and audio data. In some implementations, it is conceivable that the implementation may be carried out using a single instance of computer system 4000, while in other implementations, multiple such systems or multiple nodes constituting computer system 4000 may be configured as hosts for different parts or instances of the implementation.

[0207] In various implementations, computer system 4000 may be a single-processor system including one processor 4010, or a multiprocessor system including several processors 4010 (e.g., two, four, eight, or another suitable number). Processor 4010 may be any suitable processor capable of executing instructions. For example, in various implementations, processor 4010 may be a general-purpose or embedded processor implementing any of the various instruction set architectures (ISAs) (such as x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA). In a multiprocessor system, each processor 4010 may, but is not required to, implement the same ISA.

[0208] In some implementations, at least one processor 4010 may be a graphics processing unit (GPU). A GPU can be considered a dedicated graphics client device for a personal computer, workstation, game console, or other computing or electronic device. Modern GPUs are highly efficient at manipulating and displaying computer graphics, and their highly parallel architecture makes them more efficient than a typical CPU for a range of complex graphics algorithms. For example, a GPU can implement multiple graphics primitive operations in such a way that executing them is much faster than drawing directly to the screen using a host central processing unit (CPU). In various implementations, the image processing methods disclosed herein can be implemented at least in part by program instructions configured to execute on one such GPU or in parallel on two or more such GPUs. One or more GPUs may implement one or more application programming interfaces (APIs) that allow programmers to invoke the functionality of the one or more GPUs. Suitable GPUs are commercially available from NVIDIA Corporation, ATI Technology (AMD), and other companies.

[0209] System memory 4020 may be configured to store program instructions and / or data accessible to processor 4010. In various embodiments, system memory 4020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data that implement the desired functions (such as those described above for video communications as described herein) are shown as being stored in system memory 4020 as program instructions 4025 and data storage 4035, respectively. In other embodiments, program instructions and / or data may be received, transmitted, or stored on a different type of computer-accessible medium or similar medium separate from system memory 4020 or computer system 4000. Generally, computer-accessible media may include storage media or memory media, such as magnetic or optical media, like a disk or CD / DVD-ROM, coupled to computer system 4000 via I / O interface 4030. Program instructions and data stored on a computer-accessible medium can be transmitted via a transmission medium or signal (such as an electrical signal, electromagnetic signal, or digital signal) that can be transmitted via a communication medium (such as a network and / or a wireless link, such as that which can be implemented via a network interface 4040).

[0210] In one embodiment, I / O interface 4030 may be configured to coordinate I / O communication between processor 4010, system memory 4020, and any peripheral devices (including network interface 4040 or other peripheral device interfaces, such as input / output devices 4050) within the device. In some embodiments, I / O interface 4030 may perform any necessary protocol, timing, or other data conversions to convert data signals from one component (e.g., system memory 4020) into a format suitable for use by another component (e.g., processor 4010). In some embodiments, I / O interface 4030 may include support for devices attached, for example, via various types of peripheral bus (e.g., variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, the functionality of I / O interface 4030 may be partitioned across two or more separate components, such as a northbridge and a southbridge. Furthermore, in some embodiments, some or all of the functionality of I / O interface 4030 (such as an interface to system memory 4020) may be directly incorporated into processor 4010.

[0211] Network interface 4040 can be configured to allow data exchange between computer system 4000 and other devices (such as other computer systems) attached to the network, or between nodes of computer system 4000. In various implementations, network interface 4040 may support communication via wired or wireless general-purpose data networks, such as, for example, Ethernet of a suitable type; communication via telecommunications / telephone networks, such as analog voice networks or digital fiber optic communication networks; communication via storage area networks, such as Fibre Channel SANs; or communication via any other suitable type of network and / or protocol.

[0212] Input / output device 4050 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or retrieving data by one or more computer systems 4000. Multiple input / output devices 4050 may be present in the computer system 4000 or distributed across various nodes of the computer system 4000. In some embodiments, similar input / output devices may be separate from the computer system 4000 and may interact with one or more nodes of the computer system 4000 via wired or wireless connections (such as through network interface 4040).

[0213] like Figure 25As shown, memory 4020 may include program instructions 4025 and data storage 4035. The program instructions 4025 are configured to implement embodiments of the video communication technology described herein, and the data storage 4035 includes various data accessible by the program instructions 4025. Data storage 4035 may include data that can be used in the embodiments. In other embodiments, other or different software elements and data may be included.

[0214] Those skilled in the art will recognize that the computer system 4000 is merely exemplary and not intended to limit the scope of the methods and techniques described herein. Specifically, the computer system and device may include any combination of hardware or software capable of performing the functions described, including computers, personal computer systems, desktop computers, laptops, notebooks, or netbooks, mainframe computers, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, network devices, internet devices, PDAs, cordless phones, pagers, consumer devices, video game consoles, handheld video game devices, application servers, storage devices, peripherals (such as switches, modems, routers), or any type of computing or electronic device in general. The computer system 4000 may also be connected to other devices not shown, or may operate as a standalone system. Furthermore, the functionality provided by the illustrated components may, in some embodiments, be combined into fewer components or distributed across additional components. Similarly, in some embodiments, the functionality of some illustrated components may not be provided, and / or other additional functionalities may be available.

[0215] Those skilled in the art will also recognize that, although various items are shown as being stored in memory or on storage devices during use, these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software components may be executed in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable article of manufacture for reading by a suitable drive, many examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from computer system 4000 may be transmitted to computer system 4000 via a transmission medium or signal (such as an electrical signal, electromagnetic signal, or digital signal) transmitted via a communication medium (such as a network and / or wireless link). Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented according to the above description on a computer-accessible medium. Therefore, the invention can be implemented in other computer system configurations.

[0216] In the detailed description above, numerous specific details have been provided to offer a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be implemented without these specific details. In other instances, methods, apparatuses, or systems known to a person of ordinary skill have not been described in detail in order not to obscure the claimed subject matter.

[0217] It will also be understood that while the terms "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of the invention, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first and second contacts are contacts, but they are not the same contact.

[0218] The terminology used in the detailed description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms “an,” “a,” and “described” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the items listed in association. It will also be understood that the terms “includes,” “including,” “comprises,” and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0219] As used herein, depending on the context, the term “if” can be interpreted as meaning “when”, “in the event of”, “in response to determination”, or “in response to detection”. Similarly, depending on the context, the phrase “if determination” or “if detection [the stated condition or event]” can be interpreted as meaning “in the event of determination”, “in response to determination”, “when detection [the stated condition or event]”, or “in response to detection [the stated condition or event]”.

[0220] Some parts of the following detailed description are provided as algorithms or symbolic representations of operations on binary digital signals stored in the memory of a particular device or dedicated computing device or platform. In the context of this particular specification, the term "particular device," etc., includes a general-purpose computer, provided that the general-purpose computer is programmed to perform a particular function according to instructions from program software and other programmable electronic devices. Algorithm descriptions or symbolic representations are examples of techniques used by those skilled in the art of signal processing or related fields to convey the substance of their work to others skilled in the art. An algorithm here, and generally, is considered as a self-consistent sequence of operations or similar signal processing that leads to a desired result. In this context, the operation or processing involves the physical manipulation of physical quantities. Typically, but not necessarily, such quantities may take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, or otherwise manipulated. It has proven convenient to sometimes (primarily for the sake of habitual use) refer to such signals as bits, data, values, elements, symbols, characters, items, numbers, digits, etc. However, it should be understood that all these or similar terms are to be associated with appropriate physical quantities and are merely convenient notations.

[0221] The embodiments of this disclosure can be described based on the following claims.

[0222] 1. A system comprising:

[0223] One or more audio sensors, the one or more audio sensors being configured to capture audio data;

[0224] One or more video sensors, the one or more video sensors being configured to capture image data;

[0225] One or more processors;

[0226] The memory includes program instructions that, when executed by the one or more processors, cause the one or more processors to implement:

[0227] The video communication module is configured as follows:

[0228] Receive video communication data streams for current video communication from at least one audio sensor and at least one video sensor of the audio sensors at the transmission participant, for transmission to one or more receiving participants of the current video communication, wherein the current video communication is a peer-managed connection between the transmission participant and the one or more receiving participants;

[0229] The video communication data stream is dynamically evaluated to detect a pause event of the transmission participant in the current video communication; and in response to detecting the pause event, the current video communication is paused for the transmission participant, such that at least some of the video communication data stream is not transmitted from the transmission participant to the one or more receiving participants.

[0230] 2. The system of claim 1, wherein, in order to dynamically evaluate the video communication data stream, the video communication module is configured to:

[0231] Perform at least one of video analysis or audio analysis to determine the active exchange indicator of the transmission participants; and

[0232] The active exchange indicator is determined to be below the exchange threshold for the transmission participant in order to trigger the pause event.

[0233] 3. The system of claim 1, wherein the dynamic evaluation of the video communication data stream is based on the identifier of the at least one audio sensor or the at least one video sensor obtained from a portion of the video communication data.

[0234] 4. The system according to claim 1,

[0235] The program instructions therein cause the one or more processors to implement a user interface module configured to detect a selection to resume a user interface element or pause a user interface element;

[0236] The video communication module is further configured as follows:

[0237] Receive an instruction to select the restored user interface element;

[0238] In response to receiving the selection, the current video communication is resumed, such that at least a portion of the video communication data stream is retransmitted to the one or more receiving participants.

[0239] 5. The system according to claim 1, wherein the video communication module is further configured to:

[0240] Dynamically evaluate the video communication data stream to detect recovery events of the transmission participants in the current video communication; and

[0241] In response to the detection of the recovery event, the current video communication is resumed, such that all video communication data streams are transmitted to the one or more receiving participants.

[0242] 6. The system of claim 1, wherein the system is a mobile phone.

[0243] 7. A method comprising:

[0244] Executed by one or more computing devices:

[0245] Capture video communication data streams from the transmitting participants in the current video communication for transmission to one or more receiving participants in the current video communication, wherein the current video communication is a peer-to-peer managed connection between the transmitting participants and the one or more receiving participants;

[0246] The video communication data stream is dynamically evaluated at the transmission participant to detect pause events at the transmission participant in the current video communication; and

[0247] In response to the detection of the pause event, the current video communication is paused for the transmission participant, such that at least some of the video communication data stream is not transmitted from the transmission participant to the one or more receiving participants.

[0248] 8. The method of claim 7, wherein dynamically evaluating the video communication data stream to detect the pause event comprises:

[0249] Perform at least one of video analysis or audio analysis to determine the active exchange indicator of the transmission participants; and

[0250] The active exchange indicator is determined to be below the exchange threshold for the transmission participant in order to trigger the pause event.

[0251] 9. The method of claim 8, wherein at least one of the video analysis or the audio analysis is performed to determine the active exchange indicator of the transmission participant based on a specific capture source of a portion of the video communication data stream.

[0252] 10. The method of claim 7, further comprising:

[0253] The video communication data stream is dynamically evaluated at the transmission participant to detect a recovery event of the current video communication at the transmission participant; and in response to detecting the recovery event, the current video communication is resumed such that at least a portion of the video communication data stream is retransmitted to the one or more receiving participants.

[0254] 11. The method of claim 7, wherein the portion of the video communication data stream retransmitted to one of the one or more receiving participants is different from the portion of the video communication data stream retransmitted to the other of the one or more receiving participants.

[0255] 12. The method of claim 7, wherein the peer-managed connection between the transmitting participant and the one or more receiving participants is maintained in response to the detection of the pause event.

[0256] 13. The method of claim 7, wherein the one or more receiving participants are multiple receiving participants, and wherein the current video communication is a multi-party video communication between the multiple receiving participants and the transmitting participant.

[0257] 14. A non-transitory computer-readable storage medium storing program instructions, said program instructions causing the mobile computing device to perform, when executed by the mobile computing device:

[0258] Capture video communication data streams from a transmitting participant in a current video communication for transmission to one or more receiving participants in the current video communication, wherein the current video communication is a peer-managed connection between the transmitting participant and the one or more receiving participants, wherein the transmitting participant is in a suspended state such that at least some of the video communication data streams are not transmitted to the one or more receiving participants.

[0259] Dynamically evaluate the video communication data stream to detect recovery events of the transmission participants in the current video communication; and

[0260] In response to the detection of the recovery event, the transmission of at least a portion of the current video communication data stream to the one or more receiving participants is resumed.

[0261] 15. The non-transitory computer-readable storage medium of claim 14, wherein dynamically evaluating the video communication data to detect the recovery event comprises:

[0262] Perform at least one of video analysis or audio analysis to determine the active exchange indicator of the transmission participants; and

[0263] The active exchange indicator is determined to be higher than the exchange threshold used by the transmission participant in order to trigger the recovery event.

[0264] 16. The non-transitory computer-readable storage medium of claim 14, wherein the program instructions cause the mobile computing device to further perform:

[0265] Before the transmission participant enters the pause state, the video communication data stream is dynamically evaluated at the transmission participant to detect a pause event of the transmission participant that triggers the pause state of the transmission participant.

[0266] 17. The non-transitory computer-readable storage medium of claim 14, wherein the program instructions cause the mobile computing device to further perform:

[0267] Before the transmission participant enters the pause state, it receives an instruction for a user selection of a pause user interface element, which triggers the pause state of the transmission participant.

[0268] 18. The non-transitory computer-readable storage medium of claim 14, wherein at least some of the video communication data streams not transmitted to the one or more receiving participants are video data.

[0269] 19. The non-transitory computer-readable storage medium of claim 14, wherein the dynamic evaluation of the video communication data stream is performed in response to determining that the current video communication is displayed according to an integrated display mode.

[0270] 20. The non-transitory computer-readable storage medium of claim 14, wherein the one or more receiving participants are multiple receiving participants, and wherein the current video communication is a multi-party video communication between the multiple receiving participants and the transmitting participant.

[0271] 21. A system comprising:

[0272] Electronic displays;

[0273] One or more processors;

[0274] The memory includes program instructions that, when executed by the one or more processors, cause the one or more processors to implement a video communication module and a user interface.

[0275] The user interface is configured to detect user input;

[0276] The video communication module is configured as follows:

[0277] Receive video communication data as part of a video message from a remote mobile computing device via a video messaging protocol;

[0278] When the video communication data is received, the video message is displayed on the electronic display.

[0279] The video communication data is recorded for subsequent display on a portable multi-functional device;

[0280] Receive, via the user interface, an instruction for initiating video communication with the remote mobile computing device; and

[0281] In response to receiving the instruction to initiate the video communication, a communication connection is established with the remote mobile computing device, such that video communication data subsequently received from the remote computing device and video communication data subsequently transmitted to the remote computing device are sent via the established video communication connection, wherein the video communication connection is different from the video messaging protocol.

[0282] 22. The system of claim 21, wherein the electronic display is a touch-sensitive display, and wherein the user interface is configured to detect touch gestures on the electronic display in order to detect the user input.

[0283] 23. The system of claim 21, wherein the video communication module is further configured to:

[0284] Monitor the video communication connection to determine the connection quality value;

[0285] It was determined that the connection quality value was below the connection tolerance threshold;

[0286] In response to determining that the connection quality value is below the connection tolerance threshold:

[0287] The electronic display shows an indication of transition to the video messaging protocol; and

[0288] Additional video communication data is recorded to be sent to the remote mobile computing device as another video message via the video messaging protocol.

[0289] 24. The system of claim 21, further comprising:

[0290] Data storage device;

[0291] In order to record the video communication data, the video communication module is configured to store a version of the video communication data in the data storage device for playback.

[0292] The video communication module is further configured to store versions of the subsequently received video communication data and versions of the subsequently transmitted video communication data for playback.

[0293] 25. The system according to claim 21,

[0294] The program instructions cause the one or more processors to further implement a currently executing application module, which is configured to display image data on the electronic display.

[0295] In order to display the video message on the electronic display, the video communication module is configured to determine the display position within the electronic display where the image data of the application module is currently displayed.

[0296] 26. The system of claim 21, wherein the system is a mobile phone.

[0297] 27. A method comprising:

[0298] Performed by a portable multi-functional device:

[0299] Receive video communication data as part of a video message from a remote mobile computing device via a video messaging protocol;

[0300] When the video communication data is received, the video message is displayed on the portable multi-functional device.

[0301] When the video communication data is received, the video message is recorded for subsequent display on the portable multifunction device; and in response to receiving an instruction to initiate video communication with the remote mobile computing device, a video communication connection is established with the remote mobile computing device, such that video communication data subsequently received from the remote computing device and video communication data subsequently transmitted to the remote computing device are sent via the established video communication connection, wherein the video communication connection is different from the video message protocol.

[0302] 28. The method of claim 27, further comprising:

[0303] Receive additional video communication data from another remote mobile computing device as part of another video message from the other remote mobile computing device via the video messaging protocol;

[0304] Perform the display and the recording for the other video communication data of the other video message; and

[0305] In response to receiving an instruction to stop displaying the other video, the display of the other video message at the portable multifunction device is stopped.

[0306] 29. The method of claim 27, wherein recording the video communication data further comprises recording video communication data subsequently received from the remote mobile computing device, and wherein the method further comprises:

[0307] In response to a playback request, display the recorded video communication data as well as the subsequently received video communication data.

[0308] 30. The method of claim 29, wherein at least some of the subsequently received video communication data to be displayed is obtained from a remote data storage device containing subsequently received video communication data recorded in the hosting facility.

[0309] 31. The method of claim 27, further comprising:

[0310] Monitor the video communication connection to determine the connection quality value;

[0311] It was determined that the connection quality value was below the connection tolerance threshold; and

[0312] In response to determining that the connection quality value is below the connection tolerance threshold, additional video communication data is recorded to be sent to the remote mobile computing device as another video message via the video messaging protocol.

[0313] 32. The method of claim 31, further comprising:

[0314] After determining that the connection quality value is lower than the connection tolerance threshold, it is determined that the connection quality value is equal to or higher than the connection tolerance threshold;

[0315] In response to determining that the connection quality value is equal to or higher than the connection tolerance threshold, transmission of the captured video communication data via the established video communication connection is resumed.

[0316] 33. The method of claim 27, wherein the portable multifunction device and the remote mobile computing device are two of a large number of computing devices participating in a group message thread, wherein the method further comprises:

[0317] In response to receiving the instruction to initiate video communication with the remote mobile computing device, a corresponding additional video communication connection is established with one or more other computing devices in the group message thread that are not the remote mobile computing device, wherein the established video connection with the remote mobile computing device and the corresponding additional video communication connection with the one or more other computing devices together constitute a multi-party video communication.

[0318] 34. A non-transitory computer-readable storage medium storing program instructions, which, when executed by a mobile computing device, cause the mobile computing device to perform:

[0319] Receive video communication data as part of a video message from a remote mobile computing device via a video messaging protocol;

[0320] When the video communication data is received, the video message is displayed on the portable multi-functional device.

[0321] When the video communication data is received, the video message is recorded for subsequent display on the portable multi-functional device;

[0322] Receive instructions for switching to video communication with the remote mobile computing device; and

[0323] In response to receiving the instruction from the user input, a video communication connection is established with the remote mobile computing device, such that video communication data subsequently received from and transmitted to the remote computing device is sent via the established video communication connection, wherein the video communication connection is different from the video messaging protocol.

[0324] 35. The non-transitory computer-readable storage medium of claim 34, wherein the program instructions further cause the mobile computing device to:

[0325] Monitor the video communication connection to determine the connection quality value;

[0326] It was determined that the connection quality value was below the connection tolerance threshold;

[0327] In response to determining that the connection quality value is below the connection tolerance threshold:

[0328] Provides instructions for transitioning to the aforementioned video messaging protocol; and

[0329] Additional video communication data is recorded to be sent to the remote mobile computing device as another video message via the video messaging protocol.

[0330] 36. The non-transitory computer-readable storage medium of claim 34, wherein the program instructions cause the mobile computing device to further perform:

[0331] Receive additional video communication data from another remote mobile computing device as part of another video message from the other remote mobile computing device via the video messaging protocol;

[0332] Perform the display and the recording for the other video communication data of the other video message;

[0333] Receive user input indicating to stop displaying the other videos; and

[0334] In response to receiving the instruction from the user input to stop displaying the other video, the display of the other video message at the portable multifunction device is stopped.

[0335] 37. The non-transitory computer-readable storage medium of claim 36, wherein the program instructions further cause the mobile computing device to:

[0336] In response to a playback request, the recorded video communication data of the video message is displayed.

[0337] 38. The non-transitory computer-readable storage medium of claim 34, wherein the mobile computing device and the remote mobile computing device are two of a large number of computing devices participating in a group message thread, and wherein the program instructions cause the mobile computing device to further implement:

[0338] In response to receiving the instruction to initiate video communication with the remote mobile computing device, a corresponding additional video communication connection is established with one or more other computing devices in the group message thread that are different from the remote mobile computing device, wherein the established video connection with the remote mobile computing device and the corresponding additional video communication connection with the one or more other computing devices together constitute a multi-party video communication.

[0339] 39. The non-transitory computer-readable storage medium of claim 38, wherein additional video communication data transmitted between the plurality of computing devices as part of the multi-party video communication is stored at a storage server, and wherein the program instructions cause the mobile computing device to further implement:

[0340] In response to a playback request for a portion of the multi-party video communication, a portion of the additional video communication data obtained from the storage server corresponding to the requested portion of the multi-party video communication is displayed.

[0341] 40. The non-transitory computer-readable storage medium of claim 39, comprising each of the plurality of computing devices in the multi-party video communication maintaining a corresponding record of additional video communication data transmitted from the computing device as part of the multi-party video communication, and wherein the program instructions cause the mobile computing device to further implement:

[0342] In response to a playback request for a portion of the multi-party video communication, a corresponding record of additional video communication data obtained from a specific computing device corresponding to the requested portion of the multi-party video communication is displayed.

[0343] 41. A system comprising:

[0344] A data storage device configured to maintain corresponding connection information with one or more pre-established real-time video communication connections to one or more corresponding remote mobile computing devices;

[0345] One or more processors;

[0346] The memory includes program instructions that, when executed by the one or more processors, cause the one or more processors to implement a video communication module.

[0347] The video communication module is configured as follows:

[0348] Receive a request to initiate real-time video communication with a specific remote mobile computing device among the one or more remote mobile computing devices;

[0349] In response to receiving the request to initiate the real-time video communication:

[0350] It is determined that the real-time video communication connection with the specific remote mobile computing device is authorized;

[0351] In response to the determination:

[0352] Access the data storage device for the corresponding connection information of the specific remote mobile computing device; and transmit the captured video communication data of the requested real-time video communication according to the corresponding real-time connection information, such that the captured video communication data can be displayed on the remote portable multifunction computing device when received.

[0353] 42. The system of claim 41, wherein the video communication module is further configured to:

[0354] Before receiving the instant video communication request:

[0355] Send a request to the specific remote mobile computing device for authorization to establish the instant video communication connection with the specific remote mobile computing device;

[0356] Receive the corresponding connection information of the specific remote mobile computing device; and

[0357] The corresponding connection information of the specific remote mobile computing device is stored in the data storage device.

[0358] 43. The system of claim 41, wherein the video communication module is further configured to:

[0359] Before receiving the instant video communication request:

[0360] Receive a request from the remote mobile computing device for authorization to establish the corresponding pre-established instant video communication connection;

[0361] In response to receiving the request for authorization:

[0362] In response to determining that the specific remote mobile computing device is authorized to establish the pre-established instant video communication connection, the request is responded to with corresponding connection information for establishing the pre-established instant video communication connection.

[0363] 44. The system according to claim 41,

[0364] The system also includes an electronic display;

[0365] The video communication module is further configured as follows:

[0366] Receive other video communication data from another remote mobile computing device among the one or more remote mobile computing devices via another pre-established real-time video communication connection in the pre-established real-time video communication connection;

[0367] In response to receiving the other video communication data:

[0368] In response to determining that the other pre-established real-time video communication connection is authorized, the other video communication data is displayed on the electronic display when it is received.

[0369] 45. The system of claim 41, wherein, in order to display the other video communication data on the electronic display, the video communication module is configured to determine the current...

[0370] The display location within the electronic display shows image data from another application.

[0371] 46. ​​The system of claim 41, wherein the system is a mobile phone.

[0372] 47. A method comprising:

[0373] Portable mobile computing devices perform:

[0374] Receive requests to initiate real-time video communication with a remote portable multifunction computing device;

[0375] In response to receiving the request to initiate the real-time video communication:

[0376] It is confirmed that the real-time video communication connection with the remote portable multifunction computing device is authorized;

[0377] In response to the determination:

[0378] Access to the connection information stored in the remote portable multifunction computing device; and

[0379] The captured video communication data of the requested real-time video communication is transmitted according to the stored real-time connection information, so that the captured video communication data can be displayed on the remote portable multifunction computing device when received.

[0380] 48. The method of claim 47, further comprising:

[0381] Before receiving the instant video communication request:

[0382] Send a request to the remote portable multifunction computing device for authorization to establish an instant video communication connection;

[0383] Receive connection information for establishing the instant video communication connection; and store the connection information of the remote portable multifunction computing device.

[0384] 49. The method of claim 48, further comprising:

[0385] The remote portable multi-functional computing device performs:

[0386] Receive the request for authorization to establish the instant video communication connection with the portable multifunction computing device;

[0387] In response to receiving the request:

[0388] Determine that the portable multifunction computing device is authorized to establish the real-time video communication connection; and

[0389] In response to the determination that the portable multifunction computing device is authorized to establish the instant video communication connection, the request is responded to with the instant connection information.

[0390] 50. The method of claim 48, further comprising:

[0391] The remote portable multi-functional computing device performs:

[0392] Receive the video communication data via the real-time video communication connection; and

[0393] When the video communication data is received, it is displayed on the remote portable multifunction computing device.

[0394] 51. The method of claim 47, further comprising:

[0395] Receive another request to initiate real-time video communication with another remote portable multifunction computing device;

[0396] In response to receiving the other request:

[0397] In response to determining that the real-time video communication connection with the other remote portable multifunction computing device is unauthorized:

[0398] Establish a video communication connection with the other remote portable multifunction computing device; and

[0399] Other captured video communication data of the other requested real-time video communication are transmitted via the established video communication connection with the other remote portable multifunction computing device.

[0400] 52. The method of claim 47, further comprising:

[0401] Detecting changes in the connection information of the real-time video communication connection; and

[0402] In response to the detection of the change, the connection information stored in the remote portable multifunction computing device is updated according to the detected change.

[0403] 53. The method of claim 52, wherein detecting changes in the connection information of the instant video communication connection comprises periodically or non-periodically polling the remote portable multifunction computing device to verify the instant video communication connection.

[0404] 54. A non-transitory computer-readable storage medium storing program instructions, which, when executed by a mobile computing device, cause the mobile computing device to perform:

[0405] Maintain one or more pre-established real-time video communication connections with one or more corresponding remote mobile computing devices;

[0406] Receive video communication data from the corresponding remote mobile computing device via one of the pre-established real-time video communication connections;

[0407] In response to receiving the video communication data:

[0408] In response to determining that the pre-established real-time video communication connection has been authorized,

[0409] The video communication data is displayed when it is received.

[0410] 55. The non-transitory computer-readable storage medium according to claim 54,

[0411] The program instructions described therein cause the mobile computing device to further perform:

[0412] Before receiving the video communication data:

[0413] Send a request to the corresponding remote mobile computing device for authorization to establish the real-time video communication connection with the corresponding remote mobile computing device;

[0414] Receive connection information for establishing the instant video communication connection;

[0415] and

[0416] In maintaining one or more pre-established real-time video communication connections with one or more corresponding remote mobile computing devices, the program instructions cause the mobile computing device to: store the connection information of the remote mobile computing device.

[0417] 56. The non-transitory computer-readable storage medium of claim 54, wherein in the step of displaying the video communication data when the video communication data is received, the program instructions cause the mobile computing device to: determine a display position within an electronic display currently displaying image data of another application, wherein the display position is dynamically determined based on one or more display attributes configured at runtime as part of executing the other application.

[0418] 57. The non-transitory computer-readable storage medium of claim 54, wherein the program instructions cause the mobile computing device to further perform:

[0419] Before receiving the video communication data:

[0420] Receive a request from another mobile computing device for authorization to establish another pre-established instant video communication connection among the one or more pre-established instant video communication connections;

[0421] In response to receiving the request:

[0422] Determine that the other mobile computing device is authorized to establish the other pre-established instant video communication connection; and in response to determining that the other mobile computing device is authorized to establish the other pre-established instant video communication connection, respond to the request with connection information for establishing the pre-established instant video communication connection.

[0423] 58. The non-transitory computer-readable storage medium of claim 54, wherein the program instructions cause the mobile computing device to further perform:

[0424] Receive a request to initiate real-time video communication with another of the one or more remote mobile computing devices;

[0425] In response to receiving the request to initiate the real-time video communication:

[0426] In response to determining that the real-time video communication connection with the other remote mobile computing device is authorized, the captured video communication data of the requested real-time video communication is sent to the other remote mobile computing device via the corresponding other pre-established real-time video communication connection.

[0427] 59. The non-transitory computer-readable storage medium of claim 54, wherein the program instructions cause the one or more mobile computing devices to:

[0428] In response to receiving a deauthorization instruction for another of the one or more remote mobile computing devices, the pre-established instant video communication connection for the other remote mobile computing device is removed.

[0429] 60. The non-transitory computer-readable storage medium of claim 59, wherein the deauthorization instruction is received via a user interface at the mobile computing device, and wherein, in the removal of the corresponding pre-established instantaneous video communication connection for the other remote mobile computing device, the program instructions cause the mobile computing device to: send a deauthorization notification to the other remote computing device.

Claims

1. A system for dynamically displaying video communication data, the system comprising: monitor; One or more processors; as well as The memory includes program instructions that, when executed by the one or more processors, cause the one or more processors to: The display shows video communication data of active video communications received by a video communication application, which controls the display in full-screen mode. The system detects the launch of different applications to be displayed on the monitor, wherein the launch includes changes in the control of the monitor from the video communication application to the different applications; as well as In response to the detection of the launch of the different applications, subsequent video communication data of the active video communication received by the video communication application is displayed at one or more locations corresponding to different corresponding portions of the display determined by the different applications controlling the display based on user input.

2. The system of claim 1, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: In response to detecting that the orientation of the display has been changed to landscape, the display of subsequently received video communications at the determined one or more locations is changed to a landscape view; or In response to detecting that the orientation of the display has been changed to portrait, the display of subsequently received video communications at the determined locations is changed to a portrait view.

3. The system of claim 1, wherein the video communication data includes image data received from two or more remote computing devices for two or more participants.

4. The system of claim 1, wherein, in order to display subsequent video communication data of the active video communication received by the video communication application at the determined one or more locations, the program instructions cause the one or more processors to integrate the video communication data with other data displayed for the different applications.

5. The system of claim 1, wherein the program instructions, when executed by the one or more processors, further cause the one or more processors to: In response to a detected change in control of the display from the different applications back to the video communication application, further video communication data of the active video communication received by the video communication application is displayed in full-screen mode on the display.

6. The system of claim 1, wherein the system is a mobile phone.

7. A method for dynamically displaying video communication data, the method comprising: Received video communication data of active video communication is displayed in full-screen mode on mobile devices in a display controlled by the video communication application; The control of the display detected by the mobile device changes from the video communication application to different applications; as well as In response to detecting a change in control of the display to the different applications, the mobile device displays subsequently received video communication data of the active video communication at one or more locations corresponding to different corresponding portions of the display determined by the different applications controlling the display based on user input.

8. The method of claim 7, further comprising: In response to detecting that the orientation of the mobile device has been changed to landscape, the mobile device changes the display of subsequently received video communications at the determined one or more locations to a landscape view. or In response to detecting that the orientation of the mobile device has been changed to portrait, the mobile device changes the display of subsequently received video communications at the determined locations to a portrait view.

9. The method of claim 7, wherein the video communication data comprises image data received from two or more remote computing devices for two or more participants.

10. The method of claim 7, wherein displaying subsequently received video communication data of the active video communication at the determined one or more locations comprises changing the display of the video communication data from a first location among the determined one or more locations to a second location among the determined one or more locations.

11. The method of claim 7, wherein displaying subsequently received video communication data of the active video communication at the determined locations includes allowing the different applications to integrate the video communication data with other data displayed for the different applications.

12. The method of claim 7, further comprising: In response to the detection of a change in control of the display from the different applications back to the video communication application, the mobile device displays further received video communication data of the active video communication in full-screen mode of the display.

13. The method of claim 7, wherein the display is a touch screen display and wherein the user input is touch input to the touch screen display.

14. A non-transient computer-readable storage medium storing program instructions that, when executed by a portable multifunction device, cause the portable multifunction device to perform the method as described in any one of claims 7-13.

Images