Streaming video image data from personal electronic devices to shared electronic devices

Abstract

This technology is for establishing and maintaining secure audio / video data streaming connections between electronic devices, for example, between a personal electronic device and a shared electronic device, where these devices are not necessarily associated with the same user account. For example, a first device (e.g., a shared electronic device) may enter a first state of the peer device advertising protocol, detect the presence of a second device (e.g., a personal electronic device), and then initiate a streaming session between the devices. Once the session is initiated, the first device may periodically broadcast a first session state indicator, for example via Bluetooth® Low Energy (BLE) advertising packets, indicating whether the session is currently active. The first device may also begin to periodically receive a second session state indicator from the second device, also indicating whether the session is currently active. The first device may maintain the session until session disconnection criteria are met.

Description

Technical Field

[0001] The present disclosure generally relates to the field of audio and video data streaming. More particularly, without limitation, the present invention relates to techniques for automatically establishing and maintaining a secure connection between an electronic device, such as a personal electronic device and a shared electronic device, for example, for streaming audio and / or video data.

Background Art

[0002] With the advent of portable integrated computing devices, cameras and other video capture-capable devices have become widespread. These integrated computing devices generally take the form of smartphones, tablets, or laptop computers and typically include a general-purpose computer, a camera, a high-performance user interface including a touch-sensitive screen, and wireless communication capabilities via Wi-Fi (registered trademark), Bluetooth (registered trademark), LTE, HSDPA, New Radio (NR), and other cellular-based or wireless technologies. The widespread availability of these integrated devices provides an opportunity to use the capabilities of the devices to perform tasks that would otherwise require dedicated hardware and software. (BLUETOOTH (registered trademark) is a trademark of BLUETOOTH (registered trademark) SIG, INC. and Wi-Fi (registered trademark) is a trademark of the Wi-Fi Alliance)

[0003] For example, portable integrated computing devices such as smartphones, tablets, and laptops typically have two or more built-in cameras. These cameras generally correspond to lens / camera hardware modules that can be controlled through the use of a general-purpose computer with a user interface that includes firmware and / or software (e.g., "apps"), as well as touchless controls such as touchscreen buttons, fixed buttons, and / or voice control. The integration of high-quality cameras into these portable integrated communication devices, such as smartphones, tablets, and laptop computers, has enabled users to capture and share images and videos in ways that were previously impossible. Today, it is common for users' smartphones to be their primary image capture device.

[0004] With the rising popularity of photo and video sharing via portable integrated computing devices with integrated cameras, video conferencing via such devices is also on the rise. In particular, users often engage in video conference calls or meetings, and the video image is typically captured by a forward-facing camera on the device, i.e., a camera facing the same direction as the display screen of the camera device. Most conventional cameras are optimized for either wide-angle general photography or narrow-angle photography, such as self-portraits and video conferencing streaming use cases. These cameras optimized for wide-angle are typically optimized for group and landscape compositions, but are not optimal for individual portraits due to distortion that occurs, for example, when the subject is short distance from the camera or at the edge of the camera's field of view.

[0005] Cameras optimized for portraits and video conferencing streaming (e.g., "forward-facing" cameras) are typically not ideal for landscapes and group photos (or group video conferencing calls) due to their limited field of view. Furthermore, a given camera's field of view can also affect how the user constructs the shot (i.e., how far away and at what angle the user positions themselves relative to the device's camera) and the quality of the final captured image. In addition, some devices that may be most comfortable or suitable for users during long video conferencing calls, such as desktop or laptop computers with standalone monitors, may either lack an integrated camera or have a low-quality integrated webcam.

[0006] Therefore, it is desirable to have a method and system that provides users with the ability to seamlessly and automatically create and maintain secure connections between their electronic devices, for example, to enable direct streaming of audio and / or video image data captured by a first electronic device to a second electronic device for subsequent presentation, storage, or further transmission by the second electronic device, in order to provide users with greater flexibility in image capture sources and the ability to leverage higher quality image capture devices during video conferencing sessions (e.g., via a camera integrated into one of the user's portable communication devices). Ideally, such a method and system should also have low power consumption, be resilient to interruptions and / or pauses in the shared audio and / or video stream, and be flexible enough to enable secure connections between devices that are in close proximity to each other but not necessarily associated with the same user account. [Overview of the project]

[0007] This specification discloses devices, methods, and non-temporary program storage devices (NPSDs) for enabling the establishment and maintenance of secure audio / video data streaming connections between electronic devices that are not necessarily associated with the same user or the same user account, such as a personal electronic device and a shared electronic device. For example, a first method is disclosed herein, which includes, on a first electronic device (e.g., a shared electronic device), entering a first state of a peer device advertising protocol (e.g., BLUETOOTH® Low Energy or "BLE"), detecting the presence of a second electronic device (e.g., a personal electronic device) in proximity to the first electronic device while in the first state of the peer device advertising protocol, and initiating a first session between the first electronic device and the second electronic device via a secure wireless peer-to-peer connection protocol (e.g., a Wi-Fi®-based protocol).

[0008] When a first session is initiated, the first electronic device may establish a software process (e.g., a daemon) that periodically broadcasts a first session status indicator via the peer device advertising protocol, the first session status indicator indicating whether the first session is currently active. The first electronic device may also initiate a software process to receive a second session status indicator from a second electronic device via the peer device advertising protocol, the second session status indicator indicating whether the first session is currently active. The first device may then maintain the first session until a session disconnection criterion is met, the session disconnection criterion being at least partially based on the first or second session status indicator.

[0009] According to some embodiments, the session disconnection criterion is met when at least one of the following conditions occurs: (a) the first electronic device does not receive a second session state indicator indicating that the first session is currently active for longer than a first predetermined amount of time; (b) the first electronic device does not broadcast a first session state indicator indicating that the first session is currently active for longer than a second predetermined amount of time; (c) the first electronic device initiates a disconnection event for the first session; or (d) the first electronic device receives an indication of a disconnection event for the first session from the second electronic device.

[0010] According to other embodiments, the second electronic device comprises a first image capture device (e.g., a video camera), and the method further includes receiving one or more images captured in the second electronic device by the first image capture device during a first session. According to some such embodiments, the one or more images received in the first electronic device are part of a video image stream captured in the second electronic device, and interruption or pause of the video image stream in the second electronic device does not satisfy the session disconnection criterion. According to other such embodiments, two or more applications running on the first electronic device are configured to utilize one or more images received in the first electronic device. According to yet another such embodiment, the first and second electronic devices are required to be located within a threshold distance of each other (e.g., within BLE and / or Wi-Fi range) while receiving one or more images.

[0011] In further embodiments, the first session status indicator is broadcast within the manufacturer-specific data advertisement data (AD) structure of the BLE advertisement packet. In some such embodiments, the first session status indicator is broadcast within the BLE advertisement packet which includes an authentication tag (authtag) that identifies the first electronic device. In some such embodiments, the first session status indicator includes a 1-bit value or a 2-bit value.

[0012] According to some embodiments, a software process (e.g., a daemon) is configured to periodically broadcast a first session state indicator even when the display screen of the first electronic device is off (i.e., devices can only actively "scan" when their screens are on, but they can also "advertise" even when their screens are off, for example, if they are currently in a standby / sleep state). According to other embodiments, the first electronic device is configured to enter a first state of the peer device advertising protocol in response to the activation of a device discovery user interface flow.

[0013] According to some embodiments, a secure wireless peer-to-peer connection protocol (e.g., Wi-Fi) is a different protocol from a peer device advertising protocol (e.g., BLE). According to other embodiments, initiating a first session between a first electronic device and a second electronic device may further include using a QR code®, a PIN pairing process, or other forms of authentication mechanisms that ensure that only devices positively approved by the user can connect to those electronic devices (even if such devices are not currently associated with the same user account, the same wireless network, etc.).

[0014] According to some embodiments, the first electronic device and the second electronic device are associated with different users or different user accounts. According to other embodiments, the first electronic device is not associated with any single user account.

[0015] The system comprises various electronic devices, for example, memory, a first image capture device, and one or more processors operably coupled to the memory, wherein one or more processors cause the electronic device to initiate a first video conferencing session, enter a first state of the peer device advertising protocol, detect the presence of a second electronic device adjacent to the electronic device while in the first state of the peer device advertising protocol, initiate a first session by extending the first video conferencing session to the second electronic device via a secure wireless peer-to-peer connection protocol, and extend the first session via the peer device advertising protocol to the electronic device. Disclosed herein is an electronic device in which a software process is established to periodically broadcast a session status indicator, the first session status indicator indicating whether the first session is currently active, and the software process of the electronic device causes to receive a second session status indicator from a second electronic device via a peer device advertising protocol, the second session status indicator indicating whether the first session is currently active, and causes to maintain the first session until a session disconnection criterion is met, the session disconnection criterion is configured to execute an instruction that is at least partially based on the first session status indicator or the second session status indicator.

[0016] According to some embodiments of such electronic devices, the session disconnection criterion is met when at least one of the following conditions occurs: (a) the electronic device does not receive a second session state indicator indicating that a first session is currently active for longer than a first predetermined amount of time; (b) the electronic device does not broadcast a first session state indicator indicating that a first session is currently active for longer than a second predetermined amount of time; (c) the electronic device initiates a disconnection event for a first session; or (d) the electronic device receives an indication of a disconnection event for a first session from a second electronic device.

[0017] According to other embodiments of such electronic devices, one or more processors are further configured to execute instructions causing one or more processors to transmit one or more images captured by a first image capture device to a second electronic device during a first session.

[0018] Various non-temporary computer-readable media, for example, non-temporary computer-readable media including computer-readable instructions, wherein the computer-readable instructions cause one or more processors to cause a first electronic device to enter a first state of a peer device advertising protocol, detect the presence of a second electronic device adjacent to the first electronic device while in the first state of the peer device advertising protocol, initiate a first session between the first electronic device and the second electronic device via a secure wireless peer-to-peer connection protocol, receive one or more images from a video image stream captured by a first image capture device of the second electronic device while in the first session, and periodically set a first session state indicator via the peer device advertising protocol in the first electronic device. Also disclosed herein is a non-temporary computer-readable medium in which it is possible to establish a software process that broadcasts to a specific location, where a first session status indicator indicates whether the first session is currently active, and in the software process of the first electronic device, a second session status indicator is received from a second electronic device via a peer device advertising protocol, where the second session status indicator indicates whether the first session is currently active, and in the first electronic device, the first session is maintained until a session disconnection criterion is met, where the session disconnection criterion is at least partially based on the first or second session status indicator, and interruption or pause of the video image stream in the second electronic device does not satisfy the session disconnection criterion.

[0019] Embodiments of various other non - transient computer - readable media (CRM) are also disclosed herein. Such CRM is readable by one or more processors. Instructions can be stored in the CRM to cause one or more processors to implement any of the embodiments disclosed herein. Various image processing and secure device connection methods according to the embodiments of the devices and CRM disclosed herein are also disclosed herein.

Brief Description of the Drawings

[0020] [Figure 1] An exemplary personal electronic device attempting to establish a secure connection with a shared electronic device for streaming audio and video data according to one or more embodiments is shown.

[0021] [Figure 2] An exemplary block diagram showing the establishment and maintenance of a secure connection between electronic devices for streaming audio and video data according to one or more embodiments.

[0022] [Figure 3] An exemplary process flow diagram showing the establishment and maintenance of a secure connection between electronic devices for streaming audio and video data according to one or more embodiments.

[0023] [Figure 4] An exemplary process flow diagram showing the termination of a secure connection between electronic devices for streaming audio and video data according to one or more embodiments.

[0024] [Figure 5] An exemplary extended BLE advertising packet according to one or more embodiments is shown.

[0025] [Figure 6]A flowchart showing a method for establishing and maintaining a secure connection between electronic devices for streaming audio and video data according to various embodiments.

[0026] [Figure 7] A flowchart showing a method for "handing off" an active communication session from a first electronic device to a second electronic device and then maintaining a secure connection between the devices according to various embodiments.

[0027] [Figure 8] A block diagram showing a programmable electronic device in which one or more of the techniques disclosed herein may be implemented. **DETAILED DESCRIPTION**

[0028] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention disclosed herein. It will be apparent, however, to one of ordinary skill in the art that the invention can be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order not to obscure the invention. References to numbers without subscripts or suffixes are understood to refer to all cases of subscripts and suffixes corresponding to the reference number.

[0029] Furthermore, the language used in this disclosure has been principally selected for readability and for the purpose of explanation, and not for the purpose of delineating or limiting the invention, and one may need to rely on the claims to determine such a gist of the invention. References herein to "one embodiment" or "an embodiment" (or the like) mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention, and references to "one embodiment" or "an embodiment" multiple times do not necessarily all refer to the same embodiment.

[0030] The various technologies disclosed herein enable the establishment and maintenance of secure audio / video data streaming connections between electronic devices, e.g., personal electronic devices and shared electronic devices, which are not necessarily associated with the same user account. For example, a first electronic device (e.g., a shared electronic device) can enter a first state of a peer device advertising protocol, detect the presence of a second electronic device (e.g., a personal electronic device), and then initiate a session between the devices. Once the first session is initiated, the first electronic device may periodically broadcast a first session state indicator indicating whether the first session is currently active. The first electronic device may also begin receiving a second session state indicator from the second electronic device, also indicating whether the first session is currently active. The first device may maintain the first session until session disconnection criteria are met.

[0031] An exemplary device setup for attempting to establish a secure connection between user electronic devices.

[0032] Referring now to Figure 1, a scenario is shown in which several exemplary personal electronic devices 104A-104C are attempting to establish a secure connection 107 with a shared electronic device 102 for streaming audio and video data. As shown in Figure 1, the shared electronic device 102 may comprise a “smart TV” or other similar display device, and may comprise a built-in image capture device 106 (e.g., a “webcam”), which may be connected inside other components of device 102 and may be used to capture one or more exemplary images (109) of the scene surrounding device 102, for example, including one or more human subjects.

[0033] An exemplary personal electronic device 104A may also have a built-in image capture device 105 (and may also have one or more additional built-in image capture devices). In some cases, the image capture device 105 of the personal electronic device 104A may be of higher quality than the image capture device 106 of device 102 with respect to, for example, resolution, zoom, field of view (FOV), spatial resolution, focus, color quality, or any other imaging parameters. In such cases, it may be more desirable to use the image capture device 102 of the personal electronic device 104A rather than the image capture device 106 of device 102 to capture images that are used, enhanced, displayed, stored, transmitted, etc., by device 105, for example, as part of an active AV communication (e.g., video conferencing) session.

[0034] In other cases, the user of device 102 may simply want to select the image capture device of personal electronic device 104A to provide a different (and / or additional) view of the scene around device 102, for example, to provide a different (and / or additional) view of the scene around device 102, for any number of other reasons, such as the fact that device 102 may not have its own image capture device, the image capture device of device 102 may not be functioning properly, or the image capture device of personal electronic device 104A may have special image capture functions or modes desired by the user of the second device (e.g., "portrait" or composite shallow depth of field (SDOF) photography mode, "night" capture mode, "slow motion" video capture mode, etc.).

[0035] According to some embodiments, shared electronic devices 102 can only connect to personal electronic devices (e.g., personal electronic device 104A) that are in close proximity to each other and attempting to form a secure wireless connection, such as wireless connection 107A, via an agreed wireless connection protocol. As used herein, the term “in close proximity” may refer to devices that are discoverable from each other with respect to a given wireless connection protocol, for example, as indicated by a boundary circle 108 having a radius of 110.

[0036] In some implementations, measured signal strength can be used as a proxy to estimate the distance between two devices and determine whether they are within a sufficiently close range of each other. In some such implementations, the threshold signal strength required to determine that two devices are sufficiently close to each other may not be a fixed threshold, but may also be based on filtering (e.g., averaging) signal strength values ​​over time across many signal strength samples.

[0037] In some embodiments, after a successful connection, the personal electronic device 104A can seamlessly appear along any other image capture source available for selection within various applications in device 102, for example, along an image capture device located inside (i.e., built-in) device 102 (such as image capture device 106), or along an image capture device directly connected to device 102 (for example, via a USB port).

[0038] In some embodiments, device 102 may form a secure connection session with only one personal electronic device (e.g., personal electronic device 104A) at a time. Therefore, as shown in Figure 1, a second personal electronic device (i.e., personal electronic device 104B) is currently unable to establish a secure wireless connection 107B via the agreed wireless connection protocol with device 102. Similarly, a third personal electronic device (i.e., personal electronic device 104C) is also currently unable to establish a secure wireless connection 107C via the agreed wireless connection protocol with device 102.

[0039] As described herein, a secure wireless connection between personal electronic devices (e.g., personal electronic devices 104B and 104C) may also be disconnected for a variety of other reasons, such as: device 102 not receiving a session status indicator (e.g., via BLE) from personal electronic device 104 indicating that the streaming session is currently active for longer than a first predetermined amount of time; device 102 not broadcasting a session status indicator (e.g., which may be received by any nearby personal electronic device 104) indicating that the streaming session is currently active for longer than a second predetermined amount of time; a user of device 102 positively initiating a session disconnection event; and / or device 102 receiving an indication of a session disconnection event from a nearby personal electronic device 104.

[0040] Streaming sessions may also be interrupted for a variety of other reasons, including device crashes, crashes of the software processes that manage the session and run on the device, devices moving out of close proximity to each other, devices restarting, or entering sleep / standby mode.

[0041] Exemplary block diagram of electronic devices establishing and maintaining a secure connection.

[0042] Next, referring to Figure 2, an exemplary block diagram 200 is shown illustrating the establishment and maintenance of a secure connection between electronic devices 202 and 220 for streaming audio and video data, according to one or more embodiments.

[0043] Starting with the shared electronic device 202, the shared electronic device 202 may include an audiovisual (AV) framework 204 that can function as a framework for working with all time-based AV media consumed or played on the shared electronic device 202. The shared electronic device 202 may further include an external camera agent 206, which may expose (203) details of any / all external cameras it observes (or has registered) to the AV framework 204. The shared electronic device 202 may further include a connection management software program 208 (for example, a session management daemon). The connection management software 208 may serve various purposes, such as maintaining the currently selected personal electronic device (207) and / or running session management software (209) to handle broadcasting / receiving BLE advertising packets to other electronic devices, including the currently selected personal electronic device 207. The currently selected personal electronic device 207 may also observe (205) the external camera agent 206, for example, so that the shared electronic device 202 remains updated in the list of devices in the currently connected streaming session.

[0044] According to some embodiments, the external camera selection UI module (210) may be used in the shared electronic device 202 to enable the user of the shared electronic device 202 to select a desired external personal electronic device camera on which to establish a secure streaming session.

[0045] Next, referring to the personal electronic device 220, the personal electronic device 220 may include a connection management software program 222 (which may be similar to, for example, the connection management software program 208 of the shared electronic device 202, and / or include a session management daemon). The connection management software 222 may serve various purposes, such as maintaining the currently connected personal electronic device (223) and / or running session management software (225) to handle the broadcasting / reception of BLE advertising packets to other electronic devices, including the shared electronic device 202.

[0046] The currently connected personal electronic device 223 can also observe the camera management software 224 (227), which allows the personal electronic device 220 to maintain an updated state with the exact status of the currently connected personal electronic device (229), which can be presented to the user, for example, via the camera streaming user interface software 226. The camera streaming user interface software 226 of the personal electronic device 220 can allow the user to pause / resume the streaming session, end the streaming session, "hand off" the video conferencing session to another device, search for and connect to other shared electronic devices, etc.

[0047] In some embodiments, the peer device advertising protocol state may include a “default” or “low-power” advertising state configured to have a minimum advertising duty cycle and / or a maximum time interval delay between broadcasts of advertising packets. In other circumstances, for example, if a qualified device is known (or expected to be nearby), the peer device advertising protocol state may enter an “extended” or “high-power” advertising state configured to have a maximum allowable advertising duty cycle and / or a minimum allowable time interval delay between broadcasts of advertising packets. In some embodiments, a first device operating in the “extended” or “high-power” mode of the peer device advertising protocol state may also cause another nearby device to enter the extended peer device advertising state.

[0048] In some embodiments, an electronic device streaming AV content may detect an interruption event (e.g., an incoming phone call, text message, email, etc.) and, in response to such detection, may temporarily suspend the transmission of images to the connected electronic device for the duration of the interruption event and optionally resume the transmission of images after the detected interruption event has ended. In other embodiments, for example, if the user needs to access the electronic device's functions for any other reason, there may be a user interface option to positively indicate the user's desire to pause / interrupt the AV content sharing session.

[0049] An exemplary process flow diagram illustrating the establishment, maintenance, and termination of secure connections between electronic devices.

[0050] Next, referring to Figure 3, an exemplary process flow diagram 300 is shown for establishing and maintaining a secure connection between electronic devices for streaming audio and video data, according to one or more embodiments. In particular, Figure 300 details the processes and various interactions between an exemplary shared electronic device 202 and an exemplary personal electronic device 220 involved in establishing and maintaining a secure streaming session.

[0051] First, we move to a shared electronic device 202, which may include a client application 302 used by a shared electronic device 202 to search for, connect to, and stream AV content from a personal electronic device such as a personal electronic device 220. The shared electronic device 202 may further include an external camera agent 206 and connection management software 208 (which may include a session management daemon), as described above with reference to Figure 2. Next, we move to a personal electronic device 220, which may include connection management software 222 (which may include a session management daemon), camera management software 224, and camera streaming user interface software 226, as described above with reference to Figure 2.

[0052] An exemplary process flow diagram 300 can begin in step (1), where a user of the shared electronic device 202 requests to launch an application for establishing and maintaining a secure connection between electronic devices to stream audio and video data. Next, in step (2), the client application 302 can request a list of discoverable electronic devices (e.g., personal electronic devices) from the connection management software 208. Next, in step (3), the connection management software 208 from the shared electronic device 202 can verify the presence of the personal electronic device 220 via the connection management software 222 of the personal electronic device 220. In step (4), the connection management software 222 may register the shared electronic device 202 as a connected device.

[0053] Next, in step (5), the connection management software 208 can send a callback message to the external camera agent 206 to confirm that the electronic device has been selected. Then, in step (6), the external camera agent 206 can initiate a session lifecycle with the connection management software 222 on the personal electronic device 220, and in step (7), the connection management software 222 can confirm the session lifecycle as a valid connection.

[0054] In step (8), the shared electronic device 202 and the personal electronic device 220 can formally set up an advertising broadcast / receive framework used to monitor and maintain connected sessions, such as sessions that monitor and maintain secure streaming sessions between devices that are not necessarily associated with the same user account, leveraging the sessions described herein, for example, BLE advertising packets and protocols.

[0055] In step (9), the connection management software 222 can transfer session lifecycle information to the camera management software 224, which in turn can launch the camera streaming user interface software 226 on the personal electronic device 220 in step (10).

[0056] In the first example, the camera management software 224 of the personal electronic device 220 can request a UI state from the camera streaming user interface software 226. In this first example, the camera management software 224 can, in step (12), send a message to the external camera agent 206 of the shared electronic device 202 indicating a desire to end the current session lifecycle. At that point, the personal electronic device 220 is still connected to the shared electronic device 202, but is not currently streaming content to the shared electronic device 202 (304) (for example, a user of the shared electronic device 202 or the personal electronic device 220 may have changed applications, paused the streaming session, or temporarily entered a settings menu on the device). This state allows the devices to quickly re-establish content streaming whenever either device is ready to do so.

[0057] In the second example, in step (13), the client application 302 of the shared electronic device 202 may indicate a request to resume streaming content on the personal electronic device 220. Therefore, in step (14), the external camera agent 206 may send a message to the camera management software 224 to confirm that the personal electronic device 220 should resume streaming data. In step (15), the camera management software 224 may send a message to the camera streaming user interface software 226 indicating a request to update the UI to reflect that the streaming session has resumed and is active again. At that point, the personal electronic device 220 is connected to the shared electronic device 202 and is currently streaming content to the shared electronic device 202 (306).

[0058] In the third example, in step (16), the camera streaming user interface software 226 may receive an indication of a request to pause the streaming of content to the shared electronic device 202. Thus, in step (17), the camera management software 224 may send a message to the camera streaming user interface software 226 indicating a request to update the UI to reflect that the streaming session has been paused. At that point, the personal electronic device 220 is connected to the shared electronic device 202 (308), but the streaming of content to the shared electronic device 202 is (temporarily) paused.

[0059] In the fourth example, in step (18), the camera streaming user interface software 226 may receive an indication from the shared electronic device 202 that it wants to resume the paused content stream. Thus, in step (19), the camera management software 224 may send a message to the camera streaming user interface software 226 indicating a request to update the UI to reflect that the streaming session has resumed. At that point, the personal electronic device 220 is connected to the shared electronic device 202 and is streaming content to the shared electronic device 202 again (310).

[0060] In the fifth example, in step (20), the client application 302 of the shared electronic device 202 may indicate a request to stop streaming content on the personal electronic device 220. Therefore, in step (21), the external camera agent 206 may send a message to the camera management software 224 to terminate the streaming session. In step (22), the camera management software 224 may send a message to the camera streaming user interface software 226 indicating a request to update the UI to reflect that the streaming session has ended. At that point, the personal electronic device 220 is connected to the shared electronic device 202, but is not currently streaming content to the shared electronic device 202 (312).

[0061] Next, referring to Figure 4, an exemplary process flow diagram 400 is shown, which illustrates the termination of a secure connection between electronic devices for streaming audio and video data by one or more embodiments. As can be understood, Figure 4 and the exemplary process flow diagram 400 continue the process described above with respect to Figure 3 and the exemplary process flow diagram 300, and include similar components such as the client application 302, the external camera agent 206, connection management software 208, connection management software 222, camera management software 224, and camera streaming user interface software 226, while focusing on the termination of the connection session (in contrast to the establishment and maintenance described above with respect to Figure 3).

[0062] In the first exemplary discard process flow 402 (where the shared electronic device 202 is a lost device), flow 402 may begin from step (23) when the connection management software 222 detects that a response session status indicator has not been received by the personal electronic device 220 for a longer period of time than a predetermined amount, and / or that an inaccurate or invalid session status indicator has been received. This may indicate that the shared electronic device 202 has gone out of range, been restarted, or experienced a crash in one or more of its associated connection management software processes. Thus, flow 402 may proceed to step (24) and discard the connection session and camera streaming user interface software 226.

[0063] In the second exemplary disposal process flow 404 (where the personal electronic device 220 is the lost device), flow 404 may begin at step (25), when the connection management software 208 detects that no response session status indicator has been received on the shared electronic device 202 for a longer period than a predetermined amount of time. This may indicate that the personal electronic device 220 has gone out of range, been restarted, or experienced a crash in one or more of its associated connection management software processes. Thus, flow 404 may proceed to step (26) to formally unpublish / disconnect the personal electronic device 220, and optionally, in step (27), inform the user of the shared electronic device 202 that the connection with the personal electronic device 220 has been interrupted, allowing the user of the shared electronic device 202 to check if there are any other personal electronic devices in proximity that are ready to form a secure streaming connection.

[0064] In a third exemplary termination process flow 406 (when the personal electronic device 220 initiates a termination event), flow 406 may begin in step (28) when the user interface software 226 sends an indication to the camera management software 224 that the user of the personal electronic device 220 has initiated a termination event. The camera management software 224 may then proceed in step (29) by terminating the session at an appropriate time so as not to cause undue disruption to any electronic device. Finally, in step (30), a connection may be formed between the camera management software 224 and the external camera agent 206, and as a result, the personal electronic device 220 may indicate that the session has been positively terminated.

[0065] In the fourth exemplary discard process flow 408 (when the shared electronic device 202 initiates a disconnection event), flow 408 can begin in step (31) when the connection management software 208 receives an indication that the user of the shared electronic device 202 has initiated a disconnection event (for example, by the user selecting a disconnection option in the device UI, switching to connect to a different personal electronic device, “handing off” the video conferencing session from a different personal electronic device, entering sleep / standby, etc.). Next, in step (32), the connection management software 208 can notify the external camera agent 206 that the personal electronic device 220 is no longer connected. Finally, in step (33), the external camera agent 206 can establish a connection with the camera management software 224, and as a result, it can indicate that the session on the shared electronic device 202 has been positively disconnected. In step (34), the camera management software 224 can proceed to discard the session on the personal electronic device 220.

[0066] At the end of any of the exemplary disposal process flows shown in Figure 6, the final result is that the content streaming UI enters state (410), in which state the content streaming UI no longer indicates that there is an active streaming or sharing session on the personal electronic device 220.

[0067] Exemplary extended BLE advertising packet

[0068] As mentioned above, BLUETOOTH® Low Energy, or "BLE," is a 2.4 GHz industrial, scientific, and medical (ISM) band wireless communication standard first introduced in 2010. BLE operates on 40 physical channels, three of which are dedicated to the discovery mechanism, also known as "advertisement," while the remaining 37 channels are used for data transmission. The advertisement discovery mechanism in BLE allows BLE-enabled devices to advertise their presence and characteristics to nearby devices. This relies on the broadcasting of advertisement packets on BLE's three advertisement channels.

[0069] Figure 5 shows exemplary extended BLE advertising packets 500 according to one or more embodiments. The exemplary packets 500 are shown organized from left to right, starting with the most significant bit (MSB) and ending with the least significant bit (LSB). According to some embodiments, packets 500 may begin with a typical header field (502), which may be, for example, a two-byte header field and may be used to store the length of the payload field (510) and / or other relevant metadata of packets 500.

[0070] The payload field (510) includes the advertising device address (504) and up to 31 bytes of additional data, which are organized into an advertisement data (AD) structure that carries information about the advertising device. The AD structure (e.g., the manufacturer-specific data AD structure 508 shown in Figure 5) may consist of a one-byte field (512) indicating the length of the AD (excluding itself), followed by a one-byte field (514) specifying the type of the AD, and finally a sequence of up to 29 bytes of data.

[0071] As mentioned above, one AD structure type called Manufacturer-Specific Data (represented by code 0xFF, as shown in 514) is dedicated to the transport of data for custom applications defined by the manufacturer. The Manufacturer-Specific Data AD structure begins with a corporate identifier (e.g., 0x004C, as shown in 516, for Apple Inc.) and one or more messages 5180-518 N Message 518 may be followed by (for example, a message relating to the content sharing technology described herein). Message 518 may also be stored in a Type-Length-Value (TLV) format, namely a Type field (520) followed by a Length field (522) indicating the length of the message, and then a Data field (524) storing the actual data of the particular message 518. [Flags (506) are an optional AD structure for storing various program flag values ​​that are not required or used by all BLE protocols.]

[0072] In some embodiments, it may be beneficial to store session state indicator information within one or more bits transmitted as part of a BLE advertising packet, for example, within a manufacturer-specific data AD structure. The advantages of such a technique may include that the session state indicator is always advertised (i.e., at a regular cadence), meaning that the identification information transmitted in the advertising packet handles situations where the BLE addresses of one or more devices cycle over time, and / or that no additional power or performance costs are imposed on the electronic device because the BLE advertising protocol is already being run by the electronic device.

[0073] One potential drawback of using a 1-bit or 2-bit broadcast session status indicator is that it may not convey enough information to indicate which devices a broadcast device is currently in an active streaming session with. However, this drawback can be overcome if connection management software is configured to always check when a particular electronic device enters (or leaves) a session. In this way, the device knows that, as long as the session is "active," the device in the session is the same device that was indicated when the session first started.

[0074] Exemplary method for establishing and maintaining a secure connection between devices for streaming audio and video data.

[0075] Figure 6 is a flowchart 600 illustrating a method for establishing and maintaining a secure connection between electronic devices for streaming audio and video data (from the perspective of a shared electronic device) according to various embodiments. First, in step 602, method 600 can enter a first state of the peer device advertising protocol in the first electronic device. Next, in step 604, method 600 can detect the presence of a second electronic device adjacent to the first electronic device while in the first state of the peer device advertising protocol.

[0076] Next, in step 606, method 600 may initiate a first session between the first electronic device and the second electronic device via a secure wireless peer-to-peer connection protocol.

[0077] Next, in step 608, method 600 may establish a software process (e.g., a daemon or other background process) in the first electronic device that periodically broadcasts a first session status indicator via a peer device advertising protocol, the first session status indicator indicating whether the first session is currently active.

[0078] Next, in step 610, the method 600 may, in the software process of the first electronic device, receive a second session status indicator from the second electronic device via a peer device advertising protocol, the second session status indicator indicating whether the first session is currently active.

[0079] Finally, in step 612, method 600 can maintain a first session in the first electronic device until a session disconnection criterion is met, the session disconnection criterion being at least partially based on a first session state indicator or a second session state indicator. For example, according to some embodiments, a session may be maintained until either the first session state indicator or the second session state indicator times out or indicates that the device is no longer in a secure connection session (e.g., having a value of "0" if a 1-bit session state indicator is used). As described above, during some sessions, various forms of AV content may be streamed from the first electronic device to the second electronic device for display and / or playback. In some implementations, a temporary pause or interruption (whether intentional or unintentional) of AV content streaming may cause the streaming criterion to be not met, meaning, for example, that each device involved in the streaming session may continue broadcasting a session state indicator indicating that the session is still active. On the other hand, examples of situations in which the session disconnection criterion can be met are described above with reference to Figure 4.

[0080] Next, referring to Figure 7, a flowchart 700 is shown illustrating a method for "handing off" an active communication session (e.g., an audiovisual communication session) from a first electronic device (e.g., a personal electronic device) to a second electronic device (e.g., a shared electronic device) according to various embodiments, and then maintaining a secure connection between the devices. First, in step 702, method 700 can initiate a first video conferencing session on the first electronic device. Next, in step 704, method 700 can cause the first electronic device to enter a first state of the peer device advertising protocol.

[0081] Next, in step 706, method 700 can detect the presence of a second electronic device adjacent to a first electronic device while in the first state of the peer device advertising protocol.

[0082] Next, in step 708, method 700 can initiate a first session by extending the first video conferencing session to a second electronic device via a secure wireless peer-to-peer connection protocol. As described above, according to some embodiments, “handing off” an active communication session can be more accurately described as the act of “extending” an active communication session to another device. For example, according to method 700, a personal electronic device currently involved in an active communication session can continuously extend its identification information (and its use of one or more image capture devices) to another (e.g., shared) electronic device such as a “smart TV”. In such embodiments, the user’s personal electronic device may still be organizing an active communication session (and may even remain an active device in the active communication session) even after “extending” its identification information to the shared electronic device.

[0083] Next, in step 710, method 700 may establish a software process in an electronic device that periodically broadcasts a first session status indicator via a peer device advertising protocol, the first session status indicator indicating whether the first session is currently active.

[0084] Next, in step 712, the method 700 may receive a second session status indicator from a second electronic device via a peer device advertising protocol in the software process of the electronic device, the second session status indicator indicating whether the first session is currently active.

[0085] Finally, in step 714, method 700 may maintain the first session until a session disconnection criterion is met, which is at least partially based on a first session status indicator or a second session status indicator. For example, according to some embodiments, a session may be maintained until either the first session status indicator or the second session status indicator times out or indicates that the device is no longer in a secure connection session (e.g., having a value of "0" if a 1-bit session status indicator is used). As described above, during some sessions, various forms of AV content may be streamed from the first electronic device to the second electronic device for display and / or playback.

[0086] To make it clear, for example with reference to Figures 6 and 7, the various methods described herein may be performed by an electronic device, for example, by being initiated by an application (or “App”) running on the device and / or the device’s native operating system (OS). For example, an App running on a device may initiate or perform all or at least some of the steps in the Method, while calling the device’s OS to perform other steps in the Method. Similarly, the device’s OS may receive API calls from an App or elsewhere, process / execute those calls, and allow the device to perform the Method.

[0087] Exemplary electronic devices

[0088] Referring here to Figure 8, a simplified functional block diagram of an exemplary programmable electronic device 800 according to one embodiment is shown. The electronic device 800 may be, for example, a mobile phone, a personal media device, a portable camera, or a tablet, notebook, or desktop computer system. As shown, the electronic device 800 may include a processor 805, a display 810, a user interface 815, graphics hardware 820, device sensors 825 (e.g., proximity sensor / ambient light sensor, accelerometer, inertial measurement unit, and / or gyroscope), a microphone 830, an audio codec 835, a speaker 840, a communication circuit 845, an image capture device 850, a video codec 855, a memory 860, a storage device 865, and a communication bus 870, which may include, for example, multiple camera units / optical image sensors having different characteristics or capabilities (e.g., still image stabilization (SIS), HDR, OIS system, optical zoom, digital zoom, etc.).

[0089] The processor 805 can execute instructions necessary to perform or control the operation of numerous functions performed by the electronic device 800 (e.g., the generation, processing, and / or streaming of audio and / or video data according to various embodiments described herein). The processor 805 can, for example, drive the display 810 and receive user input from the user interface 815. The user interface 815 can take various forms such as buttons, keypads, dials, click wheels, keyboards, display screens, and / or touch screens. The user interface 815 can, for example, be a conduit through which a user can view a captured video stream and / or indicate a specific image frame that the user wants to capture (e.g., by clicking a physical or virtual button at the moment a desired image frame appears on the device's display screen). In one embodiment, the display 810 can display a video stream when the video stream is being captured while the processor 805 and / or graphics hardware 820 and / or image capture circuitry are simultaneously generating and storing the video stream in memory 860 and / or storage device 865. The processor 805 may be a system-on-a-chip (SOC), such as those found in mobile devices, and may include one or more dedicated graphics processing units (GPUs). The processor 805 may be based on a reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architecture, or any other suitable architecture, and may include one or more processing cores. The graphics hardware 820 may be dedicated computing hardware for processing graphics and / or for assisting the processor 805 in performing computational tasks.In one embodiment, the graphics hardware 820 may include one or more programmable graphics processing units (GPUs) and / or one or more specialized SOCs, such as Apple's Neural Engine processing core, which are specially designed to perform neural network and machine learning computations (convolutions) in a more energy-efficient manner than either the main device's central processing unit (CPU) or a typical GPU.

[0090] The image capture device 850 may comprise one or more camera units configured to capture images, which may be processed, for example, to produce framed and / or distortion-corrected versions of the captured images as described herein. The image capture device 850 may comprise two (or more) lens assemblies 880A and 880B, each having a distinct focal length. For example, lens assembly 880A may have a shorter focal length than lens assembly 880B. Each lens assembly may have a distinct associated sensor element, e.g., sensor element 890A / 890B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture device 850 can capture still images and / or video images. The output from the image capture device 850 may be processed, at least in part, by a video codec 855 and / or processor 805 and / or graphics hardware 820, and / or a dedicated image processing unit or image signal processor incorporated within the image capture device 850. The captured images can be stored in memory 860 and / or storage device 865.

[0091] Memory 860 may include one or more different types of media used by the processor 805, graphics hardware 820, and image capture device 850 to perform the functions of the device. For example, memory 860 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 865 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and other appropriate data. Storage device 865 may include one or more non-temporary storage media, including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and digital video discs (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). Memory 860 and storage device 865 may be organized into one or more modules and used to hold computer program instructions or code written in any desired computer programming language. For example, when executed by processor 805, such computer program code can perform one or more of the methods or processes described herein. Power supply 875 may include a rechargeable battery (e.g., a lithium-ion battery) or other electrical connection to a power supply, such as a mains power supply, used to manage and / or provide power to the electronic components and associated circuits of electronic device 800.

[0092] It should be understood that the above description is illustrative and not limiting. For example, the embodiments described above can be used in combination with each other. A number of other embodiments will become apparent to those skilled in the art when considering the above description. Therefore, the scope of the present invention should be determined by referring to the appended claims and the entire scope of equivalents given to such claims.

Claims

1. It is a method, In the first electronic device, entering the first state of the peer device advertising protocol, While the peer device advertising protocol is in the first state, the presence of a second electronic device adjacent to the first electronic device is detected, Initiating a first session between the first electronic device and the second electronic device via a secure wireless peer-to-peer connection protocol, In the first electronic device, a software process is established via the peer device advertising protocol that periodically broadcasts a first session status indicator, the first session status indicator indicating whether the first session is currently active. In the software process of the first electronic device, a second session status indicator is received from the second electronic device via the peer device advertising protocol, wherein the second session status indicator indicates whether the first session is currently active. A method for the first electronic device, comprising: maintaining the first session until a session termination criterion is met, wherein the session termination criterion is at least partially based on the first session state indicator or the second session state indicator.

2. The aforementioned session disconnection criteria are the following conditions, namely: (a) The first electronic device does not receive a second session status indicator indicating that the first session is currently active for a longer period than a first predetermined amount of time. (b) The first electronic device does not broadcast a first session status indicator indicating that the first session is currently active for a longer period than a second predetermined amount of time. (c) The first electronic device initiates a disconnection event for the first session, The method according to claim 1, which is satisfied when at least one of the following occurs: (d) the first electronic device receives an indication of a first session disconnection event from the second electronic device.

3. The second electronic device comprises the first image capture device, and the method is The method according to claim 1, further comprising, during the first session, the first electronic device receiving one or more images captured in the second electronic device by the first image capture device.

4. The method according to claim 3, wherein the one or more images received by the first electronic device are part of a video image stream captured by the second electronic device, and interruption or pause of the video image stream by the second electronic device does not satisfy the session disconnection criterion.

5. The method according to claim 3, wherein two or more applications running on the first electronic device are configured to utilize one or more images received on the first electronic device.

6. The method according to claim 3, wherein the first electronic device and the second electronic device are required to be located within a threshold distance of each other during the reception of one or more images.

7. The method according to claim 1, wherein the peer device advertising protocol includes a Bluetooth Low Energy (BLE) advertising protocol.

8. The method according to claim 7, wherein the first session status indicator is broadcast within the manufacturer-specific data advertisement data (AD) structure of the BLE advertisement packet.

9. The method according to claim 7, wherein the first session status indicator is broadcast in a BLE advertisement packet that includes an authentication tag (autotag) that identifies the first electronic device.

10. The method according to claim 7, wherein the first session state indicator comprises a 1-bit value or a 2-bit value.

11. The method according to claim 1, wherein the software process is configured to periodically broadcast the first session status indicator even when the display screen of the first electronic device is turned off.

12. The method according to claim 1, wherein the first electronic device is configured to enter the first state of the peer device advertising protocol in response to the activation of the device discovery user interface flow.

13. The method according to claim 1, wherein the secure wireless peer-to-peer connection protocol is a protocol different from the peer device advertising protocol.

14. The method according to claim 1, wherein the first electronic device and the second electronic device are associated with different users or different user accounts.

15. The method according to claim 1, wherein the first electronic device is not associated with any single user account.

16. The method according to claim 1, wherein initiating the first session between the first electronic device and the second electronic device includes using a QR code (registered trademark) or a PIN pairing process.

17. It is an electronic device, Memory and A first image capture device and The system comprises one or more processors operably coupled to the memory, wherein the one or more processors are connected to the one or more processors, In the aforementioned electronic device, a first video conference session is initiated. In the aforementioned electronic device, enter the first state of the peer device advertising protocol. While the peer device advertising protocol is in the first state, the presence of a second electronic device adjacent to the electronic device is detected. The first video conferencing session is initiated by extending the first video conferencing session to the second electronic device via a secure wireless peer-to-peer connection protocol. In the electronic device, a software process is established that periodically broadcasts a first session status indicator via the peer device advertising protocol, and the first session status indicator indicates whether the first session is currently active. In the software process of the electronic device, a second session status indicator is received from the second electronic device via the peer device advertising protocol, and the second session status indicator indicates whether the first session is currently active. An electronic device configured to maintain the first session until a session disconnection criterion is met, and to execute an instruction where the session disconnection criterion is at least partially based on the first session state indicator or the second session state indicator.

18. The aforementioned session disconnection criteria are the following conditions, namely: (a) The electronic device does not receive a second session status indicator indicating that the first session is currently active for a longer period of time than a first predetermined amount of time. (b) The electronic device does not broadcast a first session status indicator indicating that the first session is currently active for a longer period than a second predetermined amount of time. (c) The electronic device initiates a disconnection event for the first session, or The electronic device according to claim 17, which is satisfied when at least one of the following occurs: (d) the electronic device receives an indication of a first session disconnection event from the second electronic device.

19. The one or more processors, The electronic device according to claim 17, further configured to execute a command causing the second electronic device to transmit one or more images captured by the first image capture device during the first session.

20. A non-temporary computer-readable medium containing computer-readable instructions, wherein the computer-readable instructions are processed by one or more processors. In the first electronic device, enter the first state of the peer device advertising protocol, While the peer device advertising protocol is in the first state, the presence of a second electronic device adjacent to the first electronic device is detected. A first session is initiated between the first electronic device and the second electronic device via a secure wireless peer-to-peer connection protocol. During the first session, the first electronic device receives one or more images from a video image stream captured by the first image capture device of the second electronic device. In the first electronic device, a software process is established that periodically broadcasts a first session status indicator via the peer device advertising protocol, and the first session status indicator indicates whether the first session is currently active. In the software process of the first electronic device, a second session status indicator is received from the second electronic device via the peer device advertising protocol, and the second session status indicator indicates whether the first session is currently active. A non-temporary computer-readable medium in the first electronic device, which maintains the first session until a session disconnection criterion is met, and which is capable of interrupting or pausing the video image stream in the second electronic device, at least in part, based on the first session status indicator or the second session status indicator, such that the session disconnection criterion is not met.

Images