Enable communication actions in a conference session

Abstract

Embodiments of the present disclosure relate to enabling communication actions in a conference session. This specification describes an apparatus, a method, and a computer program for communicating with a plurality of remote devices in a conference session, outputting a user interface on a display, the user interface including a plurality of display portions respectively associated with the plurality of remote devices, and receiving a user selection of one or more of the remote devices. Based on the received user selection, the apparatus may display, on a linked user device linked to the apparatus, the display portion associated with the selected one or more remote devices, and enable one or more communication actions for use with the selected one or more remote devices.

Description

Technical Field

[0001] Example embodiments relate to apparatuses, methods, and computer programs for enabling communication actions in a conference session, such as a video conference session. Background Art

[0002] The conference session described herein can be a communication session established between multiple devices, and thus can be a communication session established between users or parties associated with the respective devices. At a point in time, the conference session is invoked or established, and one or more communication channels can be established between the devices (possibly via a conference server). Thereafter, audio and / or video data can be sent between the devices through one or more established communication channels, and the audio and / or video data is part of a conference session that can be managed by the conference server. For example, in a conference session including first, second, and third devices, the audio and / or video data sent from the first device can be received and processed for output by each of the second and third devices in real time or near real time, and similarly, the audio and / or video data sent from, for example, a linked user device can be received and processed for output by each of the first and third devices in real time or near real time.

[0003] The conference session can be a pure audio conference session or a video conference session. In a video conference session, video data can be captured by one or more cameras of a particular device and sent through one or more established communication channels to other devices, thereby displaying the video data as a video feed. The video data can be accompanied by audio data captured by one or more microphones of a particular device. Summary of the Invention

[0004] The protection scope of the embodiments of the present invention is set forth by the independent claims. Embodiments and features (if any) described in this specification that do not fall within the scope of the independent claims should be construed as examples to facilitate understanding of the embodiments of the present invention.

[0005] According to a first aspect, this specification describes an apparatus including components for the following operations:

[0006] Communicating with a plurality of remote devices in a conference session; outputting a user interface on a display, the user interface including a plurality of display portions correspondingly associated with the plurality of remote devices; receiving a user selection of one or more of the remote devices; and based on the received user selection: causing a display portion associated with the selected one or more remote devices to be displayed on a linked user device linked to the apparatus, and enabling one or more communication actions for use with the selected one or more remote devices.

[0007] A conference session can be a video conference session in which the output component is configured to output, on a display, a user interface including a plurality of video feed display portions respectively associated with a plurality of remote devices, and wherein, based on a received user selection, the apparatus can be configured to display, on a linked user device, the video feed display portions associated with the selected one or more remote devices.

[0008] The enabling component can be configured to enable audio and / or video communication actions with the selected one or more remote devices via a dedicated communication channel that is not part of the conference session.

[0009] The implementing component can be configured to establish a dedicated communication channel in response to a received user selection.

[0010] The apparatus can also include components for: detecting a user input for initiating audio and / or video communication via the dedicated communication channel, and based on detecting the user input, causing audio and / or video data to be captured and transmitted via the dedicated communication channel.

[0011] The apparatus can be configured to, based on detecting a user input for initiating audio communication via the dedicated communication channel, cause audio data to be captured by one or more microphones of the linked user device for transmission via the dedicated communication channel, and disable the transmission of audio data as part of the conference session.

[0012] Based on detecting a user input for initiating audio communication via the dedicated communication channel, the apparatus can also be configured to pause or disable the transmission of video data from the apparatus as part of the conference session.

[0013] The apparatus can also include a component for determining the movement of the linked user device, and wherein the detecting component can be configured to detect a user input for initiating audio communication via the dedicated communication channel based on the determined movement of the linked user device corresponding to a predetermined gesture.

[0014] The apparatus can also include a component for determining the location of the linked user device, and wherein the detecting component can be configured to detect a user input for initiating audio communication via the dedicated communication channel based on the determined location of the linked user device corresponding to a predetermined location.

[0015] The detecting component can be configured to detect a user input for initiating audio communication via the dedicated communication channel based on the determined location of the linked user device being within a predetermined distance of the user's mouth.

[0016] Based on detecting a user input for initiating video communication via the dedicated communication channel, the apparatus can be configured to cause video data to be captured by one or more cameras of the linked user device for transmission via the dedicated communication channel.

[0017] The apparatus may also be configured to cause audio data to be captured by one or more microphones of the apparatus for transmission over a dedicated communication channel.

[0018] The apparatus may include a microphone array and may be configured to determine the position of a linked user device relative to the apparatus and, based on the determined relative position, apply beamforming to audio data captured by the microphone array.

[0019] In response to receiving a user selection of a plurality of remote devices, the apparatus may be configured to cause a display portion corresponding to the selected plurality of remote devices to be displayed simultaneously or sequentially on the linked user device. The apparatus may be configured to cause the display portion to be displayed sequentially on the linked user device by first causing a first display portion associated with a first remote device among the selected remote devices to be displayed and subsequently, in response to user input received by the linked user device, causing a second display portion associated with a second remote device among the selected remote devices to be displayed.

[0020] The apparatus may be configured to enable one or more conferencing actions only for the selected remote devices for which the associated display portion is being displayed at the current time.

[0021] The enabling component may be configured to output, on a user interface of the apparatus, one or more interactive user interface elements corresponding to one or more conferencing actions of the selected one or more remote devices.

[0022] When a display portion associated with the selected one or more remote devices is being displayed on the linked user device, the apparatus may be configured to remove the display portion associated with the corresponding selected one or more remote devices from the user interface of the apparatus.

[0023] The apparatus may also be configured to: receive user input for a supplementary selection of a remote device; based on the received user input for the supplementary selection: cause the currently selected one or more remote devices to be deselected, thereby removing their corresponding display portions from the linked user device and disabling one or more implemented actions for the currently selected remote devices; and cause a currently unselected remote device to be selected, thereby causing its corresponding display portion to be displayed on the linked user device and enabling one or more actions for the currently unselected remote device.

[0024] The receiving component may be configured to receive a user selection of one or more remote devices by receiving a touch or swipe input on one or more display portions respectively associated with the one or more remote devices.

[0025] The receiving component may be configured to receive a user selection of one or more remote devices in a remote device by receiving a swipe input that is substantially in the direction of the linked user device.

[0026] The linked user device may be a wearable device. For example, the wearable device may be a smartwatch. For example, the device may be a smart phone or a tablet computer.

[0027] According to a second aspect, this specification describes a method, including:

[0028] Communicating with a plurality of remote devices in a conference session through a device; outputting a user interface on a display, the user interface including a plurality of display portions respectively associated with the plurality of remote devices; receiving a user selection of one or more of the remote devices; and based on the received user selection: causing a display portion associated with the selected one or more remote devices to be displayed on a linked user device linked to the device, and enabling one or more communication actions for use with the selected one or more remote devices.

[0029] The conference session may be a video conference session, in which outputting includes outputting a user interface including a plurality of video feed display portions respectively associated with the plurality of remote devices on a display, and the method includes, based on the received user selection, displaying a video feed display portion associated with the selected one or more remote devices on the linked user device.

[0030] The enabling operation may include enabling audio and / or video communication actions with the selected one or more remote devices through a dedicated communication channel that is not part of the conference session.

[0031] The enabling operation may establish a dedicated communication channel in response to the received user selection.

[0032] The method may include components for the following operations: detecting a user input for initiating audio and / or video communication through the dedicated communication channel, and based on detecting the user input, causing audio and / or video data to be captured and transmitted through the dedicated communication channel.

[0033] The method may include, based on detecting a user input for initiating audio communication through the dedicated communication channel, causing audio data to be captured by one or more microphones of the user device for transmission through the dedicated communication channel, and disabling the transmission of audio data as part of the conference session.

[0034] Based on detecting a user input for initiating audio communication through the dedicated communication channel, the method may include pausing or disabling the transmission of audio data as part of the conference session.

[0035] The method may further include determining a movement of the linked user device, and detecting a user input for initiating an audio communication via a dedicated communication channel based on the determined movement of the linked user device corresponding to a predetermined gesture.

[0036] The method may further include determining a location of the linked user device, and detecting a user input for initiating an audio communication via a dedicated communication channel based on the determined location of the linked user device corresponding to a predetermined location.

[0037] The method may include detecting a user input for initiating an audio communication via a dedicated communication channel based on the determined location of the linked user device within a predetermined distance of the user's mouth.

[0038] Based on detecting a user input for initiating a video communication via a dedicated communication channel, the method may include causing video data to be captured by one or more cameras of the linked user device for transmission via the dedicated communication channel.

[0039] The method may include causing audio data to be captured by one or more microphones of the device for transmission via the dedicated communication channel.

[0040] The method may include determining a location of the linked user device relative to the device, and applying beamforming to the audio data captured by the microphone array based on the determined relative location.

[0041] In response to receiving a user selection of a plurality of remote devices, the method may include simultaneously or sequentially displaying on the linked user device display portions respectively associated with the plurality of selected remote devices. The method may cause the display portions to be sequentially displayed on the linked user device by first causing a first display portion associated with a first remote device among the selected remote devices to be displayed, and subsequently, in response to a user input received by the linked user device, causing a second display portion associated with a second remote device among the selected remote devices to be displayed.

[0042] The method may enable one or more conferencing actions only for a selected remote device for which an associated display portion is being displayed at the current time.

[0043] The method may output one or more interactive user interface elements corresponding to one or more conferencing actions of a selected one or more remote devices on a user interface of the device.

[0044] When a display portion associated with a selected one or more remote devices is being displayed on the linked user device, the method may remove from the user interface of the device the display portion associated with the corresponding selected one or more remote devices.

[0045] The method may further include: receiving user input for a supplementary selection of a remote device; based on the received user input for the supplementary selection: causing deselection of one or more currently selected remote devices, thereby removing their corresponding display portions from the linked user device and disabling one or more enabled actions for the currently selected remote devices; and causing selection of one or more currently unselected remote devices, thereby causing their corresponding display portions to be displayed on the linked user device and enabling one or more actions for the currently unselected remote devices.

[0046] User selection of one or more remote devices may include receiving a touch or swipe input on one or more display portions respectively associated with the one or more remote devices.

[0047] User selection of one or more of the remote devices may include receiving a swipe input substantially in the direction of the linked user device.

[0048] The linked user device may be a wearable device. For example, the wearable device may be a smartwatch.

[0049] The method may be performed by a device such as a user device, such as a smartphone or a tablet computer.

[0050] According to a third aspect, this specification describes a computer program including instructions for causing a device to perform at least the following operations: communicate with a plurality of remote devices in a conference session; output a user interface on a display, the user interface including a plurality of display portions respectively associated with the plurality of remote devices; receive user selection of one or more of the remote devices; and based on the received user selection: cause a display portion associated with the selected one or more remote devices to be displayed on a linked user device linked to the device and enable one or more communication actions for use with the selected one or more remote devices.

[0051] The example embodiments may also provide any feature of the second aspect.

[0052] According to a fourth aspect, this specification describes a computer-readable medium (such as a non-transitory computer-readable medium) storing program instructions for performing at least the following operations: communicate with a plurality of remote devices in a conference session through a device; output a user interface on a display, the user interface including a plurality of display portions respectively associated with the plurality of remote devices; receive user selection of one or more of the remote devices; and based on the received user selection: cause a display portion associated with the selected one or more remote devices to be displayed on a linked user device linked to the device and enable one or more communication actions for use with the selected one or more remote devices.

[0053] According to a fifth aspect, the present specification describes an apparatus, comprising: at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, causes the apparatus to: communicate with a plurality of remote devices in a conference session; output a user interface on a display, the user interface including a plurality of display portions respectively associated with the plurality of remote devices; receive a user selection of one or more of the remote devices; and based on the received user selection: cause a display portion associated with the selected one or more remote devices to be displayed on a linked user device linked to the apparatus, and enable one or more communication actions for use with the selected one or more remote devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Example embodiments will now be described by way of non-limiting examples with reference to the accompanying drawings, in which:

[0055] Figure 1 is a plan view of a system including a first user device and a linked user device;

[0056] Figure 2 is a block diagram of a video conferencing system including Figure 1 the system according to an example embodiment;

[0057] Figure 3 is a plan view of a first user device showing a first user interface;

[0058] Figure 4 is a plan view of a first user device showing a second alternative user interface;

[0059] Figure 5 is a flowchart indicating processing operations according to an example embodiment;

[0060] Figure 6 is a plan view of a first user device and a linked user device according to an example embodiment, indicating that the first user device has received user input;

[0061] Figure 7 is a plan view of a first user device and a linked user device according to an example embodiment, indicating the result of receiving user input;

[0062] Figure 8 is a block diagram of a video conferencing system including a dedicated communication channel according to an example embodiment; Figure 2 the video conferencing system;

[0063] Figure 9 is a perspective view of a first user using a first user device and a linked user device according to an example embodiment;

[0064] Figure 10is a perspective view of a first user using a first user device and a linked user device according to another exemplary embodiment;

[0065] Figure 11 is a plan view of a linked user device showing a user interface including a first display portion and a second display portion according to an exemplary embodiment;

[0066] Figure 12A and Figure 12B is a plan view of a linked user device showing a user interface including a first display portion and a subsequently displayed second display portion according to an exemplary embodiment;

[0067] Figure 13 is a device that can be configured to perform one or more processing operations according to one or more exemplary embodiments; and

[0068] Figure 14 is a non-transitory medium that can store computer program code for causing Figure 13 the device according to one or more exemplary embodiments to perform one or more processing operations. DETAILED DESCRIPTION

[0069] Exemplary embodiments relate to devices, methods, and computer programs for enabling communication actions, for example, in a conference session such as a video conference session.

[0070] As described herein, a conference session can include one or more communication channels established between two or more devices via a communication network and, in turn, between users or parties associated with the respective devices. A conference session can include, for example, one or more channels established between two or more devices that are participants in the conference session, and their respective users can be collectively referred to as participants.

[0071] A device or user can be a primary participant or an invited participant. One or more communication channels can be established when a conference session is established to provide a multicast data feed from a given participant to each of the other participants, typically in real time or near real time. One or more communication channels can be two-way communication channels.

[0072] The device may include a user device, which is a device operable by one or more users and configured to send and receive data via a communication network. The term "user device" will be used hereinafter. The user device may include processing functions for executing one or more applications (such as a conferencing application). The user device may also include one or more input modules and one or more output modules. For example, the user device may include one or more input transducers and one or more output transducers. For example, one or more input transducers may include one or more microphones for converting sound waves into electrical signals that can be stored, processed, and transmitted as audio data. For example, one or more output transducers may include one or more speakers for converting electrical signals into sound waves.

[0073] For example, the user device may also include one or more cameras for capturing video images, which can be stored, processed, and transmitted as video data. For example, the user device may include one or more displays, which may include any form of electronic display that may or may not be a touch-sensitive display. In the case of a touch-sensitive display, the display may also provide an input module form, such as for receiving and invoking selection commands based on detecting a touch input corresponding to a specific user interface element displayed by the touch-sensitive display.

[0074] The user device may also include one or more other input modules, such as one or more of an accelerometer and a gyroscope for generating motion data, based on which the motion characteristics of the user device can be determined. The user device may also include one or more positioning receivers, such as a GNSS (Global Navigation Satellite System) receiver, for determining the geographical location of the user device.

[0075] The user device may include, but is not limited to, a smart phone, a digital assistant, a digital music player, a personal computer, a laptop computer, a tablet computer, or a wearable device such as a smart watch. The user device may be capable of establishing a communication session with one or more other user devices via a communication network (such as an audio or video conferencing session). The user device may be configured to send and receive data using 3G, 4G, LTE, 5G protocols, or any next-generation communication protocol. The user device may include components for short-range communication using, for example, Bluetooth, Zigbee, or WiFi. The user device may include one or more antennas for communicating with external devices.

[0076] Reference Figure 1 , an example first user device is shown in the form of a smart phone 100.

[0077] The smart phone 100 may include a touch-sensitive display (hereinafter referred to as "display") 101, a microphone 102, a speaker 103, and a front camera 104. The smart phone 100 may additionally include a rear camera (not shown) on the rear side of the smart phone. During, for example, the start of a video conferencing application, the front camera 104 may be enabled, whereby video data captured by the front camera may be transmitted through an established video conferencing session.

[0078] Figure 1 Another user device 110 linked to the smart phone 100 is also shown. Another user device 110 is hereinafter referred to as the "linked user device". In this context, the term "linked" means that the linked user device 110 includes a data communication function, so that it can be connected to the first user device 100 via, for example, one or more short-range communication channels using, for example, Bluetooth, Zigbee, or WiFi as part of a local area network (LAN). For example, in a Bluetooth environment, the linked user device 110 may be "paired" with the first user device 100 in a conventional manner, such that data can be sent between the first user device 100 and the linked user device 110, and vice versa. However, it should be understood that other methods of establishing a short-range communication channel may enable data transmission between the first user device 100 and the linked user device 110.

[0079] The linked user device 110 may include a wearable user device, such as a smart watch or the like.

[0080] In this context, the terms "linked" or "paired" are not intended to imply that the linked user device 110 is the only device to which the first user device 100 can be connected or linked using a short-range communication channel. The terms "linked" or "paired" also do not imply the necessity of a short-range communication channel for communication between the first user device 100 and the linked user device 110. For example, each of the first user device 100 and the linked user device 110 may communicate with a server, and the server may forward data from one of the devices to the other device, and vice versa.

[0081] As some examples, the linked user device 110 may alternatively include one or more of another smart phone, a digital assistant, a digital music player, a personal computer, a laptop computer, or a tablet computer.

[0082] The linked user device 110 may include a display 112, which may or may not be a touch-sensitive display. The linked user device 110 may also include one or more microphones and / or one or more speakers (not shown). The linked user device 110 may also include one or more cameras (not shown).

[0083] Reference Figure 2, a video conferencing system 200 is shown. It should be understood that although the example embodiments relate to the video conferencing system shown, the example embodiments can also be applied to a pure audio (e.g., pure voice) conferencing system.

[0084] The video conferencing system 200 can include a first user device 100 (here assuming Figure 1 that the smart phone is the first user device), a second user device 202, a third user device 203, and a conference server 204. The first user device 100 is shown as being connected to a linked user device 110 using an established short-range channel 230.

[0085] For illustration, Figure 2 the shown video conferencing system 200 only includes two remote devices, namely the second user device 202 and the third user device 203, but the video conferencing system can include any number of user devices involved in a video conferencing session.

[0086] The first user 210 can use the first user device 100, the second user 211 can use the second user device 202, and one (or more) third users 212 can use the third user device 203. The user devices 100, 202, 203 can generally be located at different remote locations.

[0087] The second user device and the third user device 203 can include, for example, any one of a smart phone, a digital assistant, a digital music player, a personal computer, a laptop computer, a tablet computer, or a wearable device such as a smart watch. The second user device and the third user device 203 can include the same or similar functions as the first user device 100 and accordingly include, for example, a display screen, one or more microphones, one or more speakers, and one or more front cameras.

[0088] Each of the first user device, the second user device, and the third user device 100, 202, 203 can transmit a stream of captured audio and video data to other user devices via the conference server 204 as part of a conference session, which in this example is a video conferencing session.

[0089] For example, the first user device 100 may transmit an audio stream of the first user 210 speaking and an accompanying video stream, for example, when the first user faces the front camera 104. The audio and video streams may be sent via a first channel 220 established between the first user device 100 and the conference server 204. Then, the conference server 204 may send the audio and video streams to the second user device and the third user device 202, 203 via corresponding second and third channels 221, 222 using, or in the manner of, a multicast transmission protocol established between the conference server and the second and third user devices. Although the first, second, and third channels 220, 221, 222 are indicated by a single line representing a two-way channel, separate channels may also exist, one for sending and one for receiving. The same operating principle applies to the second user device and the third user device 202, 203 when transmitting audio and video streams as part of a video conference session.

[0090] The conference server 204 may be configured to establish a conference session upon receiving a request from one of the first user device, the second user device, or the third user device 100, 202, 203. The request may identify the first user device 100 as the host device or a participant, and the second user device and the third user device 202, 203 as the (multiple) invited user devices or participants. Thereafter, the conference server 204 may forward a signal to the second user device and the third user device 202, 203 to invite them to join the conference session. After acceptance, for example, by the second user device 202, the second channel 221 may be established. The same process may apply to the third user device 203 to establish the third channel 222. Subsequently, one or more other user devices (not shown) may be invited to join the conference session.

[0091] The audio and video streams may include audio packets and associated video packets. The audio packets and video packets may comply with any suitable conference standard, such as the Real-Time Protocol (RTP). The audio packets and video packets may, for example, include a packet header and a packet body, the packet header including control information and the packet body including audio or video data content. The packet header may, for example, include a sequence number indicating the sequential position of the packet in the transmitted packet stream. The packet header may also include a timestamp indicating the timing of the transmitted packet. The packet body may include encoded audio or video data captured during a time slot prior to transmitting the packet. For example, the video data of a packet may include an image sequence representing encoded pixels and spatial coordinates.

[0092] Each of the first user device, the second user device, and the third user device 100, 202, 203 may include, such as the following with reference to Figure 13The apparatuses shown and described. Each of the first user device, the second user device, and the third user devices 100, 202, 203 may be configured by means of hardware, software, firmware, or a combination thereof to perform the operations described below, for example, with reference to Figure 5 the operations described.

[0093] For ease of explanation, example embodiments will be described with respect to the first user device 100, which may be configured to perform operations based on the execution of a conferencing application (e.g., a video conferencing application).

[0094] When executed by the first user device 100, the video conferencing application may be used to establish a video conferencing session via a conferencing server 204 at a current time or a future time, for example, and to identify a participant list of the video conferencing session including the second user device and the third user devices 202, 203. The video conferencing application may also be used by the first user device 100 to join an existing video conferencing session. The video conferencing application may be stored on the first user device 100 or may include an applet or a plug-in that is temporarily executed by the first user device when connected to the conferencing server 204 via a web browser interface.

[0095] During a video conferencing session, when the video conferencing application is executed, a user interface may be provided on the displays of the first user device, the second user device, and the third user devices 100, 202, 203.

[0096] For example, with reference to Figure 3 , an example user interface 300 is shown on the display 101 of the first user device 100.

[0097] The user interface 300 may include a plurality of display portions 301, 302, 303. The first display portion 301 may show a video feed of the first user 210, i.e., video data captured by the front camera 104. In some example embodiments, the first display portion 301 may not be shown or may be disabled or removed by the first user 210. The second display portion 302 may show a video feed of the second user 211, and the third display portion 303 may show a video feed of one or more third users 212. Audio data received from the microphones of the second user device and the third user devices 202, 203 may be output via the speaker 103 of the first user device 100.

[0098] The user interface 300 may further include one or more interactive user interface elements (hereinafter referred to as "UI elements") 305, 306 corresponding to one or more conferencing actions.

[0099] For example, the first UI element 305 can correspond to a mute action, such that selecting the first UI element causes the application to disable the microphone of the first user device 100 and / or prevent audio data as part of a conference session from being sent from the first user device to the second and third user devices 202, 203. Selecting the first UI element 305 again can revoke the mute action. For example, the second UI element 306 can correspond to a suspend or leave action, such that selecting the second UI element can cause the first user device 100 to leave the conference session.

[0100] Figure 4 is another example user interface 400, the user interface 400 is similar to the Figure 3 user interface, except that the spatial arrangement of the first display portion, the second display portion, and the third display portion 401, 402, 403 (which can correspond to the Figure 3 first, second, and third display portions 301, 302, 303 shown) is different. For example, the background of the user interface 400 can include a map (not shown), and the spatial arrangement of the first display portion, the second display portion, and the third display portion 401, 402, 403 can indicate the positions of the corresponding first, second, and third user devices 100, 202, 203 with respect to the map.

[0101] Figure 5 is a flowchart indicating processing operations that can be performed by the first user device 100 according to one or more example embodiments. As described above, the processing operations can be performed by means of hardware, software, firmware, or a combination thereof.

[0102] The first operation 500 can include communicating with a plurality of remote devices in a conference session.

[0103] The second operation 501 can include outputting a user interface on a display, the user interface including a plurality of display portions correspondingly associated with the plurality of remote devices.

[0104] The third operation 502 can include receiving a user selection of one or more remote devices.

[0105] The fourth operation 503 can include displaying, on a linked user device linked to the apparatus, a display portion associated with the selected one or more remote devices.

[0106] The fifth operation 504 can include enabling one or more communication actions for use with the selected one or more remote devices.

[0107] The fourth and fifth operations 503, 504 can be performed in any order or simultaneously.

[0108] In the case where the conference session is a pure audio conference session, each of the plurality of display portions may include some representation of the corresponding remote device or its (multiple) associated users. For example, each display portion may include a photo, an icon, or one or more initials associated with the corresponding remote device or its (multiple) associated users.

[0109] In the case where the conference session is a video conference session, the display portion may include a video feed from the corresponding remote device, such as a plurality of video feeds corresponding to a plurality of remote devices accordingly.

[0110] In this way, the first user device 100 may operate in a mode that may be referred to as a "dual-device mode", in which, for example, a subset of the second user device and the third user device 202, 203 currently connected to the conference session may be selected to implement one or more communication actions specific to those (multiple) selected devices. Moreover, the selection may cause the linked user device 110 to display its (multiple) corresponding video feeds to indicate which user devices are enabled for one or more communication actions. This may also be used to free up user interface space, which may become cluttered when more user devices participate in the video conference session.

[0111] For enabling one or more communication actions for use with one or more selected remote devices, this may include enabling audio and / or video communication with the one or more selected remote user devices via a dedicated communication channel that is not part of the conference session.

[0112] The term "dedicated communication channel" may refer to a channel that is established such that audio and / or video communication via the dedicated communication channel is not sent to the (multiple) non-selected user devices as part of the conference session.

[0113] Additionally or alternatively, one or more UI elements may be enabled. The one or more UI elements may correspond to one or more other communication actions of one or more selected remote devices different from the (multiple) non-selected user devices. The other communication actions may include, for example, a mute action and / or a suspend or leave action.

[0114] Now reference will be made to Figures 6 to 1 2 to describe the example embodiments in more detail.

[0115] Figure 6 Illustrated are the first user device 100 and the linked user device 110 as described previously with reference to Figure 1 The first user device 100 may use the Figure 2 shown video conferencing system 200 to join a video conference session with the second user device and the third user device 202, 203.

[0116] The first user device 100 may display a user interface 600 similar to the Figure 4 user interface shown. The user interface 600 may include a plurality of display portions 601, 602, 603. Optionally, the first display portion 601 may show a video feed of the first user 210, i.e., video data captured by the front camera 104. The second display portion 602 may show a video feed of the second user 211, and the third display portion 603 may show a video feed of one or more third users 212.

[0117] Audio data received from the microphones of the second and third user devices 202, 203 may be output via the speaker 103 connected to the first user device 100, or a connected headset device, etc.

[0118] The user interface 600 may also include one or more UI elements 610 corresponding to one or more conference actions of the conference session, such as the Figure 3 and Figure 4 mute and suspend / leave actions described above.

[0119] A selection of one of the second and third user devices 202, 203 may be received via the user interface 600.

[0120] For example, a user selection of the third user device 203 may be received by means of a touch input corresponding to the third display portion 603 (e.g., via the user's finger 620). Alternatively, a user selection may be received by means of a swipe input or any other predefined gesture corresponding to selecting a remote device for the purposes described herein.

[0121] For example, a selection of the third user device 203 may be received by means of a swipe input substantially in the direction of the linked user device 110.

[0122] Alternatively or additionally, a selection of the third user device 203 may be received by means other than touch input. For example, a selection of the third user device 203 may be by means of a voice command received by the first user device 100 via, for example, Figure 1 the microphone 102 shown. For example, a selection of the third user device 203 may be by means of detecting that the linked user device 110 is positioned close to the third display portion 603 on the user interface 600.

[0123] It should be understood that the above-described method of selecting the third user device 203 is equally applicable to selecting the second user device 202 and indeed any other user device that may join the video conference session.

[0124] Refer to Figure 7, it can be seen that the selection of the third user device 203 effectively causes the third display portion 603 to be displayed on the display 112 of the linked user device 110.

[0125] When displayed on the display 112 of the linked user device 110, the third display portion 603 can be the same as the portion displayed on the user interface 600 of the first user device 100. However, in some example embodiments, the third display portion 603 can be modified for display on the display 112 of the linked user device 110, such as reducing or enlarging its size.

[0126] In some example embodiments, the third display portion 603 can also be removed from the user interface 600 of the first user device 100 to free up space. Although Figure 7 not shown, the remaining first and second display portions 601, 602 can automatically enlarge in response to the removal of the third display portion 603.

[0127] In response to selecting the third user device 203, the first user device 100 can be configured to enable audio and / or video communication with the selected third user device via a dedicated communication channel that is not part of the conference session. This means that the audio and / or video data transmitted via the dedicated communication channel will not be received by the second user device 202.

[0128] The dedicated communication channel can be automatically established in response to the received selection of the third user device 203. Alternatively, further user input may be required to establish the dedicated communication channel and / or initiate communication via the dedicated communication channel.

[0129] In response to selecting the third user device 203, the video conferencing application can also enable one or more UI elements 605 to be displayed on the user interface 600 of the first user device 100. One or more interactive UI elements 605 can correspond to one or more other communication actions, such as a mute action and / or a suspend or leave action.

[0130] When selected, one or more interactive user interface elements 605 can be applicable only to communicate with the third user device 203 via the dedicated communication channel. For example, selecting the mute action via one of the UI elements 605 can mute the audio data transmitted via the dedicated communication channel. For example, selecting the suspend or leave action via one of the UI elements 605 can end or terminate the dedicated communication channel.

[0131] One or more UI elements 605 can be distinguished from other UI elements 606 associated with a video conferencing session, such as by a graphical indicator 701. One or more UI elements 605 can be displayed on the user interface 600 of the first user device 100 rather than on the display 112 of the linked user device 110, thereby taking advantage of the generally larger display area, which means that the UI elements can be larger in size and thus incorrect selections are avoided.

[0132] Figure 8 Indicates after a dedicated communication channel 801 is established between the first user device 100 and the third user device 203 Figure 2 Block diagram of the video conferencing system 200.

[0133] The dedicated communication channel 801 can be established via the conference server 204 and can effectively include two channels, where the conference server receives data through the first channel and relays data through the second channel. However, logically, these two channels can be regarded as one dedicated communication channel 801. Alternatively, the dedicated communication channel 801 can be directly established between the first user device 100 and the third user device 203 via some other communication network, for example.

[0134] At a time when the dedicated communication channel 801 is enabled, the first user device 100 can be configured to detect a user input for initiating an audio and / or video communication through the dedicated communication channel. Based on the detected user input, the first user device 100 can be configured to capture and send audio and / or video data through the dedicated communication channel 801.

[0135] Different user inputs can be correspondingly associated with initiating pure audio, pure video, and audio and video communications through the dedicated communication channel 801.

[0136] For example, based on detecting a first user input for initiating pure audio communication through the dedicated communication channel 801, the first user device 100 can be configured to cause audio data to be captured by one or more microphones of the linked user device 110 for transmission through the dedicated communication channel.

[0137] The first user device 100 can also be configured to disable the transmission of audio data as part of the conference session such that the captured audio data is not heard at the second user device 202. The first user device 100 can also be configured to pause or disable the transmission of video data captured by, for example, the front camera 104 of the first user device as part of the conference session. In this way, the second user 211 cannot see that the first user 210 is participating in a private conversation with one or more third users 212, nor can they see the lip-reading part of the private conversation.

[0138] The first user input may be received via a user interface 600 of a first user device 100, for example.

[0139] Alternatively, or in addition, based on an action at a linked user device, such as detecting movement of the linked user device 110, the first user input may be received. The linked user device 110 may include one or more sensors, such as one or more of an accelerometer and a gyroscope, for generating movement data that is receivable by the first user device 100 using the established short-range channel 230. The first user device 100 may be configured to detect the determined movement of the linked user device 110 from the received movement data, the movement corresponding to a predetermined gesture associated with the first user input. Alternatively, or in addition, a movement sensor of the first user device 100 may be configured to detect the determined movement of the linked user device 110.

[0140] Alternatively, or in addition, the first user device 100 may be configured to determine the position of the linked user device 110 and detect that the determined position corresponds to a predetermined position associated with the first user input. Any known method may be used to determine the position of the linked user device 110. For example, if the linked user device 110 includes a positioning receiver, positioning data generated by the linked user device may be sent to the first user device 100 using the established short-range channel 230.

[0141] Alternatively, or in addition, if the distance between the linked user device 110 and the first user device 100 can be calculated, the calculated distance may be used to estimate the position of the linked user device, thereby modifying the positioning data generated by a positioning receiver of the first user device 100. The distance between the linked user device 110 and the first user device 100 may be calculated based on, for example, a received signal strength indication (RSSI) signal or data.

[0142] In some example embodiments, the first user device 100 may be configured to detect that the linked user device 110 is within a predetermined distance d1 of the user's (i.e., the first user 210's) mouth. The predetermined distance may be on the order of 0 to 20 centimeters, but may also vary using, for example, settings associated with a video conferencing application. The detection may be based on one or a combination of the above movement data and position data. Alternatively, or in addition, the first user device 100 may be configured to detect that the linked user device 110 is within a predetermined distance d1 of the user's mouth based on the amplitude of an audio signal received from the linked user device 110 being higher than a predetermined threshold.

[0143] In the case where the first user device 100 is configured to pause or prohibit the transmission of video data, this can occur before the linked user device 110 enters within a predetermined distance d1 of the user's mouth, for example, by detecting a specific motion acceleration and / or direction of the linked user device towards the user's mouth. This can prevent the second user 211 from noticing that the communication channel 801 will be used.

[0144] Figure 9 The above situation is shown, whereby the first user 210 moves the linked user device 110 to a position within a predetermined distance d1 of the user's mouth to initiate an audio communication via the dedicated communication channel 801. The first user 210 can continue to view video data associated with the ongoing conference session on the display 101 of the first user device 100.

[0145] Alternatively, or additionally, based on detecting a second user input for initiating a video communication via the dedicated communication channel 801, the video conferencing application can be configured to cause video data captured by one or more cameras of the linked user device 110 to be transmitted via the dedicated communication channel.

[0146] The second user input can be received via, for example, the user interface 600 of the first user device 100. Alternatively, the second user input can be received based on any of the above motion- and / or position-based detection methods, as long as it can be distinguished from the first user input.

[0147] Figure 10 The above situation is shown, whereby the first user 210 provides a second user input for initiating a video communication and can direct the camera of the linked user device 110 towards some object 1000 that the first user desires to show to one or more third users 212 via the dedicated communication channel 801. The first user 210 can continue to view the ongoing conference session on the display 101 of the first user device 100. The front camera 104 of the first user device 100 can continue to provide video data as part of the video conferencing session.

[0148] The first user device 100 can also be configured to cause audio data captured by one or more microphones of the first user device (and / or the linked user device 110) to be transmitted via the dedicated communication channel 801.

[0149] For example, the first user device 100 can include a microphone array, and the first user device can be configured to determine the position of the linked user device 110 relative to the first user device and, based on the determined relative position, apply beamforming to the microphone array. In this way, the audio data transmitted via the dedicated communication channel 801 can include a direction component based on the position of the linked user device 110 relative to the first user device.

[0150] In some example embodiments, there may be a selection of multiple remote devices received by the video conferencing application.

[0151] For example, referring Figure 4 , assuming there are other remote devices in the conference session and thus other display portions are shown on the user interface 400, in addition to the third user device 203, the first user 210 may also select the second user device 202. In response to the selection of the second and third remote user devices 202, 203, the first user device 100 may be configured to cause the second and third display portions 602, 603 corresponding to the second and third remote devices to be shown simultaneously or sequentially on the linked user device 110. Figure 11 Illustrates the second and third display portions 602, 603 being shown simultaneously on the linked user device 110.

[0152] If the first user device 100 is configured to sequentially show the second and third display portions 602, 603, it may be performed in the following manner: First, cause one of the second and third display portions to be shown, and subsequently, in response to user input received by the linked user device 110, cause the other of the second and third display portions to be shown. Figure 12A and Figure 12B Illustrates such sequential display, where first the third display portion 603 ( Figure 12A ) is shown, and then the second display portion 602 ( Figure 12B ) is shown.

[0153] User input received by the linked user device 110 may include, for example, a touch input on its display 112 (if it is a touch-sensitive display), or a rotation of the bezel 1200, or any other suitable input means.

[0154] In some example embodiments, the video conferencing application may be configured to enable one or more communication actions only for the selected user device for which the associated display portion is shown at the current time.

[0155] For example, referring Figure 12A , one or more communication actions such as communicating via a dedicated communication channel 801 and / or enabling a mute or suspend / leave action may be used only with the third user device 203.

[0156] In some example embodiments, the first user device 100 may also be configured to receive user input for supplementing or changing a selection of remote devices, and based on the received user input for supplementing the selection, cause (i) one or more currently selected remote devices to be deselected, thereby removing their corresponding display portions from the linked user device and disabling one or more enabled actions for the currently selected remote devices, and (ii) one or more currently unselected remote devices to be selected, thereby causing their corresponding display portions to be displayed on the linked user device and enabling one or more actions for the currently unselected remote devices.

[0157] Reference Figure 7 , for example, in response to receiving a supplementary selection, the third user device 203 may be deselected and the second user device 202 may be selected, and their corresponding display portions 603, 602 may be swapped between the user interface 601 and the display 112 of the linked user device 110. This may enable the first user 210 to switch between different private conversations in a fast and seamless manner.

[0158] Other communication actions that may be enabled for a selected subset of remote devices may include enabling recording of audio and / or video data transmitted over a dedicated communication channel 801.

[0159] Example embodiments may enable participants in a conference session (such as a video conference) to select a subset of other participants in the conference session, such that their corresponding display portions may be moved to other different linked user devices. This may free up user interface space on a particular user device and may also enable one or more communication actions, such as enabling or establishing a dedicated communication channel, such that a user of a particular user device may converse with or present information to one or more selected participants outside of the conference session. This may be to keep information confidential and / or to avoid distracting other unselected participants with irrelevant information. By forming display portions corresponding to the selected participant(s) on the linked user device, there is a clear separation between the participants. If a relatively large number of other participants are shown on the first user device 100, the selected participant(s) on the linked user device 110 may also be clearly visible compared to potentially much smaller display portions.

[0160] Exemplary apparatus

[0161] Figure 13Shows a device according to an embodiment, which may include a first user equipment 100, a linked user equipment 110, and / or a second user equipment and a third user equipment 202, 203 or their components. The device may be configured to perform the operations described herein, such as the operations described with reference to any of the foregoing processes, including Figure 5 the operations. The device includes at least one processor 1300 and at least one memory 1301 directly or closely connected to the processor. The memory 1301 includes at least one random access memory (RAM) 1301a and at least one read-only memory (ROM) 1301b. Computer program code (software) 1305 is stored in the ROM 1301b. The device may be connected to a transmitter (TX) and a receiver (RX). The device may optionally be connected to a user interface (UI) for indicating the device and / or for outputting data. At least one processor 1300, at least one memory 1301, and computer program code 1305 are arranged to cause the device to perform at least a method according to any of the foregoing processes including Figure 5 the method.

[0162] Figure 14 Shows a non-transitory medium 1400 according to some embodiments. The non-transitory medium 1400 is a computer-readable storage medium. It may be, for example, a CD, a DVD, a USB stick, a Blu-ray disc, etc. The non-transitory medium 1400 stores computer program code that causes the device to perform a method according to any of the foregoing processes including Figure 5 the method.

[0163] The names of network elements, protocols, and methods are based on current standards. In other versions or other technologies, the names of these network elements and / or protocols and / or methods may be different as long as the corresponding functions can be provided. For example, the embodiments may be deployed in 2G / 3G / 4G / 5G networks and next-generation 3GPP networks, but may also be deployed in non-3GPP wireless networks such as WiFi.

[0164] The memory may be volatile or non-volatile. It may be, for example, RAM, SRAM, flash memory, FPGA block RAM, DCD, CD, USB stick, and Blu-ray disc.

[0165] Unless otherwise specified or otherwise indicated from the context, saying that two entities are different means that they perform different functions. But this does not necessarily mean that they are based on different hardware. That is, each entity described in this specification can be based on different hardware, or some or all of the entities can be based on the same hardware. But this does not necessarily mean that they are based on different software. That is, each entity described in this specification can be based on different software, or some or all of the entities can be based on the same software. Each entity among the entities described in this specification can be implemented in the cloud.

[0166] As a non-limiting example, an implementation of any of the above boxes, devices, systems, techniques, or methods includes being implemented as hardware, software, firmware, a dedicated circuit or logic, general hardware or a controller, or other computing devices, or some combination thereof. Some embodiments can be implemented in the cloud.

[0167] It should be understood that the above is currently considered to be a preferred embodiment. However, it should be noted that the description of the preferred embodiment is given only as an example, and various modifications can be made without departing from the scope defined by the appended claims.

Claims

1. An apparatus for communication, comprising components for the following operations: Communicating with a plurality of remote devices in a video conferencing session; Outputting a user interface on a display, the user interface including a plurality of video feed display portions correspondingly associated with the plurality of remote devices; Receiving a user selection of one or more of the remote devices; and Based on the received user selection: Causing a video feed display portion associated with the selected one or more remote devices to be displayed on a linked user device linked to the apparatus, and Enabling one or more communication actions for use with the selected one or more remote devices through a dedicated communication channel that is not part of the video conferencing session: In response to detecting a user input for initiating audio communication through the dedicated communication channel, causing audio data to be captured by one or more microphones of the linked user device for transmission through the dedicated communication channel, and disabling the transmission of audio data as part of the video conferencing session.

2. The apparatus according to claim 1, wherein the component for enabling is configured to establish the dedicated communication channel in response to the received user selection.

3. The apparatus according to claim 1 or claim 2, further comprising components for the following operations: Detecting a user input for initiating video communication through the dedicated communication channel, and based on detecting the user input, capturing and transmitting video data through the dedicated communication channel.

4. The apparatus according to claim 3, wherein based on detecting the user input for initiating audio communication through the dedicated communication channel, the apparatus is further configured to pause or disable the transmission of video data from the apparatus as part of the video conferencing session.

5. The apparatus according to claim 3, further comprising components for determining the movement of the linked user device, and wherein the component for detecting is configured to detect a user input for initiating audio communication through the dedicated communication channel based on the determined movement of the linked user device corresponding to a predetermined gesture.

6. The apparatus according to claim 4, further comprising components for determining the movement of the linked user device, and wherein the component for detecting is configured to detect a user input for initiating audio communication through the dedicated communication channel based on the determined movement of the linked user device corresponding to a predetermined gesture.

7. The apparatus according to claim 3, further comprising components for determining the location of the linked user device, and wherein the component for detecting is configured to detect a user input for initiating audio communication through the dedicated communication channel based on the determined location of the linked user device corresponding to a predetermined location.

8. The apparatus according to claim 7, wherein the component for detecting is configured to detect a user input for initiating audio communication through the dedicated communication channel based on the determined location of the linked user device being within a predetermined distance of the user's mouth.

9. The apparatus according to claim 3, wherein based on detecting a user input for initiating a video communication via the dedicated communication channel, the apparatus is configured to cause video data to be captured by one or more cameras of the linked user device for transmission via the dedicated communication channel.

10. The apparatus according to claim 9, wherein the apparatus is further configured to cause audio data to be captured by one or more microphones of the apparatus for transmission via the dedicated communication channel.

11. The apparatus according to claim 10, wherein the apparatus includes a microphone array, wherein the apparatus is configured to determine a position of the linked user device relative to the apparatus and, based on the determined relative position, apply beamforming to the audio data captured by the microphone array.

12. The apparatus according to claim 1 or claim 2, wherein in response to receiving a user selection of a plurality of remote devices, the apparatus is configured to: Cause corresponding display portions associated with the plurality of selected remote devices to be displayed simultaneously or sequentially on the linked user device.

13. A method of communication, comprising: Communicating with a plurality of remote devices in a video conference session at an apparatus; Outputting a user interface on a display, the user interface including a plurality of video feed display portions respectively associated with the plurality of remote devices; Receiving a user selection of one or more of the remote devices; and Based on the received user selection: Causing a video feed display portion associated with the selected one or more remote devices to be displayed on a linked user device linked to the apparatus, and Enabling one or more communication actions for use with the selected one or more remote devices via a dedicated communication channel that is not part of the video conference session: in response to detecting a user input for initiating an audio communication via the dedicated communication channel, causing audio data to be captured by one or more microphones of the linked user device for transmission via the dedicated communication channel, and disabling the transmission of audio data as part of the video conference session.

Images