Video conference terminal system

Abstract

A video terminal system (250a) for use as an endpoint (203a) in a video conference, the video terminal system (250a) comprising one or more processors configured to conduct the video conference by performing the following steps: establishing a logical association with a media source (220) communicatively connectable with the video terminal system (250a); receiving media from a media source (220) wherein the media is directed to the video end system (250a) according to the logical association; and inserting the media into a video conference established with another video conference endpoint (203b).

Description

Technical Field

[0001] This invention relates to video conferencing systems, and more specifically to a video terminal system that can receive media from a media source to insert into a video conference. Background Technology

[0002] Video conferencing systems typically include two or more endpoints, each configured to communicate with other endpoints via data or telephone networks or other connections to implement two-way or multi-way video and / or audio teleconferences.

[0003] An endpoint (also known as a video terminal system) is configured to enable users to communicate with other users over a network by sending data streams (such as audio, video, and other content streams, e.g., data files, text, or screen sharing) from one endpoint to another during a two-way teleconference. For multi-party teleconferences, the data streams may be transmitted to a centralized or distributed conferencing network, where the data streams are exchanged or transcoded, allowing multiple participants to communicate and share information. A video terminal system may include a single-purpose endpoint device whose primary function is to support video and / or audio teleconference. A video terminal system may include separate video conferencing hardware, which may include one or more video monitors, video cameras, audio microphones, audio speakers, and a controller that enables users to control the video terminal system. These devices are typically permanently connected to the video terminal system and remain part of the video terminal system configuration.

[0004] In some cases, it can be advantageous to provide content streaming to a video conference from an external device. Video conferencing terminal systems typically achieve this via cable (such as an HDMI cable) or by using network streaming with protocols such as AirPlay, Google Cast, or Miracast.

[0005] Figure 1 A diagram of a typical known video conferencing system 100 is shown, characterized by a laptop computer 110 used to inject media into the content stream of a video conference. Two video terminal systems 103 are connected to corresponding controllers 102, displays 101 and 105, and a camera 104. They form a connection 107 across the Internet 106 to stream video and audio from the camera and microphone, thereby establishing communication across screens 101 and 105. The laptop computer 110 is connected to the video terminal systems 103 via a conventional High Definition Multimedia Interface (HDMI) cable and provides video and / or audio feeds to the content stream displayed on displays 101 and 105.

[0006] Establishing a cable connection is usually straightforward. Establishing a network connection may require additional configuration, typically including device discovery, device trust, selecting the exact video source for the sending device (such as a screen or application window), and then selecting the video on the end system.

[0007] However, the conference room may already contain another fixed piece of equipment, such as a separate video conferencing terminal system or interactive display system connected to the corporate network and managed by a centralized network management system. Therefore, it is undesirable for such equipment to require a complex associativity process before it can be used as the primary video terminal system for content streaming.

[0008] The goal is to develop an improved mechanism for granting video terminal systems access to media sources at auxiliary devices for inclusion in video conferencing. Summary of the Invention

[0009] According to a first aspect, a video terminal system is provided for use as an endpoint in a video conference. The video terminal system includes one or more processors configured to conduct a video conference by performing the following steps: establishing a logical association with a media source communicatively connected to the video terminal system; receiving media from the media source, wherein the media is directed to the video terminal system according to the logical association; and inserting the media into a video conference established with another video conference endpoint.

[0010] The one or more processors can be configured to establish a logical association with a media source that can communicate with the video terminal system by communicating with a video conferencing server. The video conferencing server can be located on a cloud server. The video conferencing server can be implemented by one or more servers located remotely from the video terminal system and auxiliary equipment and communicating with them via the Internet. Alternatively, the logical association with the media source can be established in ways other than communicating with the video conferencing server, such as via Bluetooth, voice pairing, proximity pairing, or user interaction.

[0011] By communicating with the video conferencing server, media can be inserted into a video conference established with another video conferencing endpoint. The video conferencing server may include: a management server, where the logical association can be established; and optionally a conference server through which media streams (optionally including media from a media source) can be sent as part of the video conference to another video conferencing endpoint. The management server and / or conference server may be located remotely from the video terminal system. They may be separate entities from the video terminal system. The management server may be a separate entity from the conference server. In some embodiments, the media stream (optionally including media from a media source) can be sent to another video conferencing endpoint via an alternative connection (e.g., a point-to-point connection / wired connection).

[0012] The media source can be located remotely from the video terminal system. The media source can be wirelessly connected to the video terminal system. The media source can be an auxiliary computing device that can communicate with the video terminal system. The media source can be implemented by one or more processors of an auxiliary computing device (or associated with it) that can communicate with the video terminal system.

[0013] One or more processors of the video terminal system may be configured to implement a first software entity for supporting video conferencing, the first software entity being capable of receiving media input from any of a plurality of input channels. The video terminal system may be able to implement a second software entity for receiving media from a media source and supplying media to the first software entity via one of the analog input channels.

[0014] The one or more processors can be configured to establish a control channel and / or a media channel between the video terminal system and a media source. The one or more processors can also be configured to establish a control channel and / or a media channel between the video terminal system and an auxiliary computing device. The control channel and / or media channel can be established via a video conferencing server. For example, the control channel and / or media channel can be established via a management server. The video terminal system can receive media from the media source via the media channel. Alternatively, a control channel can be established directly with the media source and / or computing device.

[0015] The video terminal system may include user interfaces for controlling various aspects of the video terminal system and / or media sources and / or auxiliary computing devices.

[0016] The user interface may include a user input device. The one or more processors may be configured to receive signals in response to user input at the user input device, and to receive media from a media source in response to receiving a signal for insertion into a video conference. The user input device may become visible on the user interface when a connection is established between the computing device and the video terminal system.

[0017] The video terminal system may include video conferencing hardware, such as one or more displays, cameras, or microphones. The one or more processors may be further configured to output media received from a media source at the video conferencing hardware.

[0018] The one or more processors can be configured to receive media via a data link that supports the transmission of media data from a media source only to the video terminal system. The data link can be a media channel separate from one or more video conferencing channels on which video conferencing data is sent to and received from another endpoint. The data link can be via a video conferencing server (e.g., a management server).

[0019] The video terminal system can be configured to establish logical associations with multiple media sources (e.g., by communicating with a video conferencing server or in other ways as described above). The video terminal system can be configured to receive media from one of the multiple media sources in response to a user selecting that source and to insert it into the video conference according to the corresponding logical association in the logical associations.

[0020] If it is determined that the auxiliary computing device acting as or implementing a media source is outside the range of the video terminal system, access to one or more media sources associated with the video terminal system at the video conferencing server can be restricted. The proximity of the auxiliary computing device to the video terminal system can be determined using technologies such as Bluetooth or audio pairing. If it is determined that the auxiliary device is outside the range of the video terminal system, this can be indicated at the server, preventing media from the media source from being inserted into the video conference.

[0021] According to another aspect, a computing device is provided, configured to communicate with a video terminal system used as an endpoint in a video conference, the computing device including one or more processors configured to perform the following steps: establishing a logical association between a media source and the video terminal system; and causing media transmission from the media source to the video terminal system to be inserted into a video conference established between the video terminal system and another video conference endpoint, wherein the media is directed to the video terminal system according to the logical association.

[0022] The one or more processors can be configured to establish a logical association with the video terminal system by communicating with a video conferencing server. The video conferencing server can be located on a cloud server. The video conferencing server can be implemented by one or more servers located remotely from the video terminal system and auxiliary equipment and communicating with them via the Internet. Alternatively, the logical association with the video terminal system can be established in other ways, such as via Bluetooth, voice pairing, proximity pairing, or user interaction.

[0023] The computing device can be an auxiliary computing device. The computing device can be located remotely from the video terminal system. The one or more processors can be configured to initiate media transmission from a media source to the video terminal system via communication with a video conferencing server, to be inserted into a video conference established between the video terminal system and another video conferencing endpoint.

[0024] A media source can be a computing device. A media source can be implemented by one or more processors of a computing device (or associated with it) that can communicate with a video terminal system.

[0025] One or more processors of the computing device may be configured to implement a first software entity (which may or may not be the same as the first software entity mentioned above) capable of outputting media from a media source on any of a plurality of output channels. The computing device may be able to implement a second software entity (which may or may not be the same as the second software entity mentioned above) for receiving media from a media source and supplying the media to a video terminal system via one of the analog output channels.

[0026] The one or more processors can be configured to process media data output from a media source to form a compressed representation of the media data, which is then supplied to a video terminal system for insertion into a video conference.

[0027] The one or more processors can be configured to establish a control channel between a computing device and a video terminal system. The one or more processors can also be configured to establish a control channel between a media source and a video terminal system.

[0028] The computing device may include a user interface for controlling various aspects of the computing device and / or media source and / or video terminal system. The one or more processors may be configured to update the user interface when a connection is established between the computing device (and / or media source) and the video terminal system. The user interface may include a user input device, wherein the one or more processors are configured to receive a signal in response to user input at the user input device, and in response to receiving the signal, to initiate media transmission to the video terminal system for insertion into a video conference. The user input device may become visible on the user interface when a connection is established between the computing device and the video terminal system. The user interface may be configured to advertise the user input device as a means of transmitting media from the media source to the video terminal system.

[0029] In response to receiving a signal, the one or more processors can be configured to establish a media channel between the media source and the video terminal system.

[0030] The computing device and / or media source can be an interactive whiteboard, mobile phone, tablet computer, or laptop computer. The media source can be configured to output one or more of the following media types: audio, video, image files, computer files, and text.

[0031] Logical associations can be persistent logical associations used across multiple video conferencing sessions.

[0032] Video conferencing servers can be cloud servers or local area networks (such as corporate networks).

[0033] According to another aspect, a computer-implemented method is provided for conducting a video conference at a video terminal system configured to act as an endpoint in a video conference, the method comprising performing the steps of: establishing a logical association with a media source, the media source being communicatively connected to the video terminal system; receiving media from the media source, wherein the media is directed to the video terminal system according to the logical association; and inserting the media into a video conference established with another video conference endpoint.

[0034] According to another aspect, a computer-implemented method is provided for implementation at a computing device configured to communicate with a video terminal system used as an endpoint in a video conference, the method comprising the steps of: establishing a logical association between a media source and the video terminal system; and inducing media transmission from the media source to the video terminal system to be inserted into a video conference established between the video terminal system and another video conference endpoint, wherein the media is directed to the video terminal system according to the logical association.

[0035] In some other examples, the assistive computing device can be configured to run an application. This application can be a third-party application. For example, the assistive computing device can be configured to run presentation software or video editing software. This can provide the media sources discussed above. As mentioned above, the output of the third-party application can be included in the content stream of the video teleconference. The third-party application can be provided in an application environment configured to provide an interface between the third-party application and the rest of the assistive computing device.

[0036] The auxiliary computing device may include a control interface device. The control interface device may include or be configured to display a user interface of the auxiliary computing device. The computer-implemented method may further include: providing an application environment on which a third-party application is to run, the application environment being configured to: transmit control signals from the control interface device to the third-party application; and transmit media signals between the third-party application and the auxiliary computing device hardware and / or a data link to a video terminal system, for inclusion in the content stream of a video teleconference. Advantageously, by providing an application environment for running the third-party application, the third-party application can be used with the auxiliary computing device without problems arising from incompatibility between the third-party application and the auxiliary computing device.

[0037] The method implemented by this computer can also include running third-party applications within the application environment. Accordingly, the functionality of the auxiliary computing device can be expanded by running new applications.

[0038] In one or more examples, the application environment may include a control signal interface, that is, programming configured to transmit control signals from a control interface device of the assistive computing device to a third-party application. The assistive computing device may include hardware such as a display screen and / or speakers. In one or more examples, the application interface may include a media signal interface, that is, programming configured to transmit media signals between the third-party application and the assistive computing device hardware and / or a data link to a video terminal system for inclusion in the content stream of a video teleconference.

[0039] Third-party applications may include collaborative applications. For example, an application may be one or more of the following: document sharing, whiteboard presentations, multimedia presentations, training, review, voting, or question submission applications. Third-party applications may be provided by or downloaded from external sources (i.e., the third-party application is not part of the system software originally provided with the assistive computing device). Third-party applications may be configured to run on mass-market hardware such as tablets or smartphones.

[0040] An application environment is defined as a framework or platform that has been configured to accept the deployment of a single application. An application environment includes a predefined set of computing resources that host the application. An application environment can be configured to provide virtualized or standardized device interfaces to the application, enabling the application to run even if the physical device (originally intended for use with the application) is disconnected or differs from what the application is expected to do. For example, a virtualized device interface can be configured to retransmit data to a different physical device while the application is running. In some examples, multiple application environments can be provided to operate multiple corresponding applications.

[0041] The application environment can be provided on the processor of the auxiliary computing device, or on a processing element connected to the processor (e.g., via an Internet connection to a cloud service). In some examples, the application environment can be provided on the control interface device of the auxiliary computing device.

[0042] The application environment may include a data mapping interface configured to: relay control signals from a control interface device to a third-party application; and relay media signals between the third-party application and auxiliary computing device hardware and / or to a video terminal system for inclusion in the content stream of a video teleconference. The data mapping may be a virtual interface that defines the boundaries of the application environment and acts as a virtualization layer by relaying data between the third-party application within the application environment and the software and hardware outside the application environment.

[0043] Media signals may include video and / or audio data that can form the content stream of a video teleconference (e.g., audiovisual signals operating in a camera-off mode). In some examples, media signals may include other types of data that can be passed to and from third-party applications. For example, media signals may include computer files or text (e.g., other types of audiovisually relevant data that can be shared via a video link, such as still images, computer files, and text). Control signals may include those for selectively enabling and disabling media signal transmission between third-party applications and auxiliary computing device hardware and / or data links to the video terminal system for inclusion in the content stream of the video teleconference.

[0044] For example, the media signal may include video from a third-party application for enabling remote viewing of the third-party application via auxiliary computing device hardware and / or a data link to the video terminal system, to be included in the content stream of the video teleconference. Therefore, a computer-implemented method may include: receiving a remote viewing control signal from a control interface device; and enabling or disabling remote viewing of the third-party application on one or more video conferencing hardware units and / or enabling or disabling remote viewing of the third-party application on one or more channels of the video teleconference based on the remote viewing control signal. In this way, the user of the video terminal system can decide on which displays the third-party application is shown and which remote participants in the video teleconference can view the third-party application.

[0045] The application environment can be configured to convert control signals of a first format from a control interface device of an auxiliary computing device into application control signals of a second format for controlling a third-party application (i.e., using a data mapping interface). In other words, the method implemented by the computer can include converting (i.e., reformatting) control signals of a first format from a control interface device into application control signals of a second format for controlling a third-party application using a data mapping interface. For example, the application environment can be configured to simulate gestures or keystrokes in the application when it receives user control signals from the control interface. This can include simulating function buttons, a virtual keyboard, a dial pad, a directory lookup, or other navigation elements of a third-party application in a virtual environment.

[0046] For example, a control interface device (or one or more such control interface devices) for an auxiliary computing device may include a keyboard for sending control signals as keyboard events. The application environment can be configured to convert keyboard events into application control signals for controlling third-party applications. In other words, a computer-implemented method may include the steps of converting control signals from the control interface device, including keyboard events and having a first format, into application control signals having a second format for controlling third-party applications. The application control signals can be configured to mimic control signals expected by the third-party application. For example, keyboard events can be converted into function signals corresponding to function buttons normally provided by the third-party application.

[0047] A control interface device may include a touchscreen of a first size for transmitting control signals, and a third-party application may be configured to receive application control signals from a touchscreen of a second size. For example, the application may be designed to run on a smartphone, and the control interface device may be a much larger screen. In this case, the application environment may be configured to scale or map control signals from the control interface device to application control signals for controlling the third-party application. In other words, a computer-implemented method may include the step of scaling control signals from the control interface device to form application control signals for controlling the third-party application. For example, the application environment may be configured to receive position and movement information from the control interface device touchscreen and scale the position and movement information proportionally to correspond to the application touchscreen.

[0048] The control interface device may include a user interface environment for running an application user interface associated with a third-party application. For example, the control interface device may include a screen or touchscreen, and the application user interface may be configured to display a representation of the third-party application on the screen. The representation of the third-party application may include function buttons, a virtual keyboard, a dial pad, a directory lookup, or other navigation elements for navigating and / or controlling the third-party application. Similarly, the application environment may be configured to convert a video signal of a first format from a third-party application into a media signal of a second format for a content stream used in video telephony and / or video conferencing hardware. In other words, the computer-implemented method may include the step of converting a media signal of a first format from a third-party application into a media signal of a second format for a content stream used in video telephony and / or auxiliary computing device hardware. For example, the media signal may include a video signal, and the application environment may be configured to reformat the video data to map screen size, video format (e.g., compressed vs. uncompressed), pixel count, etc., between formats compatible with the content stream of the application and auxiliary computing device hardware and / or video telephony. In other examples, media signals may include audio signals, image or file data, etc., converted from a first format compatible with third-party applications to a second format.

[0049] The application environment may include a data mapping interface, which is formed by an application programming interface (API) provided by a third-party application.

[0050] In some examples, the application environment may include custom instructions corresponding to third-party applications. These custom instructions may be stored in a centralized configuration service. The centralized configuration service may reside on a remote server accessible to the auxiliary computing device. In this case, the computer-implemented method may include retrieving the custom instructions from the centralized configuration service and implementing or creating a data mapping interface based on the custom instructions.

[0051] In another aspect, an application environment can be provided, configured to host a third-party application for operation of an auxiliary computing device, wherein the application environment is configured to: transmit control signals from a control interface device of the auxiliary computing device to the third-party application; and transmit media signals between the third-party application and the auxiliary computing device hardware and / or a data link to a video terminal system for use in the content stream of a video teleconference.

[0052] The video terminal system and / or computing device may have any of the optional features described above, or be configured to perform the steps of the computer-implemented methods described above.

[0053] According to another aspect, a computer program is provided that includes instructions that, when executed by a computer, cause the computer to perform the methods described above.

[0054] According to another aspect, a computer-readable storage medium is provided that stores computer-readable instructions that, when executed at a computer (e.g., including one or more processors), cause the computer to perform the methods described above. The computer-readable storage medium may be a non-transitory computer-readable storage medium. The computer may be implemented as a system of interconnected devices. Attached Figure Description

[0055] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0056] Figure 1 A diagram of a typical video conference is shown, in which a laptop computer connected to the video terminal system via cable provides a stream of content shared through the video conference;

[0057] Figure 2a A diagram showing auxiliary equipment used as a media source for a conference is provided.

[0058] Figure 2b Examples of connections and signals between various entities in the system for inserting media from a media source into a video conference are shown;

[0059] Figure 3 An example of an auxiliary computing device is illustrated schematically;

[0060] Figure 4 A diagram of a video conferencing system based on another example is shown;

[0061] Figure 5 A diagram of a video conferencing system based on another example is shown;

[0062] Figure 6 A diagram of a video conferencing system based on another example is shown;

[0063] Figure 7 The steps of an exemplary method for conducting video conferencing at a video terminal system are schematically illustrated.

[0064] Figure 8 The steps of an exemplary method for implementation at an auxiliary computing device configured to communicate with a video terminal system are schematically illustrated.

[0065] Figure 9 The illustration shows some components of an exemplary device and its associated parts configured to operate as a video terminal system and / or auxiliary computing device. Detailed Implementation

[0066] This invention relates to a logical composite system for video conferencing that shares media from a media source with a video terminal system, which can be inserted into a video conferencing media stream.

[0067] Figure 2a An example of a video conferencing system 200 is shown. In this example, two video terminal systems 250a and 250b can communicate with each other, thereby allowing a video conference to be established between the two video terminal systems.

[0068] In the following examples, the term "video terminal system" is used to refer to a dedicated video conferencing system. A video terminal system includes video conferencing endpoints and may also include function-specific hardware and software for performing video teleconferences. An endpoint can be a uniquely addressable point on a network that can call another video conferencing endpoint. A video terminal system can also be a shared video terminal system configurable for use by multiple individual users. For example, a shared video terminal system may be located in a corporate conference room that can be booked and used by multiple users for teleconferences.

[0069] A video terminal system includes one or more processors. The video terminal system may include a main processor. The main processor may be a processing element of the video terminal system configured to run system software for operating the video terminal system and performing video teleconferences. The main processor may be configured to communicate with video conferencing hardware and receive commands from the video terminal system's controller. The main processor may be a separate endpoint device, such as a single-purpose computing device or a local server, connected, for example, to the video conferencing hardware and controller of the video terminal system via a wired or wireless connection. Thus, a device including a main processor may also be referred to as an endpoint device. In other examples, the main processor may form part of a single unit that also includes a controller and / or video conferencing hardware. The main processor may include a network interface for sending and receiving content streams of video teleconferences.

[0070] therefore, Figure 2a The video terminal systems shown can be used to perform video conferencing. In this example, each video terminal system 250a, 250b includes video conferencing endpoints 203a, 203b. In this example, endpoints 203a, 203b can communicate with each other via a communication network 206. Network 206 can be, for example, the Internet or a corporate network. Video terminal system 250a can establish a video conference with the endpoint of another video terminal system 250b via communication network 206 (i.e., by sending media data via the communication network).

[0071] As described in this article, infrastructure is provided to support video conferencing between video terminal systems using joint auxiliary media sources. This infrastructure... Figure 2aThe infrastructure is generally shown as video conferencing server 215. The infrastructure may be located entirely off-site from the video terminal systems and may communicate with the endpoints of the video terminal systems via a network 206 (such as the Internet). Communication may be wireless. Alternatively, part or all of the infrastructure may be located in a shared location with one or more of the video terminal systems, for example, in an enterprise-managed implementation. The infrastructure may be provided by one or more computing devices or servers. Each server may have one or more processors for implementing its functions. Servers may be located close to each other or in dispersed locations. In one example, a physical or cloud server 222 (e.g., a live conferencing server) may support the establishment and maintenance of video conferences between two video terminal systems, and another physical or cloud server 221 (a central management server) may support the establishment of logical associations between the video terminal systems and media sources, as described below. In another embodiment, both functions may be provided by a single video conferencing server that provides live conferencing services and central management services. In some embodiments, server 222 may not exist as part of video conferencing server 215, and the connection between video terminal systems 250a and 250b for sending media as part of a video conference may be a point-to-point connection or a cable connection. In this specification, the term "video conferencing server" refers to a server configured to provide support for one or more of the video conferencing functions as described herein between video terminal systems.

[0072] exist Figure 2a In the example shown, the video terminal system can communicate with conference server 222. This connection is via a communication network 206, such as the Internet. The video terminal system can be one of multiple video terminal systems communicating with conference server 222. The video terminal system can establish a video conference with one or more other video conferencing endpoints, which can be part of a corresponding video terminal system, via server 222. The video terminal system can communicate with one or more other video conferencing endpoints via network 206. Media streams can be transmitted via... Figure 2a Channel 207 in the video terminal system 250a is transmitted between one or more other endpoints via communication network 206.

[0073] One or more processors (e.g., a main processor) of a video terminal system can be configured to implement one or more software entities configured to operate the video terminal system and enable it to perform teleconferencing. The video terminal system can be operated to conduct video and / or audio teleconferencing by running system software. The system software can be an application or computer program that includes proprietary software provided with and configured to operate with the video terminal system. The video terminal system can be managed by a standard operating system. For example, a standard operating system can be a mass-market operating system such as Windows, Apple iOS, Linux, or Android.

[0074] One or more processors (e.g., a main processor) of each video terminal system may be configured to implement a first software entity for supporting video conferencing. The first software entity may be an application, computer program, or software package running on the video terminal system. The first software entity may be able to receive media input from any of a plurality of input channels. The plurality of input channels may be, for example, input channels for media output via video conferencing hardware such as a camera or microphone, or one or more input channels may be ports, such as a High Definition Multimedia Interface (HDMI) port, to which an additional media source may be connected and to provide media output by the media source for inclusion in the video conference. Media input from the input channels may be transmitted to another video terminal system during the video conference. One or more processors may also be configured to implement additional software entities (such as applications), as described in more detail below.

[0075] Each video terminal system may include video conferencing hardware. The video conferencing hardware may include one or more devices that communicate with one or more processors of the video terminal system for video conferencing. For example, the video conferencing hardware may include a camera and / or a display screen for sharing images and / or video streams during a video conference call. Specifically, a video stream may be transmitted from a camera via one of the media input channels to be included in the video stream of the video conference call. One or more additional video streams may be received from the video stream of the video conference call and displayed on the display screen. In some examples, the video terminal system may include multiple cameras and displays.

[0076] Video conferencing hardware may also include amplifiers and / or microphones for sending and receiving audio streams of conference calls. Specifically, video conferencing hardware may include one or more audio amplifiers (e.g., as a local broadcast speaker or as an amplifier included in a personal headset) for outputting the audio content of the conference call. Video conferencing hardware may include one or more audio microphones for providing an audio stream for the content stream of the conference call. In another example, the control interface and video conferencing hardware may include separable units that communicate with each other (e.g., via a wired or wireless connection).

[0077] Each component of the video conferencing hardware (i.e., camera, display screen, microphone, amplifier, etc.) can be provided as one or more integrated units or as separate devices. Video conferencing hardware and control interfaces can also be provided as one or more integrated units or as separate devices. For example, a controller with a touchscreen can serve as both a display screen and a control interface.

[0078] Each video terminal system may also include a controller, which may also be referred to as a control interface or control interface device. The controller can be any device that includes user input controls (such as a user interface) for enabling a user to control the video terminal system. For example, a control interface device may be a tablet computer including a touchscreen. The control interface device may include a user input device that a user can use to issue control signals to the video terminal system by engaging with a user input device on the touchscreen (e.g., by pressing a button). Other examples of control interface devices may include a keyboard, a mouse, a control panel with mechanical buttons, and a voice control device. For example, a user can use the control interface device to start and end a conference call, add or remove a participant from a conference call, adjust the volume level of the speakers in the video terminal system, and configure display settings. Multiple control interface devices may be provided that are connected (e.g., via wired or wireless connections) to the rest of the video terminal system.

[0079] The controller and video conferencing hardware can also be provided as a single unit. For example, the video terminal system can be a single desktop or handheld unit, which can also be referred to as a video terminal device. In yet another example, the control interface and video conferencing hardware can include separable units that communicate with each other, for example, via endpoints (e.g., via wired or wireless connections).

[0080] exist Figure 2aIn the example shown, each corresponding endpoint 203a, 203b can be connected to a corresponding controller 202a, 202b, displays 201a, 201b, 205a, 205b, and a corresponding camera 204a, 204b. In this example, endpoints 203a, 203b form a connection 207 across network 206 (e.g., the Internet) to establish communication by streaming video and audio from cameras 204a, 204b and / or microphones. Audio and video can be output at displays 201a, 201b, 205a, 205b and / or speakers.

[0081] The video terminal system is associated with a centralized management system (also known as a centralized service or server), which in turn... Figure 2a It is shown at position 221 in the text.

[0082] Endpoint 203a can communicate with a media source. In one embodiment, the media source can be a physical device dedicated to providing media. For example, it can be a camera, microphone, or digital whiteboard. In another embodiment, the media source can be a computing device that can provide media and also perform other functions. The computing device can be, for example, a tablet, smartphone, or other personal computing device. The computing device can store, in a non-transient form, code for providing media that can be executed by one or more processors of the computing device. For example, the code can implement a whiteboard function that can receive drawing input from a user and output a media image representing that drawing input. Alternatively, the code can implement a slideshow function that can output images of presentation slides stored on the device. In this embodiment, the media source can be an application defined by the stored code and executed by the device's processor.

[0083] exist Figure 2a In the example shown, endpoint 203a can be communicatively connected to auxiliary computing device 220, which serves as a media source. The auxiliary computing device can be located remotely from the video terminal system 250a. The auxiliary computing device can be a separate entity. Auxiliary computing device 220 can be, for example, an interactive whiteboard, mobile phone, tablet computer, or laptop computer. Auxiliary device 220 can also be communicatively connected to network 206. Device 220 can be communicatively connected to a central management system 221. Endpoint 203a can be connected to device 220 via direct channel 225, and can also be connected via management system 221, through... Figure 2a Channels 223 and 224, as shown, communicate with device 220. The centralized management system 221 will be described in more detail below.

[0084] One or more processors of computing device 220 are configured to implement a first software entity capable of outputting media from a media source on any of a plurality of output channels. The one or more processors of computing device may also be capable of implementing one or more other software entities, as described in more detail below. The first software entity may be an application, computer program, or software package running on auxiliary computing device 220. The application may be implemented by one or more processors of computing device 220. For example, the application may be one or more of the following: document sharing, whiteboard presentation, multimedia presentation, training, review, voting, or question submission application. The media source may be configured to output media through one or more of a plurality of output channels. The media may include video data and / or audio data. In some examples, the media may include other types of data that can be received from the media source. For example, the media may include computer files, still images, or text.

[0085] It may be desirable to send media from a media source to a video terminal system for inclusion in a video conference established between endpoint 203a and another endpoint 203b. For example, a user may want to share notes or drawings from a virtual whiteboard or tablet. Therefore, the media source can provide supplementary media streams to be included in the video conference.

[0086] Auxiliary equipment 220 and video terminal system can directly (e.g., via) Figure 2a (225) or establish a control connection through the central management service 221.

[0087] Auxiliary device 220 can connect to centralized service 221 via channel 223. Centralized service 221 can manage the connection and / or logical association between the video terminal system and one or more media sources (such as auxiliary computing device 220). Logical association can be instantiated by storing data defining the link between the terminal system and the corresponding media source. Once the logical association has been instantiated, it can be used to help and / or allow the establishment of an active media link between the terminal system and the corresponding media source. The logical association may include the address of the terminal system and / or the corresponding media source. In one mode of operation, when the terminal system wants to establish an active media link with a media source, the terminal system can query server 221 to obtain the address of the terminal system and the media source with which it has a stored logical association. The terminal system can then use that address to access the media source over the network. In another mode of operation, the terminal system can request server 221 to establish a media link with the media source, and in response, server 221 can determine the address of the terminal system and the media source with which it has a stored logical association, and can then use that address to establish the media link. Channel 223 can be a control channel and / or a media channel. Endpoint 203a can also be connected to centralized service 221 via channel 224. Channel 224 can be a control channel and / or a media channel. Therefore, one or more processors of the video terminal system can be configured to establish a control channel between the video terminal system and the auxiliary device, and to establish a media channel between the video terminal system and the auxiliary computing device.

[0088] Channels 223 and 224 are channels through which media and / or control signals can be transmitted from device 220 to the video terminal system via server 221. Channel 223 is a separate channel from channel 207, on which media streams for video conferencing are transmitted via network 206 to another endpoint 203b. Therefore, the channel for transmitting media and / or control signals between the video terminal system and the media source at auxiliary device 220 is a separate channel for transmitting media to another endpoint 203b during video conferencing.

[0089] A secure connection can exist between the video terminal system and the centralized service 221. A secure connection can also exist between the computing device 220 and the centralized service 221. A secure connection between the video terminal system and the centralized management service 221 can be established via channel 223. A secure connection between the auxiliary device 220 and the centralized management service 221 can be established via channel 223. The communication channel 223 between the auxiliary device and the centralized management system, and the communication channel 224 between the video terminal system and the centralized management system, enable the two devices to securely exchange parameters, such as identity, network address, and information about capabilities. These communication channels can also be used to exchange session keys for secure direct communication.

[0090] Within centralized service 221, an association is configured between auxiliary device 220 and endpoint 203a. By associating the auxiliary device with the centralized service and creating a logical association between the auxiliary device and the video terminal system within the centralized service, simplified access to auxiliary media streams used in video conferencing can be achieved. This association between devices allows the centralized management system to record details of the auxiliary device, such as its location and authorization. This allows media from media sources to be easily directed to the video terminal system based on the logical association.

[0091] Logical association allows video terminal systems to be authorized to access media sources at auxiliary devices without requiring processes such as device discovery, device trust, and selection of the exact media source (such as a screen or application window) for the sending device.

[0092] Furthermore, a centralized management system can allow persistent logical associations to be established between auxiliary devices (such as those co-located with the video terminal system). These associations can then be used to supply media output by the auxiliary devices to multiple different video conferences. In other words, the logical associations formed at the video conferencing server between the computing device and the video terminal system can persist across multiple video conferences between the video terminal system and one or more other video conferencing endpoints, without requiring the formation of a logical association for each individual video conference session.

[0093] The logical association between the video terminal system and the computing device 220 also allows the exchange of capabilities between the computing device 220 and the video terminal system by recording details of the capabilities of each device at a centralized service location.

[0094] During a video conference, the video terminal system can also discover one or more new auxiliary devices within its communication range, for example, via Bluetooth or audio pairing, and establish a logical association between the media source at the location of the one or more new devices and the video terminal system at the video conferencing server. Existing logical associations can also be updated if one or more attributes (such as the location or authorization of the auxiliary device) change.

[0095] As mentioned above, the central service can store the logical associations of multiple auxiliary devices, such as auxiliary devices coexisting with the video terminal system (e.g., in the same room or building), so that commonly used devices do not need to be configured for each video conference call.

[0096] The list of auxiliary computing devices associated with the video terminal system at the video conferencing server can be restricted for specific periods, for example, restricting devices outside the range of the video terminal system from being used as media sources. The proximity of the auxiliary computing device to the video terminal system can be determined using technologies such as Bluetooth or audio pairing. If it is determined that the auxiliary device is outside the range of the video terminal system, this can be indicated at the centralized management service, preventing the device from being used to share its media.

[0097] As described above, one or more processors of the video terminal system can be configured to implement a first software entity for supporting video conferencing. The first software entity may be able to receive media input from any of a plurality of input channels. For example, the first software entity may receive media from camera 204a. One or more processors of the video terminal system (e.g., a main processor) may be able to implement a second software entity for receiving media from a media source (e.g., via server 206, via channels 223 and 224) and supplying that media to the first software entity via one of the analog input channels. For example, the video terminal system may emulate an HDMI port, to which a computing device (such as a laptop computer) can traditionally be connected to supply media for inclusion in a video conference. Generally, one way to supply media from a media source to a software entity supporting video conferencing is through software emulation of the software entity's input. This approach allows media to be supplied using the techniques described herein, even if the software entity does not have an input port pre-configured to receive data from a media source. The input to be emulated may be supported by a logical port provided by the terminal system's operating system. In normal operation, a driver for this port may link the logical port to a pre-existing input (such as a physical port, e.g., an HDMI or USB port). An alternative driver can be implemented. This driver can supply media from a media source to the same logical port. The software entity can then process the data received at the logical port in a normal manner, even if the data originates from the media source. In this mode of operation, the input from the media source can be referred to as a virtual port.

[0098] One or more processors of computing device 220 may be configured to implement a first software entity capable of outputting media from a media source on any of a plurality of output channels. The first software entity may be an application, such as a conferencing collaboration application, a virtual whiteboard, or a presentation system. This application is configured to provide media output to a local physical device (such as a screen or speaker) of computing device 220 via any of the plurality of output channels. The computing device may also be able to implement a second software entity for receiving media from a media source and supplying the media to a video terminal system via one of the analog output channels. This may allow media to be redirected to the video terminal system.

[0099] Centralized service 221 can be used to simulate media output devices on auxiliary equipment to redirect output media to the video terminal system, and to simulate media input devices on the video terminal system to receive redirected media from the auxiliary equipment.

[0100] In other words, the processor and other hardware implementing the physical endpoints for the video terminal system are typically configured to run software that implements a logical video conferencing endpoint of one or more video conferencing platforms. For the software to operate in normal use, it receives input from a physical receiving or recording entity connected to a physical port of the hardware. For example, the software may receive audio from a microphone attached to an audio support port of the hardware, or video from a video support port of the hardware. These can be physical ports, such as HDMI or USB ports, exposed externally to allow receiving or recording devices to connect to them. The video conferencing software may be provided by the operator of the video conferencing platform, which may be independent of the hardware manufacturer. The video conferencing software may be configured to support protocols specified by the operator of the video conferencing platform. Therefore, the video conferencing software may be configured to receive audio and video data from the physical ports of the video conferencing endpoint hardware. As described herein, the operation of the endpoint can be enhanced by additional software running on the processor of the hardware. This software may be part of an application, driver, or operating system running on the hardware. This additional software may emulate the physical ports of the hardware, preferably physical ports exposed or potentially exposed externally to the hardware, such as USB or HDMI ports. Through a simulated port, the additional software presents data to a software entity running on the hardware, which consumes the data received by the port, such that the consuming software can consume the data as if it had already been received at the physical port. This additional entity can implement a logical interface, thereby allowing it to receive data from sources external to the hardware. For example, the additional entity can receive the data via a wireless data link with a wireless transceiver included in the hardware. This could be a WiFi (IEEE 802.11) link or a Bluetooth link, for example. Video sources (including still image sources) and / or audio sources (which may be the same entity) can connect to this logical interface to provide data to the additional entity. This data can be video and / or audio data. Thus, a media source (e.g., video and / or audio) can provide data to video conferencing software as if the video conferencing software were connected to a port from which it could consume its data. The video conferencing software can behave as if the media source were connected to that port, when in fact the media source may not be connected to that port but is instead providing media data to another software that simulates the port from which the video conferencing software is configured to receive media data.

[0101] exist Figure 2aIn the example shown, the logical association between the video terminal system 250a and the auxiliary device 220 is established at a centralized service 221, which in this example is located in the cloud. That is, the centralized service is implemented by one or more servers located remotely from the video terminal system and the auxiliary device, and communicates with the video terminal system and the auxiliary device via the Internet. However, the system can also operate without communicating with a video conferencing server and / or without a cloud. For example, the logical association between the video terminal system and the auxiliary device can also be established using other technologies, such as Bluetooth, voice pairing, proximity pairing, or user interaction.

[0102] Figure 2b An example is shown of connections and / or signals transmitted between various entities in system 200 for inserting media from a media source at auxiliary device 220 into a video conference between video terminal system 250a and another video terminal system. For simplicity, Figure 2b Only endpoint 203a of the video terminal system 250a is shown. The processor at endpoint 203a can implement multiple software entities. In this example, the processor implements mechanisms for supporting video conferencing (i.e., mechanisms for communication between endpoint 203a and another endpoint of a video conferencing system, such as...). Figure 2a A first software entity 241 and a second software entity 242 transmit media between endpoints 203b) in the system. The second software entity can, as described above, simulate one of the input channels on which the first software entity 241 can receive media input. The second software entity 242 receives media from a media source at auxiliary device 220 and supplies media to the first software entity 241 via one of the simulated input channels. The second software entity can operate independently of the first software entity. The first software entity and / or the second software entity can be independent of the hardware manufacturer.

[0103] In this example, the logical association between the media source and the video terminal system is established via communication with management service 221 in the cloud over the Internet. The two-way link between the second software entity 242 and the media source at auxiliary device 220, shown at 251, establishes an authenticated connection between endpoint 203a and the media source. Using the signal shown at 252, the second software entity 242 requests a shared session ID from management service 221. Using the signal shown at 253, management service 221 transmits the shared session ID to the second software entity 242. Using the signal shown at 254, the second software entity 242 transmits the shared session ID to auxiliary device 220. The two-way link shown at 255 establishes a connection between the media source at auxiliary device 220 and management service 221 using the shared session ID. Using the signal shown at 256, the media source at auxiliary device 220 sends media to the second software entity 242 via management service 221 using the shared session ID. Using the signal shown at 257, the second software entity 242 sends media to the first software entity 241 to support video conferencing. The first software entity 241 can then include the media in a video conference established with another video terminal system.

[0104] exist Figure 2b In the example shown, media is transmitted via cloud and sent to endpoint 203a for insertion into the video conference. However, in other embodiments, media may not be transmitted via cloud and can be sent from the media source to the endpoint of the video terminal system via a direct channel. For example, endpoint 203a and auxiliary device 220 can be directly connected via a wireless or wired connection, and media for insertion into the video conference can be sent from the media source at auxiliary device 220 to endpoint 203a of the video terminal system via this direct connection.

[0105] The media output from the media source can be transmitted to be output at the video conferencing hardware of the video terminal system and inserted into a video conference established with another video conferencing endpoint, so as to facilitate sharing the media output from the media source with remote conferencing parties.

[0106] One or more processors of the video terminal system can be configured to receive control signals from control unit 202a or auxiliary computing device 220 to enable remote viewing of media via video conferencing hardware and / or the content stream of a video conference. For example, one or more processors of the video terminal system can receive control signals from control interface device 202a and, based on these control signals, determine whether or not media on one or more video conferencing hardware units can be remotely viewed and / or whether or not media on one or more channels of a video conference can be remotely viewed. In this way, the user of the video terminal system can control which components of the video conferencing hardware of the video terminal system output media and / or which remote participants in the video conference can view the media (i.e., which endpoints the media is sent to).

[0107] Optionally, the video terminal system may establish a user interface extension for controlling various aspects of the auxiliary equipment. Alternatively, the computing device may establish a user interface extension on the auxiliary equipment for controlling various aspects of the video terminal system.

[0108] The control devices 202a and 202b of each video terminal system may each include a user interface. The auxiliary device 220 may also include a user interface. In some embodiments, these user interfaces may each include a touchscreen on the display screen. Users can interact with one or more user input devices on the user interface (e.g., buttons displayed on the touchscreen). The user interfaces for the video terminal system and computing device 220 may be configured to provide simplified access to auxiliary media streams for use within video conferencing, as described in more detail below.

[0109] The user interface of the video terminal system may include controls for requesting a media source at an auxiliary device to send its current media output to the video terminal system for inclusion in the current video conference, or for stopping the sending of the current media output if it is currently being inserted into the video conference.

[0110] The user interface of the video terminal system (which may be part of the controller 202a of the video terminal system) may include a user input device. For example, if the user interface is in the form of a touch screen, the user input device may include a button located on the touch screen. Alternatively, the user input device may include buttons, switches, or levers located on a mechanical control panel. One or more processors of the video terminal system may be configured to receive signals in response to user input at the user input device. In response to receiving a signal, the video terminal system may receive media from a media source to insert into a video conference. One or more processors of the video terminal system may establish a media channel (e.g., via network 206) between the video terminal system and the media source in response to receiving a signal.

[0111] In some implementations, the user interface of the auxiliary device includes controls for transmitting current media output (such as audio and video) to a video terminal system via content streaming for transmission to remote participants in the current video conference, or for stopping the transmission of current media output to the video terminal system. The user interface of computing device 220 may include a user input device. For example, in the case of a touchscreen user interface, the user input device may include buttons located on the touchscreen. Alternatively, the user input device may include buttons, switches, or levers located on a mechanical control panel. One or more processors of the auxiliary computing device may be configured to receive signals in response to user input at the user input device. In response to receiving a signal, the auxiliary computing device may transmit media from a media source to the video terminal system for insertion into the video conference.

[0112] The user interface of the auxiliary device and / or terminal system can be updated due to the establishment of a connection between the video terminal system and device 220. This user interface update can be triggered by negotiation between the auxiliary device and the terminal system, or by an instruction from centralized service 221.

[0113] The updated user interface allows users to select an auxiliary device as the media source for the current video conference, for example, by pressing a button on the user interface. When a user enters a request into the user input device, a media channel can be created between the auxiliary device 220 and the terminal system 203a. This media channel allows media from the media source to be streamed from the auxiliary device 220 to the content stream of the terminal system 203a and transmitted to another endpoint 203b via the video conferencing communication channel 207.

[0114] In another example, in response to the detection of a virtual port on the video terminal system, an input device (e.g., a button) can be created. The virtual port can be selected according to general media selection criteria of the user interface, and the action of selecting the virtual port can cause the creation of a media stream from the auxiliary device to the video terminal system. The user interface of the auxiliary computing device can advertise the virtual port as a way to send media from the computing device to the video terminal system.

[0115] Figure 3 An example of auxiliary device 220 is shown in more detail. In this example, application 532 is launched in environment 530 running on device 220. Application 532 may correspond to a first software entity implemented by the computing device discussed above.

[0116] The environment may include a control signal interface, that is, programming configured to transmit control signals from the user interface of the auxiliary computing device and / or video terminal system to the application. The environment may include a media signal interface, that is, programming configured to transmit media signals between the application and the video conferencing hardware and / or the content stream of the video conferencing at the video end / system.

[0117] The environment may include a data mapping interface configured to relay control signals from the user interface of the auxiliary computing device and / or video terminal system to the application, and to relay media signals between the application and the content stream of video conferencing hardware and / or video telephony. The data mapping interface may be a virtual interface that defines the boundaries of the application environment and acts as a virtualization layer by relaying data between the application within the application environment and the software and hardware outside the application environment.

[0118] The environment can demonstrate a virtualized or standardized operating system interface to application 532 while interfacing with physical devices (e.g., media output devices such as screens and speakers) provided by device 220. This virtualization layer includes a virtualized media output layer 531 and a virtualized media input layer 533.

[0119] A user interface for the auxiliary computing device 220 is shown at 534. User interface 534 can provide interfaces such as those programmed in application 532, such as touchscreen controls required by the application. Furthermore, user interface 534 can be enhanced to control video transmission, such as sending video and audio output to a video terminal system for inclusion in a video conference. Environment 530 communicates with cloud service 221 and can use this connection to determine user interface adaptation either through negotiation with the cloud service or through negotiation with the video terminal system. In this example, media output layer 531 provides video and audio transmission and compression. When the video terminal system (e.g., via server 206) receives an instruction from user interface 534 that the user expects media to be sent from the application to a video conference, it can do so via cloud service 221 (through... Figure 2a Signals are sent to video conferencing systems (such as channels 223 and 224) or via direct control channels. Figure 2a In connection 225 (which can be a wireless connection), a media channel is established and the video terminal system is instructed to incorporate that media channel as the content media source for the video conference (in connection 207). In this way, the device is always available for the video terminal system to use as a media source. The video conferencing server can have multiple potential devices pre-configured before the video conference. One or more of these devices can be activated to send media data to be incorporated into the video conference.

[0120] The application environment can be configured to convert media signals from a media source, having a first format, into media signals of a second format that are part of the content stream of video conferencing and / or video conferencing hardware. In other words, the media source can be configured to output media signals having a first format, which can then be converted into media signals of a second format that are part of the content stream of video telephony and / or video conferencing hardware. For example, the output media signals may include video signals, and the application environment can be configured to reformat the video data to map screen size, video format (e.g., compressed vs. uncompressed), pixel count, etc., between formats compatible with the output device of the auxiliary computing device and the content stream of the video conferencing hardware and / or video telephony. In other examples, the media signals may include audio signals, image or file data, etc., converted from a first format compatible with the output device of the auxiliary computing device to a second format.

[0121] Therefore, audio and / or video output from the application can be provided by the application environment to be included. Figure 2a The video call content stream established between endpoints 203a and 203b enables the application's video output to be simultaneously sent to the local display 205a and the remote endpoint 203b for display on the remote display 205b and / or enables audio output to be output at both the local and remote speakers.

[0122] In one implementation, the media output ports (such as video and audio) of the auxiliary device can be copied to the corresponding virtual device. In this way, applications running on the auxiliary device providing the media stream are unaware of the existence of media integration with the video terminal system.

[0123] In another implementation, the application running on the auxiliary device can be encapsulated within an environment that simulates the media input and output services of the application, while simultaneously reformatting the media, such as by scaling, cropping, or completely replacing the user interface, to make the media suitable for delivery to the local physical device and create a media stream for use by the video terminal system. This environment can be configured to transmit control signals from the user interface of the computing device and / or the video terminal system to the application, and to transmit media signals between the application and the video terminal system.

[0124] As described above, in some implementations, the auxiliary device can process the media output from the media source to provide a compressed representation of the media to be sent to the video terminal system. The aforementioned virtual device or encapsulation environment (i.e., the application environment) can utilize media compression algorithms such as H.265 and Advanced Audio Coding (AAC) to compress the media output from the application to provide a more suitable stream for transmission to the video terminal system.

[0125] In some implementations, one or more virtual media source devices may be provided on the video terminal system. These virtual media source devices emulate media sources at auxiliary devices. Media from media sources at auxiliary computing devices can be presented to the video terminal system firmware via the virtual media source devices. The virtual media sources may also optionally decompress the media. In this way, the video terminal system firmware (which will typically understand the raw media stream introduced from a physical media port such as HDMI) requires minimal adaptation to acquire the media stream from the media source at a separate device and inject the media stream into the video conference in a manner similar to a physically provided media stream (e.g., a media source connected to the video terminal system via an HDMI port on the system).

[0126] This can be implemented as an additional media sink component incorporated into an existing media presentation system for auxiliary equipment, and / or as an additional media source device integrated into a video terminal system. Sources and sinks can be associated through logical relationships established at a centralized management service. Media can be sent from the sink to the source based on these logical relationships.

[0127] In other words, in normal operating mode, the video terminal system can receive media from devices connected to it, typically through a dedicated physical port. These devices can be, for example, cameras, microphones, or digital whiteboards. The endpoints can be capable of supporting video conferencing with remote endpoints supported by an alternative connection between the conferencing server 222 and the endpoints. The conferencing server 222 can perform tasks such as authenticating endpoints or their users, forwarding video conferencing traffic between endpoints, and buffering video conferencing traffic in the event of retransmissions. As an alternative to providing media to one of the endpoints, media can be provided via a data connection to a media provider (e.g., a packet-based connection). The central management server 221 can help link media providers to endpoints. The endpoint can then add media from the media provider to the video conferencing traffic and send that traffic to the server for forwarding to another endpoint in the video conference. The central management server can be implemented as a single physical processing unit or can be a logical function distributed among multiple physical processing units.

[0128] Figures 4 to 6 A video conferencing system according to another embodiment is illustrated, wherein an auxiliary computing device is configured to operate a third-party application. This document combines... Figures 4 to 6 Each of the described features can be combined with the above text and Figure 2. Figure 3 Any or all of the described combinations of features.

[0129] Figure 4A diagram illustrating some components of a video conferencing system is shown. The video terminal system shown at 300 is communicatively connected to an auxiliary computing device 350, which is configured to operate a third-party application. The third-party application provides a media source at the auxiliary computing device. In this example, the video terminal system 300 includes a video endpoint device 310, connected video conferencing hardware 330, and a control interface device 320. The endpoint device 310 is connected to the Internet 360 to exchange video telephony content streams with remote conferencing parties. The endpoint device 310 is configured to send media signals to and receive media signals from the video conferencing hardware, which can form part of the video telephony content stream. Figure 4 In this system, video conferencing hardware 330 includes a camera 3332 and a display screen 3334 for recording and displaying video clips. Endpoint device 310 is also configured to receive control signals from control interface device 320 to control video terminal system 300.

[0130] In this example, auxiliary computing device 350 is configured to run a third-party application 354 hosted within an application environment 352 provided on auxiliary computing device 350. Furthermore, the application environment 352 on auxiliary computing device 350 is connected to a management agent service 370 via the Internet 360. Management agent service 370 contains information about the identity and permitted behaviors of each auxiliary computing device that can communicate with video terminal system 300. As described above, a logical association between video terminal system and one or more auxiliary computing devices can be established at service 370. Examples of permitted behaviors may include a list of authorized third-party applications that can run on auxiliary computing device 350, details about how each application 354 can run, such as whether user authorization from the user is required, whether application 354 can be shared with any or specific remote conferencing party, and / or whether application 354 must be restricted in terms of device or network access. Management agent service 370 is configured to send configuration management controls to application environment 352.

[0131] Figure 5 Another example of a video conferencing system is shown. Figure 5 In this system, the video conferencing hardware 330 of the video terminal system 300 also includes a speaker 3336 and a microphone 3338 for playing and recording audio data of the video call content stream. Furthermore, the video terminal system 300 includes a network interface 358 for connecting to the Internet 360. (As...) Figure 5As shown, the application environment 352 includes an environment interface 356 for relaying signals between the third-party application 354 and other components of the auxiliary computing device 350. The environment interface 356 (which may also be referred to as the data mapping interface 356) is configured to transmit control signals from the control interface device 325 of the auxiliary computing device to the third-party application 354. The environment interface 356 is also configured to transmit media signals 385 and 386, where 385 is an audio signal and 386 is a video signal, via a server 370 between the third-party application 354 and the video endpoint 310, such that media can be output via the network interface 380 to one or more components of the video conferencing hardware 330 and / or the content stream of a video telephony conference. The audio stream 385 and video stream 386 received from the third-party application 352 via the data mapping interface 316 to be included in the content stream of the video telephony are relayed to the speaker 3336 and the display screen 3334, respectively. Similarly, the audio stream 348 received from the microphone 3338 and the video stream 347 received from the camera 3332 are relayed to endpoint 310 to form part of the video call content stream, which is sent to another endpoint via network interface 358 and the Internet 360.

[0132] In use, a user can request the installation of a third-party application 354 (e.g., an Android application) on the auxiliary computing device 350 via the user interface of the control interface device 325. The control interface device 325 then instructs the auxiliary computing device to create an application environment 352 for the third-party application 354 to run, and then downloads the third-party application 354 into the application environment 352 on the auxiliary computing device 350.

[0133] The third-party application 354 may have prerequisites for running on a typical device. Creating the application environment 352 may include allocating CPU and memory resources for the application 354, as well as virtual devices and / or interfaces (such as a display, keyboard, and touchscreen). When the third-party application 354 is launched into the created application environment 352, the prerequisites for the third-party application 354 are provided, such that from the perspective of the third-party application 354, the application environment 352 emulates a typical device, such as a regular mass-market tablet or phone.

[0134] The user interface of the auxiliary computing device can then be used to connect to the third-party application 352 in various ways. In one example, a virtual video stream can be relayed across the connection between the application environment 352 and the control interface device 325, allowing the application video output of the third-party application 354 to be displayed on the control interface device 325. If the aspect ratio and resolution of the screen provided on the control interface device 325 are similar to the requirements of the third-party application, a full-screen video view can be displayed. In other examples, the user interface environment on the control interface device 325 can be configured to display the application video output in a window on the screen of the control interface device 325. In some examples, a user of the control interface device 325 can zoom in and out of the video using zoom gestures such as pinching two fingers. This example is useful, for example, for setting up the third-party application 354 by entering user credentials, where the information should not be shared with other participants in the conference room. In another example, video output from the third-party application 354 can be connected to or redirected to the content stream of a display screen 3334 or a video teleconference. If the display screen 3334 includes a touchscreen, touchscreen gestures can be relayed back to the third-party application 354.

[0135] In some examples, when the auxiliary computing device 350 includes a graphics processing unit (GPU) and / or a screen and / or a touch screen on the same unit, a third-party application 354 hosted on the device 350 can directly access the GPU, display, and / or touch screen (i.e., without a virtualization layer). This can be useful for improving the efficiency and performance of the third-party application 354.

[0136] Data mapping interface 356 can be configured to convert control signals of a first format from control interface device 325 into application control signals of a second format for controlling third-party application 354. For example, control interface device 325 may include a physical touchpad or virtual touchpad simulated by the touchscreen of control interface device 325. Touch gestures from the virtual touchpad can be relayed as touch gestures via the connection between control interface device 325 and third-party application 354 through data mapping interface 356. In this example, application environment 352 can be configured to correctly map the gesture positions between the touchpad or virtual touchpad of control interface device 325 and the target virtual display of third-party application 354, which may have different sizes, taking into account any active clipping, scaling, screen size differences, and aspect ratios between the touchscreen of control interface device 325 and the target virtual display of third-party application 354.

[0137] In another example, the auxiliary computing device 350 can be programmed with custom instructions associated with a specific third-party application 354 to form an application environment and / or user interface environment. The custom instructions can be configured to set up an enhanced user interface on the control interface device 325, replacing the conventional user interface of the third-party application 354. This allows for a user interface more suited to the hardware of the auxiliary computing device 350, instead of a user interface for a third-party application that might be better suited to different hardware (e.g., a smartphone).

[0138] For example, an enhanced user interface (API) provided on an auxiliary computing device may include application-specific function buttons, a virtual keyboard, a dial pad, or directory lookup or navigation elements configured to control a third-party application 354. The application environment 352 may be configured to translate control signals from the enhanced API into control signals that mimic the gestures and keystrokes expected by the third-party application 354.

[0139] Custom instructions for specific third-party applications can be provided by a central server (e.g., on Internet360), which can be downloaded by an auxiliary computing device. Custom instructions can include a combination of structured data, code, and / or portable code. In some examples, custom instructions can include an application programming interface (API) provided by the third-party application.

[0140] In some examples, the list of authorized third-party applications permitted to be operated by the assistive computing device can be centrally managed by a management server or cloud service (such as services 221 or 370 in this document). The assistive computing device can connect to the management server or cloud service to obtain the list of authorized applications. If the desired third-party application is included in the list of authorized applications, the assistive computing device can download the third-party application from a publicly accessible repository (e.g., from the Google Play Store), from a private repository (such as a private server), from the management server itself, or from the cloud service. This is useful for reducing the likelihood of malicious third-party applications being installed on the assistive computing device. Additionally or alternatively, a management interface can be provided to allow administrators to enable or disable authorization for specific third-party applications. In some examples, user login or a video terminal system device identifier may be required to enable or disable the download of third-party applications.

[0141] In another example, the management server or management interface can be configured to allow guest applications not included in the list of authorized applications to be operated by the assisted computing device. This can be based on user login or on the video terminal system device identifier. In yet another example, the management server or cloud service can be configured to monitor the use and behavior of third-party applications. In yet another example, an application environment containing third-party applications can be provided on the control interface device of the assisted computing device, which can also host a user interface environment for user interface applications.

[0142] Figure 6 A diagram is shown of a video terminal system 500 communicatively connected to an assistive computing device 550, according to another example. The video terminal system 500 includes an endpoint 510, a control interface device 520, a camera 5332, and a display screen 5334. In this example, an application environment 512 containing a third-party application 514 is provided on a cloud service 580, to which the assistive computing device 550 communicatively connects. A video stream from the third-party application 514 in the application environment 512 is transmitted from the cloud service 580 to the assistive computing device 550 via the Internet 560. The video stream from the application can then be transmitted via a service 577 (which stores the logical association between the assistive computing device 550 and the video terminal system 500) to the video endpoint device 510 and to the display screen 5334, and can be included in a media stream to another endpoint as described above.

[0143] Figure 7 The steps of an exemplary computer implementation of a method for conducting a video conference at a video terminal system configured to act as an endpoint in a video conference are illustrated. In step 701, the method includes establishing a logical association with a media source (which may be located at an auxiliary computing device) communicatively connected to the video terminal system. In step 702, the method includes receiving media from the media source, wherein the media is directed to the video terminal system according to the logical association. In step 703, the method includes inserting media into a video conference established with another video conference endpoint (this can be accomplished by communicating with a video conference server). This method can advantageously allow media from a media source at an auxiliary computing device to be inserted into a video conference according to the logical association established between the media source and the video terminal system.

[0144] Figure 8The steps of an exemplary computer-implemented method for communicating with a video terminal system configured to serve as an endpoint in a video conference are illustrated. In step 801, the method includes establishing a logical association between a media source (which may be located at or associated with the computing device) and the video terminal system. In step 802, the method includes inducing media transmission from the media source to the video terminal system to be inserted into a video conference established between the video terminal system and another video conference endpoint (this can be accomplished by communicating with a video conference server), wherein the media is directed to the video terminal system according to the logical association.

[0145] Figure 9 The device 901 shown can be configured to operate as a processing element of the video terminal system or auxiliary computing device described herein. Device 901 includes a processor 902, a memory 903, and a transceiver 904. The processor can be implemented as dedicated hardware within the device, such as a processing chip. The memory is arranged to communicate with a corresponding processor. The memory can be non-volatile memory. Each device can include more than one processor and more than one memory. The memory can store data executable by the processor. By executing program code contained in such data, one or more processors can perform the functions described herein. The memory can store such program code in a non-transitory manner. The processor can be configured to operate according to a computer program stored in a non-transitory form on a machine-readable storage medium. The computer program can store instructions for causing the processor to perform its methods in the manner described herein. Device 901 also includes a transceiver 904, which is used for communication with... Figure 2a Channel 207 receives and / or sends data to one or more other entities, such as centralized service 221, communication network 206, or video conferencing endpoints. Each device also includes a power source (such as a battery) or can be connected to mains power.

[0146] The method described in this paper enables media integration between a set of video terminal systems and other networked auxiliary media source devices using a centralized management system at a video conferencing server or otherwise. The centralized management system allows for the establishment of persistent logical associations between auxiliary devices co-located with the video terminal systems, enabling these associations to be used to supply media to video conferences across multiple different video teleconferences. This method is compatible with standard video conferencing software and hardware, as well as existing conference room presentation systems such as interactive whiteboards, and therefore allows for easy integration of new capabilities into pre-existing video conferencing hardware and software.

[0147] The applicant hereby separately discloses each individual feature described herein, as well as any combination of two or more such features, to the extent that such features or combinations can be implemented based on this specification in their entirety according to common general knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein. The applicant notes that aspects of the invention may include any such individual feature or combination of features. In view of the foregoing description, it will be apparent to those skilled in the art that various modifications can be made within the scope of the invention.

Claims

1. A video terminal system for use as an endpoint in a video conference, the video terminal system comprising one or more processors configured to conduct a video conference by performing the following steps: Establish a logical association with the media source, which can communicate with the video terminal system; Media is received from the media source, wherein, The media is directed to the video terminal system according to the logical association; as well as Insert the media into a video conference established with another video conferencing endpoint; The video terminal system's one or more processors are configured to implement a first software entity for supporting the video conference, the first software entity being capable of receiving media input from any of a plurality of input channels, and wherein the video terminal system is capable of implementing a second software entity for receiving media from the media source and supplying the media to the first software entity by simulating one of the input channels.

2. The video terminal system as claimed in any of the preceding claims, wherein, The one or more processors are configured to establish a control channel between the video terminal system and the media source.

3. The video terminal system as claimed in any of the preceding claims, wherein, The video terminal system includes a user interface for controlling various aspects of the video terminal system and / or the media source.

4. The video terminal system as described in claim 3, wherein, The user interface includes a user input device, wherein the one or more processors are configured to receive a signal in response to user input at the user input device, and to receive media from the media source in response to receiving the signal for insertion into the video conference.

5. The video terminal system as claimed in any of the preceding claims, wherein, The video terminal system includes video conferencing hardware, wherein the one or more processors are further configured to cause the media received from the media source to be output at the video conferencing hardware.

6. The video terminal system as claimed in any of the preceding claims, wherein, The one or more processors are configured to establish the logical association with the media source by communicating with a video conferencing server.

7. The video terminal system as described in claim 6, wherein, The video conferencing server is a cloud server or a local area network.

8. The video terminal system as claimed in any of the preceding claims, wherein, The one or more processors are configured to receive the media via a data link that supports transmitting media data from the media source only to the video terminal system.

9. The video terminal system as claimed in any of the preceding claims, wherein, The video terminal system is configured to establish logical associations with multiple media sources, and wherein the video terminal system is configured to receive media from the multiple media sources for insertion into the video conference in response to a user selecting one of the multiple media sources and according to a corresponding logical association in the logical associations corresponding to the one of the multiple media sources.

10. An auxiliary computing device configured to communicate with a video terminal system used as an endpoint in a video conference, the computing device comprising one or more processors configured to perform the following steps: Establish a logical connection between the media source and the video terminal system; and This causes media transmission from the media source to the video terminal system to be inserted into a video conference established between the video terminal system and another video conferencing endpoint, wherein, The media is directed to the video terminal system according to the logical association; The computing device is configured to implement a first software entity capable of outputting media from the media source on any of a plurality of output channels, and wherein the computing device is capable of implementing a second software entity for receiving media from the media source and supplying the media to the video terminal system by simulating one of the output channels.

11. The computing device of claim 10, wherein, The one or more processors are configured to process media data output from the media source to form a compressed representation of the media data, which is to be supplied to the video terminal system for insertion into the video conference.

12. The computing device of claim 10 or claim 11, wherein, The one or more processors are configured to establish a control channel between the media source and the video terminal system.

13. The computing device as claimed in any one of claims 10 to 12, wherein, The one or more processors are configured to establish the logical association with the video terminal system by communicating with the video conferencing server.

14. The computing device according to any one of claims 10 to 13, wherein, The computing device includes a user interface for controlling various aspects of the media source and / or the video terminal system.

15. The computing device of claim 14, wherein, The one or more processors are configured to update the user interface when a connection is established between the media source and the video terminal system.

16. The computing device of claim 14 or claim 15, wherein, The user interface includes a user input device, wherein the one or more processors are configured to receive a signal in response to user input at the user input device, and in response to receiving the signal, to cause media transmission to the video terminal system for insertion into the video conference.

17. The computing device of claim 16, wherein, The user interface is configured to advertise the user input device as a means of sending media from the media source to the video terminal system.

18. The computing device of claim 16 or claim 17, wherein, In response to receiving the signal, the one or more processors are configured to establish a media channel between the media source and the video terminal system.

19. The video terminal system as claimed in any one of claims 1 to 9 or the computing device as claimed in any one of claims 10 to 18, wherein, The media source is an interactive whiteboard, mobile phone, tablet computer, or laptop computer.

20. The video terminal system as claimed in any one of claims 1 to 9 or the computing device as claimed in any one of claims 10 to 18, wherein, The media source is configured to output one or more of the following media types: audio, video, image files, computer files, and text.

21. The video terminal system as described in any one of claims 1 to 9 or the computing device as described in any one of claims 10 to 18, wherein, The logical association is a persistent logical association used in multiple video conferencing sessions.

22. A computer-implemented method for conducting a video conference at a video terminal system, the video terminal system being configured to act as an endpoint in the video conference, the method comprising the steps of: Establish a logical association with the media source, which can communicate with the video terminal system; A first software entity for supporting video conferencing is implemented, the first software entity being able to receive media input from any of a plurality of input channels; A second software entity is implemented, the second software entity being configured to receive media from the media source and supply the media to the first software entity by simulating one of the input channels; Media is received from the media source, wherein the media is directed to the video terminal system according to the logical association; as well as Insert the media into a video conference established with another video conferencing endpoint.

23. A computer-implemented method for use at an auxiliary computing device configured to communicate with a video terminal system used as an endpoint in a video conference, the method comprising the steps of: Establish a logical connection between the media source and the video terminal system; Implement a first software entity that is capable of outputting media from the media source on any of the multiple output channels; A second software entity is implemented, the second software entity being used to receive media from the media source and supply the media to the video terminal system by simulating one of the output channels; as well as The media from the media source is transmitted to the video terminal system to be inserted into a video conference established between the video terminal system and another video conferencing endpoint, wherein the media is directed to the video terminal system according to the logical association.

24. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method as claimed in claim 22 or claim 23.

25. A computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform the method as claimed in claim 22 or claim 23.

Images