Computer implementation method

Abstract

A computer-implemented method of running a third party application (114) using a video terminal system (100) is provided. The video terminal system (100) is used for a video conference and comprises video conference hardware (130) and a control interface device (120). The computer-implemented method includes providing a running application environment (112) for a third party application (114). The application environment (112) is configured to: route a control signal from the control interface device (120) to the third party application (114); and routing media signals between the third party application (114) and the video conference hardware (130) and / or a content stream of the video teleconference.

Description

Technical Field

[0001] This invention relates to a computer implementation method for running third-party applications using a video terminal system, a computer implementation method for installing third-party applications, and a video terminal system. Background Technology

[0002] Video conferencing systems typically include two or more endpoints, each configured to communicate with other endpoints via a data network, telephone network, or other connection to enable two-way or multi-way video and / or audio teleconferences.

[0003] An endpoint (also known as a video terminal system) is configured to enable users to communicate with other users over a network, sending data streams, including audio, video, and other content streams (such as data files, text, or screen sharing), from one endpoint to another during a two-way teleconference. For multi-directional teleconferences, the data streams can be transmitted to a centralized or distributed conferencing network, where they are converted or transcoded to allow multiple participants to communicate and share information.

[0004] Video terminal systems may include dedicated terminal equipment whose primary function is to support video and / or audio teleconferencing. In addition, video terminal systems typically include additional video conferencing hardware, including one or more video monitors, cameras, audio microphones, audio speakers, and controllers that enable users to conduct video teleconferencing.

[0005] In some examples, the video terminal system may be an integrated unit, where a single-function terminal device is integrated with video conferencing hardware in the same device, such as a desktop unit or handheld device. In other examples, the video terminal system may include terminal devices connected to a large screen, control interface devices, and cameras. This configuration is suitable for scenarios such as small conference rooms. In other examples, video terminal systems designed for larger conference rooms or lecture halls may include terminal devices connected to multiple screens, one or more control interface devices, and multiple cameras and microphones. It should be noted that in all the above examples, the terminal device and the control interface device may be the same device.

[0006] The software running on the video terminal devices of a video terminal system is traditionally proprietary software, designed solely for video conferencing, i.e., sharing video and content streams with other endpoints. In some cases, video terminal systems can be used to simply display content from a computer (e.g., a laptop) to participants in a local conference room without establishing a telephone conference.

[0007] Due to the specialized nature of such terminal devices, they cannot perform additional functions beyond their preset programs. If additional functions beyond the capabilities of the dedicated terminal device are required, an external device such as a computer or laptop can be used to perform these functions. Subsequently, the external device can share the content generated by these additional functions as a video conference call stream.

[0008] Therefore, the inventors discovered a need to implement functions beyond those designed by video terminal system manufacturers using the hardware of traditional video terminal systems.

[0009] The current trend is to implement video terminal systems by running video conferencing applications on standard real-time embedded operating systems (rather than proprietary operating systems such as VxWorks, RTLinux, or Windows CE). Designs based on standard mass-market operating systems (such as Android) offer several advantages, including support for diverse hardware architectures and the elimination of licensing fees. Furthermore, standard operating systems (such as Android) support the installation of a wide variety of third-party applications. However, such third-party applications are typically configured to run on tablets and smartphones, using only a single display with a touchscreen. In contrast, video terminal systems may feature multiple displays and separate control interface devices, meaning that the video terminal system may be incompatible with most third-party applications.

[0010] Based on the above considerations, this invention came into being. Summary of the Invention

[0011] According to a first aspect of the present invention, a computer implementation method for running a third-party application using a video terminal system is provided, the video terminal system being used for conducting video teleconferences; the video terminal system includes video conferencing hardware and control interface devices;

[0012] Computer implementation methods include:

[0013] Provides an application environment for a third-party application to run, and the application environment is configured as follows:

[0014] Routing control signals from the control interface device to third-party applications; and

[0015] Routing media signals between third-party applications and video conferencing hardware and / or video telephony content streams.

[0016] The advantage lies in providing an application environment for running third-party applications, allowing these applications to work seamlessly with the video terminal system without encountering incompatibility issues. This enables users of the video terminal system (e.g., conference room equipment) to utilize additional functions beyond video conferencing. For example, the video terminal system can be used to host local and / or in-conference collaboration applications, thereby enhancing the user's meeting experience. Furthermore, this first approach also provides opportunities for third-party application developers to create new applications specifically designed for video conferencing using video terminal systems.

[0017] Importantly, this method enables third-party applications to execute as native applications hosted locally on the video terminal system. This contrasts with existing methods, which may require the video terminal system to access remotely hosted applications via a browser.

[0018] The computer implementation method may also include installing and optionally executing third-party applications within the application environment. Therefore, the functionality of the video terminal system can be expanded by running new applications.

[0019] In one or more embodiments, the application environment may include a control signal interface, i.e., a programming interface configured to route control signals from a control interface device to a third-party application.

[0020] In one or more embodiments, the application programming interface may include a media signaling interface, i.e., a programmable interface configured to route media signals between a third-party application and the content stream of video conferencing hardware and / or video telephony.

[0021] In this invention, the video terminal system refers to a dedicated video conferencing system, which includes function-specific (i.e. dedicated) hardware and / or software for conducting video teleconferences.

[0022] The video terminal system may include system software configured to operate the video terminal system and allow teleconferencing via the video terminal system. Therefore, a computer-based approach may include operating the video terminal system via system software to conduct video and / or audio teleconferencing. The system software may be an application program or a computer program, including proprietary software provided with the video terminal system and configured to operate in conjunction with the video terminal system.

[0023] The video terminal system can be managed by a standard operating system. For example, a standard operating system could be a mass-market operating system such as Android. The operating system can be configured to support a collection of third-party developed mass-market applications designed to run on mass-market hardware such as tablets or smartphones. Other examples of standard operating systems could include Huawei's HarmonyOS or Apple's iOS.

[0024] The video terminal system can be configured to run a single operating system. Through application environment settings, this single operating system can run both video conferencing system software and third-party applications. Therefore, both functions can run on the same device (such as the video endpoint described below) without requiring additional computing units to run third-party applications.

[0025] Third-party applications can be applications, computer programs, or software packages that provide additional functionality. Specifically, third-party applications may have functions beyond supporting video conferencing. For example, third-party applications may include collaboration applications. For instance, applications may be one or more of the following: document sharing, whiteboard writing, multimedia presentations, training, review, voting, or question submission applications. Applications may be configured to run locally or remotely via the internet to include other participants in the video call. Third-party applications may be provided or downloaded from external sources (i.e., third-party applications are not part of the system software originally pre-installed on the video terminal system for enabling teleconferencing).

[0026] Third-party applications can be configured to run on mass-market hardware such as tablets or smartphones (i.e., third-party applications can be configured to be compatible with hardware outside of video terminal systems).

[0027] An application environment refers to a framework or platform configured to accept the deployment of a single application or multiple applications. An application environment includes a predefined set of computing resources that host the applications. An application environment can be configured to provide virtualized or standardized device interfaces to third-party applications, enabling the third-party applications to function correctly even if the physical devices (intended to work with the applications) become disconnected or do not meet the applications' expectations. For example, a virtualized device interface can be configured to reroute data to different physical devices while the application is running.

[0028] In some examples, one or more (e.g., multiple) application environments can be configured to run multiple (each corresponding) third-party applications.

[0029] A control interface device (also known as a controller) can be any device that includes user input controls or a user interface, enabling users to control the video terminal system and / or third-party applications hosted in the application environment. For example, a control interface device can be a tablet computer including a touchscreen. Users can issue control signals to the video terminal system via touchscreen buttons. For example, users can use the touchscreen to start and end conference calls, add or remove participants from a conference call, adjust the speaker volume of the video terminal system, configure display settings, etc. Other examples of control interface devices include keyboards, mice, control panels with mechanical buttons, voice control devices, etc.

[0030] Multiple control interface devices can be configured, which connect (e.g., via wired or wireless connections) to the rest of the video terminal system. For example, the control interface devices can be personal devices such as smartphones or computers. Therefore, the computer implementation method may also include connecting the personal device to the video terminal system and routing control signals between the personal device and a third-party application. The connection between the personal device and the video terminal system can be wireless or wired, such as via Wi-Fi or Bluetooth. The computer implementation method may also include terminating the connection between the personal device and the video terminal system. In this way, the personal device can be temporarily paired with the video terminal system to control the third-party application through that device. Therefore, each user of the video terminal system can control a locally installed third-party application through their own device.

[0031] Video conferencing hardware may include speakers and microphones, where the media signal includes an audio stream. Specifically, video conferencing hardware may include one or more speakers (e.g., as broadcast speakers or personal headsets) for generating audio content for the teleconference and / or audio content received from third-party applications via the application environment. Video conferencing hardware may include one or more microphones for providing audio streams for the content stream of the teleconference and / or providing audio streams to third-party applications via the application environment.

[0032] Video conferencing hardware may include cameras and / or displays for sharing images during video conference calls, where media signals include video streams. Specifically, video streams may be transmitted from a camera to the content stream of the conference call and / or via an application environment to a third-party application. One or more additional video streams may be transmitted from the content stream of the conference call and / or the third-party application to the display via the application environment. Of course, a video terminal system may include multiple cameras and displays.

[0033] Each component of video conferencing hardware (i.e., camera, display screen, microphone, speaker, etc.) can be configured as one or more integrated units or standalone devices. Video conferencing hardware and control interface devices can also be configured as one or more integrated units or standalone devices. For example, a controller with a touchscreen can serve as both a display screen and a user input device.

[0034] The application environment can be set up on the video terminal system. However, in other examples, the video terminal system can communicate with a cloud service (or a standalone server on the local network), and the application environment can be set up on the cloud service (or standalone server). The advantage is that hosting the application environment on a cloud service improves the performance of the video terminal system because potentially resource-intensive third-party environments do not need to be hosted and run on the local hardware of the video terminal system. Furthermore, if provided by a cloud service, the application can be completely isolated from the corporate network where the video terminal system resides. This allows untrusted applications to run without posing a risk to the corporate network.

[0035] The video terminal system may include a processing element for running system software and operating the video terminal system. The application environment may be set on the processing element of the video terminal system or connected to the processing element (e.g., via the Internet to a cloud service).

[0036] For example, a video terminal system may include terminal devices configured to communicate with video conferencing hardware and control interface devices. The terminal devices may be configured to perform the computer-implemented method of the first aspect.

[0037] The terminal device may be a dedicated computing device including a processor, such as a computer or local server, connected (e.g., via a wired or wireless connection) to video conferencing hardware and control interface devices. The terminal device may include a network interface for sending and receiving content streams from video teleconferences.

[0038] In this example, the terminal device may include system software for operating the video terminal system and implementing teleconference execution. The application environment may be natively set within the application environment of the terminal device. Here, the computer implementation method of the first aspect can be executed by the terminal device.

[0039] In some examples, the application environment may be set up on a control interface device (or one of the control interface devices). Here, the control interface device may be configured to perform the computer-implemented method of the first aspect. In this example, the control interface device may also be considered as a terminal device for operating a video terminal system to implement a teleconference. In other words, the control interface device and the terminal device may be integrated into a single unit. In this example, the single unit may include system software for operating the video terminal system and implementing the teleconference.

[0040] This example applies to situations where a participant in a video call wants to download and run a third-party application, but no additional resources from the video terminal system are required to run the application. This example may also be advantageous if the operating system running on the video terminal system is incompatible with the third-party application, while the operating system running on the control interface may be compatible.

[0041] As mentioned above, the control interface device and video conferencing hardware can also be integrated into a single unit. In this example, the video terminal system can be a single desktop or handheld unit, also referred to as a video terminal device.

[0042] In some examples, when the application environment is set on a control interface device, the video terminal system may include one or more additional control interface devices. The application environment can be configured to route control signals from the one or more additional control interface devices to a third-party application. Therefore, even if a third-party application is installed on the control interface device, it can still be controlled by other devices. The application environment can also be configured to route media signals to the one or more additional control interface devices.

[0043] In a further example, as described above, the control interface device and video conferencing hardware may include separable units that communicate with each other (e.g., via wired or wireless connections). Therefore, control signals can be routed between the terminal device and the control interface device via wired or wireless connections. In this way, users can access and control locally hosted third-party applications from a separate device (e.g., their own personal device).

[0044] When the application environment is set up on a cloud service, control signals can be routed between the control interface device and the third-party application hosted in the cloud. In these examples, the video terminal system may include a terminal device that communicates with an application environment set up on a cloud service. The control interface device can be connected to the terminal device via wired or wireless means. Therefore, control signals can be routed between the control interface device and the third-party application hosted in the cloud via the terminal device. In this way, a permanent or temporary connection can be established between the control interface device and the terminal device, which may already be connected to the application environment set up on a cloud service. In this way, users can use any applicable control interface device to launch and control the application, even if the application is hosted in the cloud.

[0045] The control interface device and the terminal device can communicate via a cloud-based relay service. Therefore, in this example, the application environment can reside on the terminal device, and the application environment can receive control signals from the control interface device via the cloud-based relay service. Here, the terminal device itself can be the first control interface device, and the second control interface device can communicate with the first control interface device via the cloud-based relay service.

[0046] The advantage of communicating between terminal devices and control interface devices via the cloud is that the wireless network to which the controller is connected may differ from the network to which the video terminal system is connected, and there may be firewall isolation between the two. Therefore, direct communication between the two devices may not be possible. However, by communicating via the cloud, communication can still be achieved even if the terminal devices and control interface devices are connected to different networks.

[0047] The application environment may include a data mapping interface configured to relay control signals from a control interface device to a third-party application; and to relay media signals between the third-party application and the content stream of video conferencing hardware and / or video telephony. The data mapping can serve as a virtual interface defining the boundaries of the application environment and act as a virtualization layer by relaying data between third-party applications within the application environment and software and hardware outside the application environment.

[0048] Media signals may include video and / or audio data (such as audiovisual signals operating in camera "off" mode), which constitute the content stream of the video teleconference. In some examples, media signals may include other types of data that can be transmitted to and from third-party applications. For example, media signals may include computer files or text (such as other types of audiovisually relevant data, such as still images, computer files, or text that can be shared via a video link). Control signals may include signals for selectively enabling or disabling media signal routing between third-party applications and the video conferencing hardware and / or the content stream of the video teleconference.

[0049] For example, the media signal may include video from a third-party application, used to enable remote viewing of the third-party application via the content stream of video conferencing hardware and / or video telephony. Therefore, a computer-implemented method may include receiving a remote view control signal from a control interface device and, based on the remote view control signal, enabling or disabling remote viewing of the third-party application on one or more video conferencing hardware units, and / or enabling or disabling remote viewing of the third-party application on one or more channels of a video telephony. In this way, a user of the video terminal system can decide on which displays to show the third-party application and which remote participants in the video telephony can view the third-party application.

[0050] The application environment can be configured to convert control signals of a first format from a control interface device into application control signals of a second format for controlling a third-party application (i.e., using a data mapping interface). In other words, a computer-implemented method may include using a data mapping interface to convert (i.e., reformat) control signals of a first format from a control interface device into application control signals of a second format for controlling a third-party application. For example, the application environment can be configured to simulate gestures or key presses in an application when it receives user control signals from the control interface. This may include simulating function buttons, virtual keyboards, dial pads, directory queries, or other navigation elements for a third-party application in the application environment. In some examples, this functionality can be used to transmit control signals from a control interface device that is detachable from the terminal device to a third-party application hosted on the terminal device.

[0051] In this way, an alternative user interface (UI) can be provided for third-party applications running in an application environment. This interface maps UI signals from a first system running a first operating system (e.g., a control interface device) to a second different system running a second different operating system (i.e., the application environment of the terminal device). In some examples, a customized alternative user interface can be provided for a specific third-party application to adapt to the new environment. As described below, a customized alternative user interface can be provided by a custom configuration service.

[0052] For example, a control interface device (or one or more control interface devices) may include a keyboard for sending control signals as keyboard events. The application environment may be configured to convert keyboard events into application control signals for controlling third-party applications. In other words, a computer-implemented method may include converting control signals from the control interface device, including keyboard events and having a first format, into application control signals having a second format for controlling third-party applications. The application control signals may be configured to simulate control signals expected by the third-party application. For example, keyboard events may be converted into function signals corresponding to function buttons typically provided by the third-party application.

[0053] A control interface device may include a touchscreen of a first size for transmitting control signals, while a third-party application may be configured to receive application control signals from a touchscreen of a second size. For example, the application may be designed to run on a smartphone, while the control interface device may be a larger screen. In this case, the application environment may be configured to scale or map the control signals from the control interface device to generate application control signals for controlling the third-party application. In other words, a computer-implemented method may include scaling the control signals from the control interface device to form application control signals for controlling the third-party application. For example, the application environment may be configured to receive position and movement information from the control interface device touchscreen and scale that position and movement information proportionally to correspond to the application touchscreen.

[0054] The control interface device may include a user interface environment for running an application user interface associated with a third-party application. For example, the control interface device may include a screen or touchscreen, and the application user interface may be configured to display a graphical representation of the third-party application on the screen. The graphical representation of the third-party application may include function buttons, a virtual keyboard, a dial pad, a directory lookup, or other navigation elements for navigating and / or controlling the third-party application.

[0055] Similarly, the application environment can be configured to convert a video signal in a first format from a third-party application into a media signal in a second format for a video telephony content stream and / or video conferencing hardware. In other words, a computer-implemented method may include converting a media signal in a first format from a third-party application into a media signal in a second format for a video telephony content stream and / or video conferencing hardware. For example, the media signal may include a video signal, and the application environment may be configured to reformat the video data to map parameters such as screen size, video format (e.g., compressed vs. uncompressed), and number of pixels between formats compatible with the application and the video conferencing hardware and / or video telephony content stream. In other examples, the media signal may include audio signals, image or file data, etc., converted from a first format compatible with the third-party application to a second format.

[0056] The application environment may include data mapping interfaces consisting of application programming interfaces (APIs) provided by third-party applications.

[0057] In some examples, the application environment may include custom instructions corresponding to a third-party application. For instance, the application environment may be configured to implement a custom alternative user interface for a specific third-party application, which is configured to map control signals and / or media signals between the third-party application and the control interface device. Therefore, a control interface device with an operating system and / or user interface format different from the terminal device or the third-party application can still control and / or view the third-party application.

[0058] Customization instructions may be stored in a centralized configuration service. This centralized configuration service may reside on a remote server accessible to the video terminal system. In this case, a computer implementation method may include retrieving customization instructions from the centralized configuration service and implementing or creating a data mapping interface based on those instructions. For example, the centralized configuration service may include customization user interface instructions and a curated app store for installing appropriate third-party applications onto the video terminal system.

[0059] The video terminal system can communicate with a centralized monitoring service, and computer implementation methods may include monitoring third-party applications through the centralized monitoring service. The centralized monitoring service may reside on a remote server. The centralized monitoring service may be the same service entity as a centralized configuration service. For example, the centralized monitoring service may be configured to monitor whether a third-party application is an authorized application allowed to be installed on the video terminal system. In a further example, the centralized monitoring service may be configured to select applications that can be installed on a specific video terminal system from a curated app store, select control interface devices that can be used with each application and / or each video terminal system, and select control permissions for enabled third-party applications.

[0060] Furthermore, the third-party applications themselves can be monitored to determine and / or log their operating mechanisms and / or performance. For example, error reports related to specific third-party applications can be sent to a centralized configuration service. The third-party configuration service can be configured to send updated, customized instructions for interface interaction with third-party applications.

[0061] The video terminal system may include a management interface that allows administrators to enable or disable authorization for third-party application downloads. For example, if a user wishes to install a new third-party application, the computer implementation may include first requesting the user authorization to download the new third-party application. Subsequently, authorization can be received by entering a password through the control interface device.

[0062] In some examples, enabling or disabling the authorization of third-party application downloads on video terminal systems can be performed by a centralized configuration service.

[0063] The video terminal system can be configured to communicate with a management server or cloud service (such as the centralized configuration service mentioned above) to obtain a list of authorized applications. The video terminal system may only download a third-party application if it is included in that list. In some examples, a list of authorized applications may be presented to the user, who can then select an application to install for the video terminal system to run.

[0064] Application environments containing third-party applications can be sandboxed, thereby restricting the third-party applications' access to computer resources. For example, full access to CPU, memory, file system, network, audio, or display resources can be restricted for third-party applications. In this way, potential malicious activity from untrusted third-party applications can be reduced.

[0065] Third-party applications can be further isolated from the rest of the enterprise network or video terminal system by sandboxing them by hosting them on a guest virtual LAN or by hosting the application environment on a cloud server.

[0066] In other examples, the third-party application can be a trusted application with direct access to computing resources such as CPU, memory, file system, network, audio, or display resources. This helps eliminate latency that the virtualization layer might introduce, thereby improving the efficiency and performance of the trusted third-party application.

[0067] According to a second aspect of the present invention, a computer-implemented method is provided for installing a third-party application for use in a video terminal system for video conferencing, the method comprising:

[0068] Create the application environment;

[0069] The application environment is configured as follows:

[0070] Relay control signals from the control interface device to a third-party application, and

[0071] Relaying media signals between third-party applications and video conferencing hardware / / or video telephony content streams; and

[0072] Install third-party applications within the application environment.

[0073] In some examples, installing a third-party application for use by the video terminal system may include installing the third-party application onto the video terminal system. In other examples, installing a third-party application for use by the video terminal system may include installing the third-party application onto a cloud server that communicates with the video terminal system.

[0074] The second aspect of the computer implementation method may also include downloading custom instructions corresponding to third-party applications from a centralized configuration service for building the application environment.

[0075] In some examples, the computer implementation may also include communicating with a management server or cloud service to obtain a list of authorized applications, and downloading third-party applications only if they are included in that list. This reduces instances of potentially malicious third-party applications. Furthermore, the owner of the video terminal system can control whether to allow the download of potentially resource-intensive third-party applications to the video terminal system.

[0076] Computer-based implementations may include receiving user authentication information to allow the installation of third-party applications. Therefore, video terminal systems offer enhanced security, preventing unauthorized users from downloading potentially malicious third-party applications.

[0077] To avoid ambiguity, it should be noted that the computer implementation method of the second aspect and its optional features may constitute additional steps of the computer implementation method of the first aspect.

[0078] According to other aspects of the invention, a computer program including instructions is provided that, when executed by a computer, causes the computer to perform the method of any of the foregoing aspects. The computer program product may take the form of a non-physical computer program product containing the instructions.

[0079] Other aspects of the present invention may provide a computer-readable storage medium having a computer program of the foregoing aspects of the present invention stored thereon.

[0080] According to another aspect of the present invention, an application environment configured to host third-party applications for operation by a video terminal system can be provided, wherein the application environment is configured as follows:

[0081] Routing control signals from the control interface device of the video terminal system to third-party applications; and

[0082] Routing media signals between third-party applications and video conferencing hardware and / or video telephony content streams.

[0083] In some examples, the application environment may be configured to run on a video terminal system. In other examples, the application environment may be configured to run on a cloud service that communicates with the video terminal system.

[0084] According to another aspect of the present invention, a video terminal system for video conferencing may be provided, comprising video conferencing hardware and control interface devices; wherein the video conferencing hardware optionally includes a screen, a camera and a speaker, and the video terminal system is configured to perform the method of either the first or the second aspect.

[0085] The present invention includes combinations of the aforementioned aspects and preferred features, unless such combination is clearly impractical or explicitly avoided. Attached Figure Description

[0086] The following will discuss and illustrate embodiments and experiments of the present invention with reference to the accompanying drawings, wherein:

[0087] Figure 1 A schematic diagram of a video conferencing system including two video terminal systems according to the prior art is shown.

[0088] Figure 2 A schematic diagram of a video terminal system according to an embodiment of the present invention is shown;

[0089] Figure 3 It shows Figure 2 The diagram shown is a schematic of the video terminal system.

[0090] Figure 4 A schematic diagram of a video terminal system according to another embodiment of the present invention is shown;

[0091] Figure 5 A schematic diagram of a video terminal system according to another embodiment of the present invention is shown;

[0092] Figure 6 A schematic diagram of a video terminal system according to another embodiment of the present invention is shown;

[0093] Figure 7 A schematic diagram of a video terminal system according to another embodiment of the present invention is shown. Detailed Implementation

[0094] The various aspects and embodiments of the invention will be discussed below with reference to the accompanying drawings. Other aspects and embodiments will be apparent to those skilled in the art. All references herein are incorporated by way of citation.

[0095] Figure 1 A schematic diagram of a typical video conferencing system according to the prior art is shown, which includes a connected video terminal system 1 and a second video terminal system 1'.

[0096] Each video terminal system 1, 1' includes terminal devices 3, 3', which are connected to control interface devices 2, 2' and video conferencing hardware, which includes a display screen 5, 5' and cameras 4, 4'.

[0097] In this example, the first and second video terminal systems 1 and 1' are interconnected via the Internet 6 to enable bidirectional communication. In this way, video feeds or other media content (such as audio, computer files, or text) from the camera 4 of the first terminal system 1 can be received by the second video terminal system 1' and displayed on the screen 5'. Similarly, content from the second video terminal system 1' can also be received by the first video terminal system 1.

[0098] Terminal device 3 may be a local processing resource, including proprietary software running on a mass-market operating system such as Android. Terminal device 3 is configured to control video terminal system 1 and enable the video terminal system to perform video conferencing. In some examples, video terminal system 1 may include multiple local processing resources distributed across multiple devices. In some examples, video terminal system 1 may be connected to remote processing resources via the Internet 6.

[0099] Control interface device 2 includes a user interface, optionally as a standalone device or integrated into the same unit as terminal device 3. For example, the entire video terminal system may be a personal device (e.g., including a touchscreen), or in a conference room system scenario, control interface device 2 may be one of multiple control interface devices (e.g., touchscreens) distributed around a conference table. When control interface device 2 is a standalone unit, it may run a mass-market operating system such as Android. Other examples of control interface device 2 may include a keyboard, mouse, function buttons, or other user controls.

[0100] Control interface device 2 is configured to receive user input and relay it as a control signal to terminal devices for controlling hardware and running teleconferences.

[0101] In this example, control interface device 2, camera 4, and display screen 5 are connected to terminal device 3 via wired or wireless data connections. For example, the data connection can be wired, including USB or Power over Ethernet (e.g., 802.3at standard), or wireless, including Wi-Fi or Bluetooth. The data connections between devices in video terminal system 1 can be direct point-to-point connections or indirect connections, including data relay via one or more servers on the Internet 6.

[0102] Due to compatibility issues between video terminal systems and related third-party applications, users who wish to run third-party applications on their video terminal systems using existing technology may be unable to do so. Therefore, the inventors have devised a method for running third-party applications using a video terminal system specifically configured for video conferencing.

[0103] In the examples below, the same features are referred to by the corresponding figure labels, and the corresponding descriptions may also apply, unless the description is obviously impractical or explicitly avoided.

[0104] Figure 2 A schematic diagram of a video terminal system 100 according to an embodiment of the present invention is shown.

[0105] In this example, the video terminal system 100 includes a video terminal device 110 connected to video conferencing hardware 130 and a control interface device 120. The terminal device 110 is connected to the Internet 160 for exchanging video teleconference content streams with remote parties.

[0106] Terminal device 110 is configured to send and receive media signals with video conferencing hardware, which can form part of the content stream of a video teleconference. For example, such as Figure 2 As shown, the video conferencing hardware includes a camera 1332 and a display screen 1334 for recording and displaying video footage.

[0107] Terminal device 110 is also configured to receive control signals from control interface device 120 for controlling video terminal system 100.

[0108] In this example, terminal device 110 is configured to run a third-party application 114, which is hosted within an application environment 112 set up on terminal device 110.

[0109] In this example, the control interface device 120 is provided with an additional user interface environment 122 for hosting the application user interface of a third-party application. For example, if the control interface device 120 includes a touchscreen, a video stream of the third-party application 114 can be provided to the application user interface running in the user interface environment 122, thereby displaying a remote view of the third-party application 114 on the touchscreen.

[0110] Furthermore, the application environment 112 set up on terminal device 110 is connected to management agent service 170 via Internet 160. Management agent service 170 stores the identity information and licensing behavior specifications of each video terminal system 100 under its control. Examples of licensing behavior specifications include: a list of authorized third-party applications that can run on video terminal system 100; details of how each application 114 operates, such as whether user authorization is required, whether application 114 can be shared with any or specific remote party, and / or whether application 114's access to the device or network must be restricted. Management agent service 170 is configured to send configuration management controls to application environment 112.

[0111] Figure 3 It shows Figure 2Another schematic diagram of the video terminal system 100 shown. (See diagram below.) Figure 3 As shown, the video conferencing hardware 130 also includes a speaker 1336 and a microphone 1338 for playing and recording audio data of the video call content stream. Furthermore, the video terminal system 100 includes a network interface 150 for connecting to the Internet 160.

[0112] like Figure 3 As shown, the application environment 112 includes an environment interface 116 for relaying signals between the third-party application 112 and other components of the video terminal system 100.

[0113] Environment interface 116 (also referred to as data mapping interface 116) is configured to route control signal 124 from control interface device 120 to third-party application 114. Environment interface 116 is also configured to route media signal 140 between third-party application 112 and video conferencing hardware 130 and / or video telephony content streams via network interface 150. In some examples, application environment 112 may be configured to relay other types of data, such as data files, text, etc., between third-party application 114 and the rest of the video terminal system.

[0114] In the example shown, media signal 140 includes a video stream 142 received from a camera, which is relayed to a third-party application 112 via data mapping interface 116 and simultaneously transmitted to network interface 150, thus forming part of the content stream of the video call. Furthermore, as part of the content stream of the video call, a video stream 144 received from the third-party application 112 (or from network interface 150) via data mapping interface 116 is relayed to display screen 1134. Similarly, an audio stream 148 received from microphone 1338 is relayed to the third-party application 112 via data mapping interface 116 and simultaneously transmitted to network interface 150, thus forming part of the content stream of the video call, while an audio stream 146 received from the third-party application 112 (or from network interface 150) is provided to speaker 1336.

[0115] During use, the user can request the installation of a third-party application 112 (e.g., an Android application) on the video terminal system 100 through the user interface of the control interface device 120. Subsequently, the control interface device 120 contacts the target application hosting device (i.e., the terminal device 110 in this example), instructs the hosting device to create the application environment 112 required for the third-party application 114 to run, and then downloads the third-party application 114 to the application environment 112.

[0116] Third-party application 114 may have the prerequisites to run on a typical device. Creating application environment 112 includes allocating CPU and memory resources for application 114, as well as virtual devices and / or interfaces (such as displays, keyboards, and touchscreens). When third-party application 114 is launched in the created application environment 112, the prerequisites for third-party application 114 are met, allowing application environment 112 to simulate a typical device (such as a regular mass-market tablet or mobile phone) from the perspective of third-party application 114.

[0117] Subsequently, the user interface of the video terminal system 100 located on the control interface device 120 can be connected to the third-party application 112 through various methods described herein.

[0118] In one example, a virtual video stream can be relayed via a connection between terminal device 110 and control interface device 120, allowing the application video output of third-party application 114 to be displayed on control interface device 120. If the aspect ratio and resolution of the screen set on control interface device 120 are similar to the requirements of third-party application 120, the video can be displayed in full screen. In other examples, the user interface environment 122 on control interface device 120 can be configured to display the application video output within a screen window of control interface device 120. In some examples, the user of control interface device 120 can zoom in and out of the video using zoom gestures such as pinching with two fingers. This example applies to the configuration process of third-party application 114 (e.g., by entering user credentials), and this type of information should not be shared with other participants in the meeting room.

[0119] In another example, the video output of third-party application 114 can be connected to or redirected to display screen 1334 or the content stream of a video teleconference. If display screen 1334 includes a touchscreen, touchscreen gestures can be transmitted back to third-party application 114.

[0120] In some examples, when terminal device 110 includes a graphics processing unit (GPU) and / or a display and / or a touchscreen on the same unit, a third-party application 112 hosted on terminal device 110 can directly access the GPU, display, and / or touchscreen (i.e., without a virtualization layer). This helps improve the efficiency and performance of the third-party application 114.

[0121] The data mapping interface 116 can be configured to convert a control signal 124 with a first format from the control interface device 120 into an application control signal 124' with a second format for controlling a third-party application 112.

[0122] For example, control interface device 120 may include a physical touchpad or a virtual touchpad simulated by its touchscreen. Touch gestures from the virtual touchpad can be relayed as touch gestures to third-party application 112 via data mapping interface 116 through the connection between control interface device 120 and terminal device 110. In this example, application environment 112 may be configured to accurately map gesture positions between the touchpad or virtual touchpad of control interface device 120 and the target virtual display of third-party application 112, taking into account any dynamic cropping, scaling, screen size differences, and aspect ratios between the touchscreen of control interface device 120 and the target virtual display of third-party application 112.

[0123] In another example, terminal device 110 or control interface device 120 may pre-load custom instructions associated with a specific third-party application 112 to construct the application environment and / or user interface environment 122. The custom instructions can be configured to set up an enhanced user interface on control interface device 120, replacing the conventional user interface of third-party application 112. In this way, a user interface more adapted to the hardware of video terminal system 100 can be set up, replacing the user interface of a third-party application that might be more suitable for other hardware (such as a smartphone).

[0124] For example, the enhanced user interface provided on the control interface device 120 may include application-specific function buttons, a virtual keyboard, a dial pad, a directory lookup or navigation element, dedicated to controlling the third-party application 114. The application environment 112 may be configured to convert control signals 124 from the enhanced user interface into control signals 124' that simulate the gestures and key presses expected by the third-party application 114.

[0125] Custom instructions for a specific third-party application 112 may be provided by a central server (e.g., via the Internet 160), and may be downloaded by terminal device 110 and / or control interface device 120. Custom instructions may include a combination of structured data, code, and / or portable code. In some examples, custom instructions may include an application programming interface (API) provided by third-party application 114.

[0126] In some examples, the list of authorized third-party applications that allow the video terminal system 111 to operate may be centrally managed by a management server or cloud service. The video terminal system 100 may connect to the management server or cloud service to obtain the list of authorized applications. If the required third-party application is included in the list of authorized applications, the terminal device 110 may download the third-party application 114 from a publicly accessible repository (such as the Google Play Store), a private repository (such as a private server), the management server itself, or the cloud service. This helps reduce the likelihood of malicious third-party applications 112 being installed on the video terminal system 100.

[0127] Additionally or alternatively, a management interface (e.g., on control interface device 120) may be configured to enable or disable the authorization of specific third-party applications 114. In some examples, user login information or video terminal system device identifiers may be required to enable or disable the download of third-party applications 114.

[0128] In another embodiment, the management server or management interface may be configured to allow the video terminal system 100 to run guest applications not included in the list of authorized applications. Authorization may be obtained based on user login information or the video terminal system device identifier.

[0129] In a further example, the management server or cloud service can be configured to monitor the usage and behavior of the third-party application 114.

[0130] In some examples, terminal device 110 may be configured to sandbox third-party application 112, thereby restricting the third-party application 112's access to the computer resources of video terminal system 100. For example, full access of the third-party application 112 to CPU, memory, file system, network, audio, or display resources may be restricted. By hosting the third-party application 112 in a guest virtual LAN, its access to network resources can be restricted, allowing network traffic associated with the sandboxed third-party application to be isolated from other network traffic.

[0131] In another example, the aforementioned sandboxing can be implemented on a cloud server to further isolate the third-party application 112 from the enterprise network. In this example, the third-party application 112 can only access specific media and control signals of the video terminal system 100.

[0132] Figure 4A schematic diagram of a video terminal system 200 according to another embodiment of the present invention is shown. In this example, an application environment 212 containing a third-party application 214 is set on a control interface device 220, which also hosts an environment user interface 222. The remote view (i.e., video stream) of the third-party application 214 is routed from the application environment 212 to the display screen 2334 via the terminal device 210.

[0133] Figure 5 A schematic diagram of a video terminal system 300 according to another embodiment of the present invention is shown. In this example, an application environment 314 containing a third-party application 312 is set on a cloud service 380.

[0134] An application user interface 322 running on control interface device 320 is connected to application environment 312 on cloud service 380 via terminal device 310 and Internet 360 for controlling third-party application 312. A video stream from third-party application 314 in application environment 312 is routed from cloud service 380 to video terminal device 310 via Internet 360 and simultaneously transmitted to display screen 3334. If application user interface 322 requires a video stream provided by third-party application 314, that video stream can also be routed to control interface device 322 via the same connection, allowing application 312 to view it on control interface device 322 (e.g., as part of the user interface).

[0135] Figure 6 A schematic diagram of a video terminal system 400 according to another embodiment of the present invention is shown, wherein the control interface device 420 is not directly connected to the terminal device 410 via a wired or wireless link, but is indirectly connected to the terminal device 410 via the Internet 460 through a control relay service 490. Control signals issued by the application user interface 422 running on the control interface device 420 can be routed to the application environment 412 through the control relay service 490. If the application user interface 422 requires a video stream provided by the application environment 412, the video stream can be transmitted back through the same connection, thereby enabling a third-party application 412 to view it on the control interface device 422 (e.g., as part of the user interface).

[0136] Figure 7A schematic diagram of a video terminal system 500 according to an embodiment of the present invention is shown, wherein the video terminal system 500 is conducting a teleconference with a remote video terminal system 500'. In this example, the video output of a third-party application 514 is provided as the content stream of the video call by an application environment 512, such that the video output of the third-party application 514 can be sent to both a local display screen 5334 and a remote video terminal system 500' for display on the remote display screen 5334'.

[0137] The features disclosed in the above description or the following claims or drawings, appropriately expressed in their specific form or by means of performing the disclosed functions or by methods or processes for obtaining the disclosed results, may be used alone or in any combination of these features to implement different embodiments of the invention.

[0138] Although the present invention has been described in conjunction with the exemplary embodiments described above, those skilled in the art will understand many equivalent modifications and variations upon obtaining this disclosure. Therefore, the exemplary embodiments of the present invention described above should be considered illustrative rather than restrictive. Various modifications may be made to the described embodiments without departing from the spirit and scope of the invention.

[0139] To avoid ambiguity, any theoretical explanations provided herein are intended to enhance the reader's understanding. The inventors do not intend to be bound by these theoretical explanations in any way.

[0140] Any chapter headings used in this document are for organizational purposes only and should not be construed as limiting the content of the topics discussed.

[0141] Throughout this specification, including the following claims, unless the context otherwise requires, the terms “comprise” and “include” and their variations (e.g., “comprises”, “comprising”, and “including”) shall be understood to imply inclusion of the said integer or step or group of integers or steps, but not to exclude any other integer or step or group of integers or steps.

[0142] It should be noted that the singular forms “a,” “an,” and “described” used in this specification and the appended claims include plural references unless the context clearly specifies otherwise. Ranges herein may be expressed as from “about” a particular value and / or to “about” another particular value. When such a range is expressed, another embodiment includes from one particular value and / or to another particular value. Similarly, when a numerical value is expressed as an approximation using the antecedent “about,” it should be understood that the particular value constitutes another embodiment. The term “about” associated with numerical values ​​is optional and means, for example, ±10%.

Claims

1. A computer-implemented method for running a third-party application using a video terminal system, the video terminal system being used for conducting video teleconferences; the video terminal system comprising video conferencing hardware and control interface devices; The computer implementation method includes: Provide an application environment for the third-party application to run, the application environment being configured as follows: Routing control signals from the control interface device to the third-party application; and Media signals are routed between the third-party application and the video conferencing hardware and / or the content stream of the video telephony.

2. The computer implementation method according to claim 1, wherein the video conferencing hardware includes a camera and a display screen, and the media signal includes a video stream.

3. The computer implementation method according to claim 1 or 2, wherein the video conferencing hardware includes a speaker and a microphone, and the media signal includes an audio stream.

4. The computer implementation method according to any one of the preceding claims, wherein the application environment is set on the video terminal system.

5. The computer implementation method according to claim 4, wherein the application environment is set on the control interface device.

6. The computer implementation method according to claim 4, wherein the video terminal system includes a video terminal device that communicates with the control interface device and the video conferencing hardware, wherein the application environment is set on the video terminal device.

7. The computer implementation method according to claim 6, wherein the control interface device and the video terminal device communicate via a cloud-based relay service, and the application environment is configured to receive control signals from the cloud-based relay service.

8. The computer implementation method according to any one of claims 1 to 3, wherein the video terminal system communicates with a cloud service, and the application environment is set on the cloud service.

9. The computer-implemented method according to any one of the preceding claims, wherein: The control signals include signals for selectively enabling and disabling media signal routing between the third-party application and the video conferencing hardware and / or the content stream of the video telephony.

10. The computer-implemented method according to any one of the preceding claims, wherein: The application environment is configured to convert the control signals from the control interface device, which have a first format, into application control signals with a second format for controlling the third-party application.

11. The computer implementation method according to claim 10, wherein: The control interface device includes a keyboard for sending control signals as keyboard events, and The application environment is configured to convert the keyboard events into application control signals for controlling the third-party application.

12. The computer implementation method according to claim 10 or 11, wherein: The control interface device includes a touchscreen of a first size for sending control signals; and The third-party application is configured to receive application control signals from a touchscreen with a second size; in The application environment is configured to scale the control signals from the control interface device into application control signals for controlling the third-party application.

13. The computer-implemented method according to any one of the preceding claims, wherein: The application environment is configured to convert media signals of a first format from the third-party application into content streams for the video teleconference and / or media signals of a second format from the video conferencing hardware.

14. The computer-implemented method according to any one of the preceding claims, The application environment described therein consists of the application programming interface (API) provided by the third-party application.

15. The computer implementation method according to any of the preceding claims, wherein the application environment includes custom instructions corresponding to the third-party application.

16. The computer implementation method of claim 15, wherein the custom instructions are stored in a centralized configuration service, and the method includes retrieving the custom instructions from the centralized configuration service.

17. The computer-implemented method according to any of the preceding claims, wherein the video terminal system communicates with a centralized monitoring service, the method comprising monitoring the third-party application through the centralized monitoring service.

18. The computer implementation method according to any of the preceding claims, wherein the video terminal system includes a management interface that enables an administrator to enable or disable authorization for third-party application downloads.

19. The computer implementation method according to any of the preceding claims, comprising sandboxing the application environment of the third-party application to restrict the third-party application's access to computer resources.

20. A computer-implemented method for installing a third-party application for use in a video terminal system, the video terminal system being used for video conferencing, the method comprising: Create the application environment; The application environment is configured as follows: The control signals are routed from the control interface device to the third-party application, and Routing media signals between the third-party application and the content stream of the video conferencing hardware and / or video telephony; and The third-party application is installed within the application environment.

21. The computer implementation method according to claim 20, further comprising: Download customized instructions corresponding to the third-party application from the centralized configuration service to build the application environment.

22. The computer implementation method according to claim 20 or 21, further comprising: Communicate with the management server or cloud service to obtain a list of authorized applications, and The third-party application will only be downloaded if it is included in the list.

23. The computer-implemented method according to any one of claims 20 to 22, further comprising: Receive user authentication information to allow the installation of the third-party application.

24. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 23.

25. A video terminal system for video conferencing, comprising: Video conferencing hardware and control interface devices; The video terminal system is configured to perform the method according to any one of claims 1 to 23.

Images