Third-party conference terminal control method and proxy server

Abstract

The application provides a third-party conference terminal control method and a proxy server. After a third-party conference terminal and the proxy server establish a long connection, the state of the third-party conference terminal is set as an online state, and the online state of the third-party conference terminal is broadcast to a conference application terminal. If the third-party conference terminal and the proxy server disconnect the long connection, the state of the third-party conference terminal is set as an offline state, and the offline state of the third-party conference terminal is broadcast to the conference application terminal. Similarly, if the conference application terminal and the application management server disconnect the long connection, the state of the conference application terminal is set as an offline state, and the offline state of the conference application terminal is broadcast to other conference application terminals and the third-party conference terminal. In this way, even if the third-party conference terminal does not establish a long connection with the application management server, the third-party conference terminal can respond to the conference management and control signaling in the conference system, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a third-party conference terminal control method and a proxy server. Background Technology

[0002] Video conferencing refers to meetings where people in two or more locations conduct face-to-face conversations via communication devices and networks. Depending on the number of participating locations, video conferencing can be divided into point-to-point meetings and multi-point meetings. Using a video conferencing system, participants can hear audio from other meeting rooms, see the images, actions, and expressions of other participants in those rooms, and send electronic presentations, giving them a sense of immersion.

[0003] Different video conferencing products and terminals typically follow the same audio and video decoding methods. Therefore, current video conferencing systems can connect third-party terminals (terminals that are not using the same conferencing application as other participants) to the same conferencing system and achieve audio and video interoperability.

[0004] However, current video conferencing systems can only achieve audio and video interoperability; third-party terminals cannot respond to functions such as meeting management and control signaling within the system. For example, if the meeting host changes the video layout during the meeting, the third-party terminal cannot respond accordingly, resulting in a poor user experience. Summary of the Invention

[0005] This application provides a third-party conferencing terminal control method and a proxy server to solve the problem that current video conferencing systems can only achieve audio and video interoperability, while third-party terminals cannot respond to functions such as conference management and control signaling in the conferencing system, resulting in a poor user experience.

[0006] Firstly, this embodiment provides a proxy server, which is used to perform:

[0007] After a third-party conference terminal establishes a long connection with the proxy server, the status of the third-party conference terminal is set to online, and the online status of the third-party conference terminal is broadcast to at least one conference application terminal. The third-party conference terminal is a terminal that has not established a long connection with the application management server, and the conference application terminal is a terminal that has established a long connection with the application management server. The application management server and the proxy server are different servers.

[0008] If the third-party conference terminal disconnects from the proxy server, the status of the third-party conference terminal is set to offline, and the offline status of the third-party conference terminal is broadcast to at least one of the conference application terminals.

[0009] If the conference application terminal loses its long connection with the application management server, the conference application terminal is set to an offline state, and the offline state of the conference application terminal is broadcast to other conference application terminals and the third-party conference terminal.

[0010] Secondly, this embodiment provides a third-party conference terminal control method, which is applied to a proxy server, and the method includes:

[0011] After a third-party conference terminal establishes a long connection with the proxy server, the status of the third-party conference terminal is set to online, and the online status of the third-party conference terminal is broadcast to at least one conference application terminal. The third-party conference terminal is a terminal that has not established a long connection with the application management server, and the conference application terminal is a terminal that has established a long connection with the application management server. The application management server and the proxy server are different servers.

[0012] If the third-party conference terminal disconnects from the proxy server, the status of the third-party conference terminal is set to offline, and the offline status of the third-party conference terminal is broadcast to at least one of the conference application terminals.

[0013] If the conference application terminal loses its long connection with the application management server, the conference application terminal is set to an offline state, and the offline state of the conference application terminal is broadcast to other conference application terminals and the third-party conference terminal.

[0014] The third-party conference terminal control method and server provided in this application embodiment establish a long connection between the third-party conference terminal and the proxy server. The third-party conference terminal is then set to an online state and its online status is broadcast to at least one conference application terminal. The third-party conference terminal is a terminal that has not established a long connection with the application management server, while the conference application terminal is a terminal that has established a long connection with the application management server. The application management server and the proxy server are different servers. If the third-party conference terminal disconnects from the proxy server, its status is set to offline and its offline status is broadcast to at least one conference application terminal. Similarly, if the conference application terminal disconnects from the application management server, its status is set to offline and its offline status is broadcast to other conference application terminals and the third-party conference terminal. By utilizing the proxy server, even if the third-party conference terminal has not established a long connection with the application management server, it can still send its status to the conference application terminal, and it can also receive the status of the conference application terminal using the proxy server. This enables the third-party conference terminal to respond to conference management and control signaling in the conference system, thereby improving the user experience. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] Figure 1 An operational scenario between a display device and a control device according to some embodiments is illustrated;

[0017] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0018] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0019] Figure 4 A software configuration diagram of a display device 200 according to some embodiments is shown;

[0020] Figure 5 The diagram illustrates application scenarios according to some embodiments;

[0021] Figure 6 A schematic diagram of a conference system framework according to some embodiments is shown;

[0022] Figure 7 A schematic diagram of yet another conference system framework according to some embodiments is shown;

[0023] Figure 8 A schematic diagram illustrating yet another application scenario according to some embodiments is shown;

[0024] Figure 9 The diagram illustrates the signaling interaction between the conference terminal and the server according to some embodiments;

[0025] Figure 10 This diagram illustrates another type of signaling interaction between a conference terminal and a server, according to some embodiments.

[0026] Figure 11 This diagram illustrates another type of signaling interaction between a conference terminal and a server, according to some embodiments.

[0027] Figure 12 This diagram illustrates another type of signaling interaction between a conference terminal and a server, according to some embodiments.

[0028] Figure 13 A schematic diagram of the user interface of the display device is shown according to some embodiments;

[0029] Figure 14 A schematic diagram of a user interface for another display device according to some embodiments is shown;

[0030] Figure 15 A schematic diagram of a user interface for another display device according to some embodiments is shown;

[0031] Figure 16 This diagram illustrates another type of signaling interaction between a conference terminal and a server, according to some embodiments.

[0032] Figure 17 This diagram illustrates another type of signaling interaction between a conference terminal and a server, according to some embodiments.

[0033] Figure 18 A flowchart of a third-party conference terminal control method according to some embodiments is shown;

[0034] Figure 19 A flowchart of another third-party conference terminal control method according to some embodiments is shown. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of some embodiments of this application clearer, the technical solutions of some embodiments of this application will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0036] It should be noted that the brief descriptions of terms in some embodiments of this application are only for the convenience of understanding the implementation methods described below, and are not intended to limit the implementation methods of some embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0037] In some embodiments of this application, the terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms can be used interchangeably where appropriate.

[0038] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to all the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0039] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0040] The term "remote control" as used in this application refers to a component of a display device (such as the display device disclosed in this application) that typically enables wireless control of the display device over a short range. It generally connects to the display device using infrared and / or radio frequency (RF) signals and / or Bluetooth, and may also include functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors. For example, a handheld touch remote control replaces most of the physical built-in hard buttons in a typical remote control device with a user interface on a touchscreen.

[0041] The electronic device in this application can be a display device or other electronic devices with voice assistant functions. The following description uses a display device as an example. Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a terminal device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 through the mobile terminal 300 and the terminal device 100.

[0042] In some embodiments, the terminal device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via other wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0043] In some embodiments, the mobile terminal 300 may install software applications with the display device 200 to establish a connection and communication via a network communication protocol, thereby achieving one-to-one control operation and data communication. Alternatively, the audio and video content displayed on the mobile terminal 300 can be transmitted to the electronic device 200 to achieve synchronous display.

[0044] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0045] In addition to providing broadcast television reception functions, the display device 200 can also be equipped with intelligent network television functions that provide computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0046] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0047] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0048] In some embodiments, the detector 230 is used to collect signals from the external environment or to interact with the outside world.

[0049] In some embodiments, the display 260 includes a display screen component for presenting an image, a driving component for driving image display, a component for receiving image signals output from a controller, and a user control UI interface, etc.

[0050] In some embodiments, the communicator 220 is a component for communicating with external devices or the server 400 according to various communication protocol types.

[0051] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0052] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0053] In some embodiments, user interface 280 is an interface that can be used to receive control input.

[0054] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the terminal device. (e.g.) Figure 3 As shown, the terminal device 100 includes a controller 111, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0055] Terminal device 100 is configured to control display device 200, and to receive user input operation commands and convert the operation commands into commands that display device 200 can recognize and respond to, thus acting as an intermediary for interaction between user and display device 200.

[0056] In some embodiments, the terminal device 100 may be a smart device. For example, the terminal device 100 may be equipped with various applications that control the display device 200 according to user needs.

[0057] In some embodiments, such as Figure 1 As shown, mobile terminal 300 or other smart display devices can perform similar functions to terminal device 100 after installing the application that controls display device 200.

[0058] The controller 111 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 111 is used to control the operation of the terminal device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0059] Under the control of the controller 111, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC module 133.

[0060] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, button 144, etc.

[0061] In some embodiments, the terminal device 100 includes at least one of a communication interface 130 and an input / output interface 140. The terminal device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via WiFi, Bluetooth, or NFC protocols and send them to the display device 200.

[0062] The memory 190 is used to store various operating programs, data, and applications of the driving and control terminal device 100 under the control of the controller. The memory 190 can also store various control signal instructions input by the user.

[0063] The power supply 180 is used to provide operating power support for the various components of the terminal device 100 under the control of the controller.

[0064] Figure 4 Provided for some embodiments of this application Figure 1 The diagram shows the software configuration in the display device. In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and the System Library layer (referred to as the "System Runtime Library Layer"), and the kernel layer.

[0065] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs that come with the operating system, system settings programs, clock programs, camera applications, etc., or they may be applications developed by third-party developers.

[0066] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer.

[0067] like Figure 4 As shown, in some embodiments of this application, the application framework layer includes managers, content providers, and a view system.

[0068] In some embodiments, the Activity Manager is used to: manage the lifecycle of individual applications and the usual navigation back functionality.

[0069] In some embodiments, the window manager is used to manage all window programs.

[0070] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is accessed, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0071] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, touch sensor, pressure sensor, etc.).

[0072] In some embodiments, the kernel layer also includes a power driver module for power management.

[0073] In some embodiments, Figure 4 The software programs and / or modules corresponding to the software architecture in the document are stored in [the relevant database]. Figure 2 or Figure 3 In the first or second memory shown.

[0074] Video conferencing refers to meetings where people in two or more locations conduct face-to-face conversations via communication devices and networks. Depending on the number of participating locations, video conferencing can be divided into point-to-point meetings and multi-point meetings. Using a video conferencing system, participants can hear audio from other meeting rooms, see the images, actions, and expressions of other participants in those rooms, and send electronic presentations, giving them a sense of immersion.

[0075] Different video conferencing products and terminals typically follow the same audio and video decoding methods. Therefore, current video conferencing systems can connect third-party terminals (terminals that are not using the same conferencing application as other participants) to the same conferencing system and achieve audio and video interoperability.

[0076] However, current video conferencing systems can only achieve audio and video interoperability; third-party terminals cannot respond to functions such as meeting management and control signaling within the system. For example, if the meeting host changes the video layout during the meeting, the third-party terminal cannot respond accordingly, resulting in a poor user experience.

[0077] like Figure 5In the application scenario diagram shown, the conference system includes an application management server 400 and multiple participating terminals 200A-200C. The application management server 400 and the participating terminals 200A-200C can be connected via a communication network. For any participating terminal, the user can log in to the application management server 400 using a pre-registered user ID and password through a client or web application on that terminal, thereby establishing a long-term connection channel between the application management server 400 and the participating terminal, enabling interaction between the two. The user ID uniquely identifies the user using the participating terminal; that is, the application management server 400 can uniquely identify the corresponding user based on the user ID. The user ID may include, but is not limited to, the user's identity information, the user's account information, or the account information of a user group.

[0078] Furthermore, to distinguish different participating terminals, the application management server 400 can store the terminal identifier of each participating terminal. This terminal identifier uniquely identifies the corresponding participating terminal, allowing the application management server 400 to determine the long-term connection channel between the participating terminal and the terminal, thus enabling interaction between the application management server 400 and the participating terminal. In this embodiment, the specific implementation of the terminal identifier is not limited; the terminal identifier may specifically include device address, device type, and device port number, etc.

[0079] The application management server 400 can specifically be a conference application management server (e.g., the server for a conference application APP). The method for updating the terminal status in an online conference includes the following steps: After terminal 200A joins the online conference, the status of terminal 200A is set to online, and the online status of terminal 200A is broadcast to other terminals that have joined the online conference (terminals 200B, 200C, etc.). The online conference can be a video conference, audio conference, or a video call / audio call. The terminal refers to the terminal used by the participants; terminals 200A, 200B, etc., include, but are not limited to, mobile phones, laptops, tablets, personal computers, in-vehicle terminals, and other electronic devices.

[0080] During online meetings, the application management server 400 needs to know the status of each participating terminal, including online, offline, and other statuses, and broadcasts the status of each participating terminal to other participating terminals. This allows other participating terminals to know the status of a particular participating terminal, and a participating terminal to know the status of other participating terminals. Furthermore, the server updates the status of participating terminals when their status changes.

[0081] In this embodiment, the application management server 400 and terminals 200A, 200B, and 200C can interact based on long connections, such as sending and receiving conference signaling. The steps for the application management server 400 to establish a long connection with a terminal may include: receiving a long connection request from a terminal and then establishing a long connection with the terminal. The long connection is used to receive a conference join signaling from the terminal to confirm that the terminal has joined the online conference. In specific implementations, before joining the online conference, the terminal needs to establish a long connection with the application management server 400, such as a WebSocket connection, through which conference signaling can be sent and received. For example, the terminal can send a long connection request carrying a user identifier and a terminal identifier to the application management server 400 so that the application management server can determine the terminal joining the conference and its corresponding user. If the user identifier verification is successful, a long connection can be established with the terminal.

[0082] In the above embodiments, terminals 200A, 200B, and 200C are all terminals that have established a long connection with the application management server 400 (i.e., terminals that log in to the application management server using a pre-registered user ID and password), such as... Figure 6 In the diagram shown, points 1 and 2 can initiate or join a meeting through the conference signaling service (NMSS, Net Meeting Service Server) to obtain a list of meeting participants. Subsequently, participants join a media service room based on the meeting ID to publish their own audio and video streams (e.g., publishing audio and video streams from the media service and obtaining audio and video streams from other terminals), and subscribe to the audio and video streams of other meeting participants based on their UUID (unique identifier for each meeting participant). The meeting host manages and controls the meeting, such as setting the screen layout, turning microphones / cameras on / off, and managing the identities of meeting guests / audience members.

[0083] However, third-party conference terminals (i.e., terminals that have not logged into the application management server using a pre-registered user ID and password) fail to establish a long connection with the application management server 400 according to the method described in the above embodiments. Consequently, these third-party conference terminals cannot obtain signaling from the conference signaling service. Therefore, the video feed from the third-party conference terminals cannot be viewed in the conference application, and conference control over them is impossible. Audio and video communication with other conference terminals can only be achieved using the media access service, but the conference management and control signaling functions within the conference system cannot be responded to.

[0084] To address the problems in the above embodiments, such as Figure 7In the framework diagram shown in 2 and 3, a conference signaling proxy server (i.e., the proxy server of this application) is set up in the conference system. The proxy server can not only keep the third-party conference terminal and the conference application terminal (the terminal that has established a long connection with the application management server) in the same state as in the conference and enable audio and video communication with the conference application terminal, but also send, receive and process management and control signaling during the conference.

[0085] like Figure 8 In the illustrated application scenario, the conference system includes: an application management server 400, multiple conference application terminals 200A-200C, a proxy server 600, and a third-party conference terminal 200D. The application management server 400 and the conference application terminals 200A-200C can connect via a communication network, but cannot connect to the third-party conference terminal 200D via the same network. For any conference application terminal 200A, a user can log in to the application management server using a pre-registered user ID and password through a client or web application on that terminal, thereby establishing a long-lived connection between the application management server 400 and the 200A-200C terminals, enabling interaction between them. The third-party conference terminal 200D cannot interact with the application management server 400. Therefore, this application introduces a proxy server 600, which the third-party conference terminal 200D uses to interact with the application management server 400, and subsequently with the conference application terminals 200A-200C.

[0086] The specific interaction process is as follows: Figure 9 In the signaling diagram shown, after the third-party conference terminal 200D establishes a long connection with the proxy server 600, the proxy server 600 sets the status of the third-party conference terminal 200D to online. The proxy server 600 broadcasts the online status of the third-party conference terminal 200D to at least one conference application terminal 200A-200C (or the proxy server 600 sends the online status of the third-party conference terminal 200D to the application management server 400, and then the application management server 400 broadcasts the online status of the third-party conference terminal 200D to other conference application terminals 200A-200C).

[0087] If the third-party conference terminal 200D loses its long-term connection with the proxy server 600, the proxy server 600 sets the status of the third-party conference terminal 200D to offline and broadcasts the offline status of the third-party conference terminal 200D to at least one conference application terminal 200A-200C. Similarly, if any of the conference application terminals 200A-200C loses its long-term connection with the application management server 400, the application management server 400 sets the status of the conference application terminals 200A-200C to offline, and the proxy server 600 broadcasts the offline status of the conference application terminals 200A-200C to the third-party conference terminal 200D.

[0088] In this embodiment, both the conference application terminal 200A-200C and the third-party conference terminal 200D remain in the conference. Each conference application terminal maintains its own conference state. Specifically, after successfully joining the conference, conference application terminal 200A-200C periodically sends ping (Packet Internet Groper) signaling (i.e., heartbeat signaling) to the application management server 400. If the application management server 400 detects that the WebSocket connection with conference application terminal 200A-200C is not broken, but does not receive ping signaling from conference application terminal 200A-200C within a preset time range (e.g., 90 seconds), it considers the heartbeat signaling of conference application terminal 200A-200C to have timed out, determines that conference application terminal 200A-200C has abnormally left the conference, sets the status of conference application terminal 200A-200C to offline, and notifies other terminals (including conference application terminals and third-party conference terminals) of the offline status.

[0089] Proxy server 600 maintains the conference connection for third-party conference terminal 200D. After joining the conference, third-party conference terminal 200D periodically sends ping signals to proxy server 600. If proxy server 600 does not receive a ping signal from third-party conference terminal 200D within a preset time range, it considers the heartbeat signal of third-party conference terminal 200D to have timed out, confirming that third-party conference terminal 200D has abnormally left the conference. It then sets the status of third-party conference terminal to offline and notifies other terminals (conference application terminals) of the offline status.

[0090] It should be noted that the third-party conference terminal 200D and the conference application terminals 200A-200C use the same audio and video codecs. Therefore, the third-party conference terminal 200D and the conference application terminals 200A-200C can exchange audio and video streams via DAS (Device Access Server). The third-party conference terminal 200D can obtain the audio and video streams of the participants based on the conference ID, and users can see the conference screen on the third-party conference terminal 200D. However, at this time, the conference application terminals 200A-200C cannot see the screen of the third-party conference terminal 200D, nor can they perform conference control on the third-party conference terminal 200D.

[0091] The proxy server 600 configured in this application can be configured such that after the third-party conference terminal 200D accesses the DAS, it notifies the NMProxy (Net Meeting Proxy) (i.e., proxy server 600). The proxy server 600 then sends the join signaling to the NMSS (i.e., application management server 400) to enable the third-party conference terminal 200D to join the conference, maintains the status of the third-party conference terminal 200D in the conference, and receives and responds to custom signaling messages in the conference. In addition, the various conference application terminals 200A-200C using the conference application can also receive the join signaling of the third-party conference terminal 200D and retrieve the audio and video streams of the third-party conference terminal 200D according to the UUID. The video from the third-party conference terminal 200D can also be played in the conference application.

[0092] The proxy server 600 provided in this embodiment allows a third-party conference terminal 200D to establish a long connection with the proxy server 600. After the third-party conference terminal 200D establishes a long connection with the proxy server 600, the proxy server 600 sets the status of the third-party conference terminal 200D to online and broadcasts this online status to at least one conference application terminal 200A-200C. The third-party conference terminal 200D is a terminal that has not established a long connection with the application management server 400, while the conference application terminals 200A-200C are terminals that have established a long connection with the application management server 400. The application management server 400 and the proxy server 600 are different servers. If the third-party conference terminal 200D disconnects from the proxy server 600, the proxy server 600 sets the status of the third-party conference terminal 200D to offline and broadcasts this offline status to at least one conference application terminal 200A-200C. Similarly, if the conference application terminal 200A-200C loses its long-term connection with the application management server 400, its status is set to offline, and this offline status is broadcast to other conference application terminals 200A-200C and third-party conference terminal 200D. In this way, using the proxy server 600, even if the third-party conference terminal 200D has not established a long-term connection with the application management server 400, it can still send its status to the conference application terminals 200A-200C, and similarly, it can receive the status of the conference application terminals 200A-200C using the proxy server 600. This allows the third-party conference terminal 200D to respond to conference management and control signaling in the conference system, thereby improving the user experience.

[0093] In some embodiments, conference application terminals 200A-200C maintain a persistent connection with application management server 400, and third-party conference terminal 200D maintains a persistent connection with proxy server 600. After maintaining a persistent connection with application management server 400, conference application terminals 200A-200C obtain a list of conference participants and subscribe to the audio and video signaling of other conference participants, including the audio and video signaling of all conference application terminals 200A-200C and the audio and video signaling of third-party conference terminal 200D. Third-party conference terminal 200D can also obtain the audio and video signaling of conference application terminals 200A-200C from proxy server 600. It should be noted that proxy server 600 and application management server 400 only interact with audio and video signaling and conference control signaling; the interaction of audio and video streams still takes place between Media Access Service (DAS) and Media Service (MFS).

[0094] Specifically, such as Figure 10In the signaling diagram shown, the audio and video signaling interaction process of the conference application terminal 200A (conference application terminals 200A-200C maintain a long connection with the application management server 400, and the third-party conference terminal 200D maintains a long connection with the proxy server 600) can be as follows:

[0095] Conference application terminal 200A sends the first audio / video signaling to application management server 400. Application management server 400 broadcasts the first audio / video signaling to other conference application terminals 200B and 200C, and simultaneously sends it to proxy server 600. Proxy server 600 then sends the first audio / video signaling from conference application terminal 200A to third-party conference terminal 200D. This enables simultaneous playback of audio / video streams on conference application terminals 200B, 200C, and third-party conference terminal 200D based on the first audio / video signaling from conference application terminal 200A (the audio / video streams from the conference application terminals are received and forwarded by the media access service). Figure 11 In the signaling diagram shown, the audio and video signaling interaction process of the third-party conferencing terminal 200D (conferencing application terminals 200A-200C maintain a long connection with the application management server 400, and the third-party conferencing terminal 200D maintains a long connection with the proxy server 600) can be as follows:

[0096] The third-party conferencing terminal 200D sends the second audio / video signaling to the proxy server 600, which then sends it to the application management server 400. The application management server 400 then broadcasts the second audio / video signaling to the conferencing application terminals 200A-200C. This enables simultaneous playback of audio / video streams on the conferencing application terminals 200A-200C based on the second audio / video signaling from the third-party conferencing terminal 200D (the audio / video streams from the third-party conferencing terminal are received and forwarded by the media access service).

[0097] In some embodiments, the conferencing application of this application can also implement control functions. Conferencing application terminals 200A-200C using the conferencing application can directly perform control operations, but third-party conferencing terminals 200D cannot directly perform control operations. After receiving the conferencing control signaling sent by the conferencing application terminals 200A-200C, the proxy server 600 sends the conferencing control signaling to the third-party conferencing terminal 200D, and the third-party conferencing terminal 200D performs the control operation corresponding to the conferencing control signaling.

[0098] For example, such as Figure 12In the signaling diagram shown, among the conference application terminals 200A-200C, 200A is the host terminal for the conference application, and conference application terminals 200B and 200C are the participating terminals. Each of the conference application terminals 200A-200C maintains its own conference state, while NMProxy maintains the conference state for the third-party conference terminal 200D. The host terminal 200A sets the conference layout to a six-screen configuration and sends the control signal for setting the six-screen layout to the NMSS.

[0099] On one hand, NMSS broadcasts the control signal to set the conference layout to a six-screen configuration to participating terminals 200B and 200C. Terminals 200B and 200C then adjust their layout according to the control signal. On the other hand, NMSS sends the control signal to NMProxy. Upon receiving this control signal, NMProxy recalculates the position coordinates of each screen based on the six-screen layout. Using the participant list, it selects six participating terminals to be placed in the six-screen layout based on the principles of priority for the host, microphone activation, and camera activation. Additionally, it sends the streamID (audio / video stream ID) and position coordinates (the coordinates of the six participating terminals in the adjusted layout) set in the six-screen layout to DAS.

[0100] DAS re-pulls the video streams based on the received list of video stream IDs, rearranges them according to the set position coordinates, and sends this information to the third-party conferencing terminal 200D. Finally, the third-party conferencing terminal 200D displays the adjusted meeting layout on the monitor, such as... Figure 13 The meeting layout in the user interface is shown. The host terminal 200A is displayed first in the page after the layout, terminals 200B and 200C are meeting application terminals, and terminals 200D-200F are third-party meeting terminals.

[0101] In some embodiments, if a new conference application terminal 200C joins the conference, NMProxy receives the start sharing signaling from the conference application terminal 200C and sends the audio and video streams of the conference application terminal 200C to DAS, which then sends them to the third-party conference terminal 200D. Upon receiving the control signaling and the audio and video streams, the third-party conference terminal 200D displays the shared screen on its monitor. In this way, the third-party conference terminal 200D can receive start (or end) sharing signaling from the host terminal or other conference application terminals, enabling conference management and control functions for the third-party conference terminal 200D within the conference system.

[0102] In some embodiments, if the camera of the conference application terminal 200C is turned off on the hosting terminal, or if the camera on the conference application terminal 200C is turned off (e.g.) Figure 14If the user interface shown is accessed, NMProxy will receive a control command to turn off the camera of the conference application terminal 200C and remove the conference application terminal 200C from the streamID list. The third-party conference terminal 200D will then stop displaying the screen of the conference application terminal 200C based on this control command. In this way, the third-party conference terminal 200D can receive camera-on (or-off) commands from the host terminal or other conference application terminals, enabling conference management and control functions for the third-party conference terminal 200D within the conference system.

[0103] In some embodiments, if the microphone of the conference application terminal 200C is turned off on the hosting terminal, or if the microphone is turned off on the conference application terminal 200C (e.g.) Figure 15 If the user interface shown is accessed, NMProxy will receive a control command to mute the microphone of the conference application terminal 200C. After NMProxy sets the audio status of the conference application terminal 200C to mute, it sends the updated streamID list to DAS. DAS then sends the updated streamID list to the third-party conference terminal. The third-party conference terminal 200D, based on the updated streamID list, will no longer receive the audio stream from the conference application terminal 200C, and therefore will not hear the sound from the conference application terminal 200C. In this way, the third-party conference terminal 200D can receive microphone-on (or-off) commands from the host terminal or other conference application terminals, enabling conference management and control functions for the third-party conference terminal 200D within the conference system.

[0104] In some embodiments, if the nickname of the conference application terminal 200C is modified on the hosting terminal or on the conference application terminal 200C itself, NMProxy will receive a control signaling message to modify the nickname of the conference application terminal 200C. After modifying the nickname of the conference application terminal 200C according to the control signaling message, NMProxy sends the updated streamID list to DAS, and DAS then sends the updated streamID list to the third-party conference terminal 200D. The conference application terminal 200C with the modified nickname will then be displayed on the third-party conference terminal 200D. In this way, the third-party conference terminal 200D can receive the nickname modification signaling sent by the hosting terminal or other conference application terminals, realizing the conference management and control functions of the third-party conference terminal 200D in the conference system.

[0105] In some embodiments, if "All View D" (third-party conference terminal 200D) is set on the host terminal, NMProxy will receive the control signaling for "All View D". NMProxy will then remove the existing streamID list based on this control signaling and add only the streamID of the third-party conference terminal 200D to the list. Upon receiving this list containing only the streamIDs of the third-party conference terminal 200D, the third-party conference terminal 200D will only display its own screen. In this way, the third-party conference terminal 200D can receive the "All View TA" (or "Cancel All View TA") signaling sent by the host terminal, enabling conference management and control functions for the third-party conference terminal 200D within the conference system.

[0106] If a participant joins or leaves the meeting, or if the host terminal removes a participant from the meeting, or sets a participant as a guest / audience member, NMProxy can update the streamID list according to the corresponding control signaling (adding or removing participant streamIDs, or modifying information in the streamID list), and send the updated streamID list to the third-party meeting terminal 200D. This allows the third-party meeting terminal 200D to display a screen based on the updated streamID list, thereby achieving the purpose of controlling the third-party meeting terminal 200D.

[0107] In the above embodiment, the conference control signaling and audio / video signaling are forwarded by the proxy server 600 to achieve the management and control of the third-party conference terminal 200D. However, the conference control signaling and audio / video signaling belong to different system terminals. Therefore, when the third-party conference terminal 200D joins the conference, the timing of the conference control signaling and audio / video signaling will be inconsistent.

[0108] To solve the above problems, control signaling and audio / video signaling can be processed in two ways, such as... Figure 16 In the signaling diagram shown, after the third-party conference terminal 200D joins the conference, the NMProxy state is changed from init (initial state) to start (start state). If the first received signal is conference control signal, since the audio and video signal has not yet arrived, the streamID list cannot be matched with the audio and video signal. Therefore, the streamID list is not sent to DAS according to the conference control signal. Instead, the conference control signal is buffered while waiting for the audio and video signal to arrive.

[0109] While waiting for audio and video signaling to arrive, if new conference control signaling continues to be sent to NMProxy, the new conference control signaling will continue to be cached while waiting for the audio and video signaling to arrive. If NMProxy receives audio and video signaling, it updates the streamID list according to all currently cached conference control signaling, matches the streamID list with the received audio and video signaling, and finally, the third-party conference terminal 200D plays the received audio and video signaling according to the streamID list.

[0110] For example, while waiting for audio / video signaling, NMProxy first receives the current streamID list (which could be generated from the start-sharing command received from the current conference application terminal), then receives the first conference control command (e.g., turning on the microphone of conference application terminal 200B), and then receives the second conference control command (e.g., turning off the microphone of conference application terminal 200C). Upon receiving audio / video signaling, it generates a superimposed conference control command based on the first and second conference control commands, and then updates the streamID list based on the superimposed conference control command (turning on the microphone of conference application terminal 200B and turning off the microphone of conference application terminal 200C). Finally, after performing the corresponding control operations on the third-party conference terminal 200D, the sound collected by conference application terminal 200B can be heard on the third-party conference terminal 200D, but the sound collected by conference application terminal 200C cannot be heard.

[0111] like Figure 17 In the signaling diagram shown, after the third-party conference terminal 200D joins the conference, the NMProxy's state changes from init (initial state) to start (started state). If the first received signal is audio or video, it can first synchronize participant information and publish the streamID list. Since the conference control signal has not yet arrived, it is impossible to match the streamID list with the audio or video signal. Therefore, the streamID list is not sent to DAS, but the audio and video signal is buffered while waiting for the conference control signal to arrive.

[0112] After receiving the conference control signaling, NMProxy updates the streamID list accordingly. During this time, NMProxy continues to receive audio and video signaling. It then sends the updated audio and video signaling, along with the updated streamID list, to the third-party conference terminal 200D. Finally, after performing the corresponding control operations on the third-party conference terminal 200D, the latest audio and video signaling can be played on the third-party conference terminal 200D based on the updated streamID list.

[0113] This application provides a third-party conference terminal control method. Figure 18 This is a flowchart illustrating a third-party conferencing terminal control method according to an exemplary embodiment. The song information recognition method is applicable to... Figure 8 The system shown includes a display device proxy server 600. For example... Figure 18 As shown, the third-party conference terminal control method may include the following steps:

[0114] In step S101, after the third-party conference terminal 200D establishes a long connection with the proxy server 600, the status of the third-party conference terminal 200D is set to online, and the online status of the third-party conference terminal 200D is broadcast to at least one conference application terminal 200A-200C. The third-party conference terminal 200D is a terminal that has not established a long connection with the application management server 400, and the conference application terminals 200A-200C are terminals that have established a long connection with the application management server 400. The application management server 400 and the proxy server 600 are different servers.

[0115] In step S102, if the third-party conference terminal 200D disconnects from the proxy server 600, the state of the third-party conference terminal 200D is set to offline, and the offline state of the third-party conference terminal 200D is broadcast to at least one of the conference application terminals 200A-200C.

[0116] In step S103, if the conference application terminal 200A-200C loses its long connection with the application management server 400, the status of the conference application terminal 200A-200C is set to offline, and the offline status of the conference application terminal 200A-200C is broadcast to other conference application terminals 200A-200C and the third-party conference terminal 200D.

[0117] The above method embodiment is a process for controlling the online status of a third-party conference terminal. Based on the above method, this application provides another method for controlling a third-party conference terminal. Figure 19 This is a flowchart illustrating a third-party conferencing terminal control method according to an exemplary embodiment. This third-party conferencing terminal control method is applicable to... Figure 8 The system shown uses proxy server 600. (For example...) Figure 19 As shown, the third-party conference terminal control method may include the following steps:

[0118] In step S201, the first audio and video signaling sent by the conference application terminal 200A-200C is received.

[0119] In step S202, the application management server 400 broadcasts the first audio and video signaling to the other conference application terminals 200A-200C, and simultaneously sends the first audio and video signaling to the third-party conference terminal 200D, so that the audio and video streams are played simultaneously on the conference application terminals 200A-200C and the third-party conference terminal 200D according to the first audio and video signaling.

[0120] In step S203, the second audio and video signaling sent by the third-party conference terminal 200D is received.

[0121] In step S204, the second audio and video signaling is sent to the conference application terminal 200A-200C so that the audio and video streams are played simultaneously on the conference application terminal 200A-200C and the third-party conference terminal 200D according to the second audio and video signaling.

[0122] In some embodiments, the method further includes: receiving conference control signaling sent by the conference application terminal 200A-200C, wherein the conference control signaling is used to instruct control operations on the participating terminals;

[0123] The conference control signaling is sent to the third-party conference terminal 200D so that the third-party conference terminal 200D performs the control operation corresponding to the conference control signaling.

[0124] Those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Accordingly, aspects of this application can be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. All of the above hardware or software may be referred to as a “data block,” “controller,” “engine,” “unit,” “component,” or “system.” Furthermore, aspects of this application may manifest as a computer product located on one or more computer-readable media, the product including computer-readable program code.

[0125] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure through various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely through software solutions, such as installing the described system on existing servers or mobile devices.

[0126] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of the application requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

Claims

1. A proxy server, characterized in that, The proxy server is used to perform: After a third-party conferencing terminal establishes a long connection with the proxy server, the status of the third-party conferencing terminal is set to online, and the online status of the third-party conferencing terminal is sent to the application management server. This allows the application management server to broadcast the online status of the third-party conferencing terminal to at least one conferencing application terminal. The third-party conferencing terminal is a terminal that cannot establish a long connection with the application management server, while the conferencing application terminal is a terminal that has already established a long connection with the application management server. The conferencing application used by the third-party conferencing terminal is different from the conferencing application terminal used by the conferencing application terminal. The application management server and the proxy server are different servers. The application management server and the proxy server exchange audio and video signaling and conferencing control signaling. Receive conference control signaling sent by the conference application terminal, wherein the conference control signaling is used to instruct control operations on the participating terminals; The conference control signaling is sent to the third-party conference terminal so that the third-party conference terminal performs the control operation corresponding to the conference control signaling. If the audio / video signaling sent by the conference application terminal is not received when the first conference control signaling is received, the first conference control signaling is cached while waiting for the audio / video signaling sent by the conference application terminal. If the audio / video signaling sent by the conference application terminal is not received when the second conference control signaling is received, the second conference control signaling is cached while waiting for the audio / video signaling sent by the conference application terminal. If the audio and video signaling sent by the conference application terminal is received, the first conference control signaling and the second conference control signaling are superimposed to obtain a third conference control signaling, and the third conference control signaling and the audio and video signaling are sent to the third-party conference terminal so that the third-party conference terminal performs the control operation corresponding to the third conference control signaling and then plays the audio and video stream according to the audio and video signaling.

2. The proxy server according to claim 1, characterized in that, The proxy server is also used to perform: Receive the first audio and video signaling sent by the conference application terminal; The application management server broadcasts the first audio and video signaling to other conference application terminals, and simultaneously sends the first audio and video signaling to the third-party conference terminal, so that audio and video streams are played simultaneously on the conference application terminal and the third-party conference terminal according to the first audio and video signaling.

3. The proxy server according to claim 1, characterized in that, The proxy server is also used to perform: Receive the second audio and video signaling sent by the third-party conference terminal; The second audio and video signaling is sent to the conference application terminal so that audio and video streams are played simultaneously on the conference application terminal and the third-party conference terminal according to the second audio and video signaling.

4. The proxy server according to claim 1, characterized in that, The proxy server is also used to perform: If the third-party conference terminal disconnects from the proxy server, the status of the third-party conference terminal is set to offline, and the offline status of the third-party conference terminal is broadcast to at least one of the conference application terminals. If the conference application terminal loses its long connection with the application management server, the conference application terminal is set to an offline state, and the offline state of the conference application terminal is broadcast to other conference application terminals and the third-party conference terminal.

5. The proxy server according to claim 1, characterized in that, The conference control signaling includes at least: start / end sharing signaling, turn on / off image acquisition signaling, turn on / off audio acquisition signaling, and modify conference nickname signaling.

6. The proxy server according to claim 1, characterized in that, The proxy server is also used to perform: After the third-party conferencing terminal establishes a long connection with the proxy server, if it receives audio and video signaling but does not receive conferencing control signaling sent by the conferencing application terminal, it caches the audio and video signaling while waiting for conferencing control signaling sent by the conferencing application terminal. If a conference control signaling is received from the conference application terminal, the conference control signaling and the cached audio and video signaling are sent to the third-party conference terminal so that the third-party conference terminal performs the control operation corresponding to the conference control signaling and then plays the audio and video stream according to the audio and video signaling.

7. A third-party conference terminal control method, characterized in that, The control method is applied to a proxy server, and the method includes: After a third-party conferencing terminal establishes a long connection with the proxy server, the status of the third-party conferencing terminal is set to online, and the online status of the third-party conferencing terminal is sent to the application management server. This allows the application management server to broadcast the online status of the third-party conferencing terminal to at least one conferencing application terminal. The third-party conferencing terminal is a terminal that cannot establish a long connection with the application management server, while the conferencing application terminal is a terminal that has already established a long connection with the application management server. The conferencing application used by the third-party conferencing terminal is different from the conferencing application terminal used by the conferencing application terminal. The application management server and the proxy server are different servers. The application management server and the proxy server exchange audio and video signaling and conferencing control signaling. Receive conference control signaling sent by the conference application terminal, wherein the conference control signaling is used to instruct control operations on the participating terminals; The conference control signaling is sent to the third-party conference terminal so that the third-party conference terminal performs the control operation corresponding to the conference control signaling. If the audio / video signaling sent by the conference application terminal is not received when the first conference control signaling is received, the first conference control signaling is cached while waiting for the audio / video signaling sent by the conference application terminal. If the audio / video signaling sent by the conference application terminal is not received when the second conference control signaling is received, the second conference control signaling is cached while waiting for the audio / video signaling sent by the conference application terminal. If the audio and video signaling sent by the conference application terminal is received, the first conference control signaling and the second conference control signaling are superimposed to obtain a third conference control signaling, and the third conference control signaling and the audio and video signaling are sent to the third-party conference terminal so that the third-party conference terminal performs the control operation corresponding to the third conference control signaling and then plays the audio and video stream according to the audio and video signaling.

8. The third-party conference terminal control method according to claim 7, characterized in that, The method further includes: Receive the first audio and video signaling sent by the conference application terminal; The application management server broadcasts the first audio and video signaling to other conference application terminals, and simultaneously sends the first audio and video signaling to the third-party conference terminal, so that audio and video streams are played simultaneously on the conference application terminal and the third-party conference terminal according to the first audio and video signaling.

9. The third-party conference terminal control method according to claim 7, characterized in that, The method further includes: receiving a second audio / video signaling sent by the third-party conferencing terminal; The second audio and video signaling is sent to the conference application terminal so that audio and video streams are played simultaneously on the conference application terminal and the third-party conference terminal according to the second audio and video signaling.

10. The third-party conference terminal control method according to claim 7, characterized in that, The method further includes: If the third-party conference terminal disconnects from the proxy server, the status of the third-party conference terminal is set to offline, and the offline status of the third-party conference terminal is broadcast to at least one of the conference application terminals. If the conference application terminal loses its long connection with the application management server, the conference application terminal is set to an offline state, and the offline state of the conference application terminal is broadcast to other conference application terminals and the third-party conference terminal.

Images