Server, terminal equipment and video conference remote control method

Abstract

Some embodiments of the invention show a server, a terminal device and a video conference remote control method, and relate to the field of remote control, the method comprising: receiving a remote control request sent by a first terminal device, and sending the remote control request to a second terminal device to establish remote control of the first terminal device to the second terminal device; if the audio and video data uploaded by the second terminal device is received, storing the audio and video data to a preset position; and if the keyboard and mouse control message uploaded by the first terminal device is received, only sending the keyboard and mouse control message to the second terminal device, so that the second terminal device executes an operation corresponding to the keyboard and mouse control message. When multiple parties participate in the video conference, the server can store the audio and video data sent by the controlled end for other equipment to pull streams, and can only send the keyboard and mouse control message sent by the control end to the terminal equipment in remote control connection with the control end, so that the purpose of remote control is achieved, other software does not need to be installed, and user operation is simplified.

Description

Technical Field

[0001] The present application relates to the field of remote control technology, and in particular to a server, a terminal device, and a video conference remote control method. Background Art

[0002] With the development of information technology, the emergence of online video conferencing has broken the spatial barriers, improved the work efficiency of users, and has become closely related to the work and life of users. Video conferencing office software supports the function of sharing computer desktops in meetings, which is convenient for participants to watch meeting materials together and improve meeting efficiency. In some scenarios, language communication cannot accurately express the meaning, and a lot of time is spent on small issues such as determining the position of text, which slows down the progress of the meeting and wastes user time.

[0003] In order to save user time, you can choose to use special remote assistance software to control the computer of the sharer to perform actual operations. However, this solution requires that both the control end and the controlled end participating in the meeting install the same remote assistance software in order for other participants to control the computer of the sharer, and the user operation is relatively cumbersome. Summary of the invention

[0004] Some embodiments of the present application provide a terminal device, a server and a remote control method for a video conference. When multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices, store the audio and video data sent by the controlled end for other devices to pull the stream, and send the keyboard and mouse control messages sent by the control end only to the terminal device with which the remote control connection is established, so as to achieve the purpose of remote control without installing other software, thereby simplifying user operations.

[0005] In a first aspect, some embodiments of the present application provide a server, including:

[0006] Communication module;

[0007] The processing module is configured as follows:

[0008] Receiving conference requests respectively sent by the first terminal device, the second terminal device and the third terminal device, so that the first terminal device, the second terminal device and the third terminal device join the same video conference;

[0009] Receiving a remote control request sent by the first terminal device, the remote control request including a second device identifier of the second terminal device, and sending the remote control request to the second terminal device corresponding to the second device identifier to establish remote control of the second terminal device by the first terminal device;

[0010] In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play;

[0011] If a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

[0012] The above technical solution has the following advantages or beneficial effects: when multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices, store the audio and video data sent by the controlled end for other devices to pull the stream, and send the keyboard and mouse control messages sent by the control end only to the terminal device with which the remote control connection is established, so as to achieve the purpose of remote control without installing other software, simplifying user operations.

[0013] In some embodiments, the processing module executes sending the keyboard and mouse control message to the second terminal device, and is further configured to:

[0014] Reading a keyboard and mouse control message at the head of a keyboard and mouse message queue corresponding to the second device identifier, where the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device;

[0015] When the difference between the reception time of the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, sending the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier;

[0016] When the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is greater than a first preset difference, delete the keyboard and mouse control message at the head of the keyboard and mouse message queue, and continue to read the next keyboard and mouse control message at the head of the keyboard and mouse message queue, until the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, and then send the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier.

[0017] The above technical solution has the following advantages or beneficial effects: when the network is stuck or multiple first terminal devices send keyboard and mouse control messages at the same time, resulting in too many keyboard and mouse control messages in the keyboard and mouse message queue, the keyboard and mouse control message at the head of the queue can be deleted and not sent to the second terminal device, thereby avoiding the second terminal device from being overloaded for a short time when restoring the network.

[0018] In some embodiments, the processing module executes sending the keyboard and mouse control message to the second terminal device, and is further configured to:

[0019] Reading a keyboard and mouse control message at the head of a keyboard and mouse message queue corresponding to the second device identifier, where the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device;

[0020] Determine a keyboard and mouse control message whose reception time difference with the keyboard and mouse control message at the head of the keyboard and mouse queue is less than a second preset difference as a first keyboard and mouse control message;

[0021] Obtaining a first device identifier and a priority corresponding to the first device identifier in the first keyboard and mouse control message, and determining the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determining the first keyboard and mouse control messages other than the second keyboard and mouse control message as the third keyboard and mouse control message, wherein the first device identifier is the device identifier of the first terminal device;

[0022] The third keyboard and mouse control message is deleted, and the second keyboard and mouse control message is sent to the second terminal device corresponding to the second device identifier.

[0023] The above technical solution has the following advantages or beneficial effects: when multiple first terminal devices send keyboard and mouse control messages simultaneously or successively, the keyboard and mouse control messages with relatively low priority corresponding to the first terminal devices are deleted, and the undeleted keyboard and mouse control messages are sent to the second terminal devices, so that the second terminal devices give priority to executing the keyboard and mouse control messages sent by the first terminal devices with higher priority.

[0024] In some embodiments, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, the processing module is further configured to:

[0025] Receiving a request for sharing clipboard content sent by the first terminal device, the second terminal device and / or the third terminal device, and adding the first device identifier of the first terminal device, the second device identifier of the second terminal device and / or the third device identifier of the third terminal device to the shared clipboard list;

[0026] When receiving the clipboard content sent by the first terminal device, the second terminal device or the third terminal device, the clipboard content is sent to the target terminal device corresponding to the device identifier in the shared clipboard list to change the clipboard content of the target terminal device.

[0027] The above technical solution has the following advantages or beneficial effects: the server can send the clipboard content uploaded by the participating terminal devices to the terminal devices in the shared clipboard list, thereby realizing the clipboard content sharing function.

[0028] In a second aspect, some embodiments of the present application provide a first terminal device, including:

[0029] A display; configured to display a user interface;

[0030] A communication device configured to establish a communication connection with a server;

[0031] The controller is configured as:

[0032] In response to an instruction to create or join a conference, sending a conference request to a server, so that the server enables the first terminal device to join a video conference based on the conference request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference;

[0033] In response to the instruction to control the second terminal device, a remote control request is sent to the server, so that the server sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device, wherein the remote control request includes the second device identifier of the second terminal device;

[0034] Get audio and video data from the preset location of the server and play the audio and video data in the rendering window;

[0035] The keyboard and mouse control message sent to the rendering window is obtained, and the keyboard and mouse control message is sent to the server, so that the server sends the keyboard and mouse control message to the second terminal device, and the second terminal device performs the operation corresponding to the keyboard and mouse control message.

[0036] The above technical solution has the following advantages or beneficial effects: when multiple parties participate in a video conference, a remote control connection can be directly established with the second terminal device, the screen image of the second terminal device can be displayed in the rendering window by pulling the stream from the server, and the keyboard and mouse control messages can be sent to the second terminal device through the server to achieve the purpose of remote control, without the need to install other software, simplifying user operations.

[0037] In some embodiments, the keyboard and mouse control message includes a keyboard control message and a mouse control message, and the controller executes obtaining the keyboard and mouse control message sent to the rendering window and sending the keyboard and mouse control message to the server, and is further configured as follows:

[0038] When the rendering window is the focus window, the keyboard control message is obtained by using the keyboard and mouse hook programming, and the keyboard control message is sent to the server. The focus window is used to represent the window that is interacting with the user;

[0039] When the rendering window is the focus window and the position indicated by the mouse is within the rendering window, the mouse control message is obtained by using the keyboard and mouse hook programming, and the mouse control message is sent to the server.

[0040] The above technical solution has the following advantages or beneficial effects: keyboard control messages are obtained by keyboard and mouse hook programming, which does not rely on the user interface implementation method of the rendering window. When there are many keyboard control messages, the number of keyboard control message processing can be reduced, the execution efficiency is improved, and the hook programming is highly portable.

[0041] In some embodiments, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to:

[0042] In response to the instruction to share the clipboard, sending a request to share the clipboard content to the server, so that the server adds the first device identifier of the first terminal device to the shared clipboard list;

[0043] When it is detected that the clipboard content has changed and it is not the video conferencing application that changes the clipboard content, the clipboard content is sent to the server, so that the server sends the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

[0044] The above technical solution has the following advantages or beneficial effects: when the clipboard content changes and the change is not caused by the video conferencing application, the clipboard content can be sent to the server and the server can share the clipboard content with the terminal devices in the shared clipboard list, thereby realizing the shared clipboard function and facilitating user operation.

[0045] In a third aspect, some embodiments of the present application provide a second terminal device, including:

[0046] A display; configured to display a user interface;

[0047] A communication device configured to establish a communication connection with a server;

[0048] The controller is configured as:

[0049] In response to an instruction to create or join a conference, sending a conference request to a server, so that the server enables the second terminal device to join the video conference based on the conference request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference;

[0050] In response to an instruction to approve the remote control request, a control approval message and audio and video data are sent to the server, so that the server establishes remote control of the second terminal device by the first terminal device based on the control approval message, and stores the audio and video data in a preset location for the first terminal device and the third terminal device to obtain and play;

[0051] Receive the keyboard and mouse control message sent by the server, and execute the operation corresponding to the keyboard and mouse control message, wherein the keyboard and mouse control message is uploaded to the server by the first terminal device.

[0052] The above technical solution has the following advantages or beneficial effects: when multiple parties participate in a video conference, a remote control connection can be directly established with the first terminal device, the screen image can be uploaded to the server for streaming and playback by other participating devices, and the keyboard and mouse control messages sent by the first terminal device via the server can be received to perform corresponding operations to achieve the purpose of remote control without the need to install other software, thereby simplifying user operations.

[0053] In some embodiments, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to:

[0054] In response to the instruction to share the clipboard, sending a request to share the clipboard content to the server, so that the server adds the second device identifier of the second terminal device to the shared clipboard list;

[0055] When it is detected that the clipboard content has changed and it is not the video conferencing application that changes the clipboard content, the clipboard content is sent to the server, so that the server sends the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

[0056] The above technical solution has the following advantages or beneficial effects: when the clipboard content changes and the change is not caused by the video conferencing application, the clipboard content can be sent to the server and the server can share the clipboard content with the terminal devices in the shared clipboard list, thereby realizing the shared clipboard function and facilitating user operation.

[0057] In a fourth aspect, some embodiments of the present application provide a video conference remote control method, applied to a server, comprising:

[0058] Receiving conference requests respectively sent by the first terminal device, the second terminal device and the third terminal device, so that the first terminal device, the second terminal device and the third terminal device join the same video conference;

[0059] Receiving a remote control request sent by the first terminal device, the remote control request including a second device identifier of the second terminal device, and sending the remote control request to the second terminal device corresponding to the second device identifier to establish remote control of the second terminal device by the first terminal device;

[0060] In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play;

[0061] If a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

[0062] The above technical solution has the following advantages or beneficial effects: when multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices, store the audio and video data sent by the controlled end for other devices to pull the stream, and send the keyboard and mouse control messages sent by the control end only to the terminal device with which the remote control connection is established, so as to achieve the purpose of remote control without installing other software, simplifying user operations.

[0063] In the technical solution provided by the embodiment of the present application, after the first terminal device, the second terminal device and the third terminal device join the same video conference, the remote control request sent by the first terminal device is received, and the remote control request is sent to the second terminal device to establish remote control of the second terminal device by the first terminal device. If the audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play; if the keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device, so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device. When multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices, store the audio and video data sent by the controlled end for other devices to pull the stream, and can also send the keyboard and mouse control message sent by the control end only to the terminal device with which the remote control connection is established, so as to achieve the purpose of remote control, without the need to install other software, simplifying user operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A timing diagram of a video conference remote control method provided in some embodiments of the present application;

[0066] Figure 2 A flowchart of a video conference remote control method provided for some embodiments of the present application;

[0067] Figure 3A schematic diagram of a video conference page of a first terminal device provided in some embodiments of the present application;

[0068] Figure 4 A schematic diagram of screen sharing data transmission provided in some embodiments of the present application;

[0069] Figure 5 A schematic diagram of a video conference page of a first terminal device provided in some embodiments of the present application;

[0070] Figure 6 A schematic diagram of a prompt page of a second terminal device provided in some embodiments of the present application;

[0071] Figure 7 A schematic diagram of a video conference page of a second first terminal device provided in some embodiments of the present application;

[0072] Figure 8 A flowchart of a keyboard and mouse event processing method provided for some embodiments of the present application;

[0073] Fig. 9 A flowchart of a hardware message processing method provided for some embodiments of the present application;

[0074] Fig.10 A schematic diagram of a video conference page of a third first terminal device provided in some embodiments of the present application;

[0075] Fig.11 A schematic diagram of another screen sharing data transmission provided in some embodiments of the present application;

[0076] Fig.12 A flowchart of a method for sharing a clipboard provided in some embodiments of the present application;

[0077] Fig.13 A timing diagram of another video conference remote control method provided for some embodiments of the present application. DETAILED DESCRIPTION

[0078] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0079] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0080] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0081] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0082] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0083] In the embodiments of the present application, the terminal device is a device that inputs programs and data into a computer or receives computer output processing results via communication facilities. The terminal device is usually set up in a convenient place where it can use communication facilities to connect to a remote computer for work. It is mainly composed of a communication interface control device and a dedicated or selected input and output device.

[0084] Terminal devices include but are not limited to smart TVs, mobile terminals, computers, wearable devices, virtual reality devices, augmented reality devices, etc. Mobile terminals include but are not limited to laptops, tablets, and smart phones, etc.

[0085] Terminal devices include control terminal devices and controlled terminal devices. Control terminal devices generally refer to terminal devices that can control other terminal devices, that is, terminal devices that send control instructions to other terminal devices. Controlled terminal devices generally refer to terminal devices that can be controlled by other terminal devices, that is, terminal devices that receive control instructions sent by other terminal devices. Terminal devices can be both controlled terminal devices and control terminal devices.

[0086] The terminal device may include a display, a communication device, a controller and a user input interface.

[0087] In some embodiments, the display includes a display function component for presenting a picture, and a driving component for driving an image display. The display is used to receive an image signal output from the controller for display. For example, the display can be used to display video content, image content, and components of a menu control interface and a user control user interface (UI).

[0088] In some embodiments, the communication device is a component for communicating with an external device or server according to various communication protocol types. The terminal device may be provided with multiple communication devices according to the different communication modes supported. For example, when the terminal device supports wireless network communication, the terminal device may be provided with a communication device including a WiFi function. When the terminal device supports Bluetooth connection communication, the terminal device needs to be provided with a communication device including a Bluetooth function.

[0089] The communication device can enable the terminal device to communicate with the external device or server through wireless or wired connection. Among them, the wired connection can connect the terminal device with the external device through components such as data cables and interfaces. The wireless connection can connect the terminal device with the external device through wireless signals or wireless networks. The terminal device can establish a connection relationship with the external device directly, or indirectly through a gateway, router, connection device, etc.

[0090] In some embodiments, the controller may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller controls the operation of the terminal device and responds to user operations through various software control programs stored in the memory. The controller controls the overall operation of the terminal device.

[0091] In some embodiments, the user input interface may be used to receive instructions from a user input.

[0092] With the development of information technology, the emergence of online video conferencing has broken the spatial barriers, improved the work efficiency of users, and has become closely related to the work and life of users. Video conferencing office software supports the function of sharing computer desktops in meetings, which is convenient for participants to watch meeting materials together and improve meeting efficiency. In some scenarios, language communication cannot accurately express the meaning, and a lot of time is spent on small issues such as determining the position of text, which slows down the progress of the meeting and wastes user time. In order to save user time, you can choose to use special remote assistance software to control the computer of the sharer for actual operation, for example, adding remote desktop control function in video conferencing.

[0093] In order to save user time, you can choose to use special remote assistance software to control the computer of the sharer to perform actual operations. However, this solution requires that both the control end and the controlled end participating in the meeting install the same remote assistance software in order for other participants to control the computer of the sharer, and the user operation is relatively cumbersome.

[0094] In some embodiments, Figure 1 As shown, in the case of joining a video conference, after the first control terminal device receives the instruction input by the user to control the controlled terminal device, it sends a first control request to the controlled terminal device. After receiving the first control request, the controlled terminal device establishes a communication connection with the first control terminal device. After receiving the first control instruction input by the user for the controlled terminal device, the first control terminal device sends the first control instruction to the controlled terminal device, so that the controlled terminal device performs the operation corresponding to the first control instruction. After receiving the instruction input by the user to control the controlled terminal device, the second control terminal device sends a second control request to the controlled terminal device. After receiving the second control request, the controlled terminal device disconnects the communication connection with the first control terminal device and establishes a communication connection with the second control terminal device. After receiving the second control instruction input by the user for the controlled terminal device, the second control terminal device sends the second control instruction to the controlled terminal device, so that the controlled terminal device performs the operation corresponding to the second control instruction.

[0095] In the above embodiment, only a single control terminal device is supported to remotely control a shared computer (controlled terminal device) at the same time. If other users also have control needs, they must first end the existing remote control and then invite other users to re-perform remote control. The user operation is relatively cumbersome and multiple users cannot control the controlled terminal device at the same time.

[0096] In order to achieve remote control and simplify user operations in a video conference, an embodiment of the present application provides a server. The server includes a communication module and a processing module. Figure 2 As shown, the processing module performs the following steps:

[0097] Step S201: receiving conference requests respectively sent by the first terminal device, the second terminal device and the third terminal device, so that the first terminal device, the second terminal device and the third terminal device join the same video conference.

[0098] In some embodiments, the first terminal device, the second terminal device, and the third terminal device respectively respond to the start instruction of the video conferencing application to display the video conferencing home page, and the video conferencing home page includes a create conference control and a join conference control.

[0099] After receiving the user's confirmation operation on the create conference control, that is, the instruction to create a conference, the first terminal device, the second terminal device, or the third terminal device sends a request to create a conference to the server. The server creates a video conference according to the request to create a conference, and sends page data to the first terminal device, the second terminal device, or the third terminal device, so that it displays the video conference page corresponding to the page data.

[0100] In some embodiments, the audio call data of each participating terminal and / or the video call data collected by the camera component will be uploaded to the room corresponding to the video conference and cached. Each terminal pulls the audio and video call data of the other end according to the identifier of the other end in the video conference to obtain the audio data and / or video data of the other end for playback.

[0101] After receiving the user's confirmation operation on the join conference control, that is, the instruction to join the conference, the first terminal device, the second terminal device, or the third terminal device displays the join conference page, which includes a conference number input box. After receiving the conference number entered by the user in the conference box, a join conference request is sent to the server, and the conference request includes the conference number. The server enables the first terminal device, the second terminal device, or the third terminal device to join the video conference corresponding to the conference number, and sends page data to the first terminal device, the second terminal device, or the third terminal device, so that it displays the video conference page corresponding to the page data.

[0102] In other embodiments, after receiving a user's confirmation operation on a video conference link in a certain page, the first terminal device, the second terminal device, and the third terminal device may start a video conference application and jump to a video conference page corresponding to the video conference link.

[0103] The first terminal device, the second terminal device and the third terminal device can join the same video conference through the same conference number or the same video conference link.

[0104] Step S202: receiving a remote control request sent by the first terminal device, and sending the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the first terminal device over the second terminal device.

[0105] Step S203: When the first terminal device remotely controls the second terminal device, if the audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play. The audio and video data is a screen image captured by the second terminal device at a preset time interval.

[0106] In some embodiments, after the second terminal device initiates screen sharing, the first terminal device sends a remote control request and sends the remote control request to the second terminal device corresponding to the second device identifier to establish remote control of the second terminal device by the first terminal device.

[0107] The step of initiating screen sharing by the second terminal device includes: responding to the screen sharing instruction input by the user, capturing the screen image at a preset interval and uploading the screen image to the server in the form of a video stream. The server stores the video stream to a preset location for the first terminal device and the third terminal device to obtain and play. The video conference page includes a screen sharing control, which can receive the user's confirmation operation on the screen sharing control to input the screen sharing instruction.

[0108] For example, Figure 3 As shown, the video conference page includes a participant control 31 and a toolbar 32. The participant control 31 can display the corresponding account name, account avatar and / or image data collected by the image acquisition device of the terminal device used by the participant. The toolbar 32 includes a mute control 321, a video start control 322 and a screen sharing control 323. When a confirmation operation on the mute control 321 is received, so that the mute control 321 is changed to an unopened state, the sound collection device is started, and the audio call data collected by the sound collection device is sent to the server. The terminal devices of other participants obtain the audio call data from the server and play the audio call data. When a confirmation operation on the mute control 321 is received, so that the mute control 321 is changed to an open state, the sound collection device is turned off.

[0109] When the confirmation operation of opening the video control 322 is received, so that the opening video control 322 is changed to the opening state, the image acquisition device is started, and the video call data collected by the image acquisition device is sent to the server. The terminal devices of other participants obtain the video call data from the server, and display the image corresponding to the video call data in the image display area on the participant control 31 corresponding to the video call data sending end. When the confirmation operation of opening the video control 322 is received, so that the opening video control 322 is changed to the closing state, the image acquisition device is closed.

[0110] It should be noted that the terminal devices of the participants can package the image data collected by the image acquisition device and / or the audio data collected by the sound acquisition device into a call video stream and / or a call audio stream, and transmit it to the server through the Real-time Transport Protocol (RTP). The terminal devices of the other participants obtain the call video stream and / or the call audio stream from the preset position of the server through the real-time transport protocol, and then decode and render it in the window corresponding to the call video stream sender identifier in the video conference page, and / or play the audio, so as to understand the images and sounds shared by the terminal devices of other participants in real time.

[0111] The screen sharing control 323 is used to trigger whether to share the screen with the participants, and can receive a user's click or selection operation on the screen sharing control 323.

[0112] like Figure 4 As shown, after receiving the confirmation operation of the screen sharing control, the second terminal device captures the screen image at a preset interval, packages the screen image into a shared screen video stream, and transmits it to the server through the real-time transport protocol. The first terminal device and the third terminal device obtain the shared screen video stream from the server through the real-time transport protocol, and then decode and render it in the rendering window of the screen sharing in the video conference page, so as to watch the screen image shared by the second terminal device in real time.

[0113] In some embodiments, after receiving the user's confirmation operation on the screen sharing control 323, the second terminal device can directly display the terminal screen display page, and package the terminal screen display page into a shared screen video stream and upload it to the server, which is then pulled and played by the first terminal device and the third terminal device.

[0114] In other embodiments, after receiving the user's confirmation operation on the screen sharing control 323, the second terminal device can obtain the background page corresponding to the application running in the background, and display the page control corresponding to the background page on the video conference page. After receiving the user's selection of the page control, the background page corresponding to the page control is displayed in the foreground and the terminal screen display page at this time is packaged into a shared screen video stream and uploaded to the server, and the server is used by the first terminal device and the third terminal device to pull and play.

[0115] In some embodiments, when the second terminal device is not sharing the screen, the first terminal device and the third terminal device only need to pull the call video stream and / or call audio stream from the server, and display the image corresponding to the call video stream in the display area corresponding to the call video stream sending end in the video conference page, and / or play the audio corresponding to the call audio stream.

[0116] In some embodiments, after the second terminal device shares the screen, the first terminal device and the third terminal device need to pull the shared screen call video stream and / or call audio stream from the server, and also need to pull the shared screen video stream. The shared screen video stream is used to display the screen image of the second terminal device in the rendering window of the screen sharing in the video conference page.

[0117] After the first terminal device displays the video conference page through the video conference number or video conference link, it pulls the shared screen video stream uploaded by the second terminal device from the server, and displays the screen sharing rendering window in the video conference page, and displays the image corresponding to the shared screen video stream in the rendering window, that is, the screen image of the second terminal device.

[0118] The rendering window includes a request control control. After receiving the user's confirmation operation on the request control control, the first terminal device sends a remote control request to the server, and the remote control request includes the second device identifier of the second terminal device. The server sends the remote control request to the second terminal device corresponding to the second device identifier. The remote control request may also include the name of the participant corresponding to the first terminal device, which can be used to display the participant who wants to control the device when displaying the control prompt information on the second terminal device.

[0119] In some embodiments, after receiving the remote control request sent by the first terminal device, the second terminal device may directly send a connection approval request to the server. The server may directly establish remote control of the second terminal device by the first terminal device.

[0120] In other embodiments, after receiving the remote control request sent by the first terminal device, the second terminal device displays an approval control in the video conference page. After receiving a confirmation operation on the approval control, a connection approval request is sent to the server. The server can directly establish remote control of the second terminal device by the first terminal device.

[0121] For example, Figure 5 As shown, the video conference page of the first terminal device includes a rendering window 51 and a request control control 52. After receiving the user's confirmation operation on the request control control 52, a remote control request is sent to the server, and then sent to the second terminal device by the server. Figure 6 As shown, the prompt page of the second terminal device includes prompt information 61, an approval control 62 and a rejection control 63. The user can receive a confirmation operation on the approval control 62 and send a connection approval request to the server. The server can directly establish remote control of the first terminal device to the second terminal device.

[0122] In some other embodiments, after sharing the screen image or specifying the background page to the first terminal device and the third terminal device participating in the conference, the second terminal device can receive a user input instruction to switch to the video conference page displaying the information of the participants, and display the participant controls on the video conference page. After receiving the confirmation operation of the participant control, the personnel setting box is displayed, and the personnel setting box includes the control requesting to be controlled. After receiving the confirmation operation of the control requesting to be controlled, the request to apply to be controlled is sent to the first terminal device corresponding to the participant control. After receiving the instruction to agree to the request to be controlled, the first terminal device can send an agreement message to the server so that the server establishes remote control of the second terminal device by the first terminal device.

[0123] In other embodiments, after joining the video conference, the first terminal device can directly send a remote control request without the second terminal device initiating screen sharing, and send the remote control request to the second terminal device corresponding to the second device identifier to establish remote control of the second terminal device by the first terminal device.

[0124] After receiving a confirmation operation on the participant control in the video conference page, the first terminal device can display a personnel setting box, which includes a control requesting to be controlled and a control requesting control. After receiving a confirmation operation on the control requesting control, a remote control request is sent to the server. The remote control request includes the second device identifier of the second terminal device corresponding to the participant control. The server sends the remote control request to the second terminal device corresponding to the second device identifier. The second terminal device receives an instruction to agree to the request to be controlled, sends an agreement message to the server, and determines whether to share the screen. If the screen is not shared, the screen image is captured at a preset interval and the screen image is sent to the server in the form of a video stream. After receiving the agreement message, the server establishes remote control of the second terminal device by the first terminal device. After receiving the video stream, the server stores it in a preset location for the first terminal device and the third terminal device to pull and play.

[0125] After receiving the confirmation operation for requesting the controlled control, the first terminal device sends a request to apply for control to the server, and the request to apply for control includes the second device identifier of the second terminal device corresponding to the participant control. The server sends the request to apply for control to the second terminal device corresponding to the second device identifier. At this time, the first terminal device is the controlled device, and the second terminal device is the controlling device. After receiving the instruction to agree to the control request, the second terminal device sends an agreement message to the server. The server establishes remote control of the first terminal device by the second terminal device and notifies the first terminal device. The first terminal device determines whether to share the screen. If the screen is not shared, the screen image is captured at a preset interval and the screen image is sent to the server in the form of a video stream, so that the server sends the video stream to the second terminal device and the third terminal device.

[0126] For example, Figure 7 As shown, the personnel setting box 71 of the first terminal device includes a control request 711, a control request 712, a shared clipboard control 713, and a control 714 for removing from the conference. After receiving a confirmation operation for the control request 711, a control request is sent to the server, and the server sends it to the second terminal device corresponding to the participant control to establish remote control of the first terminal device by the second terminal device. After receiving a confirmation operation for the control request 712, a remote control request is sent to the server, and the server sends it to the second terminal device corresponding to the participant control to establish remote control of the second terminal device by the first terminal device.

[0127] In some embodiments, the server establishes remote control of the second terminal device by the first terminal device through the Websocket protocol. The server can use the Websocket protocol for remote control. The first terminal device establishes a Websocket long connection with the server. The user name of the second terminal device needs to be added to the message sent, and the server performs directional forwarding according to the user name contained in the message. However, this method has high overhead and low efficiency, and the priority of long connection transmission is much higher than the real-time transport protocol. In poor network conditions, that is, when the video conference is stuck, due to the high priority of the long connection, the control instructions transmitted by the first terminal device through the long connection may not be affected by the network and are transmitted normally to the second terminal device, while the latest screen image of the second terminal device may not be transmitted to the first terminal device, resulting in the control instructions being invalid or not matching the user's intention.

[0128] In some embodiments, in order to solve the problems of high overhead, low efficiency and mismatch between control instructions and user intentions, the server establishes remote control of the first terminal device over the second terminal device through a stream control transmission protocol (SCTP) tunnel. The server can establish a stream control transmission protocol tunnel between multiple first terminal devices and a virtual video conference room in the server, as well as a stream control transmission protocol tunnel between the second terminal device and the virtual video conference room according to the remote control request. The first terminal device needs to add the device identifier of the second terminal device to the message sent, and upload it to the message queue corresponding to the second device identifier of the second terminal device in the virtual video conference room through the stream control transmission protocol tunnel. The server reads the message in the message queue corresponding to the second device identifier, and sends the message to the second terminal device, thereby realizing directional forwarding to the second terminal device instead of forwarding to the third terminal device. The stream control transmission protocol tunnel can transmit the keyboard and mouse control message of the first terminal device to the second terminal device.

[0129] Stream Control Transmission Protocol is a protocol for transmitting multiple data streams simultaneously between the two ends of a network connection. As a transport layer protocol, SCTP can be understood as a protocol similar to TCP (Transport Control Protocol) and UDP (User Datagram Protocol). The service it provides is a bit like TCP, while combining some of the advantages of UDP. SCTP is a protocol that provides reliable, efficient, and orderly data transmission.

[0130] It should be noted that in the embodiment of the present application, the second terminal device can simultaneously receive remote control requests sent by multiple first terminal devices, that is, remote control of the second terminal device by multiple first terminal devices can be established at the same time, and remote control can be established with a new terminal device without disconnecting the current remote control.

[0131] Step S204: If a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

[0132] Keyboard and mouse control messages include keyboard control messages (keyboard events) and mouse control messages (mouse events). The keyboard and mouse event processing flow of the terminal device can be as follows: Figure 8 After receiving the user's operation on the mouse and keyboard, a hardware event is generated, which is stored in the system message queue and then distributed to the message queues of various applications. The applications process each message in turn and trigger response events, such as keyboard input, mouse movement, mouse click, etc.

[0133] In some embodiments, after the first terminal device receives the hardware message sent to the rendering window in the video conferencing application, the window callback function of the rendering window determines whether the hardware message is a keyboard and mouse control message. If the hardware message is a keyboard and mouse control message, the keyboard and mouse control message is analyzed and converted, and then the analyzed and converted message is sent to the remote control module, and the remote control module sends it to the server through the stream control transmission protocol tunnel. The server sends the keyboard and mouse control message only to the second terminal device.

[0134] However, the analysis and processing of keyboard and mouse events depends on the UI implementation of the rendering window. For example, the processing methods of the two are different in applications developed by Microsoft Foundation Classes (MFC) and cross-platform application development framework (Qt). In addition, the processing method of the UI implementation of the rendering window is serially executed, that is, each piece of hardware information must be processed in turn, resulting in low execution efficiency.

[0135] In some embodiments, in order to be independent of the UI implementation of the rendering window and improve execution efficiency, a hook programming technology is used to obtain a keyboard and mouse control message sent to the rendering window. The specific method may include: the first terminal device uses the keyboard and mouse hook programming to obtain the keyboard control message when the rendering window is the focus window, and sends the keyboard control message to the server. The focus window is used to represent the window that is interacting with the user;

[0136] When the rendering window is the focus window and the position indicated by the mouse is within the rendering window, the mouse control message is obtained by using the keyboard and mouse hook programming, and the mouse control message is sent to the server.

[0137] Hooking, also known as "hook" technology, is a computer programming term that refers to various technologies that modify or extend the behavior of operating systems, applications, or other software components by intercepting function calls, message passing, and event passing between software modules. The code that handles the intercepted function calls, events, and messages is called a hook.

[0138] like Fig. 9 As shown, the embodiment of the present application adds a keyboard and mouse hook process to the rendering window of the video conferencing application. The keyboard and mouse hook takes over all hardware messages sent by the system to the rendering window and determines whether the hardware message is a mouse or keyboard event. If the hardware message is a mouse or keyboard event, the mouse or keyboard event is analyzed and processed or transmitted to the remote control module for analysis and processing. If the hardware information is not a mouse or keyboard event, the hardware information is still transmitted to the rendering window.

[0139] A keyboard event usually contains information such as virtual key code and key status. The virtual key code indicates which key triggered the event (such as keyboard keys such as X, enter, and esc), and the key status indicates whether the current key is pressed or released. By analyzing the keyboard event, the virtual key code and key status are extracted and sent to the second terminal device. The second terminal device converts this information into a keyboard event for execution, thereby achieving the effect of remote keyboard control.

[0140] A mouse event usually contains information such as cursor position, virtual key code, and button status. The cursor position indicates the coordinates of the mouse in the display, the virtual key code indicates the mouse button that triggered the event (such as the left button, right button, and middle button), and the button status indicates whether the current mouse button is pressed or popped up. If the user only moves the mouse, the virtual key code and button status in this mouse event are empty. Usually, in order to ensure the viewing experience, the rendering display will be performed under the premise of ensuring the aspect ratio of the shared screen. At this time, the screen of the controlled terminal device will not cover the entire rendering window, such as Figure 5As shown. According to the position coordinates of the rendering window, the position of the mouse relative to the rendering window can be calculated, and then according to the coordinates of the actual rendering area, the position of the mouse relative to the rendering screen can be calculated. Finally, according to the ratio of the rendering screen to the original screen (such as the original 1920x1080 screen rendered in the 640x360 area), the position of the mouse relative to the screen of the controlled terminal device can be calculated. The converted cursor position, virtual key code, and key status information are sent to the second terminal device, and the second terminal device converts this information into mouse events for execution, thereby achieving the effect of remote mouse control.

[0141] In some embodiments, after the server establishes remote control of the first terminal device over the second terminal device, a control bar can be displayed on the upper layer of the terminal page of the second terminal device, and at least one participant control corresponding to the user account logged in to the first terminal device can be displayed on the control bar, which can prompt the user which participants are allowed to control the terminal.

[0142] After receiving the keyboard and mouse control message, the second terminal device can determine the sending first terminal device corresponding to the keyboard and mouse control message, and display it on the participant control corresponding to the sending first terminal device to prompt the user which participant currently controls the second terminal device.

[0143] For example, Fig.10 As shown, when the second terminal device shares the screen, a control bar 101 is displayed on the upper layer of the terminal page, and the control bar 101 includes a first participant control 1011 and a second participant control 1012. Currently, it is indicated that the participant user A corresponding to the first participant control 1011 is controlling the second terminal device.

[0144] It should be noted that when the hook programming receives multiple keyboard and mouse control messages within a preset time, it will delete multiple keyboard and mouse control messages before the last keyboard and mouse control message, reducing the number of keyboard and mouse control message processing and improving execution efficiency. For example, during the mouse movement process, the calculation of the mouse position can be reduced and only the last mouse position within the preset time is retained. Hook programming has strong portability.

[0145] The first terminal device sends the keyboard and mouse control message to the server, and the server stores the keyboard and mouse control message in the keyboard and mouse message queue. At the same time, the server reads the keyboard and mouse control messages in the keyboard and mouse message queue in sequence and sends the keyboard and mouse control message to the second terminal device.

[0146] After the first terminal device obtains the keyboard and mouse control message sent to the rendering window, the data type, the sender identifier, i.e., the first device identifier of the first terminal device, and the designated receiver identifier, i.e., the second device identifier of the second terminal device, are added to the keyboard and mouse control message, and sent to the virtual video conference room of the server through the established stream control transmission protocol tunnel. The data type is the keyboard and mouse control message.

[0147] After receiving the data sent by the first terminal device, the server first identifies the data type. If the data type is a keyboard and mouse control message, the server identifies the designated receiving terminal identifier and determines whether the designated receiving terminal identifier is the device identifier of the virtual video conference room. If the designated receiving terminal identifier is not the device identifier of the virtual video conference room, the server discards the keyboard and mouse control message. If the designated receiving terminal identifier is the device identifier of the virtual video conference room, the server places the keyboard and mouse control message in the designated receiving terminal identifier, that is, the keyboard and mouse message queue corresponding to the second device identifier of the second terminal device.

[0148] Among them, each time a terminal device joins the virtual video conference room, its device identification will be stored in the device list of the virtual video conference room. It can be determined whether the specified receiving end identification is the device identification of this virtual video conference room by judging whether the specified receiving end identification is in the device list.

[0149] Once the server detects that the keyboard and mouse control message is stored in the keyboard and mouse message queue, it will read the keyboard and mouse control message in the keyboard and mouse message queue, and send the read keyboard and mouse control message to the terminal device corresponding to the device identifier to which the keyboard and mouse message queue belongs through the established stream control transmission protocol tunnel. For example, the keyboard and mouse control message in the keyboard and mouse message queue of the second device identifier is sent to the second terminal device.

[0150] In some embodiments, when the network condition is poor or multiple first terminal devices send keyboard and mouse control messages at the same time, many keyboard and mouse control messages are accumulated in the keyboard and mouse message queue and cannot be sent in time. A specific implementation method of the server sending the keyboard and mouse control message to the second terminal device may include:

[0151] Reading a keyboard and mouse control message at the head of a keyboard and mouse message queue corresponding to the second device identifier, where the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device;

[0152] When the difference between the reception time of the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, sending the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier;

[0153] When the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is greater than a first preset difference, delete the keyboard and mouse control message at the head of the keyboard and mouse message queue, and continue to read the next keyboard and mouse control message at the head of the keyboard and mouse message queue, until the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, and then send the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier.

[0154] Exemplarily, the first preset difference is 500ms, and there are 10 keyboard and mouse control messages in the keyboard and mouse message queue, namely, keyboard and mouse control message 1, keyboard and mouse control message 2, ..., keyboard and mouse control message 10. If the difference between the receiving time of keyboard and mouse control message 1 and keyboard and mouse control message 10 is greater than 500ms, then delete keyboard and mouse control message 1, and then determine whether the difference between the receiving time of keyboard and mouse control message 2 and keyboard and mouse control message 10 is greater than 500ms. At this time, no new keyboard and mouse control message is stored in the keyboard and mouse message queue. If the difference between the receiving time of keyboard and mouse control message 2 and keyboard and mouse control message 10 is less than 500ms, then keyboard and mouse control message 2 can be sent to the second terminal device.

[0155] In an embodiment of the present application, when the network is stuck or multiple first terminal devices send keyboard and mouse control messages at the same time, resulting in too many keyboard and mouse control messages in the keyboard and mouse message queue, the keyboard and mouse control message at the head of the queue can be deleted and not sent to the second terminal device, thereby avoiding a short-term overload on the second terminal device when restoring the network.

[0156] In some embodiments, when multiple first terminal devices send keyboard and mouse control messages at the same time, the keyboard and mouse message queue will contain many keyboard and mouse control messages. A specific implementation of the server sending the keyboard and mouse control message to the second terminal device may include:

[0157] Reading a keyboard and mouse control message at the head of a keyboard and mouse message queue corresponding to the second device identifier, where the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device;

[0158] A keyboard and mouse control message whose reception time difference with the keyboard and mouse control message at the head of the keyboard and mouse queue is less than a second preset difference is determined as the first keyboard and mouse control message.

[0159] Obtaining a first device identifier and a priority corresponding to the first device identifier in the first keyboard and mouse control message, and determining the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determining the first keyboard and mouse control messages other than the second keyboard and mouse control message as the third keyboard and mouse control message, wherein the first device identifier is the device identifier of the first terminal device;

[0160] The third keyboard and mouse control message is deleted, and the second keyboard and mouse control message is sent to the second terminal device corresponding to the second device identifier.

[0161] The first keyboard and mouse control message may include the keyboard and mouse control message at the head of the keyboard and mouse queue, or may not include the keyboard and mouse control message at the head of the keyboard and mouse queue. In the case where the first keyboard and mouse control message does not include the keyboard and mouse control message at the head of the keyboard and mouse queue, the keyboard and mouse control message at the head of the queue is first sent to the second terminal device, and then the second keyboard and mouse control message is sent to the second terminal device.

[0162] Exemplarily, the second preset difference is 100ms, and there are 10 keyboard and mouse control messages in the keyboard and mouse message queue, namely, keyboard and mouse control message 1, keyboard and mouse control message 2, ..., keyboard and mouse control message 10. Keyboard and mouse control message 2, keyboard and mouse control message 3, and keyboard and mouse control message 4 have a reception time difference of less than 100ms with keyboard and mouse control message 1, and their corresponding priorities for sending to the first terminal device are level 2, level 1, and level 3, respectively. Keyboard and mouse control message 1 has a priority of level 3, so keyboard and mouse control message 1, keyboard and mouse control message 2, and keyboard and mouse control message 4 are deleted, and keyboard and mouse control message 3 is sent to the second terminal device. Then continue to obtain keyboard and mouse control messages whose reception time difference with the keyboard and mouse control message at the head of the queue is less than 100ms...

[0163] The priority of the first keyboard and mouse control message is determined by the first terminal device included in the first keyboard and mouse control message, and the priority of the first terminal device can be selected by the second terminal device in the video conference page.

[0164] For example, Fig.10 As shown, the priority levels are displayed on the first participant control 1011 and the second participant control 1012 in the control bar 101. After receiving a confirmation operation on the priority setting control 1013, the priorities of the first participant and the second participant can be changed, that is, the priorities of the first terminal devices corresponding to the first participant and the second participant can be changed.

[0165] In some embodiments, when multiple first terminal devices send keyboard and mouse control messages at the same time, the keyboard and mouse message queue will contain many keyboard and mouse control messages. A specific implementation of the server sending the keyboard and mouse control message to the second terminal device may include:

[0166] Read the keyboard and mouse control message at the head of the keyboard and mouse message queue corresponding to the second device identifier, and then determine whether there is a fourth keyboard and mouse control message whose reception time difference with the keyboard and mouse control message at the head of the queue is less than a third preset difference. If there is no fourth keyboard and mouse control message, send the fourth keyboard and mouse control message to the second terminal device. If there is a fourth keyboard and mouse control message, delete the fifth keyboard and mouse control message, and send the fourth keyboard and mouse control message with the latest reception time to the second terminal device, where the fifth keyboard and mouse control message is a keyboard and mouse control message other than the fourth keyboard and mouse control message with the latest reception time.

[0167] Exemplarily, the third preset difference is 50ms, and there are 10 keyboard and mouse control messages in the keyboard and mouse message queue, namely, keyboard and mouse control message 1, keyboard and mouse control message 2, ..., keyboard and mouse control message 10. If there is no keyboard and mouse control message whose receiving time difference with keyboard and mouse control message 1 is less than 50ms, then keyboard and mouse control message 1 is sent to the second terminal device. If there are three keyboard and mouse control messages whose receiving time difference with keyboard and mouse control message 1 is less than 50ms, namely, keyboard and mouse control message 2, keyboard and mouse control message 3 and keyboard and mouse control message 4 in order of receiving time, then keyboard and mouse control message 1, keyboard and mouse control message 2 and keyboard and mouse control message 3 are deleted, and keyboard and mouse control message 4 is sent to the second terminal device. Then continue to determine whether there is a keyboard and mouse control message whose receiving time difference with the keyboard and mouse control message at the head of the queue is less than 50ms...

[0168] In some embodiments, Fig.11 As shown, the image data and audio data transmission method is the same as Figure 4 The first terminal device sends the keyboard and mouse control message to the server through the stream control transmission protocol, and then sends the message to the terminal device sharing the screen, that is, the second terminal device, through the server.

[0169] In some embodiments, during screen sharing on the second terminal device, the first terminal device may need to display specified text or pictures on the screen of the second terminal device. The specified text or pictures may be input into the chat area of ​​the instant messaging software or the public chat area in the video conference, and then the specified text or pictures may be pasted from the chat area of ​​the second terminal device to the specified area, which is cumbersome for the user to operate.

[0170] In some embodiments, in order to simplify user operations, a clipboard can be shared. After receiving a confirmation operation on a shared clipboard control in a video conference page, the first terminal device, the second terminal device, and / or the third terminal device send a request to share clipboard content to the server. The request to share clipboard content includes a device identifier. The server adds the first device identifier of the first terminal device, the second device identifier of the second terminal device, and / or the third device identifier of the third terminal device to a shared clipboard list.

[0171] Among them, if the third terminal device has not established a Stream Control Transmission Protocol tunnel with the virtual video conference room, it is necessary to establish a Stream Control Transmission Protocol tunnel between the third terminal device and the virtual video conference room after the third device identifier is added to the shared clipboard list to transmit the clipboard content.

[0172] For example, in Figure 7 In the example, the first terminal device sends a request to share clipboard content to the server after receiving the confirmation operation on the shared clipboard control 713. The server adds the first device identifier of the first terminal device to the shared clipboard list.

[0173] like Fig.12 As shown, a specific implementation of the shared clipboard may include: the first terminal device, the second terminal device or the third terminal device may determine whether the clipboard content has changed. If the clipboard content has changed, determine whether it is the video conferencing application that changes the clipboard content; if it is not the video conferencing application that changes the clipboard content, then add the data type to the clipboard content and send it to the server through the stream control transmission protocol tunnel, and the data type is the clipboard content. After the server recognizes that the uploaded data type is the clipboard content, it sends the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content. If it is the video conferencing application that changes the clipboard content, continue to determine whether the clipboard content has changed.

[0174] The clipboard is a continuous memory space in the system that can change with the size of the stored information. It is used to temporarily store exchange information. Different applications running on the system share the same clipboard. A hidden window can be created on the first terminal device, the second terminal device, and the third terminal device to monitor the changes in the clipboard content. When the clipboard content changes, the window will receive a message. If the video conferencing application does not change the clipboard content, it means that the clipboard content is completed by the user operation. Then, the clipboard content needs to be extracted and sent to the server through the stream control transmission protocol tunnel, and the server forwards it to other terminal devices. If the video conferencing application changes the clipboard content, it means that the clipboard content transmitted by other terminal devices causes it to change. Therefore, there is no need to send the clipboard content to other terminal devices, and continue to monitor. When the first terminal device, the second terminal device, or the third terminal device receives the clipboard content forwarded by the server, it is written into the system clipboard memory space, thereby realizing the function of sharing the clipboard.

[0175] For example, in Figure 5 In the example, when receiving the user's confirmation operation on the shared clipboard control 53, the clipboard content sharing function of the first terminal device, the second terminal device and the third terminal device participating in the same video conference can be realized.

[0176] When the clipboard content is a file, the file is sliced ​​and transmitted in sequence through the stream control transmission protocol tunnel. After the receiving end receives all the slices, it performs integrity verification and then synthesizes the final file, which includes pictures, documents, audio and video.

[0177] In some embodiments, after receiving a confirmation operation on the shared clipboard control in the video conference page, the second terminal device can share the clipboard content with the first terminal device for which remote control has been established, that is, automatically add the first device identifier of the first terminal device to the shared clipboard list, and can also select participants to share the clipboard. If the terminal device of the participant selected by the user has not established a stream control transmission protocol tunnel with the virtual video conference room, it is necessary to establish a stream control transmission protocol tunnel first.

[0178] In other embodiments, after receiving a confirmation operation on the shared clipboard control in the video conference page, the second terminal device can send a request for sharing the clipboard to the first terminal device over which remote control has been established. After the first terminal device agrees to the shared clipboard request, the first device identifier of the first terminal device is automatically added to the shared clipboard list, so that the clipboard contents can be shared with the first terminal device.

[0179] In some other embodiments, after receiving a confirmation operation on the shared clipboard control in the video conference page, the first terminal device and the third terminal device may share the clipboard content with the second terminal device by default, and it may be set that the device identifiers of the first terminal device and the third terminal device can be added to the shared clipboard list of the second terminal device only after the second terminal device agrees to the sharing request. Among them, if the owner of the shared clipboard list is the second terminal device, the clipboard content uploaded by the device identifier in the shared clipboard list can only be shared with the second terminal device, and the clipboard content uploaded by the second terminal device can be shared with all terminal devices corresponding to the device identifiers in the shared clipboard list.

[0180] In some embodiments, after receiving a confirmation operation for the shared clipboard control in the video conference page, the upper layer of the terminal page of the second terminal device can display a clipboard bar, and display at least one participant control corresponding to the user account logged in to the first terminal device on the clipboard bar, which can prompt the user with which participants to control the terminal clipboard sharing.

[0181] After receiving the clipboard content sent by the server, the second terminal device can determine the sending first terminal device corresponding to the clipboard content, and display it on the participant control corresponding to the sending first terminal device to prompt the user which participant's terminal device the current clipboard content is shared by.

[0182] For example, Fig.10 As shown, when the second terminal device shares the screen, a clipboard bar 102 is displayed on the upper layer of the terminal page, and the clipboard bar 102 includes a first participant control 1021 and a second participant control 1022. The control terminal device of the participant user A is currently instructed to share the clipboard content.

[0183] In some embodiments, the timing diagram of the remote control method in a video conference can be as follows: Fig.13 As shown. The first terminal device, the second terminal device and the third terminal device send conference requests respectively, and the server enables the first terminal device, the second terminal device and the third terminal device to join the same video conference based on the conference request. The first terminal device receives the control instruction of the first user to the second terminal device, sends a remote control request to the server, and the server sends the remote control request to the second terminal device, and the second terminal device displays the prompt page requested by the first user to control, and the prompt page includes a consent control. The second terminal device receives the second user's confirmation operation on the consent control, that is, the consent control instruction, and sends a consent message to the server. The server establishes remote control of the second terminal device by the first terminal device through the SCTP tunnel. The second terminal device captures the screen image at a preset time interval and forms a video stream and sends it to the server. The server stores it to a preset location, and notifies the first terminal device and the third terminal device to pull the video stream from the preset location and display the image corresponding to the video stream in the rendering window of the video conference interface.

[0184] The first terminal device uses hook programming to obtain the keyboard and mouse control message, and sends the keyboard and mouse control message to the server through the stream control transmission protocol tunnel, and the server processes the keyboard and mouse control message and sends it to the second terminal device, and the processing includes deleting some keyboard and mouse control messages when there are many keyboard and mouse control messages in the keyboard and mouse message queue. The second terminal device performs the operation corresponding to the keyboard and mouse control message.

[0185] After the first terminal device and the second terminal device receive the clipboard sharing instructions input by the first user and the second user respectively, they send a sharing request to the server, and the server adds the first device identifier of the first terminal device and the second device identifier of the second terminal device in the sharing request to the clipboard sharing list. The second terminal device determines whether the clipboard content has changed. If the clipboard content has changed, it determines whether the video conferencing application has changed the clipboard content. If the video conferencing application has not changed the clipboard content, the clipboard content is sent to the server through the stream control transmission protocol tunnel, and the server sends it to the first terminal device. The first terminal device changes the clipboard content. After executing the operation corresponding to the sharing instruction, the first terminal device will also execute the judgment whether the clipboard content has changed and subsequent steps, so as to share the clipboard content of the first terminal device with the second terminal device.

[0186] The embodiment of the present application provides a multi-person remote desktop control function based on video conferencing, and also includes clipboard sharing and file transfer functions. The collaborative operation screen is synchronized in real time to other users in the meeting, and is suitable for application scenarios such as online office and teaching.

[0187] Some embodiments of the present application also provide a computer-readable storage medium, which may store a program. When the computer storage medium is configured in a display device or a server, the program may include the program steps involved in the video conference remote control method in the above embodiment when the program is executed. Among them, the computer storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0188] The embodiment of the present application provides an electronic device, which includes: a processor and a memory for storing processor executable instructions. The processor is used to read the executable instructions from the memory and execute the instructions to implement the video conference remote control method in the above embodiment.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0190] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A server, characterized in that: include: Communication module; The processing module is configured as follows: Receiving conference requests respectively sent by a first terminal device, a second terminal device, and a third terminal device, and enabling the first terminal device, the second terminal device, and the third terminal device to join the same video conference; receiving a remote control request sent by the first terminal device, the remote control request including a second device identifier of the second terminal device, and sending the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device; In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play; If a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

2. The server according to claim 1, characterized in that: The processing module executes sending the keyboard and mouse control message to the second terminal device, and is further configured to: Reading a keyboard and mouse control message at the head of a keyboard and mouse message queue corresponding to the second device identifier, where the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device; When the difference between the reception time of the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, sending the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier; When the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is greater than a first preset difference, delete the keyboard and mouse control message at the head of the keyboard and mouse message queue, and continue to read the next keyboard and mouse control message at the head of the keyboard and mouse message queue, until the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, and then send the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier.

3. The server according to claim 1, characterized in that: The processing module executes sending the keyboard and mouse control message to the second terminal device, and is further configured to: Reading a keyboard and mouse control message at the head of a keyboard and mouse message queue corresponding to the second device identifier, where the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device; Determine a keyboard and mouse control message whose reception time difference with the keyboard and mouse control message at the head of the keyboard and mouse queue is less than a second preset difference as a first keyboard and mouse control message; Obtaining a first device identifier in the first keyboard and mouse control message and a priority corresponding to the first device identifier, and determining the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determining the first keyboard and mouse control messages other than the second keyboard and mouse control message as the third keyboard and mouse control message, wherein the first device identifier is the device identifier of the first terminal device; The third keyboard and mouse control message is deleted, and the second keyboard and mouse control message is sent to the second terminal device corresponding to the second device identifier.

4. The server according to claim 1, characterized in that: After the first terminal device, the second terminal device, and the third terminal device join the same video conference, the processing module is further configured to: receiving a request for sharing clipboard content sent by the first terminal device, the second terminal device and / or the third terminal device, and adding the first device identification of the first terminal device, the second device identification of the second terminal device and / or the third device identification of the third terminal device to a shared clipboard list; When receiving the clipboard content sent by the first terminal device, the second terminal device or the third terminal device, the clipboard content is sent to the target terminal device corresponding to the device identifier in the shared clipboard list to change the clipboard content of the target terminal device.

5. A first terminal device, characterized in that: include: monitor; configured to display a user interface; A communication device configured to establish a communication connection with a server; The controller is configured as: In response to an instruction to create or join a conference, sending a conference request to the server, so that the server enables the first terminal device to join a video conference based on the conference request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference; In response to an instruction to control the second terminal device, sending a remote control request to the server, so that the server sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device, wherein the remote control request includes the second device identifier of the second terminal device; Acquire audio and video data from a preset position of the server, and play the audio and video data in a rendering window; Acquire a keyboard and mouse control message sent to the rendering window, and send the keyboard and mouse control message to the server, so that the server sends the keyboard and mouse control message to the second terminal device, and the second terminal device performs an operation corresponding to the keyboard and mouse control message.

6. The first terminal device according to claim 5, characterized in that: The keyboard and mouse control message includes a keyboard control message and a mouse control message, and the controller executes to obtain the keyboard and mouse control message sent to the rendering window, and sends the keyboard and mouse control message to the server, and is further configured as follows: In the case where the rendering window is a focus window, the keyboard control message is acquired by using keyboard and mouse hook programming, and the keyboard control message is sent to the server, wherein the focus window is used to represent the window that is interacting with the user; When the rendering window is the focus window and the position indicated by the mouse is within the rendering window, the mouse control message is acquired by using keyboard and mouse hook programming, and the mouse control message is sent to the server.

7. The first terminal device according to claim 5, characterized in that: After the first terminal device, the second terminal device and the third terminal device join the same video conference, the controller is further configured to: In response to the instruction to share the clipboard, sending a request to share the clipboard content to the server, so that the server adds the first device identifier of the first terminal device to the shared clipboard list; When it is detected that the clipboard content has changed and it is not the video conferencing application that changes the clipboard content, the clipboard content is sent to the server, so that the server sends the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

8. A second terminal device, characterized in that: include: monitor; configured to display a user interface; A communication device configured to establish a communication connection with a server; The controller is configured as: In response to an instruction to create or join a conference, sending a conference request to the server, so that the server enables the second terminal device to join the video conference based on the conference request, wherein the first terminal device, the second terminal device and the third terminal device join the same video conference; In response to an instruction to approve the remote control request, a control approval message and audio and video data are sent to the server, so that the server establishes remote control of the second terminal device by the first terminal device based on the control approval message, and stores the audio and video data in a preset location for the first terminal device and the third terminal device to obtain and play; Receive a keyboard and mouse control message sent by the server, and execute an operation corresponding to the keyboard and mouse control message, wherein the keyboard and mouse control message is uploaded to the server by the first terminal device.

9. The second terminal device according to claim 8, characterized in that: After the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to: In response to the instruction to share the clipboard, sending a request to share the clipboard content to the server, so that the server adds the second device identifier of the second terminal device to the shared clipboard list; When it is detected that the clipboard content has changed and it is not the video conferencing application that changes the clipboard content, the clipboard content is sent to the server, so that the server sends the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

10. A video conference remote control method, characterized in that: Applicable to servers, including: Receiving conference requests respectively sent by a first terminal device, a second terminal device, and a third terminal device, and enabling the first terminal device, the second terminal device, and the third terminal device to join the same video conference; receiving a remote control request sent by the first terminal device, the remote control request including a second device identifier of the second terminal device, and sending the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device; In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play; If a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

Images