A server, a terminal device and a video conference remote control method

Abstract

Some embodiments of the present application show a kind of server, terminal device and video conference remote control method, it is related to remote control field, the method comprises: receiving the remote control request sent by first terminal device, and the remote control request is sent to second terminal device, to establish the remote control of first terminal device to second terminal device;If receiving the audio and video data uploaded by second terminal device, then audio and video data are stored to preset position;If receiving the keyboard and mouse control message uploaded by first terminal device, then only keyboard and mouse control message is sent to second terminal device, to make second terminal device execute the operation corresponding to keyboard and mouse control message.In the video conference of multiple parties, server can store the audio and video data sent by the controlled terminal to pull stream for other devices, and keyboard and mouse control message sent by control terminal can be only sent to the terminal device connected with remote control, to achieve the purpose of remote control, without installing other software, simplify user operation.

Description

Technical Field

[0001] This application relates to the field of remote control technology, and in particular to a server, terminal equipment, and video conferencing remote control method. Background Technology

[0002] With the development of information technology, online video conferencing has broken down spatial barriers, improved user work efficiency, and become closely related to users' work and life. Video conferencing software generally supports sharing the computer desktop during meetings, making it convenient for participants to view meeting materials together and improving meeting efficiency. However, in some scenarios, verbal communication cannot accurately convey meaning, and much time is spent on minor issues such as determining text placement, slowing down the meeting and wasting users' time.

[0003] To save users' time, dedicated remote assistance software can be used to control the shared computer for actual operation. However, this solution requires both the control end and the controlled end of the meeting to have the same remote assistance software installed in order for other participants to control the shared computer, making the user operation relatively cumbersome. Summary of the Invention

[0004] Some embodiments of this application provide a terminal device, a server, and a video conferencing remote control method. When multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices. It can store the audio and video data sent by the controlled terminal for other devices to stream. It can also send keyboard and mouse control messages sent by the control terminal only to the terminal device with which it has established a remote control connection, so as to achieve the purpose of remote control without the need to install other software, thus simplifying user operation.

[0005] Firstly, some embodiments of this application provide a server, including:

[0006] Communication module;

[0007] The processing module is configured as follows:

[0008] Upon receiving conference requests from the first terminal device, the second terminal device, and the third terminal device, the first terminal device, the second terminal device, and the third terminal device are enabled to join the same video conference.

[0009] The system receives a remote control request from a first terminal device, the remote control request including the second device identifier of the second terminal device, and sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device.

[0010] In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play.

[0011] If a keyboard and mouse control message is received from the first terminal device, the message will only be sent to the second terminal device so that the second terminal device can perform the operation corresponding to the message. The message will not be sent to the third terminal device.

[0012] The above technical solution has the following advantages or beneficial effects: When multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices, store the audio and video data sent by the controlled end for other devices to stream, and send the keyboard and mouse control messages sent by the control end only to the terminal devices with which it has established a remote control connection, so as to achieve the purpose of remote control without the need to install other software, thus simplifying user operation.

[0013] In some embodiments, the processing module is further configured to send keyboard and mouse control messages to a second terminal device:

[0014] Read the keyboard and mouse control message located at the head of the keyboard and mouse message queue corresponding to the second device identifier. The keyboard and mouse control messages in the keyboard and mouse message queue are uploaded by the first terminal device.

[0015] If the difference in reception time between the keyboard and mouse control message at the tail of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, the keyboard and mouse control message at the head of the keyboard and mouse message queue will be sent to the second terminal device corresponding to the second device identifier.

[0016] If the time difference between receiving a keyboard and mouse control message at the tail of the keyboard and mouse message queue and the time difference between receiving a keyboard and mouse control message at the head of the keyboard and mouse message queue is greater than a first preset difference, the keyboard and mouse control message at the head of the keyboard and mouse message queue is deleted, and the next keyboard and mouse control message at the head of the keyboard and mouse message queue is read, until the time difference between receiving a keyboard and mouse control message at the tail of the keyboard and mouse message queue and the time difference between receiving a keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, then the keyboard and mouse control message at the head of the keyboard and mouse message queue is sent to the second terminal device corresponding to the second device identifier.

[0017] The above technical solution has the following advantages or beneficial effects: In the event of network lag or multiple first terminal devices sending keyboard and mouse control messages simultaneously, resulting in too many keyboard and mouse control messages in the keyboard and mouse message queue, the keyboard and mouse control messages at the head of the queue can be deleted and not sent to the second terminal device, thereby avoiding excessive load on the second terminal device in a short period of time when the network is restored.

[0018] In some embodiments, the processing module is further configured to send keyboard and mouse control messages to a second terminal device:

[0019] Read the keyboard and mouse control message located at the head of the keyboard and mouse message queue corresponding to the second device identifier. The keyboard and mouse control messages in the keyboard and mouse message queue are uploaded by the first terminal device.

[0020] The keyboard and mouse control message whose reception time difference with the keyboard and mouse control message at the head of the keyboard and mouse queue is less than the second preset difference is determined as the first keyboard and mouse control message;

[0021] The system obtains the first device identifier and the priority corresponding to the first device identifier in the first keyboard and mouse control message, and determines the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determines the first keyboard and mouse control message other than the second keyboard and mouse control message as the third keyboard and mouse control message, and the first device identifier is the device identifier of the first terminal device.

[0022] Delete the third keyboard and mouse control message, and send the second keyboard and mouse control message to the second terminal device corresponding to the second device identifier.

[0023] The above technical solution has the following advantages or beneficial effects: when multiple first terminal devices send keyboard and mouse control messages simultaneously or consecutively, the keyboard and mouse control messages with relatively lower priority corresponding to the sending first terminal devices are deleted, and the keyboard and mouse control messages that are not deleted are sent to the second terminal devices, so that the second terminal devices execute the keyboard and mouse control messages sent by the first terminal devices with higher priority first.

[0024] In some embodiments, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, the processing module is further configured to:

[0025] Receive requests from a first terminal device, a second terminal device, and / or a third terminal device to share clipboard content, and add the first device identifier of the first terminal device, the second device identifier of the second terminal device, and / or the third device identifier of the third terminal device to the shared clipboard list;

[0026] Upon receiving clipboard content from a first terminal device, a second terminal device, or a third terminal device, the clipboard content is sent to the target terminal device corresponding to the device identifier in the shared clipboard list, so as to modify the clipboard content of the target terminal device.

[0027] The above technical solution has the following advantages or beneficial effects: the server can send the clipboard content uploaded by the participating terminal devices to the terminal devices in the shared clipboard list, thereby realizing the clipboard content sharing function.

[0028] Secondly, some embodiments of this application provide a first terminal device, including:

[0029] The display is configured to show the user interface.

[0030] A communication device configured to establish a communication connection with a server;

[0031] The controller is configured as follows:

[0032] In response to an instruction to create or join a meeting, a meeting request is sent to the server so that the server enables the first terminal device to join the video conference based on the meeting request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference;

[0033] In response to an instruction to control the second terminal device, a remote control request is sent to the server, so that the server sends the remote control request to the second terminal device corresponding to the second device identifier, thereby establishing remote control of the second terminal device by the first terminal device. The remote control request includes the second device identifier of the second terminal device.

[0034] Retrieve audio and video data from a preset location on the server and play the audio and video data in the rendering window;

[0035] The system retrieves the keyboard and mouse control messages sent to the rendering window and sends them to the server, so that the server sends the keyboard and mouse control messages to the second terminal device, and the second terminal device executes the operation corresponding to the keyboard and mouse control messages.

[0036] The above technical solution has the following advantages or beneficial effects: When multiple parties participate in a video conference, a remote control connection can be directly established with the second terminal device. The screen image of the second terminal device can be displayed in the rendering window by pulling the stream from the server. Keyboard and mouse control messages can also be sent to the second terminal device through the server to achieve the purpose of remote control. No other software needs to be installed, which simplifies user operation.

[0037] In some embodiments, the keyboard and mouse control messages include keyboard control messages and mouse control messages. The controller executes the process of obtaining the keyboard and mouse control messages sent to the rendering window and sending the keyboard and mouse control messages to the server, and is further configured to:

[0038] When the rendering window is the focused window, keyboard control messages are obtained through keyboard and mouse hook programming and sent to the server. The focused window is used to represent the window that is currently interacting with the user.

[0039] When the rendering window is the focused window and the mouse pointer is within the rendering window, use keyboard and mouse hook programming to obtain mouse control messages and send them to the server.

[0040] The above technical solution has the following advantages or beneficial effects: it uses keyboard and mouse hook programming to obtain keyboard control messages, which is not dependent on the user interface implementation of the rendering window. When there are many keyboard control messages, it can reduce the number of keyboard control messages processed, improve execution efficiency, and hook programming has strong portability.

[0041] In some embodiments, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to:

[0042] In response to the instruction to share the clipboard, a request to share the clipboard content is sent to the server so that the server adds the first device identifier of the first terminal device to the shared clipboard list;

[0043] If a change in clipboard content is detected, and it is not a video conferencing application that changed the clipboard content, the clipboard content is sent to the server. This causes the server to send the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

[0044] The above technical solution has the following advantages or beneficial effects: when the clipboard content changes and the change is not caused by video conferencing applications, the clipboard content can be sent to the server, and the server can share the clipboard content with the terminal devices in the shared clipboard list, thereby realizing the shared clipboard function and facilitating user operation.

[0045] Thirdly, some embodiments of this application provide a second terminal device, including:

[0046] The display is configured to show the user interface.

[0047] A communication device configured to establish a communication connection with a server;

[0048] The controller is configured as follows:

[0049] In response to an instruction to create or join a meeting, a meeting request is sent to the server so that the server enables the second terminal device to join the video conference based on the meeting request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference;

[0050] In response to the instruction to agree to remote control request, the server sends an agreement control message and audio / video data to the server, so that the server establishes remote control of the second terminal device by the first terminal device based on the agreement control message, and stores the audio / video data in a preset location for the first terminal device and the third terminal device to access and play.

[0051] It receives keyboard and mouse control messages sent by the server and executes the corresponding operations. The keyboard and mouse control messages are uploaded to the server by the first terminal device.

[0052] The above technical solution has the following advantages or beneficial effects: When multiple parties participate in a video conference, a remote control connection can be directly established with the first terminal device, the screen image can be uploaded to the server for other participating devices to stream and play, and the keyboard and mouse control messages sent by the first terminal device through the server can be received to execute the corresponding operations to achieve the purpose of remote control without the need to install other software, thus simplifying user operation.

[0053] In some embodiments, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to:

[0054] In response to the instruction to share the clipboard, a request to share the clipboard content is sent to the server so that the server adds the second device identifier of the second terminal device to the shared clipboard list;

[0055] If a change in clipboard content is detected, and it is not a video conferencing application that changed the clipboard content, the clipboard content is sent to the server. This causes the server to send the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

[0056] The above technical solution has the following advantages or beneficial effects: when the clipboard content changes and the change is not caused by video conferencing applications, the clipboard content can be sent to the server, and the server can share the clipboard content with the terminal devices in the shared clipboard list, thereby realizing the shared clipboard function and facilitating user operation.

[0057] Fourthly, some embodiments of this application provide a video conferencing remote control method applied to a server, including:

[0058] Upon receiving conference requests from the first terminal device, the second terminal device, and the third terminal device, the first terminal device, the second terminal device, and the third terminal device are enabled to join the same video conference.

[0059] The system receives a remote control request from a first terminal device, the remote control request including the second device identifier of the second terminal device, and sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device.

[0060] In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play.

[0061] If a keyboard and mouse control message is received from the first terminal device, the message will only be sent to the second terminal device so that the second terminal device can perform the operation corresponding to the message. The message will not be sent to the third terminal device.

[0062] The above technical solution has the following advantages or beneficial effects: When multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices, store the audio and video data sent by the controlled end for other devices to stream, and send the keyboard and mouse control messages sent by the control end only to the terminal devices with which it has established a remote control connection, so as to achieve the purpose of remote control without the need to install other software, thus simplifying user operation.

[0063] In the technical solution provided in this application embodiment, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, a remote control request is received from the first terminal device, and the remote control request is sent to the second terminal device to establish remote control of the second terminal device by the first terminal device. If audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first and third terminal devices to access and play. If keyboard and mouse control messages uploaded by the first terminal device are received, only the keyboard and mouse control messages are sent to the second terminal device so that the second terminal device can execute the operation corresponding to the keyboard and mouse control messages, without sending the keyboard and mouse control messages to the third terminal device. When multiple parties participate in a video conference, the server can directly establish a remote control connection between at least two terminal devices, store the audio and video data sent by the controlled end for other devices to stream, and send the keyboard and mouse control messages sent by the controlling end only to the terminal devices with which it has established a remote control connection to achieve the purpose of remote control without the need to install other software, thus simplifying user operation. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A timing diagram of a video conferencing remote control method provided for some embodiments of this application;

[0066] Figure 2 A flowchart illustrating a video conferencing remote control method provided in some embodiments of this application;

[0067] Figure 3A schematic diagram of a video conferencing page of a first type of first terminal device provided in some embodiments of this application;

[0068] Figure 4 A schematic diagram illustrating a screen-sharing data transmission method provided for some embodiments of this application;

[0069] Figure 5 A schematic diagram of a video conferencing page of a first terminal device provided for some embodiments of this application;

[0070] Figure 6 A schematic diagram of a prompt page for a second terminal device provided in some embodiments of this application;

[0071] Figure 7 A schematic diagram of a video conferencing page for a second type of first terminal device provided in some embodiments of this application;

[0072] Figure 8 A flowchart illustrating a keyboard and mouse event handling method provided in some embodiments of this application;

[0073] Figure 9 A flowchart of a hardware message processing method provided for some embodiments of this application;

[0074] Figure 10 A schematic diagram of a video conferencing page of a third type of first terminal device provided in some embodiments of this application;

[0075] Figure 11 A schematic diagram illustrating another screen-sharing data transmission method provided in some embodiments of this application;

[0076] Figure 12 A flowchart illustrating a method for sharing a clipboard provided in some embodiments of this application;

[0077] Figure 13 A timing diagram of another video conferencing remote control method provided in some embodiments of this application. Detailed Implementation

[0078] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0079] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0080] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0081] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0082] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0083] In this embodiment, the terminal device is a device that inputs programs and data to a computer or receives processing results from the computer via communication facilities. The terminal device is typically located in a convenient location where it can connect to a remote computer using communication facilities, and it mainly consists of a communication interface control device combined with dedicated or selected input / output devices.

[0084] Terminal devices include, but are not limited to, smart TVs, mobile terminals, computers, wearable devices, virtual reality devices, and augmented reality devices. Mobile terminals include, but are not limited to, laptops, tablets, and smartphones.

[0085] Terminal devices include control terminal devices and controlled terminal devices. Control terminal devices generally refer to terminal devices that can control other terminal devices, i.e., terminal devices that send control commands to other terminal devices. Controlled terminal devices generally refer to terminal devices that can be controlled by other terminal devices, i.e., terminal devices that receive control commands sent by other terminal devices. A terminal device can be both a controlled terminal device and a control terminal device.

[0086] Terminal equipment may include a display, communication device, controller, and user input interface.

[0087] In some embodiments, the display includes display function components for presenting an image and driving components for driving the image display. The display is used to receive and display image signals from the controller. For example, the display can be used to display video content, image content, menu control interface components, and user interface (UI) components, etc.

[0088] In some embodiments, a communication device is a component used to communicate with external devices or servers according to various communication protocol types. A terminal device may have multiple communication devices depending on the communication methods it supports. For example, when a terminal device supports wireless network communication, it may have a communication device that includes WiFi functionality. When a terminal device supports Bluetooth connectivity, it needs to have a communication device that includes Bluetooth functionality.

[0089] Communication devices can connect terminal devices to external devices or servers via wireless or wired connections. Wired connections use data cables, interfaces, and other components to link the terminal device to the external device. Wireless connections use wireless signals or wireless networks. Terminal devices can establish direct connections to external devices or indirectly through gateways, routers, or other connection devices.

[0090] In some embodiments, the controller may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller controls the operation of the terminal device and responds to user operations through various software control programs stored in memory. The controller controls the overall operation of the terminal device.

[0091] In some embodiments, the user input interface can be used to receive instructions from user input.

[0092] With the development of information technology, online video conferencing has broken down spatial barriers, improved user work efficiency, and become closely related to users' work and life. Video conferencing software generally supports sharing computer desktops within the meeting, allowing participants to share meeting materials and improve efficiency. However, in some scenarios, verbal communication cannot accurately convey meaning, and much time is spent on minor issues such as determining text placement, slowing down the meeting and wasting user time. To save user time, dedicated remote assistance software can be used to control the shared computer for actual operation; for example, adding remote desktop control functionality to video conferencing.

[0093] To save users' time, dedicated remote assistance software can be used to control the shared computer for actual operation. However, this solution requires both the control end and the controlled end of the meeting to have the same remote assistance software installed in order for other participants to control the shared computer, making the user operation relatively cumbersome.

[0094] In some embodiments, such as Figure 1 As shown, when joining a video conference, after receiving a user's input instruction to control the controlled terminal device, the first control terminal device sends a first control request to the controlled terminal device. Upon receiving the first control request, the controlled terminal device establishes a communication connection with the first control terminal device. After receiving a first control instruction from the user for the controlled terminal device, the first control terminal device sends the first control instruction to the controlled terminal device, causing the controlled terminal device to execute the operation corresponding to the first control instruction. After receiving a user's input instruction to control the controlled terminal device, the second control terminal device sends a second control request to the controlled terminal device. Upon receiving the second control request, the controlled terminal device disconnects the communication connection with the first control terminal device and establishes a communication connection with the second control terminal device. After receiving a second control instruction from the user for the controlled terminal device, the second control terminal device sends the second control instruction to the controlled terminal device, causing the controlled terminal device to execute the operation corresponding to the second control instruction.

[0095] In the above embodiments, only a single control terminal device can remotely control the shared human computer (controlled terminal device) at the same time. If other users also have control needs, the existing remote control must be terminated first, and then other users must be invited to re-enable remote control. The user operation is relatively cumbersome and multiple people cannot control the controlled terminal device at the same time.

[0096] To enable remote control and simplify user operations during video conferencing, embodiments of this application provide a server. The server includes a communication module and a processing module. For example... Figure 2 As shown, the processing module performs the following steps:

[0097] Step S201: Receive conference requests from the first terminal device, the second terminal device, and the third terminal device respectively, and enable the first terminal device, the second terminal device, and the third terminal device to join the same video conference.

[0098] In some embodiments, the first terminal device, the second terminal device, and the third terminal device respectively respond to the launch command of the video conferencing application and display the video conferencing homepage, which includes a create meeting control and a join meeting control.

[0099] Upon receiving confirmation from the user regarding the "Create Meeting" control (i.e., the instruction to create a meeting), the first, second, or third terminal device sends a "Create Meeting" request to the server. The server then creates a video conference based on the request and distributes page data to the first, second, or third terminal device, enabling them to display the corresponding video conference page.

[0100] In some embodiments, the audio call data and / or video call data collected by the camera component of each participating terminal are uploaded to the room corresponding to the video conference and cached. Each terminal retrieves the audio and video call data of the other end according to the identifier of the other end in the video conference in order to obtain the audio data and / or video data of the other end for playback.

[0101] Upon receiving confirmation from the user regarding the "Join Meeting" control, the first, second, or third terminal device displays a "Join Meeting" page, which includes a meeting number input box. After receiving the meeting number entered by the user in the meeting box, the server sends a "Join Meeting" request to the server, including the meeting number. The server then enables the first, second, or third terminal device to join the video conference corresponding to the meeting number and sends page data to the first, second, or third terminal device to display the corresponding video conference page.

[0102] In other embodiments, after receiving confirmation from a user for a video conferencing link on a certain page, the first terminal device, the second terminal device, and the third terminal device can launch the video conferencing application and jump to the video conferencing page corresponding to the video conferencing link.

[0103] The first, second, and third terminal devices can join the same video conference using the same conference ID or the same video conference link.

[0104] Step S202: Receive a remote control request sent by the first terminal device, and send the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device.

[0105] Step S203: In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to access and play. The audio and video data consists of screen images captured by the second terminal device at preset time intervals.

[0106] In some embodiments, after the second terminal device initiates screen sharing, the first terminal device sends a remote control request and sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device.

[0107] The steps for the second terminal device to initiate screen sharing include: responding to a user's input of a screen sharing command, capturing screen images at preset intervals and uploading the screen images as a video stream to the server. The server stores the video stream in a preset location for the first and third terminal devices to access and play. The video conferencing page includes a screen sharing control that can receive confirmation input from the user regarding the screen sharing command.

[0108] For example, such as Figure 3 As shown, the video conferencing page includes participant controls 31 and a toolbar 32. Participant controls 31 can display the participant's corresponding account name, account avatar, and / or image data captured by the image acquisition device of the participant's terminal device. The toolbar 32 includes a mute control 321, a video-on control 322, and a screen-sharing control 323. Upon receiving a confirmation operation on the mute control 321, causing it to change to an off state, the sound acquisition device is activated, and the audio call data captured by the sound acquisition device is sent to the server. Other participants' terminal devices retrieve the audio call data from the server and play the audio call data. Upon receiving a confirmation operation on the mute control 321, causing it to change to an on state, the sound acquisition device is deactivated.

[0109] Upon receiving a confirmation operation to enable the video control 322, causing it to change to the enabled state, the image acquisition device is activated and the video call data acquired by the image acquisition device is sent to the server. Other participants' terminal devices retrieve the video call data from the server and display the corresponding image on the image display area of ​​the participant's control 31 at the video call data sending end. Upon receiving a confirmation operation to enable the video control 322, causing it to change to the disabled state, the image acquisition device is disabled.

[0110] It should be noted that all participants' terminal devices can package the image data captured by the image acquisition device and / or the audio data captured by the sound acquisition device into a call video stream and / or a call audio stream, and transmit them to the server via the Real-time Transport Protocol (RTP). The terminal devices of other participants obtain the call video stream and / or call audio stream from a preset location on the server via RTP, then decode and render them in the window corresponding to the call video stream sender identifier on the video conferencing page, and / or play the audio, so as to understand in real time the images and sounds shared by other participants' terminal devices.

[0111] The screen sharing control 323 is used to trigger whether to share the screen with participants, and can receive user clicks or selections on the screen sharing control 323.

[0112] like Figure 4 As shown, after receiving confirmation of the screen sharing control, the second terminal device captures screen images at preset intervals, packages the screen images into a shared screen video stream, and transmits it to the server via a real-time transmission protocol. The first and third terminal devices obtain the shared screen video stream from the server via the real-time transmission protocol, then decode and render it in the screen sharing rendering window on the video conferencing page, allowing them to view the screen image shared by the second terminal device in real time.

[0113] In some embodiments, after receiving confirmation from the user regarding the screen sharing control 323, the second terminal device can directly display the terminal screen display page, package the terminal screen display page into a shared screen video stream, upload it to the server, and allow the first and third terminal devices to retrieve and play it.

[0114] In other embodiments, after receiving confirmation from the user regarding the screen sharing control 323, the second terminal device can obtain the background page corresponding to the background application running in the background and display the corresponding page control on the video conferencing page. Upon receiving the user's selection of a page control, the second terminal device displays the background page corresponding to that control in the foreground and packages the currently displayed terminal screen page into a shared screen video stream, which is then uploaded to the server for the first and third terminal devices to retrieve and play.

[0115] In some embodiments, when the second terminal device does not share its screen, the first and third terminal devices only need to retrieve the call video stream and / or call audio stream from the server, and display the image corresponding to the call video stream in the display area corresponding to the sender of the call video stream on the video conferencing page, and / or play the audio corresponding to the call audio stream.

[0116] In some embodiments, after the second terminal device shares its screen, the first and third terminal devices, in addition to retrieving the shared screen call video stream and / or call audio stream from the server, also need to retrieve the shared screen video stream. The shared screen video stream is used to display the screen image of the second terminal device in the screen sharing rendering window of the video conferencing page.

[0117] After the first terminal device displays the video conference page via a video conference number or video conference link, it retrieves the shared screen video stream uploaded by the second terminal device from the server and displays a screen sharing rendering window on the video conference page. The rendering window displays the image corresponding to the shared screen video stream, which is the screen image of the second terminal device.

[0118] The rendering window includes a request control control. After receiving confirmation from the user regarding the request control control, the first terminal device sends a remote control request to the server. The remote control request includes the second device identifier of the second terminal device. The server then sends the remote control request to the second terminal device corresponding to the second device identifier. The remote control request may also include the name of the participant corresponding to the first terminal device, which can be used to display the participant who wants to control the device while displaying control prompts on the second terminal device.

[0119] In some embodiments, after receiving a remote control request from the first terminal device, the second terminal device can directly send a connection consent request to the server. The server can then directly establish remote control of the second terminal device by the first terminal device.

[0120] In other embodiments, after receiving a remote control request from the first terminal device, the second terminal device displays an agreement control on the video conferencing page. Upon receiving confirmation of the agreement control, it sends an agreement connection request to the server. The server can then directly establish remote control of the second terminal device from the first terminal device.

[0121] For example, such as Figure 5 As shown, the video conferencing page of the first terminal device includes a rendering window 51 and a request control control 52. Upon receiving confirmation from the user on the request control control 52, a remote control request is sent to the server, and then the server forwards it to the second terminal device. Figure 6 As shown, the prompt page of the second terminal device includes a prompt message 61, an agreement control 62, and a rejection control 63. It can receive the user's confirmation of the agreement control 62 and send an agreement connection request to the server. The server can directly establish remote control of the second terminal device from the first terminal device.

[0122] In some embodiments, after sharing a screen image or specifying a background page with the participating first and third terminal devices, the second terminal device can receive a user input instructing it to switch to a video conferencing page displaying participant information, and display participant controls on the video conferencing page. Upon receiving confirmation of the participant controls, a personnel settings box is displayed, which includes a request to be controlled control control. After receiving confirmation of the request to be controlled control control, a request to be controlled is sent to the first terminal device corresponding to the participant control. Upon receiving the instruction to agree to the control request, the first terminal device can send an agreement message to the server, enabling the server to establish remote control of the second terminal device by the first terminal device.

[0123] In other embodiments, after joining a video conference, the first terminal device can send a remote control request directly to the second terminal device corresponding to the second device identifier without the second terminal device initiating screen sharing, thereby establishing remote control of the second terminal device by the first terminal device.

[0124] After receiving confirmation from the participants' controls on the video conferencing page, the first terminal device displays a participant settings box, which includes a request to be controlled control and a request to control control. Upon receiving confirmation from the request to control control, it sends a remote control request to the server. The remote control request includes the second device identifier of the second terminal device corresponding to the participant's control. The server sends the remote control request to the second terminal device corresponding to the second device identifier. The second terminal device receives the instruction to agree to the control request, sends an agreement message to the server, and determines whether to share its screen. If not, it captures screen images at preset intervals and sends these images as a video stream to the server. Upon receiving the agreement message, the server establishes remote control of the second terminal device from the first terminal device. After receiving the video stream, the server stores it in a preset location for the first and third terminal devices to retrieve and play.

[0125] After receiving confirmation of the request to be controlled by the control, the first terminal device sends a request to be controlled to the server. This request includes the second device identifier of the second terminal device corresponding to the participant control. The server then sends the request to the second terminal device corresponding to the second device identifier. At this point, the first terminal device is the controlled device, and the second terminal device is the controlling device. Upon receiving the instruction to agree to the control request, the second terminal device sends an agreement message to the server. The server establishes remote control of the first terminal device by the second terminal device and notifies the first terminal device. The first terminal device determines whether to share its screen. If not, it captures screen images at preset intervals and sends these images as a video stream to the server, enabling the server to send the video stream to both the second and third terminal devices.

[0126] For example, such as Figure 7 As shown, the personnel settings box 71 of the first terminal device includes a request-to-be-controlled control 711, a request control 712, a shared clipboard control 713, and a move-out-of-meeting control 714. After receiving confirmation of the request-to-be-controlled control 711, a request to be controlled is sent to the server, and the server then sends it to the second terminal device corresponding to the participant control, thus establishing remote control of the first terminal device by the second terminal device. Similarly, after receiving confirmation of the request control 712, a remote control request is sent to the server, and the server then sends it to the second terminal device corresponding to the participant control, thus establishing remote control of the second terminal device by the first terminal device.

[0127] In some embodiments, the server establishes remote control of the second terminal device from the first terminal device via the WebSocket protocol. The server can use the WebSocket protocol for remote control; the first terminal device establishes a long-lived WebSocket connection with the server, and the username of the second terminal device needs to be included in the sent messages. The server then forwards the messages based on the username contained in the messages. However, this method has high overhead and low efficiency. Furthermore, long-lived connections have a much higher priority than real-time transmission protocols. In poor network conditions, such as when video conferencing is choppy, due to the higher priority of long-lived connections, control commands transmitted by the first terminal device via the long-lived connection may be unaffected by the network and transmitted normally to the second terminal device. However, the latest screen display from the second terminal device may not be transmitted to the first terminal device, resulting in invalid control commands or commands that do not match the user's intent.

[0128] In some embodiments, to address the issues of high overhead, low efficiency, and mismatch between control commands and user intent, the server establishes remote control of a second terminal device from a first terminal device via a Stream Control Transmission Protocol (SCTP) tunnel. The server can establish multiple SCTP tunnels between the first terminal device and a virtual video conferencing room within the server, as well as between the second terminal device and that virtual video conferencing room, based on remote control requests. The first terminal device needs to include the device identifier of the second terminal device in the sent message and upload it to the message queue corresponding to the second device identifier in the virtual video conferencing room via the SCTP tunnel. The server reads the message from the message queue corresponding to the second device identifier and sends the message to the second terminal device, thus achieving targeted forwarding to the second terminal device instead of forwarding to a third terminal device. The SCTP tunnel can also transmit keyboard and mouse control messages from the first terminal device to the second terminal device.

[0129] Stream Control Protocol (SCTP) is a protocol that allows the simultaneous transmission of multiple data streams between two ends of a network connection. As a transport layer protocol, SCTP can be understood as similar to TCP (Transport Control Protocol) and UDP (User Datagram Protocol). It provides services somewhat like TCP while also incorporating some advantages of UDP. SCTP is a protocol that provides reliable, efficient, and ordered data transmission.

[0130] It should be noted that in the embodiments of this application, the second terminal device can simultaneously receive remote control requests sent by multiple first terminal devices, that is, multiple first terminal devices can simultaneously establish remote control of the second terminal device without disconnecting the current remote control in order to establish remote control with a new terminal device.

[0131] Step S204: If a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device so that the second terminal device can perform the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

[0132] Keyboard and mouse control messages include keyboard control messages (keyboard events) and mouse control messages (mouse events). The keyboard and mouse event handling flow of a terminal device can be as follows: Figure 8 As shown, upon receiving user input to the mouse and keyboard, a hardware event is generated and stored in the system message queue. This event is then distributed to the message queues of various applications. The applications process each message sequentially, triggering response events such as keyboard input, mouse movement, and mouse clicks.

[0133] In some embodiments, after the first terminal device receives a hardware message sent to the rendering window in the video conferencing application, the window callback function of the rendering window determines whether the hardware message is a keyboard and mouse control message. If the hardware message is a keyboard and mouse control message, it analyzes and transforms the message, and then sends the analyzed and transformed message to the remote control module, which then sends it to the server through a stream control protocol tunnel. The server sends the keyboard and mouse control message only to the second terminal device.

[0134] However, the analysis and processing of keyboard and mouse events depends on the UI implementation of the rendering window. For example, the processing methods differ between applications developed using Microsoft Foundation Classes (MFC) and cross-platform application development frameworks (Qt). Furthermore, the UI implementation of the rendering window processes events sequentially, meaning each piece of hardware information must be processed in turn, leading to low execution efficiency.

[0135] In some embodiments, in order to be independent of the UI implementation of the rendering window and improve execution efficiency, hook programming technology is used to obtain keyboard and mouse control messages sent to the rendering window. Specifically, the method may include: when the rendering window is the focus window, the first terminal device uses keyboard and mouse hook programming to obtain keyboard control messages and sends the keyboard control messages to the server. The focus window is used to represent the window that is interacting with the user.

[0136] When the rendering window is the focused window and the mouse pointer is within the rendering window, use keyboard and mouse hook programming to obtain mouse control messages and send them to the server.

[0137] Hook programming, also known as "hooking" technology, is a computer programming term that refers to various techniques that modify or extend the behavior of operating systems, applications, or other software components by intercepting function calls, message passing, and event passing between software modules. The code that processes the intercepted function calls, events, and messages is called a hook.

[0138] like Figure 9 As shown in the embodiment of this application, a keyboard and mouse hook is added to the rendering window of a video conferencing application. The keyboard and mouse hook takes over all hardware messages sent by the system to the rendering window and determines whether the hardware message is a mouse or keyboard event. If the hardware message is a mouse or keyboard event, the mouse or keyboard event is analyzed and processed or transmitted to the remote control module for analysis and processing. If the hardware message is not a mouse or keyboard event, the hardware message is still transmitted to the rendering window.

[0139] A keyboard event typically includes information such as a virtual key code and key status. The virtual key code indicates which key triggered the event (e.g., X, Enter, Esc), and the key status indicates whether the key is currently pressed or released. By analyzing the keyboard event, the virtual key code and key status are extracted and sent to a second terminal device. The second terminal device then uses this information to execute a keyboard event, thus achieving remote keyboard control.

[0140] A mouse event typically includes information such as cursor position, virtual key code, and button status. The cursor position represents the mouse's coordinates on the screen, the virtual key code indicates the mouse button that triggered the event (e.g., left, right, middle button), and the button status indicates whether the mouse button is currently pressed or released. If the user only moves the mouse, the virtual key code and button status in this mouse event will be empty. To ensure a good viewing experience, rendering is usually performed while maintaining the aspect ratio of the shared screen. In this case, the screen of the controlled terminal device will not fill the entire rendering window. Figure 5As shown. Based on the position coordinates of the rendering window, the mouse position relative to the rendering window can be calculated. Then, based on the coordinates of the actual rendering area, the mouse position relative to the rendered screen can be calculated. Finally, based on the ratio of the rendered screen to the original screen (e.g., the original 1920x1080 screen rendered in a 640x360 area), the mouse position relative to the controlled terminal device's screen can be calculated. The converted cursor position, virtual key codes, and key state information are sent to the second terminal device. The second terminal device uses this information to execute mouse events, thus achieving remote mouse control.

[0141] In some embodiments, after the server establishes remote control of the second terminal device by the first terminal device, a control bar can be displayed on the upper layer of the terminal page of the second terminal device, and at least one participant control corresponding to the login user account of the first terminal device can be displayed on the control bar, which can prompt the user which participants are allowed to control the terminal.

[0142] After receiving the keyboard and mouse control message, the second terminal device can determine the sending first terminal device corresponding to the keyboard and mouse control message, and display it on the control of the participant corresponding to the sending first terminal device to prompt the user which participant is currently controlling the second terminal device.

[0143] For example, such as Figure 10 As shown, when the second terminal device shares its screen, a control bar 101 is displayed on the upper layer of the terminal page. The control bar 101 includes a first participant control 1011 and a second attendee control 1012. Currently, it indicates that the attendee user A, corresponding to the first participant control 1011, is controlling the second terminal device.

[0144] It's worth noting that when hook programming receives multiple keyboard and mouse control messages within a preset time period, it deletes all messages preceding the last one, reducing the number of messages processed and improving execution efficiency. For example, during mouse movement, it can reduce the calculation of mouse position and only retain the last mouse position within the preset time period. Hook programming is highly portable.

[0145] The first terminal device sends keyboard and mouse control messages to the server. The server stores the keyboard and mouse control messages in the keyboard and mouse message queue. At the same time, the server reads the keyboard and mouse control messages in the keyboard and mouse message queue in sequence and sends them to the second terminal device.

[0146] After receiving the keyboard and mouse control message sent to the rendering window, the first terminal device adds the data type, the sender identifier (i.e., the first device identifier of the first terminal device), and the designated receiver identifier (i.e., the second device identifier of the second terminal device) to the keyboard and mouse control message, and then sends it to the server's virtual video conferencing room through the established stream control transmission protocol tunnel. The data type is "keyboard and mouse control message".

[0147] After receiving data from the first terminal device, the server first identifies the data type. If the data type is a keyboard and mouse control message, it identifies the designated receiving end identifier and determines whether the designated receiving end identifier is the device identifier of this virtual video conferencing room. If the designated receiving end identifier is not the device identifier of this virtual video conferencing room, the keyboard and mouse control message is discarded. If the designated receiving end identifier is the device identifier of this virtual video conferencing room, the keyboard and mouse control message is placed in the keyboard and mouse message queue corresponding to the designated receiving end identifier, i.e., the second device identifier of the second terminal device.

[0148] Once a terminal device joins a virtual video conferencing room, its device identifier is stored in the virtual video conferencing room's device list. The identification of a specific receiving end is determined by checking whether the identifier is in the device list.

[0149] Once the server detects that the keyboard and mouse message queue contains keyboard and mouse control messages, it reads these messages and sends them through the established flow control protocol tunnel to the terminal device corresponding to the device identifier of the keyboard and mouse message queue. For example, keyboard and mouse control messages in the message queue of the second device identifier are sent to the second terminal device.

[0150] In some embodiments, when network conditions are poor or multiple first terminal devices send keyboard and mouse control messages simultaneously, many keyboard and mouse control messages may accumulate in the keyboard and mouse message queue and fail to be sent in time. A specific implementation of the server sending keyboard and mouse control messages to the second terminal device may include:

[0151] Read the keyboard and mouse control message located at the head of the keyboard and mouse message queue corresponding to the second device identifier. The keyboard and mouse control messages in the keyboard and mouse message queue are uploaded by the first terminal device.

[0152] If the difference in reception time between the keyboard and mouse control message at the tail of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, the keyboard and mouse control message at the head of the keyboard and mouse message queue will be sent to the second terminal device corresponding to the second device identifier.

[0153] If the time difference between receiving a keyboard and mouse control message at the tail of the keyboard and mouse message queue and the time difference between receiving a keyboard and mouse control message at the head of the keyboard and mouse message queue is greater than a first preset difference, the keyboard and mouse control message at the head of the keyboard and mouse message queue is deleted, and the next keyboard and mouse control message at the head of the keyboard and mouse message queue is read, until the time difference between receiving a keyboard and mouse control message at the tail of the keyboard and mouse message queue and the time difference between receiving a keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, then the keyboard and mouse control message at the head of the keyboard and mouse message queue is sent to the second terminal device corresponding to the second device identifier.

[0154] For example, the first preset time difference is 500ms, and the keyboard and mouse message queue contains 10 keyboard and mouse control messages: Keyboard and Mouse Control Message 1, Keyboard and Mouse Control Message 2, ..., Keyboard and Mouse Control Message 10. If the time difference between receiving Keyboard and Mouse Control Message 1 and Keyboard and Mouse Control Message 10 is greater than 500ms, Keyboard and Mouse Control Message 1 is deleted. Then, it is determined whether the time difference between receiving Keyboard and Mouse Control Message 2 and Keyboard and Mouse Control Message 10 is greater than 500ms. At this time, no new keyboard and mouse control messages are stored in the keyboard and mouse message queue. If the time difference between receiving Keyboard and Mouse Control Message 2 and Keyboard and Mouse Control Message 10 is less than 500ms, Keyboard and Mouse Control Message 2 can be sent to the second terminal device.

[0155] In this embodiment of the application, when there is a network lag or multiple first terminal devices sending keyboard and mouse control messages simultaneously, resulting in too many keyboard and mouse control messages in the keyboard and mouse message queue, the keyboard and mouse control messages at the head of the queue can be deleted and not sent to the second terminal device, thereby avoiding excessive load on the second terminal device in a short period of time when the network is restored.

[0156] In some embodiments, when multiple first terminal devices simultaneously send keyboard and mouse control messages, the keyboard and mouse message queue will contain many keyboard and mouse control messages. A specific implementation of the server sending keyboard and mouse control messages to the second terminal device may include:

[0157] Read the keyboard and mouse control message located at the head of the keyboard and mouse message queue corresponding to the second device identifier. The keyboard and mouse control messages in the keyboard and mouse message queue are uploaded by the first terminal device.

[0158] The keyboard and mouse control message whose reception time difference with the keyboard and mouse control message at the head of the keyboard and mouse queue is less than a second preset difference is identified as the first keyboard and mouse control message.

[0159] The system obtains the first device identifier and the priority corresponding to the first device identifier in the first keyboard and mouse control message, and determines the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determines the first keyboard and mouse control message other than the second keyboard and mouse control message as the third keyboard and mouse control message, and the first device identifier is the device identifier of the first terminal device.

[0160] Delete the third keyboard and mouse control message, and send the second keyboard and mouse control message to the second terminal device corresponding to the second device identifier.

[0161] The first keyboard and mouse control message may or may not include the keyboard and mouse control message at the head of the keyboard and mouse queue. If the first keyboard and mouse control message does not include the keyboard and mouse control message at the head of the queue, the keyboard and mouse control message at the head of the queue is sent to the second terminal device first, and then the second keyboard and mouse control message is sent to the second terminal device.

[0162] For example, the second preset time difference is 100ms. There are 10 keyboard and mouse control messages in the message queue: Keyboard and Mouse Control Message 1, Keyboard and Mouse Control Message 2, ..., Keyboard and Mouse Control Message 10. Keyboard and Mouse Control Message 2, Keyboard and Mouse Control Message 3, and Keyboard and Mouse Control Message 4 have a reception time difference of less than 100ms from Keyboard and Mouse Control Message 1. Their corresponding priorities for sending to the first terminal device are level two, level one, and level three, respectively. Keyboard and Mouse Control Message 1 has a priority of level three. Therefore, Keyboard and Mouse Control Message 1, Keyboard and Mouse Control Message 2, and Keyboard and Mouse Control Message 4 are deleted, and Keyboard and Mouse Control Message 3 is sent to the second terminal device. Then, the process continues to acquire keyboard and mouse control messages with a reception time difference of less than 100ms from the message at the head of the queue...

[0163] The priority of the first keyboard and mouse control message is determined by the first terminal device that sent the message, and the priority of the first terminal device can be selected by the second terminal device on the video conferencing page.

[0164] For example, such as Figure 10 As shown, the priority levels are displayed on the first participant control 1011 and the second attendee control 1012 in the control bar 101. After receiving the confirmation operation of the priority setting control 1013, the priority of the first participant and the second attendee can be changed, that is, the priority of the first terminal device corresponding to the first participant and the second attendee can be changed.

[0165] In some embodiments, when multiple first terminal devices simultaneously send keyboard and mouse control messages, the keyboard and mouse message queue will contain many keyboard and mouse control messages. A specific implementation of the server sending keyboard and mouse control messages to the second terminal device may include:

[0166] The system reads the keyboard and mouse control message at the head of the keyboard and mouse message queue corresponding to the second device identifier, and then determines whether a fourth keyboard and mouse control message exists whose reception time difference with the keyboard and mouse control message at the head of the queue is less than a third preset difference. If no fourth keyboard and mouse control message exists, it sends the fourth keyboard and mouse control message to the second terminal device. If a fourth keyboard and mouse control message exists, it deletes the fifth keyboard and mouse control message and sends the fourth keyboard and mouse control message with the latest reception time to the second terminal device. The fifth keyboard and mouse control message is any keyboard and mouse control message other than the fourth keyboard and mouse control message with the latest reception time.

[0167] For example, the third preset time difference is 50ms, and there are 10 keyboard and mouse control messages in the message queue: Keyboard and Mouse Control Message 1, Keyboard and Mouse Control Message 2, ..., Keyboard and Mouse Control Message 10. If there is no keyboard and mouse control message with a reception time difference of less than 50ms from Keyboard and Mouse Control Message 1, then Keyboard and Mouse Control Message 1 is sent to the second terminal device. If there are three keyboard and mouse control messages with a reception time difference of less than 50ms from Keyboard and Mouse Control Message 1, Keyboard and Mouse Control Message 3, and Keyboard and Mouse Control Message 4 in order of reception time, then Keyboard and Mouse Control Message 1, Keyboard and Mouse Control Message 2, and Keyboard and Mouse Control Message 3 are deleted, and Keyboard and Mouse Control Message 4 is sent to the second terminal device. Then, it continues to check if there is a keyboard and mouse control message with a reception time difference of less than 50ms from the keyboard and mouse control message at the head of the queue, ...

[0168] In some embodiments, such as Figure 11 As shown, the methods of transmitting image data and audio data are similar to... Figure 4 The process is the same and will not be repeated here. The first terminal device sends keyboard and mouse control messages to the server via the stream control protocol, and the server then sends them to the terminal device sharing the screen, i.e., the second terminal device.

[0169] In some embodiments, during the process of sharing the screen on the second terminal device, the first terminal device may need to display specified text or images on the screen of the second terminal device. The user can choose to input the specified text or images into the chat area of ​​instant messaging software or the public chat area of ​​video conferencing, and then paste the specified text or images from the chat area of ​​the second terminal device to the specified area. This operation is cumbersome for the user.

[0170] In some embodiments, clipboard sharing can be implemented to simplify user operations. After receiving confirmation of a clipboard sharing control on the video conferencing page, the first terminal device, the second terminal device, and / or the third terminal device send a clipboard sharing request to the server. The clipboard sharing request includes a device identifier. The server adds the first device identifier of the first terminal device, the second device identifier of the second terminal device, and / or the third device identifier of the third terminal device to the shared clipboard list.

[0171] If the third terminal device has not established a flow control transmission protocol tunnel with the virtual video conferencing room, then after the third device identifier is added to the shared clipboard list, a flow control transmission protocol tunnel between the third terminal device and the virtual video conferencing room needs to be established in order to transmit clipboard content.

[0172] For example, in Figure 7 In the process, upon receiving confirmation of the shared clipboard control 713, the first terminal device sends a request to the server to share the clipboard content. The server then adds the first device identifier of the first terminal device to the shared clipboard list.

[0173] like Figure 12 As shown, one specific implementation of a shared clipboard may include: a first terminal device, a second terminal device, or a third terminal device can determine whether the clipboard content has changed. If the clipboard content has changed, it is determined whether the change was made by a video conferencing application; if not, the data type is added to the clipboard content and sent to the server via a flow control protocol tunnel, with the data type being the clipboard content. After recognizing that the uploaded data type is clipboard content, the server sends the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content. If the change was made by a video conferencing application, the determination of whether the clipboard content has changed continues.

[0174] The clipboard is a contiguous memory space within the system that varies in size depending on the amount of information stored. It's used to temporarily store and exchange information, and different applications running on the system share the same clipboard. Hidden windows can be created on the first, second, and third terminal devices to monitor changes to the clipboard content. When the clipboard content changes, these windows receive a message. If the clipboard content isn't changed by a video conferencing application, it indicates a user action, and the content needs to be extracted, tunneled through a flow control protocol, and sent to the server, which then forwards it to the other terminal devices. If the video conferencing application changed the clipboard content, it means the change was caused by clipboard content transmitted from other terminal devices; therefore, there's no need to send the clipboard content to other terminal devices, and monitoring can continue. Once the first, second, or third terminal device receives the clipboard content forwarded by the server, it writes it into the system clipboard memory space, thus achieving the shared clipboard functionality.

[0175] For example, in Figure 5 In the process of receiving confirmation from the user regarding the shared clipboard control 53, the clipboard content sharing function can be realized among the first, second, and third terminal devices participating in the same video conference.

[0176] When the clipboard content is a file, the file is divided into slices, and the slices are transmitted sequentially through the flow control transmission protocol tunnel. After receiving all the slices, the receiving end performs integrity verification and then combines them into the final file. The file includes images, documents, audio, and video, etc.

[0177] In some embodiments, after receiving confirmation of the shared clipboard control on the video conferencing page, the second terminal device can share clipboard content with the first terminal device that has established remote control, that is, automatically add the first device identifier of the first terminal device to the shared clipboard list. Alternatively, the second terminal device can select participants for sharing the clipboard. If the terminal device of the selected participant has not established a flow control protocol tunnel with the virtual video conferencing room, a flow control protocol tunnel needs to be established first.

[0178] In other embodiments, after receiving confirmation of the shared clipboard control on the video conferencing page, the second terminal device can send a shared clipboard request to the first terminal device with which remote control has been established. After the first terminal device agrees to the shared clipboard request, the first device identifier of the first terminal device is automatically added to the shared clipboard list, thereby enabling the second terminal device to share clipboard content with the first terminal device.

[0179] In some embodiments, after receiving confirmation of the shared clipboard control on the video conferencing page, the first and third terminal devices can share clipboard content with the second terminal device by default. Furthermore, it can be configured that the device identifiers of the first and third terminal devices are added to the shared clipboard list of the second terminal device only after the second terminal device agrees to the sharing request. Where the owner of the shared clipboard list is the second terminal device, the clipboard content uploaded by the device identifier in the shared clipboard list can only be shared with the second terminal device, while the clipboard content uploaded by the second terminal device can be shared with all terminal devices corresponding to the device identifiers in the shared clipboard list.

[0180] In some embodiments, after receiving a confirmation operation for sharing the clipboard control in the video conferencing page, the upper layer of the terminal page of the second terminal device can display a clipboard bar, and display at least one participant control corresponding to the login user account of the first terminal device on the clipboard bar, which can prompt the user which participants control the terminal clipboard to share with.

[0181] After receiving the clipboard content sent by the server, the second terminal device can determine the sending first terminal device corresponding to the clipboard content and display it on the participant control corresponding to the sending first terminal device to prompt the user which participant's terminal device shared the current clipboard content.

[0182] For example, such as Figure 10 As shown, when the second terminal device shares its screen, a clipboard 102 is displayed on top of the terminal page. The clipboard 102 includes a first participant control 1021 and a second attendee control 1022. Currently, attendee user A is instructing the control terminal device to share the clipboard content.

[0183] In some embodiments, the timing diagram of the remote control method in a video conference can be as follows: Figure 13 As shown in the diagram, the first, second, and third terminal devices each send a conference request. Based on these requests, the server allows all three devices to join the same video conference. The first terminal device receives a control command from the first user for the second terminal device, sends a remote control request to the server, and the server forwards the request to the second terminal device. The second terminal device displays a prompt page indicating the first user's control request, including an agreement control. The second terminal device receives confirmation from the second user regarding the agreement control, i.e., an agreement control command, and sends an agreement message to the server. The server establishes remote control of the second terminal device from the first terminal device via an SCTP tunnel. The second terminal device captures screen images at preset intervals and generates a video stream, which is then sent to the server. The server stores the video stream in a preset location and notifies the first and third terminal devices to retrieve the video stream from the preset location and display the corresponding images in the rendering window of the video conference interface.

[0184] The first terminal device uses hook programming to obtain keyboard and mouse control messages, and sends these messages to the server via a stream control protocol tunnel. The server processes the messages and then sends them to the second terminal device. This processing includes deleting some messages when there are many in the message queue. The second terminal device then executes the corresponding operation for each keyboard and mouse control message.

[0185] After receiving clipboard sharing instructions from the first and second users respectively, the first and second terminal devices send sharing requests to the server. The server adds the first device identifier of the first terminal device and the second device identifier of the second terminal device to the clipboard sharing list. The second terminal device checks if the clipboard content has changed. If the clipboard content has changed, it determines whether the change was due to a video conferencing application. If not, it sends the clipboard content to the server via a stream control protocol tunnel, and the server then sends it to the first terminal device. The first terminal device then modifies the clipboard content. After executing the sharing instruction, the first terminal device also performs the check for changes in the clipboard content and subsequent steps to share its clipboard content with the second terminal device.

[0186] This application provides a multi-user remote desktop control function based on video conferencing, which also includes clipboard sharing and file transfer functions. The collaborative operation screen is synchronized to other users in the meeting in real time, and it is suitable for online office, teaching and other application scenarios.

[0187] Some embodiments of this application also provide a computer-readable storage medium that can store a program. When the computer storage medium is configured in a display device or server, the program, when executed, can include the program steps involved in the video conferencing remote control method in the above embodiments. The computer storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0188] This application provides an electronic device, which includes a processor and a memory for storing processor-executable instructions. The processor is used to read executable instructions from the memory and execute the instructions to implement the video conferencing remote control method described in the above embodiments.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0190] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of the embodiments suitable for specific application considerations.

Claims

1. A server, characterized in that, include: Communication module; The processing module is configured as follows: Upon receiving conference requests from the first terminal device, the second terminal device, and the third terminal device, the first terminal device, the second terminal device, and the third terminal device are enabled to join the same video conference. The system receives at least one remote control request from the first terminal device, the remote control request including a second device identifier of the second terminal device, and sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by at least one first terminal device; In the case of establishing remote control of the second terminal device by at least one first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location, the audio data in the preset location is retrieved, and distributed to at least one first terminal device and the third terminal device for at least one first terminal device and the third terminal device to obtain and play. If at least one keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message at the head of the keyboard and mouse message queue corresponding to the second device identifier is read. The keyboard and mouse control messages in the keyboard and mouse message queue are uploaded by at least one of the first terminal devices. The keyboard and mouse control message whose reception time difference with the keyboard and mouse control message located at the head of the keyboard and mouse message queue is less than a second preset difference is determined as the first keyboard and mouse control message; In response to the fact that the keyboard and mouse message queue includes multiple keyboard and mouse control messages, the system obtains the first device identifier and the priority corresponding to the first device identifier in the first keyboard and mouse control message, and determines the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determines the first keyboard and mouse control messages other than the second keyboard and mouse control messages as the third keyboard and mouse control messages; wherein, the first device identifier is the device identifier of the first terminal device, the number of first device identifiers is equal to the number of first terminal devices, and the first terminal device corresponds one-to-one with the first device identifier; The third keyboard and mouse control message is deleted, and the second keyboard and mouse control message is only distributed to the second terminal device corresponding to the second device identifier based on the message queue push mode, so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device. In response to the inclusion of a keyboard and mouse control message in the keyboard and mouse message queue, the keyboard and mouse control message is sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

2. The server according to claim 1, characterized in that, The processing module is further configured to send the keyboard and mouse control messages to the second terminal device. Read the keyboard and mouse control message located at the head of the keyboard and mouse message queue corresponding to the second device identifier, wherein the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device; If the difference in reception time between the keyboard and mouse control message at the tail of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to a first preset difference, the keyboard and mouse control message at the head of the keyboard and mouse message queue will be sent to the second terminal device corresponding to the second device identifier. If the reception time difference between the keyboard and mouse control message at the tail of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is greater than a first preset difference, the keyboard and mouse control message at the head of the keyboard and mouse message queue is deleted, and the next keyboard and mouse control message at the head of the keyboard and mouse message queue is read, until the reception time difference between the keyboard and mouse control message at the tail of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, then the keyboard and mouse control message at the head of the keyboard and mouse message queue is sent to the second terminal device corresponding to the second device identifier.

3. The server according to claim 1, characterized in that, After the first terminal device, the second terminal device, and the third terminal device join the same video conference, the processing module is further configured to: Receive a request to share clipboard content from the first terminal device, the second terminal device and / or the third terminal device, and add the first device identifier of the first terminal device, the second device identifier of the second terminal device and / or the third device identifier of the third terminal device to the shared clipboard list; Upon receiving clipboard content from the first terminal device, the second terminal device, or the third terminal device, the clipboard content is sent to the target terminal device corresponding to the device identifier in the shared clipboard list to modify the clipboard content of the target terminal device.

4. A first terminal device, characterized in that, include: monitor; Configured to display the user interface; A communication device configured to establish a communication connection with a server; The controller is configured as follows: In response to an instruction to create or join a meeting, a meeting request is sent to the server so that the server enables the first terminal device to join the video conference based on the meeting request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference; In response to an instruction to control the second terminal device, a remote control request is sent to the server, so that the server sends the remote control request to the second terminal device corresponding to the second device identifier, thereby establishing remote control of the second terminal device by the first terminal device, wherein the remote control request includes the second device identifier of the second terminal device; Acquire audio and video data distributed by the server, wherein the audio and video data is retrieved by the server based on a preset location and played in the rendering window; The system obtains the keyboard and mouse control message sent to the rendering window and sends the keyboard and mouse control message to the server, so that the server distributes the target keyboard and mouse control message to the second terminal device based on the message queue push mode, and the second terminal device executes the operation corresponding to the target keyboard and mouse control message; The server sends the target keyboard and mouse control message to the second terminal device in the following manner: Read the keyboard and mouse control message located at the head of the keyboard and mouse message queue corresponding to the second device identifier, wherein the keyboard and mouse control message in the keyboard and mouse message queue is uploaded to the server by at least one of the first terminal devices; wherein the second device identifier is the device identifier of the second terminal device; The keyboard and mouse control message whose reception time difference with the keyboard and mouse control message located at the head of the keyboard and mouse message queue is less than a second preset difference is determined as the first keyboard and mouse control message; In response to the fact that the keyboard and mouse message queue includes multiple keyboard and mouse control messages, the system obtains the first device identifier and the priority corresponding to the first device identifier in the first keyboard and mouse control message, and determines the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determines the first keyboard and mouse control messages other than the second keyboard and mouse control messages as the third keyboard and mouse control messages; wherein, the first device identifier is the device identifier of the first terminal device, the number of first device identifiers is equal to the number of first terminal devices, and the first terminal device corresponds one-to-one with the first device identifier; Delete the third keyboard and mouse control message, use the second keyboard and mouse control message as the target keyboard and mouse control message, and send the target keyboard and mouse control message to the second terminal device; In response to the inclusion of a keyboard and mouse control message in the keyboard and mouse message queue, the keyboard and mouse control message is sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

5. The first terminal device according to claim 4, characterized in that, The keyboard and mouse control messages include keyboard control messages and mouse control messages. The controller retrieves the keyboard and mouse control messages sent to the rendering window and sends the keyboard and mouse control messages to the server, and is further configured to: When the rendering window is the focus window, the keyboard control messages are obtained using keyboard and mouse hook programming, and the keyboard control messages are sent to the server. The focus window is used to represent the window that is interacting with the user. When the rendering window is the focus window and the mouse pointer is located within the rendering window, the mouse control message is obtained using keyboard and mouse hook programming, and the mouse control message is sent to the server.

6. The first terminal device according to claim 4, characterized in that, After the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to: In response to the instruction to share the clipboard, a request to share clipboard content is sent to the server, so that the server adds the first device identifier of the first terminal device to the shared clipboard list; If a change in clipboard content is detected, and it is not a video conferencing application that changed the clipboard content, the clipboard content is sent to the server. This causes the server to send the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

7. A second terminal device, characterized in that, include: monitor; Configured to display the user interface; A communication device configured to establish a communication connection with a server; The controller is configured as follows: In response to an instruction to create or join a meeting, a meeting request is sent to the server so that the server enables the second terminal device to join the video conference based on the meeting request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference; In response to an instruction to agree to remote control, an agreement control message and audio / video data are sent to the server, so that the server establishes remote control of the second terminal device by at least one first terminal device based on the agreement control message, and stores the audio / video data in a preset location, wherein the audio data is used to be retrieved by the server from the preset location and distributed to at least one first terminal device and the third terminal device for the first terminal device and the third terminal device to acquire and play; Receive target keyboard and mouse control messages sent by the server, wherein the keyboard and mouse control messages are distributed by the server based on a message queue push mode; The operation corresponding to the target keyboard and mouse control message is executed, and the target keyboard and mouse control message is sent by the server to the second terminal device. The server sends the target keyboard and mouse control message to the second terminal device in the following manner: Read the keyboard and mouse control message located at the head of the keyboard and mouse message queue corresponding to the second device identifier, wherein the keyboard and mouse control message in the keyboard and mouse message queue is uploaded to the server by at least one of the first terminal devices; wherein the second device identifier is the device identifier of the second terminal device; The keyboard and mouse control message whose reception time difference with the keyboard and mouse control message located at the head of the keyboard and mouse message queue is less than a second preset difference is determined as the first keyboard and mouse control message; In response to the fact that the keyboard and mouse message queue includes multiple keyboard and mouse control messages, the system obtains the first device identifier and the priority corresponding to the first device identifier in the first keyboard and mouse control message, and determines the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determines the first keyboard and mouse control messages other than the second keyboard and mouse control messages as the third keyboard and mouse control messages; wherein, the first device identifier is the device identifier of the first terminal device, the number of first device identifiers is equal to the number of first terminal devices, and the first terminal device corresponds one-to-one with the first device identifier; Delete the third keyboard and mouse control message, use the second keyboard and mouse control message as the target keyboard and mouse control message, and send the target keyboard and mouse control message to the second terminal device; In response to the inclusion of a keyboard and mouse control message in the keyboard and mouse message queue, the keyboard and mouse control message is sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

8. The second terminal device according to claim 7, characterized in that, After the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to: In response to the instruction to share the clipboard, a request to share clipboard content is sent to the server, so that the server adds the second device identifier of the second terminal device to the shared clipboard list; If a change in clipboard content is detected, and it is not a video conferencing application that changed the clipboard content, the clipboard content is sent to the server. This causes the server to send the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

9. A method for remote control of video conferencing, characterized in that, Applied to servers, including: Upon receiving conference requests from the first terminal device, the second terminal device, and the third terminal device, the first terminal device, the second terminal device, and the third terminal device are enabled to join the same video conference. The system receives at least one remote control request from the first terminal device, the remote control request including a second device identifier of the second terminal device, and sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by at least one first terminal device; In the case of establishing remote control of the second terminal device by at least one first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location, the audio data in the preset location is retrieved, and distributed to at least one first terminal device and the third terminal device for at least one first terminal device and the third terminal device to obtain and play. If at least one keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message at the head of the keyboard and mouse message queue corresponding to the second device identifier is read. The keyboard and mouse control messages in the keyboard and mouse message queue are uploaded by at least one first terminal device. The keyboard and mouse control messages are distributed based on the push mode of the message queue. The keyboard and mouse control message whose reception time difference with the keyboard and mouse control message located at the head of the keyboard and mouse message queue is less than a second preset difference is determined as the first keyboard and mouse control message; In response to the fact that the keyboard and mouse message queue includes multiple keyboard and mouse control messages, the system obtains the first device identifier and the priority corresponding to the first device identifier in the first keyboard and mouse control message, and determines the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determines the first keyboard and mouse control messages other than the second keyboard and mouse control messages as the third keyboard and mouse control messages; wherein, the first device identifier is the device identifier of the first terminal device, the number of first device identifiers is equal to the number of first terminal devices, and the first terminal device corresponds one-to-one with the first device identifier; The third keyboard and mouse control message is deleted, and the second keyboard and mouse control message is only distributed to the second terminal device corresponding to the second device identifier based on the message queue push mode, so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device. In response to the presence of at least one keyboard and mouse control message in the keyboard and mouse message queue, the keyboard and mouse control message is sent to the second terminal device to cause the second terminal device to perform the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

Images